HackerTrans
TopNewTrendsCommentsPastAskShowJobs

ethanseal

no profile record

Submissions

Subtypes and status-dependent data: pure relational approach

minimalmodeling.substack.com
2 points·by ethanseal·5 months ago·0 comments

Two ways to crack a walnut, per Grothendieck (2025)

shreevatsa.net
54 points·by ethanseal·6 months ago·19 comments

Faking Dot Density on a Map

ethanseal.com
5 points·by ethanseal·7 months ago·0 comments

Use the Index, Luke – SQL Indexing and Tuning

use-the-index-luke.com
20 points·by ethanseal·10 months ago·0 comments

A SQL Heuristic: ORs Are Expensive

ethanseal.com
170 points·by ethanseal·10 months ago·79 comments

comments

ethanseal
·10 months ago·discuss
Exactly the same one from what I see: https://github.com/ethan-seal/ors_expensive/blob/main/explai...

Given the buffer reads seem close to yours, I believe it's page cache.
ethanseal
·10 months ago·discuss
When you say cold cache, did you clear the os page cache as well as the postgres buffercaches? After setup.sql, the cache will be warmish - I get 4ms on the first run. I'm using postgres 17.5

See https://github.com/ethan-seal/ors_expensive/blob/main/benchm... where I use dd to clear the os page cache.

This article by pganalyze talks about it: https://pganalyze.com/blog/5mins-postgres-17-pg-buffercache-...
ethanseal
·10 months ago·discuss
Materialized views in Postgres don't update incrementally as the data in the relevant tables updates.^1

In order to keep it up to date, the developer has to tell postgres to refresh the data and postgres will do all the work from scratch.

Incremental Materialized views are _hard_. This^2 article goes through how Materialize does it.

MSSQL does it really well from what I understand. They only have a few restrictions, though I've never used a MSSQL materialized view in production.^3

[1]: https://www.postgresql.org/docs/current/rules-materializedvi... [2]: https://www.scattered-thoughts.net/writing/materialize-decor... [3]: https://learn.microsoft.com/en-us/sql/t-sql/statements/creat...
ethanseal
·10 months ago·discuss
GIS is underrated. This is awesome!

Have you looked into speaking with the various SHPOs in each US State/Territory?

I've worked with several of them a fair bit and they have a ton of old maps hidden internally. Especially for small, specific areas of the state, like historical districts.
ethanseal
·10 months ago·discuss
I think having a way to build statistics on the join itself would be helpful for this. Similar to how extended statistics^1 can help when column distributions aren't independent of each other.

But this may require some basic materialized views, which postgres doesn't really have.

[1]: https://www.postgresql.org/docs/current/planner-stats.html#P...
ethanseal
·10 months ago·discuss
Highly recommend https://use-the-index-luke.com/

It's very readable - I always ask new hires and interns to read it.
ethanseal
·10 months ago·discuss
Cool. I'll have to read up on that.
ethanseal
·10 months ago·discuss
For sure, there's definitely a lot of cool techniques (and I'm not aware of all of them)! And the first example is very much contrived to show a small example.

I'm not super familiar with the term index merge - this seems to be the term for a BitmapOr/BitmapAnd?

Is there another optimization I'm missing?

The article links to my code for my timings here: https://github.com/ethan-seal/ors_expensive

There is an optimization that went in the new release of PostgreSQL I'm excited about that may affect this - I'm not sure. See https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...
ethanseal
·10 months ago·discuss
Gotcha, I misunderstood your comment. The multiple counts is a definitely very contrived example to demonstrate the overhead of BitmapOr and general risk of sequential scans.
ethanseal
·10 months ago·discuss
Absolutely. Though I don't recall seeing multiple sequential scans without a self-join or subquery. A basic filter within a sequential scan/loop is the most naive/simplest way of performing queries like these, so postgres falls back to that. Also, fwiw, BitmapOr is only used with indexes: https://pganalyze.com/docs/explain/other-nodes/bitmap-or.