HackerTrans
TopNewTrendsCommentsPastAskShowJobs

ethanseal

no profile record

Submissions

Subtypes and status-dependent data: pure relational approach

minimalmodeling.substack.com
2 points·by ethanseal·5 mesi fa·0 comments

Two ways to crack a walnut, per Grothendieck (2025)

shreevatsa.net
54 points·by ethanseal·6 mesi fa·19 comments

Faking Dot Density on a Map

ethanseal.com
5 points·by ethanseal·7 mesi fa·0 comments

Use the Index, Luke – SQL Indexing and Tuning

use-the-index-luke.com
20 points·by ethanseal·10 mesi fa·0 comments

A SQL Heuristic: ORs Are Expensive

ethanseal.com
170 points·by ethanseal·10 mesi fa·79 comments

comments

ethanseal
·9 mesi fa·discuss
Exactly the same one from what I see: https://github.com/ethan-seal/ors_expensive/blob/main/explai...

Given the buffer reads seem close to yours, I believe it's page cache.
ethanseal
·9 mesi fa·discuss
When you say cold cache, did you clear the os page cache as well as the postgres buffercaches? After setup.sql, the cache will be warmish - I get 4ms on the first run. I'm using postgres 17.5

See https://github.com/ethan-seal/ors_expensive/blob/main/benchm... where I use dd to clear the os page cache.

This article by pganalyze talks about it: https://pganalyze.com/blog/5mins-postgres-17-pg-buffercache-...
ethanseal
·10 mesi fa·discuss
Materialized views in Postgres don't update incrementally as the data in the relevant tables updates.^1

In order to keep it up to date, the developer has to tell postgres to refresh the data and postgres will do all the work from scratch.

Incremental Materialized views are _hard_. This^2 article goes through how Materialize does it.

MSSQL does it really well from what I understand. They only have a few restrictions, though I've never used a MSSQL materialized view in production.^3

[1]: https://www.postgresql.org/docs/current/rules-materializedvi... [2]: https://www.scattered-thoughts.net/writing/materialize-decor... [3]: https://learn.microsoft.com/en-us/sql/t-sql/statements/creat...
ethanseal
·10 mesi fa·discuss
GIS is underrated. This is awesome!

Have you looked into speaking with the various SHPOs in each US State/Territory?

I've worked with several of them a fair bit and they have a ton of old maps hidden internally. Especially for small, specific areas of the state, like historical districts.
ethanseal
·10 mesi fa·discuss
I think having a way to build statistics on the join itself would be helpful for this. Similar to how extended statistics^1 can help when column distributions aren't independent of each other.

But this may require some basic materialized views, which postgres doesn't really have.

[1]: https://www.postgresql.org/docs/current/planner-stats.html#P...
ethanseal
·10 mesi fa·discuss
Highly recommend https://use-the-index-luke.com/

It's very readable - I always ask new hires and interns to read it.
ethanseal
·10 mesi fa·discuss
Cool. I'll have to read up on that.
ethanseal
·10 mesi fa·discuss
For sure, there's definitely a lot of cool techniques (and I'm not aware of all of them)! And the first example is very much contrived to show a small example.

I'm not super familiar with the term index merge - this seems to be the term for a BitmapOr/BitmapAnd?

Is there another optimization I'm missing?

The article links to my code for my timings here: https://github.com/ethan-seal/ors_expensive

There is an optimization that went in the new release of PostgreSQL I'm excited about that may affect this - I'm not sure. See https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit...
ethanseal
·10 mesi fa·discuss
Gotcha, I misunderstood your comment. The multiple counts is a definitely very contrived example to demonstrate the overhead of BitmapOr and general risk of sequential scans.
ethanseal
·10 mesi fa·discuss
Absolutely. Though I don't recall seeing multiple sequential scans without a self-join or subquery. A basic filter within a sequential scan/loop is the most naive/simplest way of performing queries like these, so postgres falls back to that. Also, fwiw, BitmapOr is only used with indexes: https://pganalyze.com/docs/explain/other-nodes/bitmap-or.