HackerLangs
TopNewTrendsCommentsPastAskShowJobs

levkk

2,208 karmajoined قبل 4 سنوات
Sharding Postgres. <https://pgdog.dev>

[email protected]

Submissions

Why we built yet another Postgres connection pooler

pgdog.dev
257 points·by levkk·قبل 4 أيام·64 comments

Why we built yet another Postgres connection pooler

pgdog.dev
3 points·by levkk·قبل 5 أيام·0 comments

PgDog is funded and coming to a database near you

pgdog.dev
548 points·by levkk·الشهر الماضي·261 comments

PostgreSQL and the OOM Killer: Why We Use Strict Memory Overcommit

ubicloud.com
5 points·by levkk·الشهر الماضي·3 comments

Shard Postgres with One Command

pgdog.dev
1 points·by levkk·قبل 3 أشهر·0 comments

Reshard Button

pgdog.dev
2 points·by levkk·قبل 3 أشهر·0 comments

We fixed Postgres connection pooling on serverless with PgDog

circleback.ai
3 points·by levkk·قبل 4 أشهر·0 comments

Show HN: PgDog – Scale Postgres without changing the app

github.com
326 points·by levkk·قبل 5 أشهر·63 comments

PgDog: Connection pooler, load balancer and sharder for PostgreSQL

1 points·by levkk·قبل 5 أشهر·0 comments

Shard Your Database

pgdog.dev
1 points·by levkk·قبل 8 أشهر·0 comments

Shard Your Database

pgdog.dev
3 points·by levkk·قبل 8 أشهر·0 comments

Shard Your Database

pgdog.dev
1 points·by levkk·قبل 8 أشهر·0 comments

Writing Nothing but Docs for a Week

pgdog.dev
2 points·by levkk·قبل 10 أشهر·0 comments

Pub/Sub

docs.pgdog.dev
2 points·by levkk·قبل 10 أشهر·0 comments

Two-Phase Commit

docs.pgdog.dev
3 points·by levkk·قبل 10 أشهر·2 comments

Implement two-phase commit (2pc)

github.com
2 points·by levkk·قبل 10 أشهر·0 comments

comments

levkk
·أمس·discuss
Ouf. I don't know. I don't want to call you out without evidence -- I myself make benchmark claims all the time -- but 50% improvement in OLTP seems suspicious. I get that you used a standard benchmark, and I don't even know what it entails, but my spidey sense is going off. Perhaps, some trade off somewhere that won't make it to prod because it breaks MVCC -- and yes, I saw that it passes regression tests.

Just checking, is fsync on? :) Regression tests don't catch bad IO patterns afaik.

Anyway... sounds like a fun project to work on!
levkk
·قبل 3 أيام·discuss
Pin the client temporarily. I'm assuming that our users know what they are doing. I could be wrong, but so far so good.
levkk
·قبل 3 أيام·discuss
Yes. The plugin architecture supports this currently. That being said, you can use any number of sharding functions we support to do this too.
levkk
·قبل 3 أيام·discuss
Thanks! We try to be very open and explicit about why we chose AGPL. Personally, I like it because it's an extension of GPL, which is a huge reason why I was able to self-teach programming. Just trying to give back.
levkk
·قبل 3 أيام·discuss
Handling cursors is tough - they are very much session-level objects, so even if we, say, pinned your client while it uses that cursor, which would work, that would decrease the performance of connection pooling overall.

So, what's better, breaking your app initially so you know to remove that feature, or letting it work silently while the connection pool isn't 100% in transaction mode? Tough call.
levkk
·قبل 3 أيام·discuss
I don't think so. Too much trouble. Caching is a really hard problem without context, and the context, typically, is app-specific.
levkk
·قبل 3 أيام·discuss
From the strictest CAP theorem definition, that's correct, it is not. But, it's pretty close. I know that in the database world, that's not a good answer, but in practice, it will deliver the vast majority of messages, so maybe that's good enough? We'll see.

We show that it's possible to come close without breaking the DB or the app, but I suspect, it's not quite yet at the level you'd expect from a _durable_ work queue, e.g., Kafka. Not going to replace that one anytime soon.
levkk
·قبل 8 أيام·discuss
Very cool. At first, I thought it was yet another LLM-generated demo, but no: original work of art. Super cool. Transpiling into C does seem easier than LLVM IR, and letting GCC optimize seems like this might actually work.

Excited to see the compiler implementation when it's out -- a lot to learn from.
levkk
·قبل 9 أيام·discuss
I cancelled Claude. The harness is kinda broken, GPT 5.5 is good, and GLM 5.2/Deepseek is good too (with pi, especially). Just not worth the trouble. And I'm not going to pay two subscriptions.
levkk
·قبل 27 يومًا·discuss
CRUD apps don't usually delete in bulk. It's also hard to structure partitions in a way that doesn't wipe out months of important business data -- this is why teams often ETL their DB into Snowflake/ClickHouse and only then drop partitions. That makes it hard for the app to use that data again.

The better approach is either to change your storage engine (e.g. OrioleDB is working on adding the undo log to Pg), or to shard which distributes the vacuum load across multiple servers.
levkk
·قبل 28 يومًا·discuss
You can have multiple. All sharding is config-based, so no real-time synchronization is required.
levkk
·قبل 28 يومًا·discuss
This just checks if the package is installed, not if the installed version is infected. Presumably, if you (me...) haven't run `yay -Syu` in a while (months), we're fine, right? ...Right?

Goddamit, don't make me reinstall Arch, took me a week last time.

Update: archinstall rocks, back in business after like 15min.
levkk
·قبل 30 يومًا·discuss
I didn't realize there are _any_ managed providers of PgDog out there...do tell!
levkk
·قبل 30 يومًا·discuss
Awesome, glad it worked!
levkk
·قبل 30 يومًا·discuss
We also do that! But it's not well documented at the moment.
levkk
·قبل 30 يومًا·discuss
Thanks! Glad we made it relatively easy to migrate!
levkk
·قبل 30 يومًا·discuss
For sure. They should be used for "metadata"-style tables only. High throuput writes should be direct-to-shard.
levkk
·قبل 30 يومًا·discuss
Yeah good callout. We'll add rendezvous soon enough. Until then, being compatible with Postgres partitions has been advantageous -- while we build everything out, people were able to migrate to PgDog for the query routing layer while doing the resharding in Postgres.

Adding a sharding function in our architecture is relatively straightforward. We also support plugins which can control the flow (and direction) for queries, so our users can add their own (and they do!).
levkk
·الشهر الماضي·discuss
Not yet, but actively working on this as we speak.
levkk
·الشهر الماضي·discuss
fwiw, we support cross-shard transactions. They are not magic though, just good old 2pc and a bit of coordination.