Software engineer with lots os experience building web apps and getting the most out of Postgres. Nowadays I mostly work with Docker containers, data pipelines (batch and stream), GCP and AWS. I'm a fan of functional programming and strongly typed languages.
I wish LaunchDarkly and other feature flag providers supported protocol buffers to MN define the feature flag schema. It would be a game changer when you have complex variations and end up reaching for untyped JSON.
Shout-out to PowerSync for making it easier to develop fast offline-first mobile apps. It pushes data from Postgres/MySQL/SQL Server subscriptions to a SQLite into the user's mobile device, avoiding the need for many loading animations when the data is there ahead of time. My company is a customer and we recommend it.
Those looking for a more mature solution in this space will probably enjoy SQLc [1]. It was initially developed for Go applications, but over the years it got pluggins for many other languages, including JavaScript/Typescript.
> but seeing how Lustre does HTML templating versus how Phoenix does Heex was my deciding factor to try the latter. My understanding is this is because of a current lack of any macro system.
When I worked with Ruby on Rails I was "addicted" to macros for everything, but after working for a while with statically-typed languages like Elm and Gleam, I see that there are many other ways to solve those same problems. The code can be quite repetitive sometimes, but as long as the compiler ensures everything is in place, it works quite well.
For those of you a similar SPA-type app with more type safety – which is even more useful for writing code with AI – you may want to have a look at the Gleam language and the Lustre web framework [1]. It combines the best of Elixir and Elm. You can mix and match having more logic in the backend or the frontend, as Gleam compiles both to Erlang and to JS.
>There was a time that Google cared deeply about UX. Now, on macOS Google remaps CMD-G in Google Docs to launch some LLM bullshit
That reminds me of a few years ago when Android phones replaced the behavior of "long press sleep/power button" from "shut down" to "ask AI about what's in your screen". Perhaps a manager got promoted somewhere for "raising AI usage" in Android phones.
As far as I can tell, Postgres is not designed with this inclination towards doing lighter work when clients are waiting and piling up maintenance work to do in background. I think the background work it does is mostly running vacuum on tables now and then.
Contrast that with ClickHouse, for example. It operates in a different niche than Postgres (OLAP instead of OLTP) – with their merge tree engine family [1] that does data deduplication in background.
There is one project of modernizing Postgres' storage engine called OrioleDB [2], but I think the company got acquihired by Supabase [3] and maybe the project has not been progressing very quickly since then.
Yes, partitioning will decrease a bit the read performance of queries not correlated with the partition key. That's why you need to periodically merge smaller partitions, so that you can keep the overall partition count bounded.
It is a lot of admin work, but if you really need to scale up Postgres write throughput, I don't see many other options without increasing hardware costs.
I assume you have already picked the low-hanging fruit discussed in the neighboring comments - batch writes, make sure you are using COPY instead of INSERT, tune Postgres parameters adequately and use the fastest disk you can grab for the WAL.
Say, if you know the function is a polynomial of degree N, with N+1 datapoints you can find it – e.g. with Lagrange's polynomial, although the finite precision of computer numbers might make that more complex.
With some extra admin work, you can greatly increase your insert throughput, as long as the table load is comprised mostly of inserts:
1. Partition your table by range of a monotonic ID or timestamp. Notice the primary key will have to contain this column. A BIGINT id column should work fine;
2. Remove all the other indexes from the partitioned table. Add them to all the partitions, except the latest one. This way, the latest one can endure a tough write load, while the other ones work fine for reads;
3. Create an admin routine (perhaps with pg_cron) to create a new partition whenever the newest one is getting close to the limit. When the load moves to the newer partition, add indexes concurrently to the old one;
4. You'll notice the newest partition will the optimized for writes but not reads. You can offset some of that by replacing BTREE secondary indexes with BRIN [1], particularly the one with bloom operator (not to be confused with Postgres Bloom regular indexes [2]). BRIN is a family of indexes more optimized for writes than reads. If the partition is not too large, it shouldn't be too bad to read from it.
5. Later you can merge partitions to avoid having too many of them. Postgres has commands for that, but I think they lock the whole table, so a safer bet is to copy small partitions into a new larger one and swap them manually.
That's why these niche languages need state-of-the-art compilers that enforce invariants more strongly. This way, they can catch most of the subtle bugs the LLM produces, sort of like antibodies.
For those of you using a browser to generate PDFs, the Rust crate you should look into is Typst [1]. Regardless of your application language, you can use their CLI.
It takes some time to get used to their DSL to write PDFs, but nowadays with AI that shouldn't take too long.
I find it fascinating when different people independently arrive at the same architecture when working on a hard problem like this. In my company we built our offline-first apps with PowerSync, which has the same idea of optimistic local changes while waiting for the central server to acknowledge the definitive changes. In PowerSync's case, the sync engine reads Postgres replication logs directly.
For those of you in Brazil, my company jota.ai has built a financial AI-assistant that you can chat with to open a bank account, connect with accounts from other banks, make instant Pix payments with any of them, all of that through WhatsApp right now. We're working hard to release long-running agents soon that can do increasingly complex workflows involving payments and whatnot.
Please let us know if you have suggestions of what complex workflows you would like to build.
My HN handle is also my Github handle.