At huge companies, it's hard to prioritize this because it's hard to pin dollar values to removing legacy code, while it's easier to show how building feature X will earn the company $Y amount of revenue. And because of that, there is also no incentive to do it, you don't get promoted by deleting old code, you get promoted by showing how your effort helped contribute to company revenue. At my previous company (100+ engineers, hundreds of microservices), teams that regularly clean legacy codebase tend to be platform teams (cost centers), while teams that struggle to get these prioritized are product vertical teams (revenue centers).
It wouldn't be possible to do this with LTAP architecture since (I'm assuming) the individual logical changes are not visible. But honestly I've always seen SCD type 2 table as a workaround due to lack of data modeling experience in the source database. If you design your tables correctly, you shouldn't need SCD type 2 downstream.
For example, if you know your user can change emails, and there might be events from another source that is keyed by user email (e.g. marketing-related events), then naturally you will need some sort of email_history table that has historical mapping of user id to email (you probably need it for audit purposes too). Then in this case there is no need to build SCD type 2 table of user from CDC, it's already there.
How does it compare to a full-fledged durable execution platform like DBOS[0], which follows the same philosophy? Looks like River does have workflows, but it's locked behind Pro [1].
What about Multigres[0]? It builds on top of Postgres and adds HA (based on Flexible Paxos[1]), sharding, etc. They're still not production-ready, but I'm highly optimistic they will solve a lot of the problems Postgres have.
For example, with Multigres, you should be able to achieve true zero downtime major version upgrade by simply resharding [2]. With vanilla Postgres + pgBouncer, you can only achieve near-zero downtime (few seconds at most), though it's probably good enough for most use cases.
It is also easier to make your code deterministic with Rust vs with Go, which is incredibly useful if you need to perform deterministic simulation testing + property-based testing. I recently wrote a Postgres-to-Iceberg data mirroring tool [1] in Go, but I ported it to Rust because I wanted the ability perform DST without fighting Go's runtime [2]. But if the domain is not critical that warrants DST, I would still pick Go over Rust any day.
The problem with the us-east-1 outage is that a lot of big companies are there, so even if you try your best not to depend on us-east-1, your third party providers are most likely there. In my previous company, we were completely down during us-east-1 outage because of other dependencies that are beyond our control.
In some cases you have no choice but to retain the data, e.g. due to compliance. But the good thing is it doesn't have to be in Postgres. You can periodically offload data to a lakehouse, then delete it from Postgres. If the table is partitioned, delete should be cheap.
I'm guessing with Neon, since their storage is a lakehouse, you get this for free.
How does this compare to open source Deepnote[0]? We use the cloud version (BYOC) at my previous company to replace self-hosted Jupyter notebooks, and it's pretty great.
I mean that's what the Vega team is doing no? They are building the standard grammar (Vega-Lite), along with an implementation (Vega). And they are already quite established with rich ecosystem, and supports a ton of components[0]. The only thing missing is that it expects a CSV or inline data source. But it's probably not too hard to build an extension that connects to a data warehouse with an SQL query.
You can skip by running git commit --no-verify. I know this because I also hate pre-commit checks, and I will automatically use it when working with any codebase that has one.
Having worked at enterprise with 100+ engineers across multiple teams with complex reporting lines, I definitely agree with this. There's a lot of nuance behind decisions that docs simply can't capture. I mean in theory you can probably write every considerations that led you to make a certain decision, e.g.:
- I'm writing this service even though team X has built the same thing, because my team lead doesn't trust team X since the last time we depended on their service 3 years ago, they had a major downtime that screwed us up big time
- This service is using AWS Lambda simply because I think it's cool, despite the fact that the company has a dedicated team running k8s stack with argocd, argo rollouts, KEDA, etc for the entire company
- Service Y is written in this particular way because it's a service that is shared with another team that came from a company that was acquired, and they wouldn't use it unless we write it this particular way, and making the top execs happy is more important than dealing with a small tech debt (this is probably true)
But no one is going to write these in their RFC. But Sarah knows.
Looks cool, I've been waiting for someone to build this since dbt and SQLMesh acquisition. It would be great to have model versioning and support for ClickHouse SQL.
Slightly off-topic but Datalog related: is there a way for me to query data in ClickHouse using Datalog without any ETL? I want to do fraud analysis, and I've been reading about how Datalog is a lot better at these kind of use cases than SQL.
I'm not familiar with Cursor, does it allow the agent to have access to run "curl -X POST" with no approval, i.e. a popup will show up asking you to approve/deny/always approve? AFAIK with Claude Code, this can only happen if you use something like "--dangerously-skip-permissions". I have never used this, I manually approve all commands my agent runs. Pretty insane that people are giving agents to do whatever it wants and trusting the guardrails will work 100% of the time.
I've experienced something like this at work but with data warehouse instead, and it happened multiple times (to be fair, data engineering is still fairly new where I'm from).
One example was an engineer wanted to build an API that accepts large CSV (GBs of credit reports) to extract some data and perform some aggregations. He was in the process of discussing with SREs on the best way to process the huge CSV file without using k8s stateful set, and the solution he was about to build was basically writing to S3 and having a worker asynchronously load and process the CSV in chunks, then finally writing the aggregation to db.
I stepped in and told him he was about to build a data warehouse. :P