I needed to back up terabytes of Postgres RDS data to R2 without intermediate local dumps or AWS egress fees, I could not find any ideal solution, so I built this.
DuckLake is great for the lakehouse layer and it's what we use in production. But there's a gap and thats what I'm trying to address with OpenDuck. DuckLake do solve concurrent access at the lakehouse/catalog level and table management.
But the moment you need to fall back to DuckDB's own compute for things DuckLake doesn't support yet, you're back to a single .duckdb file with exclusive locking. One process writes, nobody else reads.
OpenDuck sits at a different layer. It intercepts DuckDB's file I/O and replaces it with a differential storage engine which is append-only layers with snapshot isolation.
The project is still fairly new and not close to production tbh.
I'd actually recommend the simplest option: just write them to Parquet on S3 and query with plain DuckDB. Or you could use Ducklake - https://ducklake.select/
Yes, this is actually one of the core problems OpenDuck's architecture addresses.
The short version: OpenDuck interposes a differential storage layer between DuckDB and the underlying file. DuckDB still sees a normal file (via FUSE on Linux or an in-process FileSystem on any platform), but underneath, writes go to append-only layers and reads are resolved by overlaying those layers newest-first. Sealing a layer creates an immutable snapshot.
This gives you:
Many concurrent readers: each reader opens a snapshot, which is a frozen, consistent view of the database. They don't touch the writer's active layer at all. No locks contended.
One serialized write path: multiple clients can submit writes, but they're ordered through a single gateway/primary rather than racing on the same file. This is intentional: DuckDB's storage engine was never designed for multi-process byte-level writes, and pretending otherwise leads to corruption. Instead, OpenDuck serializes mutations at a higher level and gives you safe concurrency via snapshots.
So for your specific scenario — one process writing while you want to quickly inspect or query the DB from the CLI — you'd be able to open a read-only snapshot mount (or attach with ?snapshot=<uuid>) from a second process and query freely. The writer keeps going, new snapshots appear as checkpoints seal, and readers can pick up the latest snapshot whenever they're ready.
It's not unconstrained multi-writer OLTP (that's an explicit non-goal), but it does solve the "I literally cannot even read the database while another process has it open" problem that makes DuckDB painful in practice.