Working on Marmot https://github.com/maxpert/marmot recently added support for vector index. My local benchmarks show pretty decent QPS with less than GB of RSS on DBpedia dataset.
Interesting part is that I started off implementing a research paper for indexing and performance was not good enough. I ended up tuning things up for my own use-case and ended up with good enough replicatable RAG store.
I've got replicas now working with DML proxy. This essentially means I can now have a cluster of primaries, and then spin up replicas on demand and nodes talking to local host will never see their mutation work pretty transparently from readonly-replicas. While PoC works now the snapshot restore is extremely inefficient IMO yet.
A lot of skepticism in comments. Let me remind them doing N loops over local disk with in memory cached pages is absolutely different compared to doing RT over typical VPS network. Having said that there is no silver bullet for dumb code! So let's not conflate the argument the author is trying to make.
Kind of what I've been working on to build tenancy on top of SQLite CDC to make it a simple repayable SQLite for Marmot (https://github.com/maxpert/marmot). I personally think we have a synergy here, would drop by your discord.
I have been asked multiple times on why I chose SQLite and not Turso. I've always responded people that I don't trust an open-source project once it's backed by a VC firm. I've moved away from Redis to Val-Key for same reason, and we have seen the Redis train-wreck in slow-mo. I hope at no point in future Turso ever ends up in that state, but chances are pretty high. At this point the "compatible with SQLite" has become a marketing term IMO, we all know how easy it is to break compatibility here or SQLite to break compatibility.
Ok it's a very long discussion but I will try to keep it brief here (more than happy to chat on Marmot Discord if you wanna go deeper). Honestly I've not done head to head comparison, but if you are asking for guestimated comparison:
- Marmot can give you better easy DDL and better replication guarantees.
- You can control the guarantees around transactions. So if you're doing a quorum based transaction, you are guaranteed that quorum has written those set of rows before returning success. This takes care of those conflicting ID based rows getting overwritten that people would usually ignore. And you should be able to do transactions with proper begin and commit statements.
- Disk write amplification is way lower than what you would see in CRDT. This should usually mean that on a commodity hardware you should see better write throughput. As I mentioned on my local benchmarks I'm getting close to 6K insert ops. This was with a cluster of three nodes. So you can effectively multiply it by three and that is like 18k operations per second. I did not set up a full cluster to actually benchmark these. That requires investing more money and time. And I would be honestly frugal over here since I am spending all my $$$ on my AI bill.
- Reads as you can see, you can read directly from the SQLite database. So you are only bottlenecked by your disk speed. There are no fancy mergers that happen on CRDT level in the middle. It's written once and you're ready to read.
- The hardest part in my opinion that I faced was the auto increment IDs. It is a sad reality but turns out 99% of small to mid-size companies, are using the auto increment for IDs. In all CRDTs, in case of conflict, the LWW (based on one ID or another) happens, and I can guarantee you at some point in time without coordination, if nodes are just emitting those regular incrementing IDs, THEY WILL OVERWRITE each other. That was the exact problem in the first version of Marmot.
- SQLite is single writer database. cr-sqlite writes these delta CRDT rows in a table as well, under high write load you are putting too much pressure on WAL, how do I know? I did this in Marmot v0.x and even v2 started with that and eventually I decided to write logs in a SQLite database as well. Turns out at a high throughput even writing or dumping those logs that change logs that I'm gonna discard away is a bad idea. I eventually move to PebbleDB, with mimalloc based unmanaged memory allocator for serialization/deserialization (yes even that caused slowdowns due to GC). It doesn't stop here each row in CRDT entry is for one every column of table (changed column) + it has index for faster lookup. So there that will bog it down further on many many rows. For context I have tested Marmot on gigs of data not megs.
I do have couple of ideas on how I can really exploit the CRDT stuff, but I don't think I need it right now. I think most of stuff can be taken care of if I can build and MVCC layer on top.
I see your point. Yes the Debezium path requires more configuration and ochestration, litestream makes it very simple. Would be more than happy to provide this out of box in Marmot if enough users request it (Feel free to open ticket).
- DDL gets really tricky in these cases, that's why you see Corrosion has this weird file based system.
- cr-sqlite ain't maintained anymore but I did some benchmarks and if I remember correctly it was as slow as 4x-8x depending upon type of your data & load. Storage bloats by 2x-3x, tombstones accumulate pretty fast as well.
I mean each mutation on every column looks something like:
You point out a question that I spent months thinking about. I personally love Postgres, heck I initially even had a version that will talk postgres wire but with SQLite only syntax. But then somebody pointed me out my WordPress demo, and it was obvious to me that I have to support MySQL protocol, it's just a protocol. Underlaying technology will stay independent from what I choose.
Yes you can. I know wordpress has been trying to get SQLite as first class citizen too. But at this point I am tired of waiting for that pipe-dream. But this will let you do it today. Plus Marmot supports debezium so you can have continuous stream to your NAS.
- Marmot let's you choose consistency level (ONE/QUORUM/FULL) vs MySQL's serializable.
- MySQL requires careful setup of replication, conflict avoidance and monitoring. Fencing split brain and failover is manual in many cases. Marmot even right now is easier to spin up, plus it's leaderless. So you can actually just have your client talk to different nodes (maybe in round robin fashion) to do load distribution.
- Marmot's eventual consistency + anti-entropy will recover brain-splits with you requiring to do anything. MySQL active active requires manual ops.
- Marmot's designed for read-heavy on the edge scenarios. Once I've completed the read-only replica system you can literally bring up or down lambda nodes with Marmot running as sidecar. With replicas being able to select DBs they want (WIP) you should be able to bring up region/org/scenario specific servers with their light weight copies, and writes will be proxied to main server. Applications are virtually unlimited. Since you can directly read SQLite database, think many small vector databases distributed to edge, or regional configurations, or catalogs.
Right now I've started off with full replication of every database in cluster. On my roadmap I have:
- Ability to launch a replica on selected databases from main cluster.
- Ability for replica to only download and replicate changes of select databases (right now all or nothing).
- Ability for replica to proxy DML & DDL into write cluster transparently.
- Custom set of commands for replicas to download and attach/detach databases on the fly.
This will instantly solve 80% of the problems for most of consumption today. I will probably go after on demand page stream and other stuff once core features are done.
Not to mention this solves majority use-cases of lambdas. One can have a persistent main cluster, and then lambda's coming up or going down on demand transparently.
Interesting part is that I started off implementing a research paper for indexing and performance was not good enough. I ended up tuning things up for my own use-case and ended up with good enough replicatable RAG store.