Show HN: SlateDB – An embedded storage engine built on object storage(github.com)
github.com
Show HN: SlateDB – An embedded storage engine built on object storage
https://github.com/slatedb/slatedb
7 comments
I've been working on something super similar, but some of the arch decisions here are curious considering the clear tradeoffs made.
For example, if you have a durability flush interval, what is the WAL for? L0 is the WAL now.
For example, if you have a durability flush interval, what is the WAL for? L0 is the WAL now.
Great question! We started out with the design you described--WAL as L0. But we found that there's a bit of a tension between wanting to have L0 SSTs be larger (and having fewer of them) to reduce metadata size, while we wanted to keep WAL SSTs small and frequent (to reduce async/await latency).
Basically, we wanted to have WAL writes go on the order of milliseconds, but we wanted L0 SSTs to be larger since they actually service reads.
The architecture page has more detail if you haven't found it yet:
https://slatedb.io/docs/architecture
Basically, we wanted to have WAL writes go on the order of milliseconds, but we wanted L0 SSTs to be larger since they actually service reads.
The architecture page has more detail if you haven't found it yet:
https://slatedb.io/docs/architecture
Seems analogous to putting seaweedfs in front of a cloud S3. Then adding a database. We use(unrelated) Zenoh and Loki keeping state on S3 so it would be interesting to have a KV engine.
It's a very very cool idea, but I'm still not clear on the main benefits.
Bottomless storage: yes, but couldn't you theoretically achieve this with plenty of cloud DB services? Amazon Aurora goes up to 128 TB, and once your DB gets to that size, it's likely that you can hire some dedicated engineers to handle more complicated setups.
High durability: yes, but couldn't this be achieves with a "normal" DB that has a read replica using object storage, rather than the entire DB using object storage?
Easy replication: arguably not easier than normal replication, depending on which cloud DB you're considering as an alternative.
Bottomless storage: yes, but couldn't you theoretically achieve this with plenty of cloud DB services? Amazon Aurora goes up to 128 TB, and once your DB gets to that size, it's likely that you can hire some dedicated engineers to handle more complicated setups.
High durability: yes, but couldn't this be achieves with a "normal" DB that has a read replica using object storage, rather than the entire DB using object storage?
Easy replication: arguably not easier than normal replication, depending on which cloud DB you're considering as an alternative.
Also wondering if this would become expensive very fast if it ends up using S3 with a large number of PUT calls
tune the write flush interval down, it'd be very cheap
if the benefits are not obvious to you, then you're not the target user, or you don't understand what kind of person needs this
there's a class of folks who desperately need this. It's the KV equivalent to turbopuffer
there's a class of folks who desperately need this. It's the KV equivalent to turbopuffer
To mitigate high write API costs (PUTs), SlateDB batches writes. Rather than writing every put() call to object storage, MemTables are flushed periodically to object storage as a string-sorted table (SST). The flush interval is configurable.
To mitigate write latency, SlateDB provides an async put method. Clients that prefer strong durability can await on put until the MemTable is flushed to object storage (trading latency for durability). Clients that prefer lower latency can simply ignore the future returned by put.
To mitigate read latency and read API costs (GETs), SlateDB will use standard LSM-tree caching techniques: in-memory block caches, compression, bloom filters, and local SST disk caches.