Olric: Distributed, embeddable data structures in Go(github.com)
github.com
Olric: Distributed, embeddable data structures in Go
https://github.com/buraksezer/olric
21 comments
Relevant discussion when it comes to Redis compatible API: https://news.ycombinator.com/item?id=31796311
> Olric v0.4 and previous versions use Olric Binary Protocol, v0.5 uses Redis serialization protocol for communication and the API was significantly changed. Olric v0.4.x tree is going to receive bug fixes and security updates forever, but I would recommend considering an upgrade to the new version.
Seems like a tall order to promise updates “forever”, particularly for a v0 release.
Seems like a tall order to promise updates “forever”, particularly for a v0 release.
But considering there will be no features added to it, it's probably not a lot of effort to fix a few bugs or security issues here and there when people report them.
I was excited to read its embeddable data structure then found it did Redis serialisation. I was hoping for something close to plasma store given that one isn’t getting much love these days. Would be great to get something that can save binary formats like arrow.
Ultimately Redis just stores bytes, there's no reason you couldn't put arrow formatted data in it (or in Olric, as I understand it).
I do wish plasma wasn't dead/dying though. There is a huge gap in the (extremely crowded) key-value store space for zero-copy shared memory IPC. I assume it doesn't get built because it's not very "cloud native," but such a thing would be undeniably useful.
I do wish plasma wasn't dead/dying though. There is a huge gap in the (extremely crowded) key-value store space for zero-copy shared memory IPC. I assume it doesn't get built because it's not very "cloud native," but such a thing would be undeniably useful.
I know Java and Erlang ecosystems have been big on this at times (Hazelcast, mnesia). When are these the appropriate architectural choice, if ever?
Definitely fun/cool tech, I just don't understand how pragmatic it is.
Definitely fun/cool tech, I just don't understand how pragmatic it is.
In Erlang, mnesia has a couple of things going for it:
One, it stores Erlang terms. The consequence is that you need no data conversion when you want to store data persistently on disk. You just save whatever term you have.
Two, it provides Key/Value lookup at memory read speeds. It also does so across Erlang processes, so you don't have to call into another process to grab data in most cases. This was originally a main design constraint. You need it for low latency operation, and the only way to avoid a disk read is to keep things in memory. Nowadays, since we have fast disks with no seek-time, it matters less. But when Mnesia was built, we didn't have that luxury. Note that many other systems run in separate processes (memcached, redis, ...) which incurs a context switch and data transfer across a OS process boundary. When the K/V store is in the same memory space things are much faster.
Three, it is an excellent tool for rapid prototyping, and early development work. You can easily buy two terabytes of memory nowadays. Hence, scaling memory is relatively easy (save the time it takes to make that data hot in memory). This means your first million customers are easily served from memory, and it'll take a while before you reach a million customers. Mnesia also allows you to plug in other storage backends, so you can eventually move to a more traditional setup later.
Four, mnesia does provide query capability through a query planner. So for the occasional larger query, there are tools built-in which solves your problem in a very small amount of code.
One, it stores Erlang terms. The consequence is that you need no data conversion when you want to store data persistently on disk. You just save whatever term you have.
Two, it provides Key/Value lookup at memory read speeds. It also does so across Erlang processes, so you don't have to call into another process to grab data in most cases. This was originally a main design constraint. You need it for low latency operation, and the only way to avoid a disk read is to keep things in memory. Nowadays, since we have fast disks with no seek-time, it matters less. But when Mnesia was built, we didn't have that luxury. Note that many other systems run in separate processes (memcached, redis, ...) which incurs a context switch and data transfer across a OS process boundary. When the K/V store is in the same memory space things are much faster.
Three, it is an excellent tool for rapid prototyping, and early development work. You can easily buy two terabytes of memory nowadays. Hence, scaling memory is relatively easy (save the time it takes to make that data hot in memory). This means your first million customers are easily served from memory, and it'll take a while before you reach a million customers. Mnesia also allows you to plug in other storage backends, so you can eventually move to a more traditional setup later.
Four, mnesia does provide query capability through a query planner. So for the occasional larger query, there are tools built-in which solves your problem in a very small amount of code.
> You can easily buy two terabytes of memory nowadays. Hence, scaling memory is relatively easy (save the time it takes to make that data hot in memory). This means your first million customers are easily served from memory, and it'll take a while before you reach a million customers.
However, by this time your entire business depends on submillisecond responses from Mnesia in memory and you can't transition to a different database, or even to on-disk Mnesia.
This has happened :)
However, by this time your entire business depends on submillisecond responses from Mnesia in memory and you can't transition to a different database, or even to on-disk Mnesia.
This has happened :)
That and your app depend on 2TB being all in memory NOW so after one error you have to reload it all from disk before serving any request, because you had full fast backup right?
Libraries simplify ops compared to external services, including local development. An embedded database can be used in unit tests with zero effort.
olric. Great name :)
I don’t understand why people need another abstraction layer on top of Redis? What can this do that Redis can’t?
Two to X programes (any language that has redis client) can access the same memory via redis protocol. Without a running redis server. Or i missed something !?
Can run as both embedded and server mode. Essentially a re-imagined redis from a general point of view.
I don't think this is an abstraction layer on top of Redis? It appears to be a reimplementation of Redis in go.