Show HN: Diom – Backend primitives (queue, rate limit, etc.) in one Rust binary(github.com)
github.com
Show HN: Diom – Backend primitives (queue, rate limit, etc.) in one Rust binary
https://github.com/svix/diom
0 comments
—
https://github.com/svix/diom
—
Diom is a single Rust binary that replaces Redis, RabbitMQ, Kafka, and a lot of custom code for many common use-cases. It's open-source, self-contained, and has no external runtime dependencies.
While building Svix, we had to reimplement the same backend primitives that everyone reimplements when building a service: robust caching, rate-limiting, idempotency, and more. We also constantly felt the tension between building something custom on top of existing infra (like Redis and Postgres) and adding more dedicated services (like RabbitMQ and Kafka) which we would then need to configure, operate, back up, and maintain. This was even worse for us because Svix is open-source, so additional infrastructure meant additional burden on our customers.
Six months ago we finally decided to build Diom. It's a self-contained open-source service that people can run in their infrastructure with a focus on developer experience and ease of operation. It manages its own storage using fjall (a fast LSM-tree-based storage similar to RocksDB) and therefore requires no external runtime dependencies (no redis/postgres/kafka/etc), and supports running as a single node or a highly-available Raft based cluster.
The goal is for Diom to replace Redis, RabbitMQ, Kafka, and a lot of custom code for most use-cases, by offering pre-built implementations for common backend primitives. It currently supports cache, key-value, idempotency, rate-limiting, queues, and streams. We also plan on adding auth-tokens, distributed settings, feature flags, and other common components; as well as adding more functionality to existing components.
The closest existing project is probably NATS with JetStream, which is also a single self-contained binary with KV, streams, and queues. One of the main differences is that Diom ships higher-level primitives like idempotency, rate limiting, and cache as first-class APIs, rather than something you build on top of KV and streams.
Diom favors ease of operation over scale, so it doesn't match Kafka-level throughput or very high QPS like Redis and Dragonfly. However, most products and developers don't process multiple terabytes and billions of events per second anyway. That said, Diom can still hit high performance for its target use-cases as it implements higher-level primitives rather than basic operations. Additionally, because the primitives live in the same process as the storage, there are fewer network round-trips, which keeps latency low.
It uses HTTP/2 with msgpack as the wire protocol (works fine from browsers), and ships a CLI and SDKs for Python, TypeScript, Rust, Go, and Java, with more on the way.
We have Svix fully ported to Diom and continuously running tests and simulated workloads in one of our staging environments. GA (general availability) is planned for later this year, once we've moved Svix production workloads over.
Repo (MIT licensed): https://github.com/svix/diom
Docs: https://docs.diom.com
Live playground: https://diom.com/playground
I'm excited to finally share Diom, and would love to hear what everyone thinks, if you find it useful, and what other components we should build! Would also love help figuring out what to call this. We currently say "component platform," but I'm not a fan of the name.