Show HN: Kafka-compatible streaming on S3 with stateless brokers
2 comments
Thanks for sharing this! I'd love to see a section in the readme comparing this to other s3-backed kafka replacements such as warpstream and bufstream, and why you'd use this over those.
Did that here: https://kafscale.io/comparison/ - mainly k8s native, easy ops, direct decoupling of brokers and data streaming processing, 100% open source. Much cheaper to operate in a cloud.
KafScale is Kafka-compatible streaming, k8s native, where S3 is the source of truth and brokers hold no persistent state. Written in Go, runs on Kubernetes.
Built this after years of operating Kafka and hitting the same walls: broker failures that take hours to recover, partition rebalancing that blocks deploys, disk capacity planning that never ends.
How it works:
- Producers and consumers use standard Kafka clients - Brokers buffer in memory, flush to S3 - etcd stores metadata and consumer group state - Recovery means restarting a pod and reading from S3 - Optional Iceberg processor reads segments directly from S3, bypasses brokers entirely for batch/analytical workloads
What you give up: latency is 400-500ms (S3 round-trip), no transactions, no compacted topics. It's not a 100% replacement.
What you get: brokers are disposable, scaling is just replica count, no disk management, direct access to streamed data over S3 ACL
License: Apache 2.0 GitHub: https://github.com/novatechflow/kafscale