What are you using for distributed block storage on NVMe in production?
1 comments
This is the problem that we're building for at Archil [1]. Would be super curious to hear more about your use case (and specifically why you're not able to use the hyperscaler managed offerings). Feel free to email me at hleath [at] archil.com
[1] https://archil.com
[1] https://archil.com
Ceph RBD is the default but was designed for HDDs. Most benchmarks I've seen show 15–25% flash utilization and painful tail latencies on NVMe. Crimson/SeaStore is promising but still tech preview. LINSTOR/DRBD is solid and battle tested but not NVMe-oF native. Mayastor(OpenEBS) is SPDK-based and interesting but K8s-only, one pool per node, and I've seen users report 1/4 to 1/5 of raw NVMe performance. Vitastor has impressive performance but restrictive licensing (VNPL) and bus factor of 1. Lightbits / Simplyblock are proprietary.
Curious what others are running, especially:
If you're at a GPU neocloud or running large AI training clusters, what's your storage stack for checkpoints and shared data? If you run Ceph on NVMe, have you found it worth the overhead vs. simpler options? Anyone running NVMe-oF (TCP or RDMA) in production for shared block storage? What target are you using?