Kafka as database – is it safe?
2 comments
Can you talk more about the use case?
My question would be, why use that setup if you don't have to?
That setup is not something I would reach for unless I really really had to (and when you think you really really have to, check again, then check again, you probably don't really really have to).
How about a RDBMS like Postgres setup with streaming synchronous replication and durability-tweaked parameters?
My question would be, why use that setup if you don't have to?
That setup is not something I would reach for unless I really really had to (and when you think you really really have to, check again, then check again, you probably don't really really have to).
How about a RDBMS like Postgres setup with streaming synchronous replication and durability-tweaked parameters?
In my opinion, using RMDBS as source of truth makes more sense, because these systems were optimised for reliability, durability & data retention.
To be completely honest, some have drunk the proverbial Confluent koolaid, with kafka being the hammer to hit all nails.
I just want to communicate the risks & tradeoffs with taking this approach. Problem is, the theory sort-of holds up, if configured correctly, kafka should be as reliable/durable as PG/SqlServer/MySQL cluster. But I can't find any material on companies experience running kafka this way, I.e. 'we ran this setup for a year in prod, one of our brokers disks got corrupted, but we were able to get back with no data loss, via X,Y,Z.' or 'we couldn't recover and had to accept data loss'.
EDIT: in short, are the claims made by this article reliable: https://www.confluent.io/blog/okay-store-data-apache-kafka/
To be completely honest, some have drunk the proverbial Confluent koolaid, with kafka being the hammer to hit all nails.
I just want to communicate the risks & tradeoffs with taking this approach. Problem is, the theory sort-of holds up, if configured correctly, kafka should be as reliable/durable as PG/SqlServer/MySQL cluster. But I can't find any material on companies experience running kafka this way, I.e. 'we ran this setup for a year in prod, one of our brokers disks got corrupted, but we were able to get back with no data loss, via X,Y,Z.' or 'we couldn't recover and had to accept data loss'.
EDIT: in short, are the claims made by this article reliable: https://www.confluent.io/blog/okay-store-data-apache-kafka/
The setup is:
-kafka is the underlying source-of-truth for the data.
-querying is done on caches (I.e. Redis, ktables) hydrated from kafka
-Kafka configured for durability (infinite topic retention, 3+ replication factor etc)
We are not allowed to lose data in any circumstances
Is the proposed setup nuts?
In theory the replication should guarantee durability & resiliency. Confluent themselves encourage the above pattern.
The only flaws I can think of, are:
- cache blows up and needs to be rehydrated from scratch
-broker disk gets wiped/corrupted -> kafka rebalance, resulting in prolonged downtime if topics contain mountains of data
Has anyone run and battle tested this kind of setup in production? I.e. encountered disk corruption, brokers going down, but still retain data?
My gut feel is, this is a bad setup, because kafka wasn't originally designed for RMDBS levels of durability, but can't point to a concrete reason why this would be the case.