great question actually. There is a nuance in cassandra regarding to read-repair. If you issue a QUORUM read and the read itself failed (say, because 2/3 nodes are down), but succeeded on 1 node; and later on you read again when all three nodes are up, the failed write will actually be resurrected during read repair due to last write wins conflict resolution.
This is why it is recommended to always retry when a write fails. This way data might arrive at BQ slightly before cassandra itself for a failed write, but eventually they will be consistent.
That said, at WePay cassandra is being used by services which do not have to be strongly consistent at the moment. Not to say it’s not possible, as there are companies that use Cassandra’s light weight transaction feature for CAS operations.
As for consistency between kafka and cassandra, agree that its hard to make it 100% consistent (due to a number of limitations cassandra poses). This is why we tend to be pretty careful about which services should use cassandra, and specify clearly what is the expectation and SLA for cassandra cdc.
The high-level idea is hard-linking the commit log segment file in cdc directory as it is being written + have a separate index file to keep track of offset.
Thanks for the feedback! I totally agree with you that using a streaming approach seems like an overkill. Actually making the pipeline truly real-time is part of our future work for Cassandra CDC (This is mentioned in Part 2 of this blogpost, which is expected to be published next week). In Cassandra 4.0, the CDC feature is improved to allow real-time parsing of commit logs. We are running 3.X in production, however we plan to add 4.0 support in the future.