How to Get Stronger Consistency Out of a Datastore (2019)(blog.sentry.io)

18 pointsby camtarn4 lata temu2 comments

blog.sentry.io

How to Get Stronger Consistency Out of a Datastore (2019)

https://blog.sentry.io/2019/09/17/how-to-get-stronger-consistency-out-of-a-datastore/

3 comments

Sequencing in distributed systems is difficult to handle correctly. When faced with such a situation, it’s often a matter of philosophy rather than technical brilliance - are you willing to accept that failure is inevitable even with a neat “waiter” implementation? What happens if the waiter waits forever? Is shaving seconds tangibly benefitting customers (for Sentry it might) and worth the operational overhead?

Framing these problems slightly differently - is 10 second latency close enough to “real time”? - unlocks solutions like an online batch processor that reads out of your DB (if it doesn’t support NOTIFY) on an indexed incremental identifier.

[deleted]4 lata temu

pbalau4 lata temu

I'm not sure about the first part of the solution. Why isn't it enough for the "post process" consumer to wait for events on a "persisted events" topic? And have snuba copy there a message only after it was persisted (and available on all replicas)?

/edit: actually, why does the "post process" consumer even cares about events not persisted yet?