I am not sure “so what” equates to “why” in my mind. “Why” tells the cause of the “what”. “So what” explains the reason one should care about the “what”.
Agreed. It’s been fine during covid because I have barely left the house. The times I have it has not performed well battery-wise. Will be looking to bump size next round.
For 1, no example really comes to mind, but i guess there could be cases where a service went from publishing an event with all of its related data, then split into a service where that becomes more expensive to do (like that data is no longer in memory its behind the api of the old service). In some cases you can have very simple services that consume a message, make a few calls to services or databases to hydrate it with more information, then produce that message to another topic that the original consumers could switch to. More commonly though if the data model is making a drastic change where the database is being split and owned by two new services, you will have to get consumers in on the change to make sure everyone knows the semantics of the new changes.
For 2, it completely depends on the source of the trigger. The first event in a chain probably only has enough information to know that it should produce an event, usually as quickly possible, so no additional db or api fetches. So you might get something in the driver status topic that contains {driver_uuid, new_status, old_status}, then based on what downstream consumers may want to do in response to that event, you may need more info, so you may get more entity information in derived topics. Even pure-entity-based messages would have needed a trigger, so in our topics that tail databases, you may have the full row as a message along with the action that occurred like {op: insert, msg: {entity data… }}.
Like others have said, it is just one tool in the tool box.
We used Kafka for event-driven micro services quite a bit at Uber. I lead the team that owned schemas on Kafka there for a while. We just did not accept breaking schema changes within the topic. Same as you would expect from any other public-facing API. We also didnt allow multiplexing the topic with multiple schemas. This wasn’t just because it made my life easier. A large portion of the topics we had went on to become analytical tables in Hive. Breaking changes would break those tables. If you absolutely have to break the schema, make a new topic with a new consumer and phase out the old. This puts a lot of onus on the producer, so we tried to make tools to help. We had a central schema registry with the topics the schemas paired to that showed producers who their consumers were, so if breaking changes absolutely had to happen, they knew who to talk to. In practice though, we never got much pushback on the no-breaking changes rule.
DLQ practices were decided by teams based on need, too many things there to consider to make blanket rules. When in your code did it fail? Is this consumer idempotent? Have we consumed a more recent event that would have over-written this event? Are you paying for some API that your auto-retry churning away in your DLQ is going to cost you a ton of money? Sometimes you may not even want a DLQ, you want a poison pill. That lets you assess what is happening immediately and not have to worry about replays at all.
I hope one of the books you are talking about is Designing Data Intensive Applications, because it is really fantastic. I joke that it is frustrating that so much of what I learned over years on the data team could be written so succinctly in a book.
I think in a recent gates documentary they were planning on building a dozen of these in china for economies of scale. They specifically didnt choose the US because they would not have been able to build as many
Does anyone who has done the previous iteration of this course have any takeaways? I am interested in working through it and would be interested in the last class saw it as time well spent.
Usually you get a 4 year grant that vests monthly or weekly or whatever over the four years. You get one of those every year, so after 4 years you have 4 stacked vesting bonuses, when the oldest runs out, the new one you got that year will take its place.