100% this. Embracing Serverless for me meant embracing distributed, event driven systems. Serverless and Event Driven architecture go hand in hand. I actually find my productivity is higher than in days of MVC monoliths
> Are all downstream consumers blocked until the DLQ is emptied
No other service should know about the state of another services DLQ
> do all consumers know how to deal with late-arriving events
In my approach in architecting event driven systems, this question is meaningless. For me some of the core tenants (for me) of architecting these systems are
1. Message ordering should not matter
2. Message delays should not matter (no temporal coupling)
3. Message replays should have no side effects
This means that Service B should have no knowledge of what happens in Service A. If something in Service A fails and gets sent to Service A's DLQ, it should have no impact on Service B. This often involves rethinking business requirements and processes to account for workflows that get interrupted - I've yet to find an insurmountable issue.
If however, we are talking about a message broker trying to deliver a message from Service A to Service B, failing and forwarding to Service B's DLQ, then there is a very simple solution. Have your message broker deliver messages to queues. Service B then processes and handles messages from the queue rather than having to immediately handle all incoming messages from the broker. This protects against a lot of failure modes. The only thing(s) that arrive at Service B's DLQ are exceptions thrown internally in Service B
Yes. It's my preferred architecture for any non trivial system. The single biggest downside to it is it's really hard to find people with experience building event driven systems.
There's a bit of a training curve but it's honestly not that hard if people are willing and wanting to learn. You could level up a moderately experienced team in a matter of weeks to be able to work within a well defined event driven microservice architecture. The part that gets tricky and requires experience is carving out the boundaries and messages.
To answer the question about DLQs I think this is a valid critique. I've seen many places just set and forget DLQs and they might as well not have them. For me, I like to start each DLQ with an alert on every message published. Then manually inspect the message, trace the logs and figure out what to do from there. Once I have enough data on failure modes and paths to rectify them, you can start automating DLQ processing. In general though DLQs should not see much traffic outside of a system going down or poison messages hitting your services (broken schema changes from another service)