Backpressure (2019)(tedinski.com)
tedinski.com
Backpressure (2019)
https://www.tedinski.com/2019/03/05/backpressure.html
18 comments
Sometimes it is useful to impose different size limits on ‘subqueues’ (each consisting of elements of a certain kind).
Alternatively, autoscale your consumers based on queue size and alert if it's growing indefinitely. Better than losing messages IMO.
That sounds like instructions for turning a cheap, easy to diagnose problem into a very expensive, harder to diagnose problem.
Not to mention the second-order effects: if you pretend to drop exactly 0 % of messages, then people will build things that depend on that condition. When the unexpected happens and that condition is violated, bad becomes worse.
By deliberately designing in the option of dropping a reasonable amount of messages, you're creating a more robust system. (As well as a perfect point for a circuit breaker in an emergency.)
Other than in some very specific domains, dropping messages is not a big deal and the design implications of allowing that are far preferable to designing for a fantasy world in which nothing goes wrong ever.
Not to mention the second-order effects: if you pretend to drop exactly 0 % of messages, then people will build things that depend on that condition. When the unexpected happens and that condition is violated, bad becomes worse.
By deliberately designing in the option of dropping a reasonable amount of messages, you're creating a more robust system. (As well as a perfect point for a circuit breaker in an emergency.)
Other than in some very specific domains, dropping messages is not a big deal and the design implications of allowing that are far preferable to designing for a fantasy world in which nothing goes wrong ever.
While I agree with you, I think the parent's advice of "autoscale your consumers" is also a good advice, if interpreted in certain sense: that is, when you see that your queues start approaching the uncomfortable sizes, start dropping your clients. When they try to reconnect, they will probably get routed to a different, less busy node.
I would absolutely not want to lose log messages. I'd rather have my queue grow infinitely until I solve the ingestion issues than lose messages, _especially_ but not exclusively audit logs.
I don't think logging is _some very specific domain_ but I will concede that this might be due to my own biases because of my role.
I don't think logging is _some very specific domain_ but I will concede that this might be due to my own biases because of my role.
The best strategy is usually to tell the sender that their message could not be delivered and let them decide how to deal with this. Unfortunately, most current message queue systems are not end-to-end (e.g. re-send or modify their message); they usually have an additional intermediate service like HTTP or a WebSocket API to interface with end users; this design makes it difficult to implement this kind of error-routing mechanism, especially at scale when different end users might be connected to different hosts.
Circular dependencies causing deadlocks? That should not be a surprise. Making the request queue unbounded is just trading one problem for another.
Backpressure is definitely important--and I have even been largely sold over the past few years on the final point about queues of any size potentially being harmful--but this article failed to motivate it correctly by framing the problem in terms of a system that had an even more common (and dire) mistake: the existence of any form of automated timeout, which led requests that would eventually have worked to get retried... quite possibly now as a duplicate task; the mere removal of the timeout would probably make this work correctly, as there certainly will be some "back pressure of last resort" somewhere (such a maximum number of pending database transactions due to some maximum number of connections to the server due to some maximum number of socket descriptors or ephemeral ports or something). You simply can't predict how long something might take to happen, so you shouldn't. You shouldn't have a "default" timeout of 10 seconds "because of course it should complete by then". You don't know my networking connection and you don't know your congestion and load: the number of times I have had to fix something broken by some inane timeout--the most recent being a bug I found in the default Google Play Store Python library HTTP client setup that kept timing out my giant APK uploads that were even making progress--is ridiculous: you should have to call together everyone at your company to get consensus and then bring in a second external opinion before you even consider adding a timeout (and like, there are places where timeouts might make sense, but they have to be so so carefully done; a SYN packet handshake timeout is probably one of these places).
IME, people add these timeouts as a hack around two major things:
1) broken software (often their own) that can lock up
2) broken networks that prevent proper dead peer detection, such as by dropping ICMP packets, employing NAT or other stateful firewall rules that drop unknown packets (which happens when a router reboots or otherwise loses or evicts state), etc.
The eternal irony is that these intended robustness hacks invariably end up creating or compounding reliability problems. And people get ridiculously sophisticated with it. For example, rather than add timeouts to each piece of client software, people may an employ a forwarding application proxy (see, e.g., istio on kubernetes) to front requests and implement retries centrally. Now instead of N problems you have 2N or N^2 problems because such architectures invariably end up recapitulating all the sins committed at the IP and TCP level.
1) broken software (often their own) that can lock up
2) broken networks that prevent proper dead peer detection, such as by dropping ICMP packets, employing NAT or other stateful firewall rules that drop unknown packets (which happens when a router reboots or otherwise loses or evicts state), etc.
The eternal irony is that these intended robustness hacks invariably end up creating or compounding reliability problems. And people get ridiculously sophisticated with it. For example, rather than add timeouts to each piece of client software, people may an employ a forwarding application proxy (see, e.g., istio on kubernetes) to front requests and implement retries centrally. Now instead of N problems you have 2N or N^2 problems because such architectures invariably end up recapitulating all the sins committed at the IP and TCP level.
A lot of the ideas mentioned in this article have been incorporated into SocketCluster https://socketcluster.io/ - It's an end-to-end RPC + pub/sub client/server library and it provides a versatile API to manage backpressure.
This point is particularly relevant:
> Exerting backpressure is why we don’t retry except at the ends of a system. If we’re going to fail, we want to propagate that failure information to where it can be used.
This is an excellent observation and why SC is an end-to-end system which extends all the way to the end user (as opposed to Redis, Kafka, RabbitMQ, STOMP, NSQ and MQTT... which are all designed to be used on the backend in a permissioned environment and do not extend to the end user; they require an additional communication and authentication layer for last-mile delivery). If a message fails to be delivered, then the end user's client is the one which should re-submit the message.
The principle of putting most of the control in the hands of end users has been at the core of SC's design since the beginning.
This point is particularly relevant:
> Exerting backpressure is why we don’t retry except at the ends of a system. If we’re going to fail, we want to propagate that failure information to where it can be used.
This is an excellent observation and why SC is an end-to-end system which extends all the way to the end user (as opposed to Redis, Kafka, RabbitMQ, STOMP, NSQ and MQTT... which are all designed to be used on the backend in a permissioned environment and do not extend to the end user; they require an additional communication and authentication layer for last-mile delivery). If a message fails to be delivered, then the end user's client is the one which should re-submit the message.
The principle of putting most of the control in the hands of end users has been at the core of SC's design since the beginning.
> it provides a versatile API to manage backpressure
I think whoever wrote this library has a very different idea of what backpressure is to me:
> For these reasons, SocketCluster exposes a simple API for tracking stream backpressure and it lets you immediately kill streams or groups of streams which are becoming overly congested. SocketCluster lets you measure the backpressure of individual streams within a socket and also the aggregate backpressure of all streams within the socket.
https://socketcluster.io/docs/streams-and-backpressure/
To me, "backpressure" means that if some stage is slow, then it should communicate to upstream stages to slow down, so that queues do not grow. It is an application-level concern in the upstream stages how that slowing-down can be achieved; it might involve coalescing updates, dropping updates that are made irrelevant, reducing sample frequency, etc.
"This got slow so we killed it lol" does not sound useful.
I think whoever wrote this library has a very different idea of what backpressure is to me:
> For these reasons, SocketCluster exposes a simple API for tracking stream backpressure and it lets you immediately kill streams or groups of streams which are becoming overly congested. SocketCluster lets you measure the backpressure of individual streams within a socket and also the aggregate backpressure of all streams within the socket.
https://socketcluster.io/docs/streams-and-backpressure/
To me, "backpressure" means that if some stage is slow, then it should communicate to upstream stages to slow down, so that queues do not grow. It is an application-level concern in the upstream stages how that slowing-down can be achieved; it might involve coalescing updates, dropping updates that are made irrelevant, reducing sample frequency, etc.
"This got slow so we killed it lol" does not sound useful.
>> To me, "backpressure" means that if some stage is slow, then it should communicate to upstream stages to slow down, so that queues do not grow.
Maybe the kinds of streams you're thinking of are more like 'jobs' (e.g. task queues).
Streams in SC are focused around the pub/sub use case. They are end-to-end (user to user). So if new data is being output by a consumer stream faster than the user's front end can process it, what other option is there but to stop consuming (kill) the stream on the receiver's side?
The front end application could process messages in parallel, but if it did, the backpressure would not build up in the first place. Backpressure only builds up if you make the stream await the processing of each message in a series. If there is no explicit await, then the backpressure is always going to be 0.
End-to-end streams are far easier to manage than the backend-only 'task queues' advocated by Rabbit MQ, Kafka, NSQ, etc. Backend streams are unweildy because you can't give feedback to the user on the front end if something goes wrong, the stream has to process the message no matter what... This is because message queues are completely disconnected from front end applications. Not sure why someone would architect a system like this in the first place. That seems Kafkaesque (pun intended). Isn't it better if the system can catch an issue as soon as it happens and give the originator of the action an opportunity to resolve the issue as soon as possible? The originator of an action is usually best placed to figure out how to handle issues related to the action that they've just performed (e.g. retry, show an error...).
Dealing with realtime data end-to-end is far more straight forward.
Maybe the kinds of streams you're thinking of are more like 'jobs' (e.g. task queues).
Streams in SC are focused around the pub/sub use case. They are end-to-end (user to user). So if new data is being output by a consumer stream faster than the user's front end can process it, what other option is there but to stop consuming (kill) the stream on the receiver's side?
The front end application could process messages in parallel, but if it did, the backpressure would not build up in the first place. Backpressure only builds up if you make the stream await the processing of each message in a series. If there is no explicit await, then the backpressure is always going to be 0.
End-to-end streams are far easier to manage than the backend-only 'task queues' advocated by Rabbit MQ, Kafka, NSQ, etc. Backend streams are unweildy because you can't give feedback to the user on the front end if something goes wrong, the stream has to process the message no matter what... This is because message queues are completely disconnected from front end applications. Not sure why someone would architect a system like this in the first place. That seems Kafkaesque (pun intended). Isn't it better if the system can catch an issue as soon as it happens and give the originator of the action an opportunity to resolve the issue as soon as possible? The originator of an action is usually best placed to figure out how to handle issues related to the action that they've just performed (e.g. retry, show an error...).
Dealing with realtime data end-to-end is far more straight forward.
Backpressure is great, but there are so many common components which just don't support it, and, as the article mentions, every link in the chain has to support backpressure for it to work at all.
Reactive streams and the RSocket protocol are an attempt to build a consistent framework for backpressure, but they don't seem to have spread very far from the niche where they started:
https://www.reactive-streams.org/
https://rsocket.io/
APIs for reactive streams even made it into the JDK, but i have never seen anyone use them!
Reactive streams and the RSocket protocol are an attempt to build a consistent framework for backpressure, but they don't seem to have spread very far from the niche where they started:
https://www.reactive-streams.org/
https://rsocket.io/
APIs for reactive streams even made it into the JDK, but i have never seen anyone use them!
That's what Elixir's Broadway framework aims to do fwiw.
tex32(1)
> The first law of queues is: Every queue should have a maximum size. Queues must not grow unbounded.
Unfortunately, unbounded resource is often used as a hammer to try preventing other smaller issues, with https://github.com/grpc/grpc/issues/8603 being a perfect example.