HAProxy 3.0 Released(haproxy.com)
haproxy.com
HAProxy 3.0 Released
https://www.haproxy.com/blog/announcing-haproxy-3-0
9 comments
I’m intrigued by the support for CBOR log output: I haven’t seen CBOR support built into any other widely-used infrastructure software before. Is HAProxy unusual in this regard, or am I just unaware of CBOR support elsewhere?
I honestly don't know, all I know is that we've had demands from users at very high loads because the logs are more compact and their parsing is more efficient. And once you have JSON output encoding, it's not much work to produce another encoding. Maybe other ones will follow by the way, we'll see.
So HAProxy has a bit of a love/hate relationship in Kubernetes, the problem really being UDP and the mess(?) that is proxying stateless traffic. The Gateway API kind of expecting UDP proxying support doesn't help.
UDP through a proxy is total non-sense. There isn't any single UDP-based service that is cleanly proxyable. Not just one. Most of them rely on the source IP address itself (plus port) or announce it in the protocol, then require either to spoof the source address and reconfigure all servers to route responses through the proxy, or the proxy to mangle the contents (not even always possible). There's no notion or indication of beginning nor end of connection, packet direction, nor number of packets expected in each direction. Sometimes assigned ports on both sides are not even compatible anymore with some protocols. And it's even funnier when some users want to proxy some IPv6 traffic to an IPv4 server and they don't even understand that it's not possible to perform transparent proxying in this case. UDP has always been for light stateless stuff on direct connections, and a proxy is everything but that.
In short, only services that support being NATted by a firewall will be transportable over a proxy (sometimes with some efforts), and will always be better handled using LVS that does that natively, with less overhead, less resources and less configuration. I've yet to see a single valid case for generic UDP proxying.
The only cases that make sense are service proxies (syslog proxy, DNS proxy etc), which I'm perfectly fine with. For example our syslog proxy can listen to UDP and forward to UDP, TCP and/or stdout, message by message. But for now I'll continue to firmly oppose to the supporting generic UDP in haproxy and I will continue to tell requesters that their demand is stupid and proves a very poor understanding of networking basics.
In the end there's a nice tradeoff: internally at HAProxyTech some of my coworkers have implemented generic proxying in the enterprise edition for all those whom we don't want to say their demand is stupid, because they're willing to pay to avoid using their brain. They're happy with that (at least I hope so), and we don't have to pollute the core with code trying to plug uncovered areas for people complaining on an issue tracker that something does not work due to their uneducated architectural choices. So in the end everyone wins.
In short, only services that support being NATted by a firewall will be transportable over a proxy (sometimes with some efforts), and will always be better handled using LVS that does that natively, with less overhead, less resources and less configuration. I've yet to see a single valid case for generic UDP proxying.
The only cases that make sense are service proxies (syslog proxy, DNS proxy etc), which I'm perfectly fine with. For example our syslog proxy can listen to UDP and forward to UDP, TCP and/or stdout, message by message. But for now I'll continue to firmly oppose to the supporting generic UDP in haproxy and I will continue to tell requesters that their demand is stupid and proves a very poor understanding of networking basics.
In the end there's a nice tradeoff: internally at HAProxyTech some of my coworkers have implemented generic proxying in the enterprise edition for all those whom we don't want to say their demand is stupid, because they're willing to pay to avoid using their brain. They're happy with that (at least I hope so), and we don't have to pollute the core with code trying to plug uncovered areas for people complaining on an issue tracker that something does not work due to their uneducated architectural choices. So in the end everyone wins.
An example of what seems to be a reasonable usecase for UDP proxy is Quilkin[1] from Google.
Made for hosting game servers on Kubernetes, and supports session auth, routing, monitoring, and various other features, some of them using prefix bits in UDP packages.
Also used for DOS protection without the traffic hitting the server.
Not sure how usable it is but Google actually has a lot of open-source stuff for game hosting on kubernetes with agones + openmatch + quilkin, I am somewhat tempted to play with them at some point.
[1] https://github.com/googleforgames/quilkin
Also used for DOS protection without the traffic hitting the server.
Not sure how usable it is but Google actually has a lot of open-source stuff for game hosting on kubernetes with agones + openmatch + quilkin, I am somewhat tempted to play with them at some point.
[1] https://github.com/googleforgames/quilkin
I've been knee deep into TURN lately, so while it would be a terrible idea, I think you could do something like TURN in 'server mode', where the origins are TURN clients and the proxy is a TURN server.
Then when a packet comes in, you decide which origin to forward it to (by unspecified magic), and then send it on annotated with the source ip and port. Replies from the server would include the desired destination ip and port. TURN can also setup a 'channel' so data packets skip the addressing. The end client facing service ip and port would be implicitly linked to the internal TURN service ip and port.
I think the turn server I've been working with could do this out of the box for a single origin server with specific configuration. With a single origin server, it's not very useful other than proof of concept, of course.
But LVS or some other lower level balancing, where ideally the origins are configured with the service ip and port, and outbound packets don't need munging (DSR) sounds like a much better idea. Any sort of rules based forwarding based on a generic UDP packet seems hopelessly complex because of tracking state on stateless protocols.
Then when a packet comes in, you decide which origin to forward it to (by unspecified magic), and then send it on annotated with the source ip and port. Replies from the server would include the desired destination ip and port. TURN can also setup a 'channel' so data packets skip the addressing. The end client facing service ip and port would be implicitly linked to the internal TURN service ip and port.
I think the turn server I've been working with could do this out of the box for a single origin server with specific configuration. With a single origin server, it's not very useful other than proof of concept, of course.
But LVS or some other lower level balancing, where ideally the origins are configured with the service ip and port, and outbound packets don't need munging (DSR) sounds like a much better idea. Any sort of rules based forwarding based on a generic UDP packet seems hopelessly complex because of tracking state on stateless protocols.
Disclosure: I work at Haproxy.com
It is an interesting topic. We don't see UDP in Kubernetes that often but we do support it in various ways depending on the application. HAProxy Community supports QUIC, which is UDP and Syslog load balancing. HAProxy Enterprise (the paid one) has a UDP module that supports other applications too.
It is an interesting topic. We don't see UDP in Kubernetes that often but we do support it in various ways depending on the application. HAProxy Community supports QUIC, which is UDP and Syslog load balancing. HAProxy Enterprise (the paid one) has a UDP module that supports other applications too.
How is the CONTINUATION frame issue handled? Max number of continuations per stream?
There has never been any CONTINUATION frame issues in the first place. Only a bunch of other implementations did it a strange way resulting in abnormal memory usage, but CONTINUATION frames are handled exactly how the spec meant them to be handled, i.e. concatenated on the fly as they are read. You can even try to send many frames of 1 byte if you want, there's no issue (aside the small overhead of parsing extra frames of course).
Regardless, given that it's being used and abused these days to waste server resources (CPU and network bandwidth in this case, memory for some other servers), they're now counted as glitches and you have an option to automatically close a connection above a certain number of glitches, and you can even track them in a stick-table that will allow you to block a source IP and share that info with all other LBs. I.e. the abuse of CONTINUATION (and all other classical H2 abuses since this one was known for 8+ years) can actually be used as a marker for a bad source that leads it to being blocked early.
There are other well-known attack vectors on H2, such as constantly sending SETTINGS frames to change the initial window size, whose cost can be proportional to the number of established streams, and, of course, the fast creation of streams that are immediately reset. All of this is properly counted as glitches and will be handled likewise.
I'm aware of a few large sites which managed to lower their CPU usage with this, by getting rid of this annoying background noise.
Also it's worth keeping in mind that when someone has enough resources to use your CPU this way, the impact would be roughly 100 times higher if they used the same bandwidth to force make you generate TLS handshakes! Thus, H2 attacks, while annoying, are a just a minor detail.
Regardless, given that it's being used and abused these days to waste server resources (CPU and network bandwidth in this case, memory for some other servers), they're now counted as glitches and you have an option to automatically close a connection above a certain number of glitches, and you can even track them in a stick-table that will allow you to block a source IP and share that info with all other LBs. I.e. the abuse of CONTINUATION (and all other classical H2 abuses since this one was known for 8+ years) can actually be used as a marker for a bad source that leads it to being blocked early.
There are other well-known attack vectors on H2, such as constantly sending SETTINGS frames to change the initial window size, whose cost can be proportional to the number of established streams, and, of course, the fast creation of streams that are immediately reset. All of this is properly counted as glitches and will be handled likewise.
I'm aware of a few large sites which managed to lower their CPU usage with this, by getting rid of this annoying background noise.
Also it's worth keeping in mind that when someone has enough resources to use your CPU this way, the impact would be roughly 100 times higher if they used the same bandwidth to force make you generate TLS handshakes! Thus, H2 attacks, while annoying, are a just a minor detail.