An argument for increasing TCP's initial congestion window (2024)(jeclark.net)
jeclark.net
An argument for increasing TCP's initial congestion window (2024)
https://jeclark.net/articles/tcp-initcwnd/?tag=performance
51 comments
Google has probably sent data to almost every /24 in the last hour. Probably 99% of their egress data goes to destinations where they've sent enough data recently to make a good estimate of bottleneck link speed and queue size.
Having to pick a particular initcwnd to be used for every new TCP connection is an architectural limitation. If they could collect data about each destination and start each TCP connection with a congestion window based on the recent history of transfers from any of their servers to that destination, it could be much better.
It's not a trivial problem to collect bandwidth and buffer size estimates and provide them to every server without delaying the connection, but it would be fun to build such a system.
Having to pick a particular initcwnd to be used for every new TCP connection is an architectural limitation. If they could collect data about each destination and start each TCP connection with a congestion window based on the recent history of transfers from any of their servers to that destination, it could be much better.
It's not a trivial problem to collect bandwidth and buffer size estimates and provide them to every server without delaying the connection, but it would be fun to build such a system.
> It's not a trivial problem to collect bandwidth and buffer size estimates and provide them to every server without delaying the connection, but it would be fun to build such a system.
Tons of fun. Sadly, I don't have access to enough clients to do it anymore.
But napkin architecture. Collect per connection stats and report on connection close (you can do a lot with tcp_info or something something quic). That goes into some big map/reduce whatever data pipeline.
The pipeline ends up with some recommended initial segment limit and a mss suggestion [1], you can probably fit both of those into 8-bits. For ipv4, you could probably just put them into a 16 MB lookup table... shift off the last octet of the address and that's your index into the table. For ipv6 it's trickier, the address space is too big; there's techniques though.
At google scale, they can probably regenerate this data hourly, but weekly would probably be plenty fast.
[1] This is it's own rant (and hopefully it's outdated) but mss on a syn+ack should really start at the lower of what you can accept and what the client told you they can. Instead, consensus has been to always send what you can accept. But path mtu doesn't always work, so a lot of services just send a reduced mss. If you have the infrastructure, it's actually pretty easy to tell if clients can send you full mtu packets or not... with per network data, you could have 4 reasonable options, reflect sender, reflect sender minus 8 (pppoe), reflect sender minus 20 (ipip tunnel), reflect sender minus 28 (ipip tunnel and pppoe). If you have no data for a network, random select.
Tons of fun. Sadly, I don't have access to enough clients to do it anymore.
But napkin architecture. Collect per connection stats and report on connection close (you can do a lot with tcp_info or something something quic). That goes into some big map/reduce whatever data pipeline.
The pipeline ends up with some recommended initial segment limit and a mss suggestion [1], you can probably fit both of those into 8-bits. For ipv4, you could probably just put them into a 16 MB lookup table... shift off the last octet of the address and that's your index into the table. For ipv6 it's trickier, the address space is too big; there's techniques though.
At google scale, they can probably regenerate this data hourly, but weekly would probably be plenty fast.
[1] This is it's own rant (and hopefully it's outdated) but mss on a syn+ack should really start at the lower of what you can accept and what the client told you they can. Instead, consensus has been to always send what you can accept. But path mtu doesn't always work, so a lot of services just send a reduced mss. If you have the infrastructure, it's actually pretty easy to tell if clients can send you full mtu packets or not... with per network data, you could have 4 reasonable options, reflect sender, reflect sender minus 8 (pppoe), reflect sender minus 20 (ipip tunnel), reflect sender minus 28 (ipip tunnel and pppoe). If you have no data for a network, random select.
You're forgetting wireless access networks with varying signal strength (home wifi; phone in the basement).
Wifi does some amount of retransmits of dropped packets at the link level.
That's about local link loss; you at best get buffer bloat from confusing the wired desktop and the wireless laptop that share a 800~1200 Mbit/s DOCSIS downlink.
Or worse, different service tiers from a neighborhood getting bundled via CGNAT; though that's a clear argument for IPv6.
Or worse, different service tiers from a neighborhood getting bundled via CGNAT; though that's a clear argument for IPv6.
Spiders that send too much traffic tend to get blocked, so they are already having to contend with some sort of coordination. Whatever system they’re using for that coordination (server affinity being the simplest) can also propagate the congestion windows.
This!
They also miss the fact that even with an initcwnd of 10 the TLS negotiation isn’t going to consume it so the window starts growing long before content is actually sent.
Plus there’s no discussion of things like packet pacing
They also miss the fact that even with an initcwnd of 10 the TLS negotiation isn’t going to consume it so the window starts growing long before content is actually sent.
Plus there’s no discussion of things like packet pacing
TLS without 0-rtt gets you one round trip of not too many packets, maybe 4 unless your certificates are really large. That helps your initial window for content, but not by a whole lot.
can you please explain how packet pacing factors into this?
I'd start with a read of this https://www.ietf.org/archive/id/draft-welzl-iccrg-pacing-00....
> Google also developed QUIC, which is HTTP over UDP. There’s no longer any congestion window to deal with, so entire messages can be sent at once.
I don't think that's true. QUIC implementations typically use the same congestion control algorithms, including both CUBIC and BBR, at least nominally. The latest RFCs for those discuss use with both TCP and QUIC. Though, perhaps when used with QUIC they have more degrees of freedom to tune things.
I don't think that's true. QUIC implementations typically use the same congestion control algorithms, including both CUBIC and BBR, at least nominally. The latest RFCs for those discuss use with both TCP and QUIC. Though, perhaps when used with QUIC they have more degrees of freedom to tune things.
Also QUIC is not HTTP over UDP. HTTP/3 is HTTP over QUIC. QUIC is bidi streams and best-effort datagrams over UDP.
Vendors are pushing using QUIC and without other networking layers, thus the performance increase. For example, SMB3 over QUIC over the Internet, sans VPN.
Yeah, the article completely lost me at this point.
A somewhat-interesting, though simplistic, discussion of congestion control in TCP.
And then… "but don't worry, QUIC is magic, and it doesn't need any of this.”
No. No it's not. QUIC is not magic and it basically changes nothing in terms of congestion control.
A somewhat-interesting, though simplistic, discussion of congestion control in TCP.
And then… "but don't worry, QUIC is magic, and it doesn't need any of this.”
No. No it's not. QUIC is not magic and it basically changes nothing in terms of congestion control.
L4S[0] also helps a lot with sensing congestion before young connections have yet to suffer their first lost packet...
Basically it sqrt's your actual packet loss rate as far as feedback frequency/density is concerned, without actually even typically having to enact that loss. For example, you could get congestion feedback every 100th packet (as you would with 1% packet loss), during network conditions that would traditionally only have 0.01% packet loss. From [1]:
Unless an AQM node schedules application flows explicitly, the likelihood that the AQM drops a Not-ECT Classic packet (p_C) MUST be roughly proportional to the square of the likelihood that it would have marked it if it had been an L4S packet (p_L). That is:
p_C ~= (p_L / k)2
The constant of proportionality (k) does not have to be standardized for interoperability, but a value of 2 is RECOMMENDED. The term 'likelihood' is used above to allow for marking and dropping to be either probabilistic or deterministic.
[0]: https://www.rfc-editor.org/rfc/rfc9330.html [1]: https://www.rfc-editor.org/rfc/rfc9331#name-the-strength-of-...
Basically it sqrt's your actual packet loss rate as far as feedback frequency/density is concerned, without actually even typically having to enact that loss. For example, you could get congestion feedback every 100th packet (as you would with 1% packet loss), during network conditions that would traditionally only have 0.01% packet loss. From [1]:
Unless an AQM node schedules application flows explicitly, the likelihood that the AQM drops a Not-ECT Classic packet (p_C) MUST be roughly proportional to the square of the likelihood that it would have marked it if it had been an L4S packet (p_L). That is:
p_C ~= (p_L / k)2
The constant of proportionality (k) does not have to be standardized for interoperability, but a value of 2 is RECOMMENDED. The term 'likelihood' is used above to allow for marking and dropping to be either probabilistic or deterministic.
[0]: https://www.rfc-editor.org/rfc/rfc9330.html [1]: https://www.rfc-editor.org/rfc/rfc9331#name-the-strength-of-...
This might be one of those induced demand situations like building more lanes on a highway, which generally makes traffic worse.
The actual limiting factor for how horribly bloated frontend code becomes is that at some point, it becomes so bad that it noticably impacts your business negatively, and you need to improve it.
Increasing the TCP window so it managed at least the basic asset delivery makes sense, but if you need to cold start regularly and you have hundreds of kb of javascript, perhaps fix your stuff?
The actual limiting factor for how horribly bloated frontend code becomes is that at some point, it becomes so bad that it noticably impacts your business negatively, and you need to improve it.
Increasing the TCP window so it managed at least the basic asset delivery makes sense, but if you need to cold start regularly and you have hundreds of kb of javascript, perhaps fix your stuff?
IIRC, all latency-driven congestion control algorithms suffer from violent rtt variance, which happens frequently in wireless networks. How does BBR perform under such circumstances?
Just the day I discovered TCP Congestion Windows and spent the day tweaking and benchmarking between Vegas, Reno, Cubic and TCTP
I've also tweaked and marked benches in Vegas and Reno, to my great shame.
How would this affect DDOS attacks? Would it make you more vulnerable?
Someone needs to reread all of Sally Floyd, et al. researches from International Computer Science Institute of UC Berkeley.
She touched the TCP congestion algorithms of all TCP variants Tahoe, Reno, New Reno, Carson, Vegas, SACK, Westwood, Illinois, Hybla, Compound, HighSpeed, BIC, CUBIC, DCTCP, BBR, BCP, XCP, RCP.
And it all boils down to:
* how much propagation delay are there,
* how long are each packets, and
* whether there are sufficient storage space for “buffer bloats”.
Also, TCP congestion algorithms are neatly pegged as
* reactive (loss-based)
* proactive (delay-based)
* predictive (bandwidth estimation)
https://egbert.net/blog/articles/tcp-evolution.html
also citations:
DUAL (Wang & Crowcroft, 1992) https://www.cs.wustl.edu/~jain/cis788-95/ftp/tcpip_cong/
TCP Veno (Fu & Liew, 2003) https://www.ie.cuhk.edu.hk/wp-content/uploads/fileadmin//sta... https://citeseerx.ist.psu.edu/document?doi=003084a34929d8d2c...
TCP Nice (Venkataramani, Kokku, Dahlin, 2002) https://citeseerx.ist.psu.edu/document?doi=10.1.1.12.8742
TCP-LP (Low Priority TCP, Kuzmanovic & Knightly, 2003) https://www.cs.rice.edu/~ek7/papers/tcplp.pdf
Scalable TCP (Kelly, 2003) https://www.hep.ucl.ac.uk/~ytl/talks/scalable-tcp.pdf
H-TCP (Leith & Shorten, 2004) https://www.hamilton.ie/net/htcp/
FAST TCP (Jin, Wei, Low, 2004/2005) https://netlab.caltech.edu/publications/FAST-TCP.pdf
TCP Africa (King, Baraniuk, Riedi, 2005) https://www.cs.rice.edu/~ied/comp600/PROJECTS/Africa.pdf
TCP Libra (Marfia, Palazzi, Pau, Gerla, Sanadidi, Roccetti, 2007) https://www.cs.ucla.edu/NRL/hpi/tcp-libra/
YeAH-TCP (Yet Another High-speed TCP, Baiocchi, Castellani, Vacirca, 2007) https://dl.acm.org/doi/10.1145/1282380.1282391
TCP-Nice and other background CCAs https://en.wikipedia.org/wiki/TCP_congestion_control
TCP-FIT (Wang, 2016) https://www.sciencedirect.com/science/article/abs/pii/S10848...
She touched the TCP congestion algorithms of all TCP variants Tahoe, Reno, New Reno, Carson, Vegas, SACK, Westwood, Illinois, Hybla, Compound, HighSpeed, BIC, CUBIC, DCTCP, BBR, BCP, XCP, RCP.
And it all boils down to:
* how much propagation delay are there,
* how long are each packets, and
* whether there are sufficient storage space for “buffer bloats”.
Also, TCP congestion algorithms are neatly pegged as
* reactive (loss-based)
* proactive (delay-based)
* predictive (bandwidth estimation)
https://egbert.net/blog/articles/tcp-evolution.html
also citations:
DUAL (Wang & Crowcroft, 1992) https://www.cs.wustl.edu/~jain/cis788-95/ftp/tcpip_cong/
TCP Veno (Fu & Liew, 2003) https://www.ie.cuhk.edu.hk/wp-content/uploads/fileadmin//sta... https://citeseerx.ist.psu.edu/document?doi=003084a34929d8d2c...
TCP Nice (Venkataramani, Kokku, Dahlin, 2002) https://citeseerx.ist.psu.edu/document?doi=10.1.1.12.8742
TCP-LP (Low Priority TCP, Kuzmanovic & Knightly, 2003) https://www.cs.rice.edu/~ek7/papers/tcplp.pdf
Scalable TCP (Kelly, 2003) https://www.hep.ucl.ac.uk/~ytl/talks/scalable-tcp.pdf
H-TCP (Leith & Shorten, 2004) https://www.hamilton.ie/net/htcp/
FAST TCP (Jin, Wei, Low, 2004/2005) https://netlab.caltech.edu/publications/FAST-TCP.pdf
TCP Africa (King, Baraniuk, Riedi, 2005) https://www.cs.rice.edu/~ied/comp600/PROJECTS/Africa.pdf
TCP Libra (Marfia, Palazzi, Pau, Gerla, Sanadidi, Roccetti, 2007) https://www.cs.ucla.edu/NRL/hpi/tcp-libra/
YeAH-TCP (Yet Another High-speed TCP, Baiocchi, Castellani, Vacirca, 2007) https://dl.acm.org/doi/10.1145/1282380.1282391
TCP-Nice and other background CCAs https://en.wikipedia.org/wiki/TCP_congestion_control
TCP-FIT (Wang, 2016) https://www.sciencedirect.com/science/article/abs/pii/S10848...
I seem to remember this coming up a few times over the years and it’s always bad iirc.
I reckon bufferbloat is overhyped as a problem, it mattered to a small set of Internet connectivity in the 2010s and promptly went away as connectivity changed and improved, yet we continue to look at it like it was yesterdays problem.
Bufferbloat is alive and well. Try a t-mobile 5g home gateway. Oof.
I think cable modems have had a ton of improvement, and more fiber in our diet helps, but mobile can be tricky, and wifi is still highly variable (there's promising signs, but I don't know how many people update their access points)
I think cable modems have had a ton of improvement, and more fiber in our diet helps, but mobile can be tricky, and wifi is still highly variable (there's promising signs, but I don't know how many people update their access points)
Is it though, or is it just a scapegoat or a red herring, especially in the case with a wireless medium? That's been my experience with quick claims to bufferbloat, it's usually something else at play. But again ymmv.
There's always something else at play. Bufferbloat hides problems from the systems that can easily solve them. It doesn't cause problems, it makes them worse.
I mean, I did a speed test with t-mobile 5g home internet, download speed was impressive, but so was the difference in ping time during the download vs otherwise.
Sure, wireless is complex, but there were definitely some way too big buffers in the path. Add in some difficulty integrating their box into my network, and it wasn't for me.
Sure, wireless is complex, but there were definitely some way too big buffers in the path. Add in some difficulty integrating their box into my network, and it wasn't for me.
Fair enough, I concede with your assessment, my understanding of bufferbloat (which I have to relearn everytime I look at it) is that the telltale sign is ping to any destination that traverses the uplink exhibits higher latency than usual when you're saturating your download. It's just a tricky thing to test given variability of conditions (and what might be deemed as expected operation) which is why I'm usually hesitant and sceptical, and I don't trust those speedtest websites to gauge it properly.
Every speed test I tried that measures latency under load shows a large difference between fq_codel on and off.
This is very much a problem ISPs have to deal with as big pipes feed small pipes.
What do ISPs have to actually determine these issues outside of sketchy speedtest websites and vague reports or concerns from customers? What about placing probes in the correct places (e.g. in conditions where there is no additional loss or introduced latency between the end user and uplink). Also is this an actual problem that users are really having, or is it perceived because some benchmark / speedtest gave you a score.
There's a lot of issues and variables at play; this isn't a case of "it's always DNS". What tools do ISPs even have at their disposal and how accurate are they and does it uncover the actual problem users are experiencing? This is the real issue that ISPs of all size have to deal with.
There's a lot of issues and variables at play; this isn't a case of "it's always DNS". What tools do ISPs even have at their disposal and how accurate are they and does it uncover the actual problem users are experiencing? This is the real issue that ISPs of all size have to deal with.
You don’t need a speed test website to see this problem. Just run a ping on your own while doing a big download that saturates your connection and bufferbloat will happen unless there is some active queue management to prevent the ping packets from waiting in the queue behind the download packets. This happens anytime that there is a fast/slow transition in the internet and the slow connection cannot keep up. To prevent packet loss, the packets will be buffered, which works well for short spikes, but prolonged activity will result in a noticeable backlog and if buffers are allowed to be sufficiently big, you can get arbitrarily long delays, which are visible in ping times.
The worst that I have ever seen was about 30 seconds when visiting a foreign country where bufferbloat was occurring at peering links. The bufferbloat in peering links is likely visible from western countries if you ping residential IPs in developing countries and monitor the ping times over days. Some parts of the day will have very high ping times while others will not. The high ping times will be the buffer bloat.
In most western countries, the bufferbloat typically occurs at people’s home internet connections. As is the case in all cases of buffer bloat, the solution is to be willing to drop packets when the connection is saturated. If you limit the bandwidth just below what the connection can handle, you can do active queue management to solve the problem.
That said, I suggest you stop posting replies. Your crusade against the idea of buffer bloat makes you look bad to anyone with enough networking knowledge to understand what bufferbloat is. I also strongly suspect I wrote an explanation that you will take zero time to understand and rather than take my advice, you will post another reply to continue your crusade. :/
The worst that I have ever seen was about 30 seconds when visiting a foreign country where bufferbloat was occurring at peering links. The bufferbloat in peering links is likely visible from western countries if you ping residential IPs in developing countries and monitor the ping times over days. Some parts of the day will have very high ping times while others will not. The high ping times will be the buffer bloat.
In most western countries, the bufferbloat typically occurs at people’s home internet connections. As is the case in all cases of buffer bloat, the solution is to be willing to drop packets when the connection is saturated. If you limit the bandwidth just below what the connection can handle, you can do active queue management to solve the problem.
That said, I suggest you stop posting replies. Your crusade against the idea of buffer bloat makes you look bad to anyone with enough networking knowledge to understand what bufferbloat is. I also strongly suspect I wrote an explanation that you will take zero time to understand and rather than take my advice, you will post another reply to continue your crusade. :/
You are 100% correct.
It is not yet a "solved" problem, but 10-15 years have started to make a dent and get better tools to both observe and act on the problem.
This is seen everywhere from the inclusion of CAKE ( https://man7.org/linux/man-pages/man8/tc-cake.8.html ) in some CPE / home router, but the use of fq_codel ( https://man7.org/linux/man-pages/man8/tc-fq_codel.8.html ) in routers along the way.
Other ISPs have to go even farther, because "content" might be 80-120ms away, and the ability to be more aggressive or less aggressive in tuning certain parameters can have a large impact on overall customer Quality of Experience. If there are any LEO hops along the way, problems with TCP and delayed signaling as a byproduct can also make throughout tank while latency spikes.
DPDK and VPP have contributed to a lot of new networking devices to help observe and act on traffic.
Everytime you go from a big pipe to a small pipe (higher data rate to lower data rate) connection you will see this issue at varying levels.
It is not yet a "solved" problem, but 10-15 years have started to make a dent and get better tools to both observe and act on the problem.
This is seen everywhere from the inclusion of CAKE ( https://man7.org/linux/man-pages/man8/tc-cake.8.html ) in some CPE / home router, but the use of fq_codel ( https://man7.org/linux/man-pages/man8/tc-fq_codel.8.html ) in routers along the way.
Other ISPs have to go even farther, because "content" might be 80-120ms away, and the ability to be more aggressive or less aggressive in tuning certain parameters can have a large impact on overall customer Quality of Experience. If there are any LEO hops along the way, problems with TCP and delayed signaling as a byproduct can also make throughout tank while latency spikes.
DPDK and VPP have contributed to a lot of new networking devices to help observe and act on traffic.
Everytime you go from a big pipe to a small pipe (higher data rate to lower data rate) connection you will see this issue at varying levels.
[deleted]
Do you have links to information on what DPDK and VPP are doing in the area of bufferbloat? I have not kept up with them since I cannot use them in my day to day life, but I would love to update myself on the subject.
By the way, when I wrote in another comment that bufferbloat was solved, I meant it in the same way that IPv4 exhaustion is solved by IPv6. We have the ability to deploy solutions that largely fix things, but whether we do is another matter. You are right to say that the past 10-15 years have started to make a dent in the problem. I had not meant to suggest otherwise.
By the way, when I wrote in another comment that bufferbloat was solved, I meant it in the same way that IPv4 exhaustion is solved by IPv6. We have the ability to deploy solutions that largely fix things, but whether we do is another matter. You are right to say that the past 10-15 years have started to make a dent in the problem. I had not meant to suggest otherwise.
Take a look at any of the QoE type vendors out there right now.
https://www.bequant.com/
https://www.bequant.com/
Thanks for the reply and confirming what I had already said earlier in regards to detecting telltale signs of bufferbloat. In case you were aware, a controlled experiment to exhibit bufferbloat doesn't translate to users being materially affected.
The worst that I have ever seen was about 30 seconds when visiting a foreign country where bufferbloat was occurring at peering links. The bufferbloat in peering links is likely visible from western countries if you ping residential IPs in developing countries and monitor the ping times over days. Some parts of the day will have very high ping times while others will not. The high ping times will be the buffer bloat.
Out of curiosity, did you have full observability of these peering links, or is this a hypothesis? I could think of a few scenarios where alternative explanations could explain what you're seeing.
In most western countries, the bufferbloat typically occurs at people’s home internet connections.
Says who? How is this measured? Do we have actual numbers on people experiencing real bufferbloat issues that are affecting their service?
That said, I suggest you stop posting replies. Your crusade against the idea of buffer bloat makes you look bad to anyone with enough networking knowledge to understand what bufferbloat is. I also strongly suspect I wrote an explanation that you will take zero time to understand and rather than take my advice, you will post another reply out of ignorance. :/
Look, I will cordially suggest a more tenable approach: consider disengaging from this thread, your vacuous and vapid post hasn't really brought anything to the table.
Edit: Seems I can't reply to the child comment, so I'll just say, you should've used your own advice and not reply. There's nothing of substance and you're still continuing with your daft misinterpretation of my take. I'll leave it at that.
The worst that I have ever seen was about 30 seconds when visiting a foreign country where bufferbloat was occurring at peering links. The bufferbloat in peering links is likely visible from western countries if you ping residential IPs in developing countries and monitor the ping times over days. Some parts of the day will have very high ping times while others will not. The high ping times will be the buffer bloat.
Out of curiosity, did you have full observability of these peering links, or is this a hypothesis? I could think of a few scenarios where alternative explanations could explain what you're seeing.
In most western countries, the bufferbloat typically occurs at people’s home internet connections.
Says who? How is this measured? Do we have actual numbers on people experiencing real bufferbloat issues that are affecting their service?
That said, I suggest you stop posting replies. Your crusade against the idea of buffer bloat makes you look bad to anyone with enough networking knowledge to understand what bufferbloat is. I also strongly suspect I wrote an explanation that you will take zero time to understand and rather than take my advice, you will post another reply out of ignorance. :/
Look, I will cordially suggest a more tenable approach: consider disengaging from this thread, your vacuous and vapid post hasn't really brought anything to the table.
Edit: Seems I can't reply to the child comment, so I'll just say, you should've used your own advice and not reply. There's nothing of substance and you're still continuing with your daft misinterpretation of my take. I'll leave it at that.
Overanalysis for the sake of denying the existence of whatever you want is cliche. It does not matter how complete the information on a subject is, since you will just post more pointless questions, whose relevance is specious, for the sake of claiming there are non-existent issues in understanding. The last time I saw this used involved a very loquacious guy who denied Darwin’s theory of evolution. It can also be used to claim the world is flat.
I was being generous by advising you to stop posting, since the more you post asinine things, the worse you look. In the past, I have taken the liberty to do amateur psychoanalysis of people who post bizarre things online based on a psychology class I took in college. If I keep responding, it will only be to get you to post more so that I can work out what is wrong with you for my own curiosity. I am probably not the only one thinking this.
I was being generous by advising you to stop posting, since the more you post asinine things, the worse you look. In the past, I have taken the liberty to do amateur psychoanalysis of people who post bizarre things online based on a psychology class I took in college. If I keep responding, it will only be to get you to post more so that I can work out what is wrong with you for my own curiosity. I am probably not the only one thinking this.
I was being generous by advising you to stop posting, since the more you post asinine things, the worse you look. In the past, I have taken the liberty to do amateur psychoanalysis of people who post bizarre things online based on a psychology class I took in college. If I keep responding, it will only be to get you to post more so that I can work out what is wrong with you for my own curiosity. I am probably not the only one thinking this.
Look, let's call this what it is: gatekeeping. Furthermore you deflect and avoid answering a real question. I don't think you actually understood the crux of what I'm saying and instead resorted to ad hominems and gatekeeping, but seeing as it went over your head, I will pose the question: does bufferbloat have more than a marginal affect on the Internet experience of end users in real world conditions (not in a controlled experiment), furthermore does it affect a significant population, as of today in the 2020s as opposed to circa 2010? I'm saying no to both; a good way to gauge whether it is still relevant is to see publications in networking conferences and journals or even discussions by the *NOG, and really it's just not there. I know there's obsession over CoDel etc. and I used to follow the late Dave Taht's evangelising about the issue, but put simply the numbers don't add up - anyways a simpler solution would simply to prioritise ICMP and UDP flows over TCP. Anyways, this is not your imagined crusade against bufferbloat, it's just a pragmatic assessment. I'll leave it at that, rather than deflect and attack, consider applying some emotional intelligence.
Look, let's call this what it is: gatekeeping. Furthermore you deflect and avoid answering a real question. I don't think you actually understood the crux of what I'm saying and instead resorted to ad hominems and gatekeeping, but seeing as it went over your head, I will pose the question: does bufferbloat have more than a marginal affect on the Internet experience of end users in real world conditions (not in a controlled experiment), furthermore does it affect a significant population, as of today in the 2020s as opposed to circa 2010? I'm saying no to both; a good way to gauge whether it is still relevant is to see publications in networking conferences and journals or even discussions by the *NOG, and really it's just not there. I know there's obsession over CoDel etc. and I used to follow the late Dave Taht's evangelising about the issue, but put simply the numbers don't add up - anyways a simpler solution would simply to prioritise ICMP and UDP flows over TCP. Anyways, this is not your imagined crusade against bufferbloat, it's just a pragmatic assessment. I'll leave it at that, rather than deflect and attack, consider applying some emotional intelligence.
The problem of buffer bloat is one of many issues that affect internet users. When I visited China and pings to my VPN in the US jumped from 200ms to 30 seconds depending on the time of day, bufferbloat was severely affecting me. That could only be described as bufferbloat, since the packets were suffering from store and forward overhead to an excessive degree and my pings were able to measure it across times of day.
Historically and likely still in the present day (but not in my household as we use AQM now), whenever one person in a household does a large download, internet latencies shoot up for everyone in the household, which is also bufferbloat. Having to wait hundreds of ms per round trip brings us back to the 56k dialup days and the performance impact on interactive traffic is horrific. It is enough to make VoIP unusable. As others have told you, there can be other issues at the same time, but bufferbloat makes the issues worse. I cannot speak for others on the extent to which they are afflicted by buffer bloat, but adopting AQM had a night and day difference in performance of the internet connection in my house, since I often do big data transfers that previously would slow down basic web browsing for everyone in my house, myself included.
As for your conjecture that extant problems are visible in recent journal publications, journals have a selection bias. The idea that a problem’s existence is indicated by the degree to which people are publishing papers on it in journals is fallacious since the papers need to not just provide something new, but also be interesting to those running the journal (i.e. make them think that the papers would elevate the status if their journal and help them get increased readership, provided that they are not a junk journal that will publish literally anything). On top of that, the work needs to be funded. Bufferbloat, which is largely considered a solved problem and which predominantly affects the less affluent these days, is not something that will get much attention in journals since nobody in academia seeks funding for something that they do not think they can improve or publish.
Finally, I did not use any ad hominem remarks toward you, as my remarks had focused entirely on what you wrote. I did write that any further replies would likely be done to get you to keep talking so I can play my old game of “figure out what is wrong with someone posting bizarre things on the internet”. About 30% of the population is mentally ill and thus when someone is posting bizarre things online, it is often the result of mental illness. Figuring out which mental illness is often the only reason responding to bizarre posts is worthwhile (as it is both an intellectual challenge and a public service). This contradicts your remarks suggesting that there is no point to my replies, to use my words rather than yours. It is not an ad hominem remark to say that I am likely to do this analysis. Posting the results of the analysis would be, but it would be grounded in fact and would likely be done to suggest professional help for X condition, if my amateur analysis identifies a condition that could benefit from professional help. Honestly, I think the world would be a better place if more people who studied psychology (even 1 class like I did) played armchair psychologist when others persist in a pattern of bizarre remarks and refer those who need professional help to trained professionals.
Historically and likely still in the present day (but not in my household as we use AQM now), whenever one person in a household does a large download, internet latencies shoot up for everyone in the household, which is also bufferbloat. Having to wait hundreds of ms per round trip brings us back to the 56k dialup days and the performance impact on interactive traffic is horrific. It is enough to make VoIP unusable. As others have told you, there can be other issues at the same time, but bufferbloat makes the issues worse. I cannot speak for others on the extent to which they are afflicted by buffer bloat, but adopting AQM had a night and day difference in performance of the internet connection in my house, since I often do big data transfers that previously would slow down basic web browsing for everyone in my house, myself included.
As for your conjecture that extant problems are visible in recent journal publications, journals have a selection bias. The idea that a problem’s existence is indicated by the degree to which people are publishing papers on it in journals is fallacious since the papers need to not just provide something new, but also be interesting to those running the journal (i.e. make them think that the papers would elevate the status if their journal and help them get increased readership, provided that they are not a junk journal that will publish literally anything). On top of that, the work needs to be funded. Bufferbloat, which is largely considered a solved problem and which predominantly affects the less affluent these days, is not something that will get much attention in journals since nobody in academia seeks funding for something that they do not think they can improve or publish.
Finally, I did not use any ad hominem remarks toward you, as my remarks had focused entirely on what you wrote. I did write that any further replies would likely be done to get you to keep talking so I can play my old game of “figure out what is wrong with someone posting bizarre things on the internet”. About 30% of the population is mentally ill and thus when someone is posting bizarre things online, it is often the result of mental illness. Figuring out which mental illness is often the only reason responding to bizarre posts is worthwhile (as it is both an intellectual challenge and a public service). This contradicts your remarks suggesting that there is no point to my replies, to use my words rather than yours. It is not an ad hominem remark to say that I am likely to do this analysis. Posting the results of the analysis would be, but it would be grounded in fact and would likely be done to suggest professional help for X condition, if my amateur analysis identifies a condition that could benefit from professional help. Honestly, I think the world would be a better place if more people who studied psychology (even 1 class like I did) played armchair psychologist when others persist in a pattern of bizarre remarks and refer those who need professional help to trained professionals.
> Also is this an actual problem that users are really having, or is it perceived because some benchmark / speedtest gave you a score.
The actual problem is I'm on a voip call and someone starts a big download (steam) and latency and jitter go to hell and the call is unusable. Bufferbloat test confirms that latency dramatically increases under load. Or same call but someone starts uploading something big.
If troublesome buffers are at the last mile connection and the ISP provides a modem/router, adding QoS limiting downloads and uploads to about 90% of the acheived physical connection will avoid the issue. The buffers are still too big, but they won't fill under normal conditions, so it's not a problem. You could still fill the buffers if there's a big flow that doesn't use effective congestion control, or a large enough number of flows so that the minimum send rate is still too much; or when the physical connection rate changes, but good enough. Many ISPs do this, and so you hear a lot less complaining about bufferbloat on say Comcast these days; also, this is an effective best practice, so less need for papers, reports and case studies... it's a matter of getting the practices in the wild and maybe figuring out how to do it better for wireless systems with rapidly changing rates.
Otherwise, ISP visibility can be limited. Not all equipment will report on buffer use, and even if it does, it may not report on a per port basis, and even then, the timing of measurement might miss things. What you're looking for is a 'standing buffer' where a port always has at least N packets waiting and the buffer does not drain for a meaningful amount of time. Ideally, you'd actually measure the buffer length in milliseconds, rather than packets, but that's asking a lot of the equipment.
There's a balance to be met as well. Smaller buffers mean packet drops, which is appropriate when dealing with standing buffers; but too small of buffers leads to problems if your flows are prone to 'micro bursts', lots of packets at once potentially on many flows, and then calm for a while. It's better to have room to buffer those.
The actual problem is I'm on a voip call and someone starts a big download (steam) and latency and jitter go to hell and the call is unusable. Bufferbloat test confirms that latency dramatically increases under load. Or same call but someone starts uploading something big.
If troublesome buffers are at the last mile connection and the ISP provides a modem/router, adding QoS limiting downloads and uploads to about 90% of the acheived physical connection will avoid the issue. The buffers are still too big, but they won't fill under normal conditions, so it's not a problem. You could still fill the buffers if there's a big flow that doesn't use effective congestion control, or a large enough number of flows so that the minimum send rate is still too much; or when the physical connection rate changes, but good enough. Many ISPs do this, and so you hear a lot less complaining about bufferbloat on say Comcast these days; also, this is an effective best practice, so less need for papers, reports and case studies... it's a matter of getting the practices in the wild and maybe figuring out how to do it better for wireless systems with rapidly changing rates.
Otherwise, ISP visibility can be limited. Not all equipment will report on buffer use, and even if it does, it may not report on a per port basis, and even then, the timing of measurement might miss things. What you're looking for is a 'standing buffer' where a port always has at least N packets waiting and the buffer does not drain for a meaningful amount of time. Ideally, you'd actually measure the buffer length in milliseconds, rather than packets, but that's asking a lot of the equipment.
There's a balance to be met as well. Smaller buffers mean packet drops, which is appropriate when dealing with standing buffers; but too small of buffers leads to problems if your flows are prone to 'micro bursts', lots of packets at once potentially on many flows, and then calm for a while. It's better to have room to buffer those.
Rate limiting the CPE doesn't seem to really impact the buffer queue depth on the 100G upstream switch feeding the 1G customer port. In addition, sticking them on something like 90% customer speed plan or 90% port speed also doesn't help, and in fact with many customers, they are now pissed because they never hit their plan speeds in a speed test.
Something I have always done I actually provision to account for packet overhead, so you might speed 2-3% higher speeds than your plan limit in a speed test, but psychologically the customer is getting more than they paid for, and most seem to be very happy about that.
But, rate limits were already in place long before anything about queue depth was even discussed, so that was nothing new. CAKE OTOH has had a very noticable impact on the customer experience, when their kids XBox can download that 250G update without impacting the voip call or wifi offloading another member of the household is on. Alternatively, that same gamer can play while Mom is downloading something near max throughput without having latency spikes and packet loss.
Yes, you're on to something about the customer experience in general that I'm tracking down myself. Orb is also trying to get a look, but I'm not a fan so far of that tool/platform https://orb.net/
Something I have always done I actually provision to account for packet overhead, so you might speed 2-3% higher speeds than your plan limit in a speed test, but psychologically the customer is getting more than they paid for, and most seem to be very happy about that.
But, rate limits were already in place long before anything about queue depth was even discussed, so that was nothing new. CAKE OTOH has had a very noticable impact on the customer experience, when their kids XBox can download that 250G update without impacting the voip call or wifi offloading another member of the household is on. Alternatively, that same gamer can play while Mom is downloading something near max throughput without having latency spikes and packet loss.
Yes, you're on to something about the customer experience in general that I'm tracking down myself. Orb is also trying to get a look, but I'm not a fan so far of that tool/platform https://orb.net/
codel/CAKE also came from that project, no middlebox needed.
Internet connectivity improvement has slowed a lot. It was improving at a good clip in the 00's but then a lot of usage moved to mobile data which also caused investment to shift away from broadband speedups. If we had 00's rate of improvement, people would have 100G connections at home now.
(wifi also dampened bandwidth demand for a long time - it didn't make sense to pay for faster-than-wifi broadband)
(wifi also dampened bandwidth demand for a long time - it didn't make sense to pay for faster-than-wifi broadband)
A relative of mine runs a WISP (800+ customers). He's using LibreQoS to prevent bufferbloat (not its only feature) for his entire network.
Someone hasn't travelled a lot outside of the house I see?
Wifi is still mostly shitty in most places in the world.
Then there are countries like Philippines with just all around slow internet everywhere.
Wifi is still mostly shitty in most places in the world.
Then there are countries like Philippines with just all around slow internet everywhere.
Never said there isn't connectivity or service quality issues inside or outside the home; just that bufferbloat specifically is a trope that should be put to rest.
Google has a long history of performing networking research, making changes, and pushing those changes to the entire internet. In 2011, they published one of my favorite papers, which described their decision to increase the TCP initial congestion window from 1 to 10 on their entire infrastructure.
[deleted]
The primary tradeoff of initcwnd is setting a reasonable window before you've learned anything about the path. BBR has little say on this because it takes, in relative terms, quite a while to go through its phases. An early BBR session is therefore not really superior to other congestion controls because that is not the problem it is really focused on.
Jacking up the initcwnd, you start to risk tail loss, which is the worst kind of loss for a sliding window.. especially in the primordial connection. There are ways of trying to deal with all that but they are loss predictions.
If you are a big enough operator, maybe you have some a priori knowledge to jack this up for certain situations. But people are also reckless and do not understand the tradeoffs or overall fairness that the transport community tries to achieve.
As other comments have pointed out, QUIC stacks also replicate congestion control and other algorithms based on the TCP RFCs. These are usually much simpler and lacking features compared to the mainline Linux TCP stack. It's not a free lunch and doesn't obviate the problem space any transport protocol has to make tradeoffs on.