Manually Throttle the Bandwidth of a Linux Network Interface(mark.koli.ch)
mark.koli.ch
Manually Throttle the Bandwidth of a Linux Network Interface
http://mark.koli.ch/slowdown-throttle-bandwidth-linux-network-interface
41 comments
> Wrong: you see 10KB/s download speed because you are not throttling the incoming packets but the outgoing packets!
Yep. TC's default is to policy outgoing traffic, which in OP's example is a bunch of TCP ACKs essentially. Instead, they should be using ingress keyword, something like described here:
http://blog.stevedoria.net/20050906/ingress-policing-with-li...
Caveat emptor: ingress rate-limiting is hard. Long story short, it all boils down to what you do with non-confirming packets: There are two alternatives, and both are rather sub-optimal. You can either buffer/delay packets in kernel space (default, which leads to bufferbload and memory waste), or drop (which author linked above opted for, which leads to excessive retransmits and bandwidth waste).
Yep. TC's default is to policy outgoing traffic, which in OP's example is a bunch of TCP ACKs essentially. Instead, they should be using ingress keyword, something like described here:
http://blog.stevedoria.net/20050906/ingress-policing-with-li...
Caveat emptor: ingress rate-limiting is hard. Long story short, it all boils down to what you do with non-confirming packets: There are two alternatives, and both are rather sub-optimal. You can either buffer/delay packets in kernel space (default, which leads to bufferbload and memory waste), or drop (which author linked above opted for, which leads to excessive retransmits and bandwidth waste).
What actually happens when Internet traffic makes the jump from my ISPs big-pipe backbone connection to my much slower last-mile connection? They clearly can't cache the world, and "excessive retransmits and bandwidth waste" doesn't seem to be the case?
It's common to find that the buffers in front of the bottleneck last-mile link are sized for the highest tier of service regardless of what speed you're actually subscribed for. DSLReports has the only browser-based speed test that measures bufferbloat, and they have extensive data from that testing: https://www.dslreports.com/speedtest/results/bufferbloat?up=...
Your modem and the box on the other end of that bottleneck link are probably buffering far more than is reasonable. There's simply no reason for a cable modem to ever have in excess of 1s worth of backlog.
Your modem and the box on the other end of that bottleneck link are probably buffering far more than is reasonable. There's simply no reason for a cable modem to ever have in excess of 1s worth of backlog.
They drop your packets, both for ingoing and outgoing overflow.
But how is that different from the proposed method of rate limited which causes "excessive retransmits and bandwidth waste"?
Not being pedantic, genuinely trying to understand how this stuff works.
Not being pedantic, genuinely trying to understand how this stuff works.
Middle ground between dropping [1] and buffering is Active Queue Management as wtallis pointed out above. State of the art AQM on Linux nowadays is CoDel [2].
Big network gear vendors that ship to ISPs have adopted early forms of AQM (both RED [3] and proprietary algorithms) quite a while ago - they had to:
(a) backbone routers have much smaller buffer-to-bandwidth ratio (compared to a PC or even home router), so endless buffering is not an option;
(b) buffer tail drops (i.e. what drop keyword does when added to tc filters/disciplines) interact really poorly with TCP bandwidth control algorithms, and rather badly with RTP streams, too (so drop is not an option either - it ruins user's connections; and
(c) ISPs and carriers would typically run links (on average) much closer to saturation than your typical home router, so situation where router has to make buffer/drop decision is much, much more frequent.
[1] My point above was that you don't really want to drop packets on the receiver side, after that packet already traversed expensive part of the network.
[2] https://en.wikipedia.org/wiki/CoDel
[3] https://en.wikipedia.org/wiki/Random_early_detection
Big network gear vendors that ship to ISPs have adopted early forms of AQM (both RED [3] and proprietary algorithms) quite a while ago - they had to:
(a) backbone routers have much smaller buffer-to-bandwidth ratio (compared to a PC or even home router), so endless buffering is not an option;
(b) buffer tail drops (i.e. what drop keyword does when added to tc filters/disciplines) interact really poorly with TCP bandwidth control algorithms, and rather badly with RTP streams, too (so drop is not an option either - it ruins user's connections; and
(c) ISPs and carriers would typically run links (on average) much closer to saturation than your typical home router, so situation where router has to make buffer/drop decision is much, much more frequent.
[1] My point above was that you don't really want to drop packets on the receiver side, after that packet already traversed expensive part of the network.
[2] https://en.wikipedia.org/wiki/CoDel
[3] https://en.wikipedia.org/wiki/Random_early_detection
The drop vs buffer decision is no harder for outgoing or incoming packets. In either case: if you're trying to simulate a different kind of network, do what that network does. If you're just trying to get good QoS on your gateway router, then use a smart AQM that will buffer only to the extent that is reasonable, and then drop or ECN mark when buffering threatens to add too much latency.
[deleted]
[deleted]
If you don't need hierarchical classes[1], just use tbf (token bucket filter) instead of htb (hierararchy token bucket) - it's more efficient, more compact, and gives you access to delay in the same discipline as well. Compare:
[1] i.e. stuff like "I would like to have tcp/80 limited to 10 mbit/s, tcp/443 limited to 15 mbit/s, while sum of above should never exceed 20mbit/s, and tcp/80 should get priority when competing for that shared 20mbit/s"
EDIT: changed to "burst 2k". Having burst lower than interface MTU will delay large packets essentially forever.
# htb
tc qdisc add dev eth0 handle 1: root htb default 11
tc class add dev eth0 parent 1: classid 1:1 htb rate 1kbps
tc class add dev eth0 parent 1:1 classid 1:11 htb rate 1kbps
# tbf
tc qdisc add dev eth0 root tbf rate 1kbit latency 42ms burst 2k
[1] i.e. stuff like "I would like to have tcp/80 limited to 10 mbit/s, tcp/443 limited to 15 mbit/s, while sum of above should never exceed 20mbit/s, and tcp/80 should get priority when competing for that shared 20mbit/s"
EDIT: changed to "burst 2k". Having burst lower than interface MTU will delay large packets essentially forever.
https://github.com/urbenlegend/netimpair is a tool which implements the techniques that are described in the article, and more. In particular, jitter (variance) is often forgotten about when testing.
Huge variations in jitter that occurs randomly is a hard thing to reliably simulate. For example simulating a US/48-states consumer grade Ku or Ka-band VSAT service ($85-115/mo) which is a highly oversubscribed TDMA network.
When capacity in your particular spot beam is good, latency could be 550 to 600 ms end to end. When it's bad it could be 1300 or 1700ms and will jitter around randomly anywhere in between those two figures.
When capacity in your particular spot beam is good, latency could be 550 to 600 ms end to end. When it's bad it could be 1300 or 1700ms and will jitter around randomly anywhere in between those two figures.
I think your standard for huge variation in latency could use some updating. A factor of 2-4 increase over baseline latency would be a huge improvement for many terrestrial WiFi systems. And services like Gogo in-flight WiFi have been shown to degrade to over 10 seconds of latency when congested.
Excellent point..
one of my first gigs was troubleshooting an application failure over a low bandwidth dedicated link - everything worked fine until the traffic volume resulted in scheduling jitter due to buffer latency; the resulting added overhead of continual connection renegotiation operations killed what was rest of the link the link and caused the failure to propagate into the application itself..
thankfully this oversight led to me as an intern needing to perform the experiment to 'prove' the point, and a subsequent job offer.. so at least there was that :)
one of my first gigs was troubleshooting an application failure over a low bandwidth dedicated link - everything worked fine until the traffic volume resulted in scheduling jitter due to buffer latency; the resulting added overhead of continual connection renegotiation operations killed what was rest of the link the link and caused the failure to propagate into the application itself..
thankfully this oversight led to me as an intern needing to perform the experiment to 'prove' the point, and a subsequent job offer.. so at least there was that :)
If you're doing this for web related testing, Chrome has a very handy bandwidth throttle built into devtools: https://developers.google.com/web/tools/chrome-devtools/netw...
Does that work also for websocket connections? I'm asking because last time I checked it, I strongly got the impression this feature only works for simple http requests, but I could be wrong.
The tc command is great except for the weird command structure. I really like the comcast tool as a wrapper for tc: https://github.com/tylertreat/comcast
It makes it much easier to do throttle the way you want.
It makes it much easier to do throttle the way you want.
The comcast wrapper is very easy to use because its capabilities are so limited that it is essentially useless. It can accomplish some throttling, but it cannot produce a usefully accurate simulation of a Comcast connection or any other commonly congested bottleneck.
tc is complicated because it does more things.
tc is complicated because it does more things.
BTW, I've no idea of the technicalities of this, but I travel frequently to places with terrible internet (and of course billions of people live in such places), and many web experiences degrade considerably.
If some supercool app or videos don't work, sure, no problem, but if reading some programming documentation (with maybe 30KB of actual text) or a bank statement (with maybe 2KB of actual information) or even getting a restaurant address/phone number doesn't work because of monstrously huge sites with much back and forth (what's the technical term here...), that's frustrating.
So please please please do try to make your sites usable over bad connections :-)
If some supercool app or videos don't work, sure, no problem, but if reading some programming documentation (with maybe 30KB of actual text) or a bank statement (with maybe 2KB of actual information) or even getting a restaurant address/phone number doesn't work because of monstrously huge sites with much back and forth (what's the technical term here...), that's frustrating.
So please please please do try to make your sites usable over bad connections :-)
When working in locations with really terrible net connections, one of the things I have resorted to is a VNC session that is a 1920x1200 desktop (256 color) tunneled inside SSH. In this setup the workstation's VNC client connects to a port on localhost that is SSH forwarded to the remote host. With the right SSH settings for timeout and keepalive it can be surprisingly usable. Then open up whatever application you have that is terrible on high-jitter/high-latency connections inside Chrome or Firefox, or as a native desktop app on the machine that is hosting the VNC session.
How would you get to having the right SSH settings for timeout and keep alive?
What I've done when using a slow connection is run emacs with w3m on a remote server and get the rendered text through SSH. I'm realizing more and more that a lot of our interfaces are more visual so I might try your method.
What I've done when using a slow connection is run emacs with w3m on a remote server and get the rendered text through SSH. I'm realizing more and more that a lot of our interfaces are more visual so I might try your method.
Either by pushing the settings from the ssh client (http://www.gsp.com/cgi-bin/man.cgi?topic=ssh_config) or by changing the settings on the /etc/ssh/sshd_config file of the machine that is both hosting the VNC server and hosting the ssh daemon.
https://www.google.com/search?q=ssh+timeout+keepalive+sshd_c...
https://www.google.com/search?q=ssh+timeout+keepalive+sshd_c...
> So please please please do try to make your sites usable over bad connections :-)
This is only going to happen if people are accurately simulating the true problem. The instructions given in the article and the measures implemented by wrappers like the comcast tool mentioned in another comment do not simulate anything remotely realistic. Testing against a simulation like this can help verify that the obvious changes like reducing image sizes and the total number of requests for a page will help, but it won't fully show how things like the CDG and BBR TCP congestion control methods help.
This is only going to happen if people are accurately simulating the true problem. The instructions given in the article and the measures implemented by wrappers like the comcast tool mentioned in another comment do not simulate anything remotely realistic. Testing against a simulation like this can help verify that the obvious changes like reducing image sizes and the total number of requests for a page will help, but it won't fully show how things like the CDG and BBR TCP congestion control methods help.
I never see tools or articles like this discussing emulating bufferbloat, only static high latency. In the real world, satellite connections are relatively uncommon, and 500ms latency is usually instead due to excess buffering on a link that can deliver low latency when not saturated.
Also, don't the instructions in this article only apply to outbound traffic?
If you're trying to simulate poor network conditions, you need to have a better understanding than this of what causes poor network performance, and how to properly emulate it.
Also, don't the instructions in this article only apply to outbound traffic?
If you're trying to simulate poor network conditions, you need to have a better understanding than this of what causes poor network performance, and how to properly emulate it.
There is a way to do ingress traffic manipulation, you have to divert it via virtual device. For years this was https://github.com/imq/linuximq/wiki/WhatIs a most common way to do it.
Yes, and since 2006 there's been IFB in the upstream kernel to do most of the same work. My point was more that the one-way nature of qdiscs is a really huge thing to overlook.
Also for more details http://lartc.org/howto/
i generally found this : http://wiki.linuxwall.info/doku.php/en:ressources:dossiers:n... to be pretty useful overall.
It's a decent introduction, but a few years out of date. The discussion of bufferbloat needs to be updated to account for the BQL mechanism to automatically manage driver buffering, and the sections on more recent AQMs need to be fleshed out. In particular, fq_codel or cake should be recommended over the Rube Goldberg HTB+SFQ+PFIFO_FAST setup described.
Julia Evans (@b0rk) just had an article on that:
http://jvns.ca/blog/2017/04/01/slow-down-your-internet-with-...
http://jvns.ca/blog/2017/04/01/slow-down-your-internet-with-...
Maybe netem is more complete
[https://wiki.linuxfoundation.org/networking/netem]
It has delay, loss, duplication, corruption, re-ordering, rate control.
And take a look at this SO thread: [http://stackoverflow.com/questions/130354/how-do-i-simulate-...]
It has delay, loss, duplication, corruption, re-ordering, rate control.
And take a look at this SO thread: [http://stackoverflow.com/questions/130354/how-do-i-simulate-...]
if you want something complicated to study, here is my old DaemonShape, dynamic shaper.. https://github.com/daemonna/bashtools/blob/master/daemon_sha... ..should be self-explanatory when you run with help.
I know it's easy with a server, but is there a codeless way to tell a client app which network interface you want it to use?
It's more fun to simulate latency and packet loss for VOIP connections... it's been a while since I did that.
This does a bad job of simulating a slow network. First, as others mentioned, it only throttles outbound. It also doesn't simulate buffer bloat.
Linux network stack isn't designed for this. The best easy thing to use is BSD's Dummynet pipes.
Linux network stack isn't designed for this. The best easy thing to use is BSD's Dummynet pipes.
Wrong: you see 10KB/s download speed because you are not throttling the incoming packets but the outgoing packets!
So, what you are really doing is rate limiting outgoing ACK packets to 1KB/s, which delays your outgoing ACKs, hence preventing the remote server from sending you data at full throttle.
You can see that with a tool like "bwm-ng" to see Rx/Tx speed, and you'll see exactly 1KB/s on Tx, and something variable on Rx.
The incoming rate will fluctuates depending on the TCP window setting used by the server which translate to how many non-acknowledged packets are allowed to be sent by the server.