Ask HN: How can I tell if a site is being rate limited?
I live in a country with less than ideal Internet freedom standards and I have a suspicion that a popular publishing site is being rate limited. How can I know for sure? Is it possible to show a smoking gun?
28 comments
I don't have personal experience using it, but I believe Ooni Probe is designed for your exact use-case:
https://ooni.org/about/
https://ooni.org/about/
Thanks for the recommendation!
Just curious -- if a country didn't want a site to be seen, why would they rate-limit it instead of just blocking connections completely?
Is that a thing countries do, like if they want to "punish" a site while retaining plausible deniability?
Is that a thing countries do, like if they want to "punish" a site while retaining plausible deniability?
People's attention spans are ridiculously short. If the person is actively looking for the content, sure, that obviously won't work. However, if it's a more casual browser, then going to a site that takes a long time to load between different pages is definitely a way to get them to not visit it any more. This even works on tech-savvy people, as opposed to just blocking a site which will just get them to use a proxy.
> Is that a thing countries do, like if they want to "punish" a site while retaining plausible deniability?
China and Russia do this with a lot of western sites, from what I've heard.
> Is that a thing countries do, like if they want to "punish" a site while retaining plausible deniability?
China and Russia do this with a lot of western sites, from what I've heard.
Yes: https://news.ycombinator.com/item?id=22541960
I have seen this actually happen IRL
I have seen this actually happen IRL
Thanks so much for the link. That was really educational, answered my question completely.
It would be too obvious. If you can't get to the site you know it's blocked. If it's just super slow all the time, you won't know why and may just get frustrated and leave.
Have some fun at the same time:
1) get a vpn account (proton, other)
2) hack your worst enemy's wifi with your favourite tools
3) while outside your enemy's home, boot a linux live distro
4) change your computer's name to your enemy's name
5) connect to the enemy's wifi
6) try to visit all the sites banned by your country
7) insult your country's leader in a public forum using your enemy's name
8) test the speed of the suspected rate-limited site
9) set-up your vpn account settings
10) using your vpn, re-test the rate-limited site
11) subtracting a fair percentage slow down because of the vpn, gauge if there's a significant difference between the two tests
12) turn off computer, go home, enjoy tasty beverage
13) grab some popcorn, go watch your enemy's house from a reasonable distance
14) post video on youtube of your enemy getting busted, using your favourite heading
If that's all too much fun for you, then skip most of it and only do #s 8 to 11 from your own network or favourite hot-spot.
If that's all too much fun for you, then skip most of it and only do #s 8 to 11 from your own network or favourite hot-spot.
This is very insightful. Tip of the hat to you!
He should also modify the said enemy's browser cache for this to work well.
The thing about rate limits is, there are often hard limits. Try querying the site a bunch and see what the data transfer rate is. if it's consistently incapable of going above say, 128K/sec, you've got your rate limit.
Other times, they'll allow for bursts but sustained get rate limited - these can be detected just the same depending on their approach; if it's per connection (and not IP) you may have to find a big file to download.
It also helps to establish that traffic to other sites is not rate limited, and to use 3rd party checkers that check for connection rate across the globe. Or DIY and spin up a bunch of EC2 machines that are geographically diverse.
Other times, they'll allow for bursts but sustained get rate limited - these can be detected just the same depending on their approach; if it's per connection (and not IP) you may have to find a big file to download.
It also helps to establish that traffic to other sites is not rate limited, and to use 3rd party checkers that check for connection rate across the globe. Or DIY and spin up a bunch of EC2 machines that are geographically diverse.
I don't know about this, I've heard that limiting is often intentionally inconsistent because this makes people less likely to visit. I think Harlow did some similar stuff with his monkeys.
My country used to heavily censor internet pages, do finishing attacks on Facebook, Gmail etc..
The easiest way to prove it is by using a VPN or a proxy. Check with different ISPs, also try mobile vs home.
The easiest way to prove it is by using a VPN or a proxy. Check with different ISPs, also try mobile vs home.
I can definitely notice an improvement in performance when I use a VPN, however it would be cool to see where things are being delayed. Is traceroute or curl something that could help here?
Yea, if it’s faster via VPN that’s a strong hint. If you want proof you need to examine the internet between you and the site. If a VPN connection that uses a specific router to to the site is fast, your connection through that same router is slow, and you can connect to some other site through that same router and get a fast connection, then that’s a smoking gun that your connection is deliberately slowed.
That said, it’s not easy to prove it’s your country/ISP doing the slow down vs that website slowing down connections to your country.
That said, it’s not easy to prove it’s your country/ISP doing the slow down vs that website slowing down connections to your country.
My company has a residential proxy network that lets you test endpoints with requests from residential broadband ISPs all over the world. Some of our customers are using our network for exactly this use case: https://packetstream.io
its possible that your ISP has a bad peering connection. Checking with your ISP usually helps understand what is going on.
It could be the bandwidth limitation of your ISP itself.
Sites like Github are slow when accessed from few countries. It's just that their servers are located mainly in the US(afaik).
It could be the bandwidth limitation of your ISP itself.
Sites like Github are slow when accessed from few countries. It's just that their servers are located mainly in the US(afaik).
I ran a curl test and noticed that the TCP connect time is where things are getting hung up, taking over 80 seconds! Does anyone know if this is indicative of deliberate rate limiting or just a bad peering connection?
Possibly neither, that is, there are many possible reasons.
for example, stateful firewalls can only track a certain number of connections. If you don't have a public IP, or if they are trying to protect you, it may be necessary to wait for a connection resource to come available. Connections are generally expensive, different from data transfer, for these and other reasons, such as port starvation, syn flood defenses inappropriately targeting you, so many others.
for example, stateful firewalls can only track a certain number of connections. If you don't have a public IP, or if they are trying to protect you, it may be necessary to wait for a connection resource to come available. Connections are generally expensive, different from data transfer, for these and other reasons, such as port starvation, syn flood defenses inappropriately targeting you, so many others.
If it takes 80 seconds to TCP connect, something is pretty wrong.
tcpdump / wireshark would likely show lots of retransmits. Traceroute/mtr may give you an idea of where the loss starts. If you can find servers on similar hosting, you can get some idea of if it's targetted or not.
tcpdump / wireshark would likely show lots of retransmits. Traceroute/mtr may give you an idea of where the loss starts. If you can find servers on similar hosting, you can get some idea of if it's targetted or not.
One possible method: Write a script to request the site continuously and see if it fails to respond with any regularity. From there you can start dialing it in and determine what the rate limit actually is. A rate limit should be consistently reproducible unless it's implemented in a clever way.
Would mtr[0] help diagnose this?
[0] - https://github.com/traviscross/mtr
[0] - https://github.com/traviscross/mtr
You could test from vps in another country?
[deleted]
If implemented with respect to the http code, it should say 429 in the developer console.
State internet restrictions aren't implemented in the application layer.
That's not the kind of rate limiting he's asking about. He is talking about the data stream being throttled.