What happens when DNS in China is poisoned and pointed at your server(furbo.org)
furbo.org
What happens when DNS in China is poisoned and pointed at your server
http://furbo.org/2015/01/22/fear-china/
58 comments
for those wondering, extended title could be along the lines of "Experiencing the effects of a DNS poisoning attack"
This is why I really like having Varnish in front of anything Internet-facing: for something like returning a 400 for an un-approved Host header, you're going to be effectively bottlenecked by the network interface.
That's cute, the IP address for his demo curl command is ... daringfireball's web server.
I noticed that as well. However, they seem to be friends / colleagues / associates. A web search for "Craig Hockenberry daringfireball" shows that Craig Hockenberry is (or has been) a contributor to John Gruber's show and John Gruber has recommended one of Craig's apps [1].
[1] http://safetylightapp.com/
[1] http://safetylightapp.com/
And if you run it, you get the Daring Fireball homepage, which surprised me - it's not configured for this problem.
I wonder if this could really be harnessed in China as a weapon.
That's the only reason I can imagine they'd respond with anything else besides 127.0.0.1 or some internal web server with warnings to obey their overlords.
why is 'out' inbound, and 'in' outbound?
Because it's from the perspective of the router; traffic out of the router is TO his network.
if he owns/operates the router, it still doesn't make any sense.
in our network, 'out' means outbound from the edge, and 'in' means in from the edge, in all cases.
in our network, 'out' means outbound from the edge, and 'in' means in from the edge, in all cases.
The bandwidth chart I get from my colo (where I don't own the router that measures my bandwidth usage) works the way I described. The article mentions a single server, and nothing about owning the router also. If they did, that's probably where they'd null-route the traffic they don't want.
It seemed backwards to me too, when I first saw it.
It seemed backwards to me too, when I first saw it.
We were the lucky targets of China's DNS poisoning on Monday the 19th. Any requests from clients in China to Edgecast's CDN was instead thrown at our public IP. After deducing what was going on, we ended up blacklisting large (class A in some cases) APNIC address blocks assigned to China.
Interestingly, it wasn't our webservers that were overwhelmed but instead the Cisco firewall that sits in front of them. 25K concurrent connections made it decidedly unhappy.
Interestingly, it wasn't our webservers that were overwhelmed but instead the Cisco firewall that sits in front of them. 25K concurrent connections made it decidedly unhappy.
Same here. We received well over 150mbit/s, also on Monday the 19th, also from China. We also saw occassional spikes starting on January 9th. We saw the same BitTorrent /announce traffic, and lots of other random traffic.
We ended up mitigating it by moving our IP address on that host, and blocking all input on the old address.
We ended up mitigating it by moving our IP address on that host, and blocking all input on the old address.
you are lucky you only got 150mbit/s
Here is what I had to deal with on the 9nth Ramming my server...
http://i.gyazo.com/c2262d41f92bc97f0dd3e5e0152d32ee.png
Yes that is almost 1000mbit/s All coming from china...
Here is what I had to deal with on the 9nth Ramming my server...
http://i.gyazo.com/c2262d41f92bc97f0dd3e5e0152d32ee.png
Yes that is almost 1000mbit/s All coming from china...
My company's web server saw a similar unusual surge in traffic with host headers and URLs of Chinese sites for an 8 hour window a few months ago. All the IPs were from a single Chinese ISP, though the volume of the traffic was a lot lower than what OP observed. Seems like these events aren't uncommon.
They (the GFW) intend to have every server owner ban China IPs so that they could claim "you guys are also banning us, what position are you in to blame us for GFW?"
Big ol' [citation needed] on that one.
/* That's what he said */
Well, do you know what the Communist Party thinks?
Well, do you know what the Communist Party thinks?
this seems like a clever way of censoring internet access from within China - get the site owners to do the censorship for you.
Thanks a lot for the post.
What suprised me the most was the fact that Google is serving 40K (now 47K in live statistic) requests per second.
That was way below what I thought.
Each Apache server without real optimization running on a typical physical machine would be able to serve 40-50 requests/second; so they only need a thousand of servers to do 50K requests/second.
Or am I missing something?
Yeah, the actual search bit.
Serving a static file is pretty simple at 40-50 req/sec but searching an entire index of the Internet @ 40-50 req/sec is harder.
Serving a static file is pretty simple at 40-50 req/sec but searching an entire index of the Internet @ 40-50 req/sec is harder.
Your missing factor is "the fact that Google is serving 40K (now 47K in live statistic) requests per second" is wrong.
"The number of requests peaked out at 52 Mbps. Let’s put that number in perspective". Perspective is 52 Mbps isn't a small number, but not massive either.
"The number of requests peaked out at 52 Mbps. Let’s put that number in perspective". Perspective is 52 Mbps isn't a small number, but not massive either.
Not particularly feasible but it would be awesome to simply proxy all of the requests to their intended destination.
Oh man. That would be pretty cool. Somebody like CloudFlare could do that since they proxy for lots of smaller sites. Then again, HTTPS will add even more complexity.
Even if it likely impossible for anyone smaller than CF, it might even mean they would move the attack off sooner since it wasnt doing anything :)
that would be good, or proof of something not widely known in china that is officially denied by china.
I live in China.
We have a department named 工信部, it blocks a site by polluting the DNS and point the domain name to random foreign IP.
I think this explains what happened here.
A Chinese here, I'm deeply ashamed by this malicious action at a national level. though DNS poisoning wasn't new for Chinese netizens but it was always poisoning A records to varies foreign military sites like DoD or some Korea sites (which also makes no sense at all). At scale of this big, those shameless bastards running state censorship machines has been outrageously attacking on random innocent site owner, this is far beyond unacceptable.
And starting from like a month ago, a lot of people on the Chinese internet started to decorate the GFW, and concludes with "for your own good" shit. I can see from those fragments of thoughts, China is not going to stop the GFW, and as Moore's Law advancing, it's going to be even worse.
Sorry for swearing, the feeling of "there's nothing anyone can do to stop this BS" just leads to greater despair.
And starting from like a month ago, a lot of people on the Chinese internet started to decorate the GFW, and concludes with "for your own good" shit. I can see from those fragments of thoughts, China is not going to stop the GFW, and as Moore's Law advancing, it's going to be even worse.
Sorry for swearing, the feeling of "there's nothing anyone can do to stop this BS" just leads to greater despair.
I have seriously contemplated a special request handler for Chinese traffic that is being directed to the wrong host on our servers. Something along the lines of "Chinese government censors broke your internet" or the like.
I figure it either helps inform the citizens as to why this link isn't working or the message itself causes the GFW to ban our IP as well.
I figure it either helps inform the citizens as to why this link isn't working or the message itself causes the GFW to ban our IP as well.
Many of us outside GFW/Cn are aware of the "decoration" (we call it Astroturfing in the US) - it's so transparent that it's often joked about on Reddit and similar sites.
Also Chinese here, GFW is really a shame for us.
[deleted]
Today, GFW start block VPN. So it only would become worse in the future..............
[deleted]
Seeing /announce reminded me of this:
https://torrentfreak.com/bittorrent-users-ddos-websites-with...
Using DHT is also a particularly effective way of getting massive amounts of UDP traffic directed at you, which does not stop even after stopping the torrents. It subsides eventually, but can be quite irritating.
https://torrentfreak.com/bittorrent-users-ddos-websites-with...
Using DHT is also a particularly effective way of getting massive amounts of UDP traffic directed at you, which does not stop even after stopping the torrents. It subsides eventually, but can be quite irritating.
I wrote a blog post about a DDoS on a website by the Chinese at http://dvps.me/ddos-attack-by-torrent. Falsely I assumed that it was torrent tracker injection, while in fact this looks like a spike in the affect of DNS poisoning in China.
http://serverfault.com/questions/656093/mod-security-block-r... describes the problem as well.
https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepirat... also describes the same thing.
As well as http://www.webhostingtalk.com/showthread.php?p=9351951
http://serverfault.com/questions/656093/mod-security-block-r... describes the problem as well.
https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepirat... also describes the same thing.
As well as http://www.webhostingtalk.com/showthread.php?p=9351951
One possibility is to block requests coming with a different 'Host:' header than your website. Or just block the URLs like described in dvps.me/ddos-attack-by-torrent
Nitpick:
>One thing I learned is that Apache can have problems figuring out which virtual host to use in some cases:
>>If no ServerName is specified, then the server attempts to deduce the hostname by performing a reverse lookup on the IP address.
ServerName is the name Apache reports in Error messages and using for redirects. It has nothing to do with the Host-Header sent by the client. If it did, a reverse lookup definitely wouldn't be of any help either.
His remedy was still ok though - just for the wrong reasons.
>One thing I learned is that Apache can have problems figuring out which virtual host to use in some cases:
>>If no ServerName is specified, then the server attempts to deduce the hostname by performing a reverse lookup on the IP address.
ServerName is the name Apache reports in Error messages and using for redirects. It has nothing to do with the Host-Header sent by the client. If it did, a reverse lookup definitely wouldn't be of any help either.
His remedy was still ok though - just for the wrong reasons.
I'm the author of the post linked above: thanks for the clarification about ServerName. I spent a lot of time reading that documentation and got pretty confused about how it was actually being used.
Complementing it, when there is no Host HTTP header, apache displays the default virtual server for the port used. Unless you've set it, it's the first entry on your config file.
It's not a nitpick. It's the difference between making stuff up and reading the documentation. It does matter when reason about how to configure your server.
New @ GFW: Now providing infrastructure stress testing for the western world.
Surely at this level of traffic it would make sense to get in contact with your provider and see if they can block it upstream?
This is why i cloud front personal hosts, and fastly work hosts. I never open my server to the worlds with cdn' being so cheap or free now
> thousands of Apache child processes
Were you still using preforking MPM or something? Surely switching to threaded or even better, event MPM would have mitigated this to a degree...
Were you still using preforking MPM or something? Surely switching to threaded or even better, event MPM would have mitigated this to a degree...
Or put a nginx in front of the Apache, disable keepalives in the Apache server (letting nginx deal with keepalive from the clients), and reduce MaxClients so even if they're all active the server does not go into swap.
(I've done exactly that for a friend, it made a huge difference in how much traffic his box can handle).
(I've done exactly that for a friend, it made a huge difference in how much traffic his box can handle).
This is something I was wondering, too. Using nginx, I can easily top 50,000 req/s (45Mbit/s) on smaller, static/cached content, using 4 cores and < 1GB RAM.
Maybe China are trying a new approach to censorship. Instead of blocking things themselves, perhaps they're trying to get sysadmins to block all of China. I imagine it would save them a lot of work.
We started seeing this around the 9th I believe. We were not the only people as well, based on a somewhat late ISC post[1].
We thought it was a new form of intelligent blackholing. Instead of sending traffic to IPs that could easily be blacklisted by tools to get around the firewall, the Great Firewall would start sending them to random "good" IPs for the same result. Others seem to think the same thing[2].
[1] https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepirat... [2] https://en.greatfire.org/blog/2015/jan/gfw-upgrade-fail-visi...
We thought it was a new form of intelligent blackholing. Instead of sending traffic to IPs that could easily be blacklisted by tools to get around the firewall, the Great Firewall would start sending them to random "good" IPs for the same result. Others seem to think the same thing[2].
[1] https://isc.sans.edu/forums/diary/Are+You+Piratebay+thepirat... [2] https://en.greatfire.org/blog/2015/jan/gfw-upgrade-fail-visi...
Then the next day the same thing happened but much more traffic. The 444 change helped some but there was just too much traffic for the web servers to handle quickly so they bogged down to a crawl. Luckily we were able to figure out through severfault and some other searches that it appears to be DNS poisoning coming from China. We ended up banning the entire country.
The third day the same thing happened but because we had blocked the traffic from China at our outside firewall the servers were unaffected. Since then we have seen some flashes of traffic being blocked but not nearly as much as before.
In all cases the increased volume of traffic only lasted about an hour. The only thing I can surmise about the length of time is DNS records only being cached for about an hour. So after that time the poisoned cached DNS records were replaced by the real resolving ip address. In the case of the article's author their servers suffered a much longer attack than we did. I am not sure if it is the Chinese government doing it or an attack for hire scheme using holes in many Chinese based DNS servers.
TLDR; Remove default server settings from your web servers and have the default server block respond 444 or 404. This may help stave off the attack until your incoming traffic takes up all available resources of the web servers. Of course you could always block all traffic based in China like we did.
Here is info on the status codes incase you are curious: http://en.wikipedia.org/wiki/List_of_HTTP_status_codes#4xx_C...