Ask HN: Is there something wrong with internet right now?
77 comments
Malaysian here. I'm surprising a Comcast issue would reach all the way here.
I noticed a lot of sites were working really badly, thought it was some problem with AWS or some damaged internet artery cable.
I noticed a lot of sites were working really badly, thought it was some problem with AWS or some damaged internet artery cable.
There should be a service sysadmins can run that sends controlled pings, download/upload requests, etc, to specific hosts. The idea being that the results are shared publicly.
--
Copypasting https://www.wired.com/story/how-a-tiny-error-shut-off-the-in... (dated today):
--------------
A YEAR AGO, a DDoS attack caused internet outages around the US by targeting the internet-infrastructure company Dyn, which provides Domain Name System services to look up web servers. Monday saw a nationwide series of outages as well, but with a more pedestrian cause: a misconfiguration at Level 3, an internet backbone company—and enterprise ISP—that underpins other big networks. Network analysts say that the misconfiguration was a routing issue that created a ripple effect, causing problems for companies like Comcast, Spectrum, Verizon, Cox, and RCN across the country. Level 3, whose acquisition by CenturyLink closed recently, said in a statement to WIRED that it resolved the issue in about 90 minutes. "Our network experienced a service disruption affecting some customers with IP-based services," the company said. "The disruption was caused by a configuration error." Comcast users started reporting internet outages around the time of the Level 3 outages on Monday, but the company said that it was monitoring "an external network issue" and not a problem with its own infrastructure. RCN confirmed that it had some network problems on Monday because of Level 3. The company said it had restored RCN service by rerouting traffic to a different backbone.
The misconfiguration was a "route leak," according to Roland Dobbins, a principal engineer at the DDoS and network-security firm Arbor Networks, which monitors global internet operations. ISPs use "Autonomous Systems," also known as ASes, to keep track of what IP addresses are on which networks, and route packets of data between them. They use the Border Gateway Protocol (BGP) to establish and communicate routes. For example, packets can route between networks A and B, but network A can also route packets to network C through network B, and so on. This is how internet service providers interoperate to let you browse the whole internet, not just the IP addresses on their own networks.
In a "route leak," an AS, or multiple ASes, issue incorrect information about the IP addresses on their network, which causes inefficient routing and failures for both the originating ISP and other ISPs trying to route traffic through. Think of it like a series of street signs that help keep traffic flowing in the right directions. If some of them are mislabeled or point the wrong way, assorted chaos can ensue.
Route leaks can be malicious, sometimes called "route hijacks" or "BGP hijacks," but Monday's incident seems to have been caused by a simple mistake that ballooned to have national impact. Large outages caused by accidental route leaks have cropped up before. "Folks are looking to tweak routing policies, and make mistakes," Arbor Networks' Dobbins says. The problem could have come as CenturyLink works to integrate the Level 3 network, or could have stemmed from typical traffic engineering and efficiency work.
Internet outages of all sizes caused by route leaks have occurred occasionally, but consistently, for decades. ISPs attempt to minimize them using "route filters" that check the IP routes their peers and customers intend to use to send and receive packets, and attempt to catch any problematic plans. But these filters are difficult to maintain on the scale of the modern internet, and can have their own mistakes. Monday's outages reinforce how precarious connectivity really is, and how certain aspects of the internet's architecture—offering flexibility and ease-of-use—can introduce instability into what has become a vital service.
--
Copypasting https://www.wired.com/story/how-a-tiny-error-shut-off-the-in... (dated today):
--------------
A YEAR AGO, a DDoS attack caused internet outages around the US by targeting the internet-infrastructure company Dyn, which provides Domain Name System services to look up web servers. Monday saw a nationwide series of outages as well, but with a more pedestrian cause: a misconfiguration at Level 3, an internet backbone company—and enterprise ISP—that underpins other big networks. Network analysts say that the misconfiguration was a routing issue that created a ripple effect, causing problems for companies like Comcast, Spectrum, Verizon, Cox, and RCN across the country. Level 3, whose acquisition by CenturyLink closed recently, said in a statement to WIRED that it resolved the issue in about 90 minutes. "Our network experienced a service disruption affecting some customers with IP-based services," the company said. "The disruption was caused by a configuration error." Comcast users started reporting internet outages around the time of the Level 3 outages on Monday, but the company said that it was monitoring "an external network issue" and not a problem with its own infrastructure. RCN confirmed that it had some network problems on Monday because of Level 3. The company said it had restored RCN service by rerouting traffic to a different backbone.
The misconfiguration was a "route leak," according to Roland Dobbins, a principal engineer at the DDoS and network-security firm Arbor Networks, which monitors global internet operations. ISPs use "Autonomous Systems," also known as ASes, to keep track of what IP addresses are on which networks, and route packets of data between them. They use the Border Gateway Protocol (BGP) to establish and communicate routes. For example, packets can route between networks A and B, but network A can also route packets to network C through network B, and so on. This is how internet service providers interoperate to let you browse the whole internet, not just the IP addresses on their own networks.
In a "route leak," an AS, or multiple ASes, issue incorrect information about the IP addresses on their network, which causes inefficient routing and failures for both the originating ISP and other ISPs trying to route traffic through. Think of it like a series of street signs that help keep traffic flowing in the right directions. If some of them are mislabeled or point the wrong way, assorted chaos can ensue.
Route leaks can be malicious, sometimes called "route hijacks" or "BGP hijacks," but Monday's incident seems to have been caused by a simple mistake that ballooned to have national impact. Large outages caused by accidental route leaks have cropped up before. "Folks are looking to tweak routing policies, and make mistakes," Arbor Networks' Dobbins says. The problem could have come as CenturyLink works to integrate the Level 3 network, or could have stemmed from typical traffic engineering and efficiency work.
Internet outages of all sizes caused by route leaks have occurred occasionally, but consistently, for decades. ISPs attempt to minimize them using "route filters" that check the IP routes their peers and customers intend to use to send and receive packets, and attempt to catch any problematic plans. But these filters are difficult to maintain on the scale of the modern internet, and can have their own mistakes. Monday's outages reinforce how precarious connectivity really is, and how certain aspects of the internet's architecture—offering flexibility and ease-of-use—can introduce instability into what has become a vital service.
Yes, 4.2.2.1 DNS is down. Probably your problem.
Might as well be a symptom.
Symptom of it being a primary cause would be no loading, then all of a sudden loading of site. However, I've had intermittent connection with HN, so not sure that is necessarily the primary. Edit: clearly spell check is down.
Thanks for the tip. I'll pass this along to our users.
I thought it was just me. I’m in Colorado. Bandwidth seems to be reporting as normal, however I’m having issues loading Facebook, imgur, some advertisements on reddit, etc.
It may have something to do with a large, prolonged power outage in downtown Denver. I've seen at least one ISP tweet about possible disruption to services due to this, and it may be impacting others.
http://www.9news.com/news/local/power-outage-in-downtown-den...
I'm in Colorado as well! Glad to know it's not just me. Are you on CenturyLink or Comcast?
Comcast
I'm in the province of Quebec (Eastern Canada), and we're having huge problems with most websites. Strangely enough, LTE network is working on the three providers I've tried.
Same here. I was going crazy trying to diagnose potential WiFi issues. Since LTE worked fine on my phone I assumed the culprit had to be the WiFi link.
Its really weird.
Its really weird.
Everything's back online now. It's been like 5 minutes.
Southern Ontario, everything is slow as all hell
Oh no, the Internet Monopolies are instituting "Rolling Blackouts" ahead of the FCC decision on Net Neutrality!!
Also Southern Ontario (Toronto), I've noticed nothing. Been or reddit, HN, many random programming websites, starcraft, etc.
I've checked all the urls people said they had problems with so far (spotify, cloudflare, downdetector) and haven't had any.
I've checked all the urls people said they had problems with so far (spotify, cloudflare, downdetector) and haven't had any.
downdetector.com now seems to be down. never a good sign.
facebook, discord, reddit; all unusable
what the hell?
what the hell?
Related thread: https://news.ycombinator.com/item?id=15637412
There are I think three threads talking about this right now. This one won the front page and topical title fight.
There are I think three threads talking about this right now. This one won the front page and topical title fight.
Sweden here. Facebook and other select sites down.
Dang I can't even access downdetector. I'm in California in the central valley. Everything is very sluggish and the ol' router/modem reboot did nothing. :/
yes, lots of sites offline for me. I'm in Atlanta GA.
https://cloudflare.com is down for me. Comcast in California. HN is really slow.
I thought it was just me - was about to head to a cafe. On comcast here in Berkeley FWIW.
None of the sites mentioned in this or other threads are down from the vantagepoint of Cox in Arizona.
Sounds like an ISP issue.
Sounds like an ISP issue.
More likely BGP or similar border gateway routing things. In some cases, gateways can become overloaded because of bad destination routes that other gateways are routing correctly. Not that I'm an expert at this stuff. I'm not.
Comcast Business in Nashville is terrible right now for anything that's not local (Facebook, Reddit, etc.). Some are worse than others (HN is just slow; Reddit's very slow but loads eventually; Down Detector won't load at all on our work connection now).
AT&T (in Nashville) doesn't seem to be impacted; I'm RDP'd into one of my home machines with Uverse and everything's working great there.
AT&T (in Nashville) doesn't seem to be impacted; I'm RDP'd into one of my home machines with Uverse and everything's working great there.
I’m in the SF area and Comcast is between slow and not working. Cellular data is working fine though
No problems reaching any site from the UK
Spotify has been down for me since ~1:00 EST in Toronto.
Second Life is reporting major problems. [1]
[1] https://status.secondlifegrid.net/
[1] https://status.secondlifegrid.net/
I'm having significant issues as well -- it happened quite suddenly an hour ago (9:50 Pacific Time, northern California). Thought it was a local issue at first and spent the last hour debugging my network. I think my apartment is on Comcast, although I'm not 100% certain (one of my roommates is in charge of it).
Spamming `date` at an affected SSH connection suggests it's entirely a download problem, since all of the timestamps are very close, but they're received multiple seconds apart.
Like some other commenters here, my mobile connectivity is just fine.
EDIT: Certain sites (including Hacker News) are now qualitatively more accessible, and I'm now hitting expected speeds at http://testmy.net/download. SSH to my DigitalOcean is also better. Here's hoping everything else smoothes out soon!
Spamming `date` at an affected SSH connection suggests it's entirely a download problem, since all of the timestamps are very close, but they're received multiple seconds apart.
Like some other commenters here, my mobile connectivity is just fine.
EDIT: Certain sites (including Hacker News) are now qualitatively more accessible, and I'm now hitting expected speeds at http://testmy.net/download. SSH to my DigitalOcean is also better. Here's hoping everything else smoothes out soon!
Same here on Comcast in Menlo Park (CA). T-Mobile is zippy, though, so I guess I'll switch to mobile hotspot for a while.
Salem, Oregon here. Everything is zippy. Seems like it's location dependent? Just checked reddit and facebook.
I was just thinking, shouldn't there/couldn't there be a site that aggregates actual statistics (as opposed to the crowd-sourced DownDetector) regarding internet connectivity/health globally to provide an accurate snapshot of what might actually be happening?
Edit:
Sorta like this[0] I suppose.
[0]https://www.akamai.com/us/en/solutions/intelligent-platform/...
Edit:
Sorta like this[0] I suppose.
[0]https://www.akamai.com/us/en/solutions/intelligent-platform/...
For many years there was a site called the Internet Health Report which would show packet loss and problems between multiple carriers. It was pretty useful until Dynatrace deep-sixed it this year.
I haven't found a suitable replacement, would be interested to hear any suggestions.
I haven't found a suitable replacement, would be interested to hear any suggestions.
Yeah, I just headed there for the first time in a while.
DISAPPOINTED.
DISAPPOINTED.
Same here! I seriously miss that site.
Issues here in San Jose area - it appears to be Comcast from what I can tell. Same as others, cellular via ATT is fine.
some in Seattle are complaining: https://www.geekwire.com/2017/internet-issues-plague-users-a...
Lots of similar reports on the outages mailing list:
https://puck.nether.net/pipermail/outages/2017-November/0108...
"Anyone having packet loss, please send traceroutes to the list. There is clearly something happening, but not much useful data yet. Some traceroutes might help narrow down the cause of the problem."
"Anyone having packet loss, please send traceroutes to the list. There is clearly something happening, but not much useful data yet. Some traceroutes might help narrow down the cause of the problem."
I'm a Comcast user in Hayward (Bay Area), CA, USA.
Some sites are easily available: Google, Yahoo.
Some sites are partially available: Twitter, Hacker News.
Many sites are completely unavailable: Down Detector.
Some sites are easily available: Google, Yahoo.
Some sites are partially available: Twitter, Hacker News.
Many sites are completely unavailable: Down Detector.
I was having maddening issues with Comcast Business in Portland, OR this morning.
Switched over to my VPN and haven’t had any problems since.
My initial pokes at the issues I was seeing were DNS related, but even switching the DNS hosts to Google or OpenDNS didn’t completely fix it. Maybe they’ve got some deep packet inspection issues?
Switched over to my VPN and haven’t had any problems since.
My initial pokes at the issues I was seeing were DNS related, but even switching the DNS hosts to Google or OpenDNS didn’t completely fix it. Maybe they’ve got some deep packet inspection issues?
Also Comcast (in Portland, OR) also having major issues, resolved by tethering.
I've read anecdotal reports of an Level3 outage that makes Comcast have poor connectivity outside its network.
I have websites that are up right now according to monitoring tools, but they are inaccessible to me from home.
I have websites that are up right now according to monitoring tools, but they are inaccessible to me from home.
Maybe. I am getting a weird issue when trying to open a page for research I am doing for paid work. I can't get to the page and it offers to open as a PDF or something and then fails to open anything. Just more error messages follow.
Everything loading blazing fast (Romania).
It appears to be a Comcast issue: https://news.ycombinator.com/item?id=15637412
https://twitter.com/ReportOutage/status/927600102918811648 shows a problem with Comcast.
That tweet links to http://outage.report/comcast .
That tweet links to http://outage.report/comcast .
I'm in Northern California and also having issues.
One thing I noticed is that it is only Downloads having issues. My uploads are as fast as ever.
I then connected to my VPN, and everything is working great. I think it is their routing that is having the issues, not the connection itself.
One thing I noticed is that it is only Downloads having issues. My uploads are as fast as ever.
I then connected to my VPN, and everything is working great. I think it is their routing that is having the issues, not the connection itself.
I contacted Comcast Network Operations Center. They confirmed they're aware of the issue and working on fixing it, but didn't provide any more details.
I can’t connect to VPN (on AWS), but AWS CLI is working fine. Weird. But, I’ll just do something else for a while ...
The situation here started to improve within the last two minutes.
Gizmodo is reporting that it is a DDoS attack.
https://gizmodo.com/this-is-probably-why-half-the-internet-s...
[edit] Dang, I thought I had checked the date. Sorry everyone!
https://gizmodo.com/this-is-probably-why-half-the-internet-s...
[edit] Dang, I thought I had checked the date. Sorry everyone!
That's a year old article, FYI.
Goes to show you how incredible slow this attack has made the Internet!
That article is from 2016.
That's from the outage on 10/21/16.
That’s from 2016?
Comcast says it's been resolved for most customers now: https://twitter.com/comcastcares/status/927636388517838848
This is how WW3 will start
Last night I had trouble with Google, Netflix, Reddit redirect links, and some other sites. I was trying to think of places an explosion could happen that would cause my issues.
This morning I've narrowed the explosion's epicenter down to the Wifi card in my PC. :/
This morning I've narrowed the explosion's epicenter down to the Wifi card in my PC. :/
[deleted]
Anyone else having this issue or any insight as to what's going on?