East Coast Internet Outage
Anyone else noticing a widespread internet outage on the East Coast of the US?
263 comments
https://twitter.com/VerizonSupport/status/135410988957298278...
There is a fiber cut in Brooklyn. We have no ETR, as of yet. You can use the MY Fios app for updates. *EAG
There is a fiber cut in Brooklyn. We have no ETR, as of yet. You can use the MY Fios app for updates. *EAG
As per WSJ this was not the cause.
https://www.wsj.com/articles/verizon-internet-outage-disrupt...
https://www.wsj.com/articles/verizon-internet-outage-disrupt...
Its wild that after ~9 hours, there is still not a clear cause:
> The company hadn’t pinned down the source of the failure as of Tuesday afternoon.
> The company hadn’t pinned down the source of the failure as of Tuesday afternoon.
Does a cut cable in BK account for outages up and down the east coast?
This is speculation, of course, but: there are several large Internet Exchanges in NYC, and US/EU peering points. It's probable that a fiber cut could have cut several other links, and the failover links are now completely overwhelmed.
I'm not knowledgeable enough to answer that question. However, this is the only Twitter comment that mentions this on their page, and I don't have access to a Verizon account to verify their outage status memo. We had a flurry of employees saying their connections were going haywire to the business and this is the only thing I've been able to gather outside of downdetector.
Verizon had (and apparently still has) a few vulnerable links in the NYC metro area that have wide ranging impacts.
If it's a fiber that goes into major data centers it certainly could.
Apparently, Verizon's outage page doesn't actually say if there's an outage. You have to log in to see if there's an outage. Too bad logins are failing after ~10 minutes for me.
https://www.verizon.com/support/residential/service-outage
https://www.verizon.com/support/residential/service-outage
Yeah, I'm getting a 504 from their status site even after managing to get through the authn process.
Although my packet loss to my IPv6 tunnel provider just went to zero, so that's a good sign here (Manhattan).
Although my packet loss to my IPv6 tunnel provider just went to zero, so that's a good sign here (Manhattan).
When houses are created there are building codes to adhere to. For example, when there is a water pipe behind the walls a plumber (a good one) would also put a plate in front of it. In this way, once the walls are in place, drilling away into the wall, or nailing, would hit the metal plate, thus provides some safety mechanism. As another example, underground, in public areas are markings (or at least procedures) to identify such areas gas pipe areas before excavating.
How are the fiber optics being cut? Are there no procedures for this or is there some challenges how the infra is setup today?
How are the fiber optics being cut? Are there no procedures for this or is there some challenges how the infra is setup today?
I'm probably the only person reading this thread who owns a backhoe, and it's only a small one. There's really no material that will resist being dug through and is also practical to bury over miles and miles of fiber. There obviously are processes and measures in place to prevent "backhoe-fade" events, including armored cable, tracer wires and tape above the main cable but humans being humans things sometimes don't work out. Contractors can be liable for the cost to spice repairs which will be very costly so they're incented to not cut fibers.
Backhoes also can't be blamed for every outage. Cars hitting poles that carry aerial fiber, people ripping up fiber thinking it is copper they can steal, boat anchors at landing sites, etc.
Level 3 at one point stated 17% of all fiber cuts were from squirrels... https://web.archive.org/web/20151213214019/http://blog.level...
Level 3 at one point stated 17% of all fiber cuts were from squirrels... https://web.archive.org/web/20151213214019/http://blog.level...
One of my favorite network incident reports involved someone using aerial fiber for target practice...
I didn't see your post before mine. I've had circuits down TWICE due to this.
New York is particularly bad for having lots of arial fibre that gets taken down by car crashes or squirrels.
Idiots shooting rifles at aerial fiber...
I like to joke that the North-American Fiber-Seeking-Backhoe is not, and will never be an endangered species.
You are not the only person reading this thread who owns a backhoe.
Everything else is 100% correct!
Everything else is 100% correct!
I don't have a backhoe, does an excavator count?
I once hit a gas line. With a jackhammer. 8” below the surface. That should never happen, but did.
I once hit a gas line. With a jackhammer. 8” below the surface. That should never happen, but did.
Wondering what the ad targeting AI is making of this thread..
I thought they were supposed to lay that "cable/pipe/Jimmy Hoffa buried here" warning tape somewhere above the fragile thing that's buried, so if digging you'd catch the tape, see it, and stop?
Used to work for a few ISPs. 90% of the fiber cuts that were experience while I worked there were from municipal construction crews who either weren't fully paying attention when working around fiber lines, or didn't know they were there in the first place.
The lines are supposed to be known/marked to some extent but you can't avoid the human factor.
In our worst case the same city cut the fiber line in multiple places over the course of a few weeks. Our crews would finish patching one fiber break only for the same (city) crew to cut it again a few hours later.
I wasn't on the ground so I don't know exactly how they managed it, but it was a real headache for us to manage on the customer end.
The lines are supposed to be known/marked to some extent but you can't avoid the human factor.
In our worst case the same city cut the fiber line in multiple places over the course of a few weeks. Our crews would finish patching one fiber break only for the same (city) crew to cut it again a few hours later.
I wasn't on the ground so I don't know exactly how they managed it, but it was a real headache for us to manage on the customer end.
Scale: wrapping all fiber optic cables in a protective metal sheath would be cost prohibitive. Whereas adding a metal sheet to prevent a house from flooding or someone from getting electrocuted makes sense.
One idea, borrowed from some chainsaw safety pants, would be to wrap the cable in a fiber web sheath that tangles in whatever cutting equipment
https://www.youtube.com/watch?v=r5VSivQe760
One idea, borrowed from some chainsaw safety pants, would be to wrap the cable in a fiber web sheath that tangles in whatever cutting equipment
https://www.youtube.com/watch?v=r5VSivQe760
Fibre isn’t usually cut by something like a chainsaw but rather by something like a backhoe, rodent’s teeth (squirrels on overhead lines and rats if they run through utility tunnels), or a gunshot. I doubt these super-long chainsaw chaps would even stand up to a determined rodent.
TIL that Aramid fibers are not just something made up in bideo games. (: Thanks for sharing that.
Literally took off the pants testing that! I'm unsure how much is worth getting paid to run a self-test. This product looks pretty neat. Half joking, anyone ever, or witnessed getting eletrocuted trying to get the internet / computer working again?
This spun threads reminds me of nature. Spiders, silk works spinning web. Wonder if that's where the inspiration came from.
This spun threads reminds me of nature. Spiders, silk works spinning web. Wonder if that's where the inspiration came from.
Old Joke: How do you get rescued from a desert island if all you have is some networking gear?
Simple! Bury a network cable and wait for the backhoe to show up.
Backhoes are VERY strong and will easily go through gas pipe let alone anything that might protect fiber. And if the fiber is not where the plans say it is, even calling 811 won't save you.
Simple! Bury a network cable and wait for the backhoe to show up.
Backhoes are VERY strong and will easily go through gas pipe let alone anything that might protect fiber. And if the fiber is not where the plans say it is, even calling 811 won't save you.
Even worse is that (here in Canada at least) we have infrastructure surveyed, its GPS coords recorded, then manually entered into CAD (or similar) drawings using a lot of guess work. Things like metal detectors are used to more or less guess that a detected object is in fact a gas line or something, GPS location is recorded, then someone plays connect the dots in CAD. It's unreal. So many opportunities to introduce error, and the initial locating of the underground object could easily be incorrect.
I know newer infrastructure is more diligently recorded, but there is a TON of old crap in the ground ready to be ripped up.
I know newer infrastructure is more diligently recorded, but there is a TON of old crap in the ground ready to be ripped up.
We did some work on the foundation right next to the gas meter which had a pipe going down - which unexpectedly went off direct into the yard to who knows where. They hand dug until they were confident (still took out the power line to the garage. Twice).
The vast majority of fiber optic cuts are someone digging through the cable underground.
You can't realistically prevent this with additional armoring, because it's quite a different failure to someone drilling in the wrong place - the forces are a lot higher, or heavy machinery is involved etc. (see also: the rate at which workers go through water mains in footpaths).
You can't realistically prevent this with additional armoring, because it's quite a different failure to someone drilling in the wrong place - the forces are a lot higher, or heavy machinery is involved etc. (see also: the rate at which workers go through water mains in footpaths).
Fiber optics rely on obscurity to remain secure. Not far from my home, there’s at least 2-3 significant fiber optics buried in a trench next to the street in the outskirts of the city, probably 30-40 inches under the ground.
The dig safe people can and do mark them, but people tend to do stupid things.
Some highways have fiber rights of way too. You can see fiber conduit on bridges on the mass pike, for example.
The dig safe people can and do mark them, but people tend to do stupid things.
Some highways have fiber rights of way too. You can see fiber conduit on bridges on the mass pike, for example.
> when there is a water pipe behind the walls a plumber (a good one) would also put a plate in front of it
Where are you located (roughly)? Having assisted in remodeling of 7 homes in the US (IL, WI, CA) built between 1880 and 2005 I haven't seen this.
Where are you located (roughly)? Having assisted in remodeling of 7 homes in the US (IL, WI, CA) built between 1880 and 2005 I haven't seen this.
I've heard of such a thing but never seen it. I was a satellite installer and drilled into a lot of walls. I'd just drill thru the drywall then feel around with a drill bit.
I've never heard of it for plumbing, but kinda in a similar vein most electrical codes require a certain setback for all cabling running through studs and protection where it can't be maintained.
If the cable is passing through a stud less than 1.25" from either face of the stud (giving you ~1.75" once drywall is installed), then a metal plate has to be installed to stop someone trying to hang a picture or shelf or something innocuous up from driving a screw into the cable.
If the cable is passing through a stud less than 1.25" from either face of the stud (giving you ~1.75" once drywall is installed), then a metal plate has to be installed to stop someone trying to hang a picture or shelf or something innocuous up from driving a screw into the cable.
Lots of ways, unfortunately. But generally a quality network will be built to have physical redundancy. Yet, the ISP game at the top is all about cost cutting. So we'll never get what we paid for back in the 90s.
Plates are generally only placed when the pipe or wire is within drywall-screw depth once you add drywall to the studs. They aren't infallible -- For example, they don't protect against carpenters with nailguns patching sheathing on the outside who miss the stud entirely and hit a water pipe instead (you may wonder how I know this..)
As for fiber cuts, one case I'm aware of involved a large diameter concrete drill/hole saw with a diamond-tipped blade. Not much you can do to armor against that.
As for fiber cuts, one case I'm aware of involved a large diameter concrete drill/hole saw with a diamond-tipped blade. Not much you can do to armor against that.
Liability is pretty good protection -- pretty much everyone that uses heavy equipment to dig is aware of the need to use the "call before you dig" numbers to have someone come out and mark underground utilities.
There's always the person with a rented ditch digger or over confidence "Well I dug out here last year and didn't hit anything, so I can dig again" who doesn't call or notice the trace cable, but overall that service works pretty well at preventing disruptions.
There's always the person with a rented ditch digger or over confidence "Well I dug out here last year and didn't hit anything, so I can dig again" who doesn't call or notice the trace cable, but overall that service works pretty well at preventing disruptions.
Construction crews need to pay attention to what is below them before they start digging. A backhoe is going to defeat your cable every time.
In a most states, there are blue stake laws requiring you to call and set an appointment to have existing utilities marked with spray paint before you dig. Even though this service is usually free to the contractor, it's sometimes viewed as one more hassle. Usually they didn't plan in advance and so their trying to cut corners to save time.
When half of my coworkers on a Zoom call (including myself) got kicked within ten seconds of each other, I figured something major had happened. Western PA. Was somewhere around 11:24 EST.
Network never went down completely, bitrate just took a nosedive.
Network never went down completely, bitrate just took a nosedive.
Issues with FiOS in the Boston area. Traceroute seems to point to an issue originating in NY but that's just speculation. Apparently a lot of other services are down. Might be a Tier 1 or 2 ISP.
Same here. Seeing heavy packet loss to most (but not all) sites that route through verizondigitalmedia.com.customer.alter.net in NYC.
Imagine a world where all our enemies DIDN'T know how fragile our infrastructure is. False security, surely, but still would be somewhat comforting. :)
Firstly, America should strive to not have enemies. We fail at this very badly.
Secondly, ZERO countries in the world have non-fragile infrastructure. America's infrastructure is better than most countries and definitely above average even among developed countries, except public transportation.
Secondly, ZERO countries in the world have non-fragile infrastructure. America's infrastructure is better than most countries and definitely above average even among developed countries, except public transportation.
> Firstly, America should strive to not have enemies. We fail at this very badly.
This will never be possible, even if America does nothing 'wrong' against anyone else. As long as America (flaws and all) is a better place to live than other countries, and can be used as an example of how things can be better (though not perfect), and so will be a target of powers that don't want their own people thinking about improving things.
* https://en.wikipedia.org/wiki/The_Road_to_Unfreedom
* https://www.theguardian.com/books/2018/apr/15/the-road-to-un...
There are also plenty of bullies out there, who are only checked by American power, and wouldn't mind the US being more isolationist.
There are all sorts of problems of how America interacts with the world, but there are many other countries who would do worse things IMHO. America simply existing generates enemies.
This will never be possible, even if America does nothing 'wrong' against anyone else. As long as America (flaws and all) is a better place to live than other countries, and can be used as an example of how things can be better (though not perfect), and so will be a target of powers that don't want their own people thinking about improving things.
* https://en.wikipedia.org/wiki/The_Road_to_Unfreedom
* https://www.theguardian.com/books/2018/apr/15/the-road-to-un...
There are also plenty of bullies out there, who are only checked by American power, and wouldn't mind the US being more isolationist.
There are all sorts of problems of how America interacts with the world, but there are many other countries who would do worse things IMHO. America simply existing generates enemies.
America will always have adversaries. But enemies, who what to destroy the country? That's entirely our own creation.
We should try to make enemies only when attacked. Not make enemies out of adversaries.
We should try to make enemies only when attacked. Not make enemies out of adversaries.
> America will always have adversaries. But enemies, who what to destroy the country? That's entirely our own creation.
There are innocent people who have done nothing wrong, and yet other folks want them wiped out of existence.
I'm sure we can all think of examples from history where this is true, both the recent and distance past.
There are innocent people who have done nothing wrong, and yet other folks want them wiped out of existence.
I'm sure we can all think of examples from history where this is true, both the recent and distance past.
I disagree it's our fault for having enemies if we want to keep our values but regardless we have enemies now and I don't see them going away for a while.
[deleted]
The problem isn't our enemies knowing how fragile our infrastructure is as much as it Americans not knowing (or not caring) and being unwilling to spend money to make it less fragile.
> Americans not knowing (or not caring) and being unwilling to spend money to make it less fragile.
And of course the fact that for the most part those most impacted by these outages aren't the ones making the decisions about what money to spend in the first place.
I have no meaningful influence on my ISP's network backbone. I don't even have a real choice in ISP so it's not like I can go to someone else, and they know it.
I currently have 500mbit/sec service from my cable company and the only other options are ~50mbit/sec 4G or Frontier DSL that claims 24mbit/sec while rarely syncing beyond 8. Those aren't actual alternatives. If I want to have a connection that can properly support a household full of gamers and streamers, I'm stuck with this unless I want to add another zero or two to my monthly bill for a dedicated fiber line.
This is the best case scenario for the majority of the US. One cable company that's mostly decent, one shitty DSL provider, and a few wireless services that have very limited spectrum and data caps. Almost no one in this country has true competition in wired broadband unless they have a municipal fiber service or are lucky enough to be in one of the few cities that Google dumped money in to.
And of course the fact that for the most part those most impacted by these outages aren't the ones making the decisions about what money to spend in the first place.
I have no meaningful influence on my ISP's network backbone. I don't even have a real choice in ISP so it's not like I can go to someone else, and they know it.
I currently have 500mbit/sec service from my cable company and the only other options are ~50mbit/sec 4G or Frontier DSL that claims 24mbit/sec while rarely syncing beyond 8. Those aren't actual alternatives. If I want to have a connection that can properly support a household full of gamers and streamers, I'm stuck with this unless I want to add another zero or two to my monthly bill for a dedicated fiber line.
This is the best case scenario for the majority of the US. One cable company that's mostly decent, one shitty DSL provider, and a few wireless services that have very limited spectrum and data caps. Almost no one in this country has true competition in wired broadband unless they have a municipal fiber service or are lucky enough to be in one of the few cities that Google dumped money in to.
Honestly, at this point I've found HN to be the best "Status Page" for this sort of thing. I came here first when I found issues and fiddling around with my custom DNS lookup hierarchy didn't help.
FiOS customer in DC Metro.
FiOS customer in DC Metro.
It would be interesting if someone could automate something to pick out HN outage discussions and make an HN driven "downdetector" page.
downdetector.com did well today. Front page showed a generalized web issue, and any one of the individual services' status pages showed it was geographically localized to the midatlantic and northeast US.
Also checking /r/sysadmin and sorting by new
Except these posts seem to regularly get taken down or flagged. This isn't even on the front page anymore it seems.
It set off the flamewar detector. We monitor that list and eventually cancel the penalty for threads that aren't actual flamewars, but it takes a while. If anyone wants to make it faster, emailing [email protected] is usually a forcing function.
Edit: Also, posts without URLs get a downweight by default. I've turned that off too, and merged the other threads into this one, since it was the first one.
Edit: Also, posts without URLs get a downweight by default. I've turned that off too, and merged the other threads into this one, since it was the first one.
> It set off the flamewar detector.
Why? There do not appear to have been any posts remotely resembling a flamewar in the comments.
Why? There do not appear to have been any posts remotely resembling a flamewar in the comments.
That's answered in my sibling (well, cousin) comment here: https://news.ycombinator.com/item?id=25921757. It can't test for flamewars directly so it ends up guessing wrong sometimes.
How does one programmatically detect a flamewar? :)
By proxy, which is why there are false positives.
> How does one programmatically detect a flamewar? :)
An NLP/Machine learning model would be a great way! Probably also taking in other features as input such as the number of comments per second, and how deeply nested those comments are.
Let's build one!
An NLP/Machine learning model would be a great way! Probably also taking in other features as input such as the number of comments per second, and how deeply nested those comments are.
Let's build one!
I'm glad I'm not the only one noticing this. I could usually rely on HN to not be so heavily "filtered" for a lack of a better term but I have noticed the slide into a more controlled/gamed PR outlet.
My corporate laptop is affected. My roommate’s corporate laptop is totally unaffected. We’re on the same router/gateway.
Any theories out there to explain this phenomenon? Fascinating.
(My theory is a network partition, where my roommate’s IP addresses happen to be in regions that are on “our side” of the partition, whereas my IP Addresses are on the “other”, unreachable side).
Any theories out there to explain this phenomenon? Fascinating.
(My theory is a network partition, where my roommate’s IP addresses happen to be in regions that are on “our side” of the partition, whereas my IP Addresses are on the “other”, unreachable side).
Are you using different DNS servers? My work laptop was using 8.8.8.8 and is getting name resolution errors, but my personal computer is using 1.1.1.1 and working fine.
Interestingly enough, they’re both deferring DNS to our router, and the router is configured to use DNS servers dynamically provided by the ISP.
its most likely your companies RDP infrastructure (or local environment) operates on AWS. Which is down because of this "cut"
[deleted]
I didn't do any digging but it seems like once you have a connection established, it's not too bad. Is your roommate on a full-tunnel VPN?
They weren’t on a VPN, but my guess is that they got IP addresses from their DNS requests that were on our side of the partition.
In any case, it looks like it’s ended for us. The window for gathering evidence slides closed...
It’ll be interesting to read the post mortem on this one.
In any case, it looks like it’s ended for us. The window for gathering evidence slides closed...
It’ll be interesting to read the post mortem on this one.
Thanks for the tip; connecting to a full-tunnel VPN seems to fix this for me. (FiOS in Brooklyn).
Completely unable to load anything today from 11:30sh to 12:30ish EST in Brooklyn, all seems to be smooth now. First ever issue with FiOS in two years, would still fully recommend them to anyone when comparing to most options, especially Comcast.
I have no technical understanding of fiber networks. The explanation that this was caused by one fiber line in Brooklyn being severed seems unlikely. Curious to see what the experts think.
So, for clarity, it's never one fiber line being cut. It's a big massive bundle of fiber carrying different circuits for different carriers different places that were run in the same trench and all got cut at once.
Then they get to dig it out better and someone gets to get down in the hole and figure out which piece of fiber needs to be spliced to which other one.
Because of DWDM, some of those individual fibers may each be carrying 64 channels that in turn each carry 400gbit/sec for different providers/services/etc.
Then they get to dig it out better and someone gets to get down in the hole and figure out which piece of fiber needs to be spliced to which other one.
Because of DWDM, some of those individual fibers may each be carrying 64 channels that in turn each carry 400gbit/sec for different providers/services/etc.
And when it's a large bundle cut, it's going to take a bit before service is restored. I have been told that a highly skilled tech can splice a single strand in a matter of minutes. Looking it up, I find figures of an average of 30 minutes per joint for 4-strand bundles, and slowing down from there. So if you have a 144-strand bundle to repair, or multiple 144-strand bundles, it's going to take several hours to effect the repair, and you're probably going to have to rotate through techs to avoid fatigue-induced errors slowing things down even more.
Would it maybe be an idea to, going forward, color the cables individually before putting them into the ground?
Of course they are colored. Like this: http://www.evertopcomms.com/144-fiber-count-gyts-fiber-optic...
It is still a lot of work, and so far not automated.
It is still a lot of work, and so far not automated.
It is also, as a matter of practicality, going to be very difficult to find 144 colors that can‘t be mistaken for one another, particularly in conditions that i would imagine wouldn‘t have great lighting.
Well, the layout has them grouped into colored sub-bundles.
If they’re all in use and it’s point to point, might be easier connecting whatever to whatever. Then simultaneously swap line cards around as they get lit up or letting IP do its thing.
Source: I have no idea.
Source: I have no idea.
Not as feasible as it sounds initially. I don't know if you've ever looked inside a cat-5/6/whatever networking cable[1], but if you do, you'll see that it is actually a bundle of 8 individual wires, in 4 twisted-together pairs, color coded. Those different wires carry different electrical signals, and if the various signals are not supposed to be on the various wires they are supposed to be on, the connection won't work.
So imagine if you cut a Cat6 cable in half, and then spliced the individual wires back together without regard to which wires you connected. Very likely, Pair 1 negative is connected Pair 3 negative, Pair 2 has its polarity reversed, etc, and so the network link never comes online.
At the individual device level, fiber cables are connected in pairs, in a crossover fashion. Device A Tx connects to Device B Rx, and vice versa. So with a random splicing, you would end up with pairs getting crossed or broken up, invalidating the connections and labeling of every patch panel downstream, making for yet more work than the initial work of splicing the cut bundle.
1. Not impugning your intelligence, knowledge, or character here, some people never have because they've never needed or wanted to, and that's totally okay.
So imagine if you cut a Cat6 cable in half, and then spliced the individual wires back together without regard to which wires you connected. Very likely, Pair 1 negative is connected Pair 3 negative, Pair 2 has its polarity reversed, etc, and so the network link never comes online.
At the individual device level, fiber cables are connected in pairs, in a crossover fashion. Device A Tx connects to Device B Rx, and vice versa. So with a random splicing, you would end up with pairs getting crossed or broken up, invalidating the connections and labeling of every patch panel downstream, making for yet more work than the initial work of splicing the cut bundle.
1. Not impugning your intelligence, knowledge, or character here, some people never have because they've never needed or wanted to, and that's totally okay.
You must be joking. Cables are
Color-coded and labeled obsessively.
You should take some time to understand the physical layer of infrastructure. There will always be work to do in meet-me-rooms!
You should take some time to understand the physical layer of infrastructure. There will always be work to do in meet-me-rooms!
This. Even if you have redundant lines with multiple providers there is a good chance they are part of the same fiber bundle. It can be really difficult to get providers to admit who they are leasing from and what the physical runs are.
And then telco's may move leased fiber to other paths without telling customers, messing up the physical redundancy they thought they had...
While I'm also not qualified to diagnose the problem, I would imagine it's something like this...
The major internet connections are like superhighways, with major central routers acting like exchanges. If a major superhighway had an accident blocking traffic, there would be personnel stationed at exchanges to help re-route the traffic, though that often occurs too late to prevent a bunch of backups. At any rate, people using Google Maps, Waze, etc. would see the accident and would likely be redirected to alternate routes that would normally be much slower - and would also be slowed down by the influx of higher than normal traffic.
The fact that the internet still worked albeit much slower than usual leads me to believe that something similar happened. While normally my internet traffic might go through a major hub in NYC and then onward to various carriers and hosts, etc. it had to be re-routed around that hub that is normally sub-optimal, and also ill-equipped to handle the spillover. (And in some cases, bandwidth was saturated, and failed altogether for some users for certain usage.)
The major internet connections are like superhighways, with major central routers acting like exchanges. If a major superhighway had an accident blocking traffic, there would be personnel stationed at exchanges to help re-route the traffic, though that often occurs too late to prevent a bunch of backups. At any rate, people using Google Maps, Waze, etc. would see the accident and would likely be redirected to alternate routes that would normally be much slower - and would also be slowed down by the influx of higher than normal traffic.
The fact that the internet still worked albeit much slower than usual leads me to believe that something similar happened. While normally my internet traffic might go through a major hub in NYC and then onward to various carriers and hosts, etc. it had to be re-routed around that hub that is normally sub-optimal, and also ill-equipped to handle the spillover. (And in some cases, bandwidth was saturated, and failed altogether for some users for certain usage.)
The odd thing about this situation though is the geographic location of Brooklyn: on an island. Why would a significant amount of traffic from outside Long Island be going across the harbor to Brooklyn, especially given all the major switching facilities in Manhattan (closer to shore).
What makes you think it is only connected to Manhattan? Long Island has submarine cables connected to New England, various points along the East Coast south of NYC, and several transatlantic cables.
My naive explanation: fiber networks handle incredible loads, and a single fiber line might correspond to 10+ Tbs (terabytes per second). So all that data needs to go somewhere else, and if it gets shunted, packets get backed up, resulting in a cascading failure.
Note: I am not a fiber technician nor do I work for a large ISP.
Note: I am not a fiber technician nor do I work for a large ISP.
I wouldn't doubt it. Single misconfigured routers have caused issues of similar scale.
I believe the internet is designed to handle broken links, but is not very robust against misconfigured (or malicious) routers.
If a 100 Gbps link goes down, and all the other links that could carry that route are only 10 Gbps... it goes downhill quickly.
Too much traffic is concentrated at the major carriers which carry a large amount of traffic over relatively little fiber (cause its cheaper) and when that fiber disappears we lose a massive amount of capacity...
Too much traffic is concentrated at the major carriers which carry a large amount of traffic over relatively little fiber (cause its cheaper) and when that fiber disappears we lose a massive amount of capacity...
Just to add to that, sometimes the redundancy doesn't work out as planned, maybe another weird set of circumstances, or some routers can't handle spikes in load as the network reconverges, causing other reconverges, confusing bgp routing peers, etc. Fun stuff!
It seems to have recovered for me. I am using FIOS in Brooklyn.
Same, also fios in Brooklyn. Not getting my full speeds but I am getting 100Mbps, which is far better than the 60 Kbps I was getting.
Ironically things recovered within 30 seconds me of messaging our systems team to ask if the development VPN was having an outage. That's like a magic rule of internet outages, isn't it? Nothing works until you try to show someone else the issue, then it all works.
Ironically things recovered within 30 seconds me of messaging our systems team to ask if the development VPN was having an outage. That's like a magic rule of internet outages, isn't it? Nothing works until you try to show someone else the issue, then it all works.
Same! Also FIOS in BK
Verizon is having issues in NYC for sure.
https://twitter.com/VerizonSupport/status/135411467509349580...
typical that link is useless verizon.com/outage. "learn how to reboot your router"
Right now in my part of Bucks County, PA, FiOS' gigabit speed internet is running about normal speed for download and about 3/4 of normal for upload.
Small world, I'm in the same place. I noticed long periods of dropped packets, it looks like existing connections are ok but establishing a new one is a problem. My wife working over a VPN doesn't notice a issue.
I was getting 2 Mbps upstream on FiOS just after noon today! But it seems largely back to normal now. (I have 200/200, not Gigabit.)
Bucks County represent!
Too much is failing from single point outages. The FCC may have to establish standards to prevent over-centralization.
In the entire history of the Bell System, no electromechanical central office was ever down for more than 30 minutes for any reason than a natural disaster or, one occasion, a major file. That record has not been maintained in the Internet era.
In the entire history of the Bell System, no electromechanical central office was ever down for more than 30 minutes for any reason than a natural disaster or, one occasion, a major file. That record has not been maintained in the Internet era.
The Bell System's job was to operate a nationwide telephone system for voice, not high-speed data. Even then, you only had one path to the telephone CO and only one CO it could go to. Outages were rare because voice-grade wire is cheap and all the complexity is handled by the CO.
Also, while my Internet may have outages from time to time, I can't recall the last time I couldn't use my cell phone other than being in a bad signal area. For voice, cellular is definitely the way to go as capacity is easily added by adding new towers; not something easy to do with landlines.
Also, while my Internet may have outages from time to time, I can't recall the last time I couldn't use my cell phone other than being in a bad signal area. For voice, cellular is definitely the way to go as capacity is easily added by adding new towers; not something easy to do with landlines.
What do you mean by saying the wire is cheap? Obviously you can’t send much high frequency data down the wire but each wire could only be used for one call at a time for much of the company’s history, so more wires were needed. What effect does that have on the cost? And surely the complexity is still in the CO with the internet, or am I missing something?
> And surely the complexity is still in the CO with the internet
Your cable modem (which has a CPU and does high-speed signaling) or your FIOS box is much more complicated than an analog phone.
The carrier-grade router on the other end of your cable modem is really just a bigger version of routers you buy off the shelf or for enterprises. It's literally routing IP packets in the same way.
Whereas you have to have a whole switching system at the other end of an analog voice CO. (Or an operator making physical connections if we're still in the 1800's). The best you can do for yourself in that world today is get an PBX system like an old NEC NEAX box or similar, but you're still not going to be on the same level as the CO.
Your cable modem (which has a CPU and does high-speed signaling) or your FIOS box is much more complicated than an analog phone.
The carrier-grade router on the other end of your cable modem is really just a bigger version of routers you buy off the shelf or for enterprises. It's literally routing IP packets in the same way.
Whereas you have to have a whole switching system at the other end of an analog voice CO. (Or an operator making physical connections if we're still in the 1800's). The best you can do for yourself in that world today is get an PBX system like an old NEC NEAX box or similar, but you're still not going to be on the same level as the CO.
Most of the early internet piggy backed on the bell system: 56k, T1, T3 lines, frame relay, ISDN... and of course dialup.
Natural disasters sometimes take out cellular service in large areas. Definitely happened for several days during Hurricane Sandy in NY.
They can get temporary cellular devices to fill in on generator power pretty quickly. I know most if not all phone lines are underground, but something like a 2000-pair "wet pulp" repair (necessary in the first place due to negligence) takes a week. I don't know the name of these devices but something of that nature was deployed in Nashville on Dec 25 when the AT&T building was bombed and I think most cell service downtown was restored the next day.
>no electromechanical central office was ever down for more than 30 minutes for any reason than a natural disaster or, one occasion, a major file
Are these systems even comparable? Today /= yesterday phone service....
Are these systems even comparable? Today /= yesterday phone service....
dylan604(2)
Yep. Second day of classes at George Mason University in Fairfax and no one can connect to Zoom or Blackboard meetings. I'm on FiOS as well.
Yes. Pittsburgh area is spotty with FiOS right now.
I find it weirder that Zoom Verizon got BGP routed through AS42861 via 0.ae15.gw14.iad8.alter.net
and that AS42861 isn’t clearly defined under Roblex
and that AS42861 isn’t clearly defined under Roblex
Very interesting. HE's BGP tool sees it https://bgp.he.net/AS42861
edit: so does https://www.peeringdb.com/net/2258
edit: so does https://www.peeringdb.com/net/2258
Noticing degradation in connection quality in Boston, MA. ISP is Verizon FIOS - operating on the 1 Gbps plan.
Etsy appeared hard down for me (though not for friends a mile over). Homepage kept returning 429
We have FiOS in Boston and noticed it. Nothing terrible but for ~15 seconds at a time, recurring sporadically, we lost connection. Happened for a few hours, maybe 10am-1pm EST.
Based on other comments we probably just didn't notice the big actual outage that happened all at once.
Based on other comments we probably just didn't notice the big actual outage that happened all at once.
I'm in Philly, and I'm having issues with Comcast.
Doesn't look like it's isolated to any one provider, or any one site. https://downdetector.com is showing many issues across many platforms.
Doesn't look like it's isolated to any one provider, or any one site. https://downdetector.com is showing many issues across many platforms.
Is this still ongoing? I'm having issues that I can confirm that as a result of this, and now three days later things are still spotty. Could be conflating multiple problems, of course.
No problems in Jersey City all day. Bleeping Computer article cites someone as saying it's a specific AS that's affected. JC routes through an AS in Newark, NJ and not NYC so, for me at least, there hasn't been an outage.
Hey neighbor, I disagree, though switching my VPN gateway to our DR one in NJ vs our primary in Chicago cleared things up. For about an hour and a half though, remote desktop was struggling, even though my speed tests were fine. I'm on FIOS.
I saw some disruptions in services that people in my household rely on, but I think that is because of disruptions in DCs located in NYC and not FiOS itself.
Yes, lots of issues connecting to various cloud services here in DC.
On Verizon FIOS here, perhaps this is another Verizon outage?
Issues here too, I'm on Verizon Fios. Right when starting class, what a mess. (As a colleague joked, Google going down (or Verizon I guess) is like the modern snow day)
Yep, on verizon fios just outside baltimore. Can't ping 8.8.8.8, but can ping 1.1.1.1 fine. Also having issues connecting to services running in AWS.
[deleted]
Pittsburgh area has spotty FiOS internet service also.
Yes, large carriers can't do much when the backhoe strikes.. it happens.. they'll fix it.. and we'll all go on with our lives..
I thought the whole point of the internet and protocols like TCP/IP is that it's supposed to be fault tolerant and traffic is supposed to automatically re-route. Seems kind of weird how a single cable being cut can cause half the country to lose internet...
TCP doesn't know anything about route failure or rerouting. All it does is handle a few dropped or reordered packets here and there.
In principle, you can have systems that very quickly route around damage like this, but in practice it happens so rarely that it's not worth it for consumer ISPs. I think it should exist, but the ISP space is so burdened with red tape that the only way anyone could compete with it was literally to start a rocket ship company and launch a bunch of satellites.
In principle, you can have systems that very quickly route around damage like this, but in practice it happens so rarely that it's not worth it for consumer ISPs. I think it should exist, but the ISP space is so burdened with red tape that the only way anyone could compete with it was literally to start a rocket ship company and launch a bunch of satellites.
Starting an ISP isn't all THAT hard.
https://arstechnica.com/information-technology/2021/01/jared...
https://arstechnica.com/information-technology/2021/01/jared...
Wow, great story I hadn't heard!
> Mauch has been charging $65 a month for symmetrical 50Mbps service, $75 for 250Mbps, and $99 for 500Mbps,
That's less than Comcast consumer rates in Baltimore.
> Mauch has been charging $65 a month for symmetrical 50Mbps service, $75 for 250Mbps, and $99 for 500Mbps,
That's less than Comcast consumer rates in Baltimore.
The Internet is not as redundant as you might think, and in fact there is less spare capacity to go around as a result of the pandemic (a lot more time spent streaming and on video chats by a larger number of people). Internet routing is also non-trivial (convergence has not been reached in years), and when a major link goes down it can take time for routes to stabilize. Also keep in mind that the Internet itself was not down for most people who were affected by this; rather, connectivity between users and various services was affected. So people could have continued torrenting, but their Zoom sessions would get dropped because connectivity to some data center was impacted.
I wonder how much has to do with the cost-optimization pressure on a private carrier. Building robust infrastructure is only valuable to them insofar as it enables them to continue to earn money. Reliable service for the sake of reliable service--especially to residential regions--is a non-starter.
Less pessimistically, it's possible that the regions that're affected have natural "choke points" for infrastructure, either moving through the mountains or along highways, etc., which of course leads to a single-point-of-failure, though it may be unlikely.
Less pessimistically, it's possible that the regions that're affected have natural "choke points" for infrastructure, either moving through the mountains or along highways, etc., which of course leads to a single-point-of-failure, though it may be unlikely.
The ability for the internet to reroute after N path failures is there but someone has to pay for N paths worth of unused network to be ready for it. Most would rather just pay low cost and utilize all the bandwidth that can get them 364/365 days a year. For others (usually businesses) there are more expensive options.
I am on Fios in northeast NJ, issues with some services (Netflix) but not others (Google except significant variance in ping times, Bing, AWS).
Spikes everywhere https://downdetector.com/
I am experiencing connectivity issues on Fios outside of Philadelphia in Pennsylvania.
edit: as of 13:20 (GMT-5) it appears to be resolved.
edit: as of 13:20 (GMT-5) it appears to be resolved.
Yes, sites are loading very slowly, if at all. Thought it was AWS at first but Google services are having the same issue.
Props to Slack, it's the only thing working well for me. GMail, GCP, Meet, all dragging and timing out in Boston.
Getting reports that it's improving. It's nice that it never actually went down down... just degraded
Verizon FiOS. 50% packet loss with many "good" packets in the hundreds of ms when pinging Google. NYC.
Same thing in NYC. on Verizon FiOS fiber
For me, mtr reports quick pings (single digit ms mostly), but HTTP traffic is very spotty. Brooklyn FiOS.
Restored for me (FIOS Brookly NY). Someone get that tech's venmo handle. Human deserves $1.
Not to say their time is only worth $1. Just to emphasize my thanks. Anyway. Please do not skewer me. :(
Philly suburbs: Fios has been inconsistent for me for the last hour or so, starting around 11:00
Comcast seems to be fine (at around 11:50 a.m. EST), so maybe it's just Verizon.
Got knock out from Google Meeting a lot recently, even online speedtest show normal.
I'm on west coast and my entire community lost internet right now.
Observing the same issue from various East Coast locations, all on Verizon Fios.
Connecting to a full-tunnel VPN seems to fix this for me. (FiOS in Brooklyn).
Yes! Thank you for this (in hindsight, obvious) idea.
Thank you! This also fixed my slowdowns.
FIOS Here in the Philadelphia area, having issues hitting AWS as well.
boston fios here, having intermittent issues, mostly slow connection.
I have FiOS and my town is listed here. Internet is still working.
Fios outside Baltimore and yeah it's screwed currently.
[deleted]
Spotify is loading weird for me (New England).
Looks like it's Verizon having trouble.
Restored for me (FIOS Brooklyn, NY)
Fios as well.
I was just about to call FiOS to switch from Xfinity because these data caps are absolutely ridiculous. I am sad now.
Aside: Comcast, wtf? 1.2 TB limit per month? Seriously if anyone works at Comcast know that I hate your company with an overwhelming passion.
Aside: Comcast, wtf? 1.2 TB limit per month? Seriously if anyone works at Comcast know that I hate your company with an overwhelming passion.
FIOS is WAY better, they will fix their broken cable. I get (so far unlimited) 30gb up/down for $40/mo but it typically runs closer to 60 down 70 up. Fiber optic is baller. Though I do feel like they throttle me (web browsing becomes slow) if I am watching a streaming video service or mining crypto without a VPN on, could be an illusion but it always tends to clear up as soon as I activate the VPN.
60-70 gigabits per second for $40/month? Guessing you mean megabits. Not worth being packet inspected. 1gbit on comcast seems to give 42 mbps upload, was 50 a few months ago. ~$80/mo
edit: hitting 48-50 mbps again today, sweet
edit: hitting 48-50 mbps again today, sweet
Comcast upload is a farce. They don't bother stating it anywhere, I presume because they modify it as needed to satisfy their needs for providing TV channels or their "burst" download speeds.
It's usually 10% of your download it seems, apparently until you go past about 250 mbps.
its 3/5/10/35mbit up, but consistent. 35 is the docsis limit on their gigabit cable offering
They never advertise upload bandwidth anywhere, and it varies hugely when I'm at other people's houses that have cable internet. I assumed it's because they might provision 200mbps down / 10mbps up for a neighborhood, and then throttle you down as necessary.
Comcast "gigabit" is a lie.
I reliably get 900mb down and 900mb up with Verizon in DC.
I reliably get 900mb down and 900mb up with Verizon in DC.
The download isn't a lie and it's what they advertise. I was getting 800mbps before I updated my home router from a 2014 model to a 2020 one, now I hit 950-960. I wish the upload was symmetric, but I understand that's not always possible on copper, getting about 50 now. Not bad for a ~50-year-old apartment.
The problem isn't with copper, it's with the cost of building out amplifiers. It's a lot more cost effective to amplify downstream than upstream, especially when most use cases are heavily asymmetric.
(I'd expect that only in the past year or so has upstream been a lot more heavily utilized, what with massively increased work-from-home due to the pandemic. Before then, the primary stressors of the network were things like streaming 4k60 video.
And I'll be very surprised if any ISPs thought torrent uploads to be a use case worth optimizing for.)
(I'd expect that only in the past year or so has upstream been a lot more heavily utilized, what with massively increased work-from-home due to the pandemic. Before then, the primary stressors of the network were things like streaming 4k60 video.
And I'll be very surprised if any ISPs thought torrent uploads to be a use case worth optimizing for.)
Wired connection?
Oh yeah my bad should be mb
I love these posts. They remind me that there are places in the world where technology is moving forward. Where we live in Tennessee I have to run 2 internet connections to be able to have a service when I need it. 3gb DSL and a 5gb satellite connection. The sat connection has the additional bonus of being capped at 150Gb per month. All these new streaming services may as well not even exist for those of us in rural areas.
3 gigabits per second DSL and 5 gigabits per second to a satellite? You (like the other commenter...) probably mean megabits. And also Gb is gigabits, but your cap is probably in bytes, gB or GB.
You should move down to Hamilton county. We have 1Gb fiber for around $60 from EPB (the municipal power company). Even out in the more rural areas.
I’ve got Comcast 2 gig fiber and Verizon 1 fig fiber to my house, plus 1 gig cable. I live 10 minutes from horse farms in MD. (The horse farms also have fiber.)
I've never seen Comcast offer fiber, except to businesses at astronomical costs. I can't even find it on their website. Do they advertise upload bandwidth for their fiber?
https://www.xfinity.com/support/articles/requirements-to-run...
It’s actually 3g up and down. It’s 10 gig Metro-E into a Juniper switch, with a 1g port and 10g port throttled to 2g.
It’s actually 3g up and down. It’s 10 gig Metro-E into a Juniper switch, with a 1g port and 10g port throttled to 2g.
That’s awesome! How much is that, if you don’t mind sharing?
I wish they had it in more places.
I wish they had it in more places.
It’s expensive, $300/month. I try to justify it to myself by noting it’s on their business Metro-E network so it’s completely reliable and has amazing support. Also an entire /48 of static IPv6. It was $150/month for the first two years, which was much more palatable.
Their model is expensive because they don’t wire while neighborhoods. For me they had to run fiber a quarter mile, down my street, and trench under my driveway. I imagine all these one-offs make the per house investment quite high.
Their model is expensive because they don’t wire while neighborhoods. For me they had to run fiber a quarter mile, down my street, and trench under my driveway. I imagine all these one-offs make the per house investment quite high.
That's not bad, I'd be willing to pay that and even more. I was simply unable to contact anyone at Comcast who was willing to entertain running fiber to a home. So I simply had to buy a home that had an option other than Comcast.
In another instance, I needed symmetric fiber at one of my businesses. I contacted Comcast, the business/commercial departments multiple times, got numerous peoples' names, left voice and email messages left and right and got nowhere after weeks. I ended up contacting a different ISP, signed up with them, and then on the install date, Comcast showed up to install the fiber modem (who were, of course, reselling it to the ISP I contracted with). I ended up paying the other ISP $1,700 per month.
In another instance, I needed symmetric fiber at one of my businesses. I contacted Comcast, the business/commercial departments multiple times, got numerous peoples' names, left voice and email messages left and right and got nowhere after weeks. I ended up contacting a different ISP, signed up with them, and then on the install date, Comcast showed up to install the fiber modem (who were, of course, reselling it to the ISP I contracted with). I ended up paying the other ISP $1,700 per month.
The Gigabit Pro support story is very odd. The initial contact goes through the consumer support folks, who are clueless the service even exists. The preferred solution is actually getting on the Comcast Xfinity reddit site to reach a support tech who knows what's going on. After that you're handed off to a Comcast Small Business support agent, who is a real person who maintains contact with you throughout the process (which is a bit involved because it can require permitting, etc.). After that you get a support number that goes to the Metro-E NOC, which is phenomenal. But the billing website still goes through the customer online portal, which doesn't quite understand what's going on.
They offer a 2Gibps fiber optic plan in certain locations for ~299$/month.
https://todayamerican.medium.com/the-definitive-guide-on-how...
https://todayamerican.medium.com/the-definitive-guide-on-how...
This section is an amazing indictment of the company that tens of millions of Americans have to deal with to get access to the internet.
>The Important Part Because so many people had this question, and I finally talked to someone at Comcast who knew what Gigabit Pro actually is, here is how to get in touch with them:
>The Important Part Because so many people had this question, and I finally talked to someone at Comcast who knew what Gigabit Pro actually is, here is how to get in touch with them:
I'm pretty sure I have Comcast fiber, its 1Gbps down and around 750Mbps upload and there's a wire coming out of the wall into the router so... ¯\_(ツ)_/¯
When I bought a home, I limited my searches to ones that had symmetric fiber connections.
I priced having a 10m fiber connection brought out here. I was quoted just over $5k/month with a minimum 3 year contract. I assume they were expecting me to just pay for the new service to the entire area for them to resell.
Wife wanted to live out here. So we traded my being able to work easily for amazing views and lots of land. Not entirely unhappy with the trade, but there are days when I think about just renting an office in town. (30 minutes away).
Wife wanted to live out here. So we traded my being able to work easily for amazing views and lots of land. Not entirely unhappy with the trade, but there are days when I think about just renting an office in town. (30 minutes away).
I was basically restricted to buying homes in new developments since 2010. Those all tended to have fiber, and anything before did not, and they had no plans to install it. I pay $65 per month for 940mbps up and down from Centurylink, and it's been as advertised so far.
Unfortunately, the cable company usually offers some ludicrous 500mbps (burst) download / 5mbps upload connection that people are willing to accept for cheaper, so if the phone company comes in and pays to install fiber and charge accordingly for the new fiber runs, insufficient people would switch to the fiber to make the numbers work since they are okay with the shitty cable company connection.
Unfortunately, the cable company usually offers some ludicrous 500mbps (burst) download / 5mbps upload connection that people are willing to accept for cheaper, so if the phone company comes in and pays to install fiber and charge accordingly for the new fiber runs, insufficient people would switch to the fiber to make the numbers work since they are okay with the shitty cable company connection.
If you search HN you will find that negotiating plays a big role in the cost of leased line access.
https://news.ycombinator.com/item?id=3892854
Had to make this after seeing your post. https://i.imgflip.com/4vesal.jpg
Haha. I have a multi generational household. 4 adults, 2 children, and we're heavy Facetime/Apple TV users. 4 to 8 HD simultaneous video streams is normal, plus the security cameras. I also wanted to be able to stream from my server at home anywhere I am. And with 2 people working from home, it was an amazing decision.
haha I've been dreaming about a 10GB LAN setup; getting that on WAN would be awesome. I'd immediately establish elaborate multi-site NAS systems.
I bought a house 1+ year ago and had "good internet" connection on my must-haves. I remember visiting a house during the search and asking the listing agent about it. The guy was super candid, he basically told me no, there was no cable internet and that was the reason he knew it would be sitting on the market for a year at least.
To be fair, I've had FiOS for more than a year now, and this is the first incident I've had with them. I'm pretty comfortable saying they have a much better reputation than Comcast...
Unlimited data is $30 extra.
https://www.xfinity.com/support/articles/exp-unlimited-data
https://www.xfinity.com/support/articles/exp-unlimited-data
Sweet, I actually rent their modem so they said that they can give it to me for like $11 a month, just got it activated today. Thanks for the tip!
I've had FIOS service since 2007 or so here in SoCal and I can count the outages over this period on one hand.
I was a long-term Fios customer in SoCal, and they were indeed very reliable up until Verizon sold the local service to Frontier. Glad to hear they're not a problem for everyone at least.
[deleted]
Fios service is fantastic, and this morning's outage barely impacted me in Jersey City (only some disruption that seemed to be related to NYC area DCs more than Fios service itself). I have found their service to be very reliable for years now, with throughput that does not fluctuate (I get what I pay for regardless of the time of day) and lower latency than I had with Comcast. No regrets at all.
The only real complaint I have is that Verizon still seems to be unable to deploy IPv6. They have promised it for years and never seem to get past the testing phase.
The only real complaint I have is that Verizon still seems to be unable to deploy IPv6. They have promised it for years and never seem to get past the testing phase.
Wow, that's crazy. I use over 1TB a day. Forever thankful for Google Fiber.
How does does one use over 1TB a day?
Running scrapers can get you there pretty fast ;)
Wow 1TB per day? What is using all that bandwidth? (if you don't mind me asking)
I think the price dropped though. I was furious until I saw a lower bill. I also got a speed increase to 1.2gbps
I still wish I had other options in my area (nh)
I still wish I had other options in my area (nh)
What's worse is the 35 Mbps upload.
A fiber cut can affect any ISP.
The FCC said that 3mbps upload speed is all internet users need. They might not be in tune with modern internet usage
https://www.theverge.com/2021/1/21/22242635/fcc-ajit-pai-int...
https://www.theverge.com/2021/1/21/22242635/fcc-ajit-pai-int...
To be fair, that was just Ajit Pai, who also thought net neutrality was not important. Now that he's gone, I hope he gets subjected to 3mbps upload speeds for the rest of his days.
Well Ajit Pai only has 640K of memory so it’s understandable.
> “We find that the current speed benchmark of 25/3 Mbps remains an appropriate measure by which to assess whether a fixed service is providing advanced telecommunications capability,” the report reads.
Not "3mbps upload speed is all internet users need".
Not "3mbps upload speed is all internet users need".
That’s effectively the same thing, otherwise I don’t see the purpose of that statement.
We detached this subthread from https://news.ycombinator.com/item?id=25920055.
You vastly underestimate the cost to provide broadband to rural communities.
Why is this post not showing up on the HN homepage?
[deleted]
Maybe affected users can't upvote and not affected users don't care enough?
I'm nitpicking on the title, but "widespread" is one word, not two.
>There is a fiber cut in Brooklyn. We have no ETR, as of yet.
A fiber cut auto heals automatically via bgp. Absolutely no truth in this claim from verizon.
Meanwhile google, zoom, aws, and microsoft went down?
There isn't any shared datacenters here. There's at least 3 distinct datacenters.
So what's the truth? what actually went down?
A fiber cut auto heals automatically via bgp. Absolutely no truth in this claim from verizon.
Meanwhile google, zoom, aws, and microsoft went down?
There isn't any shared datacenters here. There's at least 3 distinct datacenters.
So what's the truth? what actually went down?
If 100 Gbps of capacity suddenly went away, then BGP attempted to route around it, and sent that traffic down 10 Gbps routers instead, which overloaded, or were not ready for the deluge of traffic, then those BGP routes could start flapping, at which point those routes would get pulled, forcing it onto yet other circuits, which may be 1 Gbps and that's how the outage continues to grow.
Too much capacity is centralized in certain providers, and if they go down there simply is no backup route that can handle the same traffic patterns.
Too much capacity is centralized in certain providers, and if they go down there simply is no backup route that can handle the same traffic patterns.
Yep, and it's possible that running low on ipv4 address space exacerbated the situation, so many more small routes than there used to be, maybe not all getting summarized during an outage they way they used to be, causing problems on peer bgp routers like you said.
Traffic is passing, although slowly.
I think this revealed that there is insufficient failover capacity.
I think this revealed that there is insufficient failover capacity.
Could be that. Also could be too much complexity, as opposed to capacity. Could have been timing didn't help, sometimes you take calculated risks or have to deal with additional complexity during migrations, etc.
I've even seen where playing it safe while planning to back out of a manual fix used during a weird outage (circuit went down in one direction or something, just needed to be manually shutdown to re-route everything) brought down the entire network when another outage happened before that evening, i.e. most redundancy plans protect against single failures, not all combination of two failures. They wanted to do all planned changes after business hours, like manually swinging traffic back to the primary hub site. They didn't check the weather on the other side of the country, and the power at their new super-duper data center went down that afternoon during a thunderstorm...
I wonder what percentage of fiber cuts don't cause noticeable outages. We'll never know, unless they let us root around in their ticket system.
As a customer it is fair to judge them on what you see, though. Some telcos screw up more than others. Best to use two.
(Typed over my phone, cuz my broadband is down!)
I've even seen where playing it safe while planning to back out of a manual fix used during a weird outage (circuit went down in one direction or something, just needed to be manually shutdown to re-route everything) brought down the entire network when another outage happened before that evening, i.e. most redundancy plans protect against single failures, not all combination of two failures. They wanted to do all planned changes after business hours, like manually swinging traffic back to the primary hub site. They didn't check the weather on the other side of the country, and the power at their new super-duper data center went down that afternoon during a thunderstorm...
I wonder what percentage of fiber cuts don't cause noticeable outages. We'll never know, unless they let us root around in their ticket system.
As a customer it is fair to judge them on what you see, though. Some telcos screw up more than others. Best to use two.
(Typed over my phone, cuz my broadband is down!)
Uh presuming that the connection is multihomed which is not the case for residential connections in NY.
It might sound conspiratorial, but when the WTC was attacked in 2001, the Internet slowed down a few days after. We'll probably never know, but I assumed it was deeper surveillance being implemented.
Every time an outage like this happens without a following postmortem, I wonder if it was a surveillance oopsie.
Every time an outage like this happens without a following postmortem, I wonder if it was a surveillance oopsie.
Why not take the simpler interpretation that millions of dollars of equipment was in the buildings that were destroyed, and it took a while for the network's self-healing algorithms to discover a new steady-state configuration that approximated the performance of what was suddenly lost?
Occam's razor: when presented with competing hypotheses about the same prediction, one should select the solution with the fewest assumptions.
https://en.wikipedia.org/wiki/Occam%27s_razor
Why is it that conspiracy theories seem to always require a more complex answer than the truth?
https://en.wikipedia.org/wiki/Occam%27s_razor
Why is it that conspiracy theories seem to always require a more complex answer than the truth?
Conspiracy theories are more fun to read.
One thing they always fall down is the idea that government bureaucracy is so quick and competent that they could do something like installing mass surveillance at a moments notice while the country is in chaos without it leaking due the next 20 years
One thing they always fall down is the idea that government bureaucracy is so quick and competent that they could do something like installing mass surveillance at a moments notice while the country is in chaos without it leaking due the next 20 years
> installing mass surveillance
It didn't have to be installed, it was already, although in a pre-Snowden era, governments doing Internet surveillance was the conspiracy theory. My speculation is that more international traffic got routed through the existing surveillance infrastructure, causing a bottleneck that slowly got better.
It didn't have to be installed, it was already, although in a pre-Snowden era, governments doing Internet surveillance was the conspiracy theory. My speculation is that more international traffic got routed through the existing surveillance infrastructure, causing a bottleneck that slowly got better.
Occam's razor is a guide to start with the simpler explanation. If the simpler explanation is not sufficient, you add more elements until it is.
My point is, the Internet rerouting around damage doesn't explain the slowdown, mostly because it happened days after, not immediately, meaning the damaged infrastructure was not critical for the global internet.
The attacks unleashed a huge intelligence operation, and it's reasonable to expect a parallel intelligence operation online. If the latter caused a traffic bottleneck or not, we can't know because it was never addressed, it's speculation on my part.
My point is, the Internet rerouting around damage doesn't explain the slowdown, mostly because it happened days after, not immediately, meaning the damaged infrastructure was not critical for the global internet.
The attacks unleashed a huge intelligence operation, and it's reasonable to expect a parallel intelligence operation online. If the latter caused a traffic bottleneck or not, we can't know because it was never addressed, it's speculation on my part.
Because it happened a few days after, not immediately, and there was nothing on the news about that. It was working fine immediately after, although people then mostly got their news from TV; a major event didn't increase bandwidth usage as drastically as today.
[deleted]