Hetzner subsea fibre cut outage(status.hetzner.com)
status.hetzner.com
Hetzner subsea fibre cut outage
https://status.hetzner.com/incident/de97a4ef-deb6-416e-92a7-9264d160519a
93 comments
This isn't Hetzner's own cable, and it's not just "any" cable - it's an Alcatel-Lucent + Cinia + EU project, with 150 terabits worth of bandwidth. It's a high profile target, but it's just as likely that it's just an anchor accident.
As Hetzner is a co-investor in the cable, it is partially their cable.
So do they own a solid chunk of it somewhere, or like a really long pie-slice sliver down the whole thing?
Neither, it's shared ownership. It's similar to if you buy a house with your partner: neither of you owns specific rooms, but you own the whole thing together. There's likely agreements about which fibers (there's 8 fiber pairs in the cable) and/or how much bandwidth everyone can use, though.
Why only 8? Why don't they put like 100 in there?
Look up how DWDM works; they don't need more than 8. The optical gear that goes on land, keeps getting more capable of driving ever more bandwidth over the same fiber...
Don't the inline amplifiers need to be upgraded too?
Presumably because the cable would be (much) heavier and thicker.
There is some movement towards having more pairs in newer cables, e.g. the Grace Hopper will have 16 pairs, cf. https://cloud.google.com/blog/products/infrastructure/announ...
There is some movement towards having more pairs in newer cables, e.g. the Grace Hopper will have 16 pairs, cf. https://cloud.google.com/blog/products/infrastructure/announ...
IIRC, it's mostly to do with power... every Xkm (100?) they need to put a power amp and have to splice 100 pairs at each point...
They didn't need 500+ terabits?
Well, cost for one. If that fiber isn't being used it's a terrible investment. But more generally it's probably not needed. The bandwidth of optical fibre is theoretically unlimited. A single fibre carries many wavelengths, a wave length is like a channel. This is known as DWDM(dense wave division multiplexing.) A single wavelength commonly caries 100 Gbs. Although recently advances have shown 700 Gbs for a single wavelength[1]. The number of wavelengths and hence bandwidth depends on the transmission gear used on both ends of the cable. You can upgrade the bandwidth by upgrading the transmission gear. The state of the art for years has been 8 pair systems but more recently developments have produced 24 and 36 pair systems.
[1] https://www.lightwaveonline.com/network-design/high-speed-ne...
[1] https://www.lightwaveonline.com/network-design/high-speed-ne...
Most submarine cables are consortiums. A very recent consortium for example is SEA-ME-WE 6:
https://www.submarinenetworks.com/en/systems/asia-europe-afr...
https://www.submarinenetworks.com/en/systems/asia-europe-afr...
Since it's a cable, I would imagine the latter.
It's probably split up by wavelengths.
So if they only have specific wavelengths, how do they keep them in the pie slice of cable they own? I just realized they likely don't own a cylindrical section of it somewhere because the light has to go in and out of it, only thing that makes sense is the long pie slice.
The light doesn't really work that way.
u/Denvercoder9 mentioned that there are 8 strands of fiber in this particular cable. Each strand can host multiple wavelengths of light, that do not really interfere with each other. There are different ways of dividing up the usable frequency range; here's a site with some examples: http://www.3coptics.com/News/11.html (Remember, light has a frequency & a wavelength. Human vision covers wavelengths from around 400 nm (nanometers, the UV end) to 700 nm (the IR end)).
The beams of laser light sent down a fiber strand have a wavelength (a central wavelength), and a width. The beam essentially needs exclusive ownership of wavelength ± width. The diagrams on the linked page show how channels are separated to give each channel a set width.
So, when an entity owns a part of a cable, they own a set of wavelengths, likely one or more contiguous blocks of wavelengths, that they can use for whatever. All owners will pay (probably based on the % of wavelengths they control) for a company to operate the cable (running the stations at each end, where the wavelengths are split out on to separate fibers for each owner). The owners will also pay for another company to do repairs as needed.
u/Denvercoder9 mentioned that there are 8 strands of fiber in this particular cable. Each strand can host multiple wavelengths of light, that do not really interfere with each other. There are different ways of dividing up the usable frequency range; here's a site with some examples: http://www.3coptics.com/News/11.html (Remember, light has a frequency & a wavelength. Human vision covers wavelengths from around 400 nm (nanometers, the UV end) to 700 nm (the IR end)).
The beams of laser light sent down a fiber strand have a wavelength (a central wavelength), and a width. The beam essentially needs exclusive ownership of wavelength ± width. The diagrams on the linked page show how channels are separated to give each channel a set width.
So, when an entity owns a part of a cable, they own a set of wavelengths, likely one or more contiguous blocks of wavelengths, that they can use for whatever. All owners will pay (probably based on the % of wavelengths they control) for a company to operate the cable (running the stations at each end, where the wavelengths are split out on to separate fibers for each owner). The owners will also pay for another company to do repairs as needed.
There's no way to own a physical "slice" of fiber optic cable. The wavelengths all travel together, and there are optical splitters/combiners at each end that separate/combine the wavelengths of light. Think of them like TV and radio broadcasts. Each station broadcasts on it's own frequency (wavelength) over the shared air (fiber).
A metaphor with light would be if you had two lights, one red and one blue. If you shine them both at the same spot, you'll see purple. If you put a prism in that spot, it'll split the red and blue light back out separately. At a very high level, that's what happens with fiber optics.
A metaphor with light would be if you had two lights, one red and one blue. If you shine them both at the same spot, you'll see purple. If you put a prism in that spot, it'll split the red and blue light back out separately. At a very high level, that's what happens with fiber optics.
When I lived in Cayman, the main fiber link (Maya-1[0]) was down for about a week (anchor drag, I think). Most of the island Caribbean has limited backup connectivity. The whole island only has one fiber line (around the whole western/southern Gulf of Mexico) and a crappy backup copper line to Jamaica. For a while, it was like being on a modem again. Forget about streaming anything.
This outage Hetzner is reporting isn't a big deal due to robust backups, but these things can cripple a small island outpost.
0: https://www.submarinecablemap.com/submarine-cable/maya-1
This outage Hetzner is reporting isn't a big deal due to robust backups, but these things can cripple a small island outpost.
0: https://www.submarinecablemap.com/submarine-cable/maya-1
> For a while, it was like being on a modem again
Internet in Congo if you're not on the UN satellite
Internet in Congo if you're not on the UN satellite
Bitten by a shark, or snagged on a russian submarine? Whichever the case, I just ran an ansible deployment over our cluster that's spread over their Frankfurt and Helsinki datacenters, and I didn't notice any difference. I guess that's to be expected, their line between those datacenters is just an optimization, the internet is still functioning as it always is and maybe Hetzner gets a slightly larger egress bill from their ISP as the inter-dc traffic gets rerouted the general internet.
> Bitten by a shark, or snagged on a russian submarine?
In all likelihood much less interesting, probably snagged in a fishing net or hit by an anchor.
In all likelihood much less interesting, probably snagged in a fishing net or hit by an anchor.
I imagined they would be hurried under the sea bed. Probably cheaper to place them on the surface?
I imagined it'd too impractical to bury them in case they need repairs, but apparently they're buried at least in some cases.
According to [1], a plough is dragged along the ocean floor behind the cable laying ship. The plough digs a trench that the cables are fed into. I can't really tell from the pictures whether the trench is covered back up.
[2] shows a really cool animation. Apparently ploughs like these can bury cables 2-3 meters into the ocean floor, and use a jet of water to help dig. Good stuff.
1: https://www.quora.com/How-are-major-undersea-cables-laid-in-... 2: https://www.youtube.com/watch?v=9hEDTRU_F2s
According to [1], a plough is dragged along the ocean floor behind the cable laying ship. The plough digs a trench that the cables are fed into. I can't really tell from the pictures whether the trench is covered back up.
[2] shows a really cool animation. Apparently ploughs like these can bury cables 2-3 meters into the ocean floor, and use a jet of water to help dig. Good stuff.
1: https://www.quora.com/How-are-major-undersea-cables-laid-in-... 2: https://www.youtube.com/watch?v=9hEDTRU_F2s
Wow that’s one very interesting specialized machine
Can’t rule out Russian military dolphins these days.
https://www.theguardian.com/world/2022/apr/27/russia-black-s...
https://www.theguardian.com/world/2022/apr/27/russia-black-s...
> Global Pandemic? Check
> Transnational Giga Companies controlling the world? Check
> Military Dolphins? Check
> Smuggling Gigabytes of Data via Implants? Not yet.
I rate the current situation as a strong 3/4 Johnny Mnemonics!
I rate the current situation as a strong 3/4 Johnny Mnemonics!
>Russian
>these days
https://en.wikipedia.org/wiki/Military_marine_mammal
>these days
https://en.wikipedia.org/wiki/Military_marine_mammal
There are not many sharks that can bite cables in the Baltic see due to low salinity, it's border-line freshwater.
The models of these cables that I've seen shows them in a jacket of steel so I doubt it's a shark. Or that was a joke that went over my head...
If any part of this interests you, you must read "Mother Earth, Motherboard" by Neal Stephenson. It's epic longform from the early Wired.
https://www.wired.com/1996/12/ffglass/
https://www.wired.com/1996/12/ffglass/
Then you will also like this: https://www.submarinecablemap.com
Happens quite often to subsea cables. I was renting a number of them and every now and again you'd get a break. You can follow the NOC's update emails:
Break detected
Ship has been commissioned
Ship is at harbor
Ship has sailed out to the cut
Fusing the ends
Service restored
Break detected
Ship has been commissioned
Ship is at harbor
Ship has sailed out to the cut
Fusing the ends
Service restored
What's the kind of scenario where you'd be renting an undersea cable?
Servers in different financial exchanges that need to talk to each other, reasonably fast.
How the hell do you fuse the ends of a massive cable like this at the bottom of the sea?!
This has pretty good information on it, and a couple videos:
https://www.onesteppower.com/post/subsea-cable-repair
https://www.onesteppower.com/post/subsea-cable-repair
> to fix a deep water cable, the ship has to use a grapnel, which grabs and cuts the cable, dragging the two loose ends to the surface. If needed, one end can then be hooked to a buoy and the other end brought on board. Cable has to be added to make the repair, since there is not enough slack to bring the cable up and cut a piece out. After the cable is retrieved and on board, in a repair room that looks like a laboratory, engineers repair the cable. Data cables can take up to 16 hours to repair, after which they are lowered back down to the sea bed in an omega or hairpin pattern to accommodate the extra length. There is new technology in development that would make in situ repair possible for power cables, preventing the need to pull the cables up to the surface.
You pull it up to the surface and fuse it on the boat.
(Second YT search hit so there might be better ones ) https://www.youtube.com/watch?v=FtEw19JX-UE
You find the ends, pick them up, and there's a kit that can fuse the ends to your new piece. I'm not sure about the details, but I think there are repeaters on the wire that need to be added as well.
A repeater on the ocean floor? Do they run power cables to it alongside the data all the way from the coast?
Yes, commonly there are high-voltage cables in there.
Nowadays, for shorter runs (that are still long enough to need repeaters) there are also purely optical repeaters where instead separate fiber strands deliver light from a pump laser.
Nowadays, for shorter runs (that are still long enough to need repeaters) there are also purely optical repeaters where instead separate fiber strands deliver light from a pump laser.
Do they just fuse any strand to any strand and resort everything at the end or?
For submarine cables, there aren't that many strands, so getting it right is probably worth it, for terrestrial fiber, there may be 100+ strands, so they usually can sort things out at the ends, and I think you just fuse any strand to any strand. Sometimes, only some strands break and you can reallocate strands by priority until the cable can be fixed.
Strands in a bundle would likely have color coded sheaths. As long as you're not colorblind, it's pretty straightforward matching what gets spliced where.
Find it, pick it up, patch it on deck, lower it back down.
Best guess this is the cable based on Hetzner investment and landings:
https://www.submarinecablemap.com/submarine-cable/c-lion1
https://www.submarinecablemap.com/submarine-cable/c-lion1
That seems to be correct.
https://www.hetzner.com/presse-berichte/2016/01/156818
> Hetzner Online GmbH, the first major Central European provider of data center services to construct a new data center park in Finland, is also a user and co-investor in the Cinia CLion1 cable connection.
https://www.hetzner.com/presse-berichte/2016/01/156818
> Hetzner Online GmbH, the first major Central European provider of data center services to construct a new data center park in Finland, is also a user and co-investor in the Cinia CLion1 cable connection.
I was today years old when I learned Hetzner has its own undersea cables.
It's not their cable.
They are a co-investor... so its partially their cable.
And as I'm an EU citizen paying top taxes, plenty of which went into funding the project, it's partially my cable as well.
Fix your damn cable @daniel_w!!!
Lol,
To be specific - you invested in EU taxes, while others invested in cable.
To be specific - you invested in EU taxes, while others invested in cable.
> Recovery of the service is in progress
(Last update: 2022-05-24 05:49 UTC+0)
> There is currently an ongoing fibercut at our subsea cable. Our cable provider is working on fixing this. There is currently no ETA. (Last update: 2022-05-23 14:09 UTC+0)
> There is currently an ongoing fibercut at our subsea cable. Our cable provider is working on fixing this. There is currently no ETA. (Last update: 2022-05-23 14:09 UTC+0)
The use of that cable is just to reduce latency. There are many other paths between their datacenters, so there won't be any real breakage.
[deleted]
When something like this happens, does the traffic get distributed on other operational fibers that have sufficient unused bandwidth or do the operators fire up some dark fibers to carry the load? Or maybe it’s both?
It depends. Generally the larger carriers and consumers of fibers have a network that is able to route around outages (fiber cuts like this are basically routine) just the same as on land. If they don't have enough, then they'll need to either reduce demand or rent some capacity from someone else.
> then they'll need to either reduce demand
Worst case everything just gets a bit slower (right?), depending on how much traffic went through that cable.
Worst case everything just gets a bit slower (right?), depending on how much traffic went through that cable.
If, like me, you didn't know how these get cut I'll save you the trouble. The most likely cause is improper boat anchoring. Never shark bites.
Sharks do bite these cables!
https://amp.theguardian.com/technology/2014/aug/14/google-un...
https://amp.theguardian.com/technology/2014/aug/14/google-un...
Can they somehow tell exactly where the cable was cut?
Yeah there's special tech for that! Has something to do with sending some frequency over the line and measuring how long it takes to "reflect" off of the breakage and back to the shore.
The term is time domain reflectometry, TDR. It used to be really expensive, and now it's filtered down into medium-cost optical adapters.
In the non-optical domain, that used to be a feature in the most expensive cable measuring gears (to the tune of €25k about 20 years ago).
I had the pleasure of using one of the devices that had that when I wired my old house. Borrowed the kit from ... an institution that allowed me to get their gear for a weekend. (Yes, I knew people in there.) Having the device tell me that a given strand in a CAT5e cable was faulty at <this many meters> from my location made it surprisingly easy to detect where I'd messed up.
I had the pleasure of using one of the devices that had that when I wired my old house. Borrowed the kit from ... an institution that allowed me to get their gear for a weekend. (Yes, I knew people in there.) Having the device tell me that a given strand in a CAT5e cable was faulty at <this many meters> from my location made it surprisingly easy to detect where I'd messed up.
Many (low-cost) Mikrotik routers have this feature built in now: https://wiki.mikrotik.com/wiki/Manual:Interface/Ethernet#Det...
I first saw the feature on a motherboard integrathed Ethernet chip over 10, possibly over 15 years ago.
I was going to say that this was invented by none other than Heaviside, but it turns out his technique relied on static resistance measurement to find the cable fault. Similar concept though. And in 1861!
This video explains and show the same concept for copper cables in a very simple and hands on way:
"Cheap and simple TDR using an oscilloscope and 74AC14 Schmitt Trigger Inverter"
https://www.youtube.com/watch?v=9cP6w2odGUc
This very old video does it all explain better than all todays videos. The models are just so simple, no animations.
https://m.youtube.com/watch?v=DovunOxlY1k
https://m.youtube.com/watch?v=DovunOxlY1k
Some NICs and managed switches can do it as well. Well, I should say that most can do it (as in the hardware is capable) but only some actually expose the functionality.
Yes, using an OTDR [0].
These used to be big, expensive, standalone devices (I'd occasionally use one in a previous job). Nowadays, there are portable, handheld units specifically for identifying where a break has occurred and the same, basic functionality is often even built into the gear used at either end of connections (e.g., even the "enterprise" switches -- from Brocade and Cisco -- that I use at home).
---
[0]: https://en.wikipedia.org/wiki/Optical_time-domain_reflectome...
These used to be big, expensive, standalone devices (I'd occasionally use one in a previous job). Nowadays, there are portable, handheld units specifically for identifying where a break has occurred and the same, basic functionality is often even built into the gear used at either end of connections (e.g., even the "enterprise" switches -- from Brocade and Cisco -- that I use at home).
---
[0]: https://en.wikipedia.org/wiki/Optical_time-domain_reflectome...
Dumb question: I thought TCP/IP was designed to route around cut fibre cuts.
Also to be clear, this is handled by BGP, not by TCP/IP. BGP is what ASs use to identify who connects to what so that they can route IP appropriately.
That's pretty much what happens but no amount of routing can make up for the loss of capacity when you physically take out a part of the network. You end up with less total available bandwidth and more latency.
It is, and a cut like this rarely causes a complete outage, but it depends on whether and what kind of backup connectivity your ISP has, and the connectivity of your destination. Bigger entities tend to have more backups. Smaller entities tend to have fewer backups, and often the backups are much lower bandwidth/quality, so it's typical for service to degrade.
Don't know about Hetzner but cloud traffic these days are not routed over the public Internet in that way.
[deleted]
Aha!
That's why my status server has been acting weird saying things were offline when they weren't.