Starlink is currently experiencing a service outage(starlink.com)
starlink.com
Starlink is currently experiencing a service outage
https://www.starlink.com/us
304 comments
Well my boss is in a cabin in the wilderness somewhere and his only access to the internet is via Starling, so that's great news!
I've heard of IP over pigeons, but using starlings is innovative.
Learn something new every day! I had never heard of RFC 1149[1] and was only familiar with Pigeonrank[2], another short-lived product.
[1] https://www.rfc-editor.org/rfc/rfc1149 [2] https://archive.google/pigeonrank/
[1] https://www.rfc-editor.org/rfc/rfc1149 [2] https://archive.google/pigeonrank/
We've got a family of nuthatches that set up an extensive neighborhood mesh network. Quite impressive!
Quid pro quo, Clarice
The RFC-1149 pigeon protocol has its own theme song.
https://www.youtube.com/watch?v=tIzLlODqyCE
They're reliable, provided there's a cherry tree orchard at the destination.
Starlings are lower bandwidth than pigeons
Sometimes you have to keep repeating a packet, they're very good at it.
I too am in the wilderness and Starlink is my only broadband, but I was smart enough to keep my 1.5Mbps WISP connection (radio mounted 170' up one of my redwoods) so I'm still in good shape for basic communication, even some streaming and videoconferencing, though those aren't great this morning.
Also, after my Starlink terminal rebooted a few times on its own over the last hour or so, things are looking good again.
Also, after my Starlink terminal rebooted a few times on its own over the last hour or so, things are looking good again.
Are you using something like a CradlePoint for your WISP?
If it 1,5 Mb and radio mount on a tower I would more guessing toward the side of PTP line of sight Link or such Mikrotik, Unifi instead of cellular cradlepoint.
In the US:
Ubiquiti, Cambium, probably not Tarana because that's way too expensive for a 1.5Mbps sub. Mikrotik radios are fairly rare here, but the routers and switches are everywhere.
Ubiquiti, Cambium, probably not Tarana because that's way too expensive for a 1.5Mbps sub. Mikrotik radios are fairly rare here, but the routers and switches are everywhere.
Thats interesting ! Mikrotik routers and switches are very cheap, I wonder why they one product line is extremely common but the radios are rare rather than roughly equivalent spread ?
Mikrotik has not (yet?) gone down the path of shipping sector radios supporting massive MIMO (Multiple Input Multiple Output -- an 8x8 array can deliver 8 times as much data as 1 stream within the same channel size) as vendors like Cambium have. The additional capacity on tower sectors from additional streams combined with MU-MIMO (Multi-User MIMO that transmits to multiple subscriber radios at the same time based on the number of streams the transmitter supports combined with client radios that are typically 2x2) is a great way to gain capacity where spectrum is exhausted. Higher order QAM support like the new 12 bits per symbol modulation in 802.11be tends not to be so useful for WISPs as getting the SNR to the level required to support it tends to be is challenging, especially in 6 GHz spectrum where transmit power limitations require much higher gain antennas for client radios (at least here in Canada where EIRP is limited to 36dBm).
Mikrotik is still quite heavily used for their routers and switches among WISPs. They are the most affordable vendor for 100Gbps switches. Pity we don't have a better open source software stack for their hardware.
Ubiquiti is also actively neglecting the WISP market. Anything that isn't UniFi seems to be mostly abandoned these days.
Mikrotik is still quite heavily used for their routers and switches among WISPs. They are the most affordable vendor for 100Gbps switches. Pity we don't have a better open source software stack for their hardware.
Ubiquiti is also actively neglecting the WISP market. Anything that isn't UniFi seems to be mostly abandoned these days.
Just generally not as reliable or as advanced as the competition for not much more, generally. They are doing some interesting work with LTE in Latvia though.
Hasn’t he heard of ravens? Much more reliable.
Yeah, but all they say is "nevermore".
[deleted]
in my day we used pigeons
Digression ... you must be using "wilderness" in it's colloquial form, because technically, if he's in a cabin he can't be in wilderness. Federally designated wilderness areas don't allow permanent human habitation.
Furthermore, the planet we are on cannot possibly be in space. Space is mostly empty, has no breathable atmosphere, there are no trees or grocery stores there, etc.; Earth is quite the opposite.
The cabin is located in the wilderness, not part of it, and it’s real wilderness, not a wilderness designated area
Pro-tip: Don't host your status / outage page on your own infrastructure. We learned this lesson the hard way at AWS with S3.
True, but nothing tells the story quite like a "no healthy upstream" status page
[deleted]
Starlink.com is hosted on Azure judging by its DNS records
Pro-tip: Host your status/outage page on your own infrastructure... :-) If you can't reach it...You are down :-)
It’s like when our vendor’s tech support number tells you you’re 300th in queue (or busy cuz their lines are saturated) instead of the usual 2nd or 3rd.
You just hang up cuz you know it’s everyone.
You just hang up cuz you know it’s everyone.
I broadly agree with what you're saying, but, that's not the issue here.
They don't even have a dedicated status/outage page, afaik.
The website being down is a more classic problem. The outage probably increased traffic to their website by 1000x if not more and the infrastructure for the website simply couldn't cope.
Good lesson on keeping your status infrastructure simple and on something which is highly scalable.
Having a CDN where the main page of their site was 99% cached globally would have probably mitigated this issue.
They don't even have a dedicated status/outage page, afaik.
The website being down is a more classic problem. The outage probably increased traffic to their website by 1000x if not more and the infrastructure for the website simply couldn't cope.
Good lesson on keeping your status infrastructure simple and on something which is highly scalable.
Having a CDN where the main page of their site was 99% cached globally would have probably mitigated this issue.
maybe its a double double triple bluff
How would you host a status page on Starlink? Are there web servers that are only connected to a satellite connection in house or something? Or is this just speculation that's how their marketing website works.
Starlink has servers for the website and subscriber sites. Importantly, they have servers for controlling the satellite network. They also have multiple gateways for satellites to connect to the network.
A problem in their network would take down the sites, and maybe the control plane for satellites.
A problem in their network would take down the sites, and maybe the control plane for satellites.
Starlink runs a terrestrial network to support the vehicles obviously. They have to do peering at POPs in cages on the ground like any other ISP.
[deleted]
https://radar.cloudflare.com/as14593?dateRange=1d shows a sharp drop in traffic, but no BGP changes before the drop. We can see BGP changes after the drop, so maybe some datacenter/control plane thing.
That is a neat tool
Curious as to what could cause a global outage like that. The system consists of birds in the air and individual ground stations. There's no big choke point that I'm aware of that could cause the whole shebang to go dark. Am I missing something here?
BGP is the canonical way to take your entire global network offline within seconds. Glancing at BGP looking glasses, Starlink's prefixes seem to still be announced, but there could still be an accidental blackhole or routing loop within their AS, or something broken in one of their transit providers.
No idea if that's what's going on, but routing protocols are one of a few effectively global control planes that can go wrong very quickly like this.
No idea if that's what's going on, but routing protocols are one of a few effectively global control planes that can go wrong very quickly like this.
If it were BGP/routing, you would think we'd be able to still get a signal and the modem would think it's healthy (although maybe not if the issue prevented us from obtaining our public IP), we just wouldn't be able to route to any dst. In the current case we don't have a signal (orange light on the modem)
Yes, before the drop out my traffic was coming from a downlink station in Bulgaria, on an IP on AS14593
Traceroute from "the internet" back to that IP reaches AS14593 just fine, and my endpoint doesn't get beyond the first hop of the local starlink router.
Whatever it is, it doesn't look like a peering problem
Traceroute from "the internet" back to that IP reaches AS14593 just fine, and my endpoint doesn't get beyond the first hop of the local starlink router.
Whatever it is, it doesn't look like a peering problem
https://mtr.ping.pe ftw for MTR from "the internet" ? :)
From various monitoring points I have on multiple internet connections.
One of the promises of starlink was it would stay in space as long as possible before being downlinked, giving far lower latency, alas that hasn't happened yet, and traffic will run thousands of miles in the wrong direction before being downlinked. For example from one location to another I have 360ms via Starlink but just 200ms rtt via local provision (5g p2p wireless then optical). On another it used to downlink in Lagos, but now it downlinks in Nairobi, meaning traffic to Lagos routes Nairobi -> Marseille -> Lagos, taking far longer than it used to. A shame really.
One of the promises of starlink was it would stay in space as long as possible before being downlinked, giving far lower latency, alas that hasn't happened yet, and traffic will run thousands of miles in the wrong direction before being downlinked. For example from one location to another I have 360ms via Starlink but just 200ms rtt via local provision (5g p2p wireless then optical). On another it used to downlink in Lagos, but now it downlinks in Nairobi, meaning traffic to Lagos routes Nairobi -> Marseille -> Lagos, taking far longer than it used to. A shame really.
Does the orange light specifically mean no RF link at all? Or does it include anything that prevents the modem from getting an IP address and route configuration? If the latter, BGP could still be at fault if it took out access to the control planes on the ground. But again all just guessing, from the outside all I see is the BGP routes are still being announced, and everyone seems to be seeing 100% packet loss and zero traffic.
Right, good point that could be the case, those were just my assumptions and probably jumping to conclusions on my part speculating that orange means no signal (don't actually have any idea :) ). Imagine it could be any of what you said too.
That sounds bad…
A network of satellites gives you entirely new and exciting ways of taking your network offline, such as bricking them with a firmware update and no way to actually get up there and fix it.
Imagine being that dev who pushed the bad patch - Crowdstrike but x100
Crowdstrike’s fuckup was at the company level.
Whoever is doing immediate global deployments and/or any prod deployments without verified testing is just wrong as a corporate culture.
Whoever is doing immediate global deployments and/or any prod deployments without verified testing is just wrong as a corporate culture.
Elon saw yesterday's article about The Promised LAN, and said "What if we connected that instead of the Internet...?"
https://news.ycombinator.com/item?id=44661682#44663409
Instead, it seems he's created the LAN of the lost
it's pretty amazing the amount of damage a BGP oopsie can do. Also, you can fit pretty much all the BGP admins for the entire Internet in one large room.
If by “room” you mean “Wembley stadium”, maybe.
Messing up with BGP makes you feel alive, that’s for sure.
But hey, if you haven’t caused an incident yet, that just means you’re still in onboarding. Those SLA downtime budgets are there to be spent.
But hey, if you haven’t caused an incident yet, that just means you’re still in onboarding. Those SLA downtime budgets are there to be spent.
I rebooted my terminal and I can’t tell for sure if it sees any satellites. It looks like it doesn't.
It says it didn’t, and it says the “which way is down” thing hasn’t converged. Occasionally, the signal to noise ratio light in the app goes gray which means < 3.
It also rebooted itself.
Before the first reboot, 30% of pings went through. It’s almost like the azimuth or some other timely but cached data was corrupted.
It says it didn’t, and it says the “which way is down” thing hasn’t converged. Occasionally, the signal to noise ratio light in the app goes gray which means < 3.
It also rebooted itself.
Before the first reboot, 30% of pings went through. It’s almost like the azimuth or some other timely but cached data was corrupted.
It's always a bad route that was introduced during a planned upgrade
I thought it was always DNS
I'm hardly an expert, but it seems like usually stuff like this happens to major backbones because of a bad configuration being rolled out too fast. Maybe the pushed a bad DNS/BGP/<insert networking acronym here> update to every satellite and/or ground station near simultaneously.
It just takes one
The fleet of satellites is still managed centrally, even if the physical infrastructure is distributed globally to be able to see satellites in different hemispheres, at some point they are very likely managing these from a single control plane
I wonder what they do for out of band on satellites.
It's not like you can have a tech go out and reboot it or reinstall windows.
It's not like you can have a tech go out and reboot it or reinstall windows.
I bet they have some pretty robust hardware watchdogs that allow rebooting, booting from firmware OTA etc. in case of local data corruption.
Here's a great talk on roughly this topic: https://media.ccc.de/v/38c3-hacking-yourself-a-satellite-rec...
Here's a great talk on roughly this topic: https://media.ccc.de/v/38c3-hacking-yourself-a-satellite-rec...
Their space infra is all disposable. Its GUGD.
Each individual satellite, yes. But killing all of them at the same time, e.g. due to a bad software or configuration update, might be the end of the company.
yeah, that would be an interesting dillema. Lucky for us, they technically arn't designed to be anything but disposable, so they'll fall from the heavens eventually.
Cyberattack. Given the usage in active wars, it's a big juicy target. There's precedent too; Russia hacked Viasat at the start of the Ukraine war. They actually bricked user terminals in that attack.
FYI, Vodafone and other major European mobile networks were down during the last few days.
This is hybrid warfare, not a coincidence.
Russia needs to learn some manners, the hard way.
This is hybrid warfare, not a coincidence.
Russia needs to learn some manners, the hard way.
As a former customer, I have complete faith in Vodafone’s ability to take themselves offline without any external help.
For now Russia is teaching us on how to use dishwasher chips in a smart way.
Incredibly, Americans still think they have any chance of teaching Russia or China "manners". The hubris of a decadent empire.
And of course, they have to downvote to hell. They need to keep the illusion that the US is still the most powerful military, the most powerful economy, and that AI will save their hegemony.
Really sad.
And of course, they have to downvote to hell. They need to keep the illusion that the US is still the most powerful military, the most powerful economy, and that AI will save their hegemony.
Really sad.
cluckindan(2)
elzbardico(1)
Not a very technical root cause analysis you have there. Can you leave geopolitics out of it until it's confirmed?
Other social media networks may tolerate this but this is not HN worthy.
Other social media networks may tolerate this but this is not HN worthy.
My most downvoted post by telling people to keep geopolitics out of a technical discussion. Gfy, thought you were all better.
Edit: funny, can't even access the replies..
Starlink VP of engineering Michael Nicolls tweeted that the service has "now mostly recovered from the network outage" after two and a half hours.
"The outage was due to failure of key internal software services that operate the core network. We apologize for the temporary disruption in our service; we are deeply committed to providing a highly reliable network, and will fully root cause this issue and ensure it does not occur again," the tweet, posted at 3:23 p.m. PT, says.
Edit: funny, can't even access the replies..
LunaSea(1)
If it is a hack there can't be that many suspects, just narrow it down to people/groups/countries that hate Elon Musk.
I see what you did there
Sure, it's too bad that almost all of that hatred is due to propaganda instead of anything real.
If anyone has benefited humanity over the last couple of decades, it's Elon Musk! I'd argue he's done more than anyone!
If anyone has benefited humanity over the last couple of decades, it's Elon Musk! I'd argue he's done more than anyone!
They still have tons of ground-based infrastructure that has all the regular failure modes. I suspect the satellites also route IP traffic statically, and they only do so to a fixed set of ground gateways all operated by Starlink (compared to e.g. Iridium, which makes in-space switching decisions and can at least continue ongoing phone calls between two devices without actually needing a reachable ground gateway).
National telcos manage to take themselves out countrywide from time to time too, so I'd assume the same usual suspects apply: Routing problems, botched global configuration changes etc.
If anything, due to centralization, ubiquitous and effectively free connectivity, and centralization/automation of configuration changes, the blast radius of any given error or malicious action has probably been steadily increasing over the years.
National telcos manage to take themselves out countrywide from time to time too, so I'd assume the same usual suspects apply: Routing problems, botched global configuration changes etc.
If anything, due to centralization, ubiquitous and effectively free connectivity, and centralization/automation of configuration changes, the blast radius of any given error or malicious action has probably been steadily increasing over the years.
Same things have been said of eg the massive Facebook years ago. It was DNS, which affected service discovery across the whole org. The massive Roblox outage was also caused by service discovery failure (Consul). If your machines don't know which other machines to talk to, shit goes bad fast.
This happens enough that it seems like someone would have created a better alternative.
Everything has failure modes, the only difference is how hard they are to trigger
A bad configuration pushed to network
that's going to be a lot of crew dragon trips to go hold the power button down for 5 seconds on all those satellites to reset to factory default.
I volunteer.
Almost always. Most people look at global networks like black magic and don’t realize this.
For most large businesses, 90%+ of major network events are caused by internally-driven network config changes.
Depending on how far along the business is towards automation, a proportion of the 90% can be attributed to a human going “off script” - I.e.: making a change that had not been reviewed.
For most large businesses, 90%+ of major network events are caused by internally-driven network config changes.
Depending on how far along the business is towards automation, a proportion of the 90% can be attributed to a human going “off script” - I.e.: making a change that had not been reviewed.
My money is on a misconfigured switch or hub that allowed a broadcast storm that shut down all of their edge devices!
Additional points for a user to cause the loop by plugging an rj-45 where they weren't supposed to, despite no warning or label telling them not to!
Additional points for a user to cause the loop by plugging an rj-45 where they weren't supposed to, despite no warning or label telling them not to!
Are you saying they are one big L2 domain?
I love when devs talk about networking.
I love when devs talk about networking.
They said "hub"
Heh.
Heh.
I think the network putting out warning labels to prevent loops is probably worse off than the place that just forgot to leave some form of loop prevention enabled.
...in space?
Cat 9 cables are rated for low crosstalk links in LEO, though I'm not sure how Starlink solved the problem where the cable would tend to wind around the earth because of the speed difference between LEO and the Earth's rotation.
Well, obviously the sats have to juggle cables. Only connect to one for a little bit and then toss it to the next in the train...
I was browsing Reddit and X and a lot of people saying that their base stations couldn’t even find a satellite. I find that pretty odd and can’t make sense out of it if it was simply a network problem.
You would expect to be able to at least connect to a satellite unless the satellite has no Internet connection so it does not make itself available to be connected to you.
Anyone have any deeper knowledge about this for the curious?
You would expect to be able to at least connect to a satellite unless the satellite has no Internet connection so it does not make itself available to be connected to you.
Anyone have any deeper knowledge about this for the curious?
The usual dark triad: DNS, BGP and IPv6
It is always the DNS.
I'd bet on a bad update or configuration change. (likely one that prevents the affected systems from reaching the internet and being automatically rolled back)
Too many HEILs near the rack power buttons
I mean, why do systems go down at all? a lot of big outages are simple misconfiguration or cascading failure from what seemed like small changes. It's rarely due to the physical constraints of the world
Software?
/s not-/s BGP?
Kessler syndrome?
shutdown now. Forgot the /r
There is a non-zero probability that Musk personally tried to use Grok to make and roll out a software update.
[deleted]
Unless Musk is outside the country, the outage started in the late morning [1]. If this happened at 3AM, your hypothesis would be more likely.
[1] https://www.pcmag.com/news/spacexs-starlink-hit-by-major-out...
[1] https://www.pcmag.com/news/spacexs-starlink-hit-by-major-out...
Hmm. It may be that I myself am too tired right now, but I'm afraid I don't follow the logic on that one?
But thanks for the link anyway.
But thanks for the link anyway.
Took ketamine, went blackout at 3Am, told hself future me can handle yhis, scheduled deployment in morning. Future self woke up this afternoon completely unaware.
I'm posting this from my Starlink connection, been working fine here. I'm close to the arctic circle so maybe the polar orbiting satellites are not affected
They're all the same satellites. It's actually impossible to have a satellite just orbit at the poles.
Seems to prove GPs point?
There are satellites on a polar orbit, which will cover connections at the poles, but those satellites still cover a lot of non-polar ground.
If polar orbiting satellites are still working we'd expect to see intermittent or degraded connectivity away from the poles rather than complete outage?
There are satellites on a polar orbit, which will cover connections at the poles, but those satellites still cover a lot of non-polar ground.
If polar orbiting satellites are still working we'd expect to see intermittent or degraded connectivity away from the poles rather than complete outage?
I saw some people report that their dishes are pointing in unusual directions so it would explain that, and that my service is fine since I only use the polar orbiting ones ¯\_(ツ)_/¯
Not without using a LOT of fuel .. <3
What a great writing prompt! Would love to read read the rest of this as a short story. Any takers?
Edit: Wow, that’s the fastest any of my HN posts have ever been buried. Still, as an old guy, I have to say it’s pretty awesome to be living in a future where polar satellites can keep someone in touch with the rest of civilization.
Edit: Wow, that’s the fastest any of my HN posts have ever been buried. Still, as an old guy, I have to say it’s pretty awesome to be living in a future where polar satellites can keep someone in touch with the rest of civilization.
[deleted]
why would there be a rest to this
So, if (just spitballing here) Russia were to have wanted to take out Starlink in order to handicap Ukrainian operations, and they didn't want to do something as visible as actually taking out the satellites, what would be the most likely way for them to have done it? And how would we be able to check if that's what happened?
No idea if this is plausible at all, just raising the question for anyone who knows more about this system than I do.
No idea if this is plausible at all, just raising the question for anyone who knows more about this system than I do.
I would still bet against the Russians taking down all of Starlink. If they had a zero day that could do that, they'd save it for just before a big war.
But Russian low-intensity hybrid warfare, including infiltration and sabotage, is very real. Particularly in Europe and especially so in eastern Europe. There's a lot of nasty things going on in border areas that isn't widely reported.
But Russian low-intensity hybrid warfare, including infiltration and sabotage, is very real. Particularly in Europe and especially so in eastern Europe. There's a lot of nasty things going on in border areas that isn't widely reported.
No good way to test your PoC without burning your 0day; I don’t imagine that the OV firmware is public or even “available”.
Things get patched over time, Nato has their own comms systems whilst Ukraine relies on this third party, and the US just accepted the EU funded weapons sale so their Trump advantage seems to have disappeared with his patience in P's promises.
Thus using the weakness to launch a coordinated summer offensive now for a decisive breakthrough before US weapons supplies reaches full speed again seems like a reasonably good choice.
Conquering Ukraine is the key for restoring the USSR, Belarus is already under the yoke and Khz,etc won't dare resist after Ukr falls. P sees a victory here as moving the linchpin that makes him immortal in historybooks.
Thus using the weakness to launch a coordinated summer offensive now for a decisive breakthrough before US weapons supplies reaches full speed again seems like a reasonably good choice.
Conquering Ukraine is the key for restoring the USSR, Belarus is already under the yoke and Khz,etc won't dare resist after Ukr falls. P sees a victory here as moving the linchpin that makes him immortal in historybooks.
> what would be the most likely way for them to have done it?
Ensure Starlink hires an agent. Establish skills / access permissions. Push an update which bricks both the update system and the running system, so the satellites become useless.
Ensure Starlink hires an agent. Establish skills / access permissions. Push an update which bricks both the update system and the running system, so the satellites become useless.
They would exfiltrate any secret technology first, then do their best to burn the whole network.
Potentially you could burn the constellation with reentry.
Potentially you could burn the constellation with reentry.
> Potentially you could burn the constellation with reentry
Just put them into a spin with a strong propellant burn while compromising ground stations. Even if the fleet recovers, its lifespan will have been significantly reduced by the propellant expenditure.
Just put them into a spin with a strong propellant burn while compromising ground stations. Even if the fleet recovers, its lifespan will have been significantly reduced by the propellant expenditure.
Spin would be indeed the most problematic, but the Starlink satellites only have krypton hall thrusters. Very efficient but also very low thrust - you you could not actually significantly waste fuel this way even if you wanted to.
You'd need to have actual knowledge of the hardware systems and serious access to do anything interesting with thrusters. It may be possible. But just bricking the updates is likely way simpler and less protected.
You have the resources of a country to support any activity. Need computer experts? Take your pick? Astronomers? Mathematician? They have all the expertise required.
The agent getting the right access or finding the right people on the inside and exploiting them successfully without trigger counterintelligence is the hard part.
The agent getting the right access or finding the right people on the inside and exploiting them successfully without trigger counterintelligence is the hard part.
First you'd have to extract the documentation which may not be available to everyone. I'm just saying that in limited time, the purely software side of starlink is likely much less protected from both access and changes than anything to do with thrusters usage.
Is there any secret technology in Starlink that is not taught in Russian Universities and not described in Soviet scientific works?
Fundamentally maybe not, but knowing what they are actually doing/using is still useful.
If you can get schematics or source code, you can put it to use as well.
And then you can also use communications logs and HR to find other human targets.
If you can get schematics or source code, you can put it to use as well.
And then you can also use communications logs and HR to find other human targets.
[deleted]
It's essentially impossible for anyone to disable the Starlink network by "taking out satellites" through physical means. There aren't enough anti satellite missiles in the world to make a dent, it would cost trillions to manufacture enough, and SpaceX can launch satellites faster than anyone could manufacture interceptors anyway. And Kessler syndrome is impossible at the altitudes Starlink uses.
Cyberattack is the only plausible way to take down Starlink.
Cyberattack is the only plausible way to take down Starlink.
The anti-satellite devices could be deployed in the same manner as the Starlink satellites. And they wouldn't need communications equipment, so they could be lighter and cheaper to launch. And you really wouldn't need to take them all out, just enough to make communication unreliable.
Starlink launches reduce costs by launching a bunch of satellites with similar orbits on the same vehicle, replacing one or a few satellites is going to cost a lot more per satellite. So just disrupting the network is a lot cheaper than fixing it.
Thought the cheapest is still probably paying an existing employee to break some stuff.
Starlink launches reduce costs by launching a bunch of satellites with similar orbits on the same vehicle, replacing one or a few satellites is going to cost a lot more per satellite. So just disrupting the network is a lot cheaper than fixing it.
Thought the cheapest is still probably paying an existing employee to break some stuff.
Taking out one or even a few dozen satellites isn't going to make communication unreliable. They can redistribute themselves to fill holes. You'd need to take out thousands, requiring hundreds of launches at least. And neither Russia nor China has reusable rockets, so the costs would be much higher than SpaceX's costs. The interceptors would take a long time to spread out to reach their targets if they were launched in groups like Starlink is, so it wouldn't be a surprise attack and SpaceX would have time to prepare. They would need to start launching on-orbit spares for each orbital shell, but there are only a few shells, not hundreds.
And in a few years when Starship is launching Starlink, the economics will be tilted even more wildly in SpaceX's favor.
And in a few years when Starship is launching Starlink, the economics will be tilted even more wildly in SpaceX's favor.
>>There aren't enough anti satellite missiles in the world to make a dent, it would cost trillions to manufacture enough
Few big nukes are enough to take out all satellites.
Few big nukes are enough to take out all satellites.
>Few big nukes are enough to take out all satellites.
Um, no. Not even vaguely remotely. "Thousands of satellites" is a lot by historical human space standards, but compare that to the number of cars you see every day and then remember these are spread out over an area bigger then the surface of the entire planet. Don't be fooled by the simulation maps you can find online, where the satellites and tracks are shown thousands of times bigger then they would be to scale because otherwise they couldn't be seen. They are very, very far apart. Even on the ground, "a few big nukes" wouldn't do diddly. And nukes do less, not more, damage in space. A lot of the damage enhancement effects of nukes comes from their interaction with atmosphere. In a vacuum, it's only direct radiation which falls by the square of the distance, and satellites are of course designed to handle plenty of thermal and ionizing radiation all the time already by virtue of being designed for space. But Starlink sats aren't so high that Kessler syndrome is a concern either, there's plenty of atmospheric drag to bring debris down without active boosting.
GP is correct and I don't understand the downvotes: as far as physical resilience goes the Starlink constellation is a pretty damn hard target on an absolute basis, even before we get into the geopolitics of some country trying to shoot down US property that is also a US strategic asset.
Um, no. Not even vaguely remotely. "Thousands of satellites" is a lot by historical human space standards, but compare that to the number of cars you see every day and then remember these are spread out over an area bigger then the surface of the entire planet. Don't be fooled by the simulation maps you can find online, where the satellites and tracks are shown thousands of times bigger then they would be to scale because otherwise they couldn't be seen. They are very, very far apart. Even on the ground, "a few big nukes" wouldn't do diddly. And nukes do less, not more, damage in space. A lot of the damage enhancement effects of nukes comes from their interaction with atmosphere. In a vacuum, it's only direct radiation which falls by the square of the distance, and satellites are of course designed to handle plenty of thermal and ionizing radiation all the time already by virtue of being designed for space. But Starlink sats aren't so high that Kessler syndrome is a concern either, there's plenty of atmospheric drag to bring debris down without active boosting.
GP is correct and I don't understand the downvotes: as far as physical resilience goes the Starlink constellation is a pretty damn hard target on an absolute basis, even before we get into the geopolitics of some country trying to shoot down US property that is also a US strategic asset.
> Starfish Prime caused an electromagnetic pulse (EMP) that was far larger than expected, so much larger that it drove much of the instrumentation off scale, causing great difficulty in getting accurate measurements. The Starfish Prime electromagnetic pulse also made those effects known to the public by causing electrical damage in Hawaii, about 900 miles (1,450 km) away from the detonation point, knocking out about 300 streetlights, setting off numerous burglar alarms, and damaging a telephone company microwave link. The EMP damage to the microwave link shut down telephone calls from Kauai to the other Hawaiian Islands.
https://en.wikipedia.org/wiki/Starfish_Prime
https://en.wikipedia.org/wiki/Starfish_Prime
Yes, very good you're capable of a simplistic wikipedia search. Now I'd like you to try thinking about the paragraph you just linked for a moment: is "Hawaii", the "300 streetlights", "numerous burglar alarms", and a "microwave link" in the atmosphere (and on the ground, or indeed literally the ground) or in Earth orbit? If you read further, you'll find that the mechanism of generation involves the interaction of the gamma ray emissions with the atmosphere and Earth's magnetic field, that at 500km+ the effect are is reduced, and then find that in your example of Starfish Prime the primary damage to the satellites it affected was not due to EMP at all but rather because the test created radiation belts that damaged solar panels as satellites passed through. This would reduce design lifetime but isn't an immediate killer of everything. You'd need far more then a couple of nukes.
Of course, purposefully detonating nuclear weapons to create large scale NEMP effects might well still disrupt Starlink to the degree that it'd effect any piece of ground electronics, because the ground effects would be far, far worse. The satellite constellation could be ok enough to still be functional at reduced capacity, but if terminals and ground stations get toasted that's not immediately helpful. But that directly leads, again, to the fact that you're talking about LAUNCHING A NUCLEAR WAR ON THE ENTIRE WORLD here. No internet is going to be doing very well!
So yes: Starlink is pretty darn robust.
Of course, purposefully detonating nuclear weapons to create large scale NEMP effects might well still disrupt Starlink to the degree that it'd effect any piece of ground electronics, because the ground effects would be far, far worse. The satellite constellation could be ok enough to still be functional at reduced capacity, but if terminals and ground stations get toasted that's not immediately helpful. But that directly leads, again, to the fact that you're talking about LAUNCHING A NUCLEAR WAR ON THE ENTIRE WORLD here. No internet is going to be doing very well!
So yes: Starlink is pretty darn robust.
> Starlink is pretty darn robust.
not saying it isn't. I was the evangelist of it in 2022 and seen the pentagon salivating at what they've seen and rightly so.
what it isn't is indestructible, though - and we don't know the capability of nukes designed specifcally to cause the Kessler syndrome in LEO for a few years. if I was China I'd have a project for this commissioned by Jan 2023.
not saying it isn't. I was the evangelist of it in 2022 and seen the pentagon salivating at what they've seen and rightly so.
what it isn't is indestructible, though - and we don't know the capability of nukes designed specifcally to cause the Kessler syndrome in LEO for a few years. if I was China I'd have a project for this commissioned by Jan 2023.
> we don't know the capability of nukes designed specifcally to cause the Kessler syndrome in LEO for a few years
I wouldn't even consider a nuke for that. Energetic plasma ball resulting from nuke is not effective method to do so and any debris that survives the explosion isn't going to still be in LEO, but also no plausible deniability because it's obviously a nuke.
Try "we launched a satelite and it RUDed (honest we pinky swear it wasn't just a loose heap of ball bearings like it appears on radar) and now LEO is marginally more dangerous".
I wouldn't even consider a nuke for that. Energetic plasma ball resulting from nuke is not effective method to do so and any debris that survives the explosion isn't going to still be in LEO, but also no plausible deniability because it's obviously a nuke.
Try "we launched a satelite and it RUDed (honest we pinky swear it wasn't just a loose heap of ball bearings like it appears on radar) and now LEO is marginally more dangerous".
I always assumed that Starshield (DoD starlink) exists so that the DoD can make strategic geopolitical decisions (like giving satcom c&c to Ukrainian suicide drones in Crimea) without painting a target on the back of SpaceX’s big moneymaker.
They famously did not want to get involved in the war.
I imagine DoD having their own constellation with equivalent tech was the deal set up to give the Russians and whoever else a different set of targets to retaliate against, push comes to shove.
They famously did not want to get involved in the war.
I imagine DoD having their own constellation with equivalent tech was the deal set up to give the Russians and whoever else a different set of targets to retaliate against, push comes to shove.
I'm guessing the previous poster meant "a few big nukes" Starfish Prime-style, used to generate EMPs.(I don't know if Starlink satellites are protected, but my guess is they are not.)
https://en.wikipedia.org/wiki/Nuclear_electromagnetic_pulse
https://en.wikipedia.org/wiki/Nuclear_electromagnetic_pulse
I don't believe this is true. And of course detonating nukes in space would be an act of war against the whole world.
nukes optimized for EMP may be able to do this just fine.
https://en.wikipedia.org/wiki/Nuclear_electromagnetic_pulse
https://en.wikipedia.org/wiki/Nuclear_electromagnetic_pulse
I'm aware of the existence of the EMP weapon concept. I doubt its effectiveness against modern satellites in LEO at long range.
That very article says "The damaging effects on orbiting satellites are usually due to factors other than EMP". Also "for equipment to be affected, the weapon needs to be above the visual horizon". Given how low Starlink orbits and the fact that they are dispersed evenly over the whole Earth, the number of weapons you would need to detonate just to ensure line of sight to every satellite at a range short enough to be effective is absolutely enormous.
That very article says "The damaging effects on orbiting satellites are usually due to factors other than EMP". Also "for equipment to be affected, the weapon needs to be above the visual horizon". Given how low Starlink orbits and the fact that they are dispersed evenly over the whole Earth, the number of weapons you would need to detonate just to ensure line of sight to every satellite at a range short enough to be effective is absolutely enormous.
Nukes produce lots of gamma and x-rays. On Earth, those are absorbed to produce the fireball. In LEO, some are absorbed by upper atmosphere to make EMP. In space, they are radiation.
Except for military satellites like GPS, most satellites aren't hardened to deal with that much radiation. My guess is that the range would be thousands of miles.
Except for military satellites like GPS, most satellites aren't hardened to deal with that much radiation. My guess is that the range would be thousands of miles.
I wondered how much energy could pass through some area 1000 miles away from the explosion, so I made some rough calculations.
Assumption: all energy of the nuclear explosion is radiation uniform in every direction.
The energy E would be spreading in a sphere with area 4*pi*R^2. Area A being a part of that sphere would capture E*A/(4*pi*R^2) of energy. So with:
From what I'm reading about high-altitude nuclear testing, it can cause artificial radiation belts around the Earth (composed of high-energy electrons). I think that may be more dangerous to satellites at distances like 1000 miles (or at any distance) than the immediate gamma/x-ray radiation.
Assumption: all energy of the nuclear explosion is radiation uniform in every direction.
The energy E would be spreading in a sphere with area 4*pi*R^2. Area A being a part of that sphere would capture E*A/(4*pi*R^2) of energy. So with:
energy E = 209 PJ (Tsar bomba for an extreme example)
area A = 1 m^2
radius of the sphere R = 1000 miles (1609 km)
2.09 * 10^17 J * 1 m^2 / (4 * pi * (1.609 * 10^6 m) ^ 2) = 6.4 kJ
From what I'm reading about high-altitude nuclear testing, it can cause artificial radiation belts around the Earth (composed of high-energy electrons). I think that may be more dangerous to satellites at distances like 1000 miles (or at any distance) than the immediate gamma/x-ray radiation.
Radiation is measured in gray, which is 1 J absorbed by 1 kg. I can't find the conversion between flux to absorbed dose. 5 gray is fatal. If even a fraction of that energy is absorbed, it would be fatal.
I found that satellites get 100-1000 rad per year. Getting that much in a moment would cause problems.
I found that satellites get 100-1000 rad per year. Getting that much in a moment would cause problems.
> Radiation is measured in gray, which is 1 J absorbed by 1 kg. I can't find the conversion between flux to absorbed dose. 5 gray is fatal. If even a fraction of that energy is absorbed, it would be fatal.
It's measured in a lot of different ways depending on the need, and the unit "sievert" is Gy times a conversion factor that depends on the kind of radiation (for equivalent dose) and which body parts (for effective dose): https://en.wikipedia.org/wiki/Sievert#Radiation_type_weighti...
But also, beta radiation (electrons) will be stopped by a thin sheet of tinfoil, while alpha radiation (helium-4 nuclei) by paper or the outer layer of dead skin on your body and therefore only matters if you eat a source of it, and a significant fraction of any gamma radiation you're exposed to will pass right through you without getting absorbed.
It's measured in a lot of different ways depending on the need, and the unit "sievert" is Gy times a conversion factor that depends on the kind of radiation (for equivalent dose) and which body parts (for effective dose): https://en.wikipedia.org/wiki/Sievert#Radiation_type_weighti...
But also, beta radiation (electrons) will be stopped by a thin sheet of tinfoil, while alpha radiation (helium-4 nuclei) by paper or the outer layer of dead skin on your body and therefore only matters if you eat a source of it, and a significant fraction of any gamma radiation you're exposed to will pass right through you without getting absorbed.
So, given that human body area viewed from the front is around 0.5 m^2, then a 70 kg human in space 1000 miles from the explosion would get 6.4 kJ * 0.5 / 70 kg = 45 gray. That would mean that even being a couple thousand miles away could be fatal. But that's assuming 100% of Tsar Bomba energy would be ionizing radiation, I don't know what % would that be in reality.
digging a bit deeper, apparently most damage is coming from irradiated material interacting with solar panels of satellites passing through the cloud. sounds like it'll work, just not immediately and not necessarily via an actual EMP.
Could some kind of microwave / laser attack from a high altitude air vehicle be possible?
Taking out GNSS satellites would probably also effectively take out starlink.
User terminals rely on GNSS for timing and position and I wouldn't be surprised if the satellites did too.
User terminals rely on GNSS for timing and position and I wouldn't be surprised if the satellites did too.
GPS is owned by the US military. Physically destroying GPS satellites would be an unambiguous act of war against the US.
If they had that capability, they could presumably do it any time, and then Starlink would recover at some point.
So it would make sense to combine this with some other offensive action where hampering communication would help them. Otherwise they are just poking at something just to do it... unless this were a more serious attack with long-term consequences.
So it would make sense to combine this with some other offensive action where hampering communication would help them. Otherwise they are just poking at something just to do it... unless this were a more serious attack with long-term consequences.
Get an employee to take the network down from the inside.
Either exploit existing employees or take years to seed an agent into the company.
State level persistent attacks are well funded and have long timelines.
Either exploit existing employees or take years to seed an agent into the company.
State level persistent attacks are well funded and have long timelines.
I'd wonder about someone targeting Musk seeing as Starlink is one of his ventures that seems to be running well up to now while his other companies have issues. Having typed that out it seems like a spy movie plot, but that seems to be the world we're living in recently.
I don’t think musk himself would be the weakest link, as the high profile brings scrutiny and thus increased security and counterintelligence. Trying would be a good way to get the FBI to look into you.
Easier and safer to go after unknown employees with access.
Easier and safer to go after unknown employees with access.
Yeah, Russia maanged to knock Viasat offline with hacking during the initial invasion. The original donated Starlinks were supposed to replace Viasat and SpaceX was surprised that they ended up being used on the front lines. Russia has certainly been trying to knock them out ever since.
If my memory is accurate, that didn't knock Viasat offline, it only made most of Viasat's customers unable to connect, but the network was just fine, only user routers were broken.
The Russians use Starlink too, saw a video on Kiev Independent that had a Ukranian drone team take out a Starlink dish atop a Russian bunker.
I imagine it is illegal for Starlink to not geofence service to prevent Russians using it for exactly that purpose. Sanctions still exist, after all.
I would not be surprised if US intelligence quietly asked Starlink to please let the Russians keep using Starlink, so long as they get to listen in.
Be serious. Look at the president, look at the head of ODNI, look at the head of DoD, finally look at musk
Yes, in Kursk, maybe. However this is in Ukraine.
Donetsk and Lugansk are both sanctioned too I believe.
You can watch the video yourself - https://www.youtube.com/watch?v=-MqoBPXzJds
Doesn’t the military use a different starlink or something ?
If not and this was a cyber attack the this an attack on the USA.
If not and this was a cyber attack the this an attack on the USA.
[deleted]
[deleted]
Starlink has been supporting Russia over Ukraine (congressional investigation into this). So I would bet against Russia, but you never know. Could be rouge. Anonymous has promised in the past to mess with Musk... could be that. Well funded and some crazy good hackers.
In terms of actual actions, Starlink has done a lot more for Ukraine than almost anyone else, since early on. Musk says stuff from time to time in order to try to keep Putin from noticing this, and pushing him out of a window or something. But his actions are that he let Ukraine use Starlink for over a year for free, and they still use it widely today.
No one is using Starlink for free. Poland has been paying for all of those "free" terminals DoD was sending over. Currently Poland is still paying for more than half of all Starlink in Ukraine.
Been using Starlink for awhile and can’t remember the last time there was an outage like this.
They were pretty common for the first few years of Starlink, but it looks like the last global coordinated outage was on May 28, 2024.
Yea it's funny because I was JUST thinking about how important StarLink has been for us here in rural Oregon. When it goes out, it's even more stark — but it hasn't happened in a very long time here.
Interestingly, this webpage doesn't load due to "no healthy upstream" and there's no status.starlink.com . That's.. quite the outage! Interested to hear what went wrong.
Probably their web servers are getting hugged to death by all of their worldwide users grabbing the phone to check who to call.
Wisconsin here,
Before it went down, I got a notice that my IP had just been changed. It didn't say who sent the message. Im wondering if changing my IP address was part of a hack or a fix?
Mine intentionally faced a obstructed area, which it has never done before (installed in the same spot for multiple years)
in-band notice or out-of-band?
For those who refuse to use X, as we all should, this is a post from Starlink that says simply:
"Starlink is currently in a network outage and we are actively implementing a solution. We appreciate your patience, we'll share an update once this issue is resolved."
"Starlink is currently in a network outage and we are actively implementing a solution. We appreciate your patience, we'll share an update once this issue is resolved."
A service outage meaning starlink is down globally.
Anecdotally my starlink is using more power currently than usual. And when I first looked at the app it mentioned having gotten a new public IP address.
Anecdotally my starlink is using more power currently than usual. And when I first looked at the app it mentioned having gotten a new public IP address.
> Anecdotally my starlink is using more power currently
That is interesting info - don't phones typically use a lot of power when they're trying to find a tower?
That is interesting info - don't phones typically use a lot of power when they're trying to find a tower?
Yea. Mine is down and listed as heating. And getting a public up, not a cgnatted 100.x.x.x I usually get. Weird...
I get new public IPs fairly frequently (maybe once every two weeks or so?)
Indeed but in this case it coincided with starlink going down. Could be a coincidence though!
Yeah, that's interesting actually, I guess one could speculate that the fact that you got a new IP might mean they rolled out/restarted something at the application layer and then things went south from there perhaps
So in AWS-speak, this is a yellow/orange status outage ?
I wonder if it correlates to the rollout of T-Mobile satellite services. They just exited their trial phase, so a global config update might make sense.
This happening on the same day as this https://www.the-independent.com/tech/ee-bt-three-vodafone-o2...
Could almost make one paranoid about some large level state-backed action being coordinated towards an attack of some kind.
Could almost make one paranoid about some large level state-backed action being coordinated towards an attack of some kind.
Amazon missed a huge pre-sale opportunity for their future sat net.
Negative marketing is risky because some day Kuiper will have an outage.
Is this possibly related to the service outages being experienced by telecom operators in the UK?
Well this is something new. Was wondering if my dish was broken, now seeing it's a global issue. Just over 24 hours ago the router reset (had to setup the network name and password again) and I have no idea why that happened. Somehow related?
Is it DNS or BGP?
ThousandEyes observed the outage from multiple vantage points deployed behind Starlink: https://www.linkedin.com/posts/thousandeyes_thousandeyes-dat...
[deleted]
Software update currently being pulled down. If not routing and terminals weren't connecting could be a terminal registration issue
[deleted]
My terminal is pulling an update. Slowly... I would guess outage was in terminals associating with satellites properly
Yeah. Mine did a couple of reboots automatically in the last hour and now it seems solid again.
Thought it was the thunderstorm and the clouds but then I saw this :) it’s been down for more than 30m now in Italy
I'm not saying there's a connection to the UK telco outage but the timing is rather coincidental.
Given the ongoing expansion of cellular Starlink, it could make sense.
https://www.theverge.com/news/712546/starlink-powered-t-sate...
https://www.theverge.com/news/712546/starlink-powered-t-sate...
I just checked my Starlink app and it looks like it's stuck at about 3/4 progress in "Software update". Just checked so not sure for how long it's been stuck that way, and not sure if that would be a software update for the modem or for the dish..
I brought my Starlink on a three week RV trip through the western U.S., and it was pretty impressive. I was surprised it worked well even in some wooded areas. I would’ve been pretty frustrated if it had stopped working.
What’s crazy to me is how fast Starlink went from not being possible to being possible to existing to being critical infrastructure for so many people in so many places.
All because of reusable rockets and cheap(er) access to orbit.
All because of reusable rockets and cheap(er) access to orbit.
Is this because they have now more then 50 user per square mile [1]?
[1] https://www.washingtonpost.com/technology/2025/07/18/starlin... https://www.ookla.com/articles/starlink-hughesnet-viasat-per... (PDF) https://thexlab.org/wp-content/uploads/2025/07/Starlink_Anal...
[1] https://www.washingtonpost.com/technology/2025/07/18/starlin... https://www.ookla.com/articles/starlink-hughesnet-viasat-per... (PDF) https://thexlab.org/wp-content/uploads/2025/07/Starlink_Anal...
CME pointed at earth, satilites in safe mode?
https://youtu.be/UvKqGnHQ_1Q?si=Mfu2zIlZ7Ira5RFe
https://youtu.be/UvKqGnHQ_1Q?si=Mfu2zIlZ7Ira5RFe
Those YouTube channels are lies.
Outages suck but for me this service has been rock solid. Much better uptime and more capable than the cellular broadband service that it replaced (Bell Canada) at a similar cost
Yea, my house is on Starlink... so at the moment I'm working in the back room of my wife's restaurant (she has cable internet). At least I get free coffee here.
Can't wait to watch Kevin Fang's report on this.
Just before it went down, I got an unidentified message saying my IP address was just changed.
I wonder if this was part of the issue or part of the fix???
Hopefully back up soon. Its a pretty spectacular product.
Wonder what will be the cost in lives in Ukraine. Communications outage could create a disorder in the worst possible time.
Russian aerial assets also use Starlink so it could similarly save lives.
Not as much as Ukrainians. The whole frontline is powered by Starlinks.
How do they bypass the sanctions?
Looks like it's down worldwide...
I bet it's configuration error, failed SW upgrade, or Russian sabotage in that order.
Starlink is back up in the UK
Business tier customers get a 20% discount this month for the SLA violation!
My global roaming unit in Afghanistan dropped off at 1914 GMT
Mine returned 2132G
Don't worry, just press and hold the power button on the satellites until they turn off, then release and press it again to power them back on. Should be fine.
Seems back now
Config error, being rolled back now
[deleted]
This happened as I was taking a brief dive on my networking stack. Been seeing "iwlwifi 0000:05:00.0: Unhandled alg: 0x71b " in my syslog spamming heavily. Thought I somehow really screwed things up to have internet flat out not working. :D
I updated my Starlink app, dish downloaded an update (while still ‘offline’), clicked install now via app, installed, rebooted, and now online. Take that Putin!
Another lawsuit, first Colbert, now our Internet
Glad that Musk torched his relationship with Trump so that the FAA / Starling contract is no longer on the table.
Recently, I’ve noticed that nothing hits the top spot as quickly as anything negative about Elon.
2OEH8eoCRo0(1)
kokorikooo(2)
RIMR(3)
Starlink.com is also experiencing an outage. Tesla.com works, so it makes me believe they are on different servers.
EDIT: Weird to get downvoted for a random observation. Are you guys full gray beard serious hackers?
EDIT: Weird to get downvoted for a random observation. Are you guys full gray beard serious hackers?
I'm in Colorado. Starlink has worked flawlessly since I installed it 2 years ago. This is ominous.
One outage is not ominous.
Or earth directed CME and SATs put in safe mode?
https://youtu.be/UvKqGnHQ_1Q?si=Mfu2zIlZ7Ira5RFe
https://youtu.be/UvKqGnHQ_1Q?si=Mfu2zIlZ7Ira5RFe
(Not necessarily related, but interesting nonetheless)
Russian Nuclear Sabotage In Space Could Blast U.S., SpaceX Satellites
https://www.forbes.com/sites/kevinholdenplatt/2025/07/23/rus...
Russian Nuclear Sabotage In Space Could Blast U.S., SpaceX Satellites
https://www.forbes.com/sites/kevinholdenplatt/2025/07/23/rus...
Everyone on this thread is so smart. I don’t have anything to add except a funny story.
My Starlink went out right when I was scheduled to join a meeting today at 2:15 CST.
My Starlink went out right when I was scheduled to join a meeting today at 2:15 CST.