Ask HN: Doesn't the Rogers outage prove – network as a single point of failure?
High Availability and Disaster Recovery and Auto-Scaling and a design in place to avoid SPOF (Single Point of Failure) always ignore Network being a SPOF. But, Network can be a Single Point of Failure too.
22 comments
This has been known for a long time. Almost every data center uses multiple ISPs for this reason.
Even multiple ISPs might not help.
I remember a DC issue where they had two physically separate ISP attachments from different sides of the building. Cable cut takes out ENTIRE DC.
Why? While the ISPs were different, at one (farther upstream) point they were both leasing fiber that ran through a single conduit -- so you need understanding of the physical structure of the network, not just "different ISPs".
I remember a DC issue where they had two physically separate ISP attachments from different sides of the building. Cable cut takes out ENTIRE DC.
Why? While the ISPs were different, at one (farther upstream) point they were both leasing fiber that ran through a single conduit -- so you need understanding of the physical structure of the network, not just "different ISPs".
Right.
My point is - "How so many Services across Canada went down as they never designed their Services to overcome the network outage ? "
Again as I wrote elsewhere it's because Canada is a deeply mediocre country just getting by with others help and technology.
Interac the main electronic money transfer system used Rogers as their primary network. Guess what they used for their backup network? Rogers.
There is also very rampant high level corruption leading to anti competitive behaviour, oligarchy, money laundering and I can go on and on.
Point is Canada is not a good example of networks and failures. It is a good example of incompetence and mediocrity.
Interac the main electronic money transfer system used Rogers as their primary network. Guess what they used for their backup network? Rogers.
There is also very rampant high level corruption leading to anti competitive behaviour, oligarchy, money laundering and I can go on and on.
Point is Canada is not a good example of networks and failures. It is a good example of incompetence and mediocrity.
Sorry to rain on your evidence free assertion fest here, but let me push back a bit here:
> Rogers as their primary network. [...] backup network? Rogers.
What idiots, right? But there is likely a bit more nuance here which this throw-away comment is ignoring, like using cellular/microwave/fibre/copper in a diverse and high-availability way. I don't know all the details of the network, but all of this stuff works pretty well MOST of the time, so I'd assume their core network is pretty well built and Interac has tight agreements that pay them big bucks for outages.
And, don't mistake this for me saying that having 3 national ISPs is good.
>"..." is not a good example of networks and failures.
Actually, it's a pretty rocking example. Right now, we have zero clue why this happened, and complex built to survive SPOF systems have complex interactions and weird failure modes. But those Canadians are all Cletus and don't know how to configure a router, right?
Could this be an incompetent configuration? Sure, and we'll find that out, as this one will likely spark hearings due to the length and impact.
Just wondering if a big AWS outage means that the US is "mediocre", or if a failure should reflect the organization that failed?
What idiots, right? But there is likely a bit more nuance here which this throw-away comment is ignoring, like using cellular/microwave/fibre/copper in a diverse and high-availability way. I don't know all the details of the network, but all of this stuff works pretty well MOST of the time, so I'd assume their core network is pretty well built and Interac has tight agreements that pay them big bucks for outages.
And, don't mistake this for me saying that having 3 national ISPs is good.
>"..." is not a good example of networks and failures.
Actually, it's a pretty rocking example. Right now, we have zero clue why this happened, and complex built to survive SPOF systems have complex interactions and weird failure modes. But those Canadians are all Cletus and don't know how to configure a router, right?
Could this be an incompetent configuration? Sure, and we'll find that out, as this one will likely spark hearings due to the length and impact.
Just wondering if a big AWS outage means that the US is "mediocre", or if a failure should reflect the organization that failed?
Sounds a lot like America.
Anybody can pay for redundant services in multiple data centers and networks. But almost nobody does. And corruption? Heh, here we just call them "donations".
Anybody can pay for redundant services in multiple data centers and networks. But almost nobody does. And corruption? Heh, here we just call them "donations".
Yes but Canada has none of the highs of America. If you want to change the world, America is one of the only places.
Gosh, I wish I could trade places. If Canada would accept us as refugees, I'd move there in a heartbeat.
Change the world? Lol, maybe if you're Bezos or Zuckerberg. The rest of us mostly just stifle under their iron boots. If you don't get shot first.
IMO a country that prioritizes its average citizens over its uber-rich isn't mediocre, it's well-managed. America is a hellscape of greed and corruption.
Guess the grass is always greener, huh?
Change the world? Lol, maybe if you're Bezos or Zuckerberg. The rest of us mostly just stifle under their iron boots. If you don't get shot first.
IMO a country that prioritizes its average citizens over its uber-rich isn't mediocre, it's well-managed. America is a hellscape of greed and corruption.
Guess the grass is always greener, huh?
I am in the US. I spent a long time in Canada. The population of the US is huge. While the shootings I agree are egregious, in every other way the US is better. The average citizens in Canada have no upward mobility, no healthcare except for absolute emergencies and no bright hope for the future.
Sounds like you’ve got a bone to pick. Upward mobility in Canada is actually better than the US. Sources:
https://www.brookings.edu/blog/social-mobility-memos/2014/07...
https://stonecenter.gc.cuny.edu/social-mobility-is-twice-as-...
https://www.oecd.org/social/broken-elevator-how-to-promote-s...
Quote from above:
> In comparison with traditional peer countries such as the United States or Great Britain, Canada displays a far less rigid class structure. The odds of moving from poverty to the middle class during an individual’s lifetime are twice as good for a Canadian as they are for an American. The fact that Canadians enjoy Scandinavian levels of equity without having to rely on Scandinavian levels of taxation or social conformity is further testament to the importance of Canada’s education, health care and welfare systems, as well as the country’s commitment to welcoming newcomers
Your healthcare comment is also provable false. Sources:
https://www.washingtonpost.com/world/2021/08/05/global-healt...
https://www.commonwealthfund.org/publications/fund-reports/2...
Quote from above:
> Key Findings: The top-performing countries overall are Norway, the Netherlands, and Australia. The United States ranks last overall, despite spending far more of its gross domestic product on health care. The U.S. ranks last on access to care, administrative efficiency, equity, and health care outcomes, but second on measures of care process
Your comment:
>[Canada has] no bright hope for the future.
Seriously, what’s your agenda? Why are you spreading obvious lies and toxic nonsense? Sounds like you had a bad experience here and are keen to blame anything but yourself.
Don’t get me wrong, there’s plenty to criticize about Canada but you come across as extremely disingenuous.
I wouldn’t disagree with the idea that opportunities (and even healthcare) are better for the highest earners (most HN users) but the evidence just doesn’t support your claims as general truths. It’s just not true.
https://www.brookings.edu/blog/social-mobility-memos/2014/07...
https://stonecenter.gc.cuny.edu/social-mobility-is-twice-as-...
https://www.oecd.org/social/broken-elevator-how-to-promote-s...
Quote from above:
> In comparison with traditional peer countries such as the United States or Great Britain, Canada displays a far less rigid class structure. The odds of moving from poverty to the middle class during an individual’s lifetime are twice as good for a Canadian as they are for an American. The fact that Canadians enjoy Scandinavian levels of equity without having to rely on Scandinavian levels of taxation or social conformity is further testament to the importance of Canada’s education, health care and welfare systems, as well as the country’s commitment to welcoming newcomers
Your healthcare comment is also provable false. Sources:
https://www.washingtonpost.com/world/2021/08/05/global-healt...
https://www.commonwealthfund.org/publications/fund-reports/2...
Quote from above:
> Key Findings: The top-performing countries overall are Norway, the Netherlands, and Australia. The United States ranks last overall, despite spending far more of its gross domestic product on health care. The U.S. ranks last on access to care, administrative efficiency, equity, and health care outcomes, but second on measures of care process
Your comment:
>[Canada has] no bright hope for the future.
Seriously, what’s your agenda? Why are you spreading obvious lies and toxic nonsense? Sounds like you had a bad experience here and are keen to blame anything but yourself.
Don’t get me wrong, there’s plenty to criticize about Canada but you come across as extremely disingenuous.
I wouldn’t disagree with the idea that opportunities (and even healthcare) are better for the highest earners (most HN users) but the evidence just doesn’t support your claims as general truths. It’s just not true.
Studies are false. I have stayed there and go back their frequently. I'm talking about ground reality.
“Studies are false.” has got to be the absolute most low effort, I’m-not-even-going-to-try response in HN history.
“[I] go back their frequently” is another absolute classic.
I’ve had more rigorous and intellectual debates with my 9 year old (and he goes to Canadian elementary school).
Thanks for the laugh.
“[I] go back their frequently” is another absolute classic.
I’ve had more rigorous and intellectual debates with my 9 year old (and he goes to Canadian elementary school).
Thanks for the laugh.
Unfortunately it's been an increasing issue.
Why do Canadians have no healthcare? Isn't there a national system?
> in every other way the US is better
> The average citizens in Canada have no upward mobility
> no bright hope for the future.
This also applies to everyone I know in the US, aside from those working in tech =/
It is especially bad in rural areas.
> in every other way the US is better
> The average citizens in Canada have no upward mobility
> no bright hope for the future.
This also applies to everyone I know in the US, aside from those working in tech =/
It is especially bad in rural areas.
No healthcare because every single doctor leaves for the US.
Many do, yes. Many also return. Why are you insisting on using absolutes in your comments?
Because I have friends who have been waiting for a stomach ultrasound for over 2 years now. Suffering in pain everyday. Since it's not an emergency, it's never scheduled. Absolutes are closer to the truth at the moment. Try getting a family doctor. You won't get one. Try going to emergency. You will wait for over a day. Try getting a passport. You will get one after 6 months.
If you have a friend who’s been waiting two years for an ultrasound, get them to contact a news outlet. It would be headline news and they would get in instantly.
If a journalist confirms and publishes the story, I will donate US $1000 to the charity of your (or your friends) choosing. I’m absolutely serious and will post the receipt here.
You won’t though… because you can’t. Because you made that up out of thin air.
I’ve been to emergency over 20 times in my life (between myself and loved ones). I’ve never waited (or known anyone to wait) “for over a day”. You’re ridiculous.
Come on. If you’re going to lie, at least make the lies believable.
If a journalist confirms and publishes the story, I will donate US $1000 to the charity of your (or your friends) choosing. I’m absolutely serious and will post the receipt here.
You won’t though… because you can’t. Because you made that up out of thin air.
I’ve been to emergency over 20 times in my life (between myself and loved ones). I’ve never waited (or known anyone to wait) “for over a day”. You’re ridiculous.
Come on. If you’re going to lie, at least make the lies believable.
While I'm all for ranting about too much control by too few players...
How is this different then any "big" player having issues? (Think AWS, Big US ISP, large payment provider such as VISA/MC)
I don't think most NEVER thought about network issues ... but rather a calculated risk, as long outages tend to be a rare occurrence. It's honestly cheaper to eat losses on a day like today then to pay for redundancy and availability.
How is this different then any "big" player having issues? (Think AWS, Big US ISP, large payment provider such as VISA/MC)
I don't think most NEVER thought about network issues ... but rather a calculated risk, as long outages tend to be a rare occurrence. It's honestly cheaper to eat losses on a day like today then to pay for redundancy and availability.
For things like ATMs I think having two network connections is pretty expensive. Companies also like to use "strategic outsourcing" where they use the same ISP for all their locations which results in the systemic outages we're seeing now.
Well, the cost of outage vs the cost to maintain multiple ISP's is the key decision factor here.
Network is not a SPOF. Poorly implemented network can be though. My team manages a global network. We run outage simulations quarterly ( to check for unintended consequences of changes ).
Part of that process is to down all circuits and services from a vendor.
Could vendors be sharing upstream Concentrations that create SPOF. Sure they could. That’s why we contract a single vendor to create truly diverse services.
We see 100x outages and failures from craplications vs network.
Part of that process is to down all circuits and services from a vendor.
Could vendors be sharing upstream Concentrations that create SPOF. Sure they could. That’s why we contract a single vendor to create truly diverse services.
We see 100x outages and failures from craplications vs network.