Rogers networks reliability and resiliency assessment after 2022-07-08 outage(crtc.gc.ca)
crtc.gc.ca
Rogers networks reliability and resiliency assessment after 2022-07-08 outage
https://crtc.gc.ca/eng/publications/reports/xona2024.htm
114 comments
Yep.
Allowing Rogers to purchase Shaw was absurd.
The Commissioner of Competition was very strongly opposed to the merger, the court ruled against him, and Canadians had to pay Rogers $13 million CAD because he opposed it.
I was a Shaw Mobile (not Freedom) customer, and Rogers wanted me to drive 600km (both ways) through the Kootenay mountains in winter to get my new SIM cards. I told them where to go.
Rogers is junk and Canada is an oligarchy.
Allowing Rogers to purchase Shaw was absurd.
The Commissioner of Competition was very strongly opposed to the merger, the court ruled against him, and Canadians had to pay Rogers $13 million CAD because he opposed it.
I was a Shaw Mobile (not Freedom) customer, and Rogers wanted me to drive 600km (both ways) through the Kootenay mountains in winter to get my new SIM cards. I told them where to go.
Rogers is junk and Canada is an oligarchy.
Small world - I live in the Kootenays. I am guessing you are in Nelson only because it seems to be 600km from everything :-)
Close enough though a bit more in the boonies.
They wanted me to drive to the Rogers store in Cranbrook or Kelowna.
They wanted me to drive to the Rogers store in Cranbrook or Kelowna.
Huh, very small world. Me too :)
I’m in Cranbrook. Perhaps we need to establish the smaller HN IRL meetup :-)
Nelson has a "tech people meetup" pretty often.
https://www.facebook.com/groups/kootenayworkers/
https://www.facebook.com/groups/kootenayworkers/
On Facebook? What irony.
No, that is just the page used to organize the face to face meetups
Split the difference, meet in Creston at Jimmy's Pub (pretty well equidistant from Nelson, Cranbrook, Trail). I can walk over there too.
What made Canada special died decades ago, were it not for the strict visa process I would be living in the states (Vermont) right now, with a significantly higher salary (as a principal engineer), a lower income tax, healthier housing market, lower cost of living, and better quality of life.
Funny thing, I lived outside of Canada for many years though I am Canadian born and raised. I moved my family back for a "better quality of life".
Jokes on me. ;)
I love the land and the people, but, the country itself is a mess. We have already planned our exit.
Jokes on me. ;)
I love the land and the people, but, the country itself is a mess. We have already planned our exit.
Where too? It's crossing my mind more and more lately.
What makes Canada strong is its people, whom I love very much.
I wish our gov't served us better, and I try my best to enact change, but it's a fool who thinks any country is without its faults.
I wish our gov't served us better, and I try my best to enact change, but it's a fool who thinks any country is without its faults.
I love Canadians. And Americans.
And I've lived in a few different countries in my life. No country is without faults, but, Canada right now is a mess.
And I've lived in a few different countries in my life. No country is without faults, but, Canada right now is a mess.
>were it not for the strict visa process I would be living in the states (Vermont) right now, with a significantly higher salary (as a principal engineer), a lower income tax, healthier housing market, lower cost of living, and better quality of life.
Western Europe has all those things, is actually welcoming, has sane healthcare, and you don't even need to own a gun!
Western Europe has all those things, is actually welcoming, has sane healthcare, and you don't even need to own a gun!
Moving from high taxes low pay shortage of housing in popular cities Canada to high taxes low pay shortage of housing in popular citied western Europe I don't think will make much of a difference. Being able to have a gun is an additional freedom.
It doesn't have the salary, eng salaries are significant deflated in Europe when compared to North America
Outside of tech it's not too different. Tech pay isn't even bad, it just doesn't exhibit the massively inflated salaries of the US. Software is an easier, less professional job than most professional fields, it doesn't make sense for it to command preposterous sums. Though American devs don't like having this fact pointed out to them.
> Software is an easier, less professional job
Are you in software development?
Are you in software development?
Yes, why? Coming from a family of actual engineers, I call myself a developer. Software workers haven't earned the title engineer.
TBH I call myself and most software devs an engineer. It gives us something to aspire to.
Canada used to say our healthcare was better but that hasnt been true for atleast a decade. Its more choose your version of crappy. Do you want to oay with your life savings (usa) or your life (canada)
I (born and raised Canadian) moved my family to Canada 5 years ago.
We've been on a waiting list for a doctor ever since. My only option to see a doctor here is to go the emergency room at the closest hospital.
We've been on a waiting list for a doctor ever since. My only option to see a doctor here is to go the emergency room at the closest hospital.
If it makes you feel any better, I've been in the healthcare IT field for 20 years. I tried to stay away from the doctors office as much as possible.
Last year I started feeling exhausted 24/7 and tried to make an appointment, but got turned away from my doctors office because my doctor retired and the new guy didn't want to let me make an appointment.
So here I am on a waiting list, in a city where 10% of the population don't have a doctor either. I've been waiting for over a year.
I wish I could change fields, "once you've seen how the sausage is made, you'll want to be a vegetarian"...
Last year I started feeling exhausted 24/7 and tried to make an appointment, but got turned away from my doctors office because my doctor retired and the new guy didn't want to let me make an appointment.
So here I am on a waiting list, in a city where 10% of the population don't have a doctor either. I've been waiting for over a year.
I wish I could change fields, "once you've seen how the sausage is made, you'll want to be a vegetarian"...
Yah you are paying with your life. Its shockingly stupid because there are lots of good examples to oick and choose from how to do a public payor system correctly (singapore does it best imo but nordics and france are options for examples of better tham here just not best)
Yes. Waited 18 months for a 15 minute cateract surgery that I needed 18 months ago. Should have had my license taken away this past year they were so bad.
Shaw didn't have enough capital to continue operating, and foreign ownership restrictions prevented them from raising more. The government decided that Rogers buying Shaw was a better outcome than Shaw going out of business, which is a position I'm inclined to agree with.
I was a Shaw Mobile (not Freedom) customer, and Rogers wanted me to drive 600km (both ways) through the Kootenay mountains in winter to get my new SIM cards.
Weird. They mailed my new SIM cards to me.
I was a Shaw Mobile (not Freedom) customer, and Rogers wanted me to drive 600km (both ways) through the Kootenay mountains in winter to get my new SIM cards.
Weird. They mailed my new SIM cards to me.
> Shaw didn't have enough capital to continue operating
?
They had $10b+ in equity to work with and had ample opportunity to capture ZIRP if they wanted.
Anything inhibiting Shaw’s ability to raise capital was its dual class structure where the Shaw family retained control despite owning a minority of the shares.
?
They had $10b+ in equity to work with and had ample opportunity to capture ZIRP if they wanted.
Anything inhibiting Shaw’s ability to raise capital was its dual class structure where the Shaw family retained control despite owning a minority of the shares.
$10B isn't enough to set up a 5G network. And... I guess it's a good thing they didn't assume ZIRP would be around forever?
Dunno if it was post-Quebecor acquisition, but Freedom Mobile did just that. Not every populated area, just their existing footprint.
Post acquisition. Part of the agreement was that Quebecor was going to put a lot of money into building new infrastructure.
> Shaw didn't have enough capital to continue operating, and foreign ownership restrictions prevented them from raising more. The government decided that Rogers buying Shaw was a better outcome than Shaw going out of business
They could also have just eliminated our archaic foreign ownership laws. I am surprised this hasn’t come up in trade negotiations. If we want improved productivity then we desperately need competition. If we don’t get serious about this then we’ll continue to languish economically as a nation.
They could also have just eliminated our archaic foreign ownership laws. I am surprised this hasn’t come up in trade negotiations. If we want improved productivity then we desperately need competition. If we don’t get serious about this then we’ll continue to languish economically as a nation.
Right, that would have been my preferred solution. But for unfortunately that's not something the current government is inclined to consider.
> Weird. They mailed my new SIM cards to me.
Rogers support is pretty notorious country wide, I'm sure it still varies at a localized level.
I had a Rogers internet install technician come to my house (for Teksavvy because the gov mandates they run the networks for small operators) and instead of drilling a hole through a wall he opened a window a crack ran the cable through and went home. The girl on the phone couldn't believe my story but I'm sure nothing happened in terms of punishment.
This is the outcome of artificial barriers to competition in the name of national benefit. It doesn't matter how absurd your customer story, neither customers nor the upstart businesses have a choice.
Rogers support is pretty notorious country wide, I'm sure it still varies at a localized level.
I had a Rogers internet install technician come to my house (for Teksavvy because the gov mandates they run the networks for small operators) and instead of drilling a hole through a wall he opened a window a crack ran the cable through and went home. The girl on the phone couldn't believe my story but I'm sure nothing happened in terms of punishment.
This is the outcome of artificial barriers to competition in the name of national benefit. It doesn't matter how absurd your customer story, neither customers nor the upstart businesses have a choice.
They did mail me SIM cards, three times, for three different numbers each time.
Their system didn't work when setting them up, and I had two different support operators tell me that my only option was to drive to Cranbrook. In the middle of winter. This is a drive that involves 3 mountain passes.
Their system didn't work when setting them up, and I had two different support operators tell me that my only option was to drive to Cranbrook. In the middle of winter. This is a drive that involves 3 mountain passes.
Ok, that's unfortunate, but clearly a technical problem rather than a policy problem. What would you have them do, keep mailing new SIM cards when they have no idea why they're not working?
> foreign competitors are still prohibited from coming into the market
There is some foreign competition via roaming, e.g. both T-Mobile and AT&T have US mobile postpaid plans with unlimited Canadian roaming, https://www.att.com/international/canada-roaming/ & https://www.t-mobile.com/cell-phone-plans/use-your-phone-in-...
For devices with low data (text/email/web) usage, there are vendors of global prepaid eSIMs that last 6-24 months, which roam on Bell and Rogers 5G networks, priced around $5/GB, e.g. Eskimo (Singapore) and AIS (Thailand) eSIMs.
There is some foreign competition via roaming, e.g. both T-Mobile and AT&T have US mobile postpaid plans with unlimited Canadian roaming, https://www.att.com/international/canada-roaming/ & https://www.t-mobile.com/cell-phone-plans/use-your-phone-in-...
For devices with low data (text/email/web) usage, there are vendors of global prepaid eSIMs that last 6-24 months, which roam on Bell and Rogers 5G networks, priced around $5/GB, e.g. Eskimo (Singapore) and AIS (Thailand) eSIMs.
>There is some foreign competition via roaming, e.g. both T-Mobile and AT&T have US mobile postpaid plans with unlimited Canadian roaming, https://www.att.com/international/canada-roaming/ & https://www.t-mobile.com/cell-phone-plans/use-your-phone-in-...
Those don't seem that much cheaper compared to local competitors. For instance the cheapest plan at&t offers is $30 USD/month, but public mobile is offering a 40GB unlimited talk text plan for $29 CAD/month ($21 USD).
Those don't seem that much cheaper compared to local competitors. For instance the cheapest plan at&t offers is $30 USD/month, but public mobile is offering a 40GB unlimited talk text plan for $29 CAD/month ($21 USD).
> Some competitors are present indirectly, e.g. both T-Mobile and AT&T have US mobile postpaid plans with unlimited Canadian roaming
These aren’t real options as anyone calling you back will be forced to pay long distance.
These aren’t real options as anyone calling you back will be forced to pay long distance.
If you need inbound PSTN calls and SMS (instead of text+audio messengers like WhatsApp, Facetime, Matrix, Signal), you can get a Canadian VOIP PSTN number for USD $3/month, e.g. https://www.callcentric.com/dids/pay_per_minute, which can "ring" your phone via SIP client app, call forward from Canada to any global number at your expense, or relay audio voicemails as mp3 via email.
Ok, now explain that to enough people that this solution actually works as a real proxy for competition.
It allows individuals who do care about competition to stop funding oligopolists.
I agree, though I would prefer a solution that actually leads to the weakening of the oligopolists.
VoIP.ms would be cheaper for a DID
> Not for extended international use; you must reside in the U.S. and primary usage must occur on our network. Device must register on our network before international use. Service may be terminated or restricted for excessive roaming. Coverage not available in some areas; we are not responsible for our partners’ networks.
> AT&T ROAM NORTH AMERICA FEATURE: Allows plan data, talk & text usage with no roaming charges in and to Mexico/Canada. Data: Allows domestic plan data usage in Mexico/Canada.
Eskimo (Singapore telco) has both North America and Global eSIMs.
Eskimo (Singapore telco) has both North America and Global eSIMs.
It’s a shame on Canada and on Canadians that foreign competitors are still prohibited from coming into the market.
No, no it isn't a shame, it's great.
No, no it isn't a shame, it's great.
Go on. Why?
You should more directly ask, why is foreign competition vital?
Preventing domestic mergers, and ensuring domestic competition exists are viable alternatives.
With viable alternatives available, when it comes to domestic telco and networking, you want domestic control. A nation is benefited by domestic ownership of its own infrastructure.
This is inline with domestic food production, manufacturing of defense products, transportation infrastructure, and on and on.
Domestic control is an aspect of sovereignty.
Now again, I want competition. The problem isn't foreign competition, it's a lack of domestic.
The Harper Conservatives worked on this, limiting spectrum auction to induce competition and so on.
I suspect someone had a pet peeve, re domestic cost of services. One thing about government, is that often politicians get into the politicking business because something pissed them off.
So I suspect someone was cheesed at costs, or customer service, and made it their thing. Sadly, it was dropped by the next government.
Another example would be the Trudeau Liberals overhauling the entire ombudsman process for banks, along with intense investigation into retail banking processes.
Prior, the ombudsman was an escalation process that ended up at a provincial ombudsman's office... jointly owned by the banks!
Again, I expect someone in the current government was cheesed at banking chicanery.
Regardless, we can get competition without foreign competitors. It just takes a government caring enough.
And it would take just as much push to drop foreign ownership requirements, as to limit spectrum, prevent mergers, etc.
Preventing domestic mergers, and ensuring domestic competition exists are viable alternatives.
With viable alternatives available, when it comes to domestic telco and networking, you want domestic control. A nation is benefited by domestic ownership of its own infrastructure.
This is inline with domestic food production, manufacturing of defense products, transportation infrastructure, and on and on.
Domestic control is an aspect of sovereignty.
Now again, I want competition. The problem isn't foreign competition, it's a lack of domestic.
The Harper Conservatives worked on this, limiting spectrum auction to induce competition and so on.
I suspect someone had a pet peeve, re domestic cost of services. One thing about government, is that often politicians get into the politicking business because something pissed them off.
So I suspect someone was cheesed at costs, or customer service, and made it their thing. Sadly, it was dropped by the next government.
Another example would be the Trudeau Liberals overhauling the entire ombudsman process for banks, along with intense investigation into retail banking processes.
Prior, the ombudsman was an escalation process that ended up at a provincial ombudsman's office... jointly owned by the banks!
Again, I expect someone in the current government was cheesed at banking chicanery.
Regardless, we can get competition without foreign competitors. It just takes a government caring enough.
And it would take just as much push to drop foreign ownership requirements, as to limit spectrum, prevent mergers, etc.
I think foreign competition has some unique features that are beneficial:
— cross pollination of talent, knowledge, technology, etc;
— less susceptible to being eliminated when a future government caves on some of the rules you’ve mentioned
— cross pollination of talent, knowledge, technology, etc;
— less susceptible to being eliminated when a future government caves on some of the rules you’ve mentioned
Compared to networking being routed and controlled external to domestic laws, as a starter?
When the patriot act was passed in the US, suddenly some Canadian credit card holders, whos companies which had processing centers in the US, had all of their transaction info shared with US homeland.
If the corporate entity is domestic, foreign powers won't have control over their domestic corps, managing foreign infra.
When the patriot act was passed in the US, suddenly some Canadian credit card holders, whos companies which had processing centers in the US, had all of their transaction info shared with US homeland.
If the corporate entity is domestic, foreign powers won't have control over their domestic corps, managing foreign infra.
Shaw customer base was acquired by Rogers, but they weren't allowed to absorb Freedom Mobile, which went to Quebecor.
Shaw Mobile on the other hand, did go to Rogers.
>It’s a shame on Canada and on Canadians that foreign competitors are still prohibited from coming into the market.
Yeah during NAFTA 2, Canadians were all for protecting high prices and very little choice just to show Trump and the USA that they could take their cheap prices and value and shove it.
Rogers, Bell and Telus went on and made Trudeau classify them as Canadian media companies so they would never have any competition from the USA.
I guess Canadians like high prices and little choice..
Yeah during NAFTA 2, Canadians were all for protecting high prices and very little choice just to show Trump and the USA that they could take their cheap prices and value and shove it.
Rogers, Bell and Telus went on and made Trudeau classify them as Canadian media companies so they would never have any competition from the USA.
I guess Canadians like high prices and little choice..
ilrwbwrkhv(2)
> Rogers staff relied on the company’s own mobile and Internet services for connectivity to communicate among themselves. When both the wireless and wireline networks failed, Rogers staff, especially critical incident management staff, were not able to communicate effectively during the early hours of the outage. Rogers had to send Subscriber Identity Module (SIM) cards from other mobile network operators to its remote sites to enable its staff with wireless connectivity to communicate with each other
ooops. that's actually quite funny
in case you didn't know, Rogers is one of canada's "big 3" telecom providers. The outage in 2022 basically crippled our economy for a couple days (most ATMs / interac didn't work)
ooops. that's actually quite funny
in case you didn't know, Rogers is one of canada's "big 3" telecom providers. The outage in 2022 basically crippled our economy for a couple days (most ATMs / interac didn't work)
That outage is actually a good argument against going cashless.
Happens semi-regularly in Australia, too. More often because of fires or floods than network incompetence.
Thankfully none of my wavelength services from Rogers were impacted. =-)
The "lessons learned" are all very basic. Things like "separate the network management layer from the data network", "provide the network operation center with backup connectivity", etc.
This is networking 101. Heck, this is engineering 101. The real question is how a network provider as large as Rogers managed to be so poorly engineered in the first place.
This is networking 101. Heck, this is engineering 101. The real question is how a network provider as large as Rogers managed to be so poorly engineered in the first place.
Monopolization and corruption
Rogers isn't the massive network it is because Canadian consumers love them and want to use them - they just have no choice, and the government does all it can to keep it that way
Rogers isn't the massive network it is because Canadian consumers love them and want to use them - they just have no choice, and the government does all it can to keep it that way
Canadians are sheep like in their mentally and incredibly risk averse. So there is zero competition.
I don't think this is an acceptable way to speak about 41 million people on the basis of their nationality.
What is a good way to speak on the basis of nationality or it is always bad (tm)?
Someone already mentioned the monopolism/corruption, but Canada also doesn't have much of an engineering/technology culture. Growing up in a BC suburb, it seemed like we were always 5-10 years behind the US on things like home computers, the Internet, cell phones, and e-commerce. And when that stuff did finally arrive, it cost far more than it did down South. And most of the people around me didn't seem to care; nobody I knew was interested in technology, either as an enthusiast or as a consumer.
And anyone who is good at technology generally moves across the border for the much better wages--and actually being able to work at a real tech company.
In short: brain drain is a factor too.
And anyone who is good at technology generally moves across the border for the much better wages--and actually being able to work at a real tech company.
In short: brain drain is a factor too.
> Growing up in a BC suburb, it seemed like we were always 5-10 years behind the US on things like home computers, the Internet, cell phones, and e-commerce.
That's interesting. Growing up in rural Ontario, we didn't know any different at the time, but in hindsight we were way ahead of the curve. For example, I didn't know anyone who didn't have a computer at home by the early-to-mid 80s, even families that were, looking back, quite poor. Apparently that would have been unthinkable for our American counterparts. We had high speed internet to the farm by the year 2000, and in the early 2010s fibre was installed to the farm. That was (and still largely is) unheard of stateside.
Similarly, programming was taught in schools not only during my time, but also my parents' time. They recount writing programs on punch cards in high school. My generation was writing BASIC already in elementary school. Meanwhile, our friends to the south were still desperately trying to get programming into the classrooms just a few years ago!
Suburbs are stereotypically where you find boring people, so perhaps the disinterest in technology that you experienced was simply down to that stereotype playing out? The people around me have always seemed quite excited about technology. Perhaps B.C. was/is behind the curve, but I'm not sure that is telling about the country as a whole. I don't see that being the case at all further east.
> Canada also doesn't have much of an engineering/technology culture.
I posit that Canada's situation is that it is primarily focused on engineering/technology culture at the cost of not being focused on business culture. Canada is quite strong in engineering (and in producing engineers), but as you allude to, since the business culture is lacking the engineering and engineers go to where there is a business culture (i.e. the USA). Engineering for engineering's sake does not stand on its own. You also need business to prop it up. And that's where Canada fails.
That's interesting. Growing up in rural Ontario, we didn't know any different at the time, but in hindsight we were way ahead of the curve. For example, I didn't know anyone who didn't have a computer at home by the early-to-mid 80s, even families that were, looking back, quite poor. Apparently that would have been unthinkable for our American counterparts. We had high speed internet to the farm by the year 2000, and in the early 2010s fibre was installed to the farm. That was (and still largely is) unheard of stateside.
Similarly, programming was taught in schools not only during my time, but also my parents' time. They recount writing programs on punch cards in high school. My generation was writing BASIC already in elementary school. Meanwhile, our friends to the south were still desperately trying to get programming into the classrooms just a few years ago!
Suburbs are stereotypically where you find boring people, so perhaps the disinterest in technology that you experienced was simply down to that stereotype playing out? The people around me have always seemed quite excited about technology. Perhaps B.C. was/is behind the curve, but I'm not sure that is telling about the country as a whole. I don't see that being the case at all further east.
> Canada also doesn't have much of an engineering/technology culture.
I posit that Canada's situation is that it is primarily focused on engineering/technology culture at the cost of not being focused on business culture. Canada is quite strong in engineering (and in producing engineers), but as you allude to, since the business culture is lacking the engineering and engineers go to where there is a business culture (i.e. the USA). Engineering for engineering's sake does not stand on its own. You also need business to prop it up. And that's where Canada fails.
> Canada also doesn't have much of an engineering/technology culture
The ghosts of Northern Telecom, Blackberry and Instant Pot (ex-Nortel) would like a word.
Also Bombardier aircraft and "AI".
https://www.utoronto.ca/news/u-t-deep-learning-pioneer-geoff...
> When you translate a sentence using Google, or ask Siri to send a text, or play a song recommended by Spotify, you are using a technology that owes much to the innovative research of Geoffrey Hinton.. “deep learning” – a form of artificial intelligence (AI) based on neural networks.. Hinton’s revolutionary contributions to the field have earned him the nickname “the godfather of deep learning,” and have made Canada a hotbed for high tech.. for his excellence as a global pioneer in deep learning, Hinton received a Doctor of Science, honoris causa from the University of Toronto, where he is a University Professor Emeritus.
The ghosts of Northern Telecom, Blackberry and Instant Pot (ex-Nortel) would like a word.
Also Bombardier aircraft and "AI".
https://www.utoronto.ca/news/u-t-deep-learning-pioneer-geoff...
> When you translate a sentence using Google, or ask Siri to send a text, or play a song recommended by Spotify, you are using a technology that owes much to the innovative research of Geoffrey Hinton.. “deep learning” – a form of artificial intelligence (AI) based on neural networks.. Hinton’s revolutionary contributions to the field have earned him the nickname “the godfather of deep learning,” and have made Canada a hotbed for high tech.. for his excellence as a global pioneer in deep learning, Hinton received a Doctor of Science, honoris causa from the University of Toronto, where he is a University Professor Emeritus.
Our universities are decent, I'll give you that. Great for educating folks like Andrej Karpathy so they can go work for American tech companies, or doing research that can be used by American tech companies.
Also, bringing up Blackberry to make Canada seem relevant kind of has the opposite effect.
Also, bringing up Blackberry to make Canada seem relevant kind of has the opposite effect.
> bringing up Blackberry to make Canada seem relevant
The topic of this subthread is not relevance, it's engineering culture.
Canada produced a device that was used by the U.S. President, plus countless executives who could afford any device of that era. It was such a specimen of engineering excellence that you can pay in 2024 to retrofit an original Blackberry keyboard onto a modern iPhone, i.e. no one has yet surpassed that physical keyboard.
U.S. tech companies continue to recruit from U of Waterloo.
The topic of this subthread is not relevance, it's engineering culture.
Canada produced a device that was used by the U.S. President, plus countless executives who could afford any device of that era. It was such a specimen of engineering excellence that you can pay in 2024 to retrofit an original Blackberry keyboard onto a modern iPhone, i.e. no one has yet surpassed that physical keyboard.
U.S. tech companies continue to recruit from U of Waterloo.
This is exactly the complacency that's such a weakness of my country. Canada had one company that made a splash 20 years ago, and that's good enough?
I guess it's fine that engineers will virtually never move from the US to Canada for better pay and more interesting projects, and always the other way around. It's fine that if you ask a Canadian kid to list off apps or tech products that are cool, they won't be able to think of a single Canadian one (most of them won't even have heard of Bombardier). It's fine that our telecoms are some of the worst and most expensive in the world, that this lack of competitiveness and desire for excellence extends to just about every sector of our economy, and Canadians just roll over and take it. It's fine that the brightest students at our universities will leave for other countries with more opportunities and lower cost of living. It's fine, because remember BlackBerry?
I guess it's fine that engineers will virtually never move from the US to Canada for better pay and more interesting projects, and always the other way around. It's fine that if you ask a Canadian kid to list off apps or tech products that are cool, they won't be able to think of a single Canadian one (most of them won't even have heard of Bombardier). It's fine that our telecoms are some of the worst and most expensive in the world, that this lack of competitiveness and desire for excellence extends to just about every sector of our economy, and Canadians just roll over and take it. It's fine that the brightest students at our universities will leave for other countries with more opportunities and lower cost of living. It's fine, because remember BlackBerry?
It's not about Canada having one (or N) successful companies at any point in time, it's that Canada has a long history of innovation followed by assets going missing. The reason for mentioning Blackberry, Nortel and Instant Pot is "remember NOT TO MAKE THOSE MISTAKES AGAIN".
AI mentioned upthread should be relevant to kids. Besides the foundational early work at U of Toronto, there is at least one Canadian AI startup ($3B+ valuation) with global investors and enterprise customers. They have US offices but HQ remains in Canada. Tenstorrent ($1B valuation) silicon startup recruited chip legend Jim Keller (designer of Ryzen & more). Canadian ATI (GPUs) remains part of AMD.
There's a fantastic woodworking tools vendor that's been around for 40 years, earning kudos for both products and company culture, Lee Valley Tools, which sells to both Canada and US customers.
The multi-millennia history of engineering includes the history of military innovation. If Canada is going to be compared to anyone, Finland is a more natural comparison than the southern neighbor with 10X population and 30X military budget.
AI mentioned upthread should be relevant to kids. Besides the foundational early work at U of Toronto, there is at least one Canadian AI startup ($3B+ valuation) with global investors and enterprise customers. They have US offices but HQ remains in Canada. Tenstorrent ($1B valuation) silicon startup recruited chip legend Jim Keller (designer of Ryzen & more). Canadian ATI (GPUs) remains part of AMD.
There's a fantastic woodworking tools vendor that's been around for 40 years, earning kudos for both products and company culture, Lee Valley Tools, which sells to both Canada and US customers.
The multi-millennia history of engineering includes the history of military innovation. If Canada is going to be compared to anyone, Finland is a more natural comparison than the southern neighbor with 10X population and 30X military budget.
The key to your statement: “the ghosts of…”
Failure to defend (e.g. against economic espionage) national treasures is not the same as never having them, or not having a creative culture that gave birth to tech assets worth stealing. Are future generations going to mourn fallen Canadian technology, or rebuild and learn from past mistakes?
Networking 101 doesn't apply when you have collusion between the only two major telecom companies... and they both are bad.
Why spend money on reliability when there are no consequences of not doing so?
Why spend money on reliability when there are no consequences of not doing so?
Rogers is a monopoly. There is no incentive to improve. No incentive to hire good people on the floor to fix fundamental issues. Just middle management and c-level suite hoarding all of the money.
At this point, the government can fine them but it’s just passed on to the consumers. Maybe a random regulatory fee. Can’t close them down otherwise risk massive downtime to critical infra (emergency services)
At this point, the government can fine them but it’s just passed on to the consumers. Maybe a random regulatory fee. Can’t close them down otherwise risk massive downtime to critical infra (emergency services)
Indeed. When a post mortem lists a bunch of engineering 101 tidbits, the reader should ask themselves why deeper questions are not being asked. The failure was not technical, it has no technical solution, it's about people and orgs.
Outages are inevitable, but the Rogers outage in 2022 had some devastating consequences.
Route leaks can happen to anyone, but the fact that it brought down their entire network, including voice and internet services, across all provinces, was unacceptable.
What's even more concerning is that they had no out-of-band access, which meant no management access to their network. This explains why the outage lasted a whopping 24 hours.
In my opinion, the lack of OOB was the most critical and yet the most preventable. Proper OOB is a must; I wouldn't operate a network without it, I don't understand why Rogers thought that was acceptable.
Route leaks can happen to anyone, but the fact that it brought down their entire network, including voice and internet services, across all provinces, was unacceptable.
What's even more concerning is that they had no out-of-band access, which meant no management access to their network. This explains why the outage lasted a whopping 24 hours.
In my opinion, the lack of OOB was the most critical and yet the most preventable. Proper OOB is a must; I wouldn't operate a network without it, I don't understand why Rogers thought that was acceptable.
In my opinion, from what I am reading, there is a root cause analysis in there under the heading "Reliability of Rogers network architecture". I removed the redundant and contradictory parts.
> [...] both the wireless and wireline networks sharing a common IP core network, the scope of the outage was extreme in that it resulted in a catastrophic loss of all services. [...] It is a design choice by [...] Rogers, that seeks to balance cost with performance.
Based on this write up and details I gathered, I believe the root cause, the fundamental reason for the failure is incorrect cost-to-performance balance at senior management.
> [...] both the wireless and wireline networks sharing a common IP core network, the scope of the outage was extreme in that it resulted in a catastrophic loss of all services. [...] It is a design choice by [...] Rogers, that seeks to balance cost with performance.
Based on this write up and details I gathered, I believe the root cause, the fundamental reason for the failure is incorrect cost-to-performance balance at senior management.
I have heard the deployment process at Rogers is something along the lines of:
1) get approval for your changes
2) ssh into a server
3) copy a backup of current jar file
4) copy the new instance
5) run some command to refresh
Can we for a second appreciate how freaking cool it is that a government agency published this?
I wish they didn’t need to and we had transparency policies in place that meant Rogers would have published it at their expense.
But yeah, better than. Nothing :)
But yeah, better than. Nothing :)
Hilarious. Rogers runs the internet and phone lines so when it went down, the devs couldn't:
1) Remote into the boxes to see what's happening.
2) Talk to other devs because their phones are on Rogers network.
My stress couldn't.
1) Remote into the boxes to see what's happening.
2) Talk to other devs because their phones are on Rogers network.
My stress couldn't.
Kye Prigg left Rogers shortly thereafter and ended up at Lumen (merger of CenturyLink and Level3), which is another company facing some issues: https://ir.lumen.com/news/news-details/2023/Lumen-Strengthen... "Kye most recently served as Senior Vice President of Access Networks and Operations for Rogers Communications. "
A Roger's exec used to brag that they were able to release major changes faster than anyone because of their strategy which was "push it out... the customer's will find the remaining bugs". This was more than a decade ago, but it seems like it is still their culture.
> The July 2022 outage was not the result of a design flaw in the Rogers core network architecture.
Talk about some grade A gaslighting here. Reading the post mortem they first tell you it wasn't a design flaw then say they routed all their data through one core router ( including a lack of a management network). Then they say they are going to fix things by separating out the wireless and wired traffic.
Why would you fix things if it wasn't a design flaw?
Out of band access is like resilient architecture 101. Hell even homelabs generally have some way to do it. It's appalling that Rogers didn't have a way to access the core IP routers out of band. Yes it might mean having to use a competitors infrastructure but they ended up having to do it anyway. And with the failure of the service now all the infrastructure providers are under additional scrutiny. Rogers should be striking some agreements with other providers to carry core traffic in case of an outage such as in this DR situation. For example Visa, MC, Amex all have agreements in place to process each others auth data in case the other party goes down. The thinking here being an outage for credit cards makes everyone look bad.
Talk about some grade A gaslighting here. Reading the post mortem they first tell you it wasn't a design flaw then say they routed all their data through one core router ( including a lack of a management network). Then they say they are going to fix things by separating out the wireless and wired traffic.
Why would you fix things if it wasn't a design flaw?
Out of band access is like resilient architecture 101. Hell even homelabs generally have some way to do it. It's appalling that Rogers didn't have a way to access the core IP routers out of band. Yes it might mean having to use a competitors infrastructure but they ended up having to do it anyway. And with the failure of the service now all the infrastructure providers are under additional scrutiny. Rogers should be striking some agreements with other providers to carry core traffic in case of an outage such as in this DR situation. For example Visa, MC, Amex all have agreements in place to process each others auth data in case the other party goes down. The thinking here being an outage for credit cards makes everyone look bad.
For fellow Canadians, I’m quite happy using Beanfield in Toronto. Cheap and decent. Their backbone is probably still Bell or Rogers though, but at least I pay a reasonable $60 for 2gbps down.
No, Beanfield is probably not solely dependent on Bell and/or Rogers for transit. Toronto is one of the few cities in Canada where there are loads of transit providers to purchase bandwidth from (including most tier 1s). There are also multiple carriers with transport fibre going east and west from Toronto that is not owned by Bell or Rogers. Bell and Rogers are not the most competitive for transit, but they may well be in the mix that Beanfield uses.
That said, if 151 Front Street West gets wiped off the face of the planet, the internet is probably going to be degraded for just about everyone in Ontario.
That said, if 151 Front Street West gets wiped off the face of the planet, the internet is probably going to be degraded for just about everyone in Ontario.
Usually smaller ISPs use the Bell/Rogers for last mile, but have their own backbone. IIRC Beanfield is completely independent, but is only available in Toronto
[deleted]
[deleted]
[deleted]
[deleted]
It’s a shame on Canada and on Canadians that foreign competitors are still prohibited from coming into the market.