Ask HN: What's behind all the UK IT failures this month?
51 comments
Politicisation of technology.
It's a layer 8 problem. Human political expectations of technology are outpacing the engineering reality. Over the past two decades overselling of "utopias" mixed with professional management arrogance and ignoring engineers, experts and what people actually want.
That, plus a mixture of crony contracts and bad project management.
We've ended up with over-complex systems that we don't have the human capacity or money to maintain and secure.
A telling remark by a Labour politician to the question "What is the greatest concern about a 'cashless economy' was "Making sure people are not left behind".
In other words, we're forcing this on people whether they want it or not, and whether it works or not. The idea that there might be "risks" or necessary safeguards was simply not conceivable.
That's what happened with the Post Office Horizon system and it's what's happening with other new systems pushed out "for our own good"
That's not a remark against technology or modernisation, it's a criticism of bloody-minded recklessness and anti-democratic hubris.
Good technology requires care. We talk about it a lot here [0].
[0] https://cybershow.uk
It's a layer 8 problem. Human political expectations of technology are outpacing the engineering reality. Over the past two decades overselling of "utopias" mixed with professional management arrogance and ignoring engineers, experts and what people actually want.
That, plus a mixture of crony contracts and bad project management.
We've ended up with over-complex systems that we don't have the human capacity or money to maintain and secure.
A telling remark by a Labour politician to the question "What is the greatest concern about a 'cashless economy' was "Making sure people are not left behind".
In other words, we're forcing this on people whether they want it or not, and whether it works or not. The idea that there might be "risks" or necessary safeguards was simply not conceivable.
That's what happened with the Post Office Horizon system and it's what's happening with other new systems pushed out "for our own good"
That's not a remark against technology or modernisation, it's a criticism of bloody-minded recklessness and anti-democratic hubris.
Good technology requires care. We talk about it a lot here [0].
[0] https://cybershow.uk
> Politicisation of technology.
> It's a layer 8 problem. Human political expectations of technology are outpacing the engineering reality. Over the past two decades overselling of "utopias" mixed with professional management arrogance and ignoring engineers, experts and what people actually want.
> That, plus a mixture of crony contracts and bad project management.
> We've ended up with over-complex systems that we don't have the human capacity or money to maintain and secure.
That is a LOT of VERY sweeping statements to a pretty specific question it seems of payment processors, can you give some examples of what you're talking about ?
> That, plus a mixture of crony contracts and bad project management.
> We've ended up with over-complex systems that we don't have the human capacity or money to maintain and secure.
That is a LOT of VERY sweeping statements to a pretty specific question it seems of payment processors, can you give some examples of what you're talking about ?
> can you give some examples of what you're talking about ?
PLease see the podcast linked above where we have spent many hours researching, analysing, writing and presenting on specific cases. In episodes to come we're working on joining the dots, finding common factors in these cases. Enjoy, and please give feedback if you feel able.
PLease see the podcast linked above where we have spent many hours researching, analysing, writing and presenting on specific cases. In episodes to come we're working on joining the dots, finding common factors in these cases. Enjoy, and please give feedback if you feel able.
McDonalds, Sainsbury, Tesco, barely matters. But Greggs, that's some serious shit depriving people of pies. We need answers.
The south is organising food drops to the North of England as we speak.
Poncey southern bastards. Still, pies come before pride ...
"Pies before pride" would be a great t-shirt.
Having said this, though, my nearest Kent town has 2 Greggs, 3 if you count a garage on the outskirts of town.
Even Canary Wharf has a Greggs now.
Even Canary Wharf has a Greggs now.
Pukka Pies can pissoff!
Is Putin hitting us where it hurts?
Please watch "The Undeclared War", on UK-Russia Cyberwar [fictional TV series] so the 2nd season gets produced.
Hannah Khalique-Brown taught herself Python and C++ as well as playing with IDA Pro to prepare for her role in that.
Work is getting outsourced to India. Most CS grads don’t know how a computer works. People get hired for reasons other than merit. And ChatGPT…
Only 3 comments into a story about a UK problem and here's a comment blaming it on foreigners. Incredible.
I take the spirit of that comment that they didn't blame the foreigners, they blamed the outsourcing. Because even if the provider was local the result would have likely been similar (and we do have recent examples, Horizon and not only)
They didn't say "Work is getting outsourced" they said said "Work is getting outsourced to India"
India is sadly at the forefront of low-quality tech sweatshops and may have even pioneered the concept... but not to worry, with remote work I'm sure other countries will catch up eventually too!
So are Ukraine, Romania, and Poland.
In McD's and ATT's case (because you commented on that as well) - their platform and security teams are located at HQ in the Central Sales Zone
In McD's and ATT's case (because you commented on that as well) - their platform and security teams are located at HQ in the Central Sales Zone
> Ukraine, Romania, and Poland.
They have to compete with India (because otherwise why would you go there instead of just going with India) so they have to justify it by offering at least a slightly higher quality bar to differentiate themselves from a market saturated by low-quality sweatshops. Personal experience, I don't (yet) get spam emails offering me dev services from Eastern Europe (or Thailand/Philippines for that matter) but I've had my fair share of "agencies" promising me the moon at cheap prices, with all signs pointing that the origin of those is India.
---
> In McD's and ATT's case
The problem with large, non-IT "legacy" companies is less outsourcing itself (although outsourcing is a symptom of it) but that there is no healthy engineering culture. Nobody competent joins/stays there because if you're competent and actually want that skill to be rewarded you'll have much better options, so the only thing that remains is mediocrity.
Since you mentioned ATT, ask yourself why there is zero innovation coming out of the telco space? It is after all just bits and bytes (and with things like IMS/VoLTE more and more of it is bog standard IP and SIP). We've seen innovation in banking (see challenger banks in the UK), we've seen it in electric utilities (Octopus Energy in the UK has a healthy Python-focused engineering arm), yet for some reason nothing ever comes out of telcos beyond excuses to misbill people and coming up with ever-outlandish use-cases for 5G.
They have to compete with India (because otherwise why would you go there instead of just going with India) so they have to justify it by offering at least a slightly higher quality bar to differentiate themselves from a market saturated by low-quality sweatshops. Personal experience, I don't (yet) get spam emails offering me dev services from Eastern Europe (or Thailand/Philippines for that matter) but I've had my fair share of "agencies" promising me the moon at cheap prices, with all signs pointing that the origin of those is India.
---
> In McD's and ATT's case
The problem with large, non-IT "legacy" companies is less outsourcing itself (although outsourcing is a symptom of it) but that there is no healthy engineering culture. Nobody competent joins/stays there because if you're competent and actually want that skill to be rewarded you'll have much better options, so the only thing that remains is mediocrity.
Since you mentioned ATT, ask yourself why there is zero innovation coming out of the telco space? It is after all just bits and bytes (and with things like IMS/VoLTE more and more of it is bog standard IP and SIP). We've seen innovation in banking (see challenger banks in the UK), we've seen it in electric utilities (Octopus Energy in the UK has a healthy Python-focused engineering arm), yet for some reason nothing ever comes out of telcos beyond excuses to misbill people and coming up with ever-outlandish use-cases for 5G.
> Personal experience, I don't (yet) get spam emails offering me dev services from Eastern Europe (or Thailand/Philippines for that matter) but I've had my fair share of "agencies" promising me the moon at cheap prices, with all signs pointing that the origin of those is India.
Personal experience does not give you the right to be prejudiced.
And ime EPAM and ELITEX have been equally as horrid as WITCH, yet I don't presume all East European programmers suck.
> why there is zero innovation coming out of the telco space
Plenty of innovation happens. They were some of the earliest adopters of Cloud and K8s, and a number of features and products that CSPs like AWS, Azure, GCP, Alibaba, etc now sell were initially developed by telcos and then resold to the CSPs.
---------
I get it, you're a freelancer/contractor and you must be jaded due to competition, yet in all honesty you aren't any different than a drone at EPAM or TCS.
If you are going to be publicly prejudiced, at least have the decency to be anonymous and not directly link your professional information.
It's a small industry
Personal experience does not give you the right to be prejudiced.
And ime EPAM and ELITEX have been equally as horrid as WITCH, yet I don't presume all East European programmers suck.
> why there is zero innovation coming out of the telco space
Plenty of innovation happens. They were some of the earliest adopters of Cloud and K8s, and a number of features and products that CSPs like AWS, Azure, GCP, Alibaba, etc now sell were initially developed by telcos and then resold to the CSPs.
---------
I get it, you're a freelancer/contractor and you must be jaded due to competition, yet in all honesty you aren't any different than a drone at EPAM or TCS.
If you are going to be publicly prejudiced, at least have the decency to be anonymous and not directly link your professional information.
It's a small industry
> Plenty of innovation happens
Over a decade ago, sure.
Nowadays, it seems like they're struggling with basic things like defending against social-engineered SIM swaps (even when the account has an explicit note about it) and not leaking data (not talking about the "contested" AT&T breach, but the fully admitted T-Mobile one for example)?
But don't take it from me, here's a more reliable source with way more experience in the field: https://berthub.eu/articles/posts/5g-elephant-in-the-room/
> you must be jaded due to competition
Outsourced sweatshops aren't meaningful competition for me - they have their customers and I have mine, and those rarely intersect. The sales pipeline and target market is completely different. My problem with them is more that they often manage to get government contracts (and thus get tax money) and then deliver no/subpar results.
---
At the end of the day it's clear we have a different view on these things so let's agree to disagree - my mistake is your opportunity. If you feel like prejudice against common outsourcing sweatshops is mistaken, you are welcome to exploit it (you can do so via the stock market, or by taking on locally-priced contracts, outsourcing them and pocketing the difference).
> have the decency to be anonymous
I'm trying to share my experience and opinion, not fire cheap shots under the cover of a throwaway. If this means some people choose not to work with me, so be it. If I wanted to be a people-pleaser and never say anything remotely negative I'd instead join a big corp and only communicate through the PR department.
Over a decade ago, sure.
Nowadays, it seems like they're struggling with basic things like defending against social-engineered SIM swaps (even when the account has an explicit note about it) and not leaking data (not talking about the "contested" AT&T breach, but the fully admitted T-Mobile one for example)?
But don't take it from me, here's a more reliable source with way more experience in the field: https://berthub.eu/articles/posts/5g-elephant-in-the-room/
> you must be jaded due to competition
Outsourced sweatshops aren't meaningful competition for me - they have their customers and I have mine, and those rarely intersect. The sales pipeline and target market is completely different. My problem with them is more that they often manage to get government contracts (and thus get tax money) and then deliver no/subpar results.
---
At the end of the day it's clear we have a different view on these things so let's agree to disagree - my mistake is your opportunity. If you feel like prejudice against common outsourcing sweatshops is mistaken, you are welcome to exploit it (you can do so via the stock market, or by taking on locally-priced contracts, outsourcing them and pocketing the difference).
> have the decency to be anonymous
I'm trying to share my experience and opinion, not fire cheap shots under the cover of a throwaway. If this means some people choose not to work with me, so be it. If I wanted to be a people-pleaser and never say anything remotely negative I'd instead join a big corp and only communicate through the PR department.
Which is even more jarring as McD's platform team is in their Chicagoland office.
The amount of unbridled racism on HN is getting annoying.
We outsource to Israel, Poland, Romania, etc as well, yet no one is acting racist about them.
The amount of unbridled racism on HN is getting annoying.
We outsource to Israel, Poland, Romania, etc as well, yet no one is acting racist about them.
Because TCS, HCL, Wipro, Infosys, and some western companies also keeping outsourcing teams there (i.e. Atos) provide consistently below-absolute-zero quality that pushes and enforces bad stereotypes.
Because the real issue is not Indian people or their skills, but corporations that established a pattern of supplying people whose only skill is alleged knowledge of english language. The best thing I can say about those companies is that their IT sweatshops are better than starving, and those who learn anything seem to escape to better workplaces.
I have yet to meet as consistent (negative) quality signal as finding out the job has been outsourced to consulting company in India. I have worked with, well, low-quality coworkers (and betcha been one myself more than once) from many countries - but nowhere else there is such negative selection.
For comparison, there's no such correlation if the coworker is indian but working elsewhere.
Because the real issue is not Indian people or their skills, but corporations that established a pattern of supplying people whose only skill is alleged knowledge of english language. The best thing I can say about those companies is that their IT sweatshops are better than starving, and those who learn anything seem to escape to better workplaces.
I have yet to meet as consistent (negative) quality signal as finding out the job has been outsourced to consulting company in India. I have worked with, well, low-quality coworkers (and betcha been one myself more than once) from many countries - but nowhere else there is such negative selection.
For comparison, there's no such correlation if the coworker is indian but working elsewhere.
Adding the country receiving the outsourced work doesn't change the rest of the sentence to mean those getting the work are to blame.
McD's Platform Team is at their HQ in Chicagoland.
At some point it is plain racism and prejudice.
At some point it is plain racism and prejudice.
The headquarters location doesn't really matter in the context of outsourcing. Was the work in question done in Chicago, India, or somewhere else?
Assuming racism isn't particularly useful without more context. Racism is an intent, simply saying the work was outsourced to India could be as benign as having incorrect information and absolutely doesn't say anything about the people of India themselves.
Assuming racism isn't particularly useful without more context. Racism is an intent, simply saying the work was outsourced to India could be as benign as having incorrect information and absolutely doesn't say anything about the people of India themselves.
The work is done IN CHICAGOLAND!
I have managed and sold products or funded companies that have sold to the very same guys who do this stuff at McDs and insert_F1000_here
The fact you automatically assume "India" is sheer racism, as if all good programmers are only in the US and all bad ones in India.
Newsflash - all humans are kinda good and kinda bad.
I have managed and sold products or funded companies that have sold to the very same guys who do this stuff at McDs and insert_F1000_here
The fact you automatically assume "India" is sheer racism, as if all good programmers are only in the US and all bad ones in India.
Newsflash - all humans are kinda good and kinda bad.
Fair enough, I think that falls into incorrect information then. I didn't make the original claim and wouldn't have assumed it was outsourced at all, I have no idea the inner dynamics of McDonald's engineering teams.
My point was simply that saying the work was outsourced to India isn't itself enough to make a very real claim that someone is racist.
Having incorrect information isn't racist, making an assumption that outsourced work went to India if that matches a person's experience with outsourced IT work ending up in India isn't racist, and saying the country to which work may have been outsourced to says absolutely nothing about the people there.
My point was simply that saying the work was outsourced to India isn't itself enough to make a very real claim that someone is racist.
Having incorrect information isn't racist, making an assumption that outsourced work went to India if that matches a person's experience with outsourced IT work ending up in India isn't racist, and saying the country to which work may have been outsourced to says absolutely nothing about the people there.
They gave 4 reasons, and I think all are guesses without knowledge of the IT failures in question.
A great quote I love.
Being British is about driving a German car to an Irish pub for a Belgian beer, then traveling home, grabbing an Indian curry or a Turkish kebab on the way, to sit on Swedish furniture and watch American shows on a Japanese TV.
All while being suspicious of foreigners.
Being British is about driving a German car to an Irish pub for a Belgian beer, then traveling home, grabbing an Indian curry or a Turkish kebab on the way, to sit on Swedish furniture and watch American shows on a Japanese TV.
All while being suspicious of foreigners.
> Belgian beer
Completely unrelated, but I love a classic British ESB. The microbreweries at attempt them here in the US just suck. I miss a nice Spitfire XPA.
Completely unrelated, but I love a classic British ESB. The microbreweries at attempt them here in the US just suck. I miss a nice Spitfire XPA.
[deleted]
Considering chat GPT is a few years old it seems unlikely those who used it in university and are now probably just Junior Developers are responsible for wide spread outages at major companies
I worked for one of the affected companies as a rare example of a permanent, in-house engineer.
They had a _lot_ of contractors, some independent based in the UK, some contracts with companies with a local presence who would work from the office alongside us, and some in India.
All were equally shit at worst, with only the independent contractors proving the exception on occasion.
I recall being stuck with “senior” colleagues who couldn’t manage to upgrade a dependency even while being handheld.
They had a _lot_ of contractors, some independent based in the UK, some contracts with companies with a local presence who would work from the office alongside us, and some in India.
All were equally shit at worst, with only the independent contractors proving the exception on occasion.
I recall being stuck with “senior” colleagues who couldn’t manage to upgrade a dependency even while being handheld.
All of these enterprises are examples where IT is a "cost center", not a "value center". So you have this conundrum where everything goes smooth and the higher ups ask "why do we spend so much money on this without anything in return?". And when something does go wrong they are likely to repeat said question.
Sainsburys said it was due to an error with "an overnight software update". Perhaps they all use the same software but updated at different times?
https://twitter.com/sainsburys/status/1768972295622553900
https://twitter.com/sainsburys/status/1768972295622553900
Maybe all the interns that were left in charge didn't properly do what the AI had said?
more likely all the interns that were left in charge DID properly do what the AI had said...
Maybe the dog[0] failed?
[0] https://quoteinvestigator.com/2022/01/30/future-factory/
[0] https://quoteinvestigator.com/2022/01/30/future-factory/
Most plausible explanation of all
Random chance and years of underinvestment and tech debt
McDonald's wasn't unique to the UK as it was global. McDonald's Japan was affected.
They could definitely all be unconnected. It’s not that suspicious. I guess we’ll find out if they ever publish post mortems.
Do you have sources about the "outages" ? (did not make the headlines internationally..)
It they all shut down because of "a payment system", then a simple explanation would be a failure of... the same payment system that they all happen to use ?
(It's not like Greggs is going to fully develop a payment system in house without relying on any infra, I suppose ?)
It they all shut down because of "a payment system", then a simple explanation would be a failure of... the same payment system that they all happen to use ?
(It's not like Greggs is going to fully develop a payment system in house without relying on any infra, I suppose ?)
Sainsbury's and Tesco: https://www.theguardian.com/business/2024/mar/16/sainsburys-...
Greggs: https://www.theguardian.com/business/2024/mar/20/greggs-shop...
The Sainsbury's and Tesco ones happened on the same day (last Saturday), but the scale was very different. Sainsbury's were unable to take card payments nationwide. My local one said it was due to the internet connection in their store not working. It also affected planned deliveries, likely because the scanners they use to fetch and pick the orders didn't have a connection.
Sainsbury's blamed this on a botched "software update".
Greggs happened today and was card payment related but doesn't seem to be nationwide (I've not been to one today so not 100% sure if that's true).
Greggs: https://www.theguardian.com/business/2024/mar/20/greggs-shop...
The Sainsbury's and Tesco ones happened on the same day (last Saturday), but the scale was very different. Sainsbury's were unable to take card payments nationwide. My local one said it was due to the internet connection in their store not working. It also affected planned deliveries, likely because the scanners they use to fetch and pick the orders didn't have a connection.
Sainsbury's blamed this on a botched "software update".
Greggs happened today and was card payment related but doesn't seem to be nationwide (I've not been to one today so not 100% sure if that's true).
McDonald was worldwide. Sainsbury, Tesco and Greggs seems to be related somehow? At least they are all payment related.
Just Eat went down Friday, too...
Mandatory cert rotation forced by a dependency that had a leak, and they all didn’t handle it properly. (Just a guess)
Probably leetcode.
Surely these can't be unconnected? Can anyone shed any light on what's happening here behind the scenes?