McDonald's Technology Outage Forces Restaurant Closures(wsj.com)
wsj.com
McDonald's Technology Outage Forces Restaurant Closures
https://www.wsj.com/business/hospitality/mcdonalds-hampered-by-system-outages-in-japan-hong-kong-7e8dd3c7
18 comments
Cloud goes up, cloud goes down. Same story, different day. At least it's not aws-east-1 today.
All of these things that refuse to work without network connection is just mind boggling. I understand the need/want for the companies to have all of that data centralized, but to not have a graceful pathway for when the network fails is just burning cash. Store data locally so it can be pushed to the cloud when network returns, but don't just stop working.
Unfortunately, I think it's more zero-sum than that.
Imagine you're a product manager at McDonald's. For the point-of-sale kiosks, you can build the server component for deployment in the cloud, or you can build it for on-prem and add the overhead of physical deployments across tens of thousands of retail locations, plus maintaining them, plus dealing with franchisees.
You do a cost-benefit analysis of potential downtime from internet outages, and based on internet reliability numbers in the markets you serve (plus an uptime estimate from your devops org), you conclude the occasional outage is worth the trade-off.
Imagine you're a product manager at McDonald's. For the point-of-sale kiosks, you can build the server component for deployment in the cloud, or you can build it for on-prem and add the overhead of physical deployments across tens of thousands of retail locations, plus maintaining them, plus dealing with franchisees.
You do a cost-benefit analysis of potential downtime from internet outages, and based on internet reliability numbers in the markets you serve (plus an uptime estimate from your devops org), you conclude the occasional outage is worth the trade-off.
Maybe it’s their lineage, but I appreciate most native, first party iPhone apps having local first data storage. 4G is increasingly spotty in my area and it’s a crapshoot whether I can reliably load a time sensitive app that requires remote data in some areas. Throwing something in Notes, Photos, or Wallet always works, though. I’m not sure why I still can’t have my full mailbox on my phone, but at least there are several local options.
> I’m not sure why I still can’t have my full mailbox on my phone, but at least there are several local options.
We all started with POP email accounts, but even way back then, that was tedious if you used multiple computers. IMAP was the trail blazing cloud app before cloud was a thing. Also, I would absolutely not want all of that crap from the SPAM folders local to my device taking up space, potentially malicious to my device, or any of the myriad other reasons why email doesn't need to be local to my device.
We all started with POP email accounts, but even way back then, that was tedious if you used multiple computers. IMAP was the trail blazing cloud app before cloud was a thing. Also, I would absolutely not want all of that crap from the SPAM folders local to my device taking up space, potentially malicious to my device, or any of the myriad other reasons why email doesn't need to be local to my device.
The article summary indicates that payment processing was affected. I'm not sure what possibilities there are for offline payment processing, but that's different than just not being able to report sales numbers or something. And opening cash-only is probably not worth the hassle for many locations.
"Back in my day..." we had registers that held the money that customers exchanged for the goods. Servers could write down orders on small pad and hand it to the cooks. There were lots of ways to not close your business because the phone company wasn't working. The move to cashless society has many pros/cons, and this is just showing a glaring item from the cons list. You live by the sword, you die by the sword. Some of us prefer some shields and armor though
Edit: it's so ancient that even I nearly forgot about them, but credit cards used to work by having the company take an impression of the card, then send in the receipts. sure, it might take a few days for transactions to complete, but it worked for several decades. there are fallbacks available to us, if we would just decide to allow for it rather than just failing
Edit: it's so ancient that even I nearly forgot about them, but credit cards used to work by having the company take an impression of the card, then send in the receipts. sure, it might take a few days for transactions to complete, but it worked for several decades. there are fallbacks available to us, if we would just decide to allow for it rather than just failing
Shields and armor are dirty, heavy, and tend to get stolen. And people can’t overspend so ticket totals are lower.
As with most decisions this one is a series of tradeoffs. Appreciate the reminisce and the fun metaphor though!
As with most decisions this one is a series of tradeoffs. Appreciate the reminisce and the fun metaphor though!
You can't get any lower than none because you're too dependent on some digital service to accept a payment. At least accepting cash is > 0. So your point is lacking.
Possibilities to receive cashless payments while the company's network is down:
- printed, static QR code: customer scans it with their payment app and enters the amount
- have an NFC enabled phone that receives payments (most android devices can)
- use the “knuckle busters”: the old school way of imprinting the card number on paper
- NFC enabled device like the one on buses in some cities, synced only at the end of the day
- receive money via one of the marketplace apps: UberEat, Doordash, Grab
- customer browses and pays the company’s online shop in the store instead (this us what Decathlon did once it had a similar outage)
- what else?
I’m quite curious why McDonald’s apparently decided to have no fallback at all. They surely have thought about it. Is the efficiency loss too high? Fraud risk? Training costs?
- printed, static QR code: customer scans it with their payment app and enters the amount
- have an NFC enabled phone that receives payments (most android devices can)
- use the “knuckle busters”: the old school way of imprinting the card number on paper
- NFC enabled device like the one on buses in some cities, synced only at the end of the day
- receive money via one of the marketplace apps: UberEat, Doordash, Grab
- customer browses and pays the company’s online shop in the store instead (this us what Decathlon did once it had a similar outage)
- what else?
I’m quite curious why McDonald’s apparently decided to have no fallback at all. They surely have thought about it. Is the efficiency loss too high? Fraud risk? Training costs?
I was thinking about this whenever I hear about airlines going down when their dinosaur-mainframe software breaks. How about each airport PRINTS an hourly update with ETAs for each inbound plane, where they intend it to depart, and the disposition for the crew of each plane. maybe even IDNs for all the baggage on each plane and each outbound passenger. Might take up 20-30 pages per hour at a non-hub airport. hubs could have their own "edge" device that can print on demand if the mainframe dies.
If the db goes down, make a bunch of copies of that paper, disseminate them, and keep running. worst case is planes don't run at 100% capacity. Just do the best you can to keep the next few legs going without the mothership telling you what to do. Shouldn't be that hard. I know one of the biggest constraints is that crew members run out of fly-able hours. So print that in the backup too: X pilot has Y hours left after HH::MM zulu time. Maybe lobby FAA to allow pilots to fly a few hr past their maximum a few times a year only if the scheduler goes down. I just dont accept that these billion-dollar networks should be this susceptible to disruption given the current state of tech.
If the db goes down, make a bunch of copies of that paper, disseminate them, and keep running. worst case is planes don't run at 100% capacity. Just do the best you can to keep the next few legs going without the mothership telling you what to do. Shouldn't be that hard. I know one of the biggest constraints is that crew members run out of fly-able hours. So print that in the backup too: X pilot has Y hours left after HH::MM zulu time. Maybe lobby FAA to allow pilots to fly a few hr past their maximum a few times a year only if the scheduler goes down. I just dont accept that these billion-dollar networks should be this susceptible to disruption given the current state of tech.
> - NFC enabled device like the one on buses in some cities, synced only at the end of the day
This is also how airlines used to work when it comes to collecting payments
This is also how airlines used to work when it comes to collecting payments
The ice cream machine is actually down today.
[dupe]
More discussion: https://news.ycombinator.com/item?id=39713270
More discussion: https://news.ycombinator.com/item?id=39713270
[deleted]