United Airlines System-Wide Computer Problem(flyertalk.com)
flyertalk.com
United Airlines System-Wide Computer Problem
http://www.flyertalk.com/forum/united-airlines-mileageplus/1693331-shares-down.html
116 comments
At former managed-hosting startup, we would deploy a pair of EMC symmetrixes into our DC cage as storage for our DB tier, with volumes mirrored over the pair. While equipment was being moved into the cage one day, someone accidentally banged the power-switch protection plate on one of the EMCs. Fortunately the power switch had to be held depressed for several seconds to turn off the equipment. Unfortunately, the protection plate itself got jammed in against power switch keeping it depressed. Hoisted by its own petard, as it were.
Good thing we deployed those EMCs in pairs. But then curiosity gets the better of a DC ops guy wondering how the plate got jammed. So he punches the plate on the 2nd EMC causing it to similarly jam and power-off. Doh.
When we told EMC about this flaw in the design, they deployed a fix to the production line - a rubber stopper under the plate next to the switch to protect the switch from the protection plate.
Good thing we deployed those EMCs in pairs. But then curiosity gets the better of a DC ops guy wondering how the plate got jammed. So he punches the plate on the 2nd EMC causing it to similarly jam and power-off. Doh.
When we told EMC about this flaw in the design, they deployed a fix to the production line - a rubber stopper under the plate next to the switch to protect the switch from the protection plate.
This is what happens when Homer Simpson works at your datacentre.
I remember one of the LiveJournal outages (IIRC) was caused by a dude in the datacenter thinking that the emergency power off button opened one of the doors.
Had a similar experience at an old company. One of the light switches in server room was placed next to a switch connected to the UPS of certain servers (I have no idea how/why this even came about).
Confused the heck out of us when we were trying to figure out why some of our servers went on UPS power randomly at night. Turns out we'd get the notifications on nights the cleaning crew decided to flick the lights off then on again.
Confused the heck out of us when we were trying to figure out why some of our servers went on UPS power randomly at night. Turns out we'd get the notifications on nights the cleaning crew decided to flick the lights off then on again.
A data center where I work had a naked big red switch without a protection plate. Until I flagged up this lack and let the IT people know about it, it was a disaster waiting to happen.
There's a plate on the switch now, praise Eris.
There's a plate on the switch now, praise Eris.
I've seen many places where "the solution" to the unprotected EPO button was just a simple plastic cup!
Similar situation here. A big red button on the wall in the basement data center - nobody knew what it was for and were afraid to turn it off. One day a night operator decided he was going to flip the switch and see what happened. It cut power to the entire datacenter, including the elevator systems. We spent a few days getting all the systems back online.
Had another instance where a pipe ran from the floor above right over the UPS. Construction workers on the first floor decided to poor some unusued paint into it (figured it was a drain pipe) and of course, we lost power again.
Had another instance where a pipe ran from the floor above right over the UPS. Construction workers on the first floor decided to poor some unusued paint into it (figured it was a drain pipe) and of course, we lost power again.
I have a nagging (intuition? feeling?) that software safety/reliability/security needs are going to explode soon (because unreliabilities multiply in non-resilient systems interacting with each other) and that these are simply foreshocks.
(yeah, I know security is already a huge deal, but as we come to trust software systems more and more, the safety/reliability factor will come more into play)
EDIT: This is also part of the reason I've been learning Elixir (http://elixir-lang.org/) since it's based on the highly-resilient Erlang and is designed to embrace failure. This was also informed by me reading Nassim Taleb's book "Antifragile" as well as "Thinking in Systems: A Primer" by the (late) Donella Meadows.
(yeah, I know security is already a huge deal, but as we come to trust software systems more and more, the safety/reliability factor will come more into play)
EDIT: This is also part of the reason I've been learning Elixir (http://elixir-lang.org/) since it's based on the highly-resilient Erlang and is designed to embrace failure. This was also informed by me reading Nassim Taleb's book "Antifragile" as well as "Thinking in Systems: A Primer" by the (late) Donella Meadows.
Nancy Leveson's been saying that for decades: http://www.amazon.com/Safeware-Computers-Nancy-G-Leveson/dp/...
I doubt it. Nobody wants to pay for that.
You might be right, unfortunately. If it's cheaper to buy insurance that will cover the (expected) losses caused by outages, most organizations will choose to do that instead of making the software more failure-resistant. The problem is that insurance only works well for isolated incidents, but a software failure can cause a cascading failure with a huge impact. Insurance companies generally aren't prepared for that and don't have the resources to pay out to everyone.
But aren't the insurance companies smart enough to figure this out and start correcting their rates to be much higher?And if they actually have their acts together, wouldn't those same insurance companies start insisting on basic audits of their client's systems?
I actually don't know about this stuff, so any correction of my thoughts is appreciated.
I actually don't know about this stuff, so any correction of my thoughts is appreciated.
Quite often operational losses aren't insured against this kind of error, for example the Knight-Ridder automated trading losses. Sufficiently big operational failures can just destroy companies, especially small companies.
System audits would have to be standardised. There's bits of this in ISO9001, PCI compliance, FIPS, and so on. But the technology changes rapidly and the insurance companies don't have the expertise.
System audits would have to be standardised. There's bits of this in ISO9001, PCI compliance, FIPS, and so on. But the technology changes rapidly and the insurance companies don't have the expertise.
A cynic in me feels that someone will figure out the problems with cascade case and insure from failure of insurance companies to pay out insure money. Just like during 2008 financial crisis.
That's what existing reinsurance companies do, if I understand correctly. They insure the insurance companies.
> But aren't the insurance companies smart enough to figure this out and start correcting their rates to be much higher?
That seems like a naive "MARKET WILL FIX IT" approach.
More likely, if the market does fix it, it'll be by having insurance companies deploy actual inspectors who know what they are doing and what sorts of problems to work for.
They might even be a fun combination of physical pentester/irl chaos monkey. Doesn't that sound like a fun job?
That seems like a naive "MARKET WILL FIX IT" approach.
More likely, if the market does fix it, it'll be by having insurance companies deploy actual inspectors who know what they are doing and what sorts of problems to work for.
They might even be a fun combination of physical pentester/irl chaos monkey. Doesn't that sound like a fun job?
Cascading failure will be covered in the expected loss, so insurance fees go up and up ?
I think we just need to build smarter. I've become disillusioned with the "runaway state" problem (as well as spaghetti-dependency problems) in OO languages which contributes to bugs and general nondeterministic behavior as well as making long term maintenance difficult, and at the same time I've become enamored of unit test suites and functional immutable languages like Elixir as well as static code analysis tools (I'm still coming around to Haskell-esque typing, but I generally think it's a good idea to write "potentially provably correct" code that has parts which provably have no side effects).
So obviously the "move fast and break things" philosophy is meant for some fun web applications. But what's the equivalent "modern best practice" for systems that are much more weighed towards stability and resilience as opposed to new features?
Systemic quality focus would be a good start. Deming-based management philosophy driving a systems-oriented model.
This actually applies to all businesses; speed improves, market fit improves (quality is just "what is good and valuable" after all), employee happiness improves (exponential gains from that alone). The effects are systemically positive.
But, no one cares, and the belief that we have to crack the whip to get people to make things faster, and we should reward the good people and punish the bad ones—will continue on as pure religious fallacy resulting in the failure or constant-operation-at-the-edge-of-failure of everything involving more than five people, probably until the end of the human race.
This actually applies to all businesses; speed improves, market fit improves (quality is just "what is good and valuable" after all), employee happiness improves (exponential gains from that alone). The effects are systemically positive.
But, no one cares, and the belief that we have to crack the whip to get people to make things faster, and we should reward the good people and punish the bad ones—will continue on as pure religious fallacy resulting in the failure or constant-operation-at-the-edge-of-failure of everything involving more than five people, probably until the end of the human race.
Well, on one hand there's six sigma and then there's the agile manifesto.
That's quite a wide gulf.
Would love to hear from people who have worked with developing / managing relatively high quality software with relatively modern methods.
Edit: Maybe NASA's Faster Better Cheaper comes to mind...
That's quite a wide gulf.
Would love to hear from people who have worked with developing / managing relatively high quality software with relatively modern methods.
Edit: Maybe NASA's Faster Better Cheaper comes to mind...
Six Sigma and Agile are indeed at pretty opposite sides of the spectrum.
Deming -- W. Edwards Deming, that is -- is somewhere in the middle, around the right balance. He advocated for spreading a philosophy of systems thinking, scientific method, and statistical understanding, while simultaneously empowering employees by recognizing the power and responsibility of management and leadership, and understanding the motivation from a scientifically accurate psychological viewpoint.
It's a correct framework, and it's all aimed at driving quality by improving the things that directly impact it at a base level: fundamentally, how people work together, how they build systems that work, and how they're motivated (and demotivated) in reality.
Deming -- W. Edwards Deming, that is -- is somewhere in the middle, around the right balance. He advocated for spreading a philosophy of systems thinking, scientific method, and statistical understanding, while simultaneously empowering employees by recognizing the power and responsibility of management and leadership, and understanding the motivation from a scientifically accurate psychological viewpoint.
It's a correct framework, and it's all aimed at driving quality by improving the things that directly impact it at a base level: fundamentally, how people work together, how they build systems that work, and how they're motivated (and demotivated) in reality.
Schuberg Philis (https://www.schubergphilis.com/) in the Netherlands has been selling 100% functional up-time for a while now. They've set their entire business model and management structure up to support this.
Doesn't come cheap though.
Doesn't come cheap though.
I worked on United's computer systems for a year (never that one though), and so I get nervous when I see a headline like that. True story: one of their systems still runs on a mainframe that has 9 bits in a byte!
Oh god stories like this scare me.
One of my friends used to intern for a very large company that maintained software for flight control towers. His entire summer was spent writing bash tests for these old Fortran apps that kept planes from running into each other. Most of the mainframes still had tape.
That's the code that keeps track of our planes.
One of my friends used to intern for a very large company that maintained software for flight control towers. His entire summer was spent writing bash tests for these old Fortran apps that kept planes from running into each other. Most of the mainframes still had tape.
That's the code that keeps track of our planes.
I must admit I don't take that as a negative thing. Code that's so old had a lot of time for corner cases to be ironed out. If it's maintained properly it's probably less buggy than any rewrite, even though fortran is less enticing than rust as a language.
I did not know that was ever a thing. I knew that bit counts had been lower - 5 or 6 - but assumed once we hit 8, the whole power-of-2 thing was too comfortable in a binary system to ever be anything besides a power of 2 again - and sure enough, we get 16 bit, 32 bit, and 64 bit systems, and double-byte and triple-byte char sets, etc. Need more space? Take another byte.
Was the 9th bit special? Or just a standard bit in the byte?
Was the 9th bit special? Or just a standard bit in the byte?
36 bit was popular for scientific computing because 35 gives you a sign bit and 10 decimal digits (yes, that's weird, but that's the argument I read everywhere, including at https://en.m.wikipedia.org/wiki/36-bit. 35 likely was skipped because it its only divisors are 5 and 7, limiting instructions to 7 bits even then was felt to be too restricting. For some architectures, the DoD had a say in this, too. See https://en.m.wikipedia.org/wiki/Unisys_2200_Series_system_ar...)
36 bits got us the 6-bit character (10 digits, 26 letters, and punctuation) with six characters in a word. Because of that, some OSes had six-character file names.
If you want to get upper- and lowercase, you need more than 6 bits. 9 is the smallest divisor larger than 6 of 36, so nine-bit characters made sense.
On such systems, file names could still use 6-bit characters, while applications used 9-bit ones. Also, some instructions could work on words, half words, quarter words, or sixth words.
36 bits got us the 6-bit character (10 digits, 26 letters, and punctuation) with six characters in a word. Because of that, some OSes had six-character file names.
If you want to get upper- and lowercase, you need more than 6 bits. 9 is the smallest divisor larger than 6 of 36, so nine-bit characters made sense.
On such systems, file names could still use 6-bit characters, while applications used 9-bit ones. Also, some instructions could work on words, half words, quarter words, or sixth words.
Someone probably read too much Iain M. Banks and decided to adopt Marain's base-9 system :)
If I'm not mistaken, some old consoles used a 9-bit RGB encoding, so this could theoretically help there. Minecraft also uses a 9-bit system for Redstone, afaik.
If I'm not mistaken, some old consoles used a 9-bit RGB encoding, so this could theoretically help there. Minecraft also uses a 9-bit system for Redstone, afaik.
The PowerPC based AS/400 systems were technically 65-bit machines, as the extra bit was a privilege flag to separate system code from user code. Hardware enforced security - one reason why it is such a stable machine.
If hex hadn't been invented yet, 9 bits works better with octal. I don't know if that was a factor.
Sabre, still runs all their reservations through ztpf... If you know anyone there ask them about how they get the data off the mainframe. The story I heard sounds like the process was designed by Rube Goldberg.
I knew someone at United that once offered to give me a tour of one of their data centers: "It is like a computer museum - we have one of everything." Hard to imagine that they would have problems as a result. United is a really, really bad airline.
I bet this is an issue with an old mainframe used somewhere in the booking system, something that has worked well but is difficult to fix when things go wrong.
I think there is / will be a lot of money to be made trying to solve the problem of software security and reliability. This is obviously an extremely difficult problem, however the number of ancient systems that we currently have interconnected I think more large scale outages like this are inevitable.
I think there is / will be a lot of money to be made trying to solve the problem of software security and reliability. This is obviously an extremely difficult problem, however the number of ancient systems that we currently have interconnected I think more large scale outages like this are inevitable.
I submit there's more money to be [saved] from not replacing old systems.
Case in point, take this HackerNews fan favorite[1] about a school district using an "ancient system" and avoiding $M's in incremental costs.
[1] https://news.ycombinator.com/item?id=9705830
edit: As mentioned in [1], I assume that at least someone is aware of the cost-benefit of potential projects; in turn, I assume that someone would've pulled the trigger if the $'s make sense.
Case in point, take this HackerNews fan favorite[1] about a school district using an "ancient system" and avoiding $M's in incremental costs.
[1] https://news.ycombinator.com/item?id=9705830
edit: As mentioned in [1], I assume that at least someone is aware of the cost-benefit of potential projects; in turn, I assume that someone would've pulled the trigger if the $'s make sense.
Truth is that most reservation systems are built on TPF and it isn't really easy to replace.
just in case we have readers to whom TPF isn't a household name: https://en.wikipedia.org/wiki/Transaction_Processing_Facilit...
That sounds like a heck of a system.
Official status page: http://www.fly.faa.gov/ois/jsp/summary_sys.jsp
Currently says:
Currently says:
ATCSCC ADVZY 027 DCC 07/08/15 UAL GROUND STOP REVISION
DESTINATION AIRPORT: ALL AIRPORTS
FACILITIES INCLUDED: ALL FACILITIES
GROUND STOP PERIOD: 08/1200Z - 08/1315Z
REASON: USER REQUEST AUTOMATION ISSUESGood to see the Sun Microsystems favicon on the FAA status page - not seen one of those for a while!
Last one I noticed was the FCC's comment system.
Ouch. And only a month after another major systems outage: http://www.wired.com/2015/06/united-flights-grounded-mysteri...
Was reading this on HN and heard it on NPR simultaneously.
I have a sneaking suspicion that booking systems for most airlines run atop legacyware. It just seems like the type of thing that would've been put in place long ago and then be very expensive to migrate/updgrade.
I have a sneaking suspicion that booking systems for most airlines run atop legacyware. It just seems like the type of thing that would've been put in place long ago and then be very expensive to migrate/updgrade.
Oh, it's not a suspicion.
http://www.pacbiztimes.com/2012/04/06/united-takes-a-step-ba...
http://www.pacbiztimes.com/2012/04/06/united-takes-a-step-ba...
> The biggest problem, one that would drive any tech-savvy user crazy, is that United junked an award-winning, state-of-the-art reservation system and adopted the Continental Airlines model based on older technology known as System One.
Well, that answers that.
Well, that answers that.
Not a word about it on United's web site. Flight status page doesn't load correctly. "Today's Operations" gives an error message. United's Twitter is silent.
Meanwhile news articles and twitter complaints abound. http://mashable.com/2015/07/08/united-computer-problems-flig...
Meanwhile news articles and twitter complaints abound. http://mashable.com/2015/07/08/united-computer-problems-flig...
United is a mess. I had the misfortune of flying with them a couple months ago. I ended up in a city a 2 hour drive from my actual destination and had to rent a car on my own to get to where I was going.
That was the worst of it, but almost every flight I saw on the way (both ones I was on and other flights at nearby gates) was delayed or overbooked or otherwise messed up in some way.
That was the worst of it, but almost every flight I saw on the way (both ones I was on and other flights at nearby gates) was delayed or overbooked or otherwise messed up in some way.
If I ever listened to horror stories like these, there wouldn't be an airline left I would fly with. This is textbook; replace United with Delta, Southwest, Air Canada, etc., at will. The only company I've used that I haven't heard exactly this type of complaint about is Widerøe, a tiny regional line in Norway with a fleet of prop planes. This is unfortunate, as I live in Pennsylvania and get motion sickness on those little aircraft. And I imagine that if I spoke Norwegian—it seemed to me that while many Norwegians speak English, they don't do much complaining in it when the native language would do—I wouldn't even have Widerøe left.
It's also worth remembering that quality of service depends on a host of variables, including departure/arrival cities (major hubs are better than regional airports), routes, and times (morning flights tend to be less delayed, Thursday afternoon flying always a bit of a shitshow), etc.
I fly out of UA hubs frequently and have had nothing but excellent service from them this year (over 50 segments flown this year, ~60K miles).
Definitely helps to have status too...
Personally, I consider SW to the be shittiest of them all. I hate having to fight for a seat...
Lastly, use google.com/flights by far my favorite booking tool now.
I fly out of UA hubs frequently and have had nothing but excellent service from them this year (over 50 segments flown this year, ~60K miles).
Definitely helps to have status too...
Personally, I consider SW to the be shittiest of them all. I hate having to fight for a seat...
Lastly, use google.com/flights by far my favorite booking tool now.
I've been flying 1-2 times a month for the past year or so, mostly on US Air or American (same thing now) and I never saw anything approaching the level of problems United had on one trip.
Obviously, with travel sometimes things go wrong, but the quantity and severity of things going wrong at United makes me think there's something off about the way they're running their airline. For instance, the two times this year that they've had to ground all flights.
Obviously, with travel sometimes things go wrong, but the quantity and severity of things going wrong at United makes me think there's something off about the way they're running their airline. For instance, the two times this year that they've had to ground all flights.
Overall JetBlue and Virgin have been great. AA and US Air have been good. United is the worst.
AA destroyed a piece of baggage of mine. They denied responsibility. I sued and won. Then I couldn't collect because that judgement was essentially nullified because AA was going through bankruptcy.
I have 1k status with United -- I fly internationally with them almost monthly. The problems I've had with United have been when bags have to be interlined to Brussels Airlines and occasionally (surprisingly) Lufthansa. I've also had Lufthansa somehow think it was a good idea for 2 toddlers to sit in scattered seats rows away from their parents (who were also rows apart as well, despite having checked in almost 24 hours before the flight and being Star Alliance Gold.) I've had Brussels Airlines say they were going to "gate check" a stroller only to have it show up days later. I've been stranded in Detroit back in the Northwest Airlines days when aircrews hadn't showed up to work. I've been stuck in Paris when the Air France pilots decide that salaries up to $300,000 per year just aren't enough. On a recent United trip from Hartford to Marseille, I was stuck in Hartford for % extra hours for an airplane that was stuck in Newark (just a <50 minute flight away.) I then missed a cascade of connections leaving me rather miserable. However, United sorted the problem and got me on my way as quickly as possible. Let's not forget Jet Blues antics on multiple occasions a few years ago: a 10.5 hour tarmac delay, a 7 hour tarmac delay among several other extremely long tarmac delays. AA had 14 long tarmac (over 3 hours) delays in February. United had zero long tarmac delays during the same period. Envoy/American Eagle was in last place for on-time arrivals last year.
I'm not defending United. I'm not disparaging the others. The fact is that the air transport industry is extremely complex and perceptions of quality are as varied as their are passengers in the sky.
Every airline sucks and every airline is great. Pick a day, pick a destination and roll the dice. When you fly often enough it seems like it all averages out to just one level of melancholic service; unless you're flying on Singapore Airlines -- then it just becomes sublime.
I have 1k status with United -- I fly internationally with them almost monthly. The problems I've had with United have been when bags have to be interlined to Brussels Airlines and occasionally (surprisingly) Lufthansa. I've also had Lufthansa somehow think it was a good idea for 2 toddlers to sit in scattered seats rows away from their parents (who were also rows apart as well, despite having checked in almost 24 hours before the flight and being Star Alliance Gold.) I've had Brussels Airlines say they were going to "gate check" a stroller only to have it show up days later. I've been stranded in Detroit back in the Northwest Airlines days when aircrews hadn't showed up to work. I've been stuck in Paris when the Air France pilots decide that salaries up to $300,000 per year just aren't enough. On a recent United trip from Hartford to Marseille, I was stuck in Hartford for % extra hours for an airplane that was stuck in Newark (just a <50 minute flight away.) I then missed a cascade of connections leaving me rather miserable. However, United sorted the problem and got me on my way as quickly as possible. Let's not forget Jet Blues antics on multiple occasions a few years ago: a 10.5 hour tarmac delay, a 7 hour tarmac delay among several other extremely long tarmac delays. AA had 14 long tarmac (over 3 hours) delays in February. United had zero long tarmac delays during the same period. Envoy/American Eagle was in last place for on-time arrivals last year.
I'm not defending United. I'm not disparaging the others. The fact is that the air transport industry is extremely complex and perceptions of quality are as varied as their are passengers in the sky.
Every airline sucks and every airline is great. Pick a day, pick a destination and roll the dice. When you fly often enough it seems like it all averages out to just one level of melancholic service; unless you're flying on Singapore Airlines -- then it just becomes sublime.
That's not accurate about the Air France pilots -- they were striking because the airline was moving to replace them with cheaper pilots.
Pity they don't fly to more places.
Was there a storm or something? Usually there is some motivating reason why flights are screwed up.
Not that I know of. My particular flight was delayed because they had to replace a piece of the navigation equipment on the plane. This delay caused me to miss a connecting flight and that's how I ended up in an entirely different city.
This happens with every airline. No airline has magical planes that never break and they all have similar maintenance schedules on the exact same planes running the exact same workload. Blame Boeing for making a crappy plane if they keep breaking down.
I wasn't so annoyed that the flight got delayed and I got stranded in the wrong city, it was the way they handled it. United's response was:
1. Do nothing. You're stuck here. We'll get you on the next flight. Oh, the next available flight isn't for over a week. Sorry. No, we can't pay for your rental car.
2. Upon renting my own car, and writing to customer service to complain I got a $125 voucher for United. Great. Not enough to buy an entire ticket, so it just ensures I'll have to continue giving money to this airline that failed to deliver what I gave them money for in the first place.
3. After several weeks of emails back-and-forth with customer service finally a manager agrees to issue a reimbursement for the rental car as a one-time exception to their policy of never doing that. (but not the $7 worth of gas. I guess they just have to draw a line somewhere? Oh, and they also rescinded the voucher. No big deal since I'm not too keen to fly with them again anyway)
So, yeah. Sometimes shit goes wrong when you travel. Everybody knows that. How airlines act to fix it makes all the difference. If you hamstring all your customer service reps so they can't actually solve someone's problem, it makes something that's already annoying way more frustrating.
1. Do nothing. You're stuck here. We'll get you on the next flight. Oh, the next available flight isn't for over a week. Sorry. No, we can't pay for your rental car.
2. Upon renting my own car, and writing to customer service to complain I got a $125 voucher for United. Great. Not enough to buy an entire ticket, so it just ensures I'll have to continue giving money to this airline that failed to deliver what I gave them money for in the first place.
3. After several weeks of emails back-and-forth with customer service finally a manager agrees to issue a reimbursement for the rental car as a one-time exception to their policy of never doing that. (but not the $7 worth of gas. I guess they just have to draw a line somewhere? Oh, and they also rescinded the voucher. No big deal since I'm not too keen to fly with them again anyway)
So, yeah. Sometimes shit goes wrong when you travel. Everybody knows that. How airlines act to fix it makes all the difference. If you hamstring all your customer service reps so they can't actually solve someone's problem, it makes something that's already annoying way more frustrating.
What does "all airports" mean? Global or just US domestic?
US domestic would be my guess. ie: All airports under FAA authority.
Very little news about this on Google News, but heard over the local Chicago ABC affiliate that the FAA attributed this to an "automation error"
Edit: And its Twitter account has been relatively inactive, with more than 30 minutes since the last reply-to or general tweet...presumably a lot complaining tweets have come in in the last 30 minutes https://twitter.com/united/with_replies
Edit: And its Twitter account has been relatively inactive, with more than 30 minutes since the last reply-to or general tweet...presumably a lot complaining tweets have come in in the last 30 minutes https://twitter.com/united/with_replies
Just a minute ago they tweeted about it: https://twitter.com/united/status/618769524544942081
Took them until just now to go "oh, we should probably post something that isn't a @reply".
https://twitter.com/united/status/618777538865799168
https://twitter.com/united/status/618777538865799168
I bet they could have saved themselves untold numbers of telephone calls in the support queue, and thousands of in-person queries to ticket agents and other airport staff with one or two tweets and a facebook post or two.
Interesting bits:
"Departing DEN; taxied and then returned to gate. Pilot says nationwide failure of "three or four" computer systems. Only information from airport staff is that since the computers are down UAL can't book pax onto any other airline ..."
"Systemwide Ground Stop posted at FAA: Due to USER REQUEST DUE TO AUTOMATION ISSUES. UAL AND SUBS ONLY., departure traffic destined to ALL airport will not be allowed to depart until at or after 13:15 UTC."
"Departing DEN; taxied and then returned to gate. Pilot says nationwide failure of "three or four" computer systems. Only information from airport staff is that since the computers are down UAL can't book pax onto any other airline ..."
"Systemwide Ground Stop posted at FAA: Due to USER REQUEST DUE TO AUTOMATION ISSUES. UAL AND SUBS ONLY., departure traffic destined to ALL airport will not be allowed to depart until at or after 13:15 UTC."
Considering just yesterday their flight system didn't believe they flew from SFO for 2+ hours in the morning (I have screenshots), i'm not all that shocked.
Currently in the air on a United flight, EWR -> SFO. Took off at 7:30ish. No mention of this in the airport or on the plane. Yikes
"Automation error" rings a funny bell in my head, since I'm currently re-reading "A Fire Upon The Deep".
Is anyone familiar with the UA tech stack? I'd be curious to see what they're running.
They're running SHARES internally; most everything else is specific applications with hooks into shares for data processing and return.
I was able to get checked in via the United android App around 9am ET.
Perhaps someone wasn't following the bug bounty rules?
Off-topic: Please use ISO 8601 format(YYYY-MM-DD) for dates in titles. The US date format hurts my poor logical soul.
https://en.wikipedia.org/wiki/ISO_8601
https://en.wikipedia.org/wiki/ISO_8601
There's little endian, big endian, and the US date format, which I like to call middle endian.
Well, to attribute that to SHARES, we should go with big endian ;)
Wouldn't the PC term be middle Native American?
See also: https://xkcd.com/1179/
I love how they write the dates when you hover over the image. XKCD.
We'll get around to it just as soon as we get around to using the metric system and providing universal health care.
Or just use words (8th July 2015). But yes, YYYY-MM-DD actually sorts correctly.
It's called "8. Juli 2015" where I live. Use numbers, please.
ISO all the way. Words have issues for different month languages (e.g. Auot vs. August), but otherwise is more elegant (no delimiter needed, e.g. 08JUL15)
For future proofing please use 5 digit years (YYYYY-MM-DD) e.g. 02015-07-08. :P
Stardate 1673.85
Or use JSON: {"year": 2015, "month":7, "day":8}
Except that month starts at 0.
Are the airlines using systems that modern?
I was guessing something a bit more
01 OUTAGE-DATE-DATA.
I always kinda liked the meme from that language of "variable as picture". Calling something a picture has inherent and obvious implications WRT call by value vs call by name.
On the other hand intrinsic functions like FUNCTION CURRENT-DATE are not as logically amusing.
I was guessing something a bit more
01 OUTAGE-DATE-DATA.
05 OUTAGE-DATE.
10 OUTAGE-YEAR PIC 9(04).
10 OUTAGE-MONTH PIC 9(02).
10 OUTAGE-DAY PIC 9(02).
05 OUTAGE-TIME.
10 OUTAGE-HOURS PIC 9(02).
10 OUTAGE-MINUTE PIC 9(02).
10 OUTAGE-SECOND PIC 9(02).
10 OUTAGE-MILLISECONDS PIC 9(02).
05 OUTAGE-DIFF-FROM-GMT PIC S9(04).
I always kinda liked the meme from that language of "variable as picture". Calling something a picture has inherent and obvious implications WRT call by value vs call by name.
On the other hand intrinsic functions like FUNCTION CURRENT-DATE are not as logically amusing.
Its sad that this is the top comment. When I visit European sites I don't demand they change things just for me. Maybe be more of a gracious visitor?
Why is this being down voted? It is true. When British sites spell things with superfluous letters (colour, for instance,) should HN readers demand that they change it to the more efficient spelling? American English is spoken natively by more people in the world than British English, so therefore must we banish British English from these pages? Of course not. We could argue that the US has the largest number of visitors to HN than any other country. HN is an American site. YC is an American organization. Thus, those not comfortable with American conventions really ought to get over it.
The United States has the world's largest economy, according to the IMF, World Bank and UN. So obviously the US is doing something right. Perhaps the rest of the world ought to adopt American conventions. I say that in jest, but the point is that criticizing the American way of doing things is a popular sport, yet at the end of the day, the American system has resulted in an economic output greater than Germany, the UK, France and Italy combined. The EU has a per-capita GDP of $36,779 and the US is at $54,601. So apparently something is working with the American way of doing things. Just because "the rest of the world" does it doesn't make it better. This whole idea that the American way of doing things is somehow inferior is just as ridiculous as the British/French rivalry.
The fact that the parent comment is the top comment is ridiculous. It's a petty thing about which to complain and adds nothing to the discussion. In fact, the parent comment ought to be down voted for being absolutely irrelevant to the posting in question.
The United States has the world's largest economy, according to the IMF, World Bank and UN. So obviously the US is doing something right. Perhaps the rest of the world ought to adopt American conventions. I say that in jest, but the point is that criticizing the American way of doing things is a popular sport, yet at the end of the day, the American system has resulted in an economic output greater than Germany, the UK, France and Italy combined. The EU has a per-capita GDP of $36,779 and the US is at $54,601. So apparently something is working with the American way of doing things. Just because "the rest of the world" does it doesn't make it better. This whole idea that the American way of doing things is somehow inferior is just as ridiculous as the British/French rivalry.
The fact that the parent comment is the top comment is ridiculous. It's a petty thing about which to complain and adds nothing to the discussion. In fact, the parent comment ought to be down voted for being absolutely irrelevant to the posting in question.
I worked for a company that did a special translation of its software from American English to British English alongside the other languages they did. The Brits loved it, although they were mystified by the pricing - it cost them in Pounds as much as it cost the US customer in Dollars.
I tried to get a former employer to switch to ISO format, since they had offices both in the US and Ireland. There was too much pushback so I didn't succeed, but I at least got them close: 2015-Jul-08.
It's still month 7 so it's hard to be confused.
PS. From Europe.
But a month from now, some people will be confused for sure. So, it's more future-proof to settle for ISO or alphanumeric format to avoid confusion and miscommunication.
Let's just be thankful that we haven't had a major air-based catastrophe due to an outage or hacking!
Planes with pilots in them are mostly autonomous and can avoid each other as well as the ground.
https://en.wikipedia.org/wiki/Ground_proximity_warning_syste...
https://en.wikipedia.org/wiki/Traffic_Collision_Avoidance_Sy...
https://en.wikipedia.org/wiki/%C3%9Cberlingen_mid-air_collis...
https://en.wikipedia.org/wiki/Ground_proximity_warning_syste...
https://en.wikipedia.org/wiki/Traffic_Collision_Avoidance_Sy...
https://en.wikipedia.org/wiki/%C3%9Cberlingen_mid-air_collis...
Yes, mostly.
https://en.wikipedia.org/wiki/Vitaly_Kaloyev
https://en.wikipedia.org/wiki/Vitaly_Kaloyev
I wouldn't get on a Russia autonomous flight for a million dollars. Between the corruption, hackey engineering, and complete disregard for safety standards, Russian accidents surprise no one. The Tu-154 accident list is long and scary. Hell, look at all the fires that broke out.
https://en.wikipedia.org/wiki/Tupolev_Tu-154#Incidents_and_a...
https://en.wikipedia.org/wiki/Tupolev_Tu-154#Incidents_and_a...
Automated, yes. Autonomous, no. The plane voices an alert, but the pilots still have to react to it.
I meant autonomous with respect to the ground.
That's twice in a short time now - 'coincidence' on Ian Fleming's scale. A third time is enemy action. Airlines do seem like a pretty juicy target for cyber war operations - you can cause a gigantic amount of disruption with a successful attack on a single system.
That seems like quite a leap given how often complex computer systems fail without any malicious act.
Indeed. Never attribute to malice that which is adequately explained by stupidity. https://en.wikipedia.org/wiki/Hanlon's_razor
I don't see any convincing rebuttal. Why are airline systems not a juicy target? Why would a successful attack on the system not cause gigantic disruption? Why have these long-running, stable systems only recently begun failing so severely?
Yes, one is wise to attribute to incompetence over malice, but these systems are demonstrably not run by incompetents: they've been in operation for decades.
Yes, one is wise to attribute to incompetence over malice, but these systems are demonstrably not run by incompetents: they've been in operation for decades.
Or, you know, the load is increasing over time, and the system is failing to scale to it, and so will fail more and more frequently as the load continues to increase?
The amount of daily UAL passenger seats doesn't change or grow radically over weeks or months. What is the scale-up here? Everyone using their phones to check rez/status/boarding passes?
Some quick googling suggests they take delivery of several-many dozen planes a year (based on various order/delivery/etc. coverage), which would absolutely cause a step function in the number of passenger seats.
I expect there's growth on all of these:
- Total planes
- Total flights
- Number of routes
- Number of passengers
- Passenger utilization of electronic boarding
- Passenger utilization of reservation modification (seat changes, upgrades, etc.)
- Passenger utilization of in-flight electronic amenities (wifi, in flight entertainment, etc.) that are billed through the system (United has a proprietary WiFi that's tied to your frequent flier account)
- Online booking
- Travel agent/reseller pricing queries
- Travel agent/reseller booking
I'm not saying any of these are dramatically increasing, but I'd bet they're all going up slowly, which will add more and more load to the system.
I expect there's growth on all of these:
- Total planes
- Total flights
- Number of routes
- Number of passengers
- Passenger utilization of electronic boarding
- Passenger utilization of reservation modification (seat changes, upgrades, etc.)
- Passenger utilization of in-flight electronic amenities (wifi, in flight entertainment, etc.) that are billed through the system (United has a proprietary WiFi that's tied to your frequent flier account)
- Online booking
- Travel agent/reseller pricing queries
- Travel agent/reseller booking
I'm not saying any of these are dramatically increasing, but I'd bet they're all going up slowly, which will add more and more load to the system.
Capacity only increased 0.01% over the first quarter of 2015. Passenger loads have remained flat at 81.1%. I would suggest that compared to last year, there hasn't been much change in scale. Also, all of the above systems are not linked together so I'm not sure an increase in one of those factors would be enough to take down and entire system. I could be wrong, I am definitely not an aviation IT expert. (Apparently Untied could use some though, so perhaps I might need to think about adding that to my skills!)
source: http://www.businesstravelnews.com/Travel-Management/Drop-In-...
source: http://www.businesstravelnews.com/Travel-Management/Drop-In-...
Went like this: Guy who shows around the new datacenter/ops guy demonstrated how the emergency power off works by lifting the protection plate. Protection plate unhinges suddenly and droppes onto emergency power off button. Hilarity ensues.