Latest 737 Max Fault That Alarmed Test Pilots Rooted in Software(bloomberg.com)
bloomberg.com
Latest 737 Max Fault That Alarmed Test Pilots Rooted in Software
https://www.bloomberg.com/news/articles/2019-07-27/latest-737-max-fault-that-alarmed-test-pilots-rooted-in-software
156 comments
That sounds like adding a heuristic which works most of the time which is something I can see myself doing to cover some pathfinding issue in a game where it's sufficient if it works 99,9% of the time or when working on a simulation where I can look on the result data afterwards and can see if something looks highly unusual despite the heuristic and improve it iteratively - a luxury you don't get with air planes, at least not without casualties. Additionally given the very limited computing power of the hardware in the 737 adding even more software to it might create even more problems.
Yeah, it’s a nice way of saying “this is a hack”.
Move fast and kill things. At least that seems to be the motto of self-driving carmakers, too.
Wasn’t aware the compute power was low on the 737. Interested if you have some more info?
Always thought they would spec a substantially more processing capability than required. I understand economics are a factor but surely it’s not through necessity to save power?
Always thought they would spec a substantially more processing capability than required. I understand economics are a factor but surely it’s not through necessity to save power?
The issue is (from what I understand as layman) that to avoid recertification they kept their old computer hardware from earlier 737 generations, and added more and more tasks to do for it.
https://www.nytimes.com/2019/06/26/business/boeing-737-max-f... (somewhere in the middle: "The issue discovered last week is linked to the data-processing speed of a specific flight control computer chip, according to the two people with knowledge of the matter. In the test, the F.A.A. pilot encountered delays in executing a crucial step required to stabilize an aircraft." )
https://www.nytimes.com/2019/06/26/business/boeing-737-max-f... (somewhere in the middle: "The issue discovered last week is linked to the data-processing speed of a specific flight control computer chip, according to the two people with knowledge of the matter. In the test, the F.A.A. pilot encountered delays in executing a crucial step required to stabilize an aircraft." )
That's surreal - avionics should run on hard real time. (Whether it's offline scheduled and thus "impossible" to miss a deadline, or if it's "soft" realtime and merely testing ensures this, is not the important part. The important part is that it should never miss a deadline.)
I can understand it looks weird but it isn't. A faster processor has no more real time guarantees than a slower processor from the 80s
Of course, the faster one can do more stuff in less time, but for flying an airplane (or going to the moon) the amount of computing power needed is much less than people imagine.
The issue here is adding more computing into an aging platform. But there's nothing inherently wrong into using an older platform into the flight computer.
Now I'm not sure if for certification purposes they would be allowed to replace it with modern hardware but just keeping the functionality the same, I believe the answer would be yes, that they are allowed (but they probably didn't want to rewrite the sw and recertify the hw)
Of course, the faster one can do more stuff in less time, but for flying an airplane (or going to the moon) the amount of computing power needed is much less than people imagine.
The issue here is adding more computing into an aging platform. But there's nothing inherently wrong into using an older platform into the flight computer.
Now I'm not sure if for certification purposes they would be allowed to replace it with modern hardware but just keeping the functionality the same, I believe the answer would be yes, that they are allowed (but they probably didn't want to rewrite the sw and recertify the hw)
Processing has very little to do with hitting realtime deadlines. Either you do or you don’t. In fact, sometimes faster CPUs are worse for realtime because their latency can be worse.
is it me or the certification looks at the hardware but not at the software ?
Certification mostly looks at the behaviour, how you get there is your IP for the most part
For example, the regulator doesn't care (much) how you built your wing, but it does care that's able to withstand 150% of the maximum expected forces.
You can have parts done by different manufacturers (but to the same spec) and this is sometimes done on purpose in redundancy settings so that unknown bugs in one model are less likely to show up in a different model.
For example, the regulator doesn't care (much) how you built your wing, but it does care that's able to withstand 150% of the maximum expected forces.
You can have parts done by different manufacturers (but to the same spec) and this is sometimes done on purpose in redundancy settings so that unknown bugs in one model are less likely to show up in a different model.
> to avoid recertification they kept their old computer hardware from earlier 737 generations,
This is unbelievably fucked
This is unbelievably fucked
No, it's not. Certification is an extremely expensive process with dubious benefits for safety. Why would you change hardware that has worked fine if it can fulfill your requirements?
That's related to an issue often found in medical devices development: the certified toolchain contains errors, but you cannot use the newer version with fixed errors because it is not certified. Of course, the new toolchain passes the same certification suite as does the faulty one (because the certification body does only check the documentation for completeness and basic plausibility and does not have any means to test anything themselves), it's just that it isn't commercially viable to "buy the official stamp".
Because it obviously did not work fine and does not fulfill the requirements.
What seems to have dubious benefits for safety is lying to the regulators and shoehorning new processes into old technology. Because hundreds of people have died from that exact thing, just on this one airplane.
What seems to have dubious benefits for safety is lying to the regulators and shoehorning new processes into old technology. Because hundreds of people have died from that exact thing, just on this one airplane.
The airplanes didn't crash because the computer was too slow. They crashed because safety critical data was read from a sensor without redundancies.
But now during simulations test pilots apparently found situations where the computer was too slow to handle their input in adequate time. So while the crash was not the caused by this problem, there seems to be a problem.
> No, it's not. Certification is an extremely expensive process with dubious benefits for safety. Why would you change hardware that has worked fine if it can fulfill your requirements?
Is it fair to say the hardware is fulfilling the requirements though?
Is it fair to say the hardware is fulfilling the requirements though?
If the requirements were for it to buck like a bronco before slamming into the ground then I say job well done.
Here's an article about it. According to this piece, 737 avionics run on a pair of 80286s.
https://www.moonofalabama.org/2019/06/boeings-software-fix-f...
https://www.moonofalabama.org/2019/06/boeings-software-fix-f...
If true it uses a 286, for some relative comparison, the fastest 286 is 1.5 MIPS. A run of the mill current Arduino is probably 20 MIPS, and a Raspberry PI3 is 2451 MIPS.
Can’t find the exact MIPS estimate for the Harris CS80C286 25Mhz 286, but given that the 1.5MIPS was quoted for the 10Mhz Intel one I’d guess it’s substantially faster.
Although still slower than an Arduino :)
Although still slower than an Arduino :)
I love how HN is taking umbrage in the rest of the comments about Boeing being too aggressive with MCAS et al. redesign.
Then taking umbrage here that they were too conservative and possibly froze the hardware spec.
Then taking umbrage here that they were too conservative and possibly froze the hardware spec.
The general consensus I've seen here on HN is that Boeing should have done a clean sheet design rather than trying to shoehorn increasingly ill-fitting features into a decades-old design. You're not accurately representing that view.
http://news.aa.com/news/news-details/2011/AMR-Corporation-An...
"American also intends to order 100 of Boeing's expected new evolution of the 737NG, with a new engine that would offer even more significant fuel-efficiency gains over today's models. American is pleased to be the first airline to commit to Boeing's new 737 family offering, which is expected to provide a new level of economic efficiency and operational performance, pending final confirmation of the program by Boeing. This airplane would be powered by CFM International's LEAP-X engine."
When a customer demands...
"American also intends to order 100 of Boeing's expected new evolution of the 737NG, with a new engine that would offer even more significant fuel-efficiency gains over today's models. American is pleased to be the first airline to commit to Boeing's new 737 family offering, which is expected to provide a new level of economic efficiency and operational performance, pending final confirmation of the program by Boeing. This airplane would be powered by CFM International's LEAP-X engine."
When a customer demands...
No umbrage here. Just interesting to me. My Commodore 64 was half a MIP.
The should at least get 1202 error..
Yep. At this point in time they should have plenty of test cases and regression tests which should catch such situations. While Boeing may feel they are doing a lot of testing in the simulator - there are way fewer simulators staffed with test pilots than there are 737 MAX out there. When such errors surface at this time in manual testing it calls into question the development process. And a lot of the trust - aka. certification - is based on that process.
As I have mentioned here before: The angle of attack sensor was never intended for the purpose it is now used for - military aircraft may do such things but not as an afterthought and not connected with a control surface which has a lot of delay (trim wheel winding. Delay is a major factor in destabilizing control loops). Equivalent sensors are the airspeed pitot tubes but they are a much more robust concept. Even they have brought down airplanes. We lack sufficient real world (heat, humidity, snow) information about the quality of the sensor data even before it may get distorted by transmission and processing on its way to the control law. Cleaning and validating sensor data is tricky under the best of circumstances. It does not sound like they have a handle on it.
Hi Boeing, what real world data from mounted angle of attack sensors have you analyzed? Why do you believe this data sufficiently represents climate zones and flight conditions (take-off, landing, turbulence, malfunction modes)? How do you clean and validate the sensor data? Please explain, because otherwise I won't step into one of these machines.
As I have mentioned here before: The angle of attack sensor was never intended for the purpose it is now used for - military aircraft may do such things but not as an afterthought and not connected with a control surface which has a lot of delay (trim wheel winding. Delay is a major factor in destabilizing control loops). Equivalent sensors are the airspeed pitot tubes but they are a much more robust concept. Even they have brought down airplanes. We lack sufficient real world (heat, humidity, snow) information about the quality of the sensor data even before it may get distorted by transmission and processing on its way to the control law. Cleaning and validating sensor data is tricky under the best of circumstances. It does not sound like they have a handle on it.
Hi Boeing, what real world data from mounted angle of attack sensors have you analyzed? Why do you believe this data sufficiently represents climate zones and flight conditions (take-off, landing, turbulence, malfunction modes)? How do you clean and validate the sensor data? Please explain, because otherwise I won't step into one of these machines.
Can you forward the data Airbus, your car or public transport vehicle manufacturer, Otis or equivalent elevator company obviously sent you for their respective operating environments?
Wow a textbook example of a bandaid solution. Reminds me of something a novice programmer would do. When faced with a complex problem....nothing beats an even more complicated solution!
When your design is a mess you're probably better off to scrap it altogether and start from a clean piece of paper. "Fixes" to a bad design are adding to the mess. And in this case the mess has already cost hundreds of lives. You cannot "patch" this...
I see this sentiment a lot, but its kind of surprising. Traditional wisdom states the opposite - never do a full rewrite, you lose the years of buit-up fixes and edge cases and have to learn from scratch in too many places.
Despite everything, I still don't think the solution to the 737 MAX problem is to start over. The base problem that Boeing has its actually their terrible engineering culture, the one that values shortcuts and profit over thoroughness and safety.
Even if they started from a blank piece of paper, they'd have that engineering culture problem (and to be fair, it is far more likely an engineering management culture problem), and they'd still turn out a design that kills people.
Despite everything, I still don't think the solution to the 737 MAX problem is to start over. The base problem that Boeing has its actually their terrible engineering culture, the one that values shortcuts and profit over thoroughness and safety.
Even if they started from a blank piece of paper, they'd have that engineering culture problem (and to be fair, it is far more likely an engineering management culture problem), and they'd still turn out a design that kills people.
The problem here is that what Boeing is selling ("737 that spends significantly less fuel and doesn't need recertification") simply can't be made safe. Whatever results from the marketing driven goal (selling something that doesn't and can't exist in reality) will remain unsafe as long as all the "already sold features" are preserved.
So it's not if they should "start from clean" but just accept the need for the new certification and then do the changes that are really needed. It will be less profitable for Boeing but it righly should be this time! They are killing people otherwise.
So it's not if they should "start from clean" but just accept the need for the new certification and then do the changes that are really needed. It will be less profitable for Boeing but it righly should be this time! They are killing people otherwise.
The funny thing is this plane already exists. It’s called the 757 and Boeing stopped making it 15 years ago because the airlines wanted stretched 737s far more than they wanted the clean sheet redesign.
It's one of those fixes where they add up the value of human life if something goes wrong and say 'OK we can handle that, ship it"
I wonder if the human life calculation worked well this time. As far as I see, Boeing lost more than the sum of the human lives; they also lost reputation for everything new they’ve designed in the last 7 years being corrupted, and they also engulfed the reputation of FAA with them, whose agents would fit the definition of “corrupted” by any people’s definition (I know, they are not, they just used agents of Boeing to inspect Boeing because they were understaffed), and the FAA showed the last step of failure by not admitting that the plane had to be stopped until a few days after the European agencies.
In other words, even in financial terms, it cost more than damages. It may have cost the entire company. They “DeHavailland”’ed their company. Ever heard of DeHavailland? No? That’s probably to do with their 4 successive deintegrating planes that “CEOs have complete trust in.” It just died, as a name. The risk is high.
In other words, even in financial terms, it cost more than damages. It may have cost the entire company. They “DeHavailland”’ed their company. Ever heard of DeHavailland? No? That’s probably to do with their 4 successive deintegrating planes that “CEOs have complete trust in.” It just died, as a name. The risk is high.
There is so much of U.K. which is incredible and depressing at the same time. Completely pummelled by war, (receiving no aid like West Germany did) they managed to, in 1949, release a Space Age passenger commercial airliner, defining the genre. It looks modern even today.
Yet, fumbled the ball.
They also had a space program, closed it down after their first satellite, which was a success.
Concorde! (With France.)
It goes on an on, I'm sure you have favourite examples.
It's heart wrenching.
Yet, fumbled the ball.
They also had a space program, closed it down after their first satellite, which was a success.
Concorde! (With France.)
It goes on an on, I'm sure you have favourite examples.
It's heart wrenching.
Generally short-sighted politics gets in the way in the UK based on my own opinion. Support/budgets/etc get withdrawn at precisely the wrong moment (and usually for some petty point-scoring or if ministers are running scared from the media/opposition)
Its happening right now too with electric cars and solar electricity - solar has generally been seen as a success with a lot of installations taking us to 3rd place in Europe for solar megawatts despite there being essentially no sun (1), and EV cars are starting to become something actually viable for most people's lives. The government has just announced that the UK is now legally bound to be carbon neutral by 2050... yet the government chose also to revoke funding and support for both solar and EVs and so subsequently update has plummeted (2, 3) right at this vital point.
1 - https://en.wikipedia.org/wiki/Solar_energy_in_the_European_U... 2 - https://www.theguardian.com/environment/2016/apr/08/solar-in... 3 - https://www.autocar.co.uk/car-news/industry/new-car-sales-fa...
Its happening right now too with electric cars and solar electricity - solar has generally been seen as a success with a lot of installations taking us to 3rd place in Europe for solar megawatts despite there being essentially no sun (1), and EV cars are starting to become something actually viable for most people's lives. The government has just announced that the UK is now legally bound to be carbon neutral by 2050... yet the government chose also to revoke funding and support for both solar and EVs and so subsequently update has plummeted (2, 3) right at this vital point.
1 - https://en.wikipedia.org/wiki/Solar_energy_in_the_European_U... 2 - https://www.theguardian.com/environment/2016/apr/08/solar-in... 3 - https://www.autocar.co.uk/car-news/industry/new-car-sales-fa...
“The largest recipient of Marshall Plan money was the United Kingdom (receiving about 26% of the total), followed by France (18%) and West Germany (11%).”
https://en.wikipedia.org/wiki/Marshall_Plan
Since others may be curious, what the UK received under the Marshall Plan looks to be about 3x what they paid back under Lend-Lease. (Which in turn was maybe 10% of the cost of what was shipped.)
You may like this take (rant?) which describes further loans from US to UK.
https://www.bbc.co.uk/history/british/modern/marshall_01.sht...
Interesting factoids:
Marshall plan replaced and thankfully rejected the previous Morgenthau plan - which would have explicitly ruled out any aid to Germany, and actually further destroyed the country.
In a fine bit of statesmanship Marshall plan aid was offered to the Soviets, who rejected it. It was offered in fairness, but in the certain knowledge Stalin would refuse. It would never have got through Congress anyway. :)
Marshall plan replaced and thankfully rejected the previous Morgenthau plan - which would have explicitly ruled out any aid to Germany, and actually further destroyed the country.
In a fine bit of statesmanship Marshall plan aid was offered to the Soviets, who rejected it. It was offered in fairness, but in the certain knowledge Stalin would refuse. It would never have got through Congress anyway. :)
Today I learned!
Last week I was at Munich, with +40 other Europeans: French, Spanish, Italian, German, Dutch, Swedes and of course Irish and British. As I was sipping my beer at a biergarten I realized we were all speaking English, in Germany, surrounded by people whose native language wasn't English, right when Brexit is about to happen, just when any British could say: we won. But no, as you said, they just fumbled the ball, again.
I thought the US won..
The United Kingdom received over twice the amount that West Germany did under the Marshall plan.
https://en.wikipedia.org/wiki/Marshall_Plan#Expenditures
The U.K. had an educational computer initiative with the BBC and Acorn. Worked wonders. Then they discontinued the effort.
Yes, although ARM came out of that and now look at that (...or your smartphone) so its not all a disaster.
The Raspberry Pi is a spiritual successor which I am generally pleased to see. The RPi was I think partly responsible for a real Computer Science curriculum getting back into schools ... I hate to say it but I think it was Michael Gove that made this happen!
The Raspberry Pi is a spiritual successor which I am generally pleased to see. The RPi was I think partly responsible for a real Computer Science curriculum getting back into schools ... I hate to say it but I think it was Michael Gove that made this happen!
Arm Holdings is now owned by Japan's Softbank, because a Conservative (business-first) administration thought selling off the family silver was a fine idea.
This is pretty typical of the UK government's approach to the computer hardware/software industry. Had the first business computer industry in Europe (anyone else remember the LEO line of mainframes?) and a promising software biz, of which only bits remain. (The gaming industry is thriving, but the rest mostly ended up being absorbed by US multinationals. I could speculate that having a common language, in combination with a series of governments who believed in leaving things to the Invisible Hand, and a business culture dominated by accountants, were at the root of the problem ...)
This is pretty typical of the UK government's approach to the computer hardware/software industry. Had the first business computer industry in Europe (anyone else remember the LEO line of mainframes?) and a promising software biz, of which only bits remain. (The gaming industry is thriving, but the rest mostly ended up being absorbed by US multinationals. I could speculate that having a common language, in combination with a series of governments who believed in leaving things to the Invisible Hand, and a business culture dominated by accountants, were at the root of the problem ...)
Coemergence of government competence in invention and supporting application seems almost as rare as companies delivering on a second blockbuster idea.
Which kind of speaks to the prevalence of luck over design in favorable outcomes at any macro level. Too many small, randomly whirling bits, and they all need to line up just so.
Which kind of speaks to the prevalence of luck over design in favorable outcomes at any macro level. Too many small, randomly whirling bits, and they all need to line up just so.
TSR2, Hawker P.1154 (Supersonic Harrier)
Royal Aircraft Establishment - merged and amalgamated a dozen times to become DERA - privatised. Now QinetiQ.
Most of the decent research sites are gone now.
GEC/Plessey
I'm going to stop now, it'll get depressing.
Royal Aircraft Establishment - merged and amalgamated a dozen times to become DERA - privatised. Now QinetiQ.
Most of the decent research sites are gone now.
GEC/Plessey
I'm going to stop now, it'll get depressing.
Dyson were doing quite well till Mr Dyson went pro brexit and moved the company to Sinagpore.
DeHavilland didn’t do that badly - the last Nimrod (based on the Comet) was retired in 2011. The name disappeared because, like a whole host of British aviation manufacturers (English Electric, Folland, Vickers, Supermarine, Hawker-Siddeley, Bristol, Hunting, Avro), they successively merged together until all that was left was a name you probably are familiar with - BAE Systems.
Except that, unlike de Havailland, Boeing is too big to fail. There's just no way the US will allow it to keel over, and leave Airbus the sole supplier. And as with the banks, you can bet that this implicit guarantee gets figured into what decisions people take.
markdown(3)
Sounds like a running joke at my work. Any hardware problem or design failure: you can fix it in software!
Do you work at Boeing? ;}
Bandaid fix if I've ever heard of one.
It's proper. This is exactly how you do things with life-critical embedded software. You detect when things are crazy, which of course shouldn't happen but nevertheless does, and you react in a way that is hopefully not disastrous.
The alternative is to ignore the problem.
Sensors go bad. Actuators go bad. Voting hardware goes bad. Message routing goes bad. Even the RAM sometimes goes bad.
When the hardware is failing ("specific streams of erroneous flight data") there isn't going to be a reliable solution. You can't even fully enumerate all the possible failures. How could you possibly guess that a message routing chip now flips bit 7 in every angle-of-attack measurement?
Boeing's answer is standard: when the values look crazy, stop doing stuff that might make the situation worse.
The alternative is to ignore the problem.
Sensors go bad. Actuators go bad. Voting hardware goes bad. Message routing goes bad. Even the RAM sometimes goes bad.
When the hardware is failing ("specific streams of erroneous flight data") there isn't going to be a reliable solution. You can't even fully enumerate all the possible failures. How could you possibly guess that a message routing chip now flips bit 7 in every angle-of-attack measurement?
Boeing's answer is standard: when the values look crazy, stop doing stuff that might make the situation worse.
The way I read it they are not detecting malfunctioning sensors, but essentially a computer fault caused by it's inability to deal with erroneous input. Rather than fixing the computer, they are essentially wrapping a great big try/catch around it....good God.
> are not detecting malfunctioning sensors
Because most of the time you can't. There is no reliable way of knowing when the sensor is lying to you.
You might try to reduce the probability of it happening and try to make it warn you (for example, redundancy, etc), but in aviation, even things that have a probability of failure of 10e-6 or even lower must be considered. 0% chance of reading an erroneous value doesn't exist.
The computer can deal with suspected erroneous input: it can ignore it. There. That's the standard way.
Because most of the time you can't. There is no reliable way of knowing when the sensor is lying to you.
You might try to reduce the probability of it happening and try to make it warn you (for example, redundancy, etc), but in aviation, even things that have a probability of failure of 10e-6 or even lower must be considered. 0% chance of reading an erroneous value doesn't exist.
The computer can deal with suspected erroneous input: it can ignore it. There. That's the standard way.
But what's erroneous input? A bit is flipped? A .1V increase over 0 makes it a logical 1, when it should be .3V? I mean, this is the domain of computer/electrical engineers in the first place, to prevent these kind of problems. This is what happens when you outsource your critical components to 3rd parties.
That interpretation seems like a stretch to me.
I suspect your take is correct but I understand the alarm from the previous poster as the way it's worded makes it sound like they'll look just for condition X, Y and Z and _then_ take action versus your explanation which looks for a lot of incoming, bad data and then handles it.
This might just be a bad case of people talking past one another.
This might just be a bad case of people talking past one another.
what happens when things are actually crazy and it causes a crash itself?
My jaw literally dropped. It actually took me a moment to realize I was sitting gape-mouthed reading that. We're talking airplanes here, not some LAMP stack.
If this is what we can expect going forward, the Jet Age isn't going to last another 50 years. This kind of software ought to be developed using a tool like TLA+.
If this is what we can expect going forward, the Jet Age isn't going to last another 50 years. This kind of software ought to be developed using a tool like TLA+.
We effectively have a front row seat to the sausage making of software development, its not pleasant.
That describes the whole 50+ years old 737 to it’s core.
It is.
But don't worry, when it is found out that it is a mess, we have a plan.
We can add a software patch to correct the bad software output. Not fix the bad software. But add some more to correct the output.
Genius!
But don't worry, when it is found out that it is a mess, we have a plan.
We can add a software patch to correct the bad software output. Not fix the bad software. But add some more to correct the output.
Genius!
its industry standard
https://en.wikipedia.org/wiki/Application_firewall
https://en.wikipedia.org/wiki/Web_application_firewall
To be fair, any hardware running a WAF in Prod these days is going to have more compute power than a 737.
I'm surprised nobody is discussing this paragraph from the article:
> The failure scenario was known previously and had been assessed in a safety analysis when the plane was certified before entering service in 2017. At that time, Boeing concluded that pilots could overcome the nose-down movement by performing a procedure to shut off the motor driving the stabilizer movement.
This isn't a new fault. Boeing certified this as safe along with MCAS back when the aircraft first flew in 2017, using the same justifications ("pilots can overcome it").
They weren't going to fix it this time either, except test pilots ran across it in late-stage simulations monitored by the FAA and found it wasn't as easy to overcome as Boeing had been asserting in their cost-safety analysis (just like MCAS) and now the FAA are requiring a fix.
Has Boeing learned anything from MCAS? The company has cultural problems vis-à-vis safety. I'm just glad the FAA are doing their job this time around.
> The failure scenario was known previously and had been assessed in a safety analysis when the plane was certified before entering service in 2017. At that time, Boeing concluded that pilots could overcome the nose-down movement by performing a procedure to shut off the motor driving the stabilizer movement.
This isn't a new fault. Boeing certified this as safe along with MCAS back when the aircraft first flew in 2017, using the same justifications ("pilots can overcome it").
They weren't going to fix it this time either, except test pilots ran across it in late-stage simulations monitored by the FAA and found it wasn't as easy to overcome as Boeing had been asserting in their cost-safety analysis (just like MCAS) and now the FAA are requiring a fix.
Has Boeing learned anything from MCAS? The company has cultural problems vis-à-vis safety. I'm just glad the FAA are doing their job this time around.
The FAA is only doing their job this time around because the rest of the entire planet, literally, was grounding the aircraft while the FAA baselessly stated the aircraft was safe. This is all stuff that should have been investigated after the first crash, not the second.
Has Boeing learned anything? Of course not. There are no executives facing jail time, or even fines! They're shameless frauds that outsourced their critical systems to the lowest bidder with predictable results.
Has Boeing learned anything? Of course not. There are no executives facing jail time, or even fines! They're shameless frauds that outsourced their critical systems to the lowest bidder with predictable results.
I’ve only recently started following the air industry more closely, but isn’t it ironic that Boeing, the one on the “direct control” side vs computerized systems from AirBus, is the one now suffering from software and automation failures?
I wonder if it might be a consequence of being a late adopter, and rushing the development of systems they never had before.
I wonder if it might be a consequence of being a late adopter, and rushing the development of systems they never had before.
The only reason Boeing is in the “direct control” business is because their best seller is from the 1960s and they don’t dare touch it.
Their newer wide bodies 777 and 787 are fly-by-wire and in the latter case even fully electric to a fault.
I really can't help to think that the design and production of this plane was rushed.
They used a one sensor input to MCAS because the FAA wouldn't certify a two sensor input without requiring a level D certification which meant sim training for pilots.
This latest revelation in the article only strengthens my belief it was rushed and possibly to the point that it was irresponsible.
They used a one sensor input to MCAS because the FAA wouldn't certify a two sensor input without requiring a level D certification which meant sim training for pilots.
This latest revelation in the article only strengthens my belief it was rushed and possibly to the point that it was irresponsible.
That's seriously overdiplomatic. The plane was rushed and it was constructed very badly. It was absolutely irresponsible to bring it into service, it's not a question anymore. It's a plane that crashes itself automatically into the ground and that killed over 300 people.
Agreed. However, the worst part is the FAA. Specifically, what baffles me is the massive hand-wave they did on this considering it is one of their mandates to check flight-worthiness of Airplanes/Airlines.
I hope all the other agencies worldwide will double check all the prior certifications issued by the FAA, because all their credibility is gone. Especially considering the US was the one the last countries to ground the 737 MAX.
I wonder what happened there. Bribes? Favors? Lobbying and ties? All of the above?
I hope all the other agencies worldwide will double check all the prior certifications issued by the FAA, because all their credibility is gone. Especially considering the US was the one the last countries to ground the 737 MAX.
I wonder what happened there. Bribes? Favors? Lobbying and ties? All of the above?
Regulatory capture. After their careers at the FAA they will go work at Boeing.
Do you think regulatory capture actually exists? If so, why is the pharmaceutical industry immune?
why do you think pharma is immune, after fiascos like Vioxx, Accutane, opioids, the fact that much regulatory machinery is in place to create drug monopolies?
Before upgrading to the latest Ubuntu, I like to wait for a few months until my friends confirm that it's for the better. I guess I would do the same for the MAX.
I can't imagine what it must be like to be one of the first pilots to fly the MAX once the grounding is lifted. Can pilots refuse to fly a specific plane "for a few months, 'cause I have kids waiting for me home"?
I can't imagine what it must be like to be one of the first pilots to fly the MAX once the grounding is lifted. Can pilots refuse to fly a specific plane "for a few months, 'cause I have kids waiting for me home"?
There were almost 400 delivered by the time it was grounded, some of them in service since 2017. Yes there were two high profile failures and this failure rate is extremely out of the norm by the very stringent standards of aviation — but these are not sudden-unavoidable-death machines that gets sensationalized.
That's exactly what they are. How many planes have malfunctioned in the same way and haven't nose-dived into the ground? So far, 0 planes that have had this issue haven't been lost (first plane had the issue, was recovered using non-standard procedures, then same plane killed everyone, even after sensor was replaced!)
You appear to be making a straw man argument that the planes are safe by claiming to refute the 'sudden-unavoidable-death machines' argument which no one is making.
The probability of a total loss of an aircraft is basically proportional to either its flight hours or number of takeoffs and landings (which are (handwave) correlated (unless you are Aloha Airlines)). A fleet of thousands of planes (especially a 'mature' design), flying thousands of hours per year, in service for decades should have very few losses due to fundamental design issues. It would be nice to say it should be <1, but the reality is the stars sometimes align and several weaknesses conspire to make a fatal event. With MCAS, the 737 Max 8 was already at a rate of 2 events in less than a year with a fleet that was a tiny fraction (<6%) of its ultimate fleet size.
Assuming nothing else changed, at full fleet size, that would be over 30 losses a year, or well over 1000 losses over the fleet's lifetime. They misengineered the system by at least 3 orders of magnitude. This is, at the very least, profound professional and managerial negligence. I'm not not yet convinced it's not also criminal. At the very least, all management who touched this subsystem or were part of the no-new-training-at-any-cost push should have already been shown the door up to and including the CEO.
And yes, things _are_ being done and they are adding 'nines' to the reliability as quickly as they can, but again, they are starting from a subsystem that kills everybody on board every 800,000 flight hours or so.
They appear to be dishonest with the public (and perhaps themselves) about how much risk is eliminated with each change and, similarly, appear to not be factoring in that some of these changes introduce other risks and failure modes.
What we have here is a giant fucking mess and a management organization that fosters no confidence in its ability to navigate itself out of it. In the end, I do expect Boeing to somehow glue-gun 3 more 9's of reliability onto this airframe and stop bleeding valuation, but it won't be pretty and it probably won't be fast.
The probability of a total loss of an aircraft is basically proportional to either its flight hours or number of takeoffs and landings (which are (handwave) correlated (unless you are Aloha Airlines)). A fleet of thousands of planes (especially a 'mature' design), flying thousands of hours per year, in service for decades should have very few losses due to fundamental design issues. It would be nice to say it should be <1, but the reality is the stars sometimes align and several weaknesses conspire to make a fatal event. With MCAS, the 737 Max 8 was already at a rate of 2 events in less than a year with a fleet that was a tiny fraction (<6%) of its ultimate fleet size.
Assuming nothing else changed, at full fleet size, that would be over 30 losses a year, or well over 1000 losses over the fleet's lifetime. They misengineered the system by at least 3 orders of magnitude. This is, at the very least, profound professional and managerial negligence. I'm not not yet convinced it's not also criminal. At the very least, all management who touched this subsystem or were part of the no-new-training-at-any-cost push should have already been shown the door up to and including the CEO.
And yes, things _are_ being done and they are adding 'nines' to the reliability as quickly as they can, but again, they are starting from a subsystem that kills everybody on board every 800,000 flight hours or so.
They appear to be dishonest with the public (and perhaps themselves) about how much risk is eliminated with each change and, similarly, appear to not be factoring in that some of these changes introduce other risks and failure modes.
What we have here is a giant fucking mess and a management organization that fosters no confidence in its ability to navigate itself out of it. In the end, I do expect Boeing to somehow glue-gun 3 more 9's of reliability onto this airframe and stop bleeding valuation, but it won't be pretty and it probably won't be fast.
I was just reading an article on New York Times that describes the process since 2005 whereby safety oversight responsibilities over Boeing were gradually moved in house — until by 2018 Boeing was self-certifying for 96% of safety testing.
If they are doing 96% of the safety validation work — then they should bear 96% of the responsibility for safety process failures ... in this case, given the break down of safety validation capacity and what I’m increasingly viewing as essentially a coverup after the first and second crashes — 96% of the responsibility is and probably should be - enough to bring Boeing down ...
I’m beginning to think that might be the only way to ensure responsibility for this failure gets allocated in a sufficiently accurate way to ensure this scenario isn’t likely to happen again ... allowing the faa to have the responsibility to bear the weight of this process breakdown would basically just allow Boeing to shift responsibility for this outcome off themselves and onto an organization which it seems has been deliberately engineered (by Boeing) to not have the capacity to perform effective oversight ...
If they are doing 96% of the safety validation work — then they should bear 96% of the responsibility for safety process failures ... in this case, given the break down of safety validation capacity and what I’m increasingly viewing as essentially a coverup after the first and second crashes — 96% of the responsibility is and probably should be - enough to bring Boeing down ...
I’m beginning to think that might be the only way to ensure responsibility for this failure gets allocated in a sufficiently accurate way to ensure this scenario isn’t likely to happen again ... allowing the faa to have the responsibility to bear the weight of this process breakdown would basically just allow Boeing to shift responsibility for this outcome off themselves and onto an organization which it seems has been deliberately engineered (by Boeing) to not have the capacity to perform effective oversight ...
Imagine if your bugfix process was broadcast to the world, every new IN added to spirateam et al, every new crash condition found by test/QA, every time the software failed in test, it got printed by news organizations the world over.
This is world the 737MAX software people are living in, for this alone, they have my utmost respect.
This is world the 737MAX software people are living in, for this alone, they have my utmost respect.
Imagine a world where a certification agency tasked to examine a plane for faults before certifying it so as to assure people aren’t killed needlessly, would have actually done their job instead of outsourcing it to the very same manufacturer whose incentive it is to certify itself ASAP.
Novel concept, I know.
But where would we be without corruption...
But where would we be without corruption...
Imagine if your software had already killed a few hundred people.
Boeing developers have been put in an untenable position and should consider walking out until management does the right thing and kills the plane.
Boeing developers have been put in an untenable position and should consider walking out until management does the right thing and kills the plane.
The plane has no fundamental mechanical flaw, it has a software bug.
The only flaw the plane had was being treated as 'just another 737'
The only flaw the plane had was being treated as 'just another 737'
It is an old air frame, and one hard mechanical fault that is impossible to solve without a completely new air frame is the ground clearance. The low ground clearance prevents new and larger engines to be used without completely ruining the aerodynamics of the 737 airframe itself.
It has a number of bugs, resulting from the bad call that was the decision to hack the original 737 for commercial reasons (whether to appease their customers or just to save money, it doesn't matter -- it was a bad call, made for the wrong reasons, and hundreds of people are now dead because of it).
> The only flaw the plane had was being treated as 'just another 737'
Passing as "just another 737" was the whole point of this plane. Which is why it should be scrapped. Boeing and its customers will scream? Tough - next time, build a plane in the proper way, so you don't kill people. If that means making a bit less money, well, money comes and goes.
> The only flaw the plane had was being treated as 'just another 737'
Passing as "just another 737" was the whole point of this plane. Which is why it should be scrapped. Boeing and its customers will scream? Tough - next time, build a plane in the proper way, so you don't kill people. If that means making a bit less money, well, money comes and goes.
It has plenty of uncertifiable components grandfathered in for 50 years, topped with ‘still within range of grandfathering’ pieces of bandaid software on top.
It’s fundamental because the whole fundament this rests one isn’t stable and all the adding instability add-ons do not make sense.
It’s fundamental because the whole fundament this rests one isn’t stable and all the adding instability add-ons do not make sense.
Since you seem to be an expert in airplanes and complex systems, which components are not certifiable?
the 737MAX could have been certified as a new airplane, with a new type certificate, and a new pilot type rating to go with it - but that would have removed much (or all) of its economic justification to exist.
the 737MAX could have been certified as a new airplane, with a new type certificate, and a new pilot type rating to go with it - but that would have removed much (or all) of its economic justification to exist.
It would be a not very good airplane. It has only 2 selling points: its type certificate, and its 737 footprint at the terminal.
Any new airplane would be a bit higher, it brings some downsides (ground operations people really like the easy access), but having bigger engines under the wings is such an advantage that it overrides all other considerations.
Any new airplane would be a bit higher, it brings some downsides (ground operations people really like the easy access), but having bigger engines under the wings is such an advantage that it overrides all other considerations.
Thats kinda my point, the shared type certificate is most of the justification for the airplane to exist, Boeing would rather build a whole new small aircraft, people who fly on them would rather fly on a whole new aircraft - airlines however strongly prefer something that costs as little as possible to put into service - and has commonality with existing fleets, which means not a whole new aircraft.
I bet if you offered Delta some brand new MD-95's they'd take up on them, as they have basically every MD-95 produced.
I bet if you offered Delta some brand new MD-95's they'd take up on them, as they have basically every MD-95 produced.
Yeah, they should have been using Javascript and React by now /s
Yes this comes with "we do everything in software".
I work in automation an I guess most of our plc code is much smaller and simpler than just one website.
Software devs still live in this "you can use this software as-is" world. This thinking is just not compatible with, we use software for everything.
Even I liked PLC programming as a younger guy and a kind of I still like to do it .. you can do cool things on the machines .. the way it goes at moment does worry me quiet a bit.
And if you would work in a different job, for ex. design a building or bridges, since ever was this "if you fail, you are in the news".
So please you just software guys, grow up.
I work in automation an I guess most of our plc code is much smaller and simpler than just one website.
Software devs still live in this "you can use this software as-is" world. This thinking is just not compatible with, we use software for everything.
Even I liked PLC programming as a younger guy and a kind of I still like to do it .. you can do cool things on the machines .. the way it goes at moment does worry me quiet a bit.
And if you would work in a different job, for ex. design a building or bridges, since ever was this "if you fail, you are in the news".
So please you just software guys, grow up.
imagine if your bug has the potential to kill 300 people.
To be fair, if another 150 people die that will be the end of the 737 program - and probably Boeing. 300 more deaths is highly unlikely.
300 people died before Boeing started acknowledging that maybe this wasn't entirely pilot's fault...
I think you underestimate Boeing size, wealth, and importance to the federal government.
This. Even is Boeing had to drop out of the airplane game _entirely_, they have a ton of other lines of business.
If another 737 Max crashed then that would be the end of the Max, IMO, but I doubt older 737s would be dropped and Boeing would still be around easily enough.
If another 737 Max crashed then that would be the end of the Max, IMO, but I doubt older 737s would be dropped and Boeing would still be around easily enough.
This.
- Boeing is certainly well financially engineered in many financial vehicles,
- It’s alone the #1 net exporter of USA,
- USA needs Boeing, whether it exports or not. Boeing also funds candidates, just mentioning.
- Existing conpanies need to maintain their existing planes, and they need Boeing’s consultancy for that,
- I’d say even Airbus needs their competitor, or they would fall in a dangerous monopolistic position.
As much as one would enjoy the spectacle of a corrupted eating the dust, it is probably too stable to happen. But ever heard of De Havailland? That’s probably to do with their 4 crashes in a row, due to its airframe shredding of metal fatigue and material constraints being concentrated on the edges of their square windows. Ever flew in a DC-10 or MD-11? That’s probably to do with their 4 crashes in a raw, for a bad doorlock. Just saying.
- Boeing is certainly well financially engineered in many financial vehicles,
- It’s alone the #1 net exporter of USA,
- USA needs Boeing, whether it exports or not. Boeing also funds candidates, just mentioning.
- Existing conpanies need to maintain their existing planes, and they need Boeing’s consultancy for that,
- I’d say even Airbus needs their competitor, or they would fall in a dangerous monopolistic position.
As much as one would enjoy the spectacle of a corrupted eating the dust, it is probably too stable to happen. But ever heard of De Havailland? That’s probably to do with their 4 crashes in a row, due to its airframe shredding of metal fatigue and material constraints being concentrated on the edges of their square windows. Ever flew in a DC-10 or MD-11? That’s probably to do with their 4 crashes in a raw, for a bad doorlock. Just saying.
But in the days of de Havailland & McD-D, there were maybe half a dozen companies which made airliners. That's a much more Adam-Smith world than when there are exactly two.
So one more plane crash is possible but two are not?? Huh?
Yes. After one more plane crash, it will probably become uneconomical for carriers to continue operating the 737.
How does a crash on the 737MAX effect operations on 737NG aircraft, or older ones - what suddenly makes it uneconomical?
Because people are made aware that Boeing wasn’t serious with engineering since 2012. And the FAA paid Boeing consultants to inspect Boeing, which by all layman’s definition fall under the terms of “corrupted”.
Who wouldn’t pay attention to the little mention in Google Flights with the brand of the plane?
Who wouldn’t pay attention to the little mention in Google Flights with the brand of the plane?
>And the FAA paid Boeing consultants to inspect Boeing, which by all layman’s definition fall under the terms of “corrupted”.
In my experience this is pretty normal for specialized stuff like this, it has nothing to do with corruption.
In my experience this is pretty normal for specialized stuff like this, it has nothing to do with corruption.
It's this way in my industry as well, it's really hard to find people who have not worked for one of the big two vendors in my field.
I think you grossly overestimate the general public, and their ability to pay attention, or care. The 737MAX isnt the first aircraft to have an issue like this - and they kept flying them.
People's refusal to fly in anything with "737" in its name.
The whitewash is already starting.
The 737 Max name is now being replaced with the 737-8200 term (1). I assume they are hoping that 737-8200 is not tarnished like 737 Max is.
1 - https://www.independent.co.uk/travel/news-and-advice/ryanair...
The 737 Max name is now being replaced with the 737-8200 term (1). I assume they are hoping that 737-8200 is not tarnished like 737 Max is.
1 - https://www.independent.co.uk/travel/news-and-advice/ryanair...
> Imagine if your bugfix process was broadcast to the world
Have you heard of the site called github?
Have you heard of the site called github?
Heard of a private GitHub?
Yes, for those who need to hide their code from public scrunity.
...or you work on closed source software in your company's private GitHub.
Closed source software should not be used in safety critical systems that affect the general public.
Most software engineers aren’t qualified to have an opinion about the quality of proprietary safety critical software systems that also happen to run against proprietary hardware.
So unless you’ve found an economical way to develop an open source jetliner... honestly, what would be the point of open sourcing safety critical systems to the general public?
I’m not at all saying it should be closed off from external safety inspectors or regulators or anything like that, but I’m not really seeing the value of opening the software up to the general public.
So unless you’ve found an economical way to develop an open source jetliner... honestly, what would be the point of open sourcing safety critical systems to the general public?
I’m not at all saying it should be closed off from external safety inspectors or regulators or anything like that, but I’m not really seeing the value of opening the software up to the general public.
You'd be surprised what fraction of a safety critical codebase is just mundane code that anybody can understand with the same amount of effort it takes for "normal" software. The real problem is that most outsiders don't understand the requirements very well (unless you make them public too!), so they can likely only find "simple" code quality issues.
That’s essentially my point. Just finding simple code quality issues isn’t going to cut it, especially when you need hundreds of thousands of dollars of hardware to even test the code against.
It doesn't hurt either.
Open sourcing my MRI scans for the general public to read "doesn't hurt" either. Doesn't mean it's a worthwhile/valuable thing to do.
And I also challenge you on "it doesn't hurt". Consider military adversaries developing targeted attacks against critical infrastructure because it's open sourced.
And I also challenge you on "it doesn't hurt". Consider military adversaries developing targeted attacks against critical infrastructure because it's open sourced.
You should assume that military adversaries already have access to the source code. Spies are a thing.
Safety critical is by design simple, it is a requirement by the norms to ensure low complexity :)
Does being qualified grant you access to the code right now? Who decides if you're qualified and if permission is granted?
I assume that currently there's several gatekeepers involved who can shut most independent investigators out.
I assume that currently there's several gatekeepers involved who can shut most independent investigators out.
I still really just don’t get where you’re going with this. You’d need hundreds of thousands of dollars worth of hardware to even test these kinds of software systems.
Simply reading the code isn’t going to help you find critical safety flaws.
Simply reading the code isn’t going to help you find critical safety flaws.
I picture a labyrinth of outsourced spaghetti, with no real ownership anywhere.
Would be very happy to be corrected here.
Would be very happy to be corrected here.
Most probably wrong. Code and requirement tracability is mandatory when we do safety critical system. If you have an audit, they can pick a source file or even line of code and ask you to find why it was added and the associated requirement. They can also ask you to show the test report where it was tested
Can anyone in the know explain why, if ground clearance due to the new engines is the issue, why Boeing cannot just add a few inches to the wheel stems?
This would raise the whole aircraft off the ground. Surely that wouldn't effect aerodynamics too much as the wheels are up most of the time.
This would raise the whole aircraft off the ground. Surely that wouldn't effect aerodynamics too much as the wheels are up most of the time.
Because you then need longer and heavier landing gear, which you need to fit somewhere in the plane as they retract. This is a ripple effect where you probably need to redesign the complete airframe, from fuselage to the wings.
They actually cleverly modified the landing gear to gain a bit of heigth, but it was not enough: https://youtu.be/F4IGl4OizM4
They actually cleverly modified the landing gear to gain a bit of heigth, but it was not enough: https://youtu.be/F4IGl4OizM4
One thing I learned many years ago, you can solve infrastructure problems with software. If prior jets have multiple sensors and the new jet has one sensor, then software isn't going to help in this situation. Gyros worked back in 1911. why not use one to back up the failed sensor? seems like a simple fix that could be mounted anywhere on the plane.
> Gyros worked back in 1911. why not use one to back up the failed sensor?
An angle of attack sensor measures something a gyroscope cannot measure: the angle of the plane relative to the wind. The amount of lift produced by the wings depends on this angle.
An angle of attack sensor measures something a gyroscope cannot measure: the angle of the plane relative to the wind. The amount of lift produced by the wings depends on this angle.
It appears the 737 Max is a total failure. Pushed through by Boing to outpace Airbus, from physical design, attitude sensors, re-certification bypass imperative, and flight control software, finally killing nearly 600 people before being ground. In short I’m never flying in one!
Move fast break things kind of doesn't work well with planes.
Also too big to fail (fall) doesn't apply here.
Also too big to fail (fall) doesn't apply here.
Au contraire, Boeing is most definitely too big too fail and will be bailed out somehow.
Laying off senior Boeing software engineers and outsourcing their work to India for $9/hour is really paying off for Boeing.
Are there any public databases that would show plane model failure statistics so that I could choose how to fly?
Sounds to some degree like a usability issue. Echoes of Three Mile Island.
Oh man. That sounds like a mess.