Flying the Unflyable Plane: The near crash of Air Astana flight 1388(admiralcloudberg.medium.com)
admiralcloudberg.medium.com
Flying the Unflyable Plane: The near crash of Air Astana flight 1388
https://admiralcloudberg.medium.com/flying-the-unflyable-the-near-crash-of-air-astana-flight-1388-88878e2eb3c4
51 comments
Mentor Pilot's videos are top notch. Watching this one made me dizzy I can't imagine debugging a system while enduring 4+ Gs.
I'd imagine it would be something akin trying to debug servers when riding rollercoaster
While your life depends on it
The obvious solution — using connectors that cannot be fastened to the wrong cable — reminds me of Bob Hoover, an absolutely stunning test and display pilot, who had a forced landing after his fuel tank was filled with jet fuel rather than standard avgas. He devised a bell-shaped nozzle that is now universally used to dispense jet fuel, and cannot be inserted into a standard fuel filling neck.
In this case, though, better design, better tooling, and better training are probably the best fix; there are too many variations in stick-to-aileron connection for a single standard to be accepted.
In this case, though, better design, better tooling, and better training are probably the best fix; there are too many variations in stick-to-aileron connection for a single standard to be accepted.
Even though the author is critical of the GPIAAF's report, it's amazing to see how thoroughly incidents like this one are investigated - With a real focus on process improvement.
Frustrating that the obvious mechanical fix for this (have different shaped connectors for each aerilon cable) wasn't adopted by Embraer, but still a great reminder of how we can do incident handling better in software dev
I find Embraer's rationalization for this (and, presumably, its acceptance by the relevant regulatory agencies) to be particularly galling. Even if it was true at the time that no procedure called for both cables to be disconnected at the same time, did they have any equivalent proof that no such procedure might be introduced later? Did they have any equivalent proof that no-one would misunderstand the instructions, or that it could go undetected if that happened? Could they be sure that this constraint, or its relevance, would not be overlooked at some point in the future? Well, we know they did not. Nobody addressed the question, "what could possibly go wrong?" with sufficient rigor.
Investigate for cause, not blame
Even something as simple as coloring them like RCA connectors would've worked. It's insane how lightly this was treated in designing it.
When I first started flying GA planes I did the control check and assumed that seeing the ailerons move was sufficient to check that item off. It never occurred to me that you _could_ reassemble the plane in such a way that the ailerons would be inverted. Some older pilot mentioned his friend wrecking a Mooney on takeoff that way (luckily walking out of the wreck) and it scared the hell out of me.
Hats off to the Air Astana guys. That's some solid debugging work and very impressive ability to keep calm and focused under pressure.
Hats off to the Air Astana guys. That's some solid debugging work and very impressive ability to keep calm and focused under pressure.
In my private pilot training, I was taught to move the wheel both ways and visually check that the ailerons moved in the expected directions. Even if the connections hadn't been changed recently, they could still jam or a cable could have broken.
They were fortunate enough that cables on both sides were inverted; therefore the ailerons "merely" worked in reverse. If they had reversed the cables on one side and not the other, that would have been even tougher to sort out and maybe impossible.
Yep, that would have been a second elevator. There would be completely no way to roll the plane in that case, though I wonder if the plane has a natural tendency to fly level anyway.
Not so much an elevator, given their position on the trailing edge of the wing, but more like a flap with limited deflection both up and down. Like flaps in general, this would have some effect on pitch, but I doubt it would be enough to make pitch control impossible (or nearly so).
My guess is that in this scenario, small angles of yoke deflection would have little effect, and the pilot's almost instinctual response would be to increase that deflection, quickly reaching the point where the spoilers would kick in, giving the sort of roll response the pilot was seeking (together with some unexpected yaw that I suspect also happened in the actual case). It would be a problem, but not quite so counterintuitive and baffling.
My guess is that in this scenario, small angles of yoke deflection would have little effect, and the pilot's almost instinctual response would be to increase that deflection, quickly reaching the point where the spoilers would kick in, giving the sort of roll response the pilot was seeking (together with some unexpected yaw that I suspect also happened in the actual case). It would be a problem, but not quite so counterintuitive and baffling.
In that case, the spoilers would still deploy when large control inputs were given, so roll control would still be present albeit with a very wide "dead zone".
What reminds me of computer systems is how the spoilers partly compensated for the opposite role, the computer modes (direct vs. assisted) lead to different behaviors, the test display (green == good?) was mostly self-referential and did not indicate what they thought it should, etc.
In short, the mental model required to understand the plane and its indicia of state was way too complex. These guys literally had to read technical manuals and follow (run-book) procedures step-by-step while the plane was cavorting.
Dynamic stability is the tendency of a system to restore stability, given external forces. There should be a term for dynamic-breaking stability, the ability to restore stability when broken, while also maintaining dynamic and static stability.
This is probably the only real benefit of devops: designing systems on the assumption people will need to intervene to fix them during moments of highest demand.
In short, the mental model required to understand the plane and its indicia of state was way too complex. These guys literally had to read technical manuals and follow (run-book) procedures step-by-step while the plane was cavorting.
Dynamic stability is the tendency of a system to restore stability, given external forces. There should be a term for dynamic-breaking stability, the ability to restore stability when broken, while also maintaining dynamic and static stability.
This is probably the only real benefit of devops: designing systems on the assumption people will need to intervene to fix them during moments of highest demand.
I watched a lot of accident videos and that kinda is the common trend. Many of them were caused not only by actual failure but by pilots not understanding what the plane was telling them, or plane sending mixed signals (whether actually or by pilot's misunderstandings)
I remember one accident where part of the cause was that both pilots were trying to yank the plane in different direction, and the system just averaged out both inputs and as it was fly by wire and there was no feedback and "too much to do", pilots didn't notice. That's the reason why many planes have pilot's controls connected with eachother - so one can feel what the other one is trying to do.
> This is probably the only real benefit of devops: designing systems on the assumption people will need to intervene to fix them during moments of highest demand.
I have noticed the interesting trend when it comes to automating and "cattlefication" of everything - the less something breaks the less everyone involved understands it. Because it's automated and because nobody needs to even read anything to deploy it (just add a line in Puppet manifest and you're done) when it does break not even the author has a clue why. Documentation helps a little bit but there is a gulf between "describing what it is and how to use it" and "knowing enough to find where the problem is.
I remember one accident where part of the cause was that both pilots were trying to yank the plane in different direction, and the system just averaged out both inputs and as it was fly by wire and there was no feedback and "too much to do", pilots didn't notice. That's the reason why many planes have pilot's controls connected with eachother - so one can feel what the other one is trying to do.
> This is probably the only real benefit of devops: designing systems on the assumption people will need to intervene to fix them during moments of highest demand.
I have noticed the interesting trend when it comes to automating and "cattlefication" of everything - the less something breaks the less everyone involved understands it. Because it's automated and because nobody needs to even read anything to deploy it (just add a line in Puppet manifest and you're done) when it does break not even the author has a clue why. Documentation helps a little bit but there is a gulf between "describing what it is and how to use it" and "knowing enough to find where the problem is.
> Documentation helps a little bit but there is a gulf between "describing what it is and how to use it" and "knowing enough to find where the problem is.
That is one of my pet peeves with modern development. A lot of things are way too easy to get started with, while very few people understands what is going on. And then when things doesn't work so well, there are even more tools and frameworks that can be deployed without understanding, but that in some ways mitigate the problems. But now it's even more complex, and suddenly there's a team of 20 people maintaining an application that doesn't do much more than moving some files between servers, reading them and saving records in a database. Sorry, not a database, five different types of databases that no one in any of the teams have any deeper understanding of.
That is one of my pet peeves with modern development. A lot of things are way too easy to get started with, while very few people understands what is going on. And then when things doesn't work so well, there are even more tools and frameworks that can be deployed without understanding, but that in some ways mitigate the problems. But now it's even more complex, and suddenly there's a team of 20 people maintaining an application that doesn't do much more than moving some files between servers, reading them and saving records in a database. Sorry, not a database, five different types of databases that no one in any of the teams have any deeper understanding of.
I guess that goes to show that no matter the warnings and training, if something can be put in reverse, it eventually will be. Reminds me a bit of Proton crash where accelerometer (which even had arrow marking "up") was mounted in reverse and the rocket wanted to accelerate in direction of the ground.
I bet designers of software were like "there is no need to check whether moving it in direction actually moves it in that direction, we have thousand procedures to make sure hardware is done right". Defense in depth really is the best approach, not just strictly for security
I bet designers of software were like "there is no need to check whether moving it in direction actually moves it in that direction, we have thousand procedures to make sure hardware is done right". Defense in depth really is the best approach, not just strictly for security
The 2013 Proton crash was more similar to the Embraer incident than you might think. Human pilots can sometimes step outside the box and re-learn how to fly but automated guidance systems aren't that flexible.
The Proton crash involved angular velocity sensors -- rotational accelerometers -- installed upside down. It's not that it wanted to fly into the ground, it was that it couldn't fly straight because (from the guidance computer's perspective) it was responding backwards to the behavior commanded by the guidance system. And rockets just don't fly sideways very well.
Watch the direction of thrust from the engines (the Proton, like most rockets, steers by thrust vectoring) in this slowed-down video of the incident:
https://www.youtube.com/watch?v=vqW0LEcTAYg
The Proton crash involved angular velocity sensors -- rotational accelerometers -- installed upside down. It's not that it wanted to fly into the ground, it was that it couldn't fly straight because (from the guidance computer's perspective) it was responding backwards to the behavior commanded by the guidance system. And rockets just don't fly sideways very well.
Watch the direction of thrust from the engines (the Proton, like most rockets, steers by thrust vectoring) in this slowed-down video of the incident:
https://www.youtube.com/watch?v=vqW0LEcTAYg
Better source: http://avherald.com/h?article=4c04438e&opt=4096
Thanks to the mhandley for posting this. I have now read a bunch of these posts (on 'admiralcloudberg') and they are absolutely fascinating.
The ones I find most interesting are those where I remember the media coverage at the time. So often, the post has fascinating details which emerged later. For example:
https://admiralcloudberg.medium.com/days-of-our-discontent-t...
I remember this crash of a Jamaica-bound plane after 9/11, and I remember the press reporting initially that the tail had been repaired at the factory, and then a few months later that the NTSB had issued a directive that pilots shouldn't respond to wake turbulence too aggressively.
However, the cloudberg post has fascinating details about how the particular characteristics of a training scenario for the A300 had led many pilots to conclude that the proper response to wake turbulence was massive inputs to the controls. It would have been easy to conclude the pilot who made them was just incompetent. But in fact, evidence was he was very competent, and that he had learned what the training taught him quite well. The trainers did not understand what the implementation of their simulation was teaching until after the accident was investigated.
Also, the industry did not fully understand what tests for structural integrity of the tail really said about its capacity to withstand inputs at lower speeds until after this crash. The details are just fascinating.
Or this one:
https://admiralcloudberg.medium.com/lost-souls-of-grammatiko...
About a plane where everyone is unconscious and it circles until it runs out of gas. I remember the coverage at the time, which presented it as a mystery. It turns out it wasn't such a mystery, the compression system on the plane was left in an incorrect state by maintenance, and the pilots repeatedly failed to recognize it. The details behind those short statements are fascinating.
Anyway, thanks for posting this.
The ones I find most interesting are those where I remember the media coverage at the time. So often, the post has fascinating details which emerged later. For example:
https://admiralcloudberg.medium.com/days-of-our-discontent-t...
I remember this crash of a Jamaica-bound plane after 9/11, and I remember the press reporting initially that the tail had been repaired at the factory, and then a few months later that the NTSB had issued a directive that pilots shouldn't respond to wake turbulence too aggressively.
However, the cloudberg post has fascinating details about how the particular characteristics of a training scenario for the A300 had led many pilots to conclude that the proper response to wake turbulence was massive inputs to the controls. It would have been easy to conclude the pilot who made them was just incompetent. But in fact, evidence was he was very competent, and that he had learned what the training taught him quite well. The trainers did not understand what the implementation of their simulation was teaching until after the accident was investigated.
Also, the industry did not fully understand what tests for structural integrity of the tail really said about its capacity to withstand inputs at lower speeds until after this crash. The details are just fascinating.
Or this one:
https://admiralcloudberg.medium.com/lost-souls-of-grammatiko...
About a plane where everyone is unconscious and it circles until it runs out of gas. I remember the coverage at the time, which presented it as a mystery. It turns out it wasn't such a mystery, the compression system on the plane was left in an incorrect state by maintenance, and the pilots repeatedly failed to recognize it. The details behind those short statements are fascinating.
Anyway, thanks for posting this.
> Fortunately, however, that’s something only pilots will have to worry about, because the first flight after heavy maintenance is never conducted with passengers on board
whew : ) but yeah, a pretty epic save
whew : ) but yeah, a pretty epic save
It's an extraordinary achievement by the pilots to keep going under the immense pressure of a 2 hour impromptu rollercoaster ride in the clouds with 6 lives on the line.
It seems color coded cables would be a nearly zero cost solution. I wonder why identical steel cables were preferred, and how this wasn't an issue in manufacturing?
I suspect having a color coding material/technique that can apply to random bits of random plane parts, that won't get rubbed off / broken off / obscured by dirt/grease/grime/etc, consistently throughout the plane, is probably harder than it appears at first glance.
Well, one less part on parts list assuming they are same length
> and how this wasn't an issue in manufacturing?
Manufacturing does same exact thing every single time in order and then someone checks it.
But this was something staff don't do often.
> and how this wasn't an issue in manufacturing?
Manufacturing does same exact thing every single time in order and then someone checks it.
But this was something staff don't do often.
What I am curious about whether it would have been possible to deactivate the spoilers, at least in roll-assistance?
From the MH-370 discussions here I had the impression that the pilot can shut-off every system.
I don't think the article was completely clear about this, but I was left with the impression that the solution was to always turn the yoke far enough that the spoilers overrode the ailerons. The alternative - to use reversed controls - is so against one's experience that it may not be feasible even if you know what is going on.
Are there any home-computer flight simulators (or their controllers) that can be hacked to simulate reversed ailerons?
Are there any home-computer flight simulators (or their controllers) that can be hacked to simulate reversed ailerons?
I don't know about simulators, but I can tell you that, out of the ten or so times I reversed the ailerons by mistake on my RC planes, zero landed without crashing.
Yes, it took me ten crashes before checking that the control surfaces move the right way before flight. Now I run a battery of tests, each one inspired by multiple crashes.
Yes, it took me ten crashes before checking that the control surfaces move the right way before flight. Now I run a battery of tests, each one inspired by multiple crashes.
You could probably do it in FlightGear, and modify the aircraft to model reversed ailerons but correct spoilers.
I wonder if you could do it by dicking around with the joystick inputs, giving "reversed" controls over the first little bit, then a dead spot, then correct inputs once you've moved more than a certain amount?
I wonder if you could do it by dicking around with the joystick inputs, giving "reversed" controls over the first little bit, then a dead spot, then correct inputs once you've moved more than a certain amount?
I am mostly interested in the simpler question of whether a pilot can fly an approach with reversed roll controls (i.e., without the counteracting spoiler action), knowing that they are reversed, but without having practiced beforehand.
Oh you could use any flight sim and just mess with the joystick wiring then! Or possibly even just the calibration.
I'm going to go with "pretty much no". I'm prepared to bet that you in your house with no pressure could probably more-or-less do it, because you know already what's going on and you're not going to die and you're not in a real plane with your primary instrument - the Mk1 Human Backside - being tossed around enough to knock it out of calibration.
Have you seen the film "Sully"? If not I won't spoil it for you, but it raises a similar issue.
I'm going to go with "pretty much no". I'm prepared to bet that you in your house with no pressure could probably more-or-less do it, because you know already what's going on and you're not going to die and you're not in a real plane with your primary instrument - the Mk1 Human Backside - being tossed around enough to knock it out of calibration.
Have you seen the film "Sully"? If not I won't spoil it for you, but it raises a similar issue.
I'm doubtful I could succeed even in a simulator, but I'm more interested in whether seasoned professionals could succeed.
I have seen "Sully", but I don't recall where the issue of inverted controls came up, either in the movie or the actual incident.
I have seen "Sully", but I don't recall where the issue of inverted controls came up, either in the movie or the actual incident.
They eventually landed (after two failed attempts) by not touching the ailerons at all and using only the rudder.
I'm not sure why a visual check procedure was never done which would have shown the issue as part of checks after maintenance was carried out.
Not an embraer pilot, but I suspect because (like many larger aircraft) you can't physically see the ailerons from the cockpit – that is why the big jets measure the deflection and (should) show it to you.
Part of me wonders if also as long as you see the ailerons _move_ you assume they've moved in the right direction. There have been similarly tragic rigging accidents in GA aircraft as well – and 38 incidences of aileron mis-connection in gliders between 1974 and 2014 reported in the UK. [1]
[1] https://members.gliding.co.uk/wp-content/uploads/sites/3/201...
Part of me wonders if also as long as you see the ailerons _move_ you assume they've moved in the right direction. There have been similarly tragic rigging accidents in GA aircraft as well – and 38 incidences of aileron mis-connection in gliders between 1974 and 2014 reported in the UK. [1]
[1] https://members.gliding.co.uk/wp-content/uploads/sites/3/201...
as a pilot, I have to really force myself to 'see' that the aileron is moving the correct direction. Not because it's hard, but because my lizard brain wants so badly to skip over the thing that it has seen so many times that it's no longer interesting.
I have not seen a glider where it is physically possible to cross-connect the controls simply through the process of de-rigging and rigging (removing and replacing the wings and stabilizer, which is often done for transport or storage), though I imagine there were examples in the past. I have, however, seen ones where the controls can be left unconnected or even insecurely connected (though such designs now seem deprecated), in which case they may move in the correct direction on the ground, but fail to do so under load (and may flutter). On account of these possibilities, the visual inspection and pull/twist test of the connections, together with positive control checks (where someone holds the surface to provide resistance), as mentioned in the article you link to, should be SOP.
>I have not seen a glider where it is physically possible to cross-connect the controls simply through the process of de-rigging and rigging (removing and replacing the wings and stabilizer, which is often done for transport or storage), though I imagine there were examples in the past
The instances in the past probably caused the newer ones to make that impossible
The instances in the past probably caused the newer ones to make that impossible
Absolutely - in aviation generally, not just gliders, and also for electrical and hydraulic connections. That's why Embraer's design choice and justification for it provokes a 'WTF?'
I would think one of the pilots would stand behind the plane and verify control surfaces via radio, but I guess that's too time consuming or error prone?
Works fine for cars though. Drive through lube shops and states with safety inspections will perform indicator checks this way. If you push the left blinker and the right light starts flashing you've got a problem...
Hopefully by now the cockpit has access to external cameras to visualize what's happening behind them in real time
Works fine for cars though. Drive through lube shops and states with safety inspections will perform indicator checks this way. If you push the left blinker and the right light starts flashing you've got a problem...
Hopefully by now the cockpit has access to external cameras to visualize what's happening behind them in real time
Watch the Mentour Pilot video; there was a display in the cockpit which showed the actual control surface movement. Why didn't they catch it? I'd guess it was because they were just a regular flight crew as opposed to acceptance test pilots, had never ever seen cross-connected controls before, and when they tested the controls they saw "yep, the surfaces moved" and it just didn't register that some of them were in the wrong direction.
The primary issue is that once the ailerons were fully deflected, the control display turned green - even if they were deflected in the wrong direction. It seems that despite this test having been performed multiple times, everyone involved was subconsciously looking for "It goes green" and not "It moves in the right direction".
> I would think one of the pilots would stand behind the plane and verify control surfaces via radio, but I guess that's too time consuming or error prone?
Pointless. Reversing of controls is not possible between flights.
After service it should've been on checklist to validate everything (really, if plane's computer have feedback on position it should immediately alert), but, well, I'd imagine after this it will be mandatory anyway
Pointless. Reversing of controls is not possible between flights.
After service it should've been on checklist to validate everything (really, if plane's computer have feedback on position it should immediately alert), but, well, I'd imagine after this it will be mandatory anyway
No, but hardware breaks and while it may not be cross-connected, you could have the issue where one aileron or the other doesn't move at all.
This is like the unrideable bicycle on steroids...
Massive respect to the pilots who managed to figure it out AND land the plane.
It's truly remarkable that they managed to land that plane, against all odds.
[1]https://youtu.be/5ywaMkMTwWk