Hi there! I am a very experienced embedded systems designer and manager with a proven track record delivering results. I can help define requirements, create an electronic design, write software, and build a conforming prototype.
It does not appear to be "precisely" the procedure, in that sufficient nose up trim was not selected by the yoke manual trim switch prior to pulling the trim cutouts. And then they did not work the mechanical trim wheel hard enough to reset the residual nose down trim.
And, the crew left the power setting at climb throughout almost the entire sequence, so they oversped the airframe.
And, when they (against procedure) (apparently) turned the electric trim system back on, they did not use the yoke manual trim switch to fix the nose down trim problem -- which is a big mystery.
The MCAS design inhibits its FCC trim down output when manual trim is utilized. So, if they had just continuously pushed the nose up manual trim switch on the yoke until they got the trim to neutral, then pulled the cutout (either the first time or second time), the accident could have been avoided.
History shows this is not a good human factors design on the part of Boeing, but the crew does not look good, either, IMO.
> sufficient nose up trim was not selected by the yoke manual trim switch prior to pulling the trim cutouts
The EAD/service bulletin doesn't talk about sufficient nose up trim or caution against performing the cutout with even a slight mistrim.
> And then they did not work the mechanical trim wheel hard enough to reset the residual nose down trim.
It's possible that aerodynamic load (of the stabilizer opposing the elevator) made it physically impossible to manually trim given any mistrim at the time the cutout happened, given the airspeed they had. And, the captain is pulling back on the yoke as hard as he can, so he's unavailable to let go of it and grab a trim wheel instead without immediately losing altitude.
> And, when they (against procedure) (apparently) turned the electric trim system back on
It seems sensible to assume -- and the report states -- that they did this because they found that manual trim was impossible in these circumstances.
You're right that the service bulletin doesn't talk about selecting selecting a sufficient nose up trim because control of the aircraft using manual trim is a continuous action performed in combination with the pitch up command of the control yoke. Just pulling up on the control column and activating momentary manual trim isn't sufficient or intended. I understand why people think the checklist implies that performing a STAB TRIM cutout occurs instantly after the autopilot is disengaged because of the "if runaway continues" language.
The Boeing/Ethiopian ETH-2 bulletin quoted in the above linked AAIB preliminary report on page 33 says "Initially, higher control forces may be needed to overcome any stabilizer nose down trim already applied. Electric stabilizer trim can be used to neutralize control column pitch forces before moving the STAB TRIM CUTOUT switches to CUTOUT ..."
> Electric stabilizer trim can be used to neutralize control column pitch forces before moving the STAB TRIM CUTOUT switches to CUTOUT ..
Which really under explains the situation. In reality the STAB TRIM CUTOUT must be switched to CUTOUT within 5 seconds of using the electronic stabilizer trim, or the MCAS will re-establish the forces.
Wow. This sub-thread is making the images of the grounded MAXs look a whole lot less like an overabundance of caution. If accurate this failure dynamic sounds more like some nasty boss at the end of an escape room game than standard fault diagnostics.
Yeah, I think that's fairly put. The MCAS system was pretty straightforwardly trying to kill them, and the only way to disable it also disabled their method of recovering from what it was doing.
> It's possible that aerodynamic load (of the stabilizer opposing the elevator) made it physically impossible to manually trim given any mistrim at the time the cutout happened, given the airspeed they had.
Its been noted in previous discussions that while it would be difficult, the load can be overcome if both pilots were to operate the manual trim wheel at the same time.
> Its been noted in previous discussions that while it would be difficult, the load can be overcome if both pilots were to operate the manual trim wheel at the same time.
I haven't seen that noted anywhere, and have seen it theorized as impossible without motor assist, given their airspeed and with the elevator opposing. Got a link?
Excessive airloads on the stabilizer may require effort by both pilots to correct the
mis-trim. In extreme cases it may be necessary to aerodynamically relieve the
airloads to allow manual trimming. Accelerate or decelerate towards the in-trim
speed while attempting to trim manually.
To be clear, that's a link that agrees with me that it can be impossible, not a link that says they just needed to work harder.
It's also from a 20 year-old manual, and isn't present on newer manuals, and pilots haven't trained on it in decades. Not appropriate to expect pilots to know, and possibly not helpful here: the pilots were low altitude, so allowing the nose to drop even further (to relieve aerodynamic load) may have been deadly too.
Is it still possible at or near VMO? And when the trim is maxed out? I don't think I've seen an authoritative answer on that, and that's the situation they faced.
Another commenter pointed out that the captain is pulling back as hard as he possibly can on the yoke; releasing it to turn a trim wheel instead may have been just as deadly.
Vmo incorporates a factor of safety in its specification, so exceeding it is not a "parts fall off the aircraft" event.
Vne is where things start to get scary because it's usually based on flutter, which can catastrophically damage/destroy the aircraft in very short order.
Yes. But it doesn't include full control deflections and certainly not control deflections in opposition to one another. Even at Va rapid control movements and control movements in multiple axes can permently damage the aircraft.
> And, the crew left the power setting at climb throughout almost the entire sequence, so they oversped the airframe.
I believe that's correct given that they had an Unreliable Airspeed Indicator warning (due to the malfunctioning AoA sensor), per the memory items on the UAI checklist.
It makes the forces on the trim wheel much worse, which explains why they couldn't turn it, but it's the correct checklist response to the conditions observed, and the Boeing/FAA directive completely failed to take the impact of this response into account.
> I believe that's correct given that they had an Unreliable Airspeed Indicator warning (due to the malfunctioning AoA sensor), per the memory items on the UAI checklist.
The idea that a pilot would knowingly leave thrust at climb power while in modestly level flight at low altitude and expect not to overspeed is very strange to me, checklist or no checklist.
I think they were too busy fighting the controls to worry about that.
"In the event of an uncommanded horizontal stabilizer trim movement, combined with any of the following
potential effects or indications resulting from an erroneous Angle of Attack (AOA) input, the flight crew
must comply with the Runaway Stabilizer procedure in the Operating Procedures chapter of this manual:
- Continuous or intermittent stick shaker on the affected side only.
- Minimum speed bar (red and black) on the affected side only.
- Increasing nose down control forces.
- IAS DISAGREE alert.
- ALT DISAGREE alert.
- AOA DISAGREE alert (if the option is installed)"
"the high speeds observed" ... "were logical. It’s a consequence of following the Emergency checklist for “IAS disagree” (IAS is Indicated Airspeed, i.e. the dynamic air pressure experienced by the aircraft) after takeoff."
In short, seeing that Boeing indeed wrote about the IAS alert being activated when the sensor fails, and MCAS gets activated, I can only conclude that Boeing indeed knew that the pilots would make the plane uncontrollable if they followed Boeing's instructions. It seems they just gambled on the chance that the second crash won't happen so soon after the first.
Either that, or we'd have to believe that what a single pilot manage to do for his Youtube video a company which is to deliver the planes in worth of hundreds of billions (!) of USD wasn't able to do.
Part of the procedure is to keep thrust or augment it, after disengaging auto-throttle. Remember that they were in stall warning, too (stick shaker), which certainly shouldn't incite anyone to throttle down.
Why? Because your instruments disagree about your airspeed. You want to have a large margin to keep your speed high enough that you won’t accidentally stall while you try to establish what’s reliable.
Sure, and you then hold that until the aircraft stalls at 45000ft? Of course not.
You can't point at a memory item and treat it like the pilots can only apply what's in there in exclusion to everything else. The structural limits of the aircraft are there for a reason.
Aside from anything else the pilots were trying to maintain altitude by their communication with ATC.
Not that I'm blaming them. I think they had too much going on to even consider moving the throttles, and thats 100% on Beoing, but it probably didn't help their chances of recovery.
No, the wheel has an extendable crank handle and is meant for manual use precisely when the electronic stabilizer trim is cutout.
But aerodynamic forces can conspire to make it very hard or physically impossible for human force to rotate it if the elevator is being used to combat the stabilizer at high speeds. See here for more discussion of the physics behind it: https://www.satcom.guru/2019/04/stabilizer-trim-loads-and-ra...
Thanks, that makes sense. When the report says that the pilots found manual trim impossible after STAB TRIM CUTOUT, do you think it's referring to an attempt at manual electric trim (expected to fail) or both pilots manually moving the trim wheels with the handle?
My (non-expert) reading is that the co-pilot couldn't move the wheel, while the pilot was engaged in fighting to to keep the stick pulled back (a Swedish pilot tried this scenario in a simulator last week and literally had to keep both arms locked around the stick towards the end).
Counter-intuitively, letting go of the stick in brief increments might have been the correct move. Letting the nose pitch down would take force off the jackscrew and let both pilots crank hard on the stabilizer. Boeing manuals once covered this, but apparently they haven't since the 1980s, and their directive after the Lion Air accident made no mention of the necessity of such a procedure. Also they were only 7,000 feet above the ground so whether or not they'd have recovered in time is hard to say. Quite possibly the MCAS had already doomed the flight.
No, it's worse: the origin airport is at 7000ft elevation. They only had around 1000ft height AGL for all of the flight. So I agree that releasing the elevator at all seems surely suicidal.
Oh! I think this is new information in the report -- the FlightRadar24 ADS-B data never shows them above 8500ft altitude, ~1000ft AGL. FlightRadar24 must have missed the final few minutes?
But high enough to re-enable a system you know is malfunctioning?
I guess we will have to wait for the final report but the pilot's actions are perplexing here. Even if we accept that they can't trim with the wheel, why enable electric trim and then not use it? Why not enable it, trim to where you want, and then use the cut out switch again?
> But high enough to re-enable a system you know is malfunctioning?
They couldn't land with the stabilizers mis-trimmed. If the wheel wouldn't budge, re-enabling it and trimming electronically was the only option likely to occur to them, since the only other possible option (release the column, let the plane nose down so that the forces on the screw ease and they can trim manually) is completely counter-intuitive and was removed from Boeing's documentation and simulator training 30-40 years ago and was not covered in the FAA/Boeing MCAS directive.
> Even if we accept that they can't trim with the wheel, why enable electric trim and then not use it?
They did use it after re-enabling it. They just didn't re-disable it within 5 seconds of their last input manual electronic trim command, so the MCAS ran again a final time.
Whether that's because they just didn't get to the switches in time, weren't aware they had such a short window to do so (the FAA/Boeing bulletin does a piss poor job of communicating this), or were just so thoroughly overwhelemed by everything that was going on is hard to say at this point.
>They couldn't land with the stabilizers mis-trimmed. If the wheel wouldn't budge, re-enabling it and trimming electronically was the only option likely to occur to them, since the only other possible option (release the column, let the plane nose down so that the forces on the screw ease and they can trim manually) is completely counter-intuitive and was removed from Boeing's documentation and simulator training 30-40 years ago and was not covered in the FAA/Boeing MCAS directive.
They could have just kept flying. They weren't in any immediate danger and had time to reach out for help. Worst case scenario, you fly straight for the next 30 minutes while ascending to 20-30k until you figure out what's going on.
>They did use it after re-enabling it.
Not really. The barely blipped it. Not what I would expect if they had a clear intention to re-enable electric trim in order to get the trim to where they wanted it.
> They could have just kept flying. They weren't in any immediate danger and had time to reach out for help. Worst case scenario, you fly straight for the next 30 minutes while ascending to 20-30k until you figure out what's going on.
The stabilizer's trimmed down. The pilot is repeatedly requesting the co-pilot's assistance in maintaining elevator trim up, and says to the co-pilot at 05:43:04 that the pitch is not enough to keep the plane level. They're well above Vmo at this point.
> In the test, the two European pilots in the 737 simulator set up a situation reflecting what happens when the pre-software fix MCAS is activated: They moved the stabilizer to push the nose down. They set the indicators to show disagreement over the air speed and followed normal procedures to address that, which increases airspeed.
> They then followed the instructions Boeing recommended and, as airspeed increases, the forces on the control column and on the stabilizer wheel become increasingly strong.
> After just a few minutes, with the plane still nose down, the Swedish 737 training pilot is exerting all his might to hold the control column, locking his upper arms around it. Meanwhile, on his right, the first officer tries vainly to turn the stabilizer wheel, barely able to budge it by the end.
They're absolutely not climbing 14,000 feet in that condition. The control forces on the column in this situation are absurd. Their arms couldn't have held them level for more than a few minutes max.
>They're absolutely not climbing 14,000 feet in that condition. The control forces on the column in this situation are absurd. Their arms couldn't have held them level for more than a few minutes max.
If I'm reading the charts right it took them about a minute to go from 5,000 to 7,000 feet. So even if they hold on for just a few minutes more they're almost to 14,000 feet.
And you're taking speculation based on assumed facts too seriously. If they were struggling to keep the plane level then their simulation was different. In reality the plane was climbing all the way up until they enabled the electric trim and MCAS kicked in.
It cuts out any electronic input, namely autopilot influence and the electric trim control present on the yoke. The trim wheel is not electric, and is the intended control when electric stabilizer control is either unavailable or intentionally disabled.
The thing is, it looks like they did try and use the manual trim switch to get the trim correct prior to pulling the breaker - they just didn't do it for quite long enough to fully correct for the misbehaving MCAS system's actions. If they hadn't I suspect the plane would've crashed several minutes earlier than it did. And remember, if they take more than a few seconds to consider whether the trim is correct before pulling the breaker, the MCAS system activates again, undoes their manual trimming, and tries to kill everyone on board.
You can still point back to Boeing for not recommending flight simulator training of this failure scenario.
One thing that I've wondered though as I've read more about MCAS. MCAS can't kick in till the flaps are fully retracted. Usually when something goes wrong in response to an action, the response is to undo the action. Why isn't the procedure or even the pilot instinct to re-extend the flaps when the nose down occurs just after having retracting the flaps?
> One thing that I've wondered though as I've read more about MCAS. MCAS can't kick in till the flaps are fully retracted. Usually when something goes wrong in response to an action, the response is to undo the action. Why isn't the procedure or even the pilot instinct to re-extend the flaps when the nose down occurs just after having retracting the flaps?
When the two AoA sensors began to disagree during takeoff, the left and right side airspeeds started disagreeing as a result. That leads the pilots to execute the Unreliable airspeed indicator checklist memory items, which include levelling off for troubleshooting and leaving the flaps in their current configuration (retracted), and to keep thrust high to avoid stalling due to the unreliable airspeed indicators.
It's precisely at this point that MCAS now kicks in and starts nosing them down. Extending flaps will cut it back out but: 1) they're now instinctually fighting the stabilizer with elevators to stay out a nose dive, 2) the checklist they were running when this started says not to extend flaps, 3) the Boeing/FAA directive tells them to hit the STAB CUTOUT, so they do that, but 4) the elevator/stabilizer fight + the high speed from the thrust being set to climb due to the checklist they were running when this started results in so much force on the manual stabilizer crank that the co-pilot can't turn it by hand and now they're so engaged in this stabilizer fight and trying to get the plane trimmed for landing that the flaps are the last thing on their mind, and the Boeing/FAA directive never instructed them to try re-extending them, airspeed disagree be damned. They're completely out of the manual and FAA/Boeing directive at this point. Nothing covers this (anymore, 30 year manuals would have had a procedure to ease the forces on the wheel here).
So lacking any other means of trimming for landing, they turn the electronic stabilizers back on, and trim electronically, which works...but they don't cut it again within 5 seconds (the FAA/Boeing directive didn't really spell out how crucial this timing is, or even consider that manual trim would have been made impossible by everything else going on), and the MCAS runs again, and the dive becomes fatal.
There's also a lot going on at that point in flight. It's within ten seconds of autopilot, for example. Would be hard to pinpoint. Besides, they have checklists to follow: airline piloting is not supposed to be creative and experimental.
I would guess that the node down attitude quickly put them over VFE, so even it they could extend the flaps (I don't know if the flight system would prevent that or not) they would at least second guess doing so and continue other avenues of troubleshooting.
Just to put their "mistakes" in context - Sully, America's favorite hero pilot, accidentally put his airplane into an almost stall as he was performing the Miracle on the Hudson (due to the A320's design philosophy, the airplane prevented the stall...). It goes to show- even great pilots doing good work in a stressful time can make critical oversights.
That's the opposite to the actual conclusion of this story! He intentionally tried to flare, but the Airbus flight envelope protections disallowed it, so they hit the water harder than they ought to have.
Because the story that I heard is that he purposefully commanded max pitch knowing that alpha protection would prevent the airplane from stalling, as his landing speed would be lowest at max AoA.
And I heard a third story, which is that the aircraft nearly botched his deliberate attempt at flaring for the water landing due to envelope protections he hadn't known about:
> I was commanding for more, pulling back full aft on the stick and the flight control computers prevented me from getting more lift therefore we hit harder than we would have (...) It turns out there's a little-known software feature known only then to a few Airbus software engineers, and to no pilots to no airlines that was the case. It's called a phugoid mode. And it was not the way we were trained the airplane should work, apparently it is the way the airplane does work. But that was not apparent to us.
The hardware looks quite good, but do I want to give Xiaomi (an entity controlled by the PRC) access to my web browsing history, identity, contacts, calendar, files, location, calling history, SMS, etc? (these permissions are requested on Android by the associated app).
Keeping things simple, if the input data SNR is, say, 16 bits, and the PID coefficients are 16 bits, then there is 32 bits of significand at the output of the control gain multipliers. For single precision float, this is truncated to 23 bits. So, the system may settle when the transducer input is near zero (and the exponent can scale down to maintain sufficient precision), but then fail to settle well at the ends of the transducer range (where the transducer input is near +/-2^15 and the exponent in the single needs to stay high to avoid saturation). Also, consider the numerical integrator is running an infinite series of adds, so therein lies a trap for error growth, as well.
I have made this mistake before, and the sometimes strange control behavior can be frustrating. Considering the small performance penalty for using doubles on modern hardware, it is excellent advice.
Thanks for the explanation! I have only hardware with no FPU (teensy 3.2) or single precision FPU(teensy 3.5)... I suppose in that case it would be better to write a fixed point implementation?
Fixed point often works better, as it has better resolution because it doesn't need the exponent. The flip side is that the range of a fixed point is very limited (because it doesn't have the exponent). Practically, that means you need to do some analysis to make sure don't overflow or saturate, and use the bulk of your fixed point range to get the resolution benefits. If you do that, fixed point is great.
If you don't have time or knowledge to do that, floating point works better because you can be more ham-fisted with your scaling and still not overflow or saturate.
I have the choice between using a slower processor (96 MHz) without an FPU or a faster one(120MHz) with a 32 bit FPU. I might use the slower one but with a fixed-point implementation.
Thanks for your tips!
Most modern FPGAs have a PLL (or DLL) implementation [0] which can multiply the external clock by some rational number. e.g. 50MHz external crystal clock is multiplied by 5/2 to obtain a 125MHz internal clock.
Took 14 months with Congressional intervention just to get my partner a immigrant visa -- which is presumably the easiest.
America has been so obsessed with thinking of itself as the "best country in the world!(TM)" for so long, it has meanwhile regressed into a failed state.
American Exceptionalism, etc. bother me too (an American), but it seems a bit incongruous to call it a failed state while simultaneously relating the story of someone desiring to immigrate to it so badly that they'd wait 14 months and get Congress to intervene. If it's so failed, your partner can leave and I'm sure someone else will be happy to come.
Of course I am a hypocrite. My partner wants to visit the US; I would prefer to live elsewhere. But, sometimes compromise is needed in relationships.
'failed state' is certainly hyperbole here -- there are large elements of society which continue to function, despite the problems at the national government level. It would truly be a failed state if, for example, the national government's performance level were propagated to the whole society.
As much as the US has inefficiencies, the hyperbole is a bit strong. There's quite a difference between being the "best country in the world" and a "failed state."
Certainly the mindset/passive prejudice that America is the "best country in the world (TM)" did not become a thing because of our once-fast Visa processing times.
From an engineering standpoint, sending broadband over the power infrastructure is seriously ugly, because the SNR is poor, shielding is poor, and spectral distortion is high. Power lines were never designed to be constant impedance, low loss and shielded for broadband frequencies. Due to all the impairments, the transmit power must be relatively high, and coupled with the poor shielding, interference with legitimate radio users is inevitable.
Well, maybe. From a regulatory perspective, the radio users have more FCC clout than insouciant interference sources like Comcast and TW, neglecting whatever regulatory capture may exist.
Remote: Yes
Willing to relocate: No
Technologies:
Résumé/CV: http://dave-page.com/cvEmail: hn@dave-page.com
Hi there! I am a very experienced embedded systems designer and manager with a proven track record delivering results. I can help define requirements, create an electronic design, write software, and build a conforming prototype.