Boeing 737 Crashes Caused by a Few Lines of Missing Code

Mish

The FAA certified the Boeing 737 Max on Wednesday. Here's the inside scoop on exactly what caused the crashes.

FAA Certifies the Boeing 737 Max

The Boeing 737 Max has been grounded since March of 2019 after two aircraft crashes killing everyone on board.

The first crash killed 189 people, the second crash killed 157.

On Wednesday, November 18, 2020, the FAA Cleared Boeing's 737 Max To Resume Passenger Service

After 20 months on the tarmac following two fatal crashes, Boeing's troubled 737 Max airliner has been given the green light to resume passenger flights, the Federal Aviation Administration announced Wednesday.

The plane's return to the skies will not be immediate, however. The FAA is requiring a series of design changes laid out in a 115-page directive. It also put forward training requirements for pilots and maintenance requirements for airlines.

"This airplane has undergone an unprecedented level of scrutiny by the FAA," Dickson said. "We have not left anything to chance here."

After the FAA announcement, the Air Line Pilots Association released a statement saying it "believes that the engineering fixes to the flight-critical aircraft systems are sound and will be an effective component that leads to the safe return to service of the 737 MAX."

Culture of Concealment

Boeing was aware of issues but did not disclose them to pilots or the FAA.  

Investigators found a "culture of concealment" as well as "grossly insufficient oversight by the FAA."

A Few Lines of Code

Leeham News author Bjorn Fehrm has interesting details in his take  FAA recertifies Boeing 737 MAX

Fehrm says "The 737 is a Safe Aircraft" and this "chain of events will not happen again on an updated 737 MAX".

Much of the discussion in from a pilot's perspective that is hard to follow but the key details are easy to understand even if you do not understand the terminology.

  1. The MCAS (Maneuver Characteristics Augmentation System) software was inaccurately classified as non “hazardous.”
  2. The inaccurate classification allowed a single sensor to control the MCAS.
  3. The MCAS was inaccurately coded.
  4. The original MCAS listened to the Speed Trim reset, “the Pilot trims,” instead of the correct “AoA is below the threshold again.” The result was MCAS trims, the Pilot trims, MCAS trims, the Pilot trims…. After 24 rounds in the Lion Air jet, the Pilots lost the race with MCAS.
  5. "MCAS went from a Pilot assist to a highly hazardous function by this single mistake in the MCAS software code. The whole drama came from the omission of a few code lines in the MAX Flight Control Computers software."

Boeing Changes

I describe the above in sufficient detail so we can understand how little in MCAS needed change to take it from a hazardous function to one that would have caused no trouble if wrongly triggered.

In addition to this change, Boeing has made additional changes to increase safety further. 

A single sensor no longer triggers MCAS. Both AoA sensors on the 737 MAX have to agree on the aircraft AoA, or Speed Trim including MCAS is deactivated (neither is needed to fly the plane. They are augmentation functions, i.e., good to have but not necessary).

On top of the dual-sensor activation of MCAS, its global authority, no matter what, is limited. The Pilot always has enough pitch control to fly the aircraft.

MCAS is Now Safe

To make MCAS safe, we only needed the correct reset criteria. But as the investigations dug deeper into how Boeing and FAA could miss how dangerous the original MCAS was, the requirements for changes grew. All eventualities, even remote ones, should be covered.

About Bjorn Fehrm

My Boeing contact who sent me the Leeham article notes Bjorn Fehrm is a former fighter test pilot and an aero engineer based in France.

Bjorn has said he would pilot the MAX as well as fly in it as a passenger.  

Most Expensive Lines of Code in History

  • Boeing is out $20 billion, not counting pending lawsuits.
  • 346 people are dead. 
  • 450 aircraft are grounded worth about $45 billion.
  • Mistrust of Boeing and the 737 Max will last for years.

Had a few lines of code been properly placed, there would not have been two crashes or 20 months of grounding even though other safety features were needed. 

In retrospect, it is not really the lines of code that were the problem. 

It was the "culture of concealment" coupled with "grossly insufficient oversight by the FAA."

Addendum

The Seattle Times has an excellent article on what happened in both crashes that is very readable by a lay person.

Q&A: What led to Boeing’s 737 MAX crisis

For those who blame pilot error, note that "Boeing assumed that the pilots would realize what was wrong and react appropriately within four seconds" without even being aware there was an MACS fighting their decisions. 

And in June 2018, before the first crash, another Boeing engineering memo acknowledged that a slow reaction by the pilots, if they took 10 seconds to react instead of four, would be “catastrophic.” These memos produced no change to the design.

The FAA did not see those memos.

Comments from my Boeing Contact

My experience in watching air crashes is, after a while, memory fades. Most people will don't pay attention to the type of aircraft they fly. Happened with the DC1- and the Lockheed Electra, and earlier, with the British Comet. After 6 months to a year of safe operations, I think the MAX will be accepted. As Bjorn Ferhm said in his LNA piece, the 737 (airframe) is basically a safe aircraft with a 50 year history. But time will tell.

Mish

Comments (51)
No. 1-23
Rocky Raccoon
Rocky Raccoon

I have talked many times with a couple of my pilot friends that pilots have become dependent on computers and are losing valuable skills in the air.

Anda
Anda

Hard to build redundancy into a computerised system where workable management is dependent on every input and calculation being correct. Centralised system gives major failure when it goes into error. The older mechanical alternative , say dual wires with one redundant, dual controls with one only for emergency etc. doesn't combine failure of any one facet of control with others. Usually it is found that disasters occur in those systems through unusual combination of simultaneous failure, whereas with the max failure was inbuilt to keep going to disaster repeatedly until the whole system was redesigned. This has happened mechanically as well on other aircraft previously, a design error for example , but then the first thing they do is ground all aircraft until cause is known. With the max and computer error they just decided to be able to blame the pilots :( .

Eddie_T
Eddie_T

I read several articles that explained that Boeing, in an attempt to compete with a new AirBus plane that was superior in design, decided to use larger engines than the Max was initially designed to use......which is what led to the need for the sophisticated fly-by-wire override in the first place.

The alternative would have been to go back to the drawing board and design a new plane from scratch....which they decided would cost too much and take too long.

"moving the engine nacelle (and a related change to the nose of the plane) changed the aerodynamics of the plane, such that the plane did not handle properly at a high angle of attack. That, in turn, led to the creation of the Maneuvering Characteristics Augmentation System (MCAS). It fixed the angle-of-attack problem in most situations, but it created new problems in other situations when it made it difficult for pilots to directly control the plane without being overridden by the MCAS.”

I expect the plane will be fine, given the scrutiny and the level of attention given to the problem and its software fix....but the underlying problem with the aircraft not handling well at certain angles of attack is not going to go away.

We can assume that any pilot who flies the airplane knows all about the problem now and has been vetted on the new system....and probably could avoid the problem in the first place by not taking the plane to the angle where it becomes vulnerable to misbehave.

I wouldn’t afraid to fly on one.....but that doesn’t make it a great airplane....and I think saying that a few lines of code “fixed the problem” is a little misleading.

One-armed Economist
One-armed Economist

I have long wondered why Boeing's purchase of McDonnel-Douglas (sp?) was not an anti-trust issue. Only 2 biggies - and they merge? Cozy Washington relations? Airbus was not a big competitor back then. Anyway, what if Boeing had have had a competitor? Could that have changed their behavior? After all they seemed callus and 'too big' to be challenged.

njbr
njbr

It is the same as the current "self driving" vehicles--where the driver is supposed to be in "watchful attendance" to the driving process.

Except, for most people, over time they drift into a deeper reverie, where the first reaction to an oncoming incident is over-reaction to something that has been ignored slightly too long.

The interface of human/machine is fraught with peril, even more so when the machine fights with the human for control.

KidHorn
KidHorn

I would guess the software wasn't developed and tested in the US. Most likely India. By people who have probably never flown in an airplane. You get what you pay for.

PreCambrian
PreCambrian

I haven't designed any flight control systems but I have designed many process automation systems. We used a PHA (Process Hazard Analysis) method which would have easily identified the issues with the MCAS. It would have been classified as a SIS (Safety Instrumented System) and with the hazard this high it would have used probably three AOA sensors, each of different type (to prevent a common mode failure) with 2oo3 (Two out of three voting) for any control action and at least two if not three processors (in case there was a failure with one processor). An alarm would have been indicated as soon as any of the three sensors did not agree within a designated tolerance. It is hard to believe that there isn't some type of international standard for the development of aircraft flight control systems like there is for process control systems (ISA 84).

Ninjango
Ninjango

The culture of concealment is former`s CEO Dennis Muilenburg way of doing things to inflate company stock valuation. He is fully responsible for the destruction of boeing reputation.

ToInfinityandBeyond
ToInfinityandBeyond

The BA CEO was shown the door taking a benefits package worth $62 Million with him. From everything I have read it sounds like someone or some folks should be doing some serious time for this fiasco. But there again you can’t believe everything you read I guess.

Casual_Observer
Casual_Observer

All well and good until the GRU is able to hack the max.

Casual_Observer
Casual_Observer

Culture of concealment is common in corporate America. Flying isn't free so caveat emptor.

Jackula
Jackula

Nothing has changed, don't get, fly, or ride first models of anything until the bugs have been worked out by a few years of use. That being said Boeing seems to have some serious QC issues. Also Boeing's space operations have been getting their doors blown off by Elon Musk/SpaceX. Again QC issues slowing them down big time.

Eddie_T
Eddie_T

I flew on these planes a few times before they were grounded. Nice seats...good wifi...lots of comfortable amenities compared with the old 737’s.....only had that one little problem....hehehe.

Mr. Purple
Mr. Purple

I used to fly a lot pre-Covid. Going forward, who knows. I swore I'd never set foot in a MAX, but I guess I'll start riding them when I forget about the crashes.

American Airlines put out a statement that they would identify the MAX during booking, and that if you found yourself switched to one at the gate, they would accommodate you if you wanted a different aircraft.

Webej
Webej

A lot more went wrong. The single sensor, changing the authority of the MCAS to 5× the original value without re-evaluating everything and without resubmitting the designs (which is a felony), regulators and industry in bed with each other, regulators outsourcing their own work to Boeing, an old design with motors too heavy and large requiring them to be hung to far to the front and changing the flights envelope.

It's complicated. With better code there wouldn't have been a crash, but a whole lot more things went wrong.

Six000mileyear
Six000mileyear

Don't blame the contractors who wrote the code. They had to provide Boeing with development plans, system requirements, module requirements, code reviews, test plans, and test reviews. Boeing had to sign off all those reviews. DO-178 is the standard process to be followed for airborne software.

American Gentile
American Gentile

I worked on 2 fighters, problem #1 was WRONG computer language being used for the planes, has to be functional language, not procedural. problem #2 was no accountability - company afraid to confess failures, self-interest, govt won't confess up, doesn't want to look bad, self-interest feeding each other's coverup. Where to start? Independent 3rd party auditing, but open to public scrutiny?

American Gentile
American Gentile

Wrong computer language used for control, has to be functional, not procedural. I talked this over with software manager for a major fighter, she admitted I was right.

Irondoor
Irondoor

As a former USAF pilot, I am mystified by the lack of a simple "disconnect" button on the control column that would allow the pilot to disconnect the computerized system in the event that the system is overriding his manual input regarding trim. In fact, there is normally a circuit breaker that would disable the trim augmentation in the event of what is known as "runaway trim". Pilots need to be able to manually override any computerized control systems and also need to be able to trim the flight controls independent of the augmentation system. Generally speaking, modern airliners are designed to be flown almost from takeoff to landing by computerized autopilot systems. If a pilot cannot fly the airplane from takeoff to landing manually, then he isn't really a qualified pilot. He's nothing more than a highly-paid computer operator.

inspectorudy
inspectorudy

If the two pilots had performed the correct procedures this would have never happened. It is called "Runaway Trim" and the procedure is to turn off the auto-trim function and trim by hand. They not only failed to do this but never pulled the power back and let the plane go uncontrollable. The problem still needed to be fixed but these pilots were not properly trained to fly this plane.

Mish
Mish

Editor

They had 4 seconds to get it right. From the addendum

"Boeing assumed that the pilots would realize what was wrong and react appropriately within four seconds."

"And in June 2018, before the first crash, another Boeing engineering memo acknowledged that a slow reaction by the pilots, if they took 10 seconds to react instead of four, would be “catastrophic.” These memos produced no change to the design,"

"Although Boeing had installed a warning light to alert the pilot if the two AOA vanes disagreed, because of a software error this didn’t work. It was functional only if the airline had paid for an optional extra that added the AOA reading on the primary flight display."

"Neither Lion Air nor Ethiopian Airlines had paid for that option. Boeing knew about this flaw in 2017, a year before the crashes, but didn’t consider it critical."

While developing the fix for MCAS, the FAA discovered a separate problem, which is that a very unlikely glitch in the microprocessor inside the jet’s flight control computer could theoretically create a similar scenario to the two crashes even without MCAS activating.

frozeninthenorth
frozeninthenorth

I disagree with the findings in two very specific ways:
(a) The aircraft is certified in the US but not in Europe or Asia -- they will see that the FAA was coopted by Boeing in its approval process. they will want to review these things themselves and not believe the Americans...who can blame them
(b) the MAX is the first commercial Boeing aircraft that employs the Airbus aircraft management system -- the aircraft is in charge and the pilot is an assistant. This is a major shift in Boeing's operation philosophy and design that has been under-discussed
(3) the aircraft was built with an inferior wing that should have been built specifically for the aircraft. Hence the requirement for MCAS so that the aircraft can operate safely.

Now the truth is that there is nothing wrong with the concept of an unstable aircraft (that's how military aircraft operate) and that's to a certain extent how Airbus aircraft function. Still, it is an important change for Boeing that was "shoved under the carpet"


Global Economics

FEATURED
COMMUNITY