On Goodhart's law and Boeing
On Goodhart's law and Boeing
Netflix has a new documentary on Boeing called Downfall, centered around the two crashes of the Boeing 737 Max planes. It is worth your time. But I wanted to talk about a lesson from the debacle: the importance of Goodhart’s law.
So why did the two Boeing 737 Max planes drop out of mid-air and kill several hundred people? A reasonable answer is that the angle of attack sensor failed, which caused the plane to misbehave. But sensors fail all the time; why did that prove fatal? We can use Systems thinking and apply Iceberg analysis to get to the root cause.
Before we get to the root cause, let me tell you a small story from the 1940s
.A paleontologist named Gustav Heinrich Ralph von Koenigswald and his team were gathering fossils in Indonesia of the Solo People (early humans). To get help from the local people, they provided an incentive: 10 cents for every hominid bone they could bring. Much to their horror, only later do they discover that the people were happily breaking large bones into many small pieces.
There is a term for this: Goodhart’s law.
It states that
When a measure becomes a target, it ceases to be a good measure.
Once you internalize this, you see this play out everywhere, especially in organizations where incentives, metrics, and policy changes are introduced all the time.
For example, a company may develop a new policy to improve customer service: bonus tied to the number of customer service phone calls handled. But, just like the locals splitting the bones up, the company may find its people finishing the calls as fast as possible, possibly without even helping the customers!
In academia, when publication count becomes the measure that gets tied to enure, you will find people publishing unimportant work. But if you then change the measure to be the citation count, hoping that important and impactful work typically gets more citations, you will end up seeing people forming citation rings to scratch each other’s backs.
Now, let’s come back to Boeing. Boeing used to take pride in making safe planes. But after the merger with McDonnell Douglas, the new CEO wanted to target the stock price growth. This means they want to sell as many planes as possible. Quality and safety take a back seat.
This kind of narrative trickles down as well. A mid-level manager may not be optimizing for the stock price. Still, they could be instructed to target a measure like the number of parts manufactured. Naturally, they are thus incentivized to take shortcuts.
Ultimately, what happened was that they built a new system called MCAS, but didn’t want to reveal to the FAA and the airlines that additional pilot training is necessary, since that increases the cost, decreases sales and ultimately affects the stock price.
Also worth your time:
Brandon Sanderson’s Advice for Doing Hard Things by Cal Newport.
From the book “A Short History of Nearly Everything” by Bill Bryson. Page 437.