At 2:10 a.m. UTC on June 1, 2009, somewhere over the Atlantic, ice crystals clogged the speed sensors on Air France Flight 447. For a moment the plane didn’t know how fast it was going. The autopilot did what it was built to do in that case: it switched itself off and handed the plane to the two copilots in the cockpit. The captain had gone to take his rest eight minutes earlier.
The fault itself was brief. The left-hand airspeed display was wrong for 29 seconds, and the last of the three readings was back within about a minute.
What happened next was not small. The pilot flying pulled the nose up. The stall warning went off, and at one point sounded without a break for 54 seconds. The plane climbed to about 38,000 feet, bled off its speed and stalled. It dropped at around 10,000 feet a minute, nose still pointing up, until it hit the ocean four minutes and twenty-three seconds after the autopilot let go. All 228 people on board died.
The plane was packed with automation built to prevent exactly this. That was part of the problem.
The job nobody can do
The better the autopilot, the less ready you are when it lets go.
A psychologist named Lisanne Bainbridge laid this out in 1983, in a five-page paper called “Ironies of Automation.” When engineers automate a job, they automate the parts they know how to automate. Whatever is left goes to the human, and what is left tends to be the oddest, rarest, hardest moments. In her words, the designer “still leaves the operator to do the tasks which the designer cannot think how to automate.”
Then the trap closes. Skills fade when they aren’t used. Someone who spends years watching a machine do the work is no longer the expert they once were. Yet the moment the machine hands control back is, almost by definition, a moment when something has already gone wrong. That, Bainbridge wrote, is exactly when the operator “needs to be more rather than less skilled, and less rather than more loaded, than average.”
It is the backup goalkeeper who sits on the bench all season and gets sent on for the penalty shootout.
Even the watching is a trap. Bainbridge pointed to vigilance studies showing that even a highly motivated person can’t keep real attention on a display where very little happens for more than about half an hour.
France’s air accident bureau, the BEA, reached a similar place about Flight 447. It put the pilot’s abrupt, excessive inputs down to the startle of the autopilot disconnecting, made worse by “the lack of practical training for crews in flight at high altitude.” Training, the report said, did not fill “the gap left by the non-existence of manual flying at high altitude.” Up there, the autopilot always flew.
Bainbridge’s final irony is the sharpest one: the most successful automated systems, the ones that rarely need a human at all, “may need the greatest investment in human operator training.”
The driver who was only watching
Then the same arrangement left the cockpit. Self-driving test cars kept a human in the driver’s seat, just in case.
On the night of March 18, 2018, an Uber test SUV driving itself in Tempe, Arizona, struck and killed a woman walking a bicycle across the road. The system had detected her 5.6 seconds before impact but never correctly worked out what she was or where she was heading. By design it would not slam on the brakes by itself to avoid a crash. It relied on the human.
The human was looking down toward her phone. According to records the investigators examined, her account was streaming a TV show for the whole trip. While the car was moving, she spent 34 percent of the time looking down there. She looked up about a second before impact.
The NTSB blamed her distraction. It also blamed Uber, among other things for not having adequate ways to deal with its operators’“automation complacency.” Five months earlier, Uber had gone from two operators per test car to one.
Two crashes, nine years apart, one in the sky and one on a road at night. In both, the machine handled the ordinary work and handed a person the one moment it couldn’t handle.
Keeping your hands warm
Picture a bookkeeper whose software now reconciles the month on its own. For a year she has only touched the handful of entries it couldn’t match. Then, at quarter-end, it flags one that fits no pattern: a single payment split across three accounts in two currencies. She used to be good at exactly this. She hasn’t done one by hand since last spring, and the auditors arrive on Monday.
The routine work was also her practice. Take it away and the exceptions get harder, because they arrive cold.
Hand the work to an AI agent and Bainbridge’s two conditions can appear. The AI is right most of the time, so you get little practice at the work itself. And your remaining job is to watch for the rare miss, through long stretches where nothing goes wrong.
After Flight 447, the BEA asked for practice. The copilots had not been trained to hand-fly a stall at high altitude, so it called for “specific and regular exercises” in exactly that. Uber, after Tempe, put a second operator back in its test cars.
The same two levers exist with an AI tool. One arrangement worth trying: do a share of the tasks yourself, say one in ten, so the skill stays warm, and spot-check a sample of the rest. Asking the tool to flag the answers it is least sure of can point your attention, but its own confidence is no substitute for an independent check.
Otherwise, when the AI finally hands you the hard case, you are the backup goalkeeper again.