In March 2019, a year after one of Uber’s self-driving test cars hit and killed a woman crossing a dark road in Tempe, Arizona, a county prosecutor wrote that there was “no basis for criminal liability for the Uber corporation.” In 2020 the person who had been sitting behind the wheel was charged with negligent homicide. In July 2023 she pleaded guilty to endangerment and was sentenced to three years of supervised probation.
She had been looking down at a phone, and the National Transportation Safety Board named that distraction as the probable cause. The same report listed three contributing failures on Uber’s side: inadequate safety risk assessment, “ineffective oversight of vehicle operators,” and no adequate way to deal with their “automation complacency.” Five months before the crash, Uber had also taken the second operator out of its test cars. At sentencing her lawyer said that with one person alone in the car, “It was not a question of if but when.”
Many people designed, tested and supervised that car. One of them stood in a courtroom as a defendant. She was the one sitting closest to the crash.
Where the car gives way
A modern car has crumple zones built to fold in a collision and soak up the force so the people inside survive. In 2019 the anthropologist Madeleine Clare Elish borrowed the image for something that folds the other way.
In a complex automated system, she wrote, the human operator can become the part that “bears the brunt of the moral and legal responsibilities when the overall system malfunctions.” Then the twist: “While the crumple zone in a car is meant to protect the human driver, the moral crumple zone protects the integrity of the technological system, at the expense of the nearest human operator.”
The day the machine was innocent
Elish’s first example was older than any self-driving car. In March 1979, a reactor at Three Mile Island in Pennsylvania partly melted down. Less than two weeks later, a Los Angeles Times front page read “Nuclear Accident Blamed Primarily on Human Error.” The presidential commission that investigated wrote that “except for human failures, the major accident at Three Miles Island would have been a minor incident.”
The operators did make bad calls, and at one point they overrode a safety system. The control room had no direct indicator of how much cooling water was in the steam generator. The operators had to infer it from a separate tank tied to pressure monitoring, and during the accident that tank stayed full and told them the wrong thing. “The operators made incorrect decisions,” Elish wrote, “because they had incorrect information.”
Her summary of the pattern: “The technology is maintained as faultless, while the human operator becomes the faulty feature of the system.”
What could she see?
It doesn’t take a reactor. Imagine a payroll clerk who runs the monthly batch and checks the totals against last month’s, as she was trained to. They look right. She clicks approve. Two weeks later it emerges that a tax table was never updated and a few hundred payslips are wrong. In the meeting, the first question is “Who approved it?”
Better questions come first. Could she see which tax table the software used? Did anyone tell her it might change? Could she hold the batch on payday without a manager’s sign-off?
If the answers are no, no and no, the approval was a signature, not a check. Elish calls this a mismatch between control and responsibility. One habit catches it: before deciding who is to blame, write down what that person could see, change and stop. If the list is short, the person is the crumple zone, and the fix belongs to whoever designed the job.
Who answers for the bot
AI pushes on this from two sides.
One is blame sliding off the company that deploys the system. When Air Canada’s chatbot gave a passenger the wrong refund rule, the airline argued before a tribunal that it wasn’t liable for what the bot said, and lost: the company was responsible for everything on its website.
The other is the familiar fold toward the nearest person. Elish noted how often design talk promises that keeping a “human in the loop” means human judgment can always step in. She found the catch in a 2015 Tesla announcement. Introducing new Autosteer software, Elon Musk said it was “almost to the point where you can take your hands off,” and in the same breath, “this is not a case of abdicating responsibility.” The machine takes the wheel, and the person keeps the blame.
The European Union’s AI Act takes the opposite view for the systems it classes as high-risk, and splits the duty in two. Article 14 is a design requirement: the system must reach its users in a way that lets the people overseeing it understand its limits, stay aware of automation bias, override its output, and halt it with “a ‘stop’ button or a similar procedure.” Article 26 is a duty on the organization using it: oversight must go to people “who have the necessary competence, training and authority, as well as the necessary support.”
That list is a fair test for any team putting a person beside an AI. Give that person the authority to say no, the information to know when, and a button that works. If you won’t, don’t ask them to sign.
The NTSB held both truths about Tempe: the operator wasn’t watching, and the company had not adequately recognized, or guarded against, the risk that a lone operator would stop watching. Only one of those truths ended up in a criminal courtroom.