In 1999, Britain’s Post Office rolled out a new computer system called Horizon to its branches. It was built by Fujitsu to handle accounting and stock. Soon the people running local branches, the sub-postmasters, were seeing money go missing on their screens: shortfalls, often of many thousands of pounds, with no sign of where it had gone.
They complained that the system had bugs. They were not believed. Their contracts made them responsible for unexplained losses, so some paid the gaps out of their own savings. Others were taken to court. Between 1999 and 2015, the Post Office prosecuted 700 people, and other bodies brought cases against more. In all, more than 900 sub-postmasters were wrongly prosecuted because of incorrect information from Horizon. Many went to prison.
The shopkeepers had the truth on their side. The computer had the law. Since 2000, the law in England and Wales has presumed that a computer was working properly unless someone shows otherwise. The government later put it in plain words: “the computer is always right”, unless someone can show it is not.
Asking the machine if the machine is right
That presumption was set in 2000. In 2023, a very different kind of computer turned up in an American courtroom. A lawyer in New York with about 30 years of practice was working on a personal injury case against the airline Avianca. His firm’s research service had limited access to federal cases, so he turned to ChatGPT, which, he later told the court, he “falsely assumed was like a super search engine.”
The brief he drafted cited six court decisions. None of them existed. At some point he did try to check: he asked ChatGPT, “Is Varghese a real case,” and whether the other cases were fake. ChatGPT answered that it had supplied real authorities that could be found on Westlaw and LexisNexis.
The judge fined the lawyers and their firm $5,000. His opinion made a point worth keeping: there is “nothing inherently improper” about using a reliable AI tool, but the rules give lawyers “a gatekeeping role” over what they file.
The shopkeepers lost to a machine that stayed silent; a ledger doesn’t argue. This one talked back. A chatbot will explain, cite and reassure, in fluent sentences. And checking its answer by asking it again is barely a check at all. The only real check comes from outside the machine: in this case, a legal database where the cases either were or weren’t.
Why the screen wins
When a machine gives the answer, people tend to stop checking it. A shopkeeper’s account loses to a ledger. A lawyer’s doubt loses to a chatbot’s reassurance. Psychologists saw the same pull in a flight simulator long before either case.
In the 1990s, Linda Skitka and Kathleen Mosier, working with NASA, set out to measure it. They named it “automation bias” and split it into two mistakes. An error of omission is missing a problem because the machine didn’t flag it. An error of commission is doing what the machine says even when other information says it’s wrong.
In one study, 80 students flew simulated flights while watching gauges for six problem events. Some had an automated assistant that pointed out events; they were told it wasn’t perfect but that their other instruments were always accurate. The students with no assistant handled the six events correctly 97% of the time. The students with the assistant managed 65%. The same events, the same gauges in front of them. The assistant made them a little better at everything else, and much worse at exactly the moments it got wrong.
A second study tried the obvious fixes. Putting two people in the cockpit didn’t help. Neither did a “Verify” prompt next to every automated instruction. The one thing that reduced commission errors was training people about these two specific errors, and even then they still made about two and a half on average out of six.
It’s the driver who follows the satnav down a road they can see is closed. The screen isn’t more convincing than their eyes. It’s just easier to obey than to think. And in Skitka’s data, more errors went with one particular feeling: that the machine had taken over the job of watching.
One fact from outside the machine
Picture a hiring manager at six in the evening with 200 applications and a screening tool that has already ranked them. Number 143 is someone she met at a conference last month and remembers as sharp. The tool has put him near the bottom. Her next interview starts in twenty minutes. The ranking is right most days, which is exactly what makes it easy to let it be right today.
She opens his application instead of the score. The tool has marked him down for a two-year gap in his work history. The gap is the startup he told her about at the conference. It took four minutes.
That is the habit the lab points to. People taught specifically about omission and commission errors made fewer of the second kind. Before you accept a machine’s answer, name one fact you checked that didn’t come from the machine.
At the Post Office, individual checks failed to change the institution’s conclusion. Hundreds of sub-postmasters said the numbers were wrong, from the first shortfalls on, and the machine’s word outranked all of them. In May 2024 a new law quashed hundreds of convictions that met its conditions. In 2025 the government opened a review of the rule the sub-postmasters had been up against all along: “the computer is always right.”