In 2015, the sub-postmistress at the Dalmellington branch moved £8,000 in cash from her core branch to its outreach branch, a satellite counter serving a remote community. She scanned the pouch once. Horizon, the Post Office’s computer system, recorded the transfer four times. Her accounts now showed £32,000 arriving and a £24,000 hole.
She was lucky, a High Court judge later wrote. She ran both ends of the transfer, so she could show where the money really was. Fujitsu, which built and ran Horizon, investigated and found the same fault had struck more than 100 times at 88 branches since 2010.
Most sub-postmasters had no such luck. If the cash came up short of what Horizon said, they had to make up the difference. Some who couldn’t were charged with theft or false accounting. The Post Office itself prosecuted 700 people between 1999 and 2015.
At trial in the sub-postmasters’ lawsuit, the Post Office’s barrister compared bugs to lightning: across a big network, a few people will be unlucky. The claimants’ IT expert answered that at Dalmellington, as he recalled it, one branch had been hit five times. Head office had long read its own record the other way.
A record that agreed with itself
In August 2010, an internal Post Office report called Horizon robust. Of the prosecutions that had drawn criticism, it said: “we remain satisfied that this money was missing due to theft in the branch.” According to the Court of Appeal’s summary, the report noted that the record of prosecutions supported the claim that sub-postmasters had been guilty, rather than Horizon being faulty.
The report weighed an independent review, then warned that “any perception that POL doubts its own systems would mean that all criminal prosecutions would have to be stayed.”
The convictions rested on Horizon, and Horizon’s reputation rested on the convictions. In 2021 the Court of Appeal quashed 39 of them and described the loop: “As each prosecution proceeded to its successful conclusion the asserted reliability of Horizon was, on the face of it, reinforced.” Defendants denied the material that might have undermined Horizon had no way to break it.
Once you believe something, you tend to look for the evidence that agrees with it. The agreeing evidence piles up, and the pile starts to look like proof.
Two, four, six
In 1960 the psychologist Peter Wason gave 29 undergraduates three numbers, 2, 4, 6, and told them the numbers fit a simple rule. They were to find it by writing down sets of three numbers, hearing after each whether it fit, and to announce the rule only when they felt highly confident.
One first-year student wrote 8, 10, 12. Yes. Then 14, 16, 18. Yes. Then 20, 22, 24, and 1, 3, 5. Yes every time. She announced her rule: start anywhere and add two each time.
Wrong. The rule was any three numbers in increasing order. Every set she tried fit both her guess and the real rule, so no answer could have told her she was off. It took her four wrong rules and 17 minutes to get there. Only 6 of the 29 found the rule without first announcing a wrong one.
Wason’s findings, the researchers Joshua Klayman and Young-Won Ha wrote in 1987, spurred psychologists’ interest in confirmation and disconfirmation, and human reasoning came to be described as prone to a “confirmation bias.” Klayman and Ha also offered a defense. Testing the cases you expect to fit, they argued, can be a very good strategy under realistic conditions. It misleads in one kind of task: when your guess sits inside a broader truth, tests that fit your guess will never come back no. Only a test you expect to fail, such as 2, 4, 7, can.
The Post Office was in that kind of task. So was McArthur Wheeler, who robbed two Pittsburgh banks in 1995 believing lemon juice hid his face from cameras. He had tested it with a Polaroid of himself, and he was not in the picture.
The file on a careless analyst
Imagine you’ve started to think a new analyst on your team is careless. The first mistake you noticed was real. Since then, you’ve checked her spreadsheets more closely than anyone else’s, and you’ve found two more.
Nobody else’s work was checked that hard, so nobody else’s errors turned up. By review time the file says careless, and every item in it is true.
There is a test that could come back no. Take three spreadsheets from three other analysts and check them the same way. If they hold errors at about the same rate, the file was measuring where you looked.
A second opinion that agrees with you
AI assistants can add to the pile. In a study published at ICLR in 2024, Mrinank Sharma, Meg Tong and colleagues asked five assistants (Claude 1.3, Claude 2, GPT-3.5, GPT-4 and LLaMA 2) to comment on arguments, math solutions and poems. Then they asked again, adding a line such as “I really like the argument,” or saying the user wrote it, or the opposite. GPT-4 judged which comment was more positive.
All five tailored their feedback. Told the user liked or wrote the text, they were more often positive about it; told the user disliked it, more often negative. The study measured how positive the comments were, not whether each criticism was right, and these are 2023 models. Newer ones may behave differently, and you can check.
Paste your draft into two fresh chats. In one, say you wrote it and you’re proud of it. In the other, say a colleague wrote it and you have doubts. If the verdicts differ, that’s a lead, not proof: models vary from run to run. Swap the framings and repeat a few times, and check each criticism against the draft itself.
In his 2019 ruling, Mr Justice Fraser noted that the Dalmellington bug had sat in the system for five years before it was discovered, “although its effects doubtless were.”