On Monday, 23 March 1998, Harold Shipman asked a doctor at the Brooke Practice in Hyde, directly across Market Street from his surgery, to countersign another cremation form. That day, the inquiry later concluded, the practice probably pulled its figures. In three months its doctors had signed 16 such forms for Shipman, who worked alone. That count left out anyone who died in hospital, was buried, or went to the coroner. The Brooke Practice itself, five doctors and about 9,500 patients, had recorded 14 or 15 deaths in the same period, counting those in hospital.
The doctors guessed Shipman had about 3,000 patients. His death rate looked, at the very least, about three times theirs. Yet for months the unease had been met with ordinary explanations. He had a lot of elderly patients. He was known for visiting them at home.
The next morning one of the partners telephoned the coroner, who asked the police to investigate. The detective, the inquiry found, “attached no importance to the numbers or death rates,” and closed the case on 17 April. Shipman killed three more patients before his arrest that September. The inquiry eventually found he had killed at least 215. The case has the rest.
When do fluctuations indicate trouble?
That question is a section heading in a 1931 book by Walter Shewhart, a member of the technical staff at Bell Telephone Laboratories. Some variation, he wrote, comes from “unknown or chance causes,” and a process can still vary within a steady range. If the past lets you predict that range, he called the process controlled. Other variation, though, comes from what he called “assignable causes,” and of those he wrote that they “may be found and eliminated.” Today the two kinds usually go by other names. In the words of NHS England’s guide to the subject, common cause variation is the “normal variation,” and special cause variation shows up as “unusual patterns” that are “unlikely to have arisen by chance.”
Ordinary wobble comes from the system. An unusual swing may have a cause of its own. The trouble is telling which one you are looking at.
Think of a commute that takes 25 to 35 minutes depending on lights and traffic. Asking why Tuesday took four minutes longer than Monday will rarely turn up anything. A 70-minute day is different; something probably happened, and it is worth finding out what.
Two ways to be wrong
Shewhart described two mistakes, as the quality engineer Mustafa Shraim summarizes them for the Deming Institute. The first is reacting to an outcome as if it had a special cause when it came from the common ones. That one is called tampering. The second is treating an outcome as ordinary when it had a special cause.
In Hyde, the second mistake had good cover. An elderly list and a doctor who made home visits were plausible explanations, and both were about the system. They were offered for a gap the figures suggested was far larger than such things usually produce.
Statistics brings the opposite risk, mistaking the ordinary for a sign of something. When researchers later applied cumulative sum charts, a tool from statistical process control, to deaths among the patients of 1,009 GPs from 1993 to 1999, 33 doctors crossed the alarm threshold, Shipman among them. The authors thought poor data quality, and factors such as patients in nursing homes or hospices, likely explained most of the other alarms. Shipman was not the most extreme outlier, a 2008 study noted. It concluded that at best such monitoring could act as “a backstop.” A chart can tell you where to look. It cannot tell you what you will find.
The bottom of the league
NHS England’s “Making data count” guides set out the limits of two habits common in the health service: comparing one data point with the one before, and red-amber-green reports. One of its examples: “Performance increased by 2% to 79% but the indicator is showing common cause variation.” Its suggested version for a decision-maker: “There has been no significant change.” On rankings, it asks whether the worst performer needs help, or is “just bottom because someone has to be.”
Imagine a regional manager who calls whichever of six shops had the lowest takings each month. Most months that call may land on a shop that simply drew the low card. If one shop then falls well below its own usual range for three months running, it can look like just another bad month, and the call that mattered might not come.
Noise on the scoreboard
AI models have scoreboards too. “Is the difference in capabilities real,” Anthropic asked in a 2024 post on evaluation statistics, “or could one model simply have gotten lucky in the choice of questions on the benchmark?” It recommended reporting error bars, and found that on popular evals, error bars that account for related questions “can be over three times as large as naive standard errors.” Ignoring that, it wrote, “may lead researchers to inadvertently detect a difference in model capabilities when in fact none exists.” That is the first mistake. It also warned of the second: with too few questions, “small differences will likely go undetected.”
Its remedies include comparing two models question by question and working out in advance how many questions it takes to see the difference you care about. A team comparing two versions of a prompt on a few dozen test cases faces Shewhart’s question in miniature: is this gap ordinary scatter, or a cause worth finding?