In March 1974, federal judges in New York, Connecticut and Vermont began finding presentence reports in their mail, five a week for six weeks. The first twenty were real files, drawn from probation offices in their own circuit, with names, places and dates changed, and every judge who sentenced one of them read the same pages. The last ten came in two versions, split at random between two halves of the bench, to test particular details. None of the sentences they wrote would ever be served.
Case 1 was a union official over 40, convicted at trial on nine counts of extortionate credit transactions and related income tax violations.
The median sentence came back at 10 years in prison and a $50,000 fine. The most severe was 20 years and a $65,000 fine. The most lenient was 3 years.
Same man, same pages, same law. On paper, seventeen years of his life depended on whose desk the file landed on.
Whose desk it lands on
Same case, different expert, different answer, often by far more than anyone guessed.
In 2016, Daniel Kahneman and three consultants, Andrew Rosenfield, Linnea Gandhi and Tom Blaser, wrote about this scatter in Harvard Business Review under the name noise, “the chance variability of judgments.” They set it apart from bias with a bathroom scale. A scale that always reads four pounds light is biased. One that gives two different readings when you step on it twice is noisy. Kahneman, Olivier Sibony and Cass Sunstein followed in 2021 with a book, Noise.
Each of the Second Circuit sentences, taken alone, could look reasonable. Only the experiment set them side by side. And without knowing the right sentence, a heavy one alone doesn’t show a judge is biased.
More than harsh and soft judges
The obvious explanation is labels: a few harsh judges at one end, a few soft ones at the other. Even that would be noise, since the same file would get a different sentence depending on which judge drew it. But would the labels account for the spread? Anthony Partridge and William Eldridge of the Federal Judicial Center, who analyzed the study, went looking.
Ranked across thirteen of the cases, only two judges came out consistently severe and one consistently lenient. Of 29 judges whose average rank sat near the middle, 26 had handed down one of the ten harshest sentences in some case and one of the ten mildest in another. The disparity, they wrote, “cannot, on the whole, be explained by labeling the judges.”
Most of the scatter wasn’t in a few people. It was in the meeting of a particular judge with a particular file.
Irrelevant inputs add to it. In the German study where young judges rolled loaded dice to set the prosecutor’s demand, told under anchoring, one identical shoplifting case drew sentences from 1 month to 12 months on probation, and the roll accounted for part of that spread.
Where the scatter shrinks, or is expected
The 1974 authors noted limits. These were paper defendants, and measured by parole eligibility, Case 1’s range would have been one year to 6 2/3 years. Still, they judged the disparity “a reasonably good approximation of what really happens in the courtrooms of the circuit.”
Not every judgment is noisy. Kahneman’s team noted that chess masters looking at the same board largely agree; years of practice with fast, clear feedback leave little room for scatter. Some spread is expected: professionals exercising judgment won’t all reach precisely the same answer, the HBR authors wrote; that is what “a matter of judgment” means. Their complaint was about scatter far above what executives would tolerate, in jobs where identical cases should be treated alike.
And agreement isn’t accuracy. A scale that always reads four pounds light agrees with itself perfectly.
The audit nobody expected to fail
Noise has one convenient property: you can measure it without knowing the right answer. Give several professionals the same realistic cases, let each judge them alone, and compare. Kahneman’s team called this a noise audit.
At two financial services firms where they ran one, executives expected their professionals’ judgments to differ by 5 to 10 percent. On average, two employees’ judgments of the same case differed by 48 percent of their mean in one firm and 60 percent in the other.
Imagine you lead a team of recruiters, and a rejected applicant writes in: a friend with a nearly identical CV got an interview. Before deciding who made the wrong call, take three past applications, strip the names, and have each recruiter score them alone. The spread is your noise, visible without knowing who the best candidate was. If it’s wide, have everyone submit a view before any meeting, as the HBR authors advised, so the talk doesn’t simply converge on whoever spoke first or most confidently.
A formula that didn’t repeat itself
The same article saw algorithms as the radical cure: “Unlike humans, a formula will always return the same output for any given input.” A language model at temperature zero is supposed to work like that, always picking the most probable next token.
In September 2025, researchers at Thinking Machines Lab tested that on one open model, Qwen3-235B. They asked it “Tell me about Richard Feynman” 1,000 times at temperature 0 and got 80 different completions. The first 102 tokens always matched. Then, after “Feynman was born on May 11, 1918, in,” 992 said “Queens, New York” and 8 said “New York City.”
In their account, the cause lay in the serving software: server load set the batch size, and the kernels in use gave slightly different numbers at different batch sizes. With batch-invariant kernels they wrote, all 1,000 came out identical. That’s one model and one prompt in their setup. At ordinary settings, chat models vary from run to run by design.
If an AI scores tickets or code for you, give it the recruiters’ audit. Run the same five cases through fresh sessions several times each and write down the scores. A wide spread on one case is a lead worth more runs, not a verdict. A narrow spread proves only that the scores agree with each other, so check a few against cases where you already know the answer.
Parole could shorten a prison term. It couldn’t undo the first question. In 16 of the 20 cases, the judges disagreed on whether the defendant should go to prison at all, and the offender sentenced to prison “does go to prison; the offender sentenced to probation or a fine does not.”