On February 1, 2015, the Seattle Seahawks were one yard from a second straight Super Bowl title. They trailed the New England Patriots 28 to 24, with 26 seconds to play and one timeout left. Marshawn Lynch, whom NFL.com’s Michael Silver called “the most powerful goal-line runner in football,” had just carried the ball to inside the 1.
Lynch lined up in the backfield. Russell Wilson took the snap and threw a quick slant to Ricardo Lockette. Malcolm Butler, an undrafted rookie cornerback with no career interceptions, jumped the route and caught the ball.
By the next morning, Jerry Brewer of The Seattle Times called it “the worst play call in Super Bowl history.” Seattle’s coach, Pete Carroll, had already taken the blame on NBC: “That’s my fault, totally.”
Four days later, he said something that sounded like an excuse. “It was the worst result of a call ever,” he told Matt Lauer on the Today show. “The call would have been a great one if we catch it.”
A word from the poker table
A good result doesn’t prove a good decision, and a bad result doesn’t prove a bad one.
Poker players have a name for forgetting it. “There’s this word that we use in poker: ‘resulting,’” the former professional player Annie Duke told Nautilus in 2017. She described it as “creating too tight a relationship between the quality of the outcome and the quality of the decision.” In chess, she said, a loss almost surely means a bad decision you can go and find. In poker you can play a hand perfectly and still lose.
Her next example was Seattle’s pass, and the publisher’s description of her 2018 book, Thinking in Bets, leads with it too. A football play is closer to a poker hand: the coach’s call plus everything he doesn’t control, starting with a rookie’s jump on the route.
Before the snap
According to ESPN Stats & Information, Wilson’s was the 109th pass thrown from the 1-yard line that season, and the first intercepted. Carroll’s reasoning was about the clock: with one timeout, Seattle couldn’t run three times, and an incomplete pass would stop the clock and leave room to run twice.
That doesn’t settle it. Over the previous five seasons, ESPN found, runs from the 1 had scored 54 percent of the time and passes just under 50 percent. FiveThirtyEight’s Benjamin Morris worked through the clock and found the gap small either way: running ahead by 0.3 percentage points of win probability under the assumptions most favorable to Lynch, passing ahead by up to 3 under others. NFL.com’s Gregg Rosenthal blamed something else, the 40 seconds Seattle took between plays.
Every one of those arguments could have been made before the snap. The headline could not.
The operation that went wrong
Psychologists call it outcome bias. In 1988 Jonathan Baron and John Hershey at the University of Pennsylvania asked 20 undergraduates to grade medical decisions. In one, a bypass operation would relieve a 55-year-old man’s heart pain and raise his life expectancy from 65 to 70, but 8 percent of patients died from the operation itself. His physician decided to go ahead. In one version the operation succeeded; in its twin, the man died.
The students knew everything the physician knew. On average, they still rated the same decisions higher when the outcome was good. Of the eight asked afterwards, seven said outcomes shouldn’t count and one was unsure. A preregistered replication in 2023, with 692 online participants, found the effect in the same direction.
Results still count. Baron and Hershey called judging by outcomes “perfectly reasonable” when there are facts only the decision maker knew. Duke draws the line at sample size. One crash says little about a driver, but “if someone has gotten in 15 car accidents in the last year, I can certainly work backward from the outcome quality to their decision quality.” One interception is one hand.
The client who went under
Imagine you approved a steep discount to win a two-year contract with a fast-growing client. Their references were good, and the terms were your usual ones. Eight months in, they go under owing you money. At the review, the only question is who signed it.
Before the review, write down what you knew at signing and what you’d have bet. Then borrow a poker habit Duke describes: lay out the situation up to the decision point for a colleague who doesn’t know what you chose or how it ended, and ask what they’d have done. If they’d have signed too, look at the terms and the checks, not the signature. If your last five deals like this all went bad, the record is telling you what one deal couldn’t.
One reply from the model
A language model can give a different answer each time it’s asked the same question, so any single reply is partly a draw. Imagine you rewrite your support bot’s prompt, try it on one hard customer email, get a perfect answer and ship it. You graded the prompt on one hand.
Anthropic researchers raised this problem for benchmark scores in 2024. When one model beats another, their post asked, “could one model simply have gotten lucky in the choice of questions on the benchmark?” They recommended reporting error bars with every score and, for evals where the model reasons step by step, resampling its answer to each question several times.
At your scale, that means running the old and new prompts on the same 20 real emails, a few times each, and grading every reply against the standard you’d hold a person to. A gap that holds across most emails and repeat runs is worth acting on. A gap on one email, in one run, is a lead.
Had Lockette caught the ball, Carroll said on the Today show, the call “would have been just fine, and nobody would have thought twice about it.”