On the evening of January 27, 1986, engineers at Morton Thiokol, which built the space shuttle’s solid rocket motors, joined a conference call with NASA. Challenger was set to launch the next morning. The forecast at launch time was about 26 degrees Fahrenheit.
Their worry was the rubber O-rings that sealed the joints of the rocket motors. After a cold launch a year earlier, in January 1985, they had found the grease between the seals turned black. “When I say black,” the engineer Roger Boisjoly later testified, “I mean black just like coal.” Their recommendation: don’t launch if it’s colder than 53 degrees, the coldest launch so far.
Lawrence Mulloy, who ran the booster project for NASA, later told the presidential commission how he had seen it. There was no launch rule for joint temperature. Thiokol was proposing a new one on the eve of launch, “after we have successfully flown with the existing Launch Commit Criteria 24 previous times.”
Twenty-four flights. The engineers were pointing at damage. The manager was pointing at the flights that damage had come home from.
Both were reading the same record.
How “acceptable” moved
Challenger launched the next morning at 36 degrees and broke apart 73 seconds later. The damage the engineers had pointed to had a history. The first erosion of a primary O-ring in flight was found after the shuttle’s second mission, in November 1981, and was not reported at the review that cleared the next one.
The January 1985 flight, at 53 degrees, came back with problems in four joints. Thiokol’s analysis ran to 18 pages. It said low temperature “enhanced probability of blow-by,” and it still ended: “Condition is acceptable.” By the time the issue reached the top-level review in February, it was one page: “acceptable risk because of limited exposure and redundancy.” The commission found no mention of temperature in that report.
Every step was small and came with a reason. Each also started from the step before, never from the design, which had not expected the seals to erode at all. Richard Feynman, a member of the commission, wrote: “The acceptance and success of these flights is taken as evidence of safety.” The commission found that NASA and Thiokol “accepted escalating risk apparently because they ‘got away with it last time.’”
Each time a warning sign passes without disaster, the next one looks more normal.
The sociologist Diane Vaughan went into the archives expecting a story of villains. “Based on the commission’s report, it looked like you had amoral, calculating managers who threw caution to the wind,” she told Columbia Magazine. The documents showed a decision process that NASA followed consistently. Because the fixes seemed to work and the shuttle kept returning, the O-ring problem came to be treated as an acceptable risk. In her 1996 book, The Challenger Launch Decision, she named the process the normalization of deviance. Her conclusion: “mistakes themselves are socially organized and systematically produced.”
No one has to break a rule openly on the day itself. By then the risk has already been accepted, one reasonable meeting at a time, through the normal process.
The waistband
Seventeen years later it happened again. Before Columbia broke apart in 2003, foam strikes at launch had been absorbed into what NASA counted as past experience so many times that the investigation board called that experience base “an elastic waistband, expanding to hold larger deviations from the original design.”
Japan’s parliamentary commission on Fukushima found a slower version on the ground. Regulators had required a safety recheck of the plant against new seismic guidelines, due in June 2009, and the operator decided internally and unilaterally to push that deadline to January 2016. The interim recheck had already shown that many reinforcements were needed, yet in March 2011 Units 1 to 3 still had none of them.
Six degrees
Say you run the kitchen at a small café. The sign on the walk-in fridge says 5°C or below, and each morning someone logs the reading. One morning in March it reads 6. The door was propped open for a delivery, and nothing smells off. You write “door open, fine.”
In April it reads 6 again, and nobody adds a note. By July the log says 7 most mornings. A new cook asks whether 7 is all right, and hears the kitchen’s version of 24 flights: it’s been like this for months.
Nobody in that kitchen chose to break the rule. Each morning was compared with yesterday, not with the sign.
That points to a check: read the log against the sign, not against last week. When “it’s always like that” answers a question, ask who decided the limit could change. Usually no one did. Then fix the fridge, and until it holds 5, move the food somewhere colder or keep less of it. The line itself can move only after someone re-examines the evidence and finds the new number still meets food-storage requirements. Without that, a manager’s signature makes 7°C official, not safe. Drift is what happens when no one makes the call.
“It worked last time”
In December 2025, the security researcher Johann Rehberger borrowed Vaughan’s term for AI agents, programs that take actions such as running code. Their outputs can be wrong, he wrote, and an attacker can steer them with instructions hidden in a file the agent reads. “Most of the time it goes well, and over time vendors and organizations lower their guard or skip human oversight entirely, because ‘it worked last time.’” The trap, in his words: “organizations confuse the absence of a successful attack with the presence of robust security.”
Part of this maps cleanly: each uneventful run gets counted as evidence, the way each returning shuttle did. Part doesn’t. An O-ring doesn’t adapt. An attacker does, so a long run of quiet days may say even less about an agent than 24 flights said about a seal.
On the shuttle, the drift left a paper trail: 18 pages, then one, then no mention of temperature. With an agent it can be a single checkbox, the review step someone switched off because nothing had gone wrong, and that nobody remembers is off.