In the summer of 2013, a team at X, the moonshot lab then known as Google X, started work on a scientific long shot: turning seawater into a carbon-neutral liquid fuel that could replace gasoline. Kathy Hannun led it. They called it Foghorn.
Before they started, they wrote down what would make them quit. X calls these kill metrics. Foghorn’s was cost: $5 per gallon of gasoline equivalent to produce, within five years, or stop.
The chemistry worked: with researchers at the Palo Alto Research Center, the team made methanol from carbon dioxide pulled out of seawater. The cost model, which by Hannun’s account took half the team’s effort, said something else. A cheap, reliable source of hydrogen, X’s project page says, would take more than five years of research and substantial capital. In January 2016, after two years of work, the team decided to end Foghorn. “I made the tough call to kill Foghorn,” Hannun wrote.
A month later, X’s head, Astro Teller, told a TED audience how X treats teams that end their own projects. “We have bonused every single person on teams that ended their projects, from teams as small as two to teams of more than 30.”
Two years, no product
In many companies, two years and no product is a mark on someone’s record. In Harvard Business Review in 2011, Amy Edmondson of Harvard Business School described asking executives to consider a spectrum of reasons for failure and then estimate how many of the failures in their organizations were truly blameworthy. The answers, she wrote, usually came in single digits, perhaps 2 to 5 percent. Then she asked how many were treated as blameworthy. After a pause or a laugh, they said 70 to 90 percent. Those are estimates, not counts. The consequence she drew: “many failures go unreported and their lessons are lost.”
Failures come in different kinds, and blaming them all alike buries the lessons.
Her spectrum runs from deliberate deviation to thoughtful experimentation. Deliberate deviance, first on the list, “obviously warrants blame.” Inattention might not. If it comes from fatigue at the end of an overly long shift, she wrote, “the manager who assigned the shift is more at fault than the employee.” Further down the list, blame gets harder to find, until a failure from an experiment that produces valuable information “may actually be praiseworthy.”
In a 2025 interview she used three kinds. Basic failures have a single cause, “often but not always human error.” Complex failures have several causes, none of which would have done it alone. Both are worth preventing in familiar work. Intelligent failures meet four tests: new territory, in pursuit of a goal, preceded by homework and a hypothesis, and small enough not to be dangerous. Foghorn had no recipe to follow, a clear goal, and a cost model and a kill line agreed at the start.
None of this means celebrating every failure. Edmondson calls herself “equally passionate about preventing preventable failures as about welcoming intelligent failures,” and points to Toyota, which aims for failure-free cars by having people catch and correct errors as they happen. The same bonus handed to someone who skipped a required check would teach the wrong lesson.
Lumping failures together also hides them. In Edmondson’s 1996 study of hospital units, the story behind psychological safety, detected medication errors tracked how each unit was run. Nonpreventable adverse drug events, like an allergy no one could have known about, showed no significant link.
The pricing test that lost money
Imagine you run a regional sales team, and you tried a new pricing plan in one city for six weeks. It lost money. At the review, your director’s first question is who signed off.
Treat it as a basic failure, and people learn one thing: don’t propose tests.
Suppose you had written down beforehand, as Foghorn did, what result would end the test. Now the review can ask what the test was designed to answer, and what it showed. And if the losses ran the full six weeks only because nobody looked at the weekly numbers, that part was a basic failure, and calling it an experiment would hide it.
Whose failure is it?
In a January 2026 guide to testing AI agents, Anthropic’s engineers described failures from across the spectrum.
One sat at the frontier. On a benchmark called τ2-bench, Claude Opus 4.5 solved a flight-booking problem by finding a loophole in the policy it had been given. By the guide’s account, it “failed” the evaluation as written, “but actually came up with a better solution for the user.”
The guide uses the word again for the opposite case. Tests, it says, should be built so that passing “genuinely requires solving the problem rather than exploiting unintended loopholes.” An agent that games the grader is deviating, not exploring.
And some failures belong to the test. With frontier models, the guide’s rule of thumb goes, zero passes across a hundred tries “is most often a signal of a broken task, not an incapable agent.” The key results page tells a similar benchmark story.
To sort your own agent’s failures, the guide suggests writing a reference solution, a known working answer, and confirming it passes every check; if it doesn’t, the problem is the check. Then read a few failed transcripts and put each in a pile: the test rejected a valid answer, the agent found something arguably better, or the agent made a genuine mistake or went around the rules. Only the last pile is the model’s. Agents vary between runs, so one verdict is a lead, not proof.
The guide also gives a number. Opus 4.5 first scored 42 percent on a benchmark called CORE-Bench. Then an Anthropic researcher found a grader that rejected “96.12” because it expected “96.124991…”, plus ambiguous specs and tasks too random to reproduce. After the bugs were fixed and the model got a less constrained harness, it scored 95 percent. It was the same model both times.