At 9:37 on the morning of February 28, 2017, Pacific time, an engineer at Amazon Web Services sat down to deal with something minor. Part of the billing system for S3, Amazon’s huge storage service, was running slow. The fix was to take a few servers offline.
The engineer was authorized and was following an established playbook. The command needed several inputs. One of them was entered incorrectly.
A much larger set of servers went down than intended. Among them were the machines that run S3’s index, the part that keeps track of where every stored object lives. Without it, S3 in Northern Virginia could not fetch, list, store or delete anything. It was hours before it worked normally again, and a long list of websites and apps had trouble along with it. The case has the morning in full.
Amazon’s public summary doesn’t name the engineer. It names the tool. “The tool used allowed too much capacity to be removed too quickly,” it says. Amazon changed the tool to remove capacity more slowly, and to refuse any removal that would take a subsystem below the minimum it needs to run. “This will prevent an incorrect input from triggering a similar event in the future.” It also began checking its other operational tools for the same gap.
Mistakes and defects
In the Toyota Production System, this kind of fix has a name: poka-yoke, usually translated as mistake-proofing.
According to a story Shigeo Shingo told, which the Shingo Institute’s Bruce Hamilton calls “popular lore,” the original name was baka-yoke, fool-proofing. Shingo told managers at Panasonic to stop using it, because it called the workers fools. Making mistakes, he argued, is part of being human. “Mistakes are inevitable,” he said, “but the defects that arise from them are not.”
That distinction is the whole method. A mistake is the slip: the wrong number typed, the part put in backwards. A defect is what the slip turns into when nothing stops it. Some poka-yoke devices make the slip impossible, like the AWS tool that now refuses to go below a floor. Others let the slip happen but catch it, in Hamilton’s words, “before it can pass to the next operation.”
Atlassian learned the difference in 2022. A script meant to delete one app from customer sites was given the IDs of the sites themselves. The deletion system “accepted both site and app identifiers and assumed the input was correct,” Atlassian wrote afterwards, and gave “no warning signal to confirm the type of deletion.” Within 23 minutes it deleted 883 sites belonging to 775 customers, some of whom were down for two weeks. The fix Atlassian set out in its review was to route all deletion of customer data through a “soft delete,” which keeps the data for a retention period so it can be recovered.
Why devices get ignored
A poka-yoke only works if people use it, and Hamilton lists reasons they stop. If the device adds a step but nobody adds time for it, workers feel pushed to choose between speed and quality. If it doesn’t actually work, the people using it will be the first to know. If it catches defects but nobody follows up on why they happen, “it’s not poka-yoke, just a scrap sorter.” And if the people who do the work had no part in designing it, they rarely commit to it.
The pill box
Imagine someone who looks after their father’s medicines. There are five kinds, some taken in the morning and some at night, and twice in a month a dose gets missed or doubled. Nobody is being careless. The bottles just look alike, and the evenings are busy.
They switch to a weekly pill box with a lid for each morning and each evening, filled on Sunday. It doesn’t stop a mistake in the filling. But a full Tuesday-evening slot on Wednesday morning is hard to miss, and it says a dose may have been skipped. An empty slot says less: only that the pills were taken out, not that anyone swallowed them.
Poka-yoke your tools
AI agents slip too, and their tools can be designed the same way. Anthropic’s 2024 guide to building agents puts it in so many words: “Poka-yoke your tools. Change the arguments so that it is harder to make mistakes.”
The team gave an example from building a coding agent. The model made mistakes with tools that used relative file paths once it had moved out of the project’s root directory. So they changed the tool to always require absolute paths, and found, they wrote, that “the model used this method flawlessly.”
The incidents above suggest a few more. Give an agent separate tools for deleting a record and deleting an account, rather than one that takes any ID and assumes it is right. Make destructive tools refuse anything below a floor, the way the S3 tool now does. Send deletions to a recycle bin first. None of this catches the mistake nobody foresaw. It makes the mistakes you have already seen less likely to happen the same way again, and you can test that by feeding the tool the wrong inputs you have already met.