At 9:37 on the morning of February 28, 2017, Pacific time, an engineer at Amazon Web Services set out to fix something small. S3, Amazon’s huge online storage service, had a billing system that was running more slowly than expected. The plan was to take a few of its servers offline.
There was an established playbook for this, and the engineer ran the command it described. One of the inputs to that command was entered incorrectly.
Instead of a few billing servers, a much larger set went down. Among them were the machines behind S3’s index, the subsystem that keeps track of what every stored object is and where it lives. Without it, S3 in Northern Virginia could not fetch a file, list one, store one or delete one.
Tens of thousands of web services keep their data on S3, so the damage spread fast. Medium, Business Insider, Slack, Quora and Giphy had problems. People who ran their homes through an automation service called Ifttt found they could not turn on their lights. Even a site called “is it down right now?”, which exists to tell you when other sites are down, was hit.
Amazon’s own status page was stuck too. The console used to update it also ran on S3, so until 11:37 a.m. Amazon could not change the status of individual services there, and it put out news through Twitter and a banner instead. According to the Guardian, the dashboard showed everything fine across the board until engineers could update it.
Getting S3 back took longer than Amazon expected. The index and a neighbor, the placement subsystem, had to be fully restarted, something Amazon had not done in its larger regions for many years, and S3 had grown enormously in that time. More than four hours after the typo, S3 was operating normally.
Amazon’s public summary never names the engineer and says nothing about what became of them. It describes them only as an authorized member of the S3 team, and it puts the problem somewhere else.
“The tool used allowed too much capacity to be removed too quickly,” Amazon wrote. So it changed the tool. It now removes servers more slowly, and it refuses to remove capacity that would take any subsystem below the minimum it needs to run. Amazon said it was checking its other operational tools for the same gap. It also moved up planned work to split the index into smaller pieces, which it calls cells, to limit how much a failure can touch and to speed up recovery.
“This will prevent an incorrect input from triggering a similar event in the future.”