At 8:01 on the morning of August 1, 2012, an email went out to a group of staff at Knight Capital. It said “Power Peg disabled.” Over the next ninety minutes, 96 more followed. Nobody acted on them. Staff usually ignored emails like these. They had never been built as an alarm.
Knight was no small shop. By the SEC’s count, about a tenth of all trading in US-listed stocks passed through it. And those emails were pointing at a problem that, within an hour of the opening bell, would nearly wipe the company out.
That day the New York Stock Exchange launched a new program for small retail orders, and Knight had updated its order router to take part. The new code borrowed a switch, a flag, that had once turned on an old feature called Power Peg. Knight had stopped using Power Peg in 2003. Nobody had deleted it.
In the days before, starting on July 27, a technician copied the new code onto the router’s servers. There were eight of them. The code went onto seven. No second person checked the work, and nothing in writing said anyone had to.
At 9:30 the market opened, with 97 warnings that nobody had acted on. On the eighth server, the borrowed switch woke Power Peg up. Back in 2005, the part of Power Peg that counted how many shares it had already bought had been moved somewhere else in the code. So now it had no way of knowing when to stop. Fed 212 small customer orders, it sent millions of orders into the market. Another part of Knight’s order system recognized that those orders had been filled, but the information never reached the router. The SEC later found that Knight had no procedure for halting the router in response to its own aberrant activity.
Engineers raced to find the problem while the system kept trading. One of their fixes was to pull the new code off the seven healthy servers. That made it worse. Now all eight were running Power Peg.
In about 45 minutes, Knight traded 397 million shares in 154 stocks. When it finally stopped, it owned $3.5 billion of stock it never meant to buy and had sold short $3.15 billion it never meant to sell. Closing out those positions cost it $460 million.
Less than three weeks earlier, Knight’s shares had traded above $12. Five days after the glitch, a group of investors put in $400 million to keep the company alive, and took a stake of more than 70 percent in return. The next summer, Knight merged with a rival, Getco, into a new company called KCG.
The SEC later fined the firm $12 million. In its order, it noted that a written procedure requiring “a simple double-check” of the deployment could have spotted the missing server and averted the whole thing.