On the morning of August 1, 2012, minutes after the US stock market opened, one of Knight Capital’s order routers was sending orders it should never have sent, millions of them, and nobody at the firm yet knew why. Engineers worked on the problem while the system kept trading. Knight had no procedures to guide staff when significant issues developed, the Securities and Exchange Commission later found, so it relied mainly on its technology team to find and fix the fault live.
One of the things they tried was to take the day’s new code off the seven servers where it had been installed correctly. In the SEC’s words, “This action worsened the problem.” The faulty old code that had been running on the eighth server now began running on the other seven too.
On the surface, the move looked reasonable. Something new had gone wrong on the day new code went live, so they pulled the new code. What they couldn’t see was how the pieces were connected. The new code had borrowed a switch that once turned on an old feature, Power Peg, which Knight had stopped using in 2003 but never deleted. In 2005, the part of Power Peg that counted filled shares had been moved elsewhere, so the old code no longer knew when to stop. One server had missed the update. Another part of Knight’s system knew the orders were filled, but the router never got the message.
And the system moved faster than anyone could think. In about 45 minutes, Knight traded 397 million shares and built up positions that cost it $460 million. The case has the rest.
Two dials
The Yale sociologist Charles Perrow studied how large organizations and their technologies produce catastrophes. In his book Normal Accidents, he sorted risky systems along two dimensions, in his publisher’s summary: “complex versus linear interactions, and tight versus loose coupling.”
Interactive complexity means the parts can affect each other in ways nobody planned and nobody can easily see. Tight coupling means there is little slack between them: no buffer, no pause, no time for a person to step in. Writing in 2011, Perrow put the two together. In some systems, “the unanticipated interaction of multiple failures may create an accident that no designer could have anticipated and no operator can understand.” And “if the system is also tightly coupled, no intervention can prevent a cascade of failures that brings it down.”
He called such accidents “normal” not because they are frequent, but “because these characteristics are built into the systems.” Adding alarms and backups doesn’t remove them, he argued. “Nonlinear, unexpected interactions of even small failures can defeat these safety systems.”
Rare, and no excuse
Perrow was firm that true normal accidents are rare. In the same 2011 article he went through a string of disasters and said only one qualified: Three Mile Island. Around 4 a.m. on March 28, 1979, a relief valve at the nuclear plant stuck open, but according to the Nuclear Regulatory Commission, instruments in the control room told the staff it was closed. Worried that part of the system would fill completely with water, the operators cut back the emergency cooling water. The core overheated and partly melted. Like Knight’s engineers, they acted on the picture in front of them.
Far more often, Perrow wrote, systems with catastrophic potential fail “because of poor regulation, ignored warnings, production pressures, cost cutting, poor training, and so on.”
Knight had plenty of those ordinary failures. No second person checked the deployment. Ninety-seven automated emails flagging the problem before the opening went unheeded. So the theory is no excuse for sloppiness. Its use is as a design question: where can parts of this system affect each other in ways we haven’t mapped, and where is there no time to catch a mistake? Perrow’s answer was to reduce the first and loosen the second, for instance by building in modules instead of one tightly integrated whole.
Coupling also shapes the recovery. In the CrowdStrike outage of 2024, the faulty file was read by a security driver that loads early in Windows startup. According to CrowdStrike’s alert at the time, the first step was to reboot and give the machine a chance to download the fixed file. If it crashed again, the alert advised booting into Safe Mode or the Windows Recovery Environment and deleting the file.
The online shop
Imagine a small online shop that links three tools to save time. The website takes orders, a spreadsheet tracks stock, and a script automatically reorders from the supplier whenever stock of anything falls below ten. One evening the owner adds a column for a new product line. The script reads stock by column position, so it might now read prices instead, see every item as nearly gone, and place a large order overnight while everyone is asleep. Reading the column by name would remove the hidden interaction. A cap on any single order, or a morning approval step, would add the slack.
Agents that cascade
Anthropic’s account of building its multi-agent research system touches both dials. It found that “small changes to the lead agent can unpredictably change how subagents behave,” and that “minor changes cascade into large behavioral changes.” Because agents run for a long time and carry state, it wrote, “errors compound,” and “without effective mitigations, minor system failures can be catastrophic for agents.”
Anthropic doesn’t use Perrow’s terms, but several of its fixes add the kind of slack he meant. Its agents can resume from where an error occurred, with retry logic and regular checkpoints, instead of starting over. New versions go out through “rainbow deployments,” shifting traffic gradually while old and new run side by side, so an update doesn’t break agents already mid-task.
Knight’s engineers made a sensible-looking move in a system that was moving faster than they could understand it. The time to make such a system forgiving is before it opens for business.