At 9:30 on the morning of August 1, 2012, the stock market opened, and one of Knight Capital’s order-routing servers began sending orders it should never have sent. It had no way of knowing that the customer orders it was working had already been filled. So it kept going.
The fact it was missing was sitting in Knight’s own systems. According to the Securities and Exchange Commission, one part of the firm’s order handling “recognized that the parent orders had been filled.” That information was never passed to the router.
Senior staff watching Knight’s main risk screen saw a huge position piling up in one account. But the screen, the SEC found, “relied entirely on human monitoring.” It was not connected to anything that could stop the orders, and Knight had no procedure “to halt SMARS’s operations in response to its own aberrant activity.” SMARS was the router.
It ran for about 45 minutes and cost Knight $460 million. The case has the rest of the morning.
The loom that stopped
In 1896 Sakichi Toyoda, the founder of what became the Toyota Group, developed a power loom with a device that stopped it automatically when a weft thread broke. Toyota says his looms did more than take over handwork: they “built the capability to make judgments into the machine itself,” cutting out defective cloth and the waste that came with it.
Toyota still calls this one of the two pillars of its production system. The word is jidoka, which it translates as “automation with a human touch.” When something abnormal happens, “the machine or equipment can detect the abnormality and stop automatically, or the operator can stop the line by pulling the stop cord themselves.” The cord is the human half, the andon. At NUMMI, the plant Toyota ran with General Motors in California, the message to workers was put simply: worry about quality, not volume.
Then comes the part that sounds backward. Toyota says a machine that can stop itself is one that “no longer need[s] to be watched over.” At Knight, senior people were watching the positions at the moment it mattered. The watching wasn’t connected to a stop.
First by hand
Jidoka has two halves: notice that something is abnormal, then stop and call a person. Both have to be built. An automatic stop needs a rule that detects the problem and a stop that is actually wired in.
Toyota’s way to the first half is an order of work. Before building a machine, “first be able to do the work smoothly and correctly by hand, determine abnormalities in the work, and replace those operations with machines.” Experience on the floor, records of past failures and testing all add to the list of what abnormal looks like.
Knight had limits. The account where the stray trades landed had a $2 million position limit, but it wasn’t linked to any automated control. Some of Knight’s trading strategies did shut themselves down when losses passed a limit. The router had nothing like that. A limit that no machine acts on is just a number someone might notice.
Stopping is also not the end. Toyota’s point is to make each abnormality visible so it can be traced and kept from coming back, which is where the five whys come in.
The flour order
Imagine a baker who sets up a spreadsheet to reorder flour whenever stock runs low. It works for a year. Then a supplier renames a product, a formula breaks, and the stock column reads zero every morning. The spreadsheet orders flour every morning. Over two weeks, twelve deliveries arrive.
The fix she adds afterwards is a hold. Before any order goes out, the system checks it, and anything more than twice the usual size, or a second order within a week, waits for her to approve it. She knows what “usual” means because she placed those orders by hand for six years. She tests it on last December’s big holiday order and on a deliberate duplicate, to make sure it doesn’t block the first or miss the second. It catches this kind of mistake. It won’t catch every kind.
Stopping conditions for agents
AI agents run in loops: plan, act, check the result, go again. Anthropic’s 2024 guide to building agents says it is “common to include stopping conditions (such as a maximum number of iterations) to maintain control,” and that agents can “pause for human feedback at checkpoints or when encountering blockers.” Its Agent SDK lets a developer set a maximum number of turns and a spending cap, and ends the run with an error when either is hit. The documentation warns that the cost estimate “can pass this value, so leave headroom.”
Caps like these are a coarse layer of jidoka, a limit on how much a run can consume. A turn counts round trips to tools, not what happens in them. A normal hard task can hit the cap, and a dangerous action can happen in the first turn.
So the checks that matter look more like the baker’s hold. Work out what wrong looks like, from doing the task by hand, from past failures and from tests: a file count that shouldn’t change, a total that should match its inputs, a command that should never run. Then put those checks where they can actually stop the agent before the action goes through, rather than in a note the agent reads. The SEC’s critique of Knight suggests one place to start: compare what leaves the system with what came in.
Toyoda’s loom never learned to mend a thread. It only stopped, and waited for someone who could.