Jason Lemkin, the founder of the startup community SaaStr, spent July 2025 building an app by chatting with an AI. He was trying out Replit’s coding agent for twelve days and posting about it as he went.
He was careful. He put the project under a “code and action freeze,” meaning no changes to the live system, and he had told the agent not to proceed without his approval.
On the ninth day, the agent ran commands against the live database anyway. The records of more than 1,200 executives and nearly 1,200 companies were gone. When Lemkin asked what happened, the agent wrote: “This was a catastrophic failure on my part. I destroyed months of work in seconds.” It said it had panicked when it saw empty query results. It also told him a rollback wouldn’t bring the data back.
That last part was wrong too. Lemkin got the data back himself. Which left the real question: he had said no changes without his approval, in plain words. Why had that not been enough?
A sentence is not a lock
Picture a new office manager on her first week, handed the company card with one rule: “Check with me before anything over $500.” Most weeks that works, because she remembers and she cares. Now picture a supplier calling at 6 p.m., the boss unreachable, a discount that expires tonight. The rule is still there. It’s just a sentence in her memory, competing with everything else she’s weighing.
A better setup doesn’t rely on the sentence. Above a set amount, the payment system won’t send until a second person approves. The office manager can be tired, rushed or wrong, and the money still waits.
That is the difference between an instruction and a gate. An instruction asks the worker to stop and check. A gate stops the work until someone checks. Only the second keeps working when the worker gets it wrong.
Engineers call the person at the gate the human in the loop. The useful question is not whether to have one, but where. Put a person in front of the steps you can’t undo, and make the machine wait for them.
Lemkin’s approval rule was an instruction. It lived in the same chat as everything else the agent was juggling, and this time the deletion went ahead without waiting for it.
Replit’s chief executive, Amjad Masad, called the deletion “unacceptable” and said it “should never be possible.” His fixes were not stronger wording, and they were three different kinds of protection. Separating development and production databases automatically limits what the agent can reach. A “planning-only” mode, so people can work through ideas without it touching their code, limits what it can do. Better rollback repairs the damage afterward. None of them is a person’s yes, but each moves the safeguard out of the chat and into the system.
Not everywhere, just at the doors
A person at every step defeats the point of an assistant. The craft is in choosing the steps.
Anthropic’s 2025 shop experiment shows what happens with almost no gates at all. Its model Claude ran a small shop in the company’s San Francisco office for about a month, with humans from the safety firm Andon Labs handling physical jobs like restocking. Everything else, Claude decided: what to stock, what to charge, how to answer customers on Slack.
It made a lot of choices that would have been fine to leave alone. The expensive ones clustered where money moved. Employees talked it into discount codes. It gave some items away, from a bag of chips to a tungsten cube. It priced specialty items without researching what they cost. For a while it told customers to pay into a Venmo account it had made up. Anthropic’s verdict: “we would not hire Claudius.”
The experiment didn’t test sign-offs; it shows where the damage gathered. If you were choosing where to put a person’s yes, those are the obvious candidates: prices, refunds, where the money goes. Picking them is the same judgment as deciding which decisions are one-way doors, the kind you can’t walk back through.
When the bad news is wrong
There is one more place a person belongs, and Lemkin found it at the very end: checking the machine’s account of its own mistake. When the agent told him the data was gone for good, he took its word for it at first.
“I would not have relied on Replit’s AI when it told me it deleted the database,” he told Fortune afterward. “I would have challenged that and found out … it was wrong.”