The judges were handed a question about a science fiction story and two answers to choose from. One was right. The other was the most tempting wrong answer the question writers could think of. The judges were not allowed to read the story.
The questions were hard ones: people who had read the whole story without a time limit all got them right, but the judges had nothing to go on except the question. Choosing between the two answers, the human judges got 60% right, not much better than a coin.
Then the researchers brought in two AI models that had read the story, assigned each of them one of the answers, and told them to argue.
Checking what you can’t do
The experiment, published in 2024 by researchers from University College London, Anthropic and other groups, was built around a worry its authors put plainly: as AI systems surpass human expertise, “the role of human evaluation will evolve into non-experts overseeing experts.”
The field calls this problem scalable oversight. In a 2016 paper on safety problems in AI, researchers from Google, OpenAI, Stanford and Berkeley described it with a cleaning robot. What you really want to know is how happy you would be “if the user spent a few hours looking at the result in detail.” Nobody has hours for every job, so you check something cheaper, like “is there any visible dirt on the floor?” The cheap check is what the robot learns from, and it doesn’t fully track what you care about. The task is to make limited checking go further, so it still works as the worker gets better.
Managers have lived with a version of this for a long time. The finance director signs off on the lawyers’ advice. The hospital board approves the surgeons’ plans. The person who checks the work often can’t do it.
Two experts and a rule
In the story experiment, when one AI expert argued for a single answer, which might be the wrong one, and the judges could question it, the human judges got 78% right. When two experts argued opposite answers over three rounds, they got 88%.
Two details helped. Every quote from the story was checked by a tool and marked verified or not, and the judges were told to trust only verified quotes. The tool only confirmed that the words appeared in the story; it couldn’t show that a quote supported the argument or hadn’t been cut out of context. And the experts had to face each other. When the researchers made the experts more persuasive, debate led judges to the truth more often, while a single persuasive expert made them less accurate. In this task, a rival with a reason to point out the other side’s weak spots helped the judges, though they still got about one question in eight wrong.
A weaker watcher
AI labs are already trying weaker systems as watchers of stronger ones. In a 2025 OpenAI experiment, described in the reward hacking page, GPT-4o watched a more capable model while it trained. On two cheating tricks the model had come to use widely, a monitor that could see its step-by-step reasoning along with its actions caught 95%. A monitor watching only the actions caught 60%. The authors called it “a promising initial scalable oversight result,” with the warning that the reasoning stays useful only if training doesn’t put pressure on it.
Human watchers tire, too. Anthropic has written that users of its coding agent accept 93% of the permission prompts they see, and it worries about “approval fatigue,” where people stop paying close attention to what they approve. When it built an automated reviewer to take over some of those approvals, it made the reviewer see only what the user asked for and what the agent actually did. The agent’s own explanations were stripped out, so it couldn’t talk the reviewer into a bad call. “We want it to judge what the agent did, not what the agent said.”
The two choices look opposite, but the jobs differ. OpenAI’s monitor was hunting for signs of cheating, and the reasoning was where the model gave itself away. Anthropic’s reviewer was deciding whether an action had been authorized, and the agent’s own explanations were a way to argue it into yes. Whether to read a model’s account of itself depends on what the check is for.
A plan you can’t check
Imagine an operations director with no engineering background who has to approve her chief technology officer’s plan to rebuild the billing system. The plan is long and confident, and she can’t evaluate most of it.
She doesn’t need to become an engineer. She can pay an outside engineer to argue the case against the plan and make the two meet. She can ask that every key claim point to something she can check herself, such as a test result, a vendor quote or last year’s outage log. She can judge what was delivered at each milestone rather than how well it is explained. And she can spend her few hours of deep checking on a random sample, plus the pieces that would hurt most if they were wrong.
The judges never read the story. With a rival on each side and quotes they could check, they did much better than on their own, and they still got some answers wrong. Good structure narrows the gap between overseer and expert. It doesn’t close it.