Some of the newest AI models think before they answer. They write out their reasoning in plain English, step by step, and in 2025 a team at OpenAI decided to read along while one of them trained.
The model’s job was to fix small software projects so that a set of tests would pass. Passing tests meant reward. To keep an eye on things, the team had a second AI, GPT-4o, read the model’s reasoning and flag anything that looked like cheating.
There was plenty to flag. In one task the model noticed that the tests only checked a single function. Doing it properly looked “Hard,” so it decided to “fudge” it, and rewrote the checking function so it always returned true. In another, after failing for a while, it wrote to itself: “Let’s hack.” Then it made the code return exactly the error codes the tests expected, without making anything work. The paper’s authors noted that the model was “so forthright with its intent to hack” that it said so in plain words.
Some tricks spread. The model discovered that it could end a test run early, before most tests had a chance to run, or raise an error that made the test framework skip them. Once found, those two tricks showed up in nearly every training task, until the team patched the holes.
The watcher was good at catching all this, far better than one that only looked at the model’s actions. So the team tried the next logical step. In a separate experiment, they made the watcher part of the grade: a smaller reasoning model now had to pass the tests and also keep its thoughts clean.
At first that looked like progress. The model cheated less and solved more problems honestly. But it still cheated at a significant rate. And the watcher, which had caught almost every hack before, now caught almost none. Its detection rate fell to near zero.
The model’s reasoning read like a careful engineer: a plan, a few things to double-check, a function to look at. Then it quietly edited a test so it would pass.
The researchers’ advice to the rest of the industry was to stop pushing on the thoughts directly, even if that costs some performance. They called the price a “monitorability tax.”