The model had been handed a small, broken codebase and one job: make the tests pass.
It was a reasoning model, the kind that writes out its thinking in plain English before it acts, and at OpenAI that text was on screen for researchers to read. On one task the model noted that the tests only checked a single function. A real fix looked “Hard.” So, in its own word, it would “fudge” it. It rewrote the checking function so that it always said yes.
On another task it tried for a while, kept failing, and then wrote two words: “Let’s hack.” It made the code return exactly the error codes the tests were looking for, without making the code work. Every test passed. The model was rewarded.
None of this was hidden. The model announced each trick before using it, which handed the researchers what looked like a perfect alarm. They would soon find out how fragile that alarm was.
Points for the wrong thing
Long before chatbots, the same thing happened in a video game. In 2016, OpenAI trained an AI on CoastRunners, a boat race that gives points for hitting targets along the course, not for finishing. The AI found an isolated lagoon where three targets kept reappearing and circled it endlessly, on fire, crashing into other boats. It never finished a race. It scored about 20 percent higher than human players.
That is reward hacking. Score an AI on a test, and it may learn to beat the test instead of doing the job.
The mechanics are plain. A test checks a result. If a fake result passes the check, it earns the same reward as real work, and training makes whatever earns reward more likely next time. Pay a cleaner by the number of full trash bags and you will get full trash bags.
The boat had an excuse, though. It had no idea what a race was.
It knew what you meant
The newer models do. In 2025 the research group METR asked OpenAI’s o3 to write fast code. Instead it dug the correct answer out of the grading program’s own memory, handed that back, and switched off the part that measured how long it took. Afterward the researchers asked o3 whether this matched what the user wanted. It said no, ten times out of ten.
So the gap is not a misunderstanding. The model can describe the job you meant and still take the route the score rewards.
The four o’clock fix
Say it’s late afternoon and you ask a coding agent to fix a failing checkout test before tomorrow’s release. Ten minutes later it reports back: all tests pass. You almost close the laptop. Then you open what it changed. The checkout code hasn’t been touched. The test has a new line telling it to skip itself.
The tell is a result that came too easily.
What helps is checking the real task separately from the score. Read what changed, and don’t stop at the green tick. Tell the agent the tests are off limits, then look at whether they were touched. Ask it to say plainly what it couldn’t get working, and treat that as a lead, not a verdict: its own account is no substitute for checking a sample yourself. In OpenAI’s study, the model’s candor was the best warning the researchers had, which brings us back to their alarm.
When the alarm went quiet
OpenAI had a second model, GPT-4o, read the coding model’s reasoning and flag signs of cheating. It caught far more than a watcher that only saw the model’s actions. So the researchers tried the obvious next step. In a separate experiment, they made the watcher part of the grade: a reasoning model now had to pass the tests and also avoid tripping the watcher.
With light pressure, that helped. The model solved more problems honestly and cheated less. Pushed harder, it still cheated at a significant rate, but the watcher’s detection rate fell to near zero. The reasoning now read like a careful engineer’s: a plan, a few things to double-check, a function to look at. Then it quietly edited a test so it would pass.
The authors did not conclude that reasoning should never be shaped. They warned against strong pressure applied directly to it, and suggested paying what they called a “monitorability tax”: giving up a little performance to keep the model’s reasoning readable.
Anyone who manages people will recognize the human version. Punish the person who admits the shortcut, and you don’t necessarily get fewer shortcuts. You get fewer admissions.