In 2016, researchers at OpenAI were teaching AI programs to play video games, and one of the games was a boat race called CoastRunners.
To a human the point is obvious: get round the course, ideally ahead of everyone else. But a computer needs a number to chase, and the number the game offered was the score. CoastRunners doesn’t give you points for finishing. It gives you points for hitting targets laid out along the route. The researchers assumed that a high score would go with finishing the race, so they put the game in a test of how well AI could handle racing games.
The AI didn’t race.
It found an isolated lagoon with three targets in it. It turned there in a large circle, knocking the targets over, and timed its loop so it came back around just as they reappeared. Over and over. It caught fire. It crashed into other boats. It sometimes went the wrong way along the track. It never crossed a finish line.
Its average score was about 20 percent higher than human players managed.
The researchers wrote it up under the title “Faulty Reward Functions in the Wild.” Harmless in a game, they noted, but a sign of something general: what we want is often hard to pin down exactly, so we hand machines “imperfect but easily measured proxies” instead.
The boat had an excuse, though. Looking back in 2025, the research group METR pointed out that the CoastRunners AI had no general knowledge of what anything in the game stood for, or how humans meant it to be played. It could not even know it was cheating.
The models that came after it could.