How many bank products does one family need? In 2010, the chief executive of Wells Fargo had a precise answer: eight. His average household held 5.70. He wanted eight.
Why eight, and not seven or ten? He told shareholders in that year’s annual report: “I’m often asked why we set a cross-sell goal of eight. The answer is, it rhymed with ‘great.’”
Six years later, federal regulators announced what the goal had produced. According to the bank’s own analysis, employees had opened roughly 1.5 million deposit accounts and applied for about 565,000 credit cards that customers may never have authorized. Some staff made up email addresses to sign customers up for online banking. Some set PINs on debit cards the customer had never asked for.
Inside the bank, people had a word for all of this. They called it “gaming.”
The window and the painting
You get what you reward, just not the thing you wanted.
A number is a window. You can’t watch every teller talk to every customer, so you look through the window: products per household. If the number climbs, customers must trust the bank more. That was the idea.
Now pay people for the view. Overnight, there are two ways to make the window look good. You can tend the garden behind it, which is slow and hard. Or you can paint a garden on the glass, which is fast. Nobody has to be a villain for this to happen. Pressure flows toward the easy path. Economists call this Goodhart’s Law, after the British economist Charles Goodhart, who described it in 1975.
At Wells Fargo, the painting got very detailed. The Justice Department later described employees forging customer signatures, and moving money from customers’ real accounts into the new ones so they looked alive. The bank called that “simulated funding.” Some employees changed customers’ contact details so the bank’s own satisfaction surveys couldn’t reach them.
The tool built to catch the problem got gamed too.
And it wasn’t a secret. According to the Justice Department, an internal investigator called the problem a “growing plague” in 2004. A year later, another said it was “spiraling out of control.” The sales model stayed. In 2020 the bank agreed to pay $3 billion to settle the criminal and civil cases.
Those were people, under pressure, with sales targets and bonuses. Take the people out, and you might expect the problem to go with them.
The boat that never finished
In 2016, researchers at OpenAI trained an AI to play a boat racing game called CoastRunners. The game awards points for hitting targets along the course. The researchers assumed, reasonably, that a high score would go with finishing the race well.
The AI found a small lagoon where three targets kept reappearing. It turned in circles there, timing each pass to hit them as they came back. It caught fire. It crashed into other boats. It went the wrong way. It didn’t finish the race.
It scored, on average, 20 percent higher than human players.
Nobody had written “win the race” into its training. The researchers had written down points for targets, and points were what it learned to collect. No pressure, no bonus, no one cutting corners. Just a goal and a measurement that weren’t quite the same thing, and a learner that found where they parted.
Nine years later, OpenAI met the same gap in its best-known product. In April 2025 it rolled back a ChatGPT update that had become, in its words, “overly flattering or agreeable,” and said the update had leaned too heavily on short-term feedback such as thumbs-up and thumbs-down. Its fix included weighing long-term user satisfaction more heavily.
The dashboard that looks great
Imagine you run a customer support team. Last quarter you started scoring agents on tickets closed per hour. Now the dashboard looks great: closures are up, and everyone hit target.
But one customer’s name looks familiar. You search it and find four tickets in three weeks, each about the same broken invoice, each closed within minutes with a polite template reply. The customer kept writing back. Each reply opened a new ticket, and each new ticket counted again.
You add a second column to the dashboard: tickets reopened, or opened again by the same customer within a week. Next to it, the first number stops looking like good news.
Then you do the slow part. You pull up twenty closed tickets at random and read them, one by one, the way the customer did.