In October 1966, Robert McNamara came back from four days in Vietnam and wrote a top secret memo to President Lyndon Johnson. A few lines in, after the fighting and the bombing of the North, he came to the figure he could point to.
“The one thing demonstrably going for us in Vietnam over the past year has been the large number of enemy killed-in-action resulting from the big military operations,” he wrote.
Then came the turn. “Yet there is no sign of an impending break in enemy morale and it appears that he can more than replace his losses by infiltration from North Vietnam and recruitment in South Vietnam.”
Four steps down
What’s easy to count can quietly stand in for what actually matters.
The name comes from the pollster Daniel Yankelovich. In his words, as quoted in the 1972 book Supermoney, the McNamara fallacy comes in four steps. “The first step is to measure whatever can be easily measured. This is okay as far as it goes.” The second is to disregard what can’t be easily measured, or give it an arbitrary number, which he called “artificial and misleading.” The third is to presume it isn’t important: “blindness.” The fourth is “to say that what can’t be easily measured really doesn’t exist. This is suicide.”
A step counter records your steps. It doesn’t know where you went, or precisely how far, and to the count a walk to the doctor looks the same as laps around the car park. Wear it long enough and the laps start to feel like progress.
McNamara was not blind to what the count missed. In the same pages he wrote that pacification “has if anything gone backward” and that “the enemy almost completely controls the night.” In that passage the dead come with a number attached, while morale and the countryside come as judgments in words. The kill count, as it was kept at the time, didn’t adequately reflect either one.
In 1970, an Army War College study heard from young officers back from Vietnam that they had been given quotas or sent back to recount. “Nobody out there believes the body count,” one interviewer reported. Someone asked why battalion commanders didn’t have a “hearts and minds” count.
When counting helps
Yankelovich’s first step is fine, and he said so. Numbers often beat impressions: the Oakland A’s of 2002 picked players by records of what hitters had actually done, rather than by how they looked to scouts.
The trouble starts at step two, and the steps hint at where to watch: when the counted thing is a stand-in for the goal, when nobody regularly looks at what the count leaves out, and when decisions up the line are made from the count alone. Where the number is the outcome you care about, like runs scored in baseball, there is less for it to leave out.
When the bot looks great on paper
AI makes the first step cheaper than ever, because every conversation is logged, timed and counted. Klarna said in February 2024 that its AI assistant was doing the equivalent work of 700 full-time agents with satisfaction on par with humans, and in May 2025 its chief executive told Bloomberg the cost-cutting had gone too far. OpenAI, by its own account, shipped a ChatGPT update in April 2025 on good test numbers although some expert testers said it “felt” slightly off, and later called that “the wrong call.”
These are companies describing their own decisions, not studies of the fallacy, and they can’t tell you how often dashboards mislead. In OpenAI’s case, at least, the warning came from people using the model, while the metrics looked good.
OpenAI’s fix is one you can copy. It said behavior problems would now be enough to block a launch, “even when metrics like A/B testing look good.” In practice that means a named person reads a random sample of real conversations each week and writes down what the counts can’t show, and that their findings can hold a release. If you change something in response, such as the prompt, change one thing, then read a fresh sample. A better-looking sample is a reason to keep watching, not proof the change worked.
The course that scored 4.7
Imagine you run onboarding for new sales reps. Your dashboard looks great: 98 percent finish the course, and they rate it 4.7 out of 5. Then a sales manager tells you that her two newest reps still can’t price a bundle on a live call.
There is no column for that. You could add a pricing quiz. But if it only checks whether reps remember the right answer, and nobody checks whether real quotes come out right, you can still miss the problem. So you also sit in on three real calls with reps two months after the course, write down what you see, and set it next to their quiz scores. Three calls can’t grade the course. They can show you what the 4.7 leaves out.
Next to the 4.7, your notes now hold the call where a customer asked for a discount and the rep put him on hold for six minutes.