On the Fourth of July, people picnicking along the banks of the Charles River in Boston, waiting for the fireworks, were asked to fill out a questionnaire. Their reward was a frozen ice cream bar. On the page with the problems, respondents were told only: “Below are several problems that vary in difficulty. Try to answer as many as you can.” The first problem read:
A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
An answer shows up almost at once: 10 cents. It is wrong. If the ball cost 10 cents, the bat would cost $1.10, and the two together $1.20. The ball costs 5 cents.
The questionnaire was Shane Frederick’s. His 2005 paper, which lists him at MIT’s Sloan School of Management, introduced a three-item Cognitive Reflection Test: the bat and ball, a question about machines making widgets, and one about lily pads spreading across a lake. Each was built so that a wrong answer springs to mind first. From January 2003, over 26 months, he gave the test to 3,428 people in 35 studies. Most were undergraduates paid $8 for a 45-minute questionnaire.
The average score was 1.24 out of 3. A third of all respondents got none right. Even at MIT, where the mean was the highest at 2.18, fewer than half got all three.
The paper sheets kept a record of how people got there. In the margins, Frederick wrote, 10 cents was often crossed out next to 5 cents, “but never the other way around.” Even among people who answered correctly, he wrote, the wrong answer was often considered first.
Frederick also asked people how hard the problem was, by having them estimate what share of other respondents would solve it. Those who answered 10 cents estimated 92 percent. Those who answered 5 cents estimated 62 percent. Both, he noted, were considerable overestimates. The people who had missed the problem thought it was the easier one.
His reading was that the 5-cent group had crossed out 10 cents in their heads and knew not everyone would. The 10-cent group, he wrote, thought the problem was too easy to miss.
He had a comparison that pointed the same way. A banana and a bagel cost 37 cents. The banana costs 13 cents more than the bagel. How much does the bagel cost? It is the same kind of problem, but these numbers invite working it out. People miss it far less often.