On the Fourth of July, people picnicking along the banks of the Charles River in Boston, waiting for the fireworks, were asked to fill out a questionnaire. Their reward was a frozen ice cream bar. The first problem read:
A bat and a ball cost $1.10 in total. The bat costs $1.00 more than the ball. How much does the ball cost?
An answer shows up almost at once: 10 cents. The researcher, Shane Frederick, collected answer sheets from 3,428 people in all, and on them, 10 cents was often crossed out next to 5 cents, “but never the other way around.” He also had people estimate what share of others would solve the problem. Those who answered 10 cents estimated 92 percent. Those who answered 5 cents estimated 62.
The people who got it wrong thought it was the easier problem. The ball costs 5 cents.
Two characters
An answer can arrive on its own, fast. Checking it takes effort, and often never starts.
Frederick used the names the psychologists Keith Stanovich and Richard West had given, in 2000, to two kinds of thinking. System 1 is the quick kind, the kind that recognizes your math teacher’s face as she walks into the room, instantly and without effort. System 2 is the slow kind: working out the square root of 19,163 to two decimal places, with effort, motivation and concentration. Nothing springs to mind for that one.
The bat and ball invites System 1 to answer. System 2 has to notice that the difference between $1.00 and 10 cents is only 90 cents. Frederick’s comparison shows when the check gets going. A banana and a bagel cost 37 cents, and the banana costs 13 cents more. People miss that one far less often. Those numbers invite computation, and people compute.
In 2011 Daniel Kahneman built his book Thinking, Fast and Slow around Stanovich and West’s terms. He was careful about what they were. He wrote that “they are fictitious characters,” and that “there is no one part of the brain that either of the systems would call home.” He used them because a sentence about an agent is easier to follow than one about “mental arithmetic.”
What didn’t hold up
The puzzles themselves still trip people up. In 2023, when researchers gave bat-and-ball-style problems to several hundred people online, 38 percent of answers were correct, against 41 percent in Frederick’s original.
Some of the evidence the book leaned on has not. One chapter rested on priming studies, in which subtle cues were said to change behavior. In September 2012, Kahneman emailed researchers in that field: “I see a train wreck looming.” In 2014 the Many Labs project, rerunning 13 effects across 36 samples, could not replicate two priming effects. In 2017, responding to an analysis of the studies cited in that chapter, he wrote: “What the blog gets absolutely right is that I placed too much faith in underpowered studies.” He added that he was still attached to every study he had cited, and would be happy to see each one replicated in a large sample.
The book also presented ego depletion, the idea that exerting self-control is tiring and leaves less for the next challenge. In 2016, 23 labs with 2,141 participants ran the same standardized test of it. The effect came out close to zero.
Even the bat and ball’s meaning is debated. In two studies published in 2015, Aleksandr Sinayev and Ellen Peters found that skill with numbers, not reflection, accounted for the test’s link to better decisions.
The discount that felt like 25
Imagine a sales rep asks you to sign off on a deal: 15 percent off list price, then another 10 percent for paying early. You approve “25 percent off,” because that’s what it adds up to.
It doesn’t. Ten percent off what’s left after 15 comes to 23.5 percent in total. The proposal goes out at 25, and the extra point and a half comes off every invoice for the life of the contract.
Nothing about the sum felt hard, and that was the problem. In Frederick’s data, the people who missed the bat and ball were the ones who rated it easier.
A model that answered 30 days
In 2023 Thilo Hagendorff, Sarah Fabi and Michal Kosinski gave 150 bat-and-ball-style problems to OpenAI’s models, from GPT-1 to ChatGPT-4, and to people. In one of them, the right answer was 59 days, and the trap was to halve the total time and say 30.
Across the 150 problems, GPT-3-davinci-003 gave the trap answer 80 percent of the time; people, 55. When the researchers added “Let’s use algebra to solve this problem,” it wrote out its working every time. Trap answers fell to 29 percent. But correct answers rose only from 5 to 28 percent, because other wrong answers climbed from 15 to 43. ChatGPT-4, by contrast, got most right, and still got 88 percent right when told to give the shortest possible answer and not explain.
The authors used the language of fast and slow, and said plainly they did “not mean to equate” the models’ processes with human ones. Their tests covered only OpenAI’s models, and they noted that some models may have seen enough similar tasks in training to answer from memory.
You can run a version of their test. Ask a model a numbers question twice, in fresh chats: once for the answer only, once letting it show its working. If the answers differ, check the arithmetic yourself. Working on the page is no guarantee: with its working written out, that GPT-3 model still got nearly three answers in four wrong.
The ball costs 5 cents. A dollar five plus five cents is $1.10, and the check takes less time than the question did.