For a study published in 1977, introductory psychology students at the University of Michigan were shown into a room with a television and told a story. The psychology department had collected end-of-term ratings of its instructors. Could a short videotaped interview predict them?
The teacher on the tape was a Belgian whose first language was French, and he spoke English with a fairly pronounced accent. He answered ordinary questions about discussion in class and how he graded.
Half the students saw him answer warmly. He enjoyed the give and take of discussion, he said, and set papers his students seemed to like. The other half saw the same man answer coldly. He didn’t allow much discussion, because “there’s a time to be a student and a time to be a professor.” He gave weekly quizzes because otherwise “you can’t trust them to do the reading.”
Then every student rated how much they would like him, along with three things that should not depend on any of that: his looks, his mannerisms, and his accent.
Both groups were rating the same man’s accent.
The accent changed sides
A substantial majority of the students who saw the warm version found his appearance appealing. A substantial majority of those who saw the cold version found it irritating. About half the warm group liked his accent. In the cold group, the overwhelming majority found it irritating.
The researchers, Richard Nisbett and Timothy Wilson, asked some of the students whether liking him had colored those ratings. Most students said no, and most held to it when questioned afterward, some heatedly. “Before he even began talking, I made my judgment about his appearance and then I stuck with it,” one young woman said.
Those who had seen him cold, asked the question the other way round, went further. On balance, they said the influence ran in reverse: his looks and accent had made them like him less. “The subjects appear to have gotten matters precisely backwards,” the authors wrote. The feeling had come first and colored the accent, and the people inside it believed the accent had come first.
One overall impression can color every separate judgment, and you may not notice.
Too high and too even
The effect was named decades earlier. In a 1920 paper, Edward Thorndike of Columbia examined ratings that army officers gave their subordinates. The form asked for separate scores on physical qualities, intelligence, leadership and character, and told raters to consider “only the qualification it covers, totally disregarding all the others.”
They couldn’t. One very conscientious flight commander rated 137 aviation cadets, and his intelligence scores tracked his physique scores at a correlation of .51, higher, Thorndike argued, than reality could give. The correlations were “too high and too even,” as if each trait were lit by “a halo belonging to the individual as a whole.”
Thorndike saw the halo only in the numbers. Nisbett and Wilson showed that from inside it feels like nothing. It works like tinted glasses: you don’t see the tint, you see a yellow world, and every detail you check comes back yellow. Nor does it take strong feelings; the impressions in this study, the authors noted, were milder than love or hate.
The review form
Say you manage seven people, and the annual review form has six boxes for each, from results to documentation. One of your people saved the year’s biggest client renewal back in March. You open her form and every box comes out a 5. Her documentation is thin, and you know it, but March is a warmer memory than a missing handover note. The next person had a steady, quiet year and gets 3s straight down the page.
Six boxes, one judgment written six times. She never hears about the handover notes, and he never hears what he did well.
Telling yourself to judge each box on its own is what Thorndike’s form already did. Changing the order may help more. Fill in one box for the whole team before moving to the next, so that when you score documentation, March is less likely to drag the number along with it. And write the evidence before the number. A score with no example under it is the likeliest to have been borrowed.
A judge you can rewind
AI models now grade other AI models, picking the better of two chatbot answers. A 2023 study led by UC Berkeley researchers tested how well that works. In one example, GPT-4 compared two answers about business etiquette in Japan and chose the first as “more detailed and organized.” Swapped, GPT-4 chose the other one, now calling it “more detailed.” On pairs of similar answers, it gave the same verdict in both orders 65% of the time. Length tilted the judges too: answers padded with reworded copies of their own list items won over GPT-3.5 and Claude-v1 in 21 of 23 tries, and GPT-4 in 2.
Part of this carries over. In both orders the judge wrote a fluent, specific reason, and the reason never mentioned what had tipped it. The students did the same.
The cause doesn’t. Nobody here is warm or cold; the authors suspect the tilt comes from training data or from the way these models read text left to right, and leave it open. And a model can be asked twice. No student can watch the tape again without remembering the first viewing. The researchers’ fix asks the judge twice with the order swapped and counts a win only when the same answer wins both times.
The students got one viewing. The machine can be shown the tape again, turned around.