The 76 Carnegie Mellon undergraduates had answered a campus advertisement for a paid study. The task: meet another student, talk for 20 minutes, then predict that student’s grade point average for a given semester.
Before they started, each was told the interviewee’s age, major, class standing and course schedule. Ten minutes in, they were offered a break to think up more questions. A researcher stayed in the room to keep time.
Some of them had been given a restriction: ask only yes-or-no or this-or-that questions. Those interviewees answered truthfully. Others had the same restriction, and after the break their interviewee quietly stopped telling the truth. For each question, the student noted the first letters of its last two words. If both fell in the same half of the alphabet, A to M or N to Z, the answer was yes, or the first option. If not, it was no.
The interviewees worried the trick would give itself away. It didn’t. The random interviews, the researchers wrote, “proceeded much as interviews in our other conditions.” In a 2017 New York Times essay, Jason Dana wrote that not one interviewer reported noticing.
After each interview came the strongest clue on offer. The interviewers were handed the student’s cumulative GPA from before that semester and told that, at this university, prior GPA by itself was the best statistical model for predicting grades. Then they made their prediction and rated statements about the conversation.
“I am able to infer a lot about this person given the amount of time we spent together.” On a four-point scale, those who had heard random answers agreed at 2.85, those who heard honest yes-or-no answers at 2.72, and a third group, free to ask anything, at 2.80. None of the differences was significant.
Then the predictions were checked against the grades. Across the interviews, predictions correlated .31 with the GPAs the students actually earned. Prior GPA alone, which every interviewer had been handed, correlated .65. Sixty-eight of the participants also predicted the grades of a second student they never met, from background information and prior GPA alone. Those predictions correlated .61.
The researchers found no sign that the random group predicted worse than the other two. Of the three interview groups, the one fed random answers was the only one whose predictions correlated significantly with the grades (.42, against .20 and .29), though the three did not differ significantly from each other.
With an interview, they had done worse than without one, and far worse than the number in their hands. The authors named two culprits. Dilution: a pile of weak clues drew weight away from the strong one. Sensemaking: interviewers could build a coherent story out of almost anything they heard.
The test was narrow: students predicting other students’ grades, seven semester GPAs in all. The authors noted that grades may be easier to predict than job performance. A second study, in which students watched recorded interviews instead of conducting them, found the interviews didn’t help, but this time didn’t find that they hurt either. Told that half the recordings were random, those viewers judged 19 of the 32 random interviews to be genuine.
In a third study, 169 other Carnegie Mellon students read how the experiment worked, then ranked what they would want for making such a prediction: an honest interview, a free-form one, a random one, or none. 96 of them, 57%, ranked the random interview ahead of no interview at all.