Twenty minutes, one question: what grade point average will this student earn this semester? At Carnegie Mellon, 76 undergraduates set out to answer it by interviewing a fellow student, with a researcher in the room to keep time.
Some were allowed only yes-or-no or this-or-that questions. For some of those, after a break at the ten-minute mark, the student across the table stopped telling the truth. Each answer now depended on the first letters of the question’s last two words: same half of the alphabet, yes; different halves, no.
The interviews carried on as if nothing had changed. In a 2017 essay, Jason Dana, the study’s first author, wrote that not one interviewer reported noticing. Asked afterward whether they could “infer a lot about this person” from their time together, those who had heard random answers agreed a shade more than those who had heard the truth: 2.85 against 2.72 on a four-point scale, a gap too small to be significant.
Then their predictions were checked against the grades.
The number on the table
Every interviewer had been handed one more clue before predicting: the student’s GPA from earlier semesters, along with the advice that at this university it was, by itself, the best statistical predictor.
Predictions made after an interview correlated .31 with the grades students actually got. The earlier GPA alone correlated .65. Sixty-eight of the same participants also predicted grades for a second student they never met, from background facts and earlier GPA only, and those predictions correlated .61. In this study, the conversation made them worse. The researchers found no sign that the random group predicted worse than the others: that group’s predictions were the only ones that correlated significantly with the grades, though the three interview groups did not differ significantly from each other.
Feeling you got to know a candidate is not evidence that you did.
The scope matters. This was students predicting students’ grades, with seven semester GPAs to guess in the first study, and the authors noted that grades may be easier to predict than job performance. In their second study, where students watched recorded interviews, the interviews didn’t help, but didn’t hurt either. For jobs, the broad evidence is gentler but not kind. In a 2022 re-analysis of hiring research, Paul Sackett and colleagues put the validity of unstructured interviews at .19, against .42 for structured ones. That is weak, not zero.
Faces in the cloud
Dana and his co-authors, Robyn Dawes and Nathanial Peterson, offered two explanations. One they called dilution: a heap of weak clues draws weight away from the one strong clue. The other was sensemaking: interviewers can build a coherent story out of almost anything they hear and, the authors suggested, keep asking follow-ups until the answers fit.
Their paper tells of a woman, known to one of the authors, who arrived five minutes early for a panel interview, was called straight in, and got the job on the spot. Afterward one interviewer praised how composed she had been after turning up 25 minutes late. She had been given the wrong time and had no idea she was late. Nobody’s questions tested the impression, and “flippant or arrogant,” the authors noted, would have explained her calm just as well.
A conversation is a cloud, and interviewers are very good at seeing faces in it. Some faces come from a single trait. In a 1977 study, students who saw a lecturer answer coldly mostly found his accent irritating, while about half of those who saw him answer warmly liked it, and most denied that liking him had anything to do with it: the halo effect. Others come from similarity. The sociologist Lauren Rivera interviewed 120 people who hired for elite banks, law firms and consulting firms. More than half said “fit” was the most important thing they judged at the interview stage, ahead of analytical thinking and communication. Fit often meant shared pastimes and a similar style outside work. One banker said he needed someone he’d be comfortable “getting stuck in an airport with.”
Matt Mullenweg reached a similar conclusion at Automattic. By his account, his team had been swayed by how candidates spoke and even how they behaved in a restaurant, so he swapped lunches for paid tryouts on real work.
The strongest thing on your desk
Say you are hiring a site manager for your building firm. One candidate brings a former employer’s reference describing three years of projects handed over on schedule. The other’s papers are thinner, but the interview is easy: same hometown, same football club, an hour that flies by.
The risk isn’t that you’ll ignore the reference. It’s that the hour quietly takes some of its weight. Dana’s interviewers had the strongest clue in their hands and still drifted from it. Writing down your ranking from the paper evidence before the interview, and moving it only for something specific you can name afterward, gives the hour less room to do that.
A candidate you can re-run
Trying out a new AI model by chatting with it is an unstructured interview too. Anthropic’s engineers write that teams building AI agents can get surprisingly far on manual testing, dogfooding and intuition. Then, once the agent is in production, users report it “feels worse” after a change, and the team is “flying blind,” unable to “distinguish real regressions from noise.”
Part of that noise is in the candidate itself. Model outputs vary between runs, the same post says, and a task that passed on one run might fail on the next. So an afternoon of chats is partly a sample of luck, and a fluent answer invites the same story-building Dana’s interviewers did.
Here the analogy turns. A model can be run on the same set of tests again and again, and its pass rate counted. The post shows why counting matters: if an agent succeeds 75% of the time, then, assuming independent trials, the chance it passes three in a row is about 42%.
An afternoon of chatting gives you an impression. A hundred runs give you a number.