In 1999 Amazon was three or four years old and, as Bill Carr remembers it, “growing like a weed.” In the tech organization, a newly hired engineering leader would hire senior managers, who would hire managers. “All these people have been here for a week,” Carr said on Lenny’s Podcast in 2023. They hired on their own judgment and their last company’s criteria. One senior leader summed it up: “We had new people hiring new people hiring new people.”
Jeff Bezos had told shareholders that setting the bar high in hiring was “the single most important element of Amazon.com’s success.” The people setting it had barely arrived.
Amazon’s fix was a stranger in the room. On every loop, one interviewer came from outside the hiring team: not the hiring manager, not someone who reported to the hiring manager, not the recruiter. That person ran the debrief. And, in Brad Stone’s account of the program’s early days, that person could veto the hire, and “even the hiring manager was unable to override” it.
Give a say in every hire to someone who won’t have to live with the empty seat.
Who pays for the empty desk
The hiring manager usually wants the hire most. The work piles up on their team today; a weak hire’s costs, if they come, arrive later. Carr named the result “the urgency bias that every hiring manager has”: the pull to fill the role with “the next warm body.” Rick Dalzell, who helped launch the program, told Stone: “Many companies as they grow begin to compromise their standards in order to fill their resource needs.”
The Bar Raiser doesn’t share that pressure. In the words of Annie Groeninger, a software development manager quoted by the company, Bar Raisers “will not be involved in the day-to-day interaction with the interviewee, so we’re completely focused on making hiring decisions for Amazon, not for a specific team or role.” The program’s stated target is that each hire be better than 50 percent of the people already in similar roles.
The estate agent is paid when the house sells. The surveyor is paid either way. Before you sign, it’s the surveyor’s report you want.
Carr adds a twist. The veto was real, but “a good Bar Raiser never uses” it; in 15 years at Amazon, he said, he never used it and never saw it used. The decision stayed with the hiring manager. What the Bar Raiser did was run the debrief and make sure people stuck to the agreed criteria. Once Amazon had written its Leadership Principles, those became the criteria, and behavior-based interviewing the method, close kin to the structured interview. Without an outsider, a debrief can slide back into the gut impressions that unstructured interviews run on. It adds a check on judgment, not a test of the work itself, which is what a work sample is for.
What it costs
Carr called the job “a huge time suck,” up to 10 hours of his week. Bar Raisers volunteer on top of their own work, unpaid for it, Fortune reported; Amazon says training takes three months to a year.
Hiring managers pay too. Carr said they could feel the whole loop was “a bureaucratic process and a group of people that I have to sell,” standing between them and the person they wanted. A recruiter told Fortune an outsider “may not understand the team culture they are looking to hire into.”
Amazon says it is “focused on hiring well, rather than hiring quickly.” Even so, during its pandemic hiring boom, Amazon dropped bar raisers from interviews for entry-level software engineers. In 2024, according to an internal memo reported by Business Insider, it brought them back to “enhance the quality” of those hires. Amazon declined to tell Fortune whether dropping them had made hires worse.
Borrowing a stranger
Say you run a twenty-person company and need to hire three engineers this quarter. Your engineering lead is swamped and likes the first candidate who can do the job.
You could ask a manager from sales or operations to sit on every loop, take written notes, chair the debrief and ask the question Groeninger favors: “What does Amazon miss out on if we don’t hire this person?” with your company’s name in it. The price may be a few weeks’ delay and an irritated engineering lead; the hoped-for gain, a hire who clears the bar instead of a seat that’s merely full.
A grader that didn’t write the answer
A version of this turns up when an AI model checks its own output. The incentive differs, since a model has no seat to fill, but the risk of grading your own work remains.
In a 2024 study, Arjun Panickssery, Samuel Bowman and Shi Feng had GPT-4, GPT-3.5 and Llama 2 write news summaries and then rate summaries from themselves, the other models and humans. The models tended to favor their own. Out of the box, GPT-4 could tell its own summaries from the others’ 73.5% of the time, and when the researchers fine-tuned GPT-3.5 and Llama 2 to recognize themselves better, their self-preference rose in step. The authors are careful about one gap: they did not check whether the models’ own summaries were actually better, which would make some of that preference justified.
Anthropic’s documentation for building evaluations reaches for the outsider. A comment in its sample code for model-graded tests reads: “Generally best practice to evaluate with a different model from the one used to generate the evaluated output.”
A different model is a Bar Raiser without a stake in the answer. It is not a neutral judge. In the same study, all three models sometimes flipped their verdict when the two summaries swapped places, GPT-4 about a quarter of the time (more on judges’ tilts under the halo effect). An outside grader avoids one risk, a model grading its own work, but it can still bring preferences of its own and a bias toward one position in the order.