Inside an experimental Amazon program, the word “women’s” was a black mark. A résumé that said “women’s chess club captain” got marked down for it by the software scoring applicants. It also downgraded graduates of two all-women’s colleges. Nobody had programmed that in. In effect, Reuters reported, the system had taught itself that male candidates were preferable.
The project began in 2014. Amazon was heading into a hiring spree: between June 2015 and 2018 its global headcount would more than triple, to 575,700. A team in its Edinburgh engineering hub, which grew to around a dozen people, set out to automate the search for talent. Their tool gave job candidates from one to five stars, much the way shoppers rate products on Amazon.
“Everyone wanted this holy grail,” one of the people involved later told Reuters. “They literally wanted it to be an engine where I’m going to give you 100 resumes, it will spit out the top five, and we’ll hire those.”
The team built 500 models, each aimed at a particular job function and location, and taught each one to recognize some 50,000 terms that showed up on past candidates’ résumés. The training material was a decade of résumés submitted to Amazon. Most of them came from men, a reflection of male dominance across the tech industry.
The models did what they were trained to do. They found the patterns. Skills that almost every tech applicant listed, such as writing various computer codes, carried little weight. What the models favored instead, one person said, were verbs that turned up more often on male engineers’ résumés, like “executed” and “captured.”
By 2015, Amazon had realized the system was not rating candidates for software developer jobs and other technical posts in a gender-neutral way. Engineers edited the programs to make them neutral to those particular terms. That fixed the words they knew about. According to the people familiar with the effort, nothing guaranteed the machines would not devise other ways of sorting candidates that could prove discriminatory.
Gender was not the only problem. Flaws in the underlying data meant unqualified candidates were often recommended for all manner of jobs. With results coming back almost at random, executives lost hope, and by the start of 2017 the team had been disbanded.
Amazon’s recruiters had looked at the tool’s recommendations when searching for new hires, the people said, but never relied solely on its rankings. Amazon told Reuters the tool “was never used by Amazon recruiters to evaluate candidates,” and did not dispute that recruiters had looked at what it recommended.
Some of the work survived. One person told Reuters that Amazon now used a “much-watered down version” of the engine for rudimentary chores, such as weeding duplicate candidate profiles out of its databases. Another said a new team in Edinburgh had been formed to try automated screening again, this time with a focus on diversity.