Amazon's Scrapped AI Recruiting Tool

Reuters reported in October 2018 that Amazon had built, and then quietly scrapped, a machine learning tool for ranking job candidates. A team began the work around 2014. The goal was to feed in a stack of résumés and get back the top few candidates to hire. The models were trained on résumés submitted to the company over ten years and learned which patterns tended to go with people who were hired. Most applicants for technical roles were men, so the patterns the model learned were male-coded. By 2015, the team saw that the system was not rating candidates for software developer and other technical jobs in a gender-neutral way. According to Reuters, it penalized résumés containing the word "women's," as in "women's chess club captain," and downgraded graduates of two all-women's colleges. It also favored verbs such as "executed" and "captured," which appeared more often on men's résumés. Engineers edited the models to be neutral to those particular terms. That gave no guarantee the system would not find other ways to sort candidates that were just as discriminatory. The project lost support and the team was disbanded by early 2017. Amazon said the tool was never used by its recruiters to evaluate candidates. Reuters reported that recruiters looked at its recommendations but did not rely on them alone. The key lesson is that the training label was the problem. The model accurately predicted whom Amazon had hired before, and those past hiring decisions contained bias. A common misconception is that the algorithm "became sexist" on its own. In fact it learned exactly what the data rewarded. Another is that removing gender fixes the problem. Text contains many correlated proxies, and a flexible model will find them. The case now comes up in regulatory debates, including New York City's bias audit law for automated hiring tools and the EU AI Act's classification of employment systems as high risk.