One of the classic examples of how AI systems can reinforce social injustice is Amazon’s A.I. hiring tool. In 2014, Amazon built an ´A.I. powered´ tool to assess resumes and recommend the top candidates that would go on to be interviewed. However, the tool turned out to be very biased, systematically preferring men over women.
The cause for this sexism is that the tool consisted of a machine learning model trained by feeding it the resumes of successful applicants of the past 10 years, who were overwhelmingly male. As a result, the tool would reject candidates it would identify as female, as women do not fit the model of the historically succesful male candidates. It did so by filtering out resumes that listed womens-colleges or that included the word ‘women’ in some way, such as ‘women football team’. In 2017, Amazon discontinued the tool due to these persistent sexist outcomes.
This example shows the danger of relying on A.I., or rather, machine learning tools. These systems are trained on historical data which reflect social inequality and historical injustices and are therefore prone to replicate these patterns. As phrased by Rachel Goodman from ACLU in response to the controversy around Amazon’s hiring tool:
Algorithms will replicate and even magnify the biases that exist in society at large — unless they are designed and monitored very, very carefully. The right kind of oversight is required to make sure that happens.
As such, automated hiring tools do not only pose a risk to female applicants but similarly to racialised people or people from other (historically) oppressed groups. With the use of these type of tools in the context of hiring on the rise, more research is coming out warning against the dangers and offering practical guidance on how to avoid (to the exent possible) these unjust outcomes. Two of such reports are the 2021 ‘Elephant in AI’ report from Penn state and the recent HvA report ‘Als de machine kiest’, co-authored by our own Hans de Zwart.
See: Amazon Created a Hiring Tool Using A.I. It Immediately Started Discriminating Against Women at Slate magazine.
Photo by Joe Mabel, published under a Creative Commons Attribution-Share Alike 4.0 International license.