Is It Discriminatory to Use Artificial Intelligence in the Hiring Process?
Data Leakage and the Risks of Learned Discrimination in Artificial Intelligence
A recent consent decree[1] secured by the U.S. Equal Employment Opportunity Commission (EEOC) against digital language tutoring company iTutorGroup demonstrates the inherent risks created by an over-reliance on artificial intelligence (AI), even for seemingly innocuous purposes such as preliminary resume screening in the hiring context. The EEOC’s investigation centered around the company’s use of an artificial intelligence algorithm to evaluate job applications from prospective tutors. While that may seem like a way to approach the hiring process in a merits-only manner, the EEOC discovered that in practice, iTutorGroup’s algorithm had automatically rejected applicants over a certain age threshold, constituting unlawful discrimination under the Age Discrimination in Employment Act (ADEA). Though legal practitioners have utilized artificial intelligence to some degree for years now, the proliferation of programs such as ChatGPT and DALL-E 2 have provided individual users with unprecedented access. By removing historical barriers to entry, this trend stands to provide a boon to businesses throughout the country, but adopters should be careful to note that even well-designed algorithms can drift toward ineffective, circular, or even discriminatory reasoning over time.
Although the iTutorGroup consent decree is the first major discrimination action of its kind, the problem is not unique. Most AI applications utilize a black box model in which the program is fed a large number of sample data points which are then used to derive generalized rules that can be used to predict those same qualities in future data sets. However, owing to the complexity of those rules, the algorithm ultimately utilized by the program is so complex that human actors typically cannot reverse engineer a given output to determine why the AI provided a given response. In a way, this is the purpose of AI, as it allows interpretation of data sets that would overwhelm a human observer. The risk posed by that approach, however, is that without the underlying rationale, human actors are unable to determine whether the AI algorithm is considering the correct factors when weighing a decision or simply taking shortcuts that sometimes – and perhaps coincidentally – arrive at a “correct” answer. Moreover, the black box nature of those decisions means that a party using that technology will be unable to offer an alternative explanation for the decision, as is often required in litigation to provide a non-discriminatory justification for an employment action under the McDonnell-Douglas[2] burden-shifting analysis.
The black-box nature of AI is particularly worrisome given its tendency to develop data leakage, whereby an algorithm develops circular reasoning by utilizing (or creating) input parameters that actually contain the solution in some hidden or unintuitive way. Since tautologies are always true, that reasoning provides high accuracy in the short term, but greatly diminishes the program’s predictive power over time. Perhaps the most well-known example of this occurred in a chest x-ray interpretation software meant to detect pneumonia. Although the program was initially successful, accuracy began to deteriorate over time. Eventually it was discovered that the program most likely made the reasonable conclusion that respiratory infections tend to be more prominent in poorer parts of the world. The problem, however, was that the program then took an additional step, concluding that the poorer parts of the world tend to utilize older imaging equipment. Thereafter, the program began to use identity of the underlying imaging machine as the most important variable for diagnosing chest scans[3] – essentially the machine learned to stereotype. Like human stereotyping, this not only corrupted the long-term reliability of the program, it also tainted the judgment of human radiologists who relied on that technology[4].
Owing to the infancy of AI technology, regulatory action is often jurisdiction-specific. For example, New York City’s recent passage of Local Law 144 of 2021, requires employers that use AI at any point in the hiring process to conduct a bias audit. Other jurisdictions are likely to adopt their own regulations on this emerging technology. Given this bottom-up regulatory approach, businesses wishing to utilize AI technology should take careful stock of the risks and regulatory requirements of that decision.
[1] https://www.eeoc.gov/newsroom/itutorgroup-pay-365000-settle-eeoc-discriminatory-hiring-suit
[2] McDonnell Douglas Corp. v. Green, 411 U.S. 792, 805-807 (1973).
[3] https://www.mountsinai.org/about/newsroom/2018/artificial-intelligence-may-fall-short-when-analyzing-data-across-multiple-health-systems
[4] https://pubmed.ncbi.nlm.nih.gov/37266657/