No retraction, correction or concern found when last checked on Oct 17, 2026.

AI recognition of patient race in medical imaging: a modelling study

Judy Wawira Gichoya, Imon Banerjee, Ananth Reddy Bhimireddy and 20 more

The Lancet Digital Health · 2022 · CC BY

Abstract

Background. Previous studies in medical imaging have shown disparate abilities of artificial intelligence (AI) to detect a person’s race, yet there is no known correlation for race on medical imaging that would be obvious to human experts when interpreting the images. We aimed to conduct a comprehensive evaluation of the ability of AI to recognise a patient’s racial identity from medical images.

Methods. Using private (Emory CXR, Emory Chest CT, Emory Cervical Spine, and Emory Mammogram) and public (MIMIC-CXR, CheXpert, National Lung Cancer Screening Trial, RSNA Pulmonary Embolism CT, and Digital Hand Atlas) datasets, we evaluated, first, performance quantification of deep learning models in detecting race from medical images, including the ability of these models to generalise to external environments and across multiple imaging modalities. Second, we assessed possible confounding of anatomic and phenotypic population features by assessing the ability of these hypothesised confounders to detect race in isolation using regression models, and by re-evaluating the deep learning models by testing them on datasets stratified by these hypothesised confounding variables. Last, by exploring the effect of image corruptions on model performance, we investigated the underlying mechanism by which AI models can recognise race.

Findings. In our study, we show that standard AI deep learning models can be trained to predict race from medical images with high performance across multiple imaging modalities, which was sustained under external validation conditions (x-ray imaging [area under the receiver operating characteristics curve (AUC) range 0·91–0·99], CT chest imaging [0·87–0·96], and mammography [0·81]). We also showed that this detection is not due to proxies or imaging-related surrogate covariates for race (eg, performance of possible confounders: body-mass index [AUC 0·55], disease distribution [0·61], and breast density [0·61]). Finally, we provide evidence to show that the ability of AI deep learning models persisted over all anatomical regions and frequency spectrums of the images, suggesting the efforts to control this behaviour when it is undesirable will be challenging and demand further study.

Interpretation. The results from our study emphasise that the ability of AI deep learning models to predict self-reported race is itself not the issue of importance. However, our finding that AI can accurately predict self-reported race, even from corrupted, cropped, and noised medical images, often when clinical experts cannot, creates an enormous risk for all model deployments in medical imaging.

Funding. National Institute of Biomedical Imaging and Bioengineering, MIDRC grant of National Institutes of Health, US National Science Foundation, National Library of Medicine of the National Institutes of Health, and Taiwan Ministry of Science and Technology.

References 39

1.
Bender EM, Gebru T, McMillan-Major A, Shmitchell S. On the dangers of stochastic parrots: can language models be too big? FAccT ‘21; March 3–10, 2021..
Not in our library10.1145/3442188.3445922
2.
Angwin J, Larson J, Mattu S, Kirchner L. Machine bias. https://www.propublica.org/article/machine-bias-risk-assessments-in-criminal-sentencing (accessed april 25, 2022).
Not in our library
3.
Koenecke A, Nam A, Lake E, Racial disparities in automated speech recognition. Proc Natl Acad Sci USA 2020; 117: 7684–89.
Not in our library10.1073/pnas.1915768117
4.
Buolamwini J, Gebru T. Gender shades: intersectional accuracy disparities in commercial gender classification. PMLR 2018; 81: 77–91.
Not in our library
5.
Adamson AS, Smith A. Machine learning and health care disparities in dermatology. JAMA Dermatol 2018; 154: 1247–48.
Not in our library10.1001/jamadermatol.2018.2348
6.
Navarrete-Dechent C, Dusza SW, Liopyris K, Marghoob AA, Halpern AC, Marchetti MA. Automated dermatological diagnosis: hype or reality? J Invest Dermatol 2018; 138: 2277–79.
Not in our library10.1016/j.jid.2018.04.040
7.
Sarkar R, Martin C, Mattie H, Gichoya JW, Stone DJ, Celi LA. Performance of intensive care unit severity scoring systems across different ethnicities in the USA: a retrospective observational study. Lancet Digit Health 2021; 3: e241–49.
Not in our library10.1016/s2589-7500(21)00022-4
8.
Obermeyer Z, Powers B, Vogeli C, Mullainathan S. Dissecting racial bias in an algorithm used to manage the health of populations. Science 2019; 366: 447–53.
Not in our library10.1126/science.aax2342
9.
Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations · 2021
No notice
10.
Seyyed-Kalantari L, Liu G, McDermott M, Chen IY, Ghassemi M. CheXclusion: fairness gaps in deep chest X-ray classifiers. arXiv 2020; published online Oct 16. (preprint).
Not in our library10.48550/arxiv.2003.00827
11.
Yi PH, Wei J, Kim TK, Radiology “forensics”: determination of age and sex from chest radiographs using deep learning. Emerg Radiol 2021; 28: 949–54.
Not in our library10.1007/s10140-021-01953-y
12.
Eng DK, Khandwala NB, Long J, Artificial intelligence algorithm improves radiologist performance in skeletal age assessment: a prospective multicenter randomized controlled trial. Radiology 2021; 301: 692–99.
Not in our library10.1148/radiol.2021204021
13.
Prediction of systemic biomarkers from retinal photographs: development and validation of deep-learning algorithms · 2020
Not checked
14.
Assessment of patient specific information in the wild on fundus photography and optical coherence tomography · 2021
Not checked
15.
Poplin R, Varadarajan AV, Blumer K, Prediction of cardiovascular risk factors from retinal fundus photographs via deep learning. Nat Biomed Eng 2018; 2: 158–64.
Not in our library10.1038/s41551-018-0195-0
16.
Greenland S, Robins JM. Identifiability, exchangeability, and epidemiological confounding. Int J Epidemiol 1986; 15: 413–19.
Not in our library10.1093/ije/15.3.413
17.
Greenland S, Pearl J, Robins JM. Confounding and collapsibility in causal inference. SSO Schweiz Monatsschr Zahnheilkd 1999; 14: 29–46.
Not in our library
18.
Wawira Gichoya J, McCoy LG, Celi LA, Ghassemi M. Equity in essence: a call for operationalising fairness in machine learning for healthcare. BMJ Health Care Inform 2021; 28: e100289.
Not in our library10.1136/bmjhci-2020-100289
19.
Tariq A, Purkayastha S, Padmanaban GP, Current clinical applications of artificial intelligence in radiology and their best supporting evidence. J Am Coll Radiol 2020; 17: 1371–81.
Not in our library10.1016/j.jacr.2020.08.018
20.
FDA cleared AI algorithms. https://report.acr.org/t/PUBLIC/views/CascadeReport/Commercial (accessed May 3, 2022).
Not in our library
21.
The state of artificial intelligence-based FDA-approved medical devices and algorithms: an online database · 2020
No notice
22.
Tadavarthi Y, Vey B, Krupinski E, The State of radiology AI: considerations for purchase decisions and current market offerings. Radiol Artif Intell 2020; 2: e200004.
Not in our library10.1148/ryai.2020200004
23.
Krieger N Shades of difference: theoretical underpinnings of the medical controversy on black/white differences in the United States, 1830–1870. Int J Health Serv 1987; 17: 259–78.
Not in our library10.2190/dby6-vdq8-hme8-me3r
24.
Cooper R, David R. The biological concept of race and its application to public health and epidemiology. J Health Polit Policy Law 1986; 11: 97–116.
Not in our library10.1215/03616878-11-1-97
25.
MIMIC-CXR, a de-identified publicly available database of chest radiographs with free-text reports · 2019
No notice
26.
Irvin J, Rajpurkar P, Ko M, CheXpert: a large chest radiograph dataset with uncertainty labels and expert comparison. AAAI 2019; 33: 590–97.
Not in our library
27.
Wagner DR, Heyward VH. Measures of body composition in blacks and whites: a comparative review. Am J Clin Nutr 2000; 71: 1392–402.
Not in our library10.1093/ajcn/71.6.1392
28.
del Carmen MG, Halpern EF, Kopans DB, Mammographic breast density and race. AJR Am J Roentgenol 2007; 188: 1147–50.
Not in our library10.2214/ajr.06.0619
29.
Office of Minority Health. Heart disease and African Americans. Jun 27, 2021. https://minorityhealth.hhs.gov/omh/browse.aspx?lvl=4&lvlid=19 (accessed April 25, 2022).
Not in our library
30.
Graham G Disparities in cardiovascular disease risk in the United States. Curr Cardiol Rev 2015; 11: 238–45.
Not in our library10.2174/1573403x11666141122220003
31.
Ettinger B, Sidney S, Cummings SR, Racial differences in bone density between young adult black and white subjects persist after adjustment for anthropometric, lifestyle, and biochemical differences. J Clin Endocrinol Metab 1997; 82: 429–34.
Not in our library10.1210/jcem.82.2.3732
32.
Hochberg MC. Racial differences in bone strength. Trans Am Clin Climatol Assoc 2007; 118: 305–15.
Not in our library
33.
DeGrave AJ, Janizek JD, Lee S-I. AI for radiographic COVID-19 detection selects shortcuts over signal. Nat Mach Intell 2021; 3: 610–619.
Not in our library
34.
Jabbour S, Fouhey D, Kazerooni E, Sjoding MW, Wiens J. Deep learning applied to chest X-rays: exploiting and preventing shortcuts. PMLR 2020; 126: 750–82.
Not in our library
35.
Ioannidis JPA, Powe NR, Yancy C. Recalibrating the use of race in medical research. JAMA 2021; 325: 623–24.
Not in our library10.1001/jama.2021.0003
36.
Vyas DA, Eisenstein LG, Jones DS. Hidden in plain sight—reconsidering the use of race correction in clinical algorithms. N Engl J Med 2020; 383: 874–82.
Not in our library10.1056/nejmms2004740
37.
Brown S, Davidovic J, Hasan A. The algorithm audit: scoring the algorithms that score us. Big Data Soc 2021; published online Jan 28..
Not in our library10.1177/2053951720983865
38.
Pierson E, Cutler DM, Leskovec J, Mullainathan S, Obermeyer Z. An algorithmic approach to reducing unexplained pain disparities in underserved populations. Nat Med 2021; 27: 136–40.
Not in our library10.1038/s41591-020-01192-7
39.
Underdiagnosis bias of artificial intelligence algorithms applied to chest radiographs in under-served patient populations · 2021
No notice

Cited by papers we hold 4

Tackling algorithmic bias and promoting transparency in health datasets: the STANDING Together consensus recommendations · 2025
No notice
The limits of fair medical imaging AI in real-world generalization · 2024
No notice
The value of standards for health datasets in artificial intelligence-based applications · 2023
No notice
AI pitfalls and what not to do: mitigating bias in AI · 2023
No notice

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.