No retraction, correction or concern found when last checked on Oct 16, 2026.

Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: A cross-sectional study

John R. Zech, Marcus A. Badgeley, Manway Liu and 3 more

PLOS Medicine · 2018 · CC BY

doi:10.1371/journal.pmed.1002683

Abstract

Background. There is interest in using convolutional neural networks (CNNs) to analyze medical imaging to provide computer-aided diagnosis (CAD). Recent work has suggested that image classification CNNs may not generalize to new data as well as previously believed. We assessed how well CNNs generalized across three hospital systems for a simulated pneumonia screening task.

Methods and findings. A cross-sectional design with multiple model training cohorts was used to evaluate model generalizability to external sites using split-sample validation. A total of 158,323 chest radiographs were drawn from three institutions: National Institutes of Health Clinical Center (NIH; 112,120 from 30,805 patients), Mount Sinai Hospital (MSH; 42,396 from 12,904 patients), and Indiana University Network for Patient Care (IU; 3,807 from 3,683 patients). These patient populations had an age mean (SD) of 46.9 years (16.6), 63.2 years (16.5), and 49.6 years (17) with a female percentage of 43.5%, 44.8%, and 57.3%, respectively. We assessed individual models using the area under the receiver operating characteristic curve (AUC) for radiographic findings consistent with pneumonia and compared performance on different test sets with DeLong’s test. The prevalence of pneumonia was high enough at MSH (34.2%) relative to NIH and IU (1.2% and 1.0%) that merely sorting by hospital system achieved an AUC of 0.861 (95% CI 0.855–0.866) on the joint MSH–NIH dataset. Models trained on data from either NIH or MSH had equivalent performance on IU (P values 0.580 and 0.273, respectively) and inferior performance on data from each other relative to an internal test set (i.e., new data from within the hospital system used for training data; P values both <0.001). The highest internal performance was achieved by combining training and test data from MSH and NIH (AUC 0.931, 95% CI 0.927–0.936), but this model demonstrated significantly lower external performance at IU (AUC 0.815, 95% CI 0.745–0.885, P = 0.001). To test the effect of pooling data from sites with disparate pneumonia prevalence, we used stratified subsampling to generate MSH–NIH cohorts that only differed in disease prevalence between training data sites. When both training data sites had the same pneumonia prevalence, the model performed consistently on external IU data (P = 0.88). When a 10-fold difference in pneumonia rate was introduced between sites, internal test performance improved compared to the balanced model (10× MSH risk P < 0.001; 10× NIH P = 0.002), but this outperformance failed to generalize to IU (MSH 10× P < 0.001; NIH 10× P = 0.027). CNNs were able to directly detect hospital system of a radiograph for 99.95% NIH (22,050/22,062) and 99.98% MSH (8,386/8,388) radiographs. The primary limitation of our approach and the available public data is that we cannot fully assess what other factors might be contributing to hospital system–specific biases.

Conclusion. Pneumonia-screening CNNs achieved better internal than external performance in 3 out of 5 natural comparisons. When models were trained on pooled data from sites with different pneumonia prevalence, they performed better on new pooled data from these sites but not on external data. CNNs robustly identified hospital system and department within a hospital, which can have large differences in disease burden and may confound predictions.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 29

1.
Wang X, Peng Y, Lu L, Lu Z, Bagheri M, Summers RM. ChestX-ray8: Hospital-scale Chest X-ray Database and Benchmarks on Weakly-Supervised Classification and Localization of Common Thorax Diseases; 2017 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1705.02315.
Not in our library
2.
Rajpurkar P, Irvin J, Zhu K, Yang B, Mehta H, Duan T, et al CheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning; 2017 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1711.05225.
Not in our library
3.
Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al Development and Validation of a Deep Learning Algorithm for Detection of Diabetic Retinopathy in Retinal Fundus Photographs. JAMA. 2016;316: 2402–2410.
Not in our library10.1001/jama.2016.17216
4.
Ting DSW, Cheung CY-L, Lim G, Tan GSW, Quang ND, Gan A, et al Development and Validation of a Deep Learning System for Diabetic Retinopathy and Related Eye Diseases Using Retinal Images From Multiethnic Populations With Diabetes. JAMA. 2017;318: 2211–2223.
Not in our library10.1001/jama.2017.18152
5.
Kermany DS, Goldbaum M, Cai W, Valentim CCS, Liang H, Baxter SL, et al Identifying Medical Diagnoses and Treatable Diseases by Image-Based Deep Learning. Cell. Elsevier; 2018;172: 1122–1131.
Not in our library10.1016/j.cell.2018.02.010
6.
Russakovsky O, Deng J, Su H, Krause J, Satheesh S, Ma S, et al ImageNet Large Scale Visual Recognition Challenge; 2014 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1409.0575.
Not in our library
7.
Lecun Y, Bottou L, Bengio Y, Haffner P. Gradient-based learning applied to document recognition In Intelligent signal processing. IEEE Press 2001 p. 306–351.
Not in our library
8.
Recht B, Roelofs R, Schmidt L, Shankar V. Do CIFAR-10 Classifiers Generalize to CIFAR-10?; 2018 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1806.00451.
Not in our library
9.
Rothwell PM. External validity of randomised controlled trials: “To whom do the results of this trial apply?” Lancet. 2005;365: 82–93.
Not in our library10.1016/s0140-6736(04)17670-8
10.
Pandis N, Chung B, Scherer RW, Elbourne D, Altman DG. CONSORT 2010 statement: extension checklist for reporting within person randomised trials. BMJ. 2017;357: j2835
Not in our library10.1136/bmj.j2835
11.
Cabitza F, Rasoini R, Gensini GF. Unintended Consequences of Machine Learning in Medicine. JAMA. 2017;318: 517–518.
Not in our library10.1001/jama.2017.7797
12.
Demner-Fushman D, Kohli MD, Rosenman MB, Shooshan SE, Rodriguez L, Antani S, et al Preparing a collection of radiology examinations for distribution and retrieval. J Am Med Inform Assoc. 2016;23: 304–310.
Not in our library10.1093/jamia/ocv080
13.
Deep learning · 2015
No notice
14.
He K, Zhang X, Ren S, Sun J. Deep Residual Learning for Image Recognition; 2015 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1512.03385.
Not in our library
15.
Huang G, Liu Z, Weinberger KQ, van der Maaten L. Densely Connected Convolutional Networks; 2016 [cited 1 July 2018]. Preprint. Available from: arXiv: https://arxiv.org/abs/1608.06993.
Not in our library
16.
Zhang C, Bengio S, Hardt M, Recht B, Vinyals O. Understanding deep learning requires rethinking generalization; 2016 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1611.03530.
Not in our library
17.
Zech J, Pain M, Titano J, Badgeley M, Schefflein J, Su A, et al Natural Language-based Machine Learning Models for the Annotation of Clinical Radiology Reports. Radiology. 2018;287(2): 570–580.
Not in our library10.1148/radiol.2018171093
18.
Paszke A, Gross S, Chintala S, Chanan G, Yang E, DeVito Z, et al Automatic differentiation in PyTorch; 2017 [cited 1 July 2018]. Preprint. Available from: https://openreview.net/forum?id=BJJsrmfCZ.
Not in our library
19.
Zhou B, Khosla A, Lapedriza A, Oliva A, Torralba A. Learning Deep Features for Discriminative Localization; 2015 Preprint. Available from: https://arxiv.org/abs/1512.04150.
Not in our library
20.
DeLong ER, DeLong DM, Clarke-Pearson DL. Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics. 1988;44: 837–845.
Not in our library
21.
pROC: an open-source package for R and S+ to analyze and compare ROC curves · 2011
No notice
22.
Pedregosa F, Varoquaux G, Gramfort A, Michel V, Thirion B, Grisel O, et al Scikit-learn: Machine Learning in Python. J Mach Learn Res. 2011;12: 2825–2830.
Not in our library
23.
Collins GS; Reitsma JB, Altman DG, Moons KGM. Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD): The TRIPOD Statement. Ann Intern Med 2015; 162:55–63.
Not in our library10.7326/m14-0697
24.
Steyerberg EW and Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. European Heart Journal 2014; 35:1925–1931.
Not in our library10.1093/eurheartj/ehu207
25.
Gale W, Oakden-Rayner L, Carneiro G, Bradley AP, Palmer LJ. Detecting hip fractures with radiologist-level performance using deep neural networks; 2017 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1711.06504.
Not in our library
26.
Mutasa S, Chang PD, Ruzal-Shapiro C, Ayyala R. MABAL: a Novel Deep-Learning Architecture for Machine-Assisted Bone Age Labeling. J Digit Imaging. 2018;31(4): 513–519.
Not in our library10.1007/s10278-018-0053-3
27.
Geras KJ, Wolfson S, Gene Kim S, Moy L, Cho K. High-Resolution Breast Cancer Screening with Multi-View Deep Convolutional Neural Networks; 2017 [cited 1 July 2018]. Preprint. Available from: https://arxiv.org/abs/1703.07047.
Not in our library
28.
Oakden-Rayner L. Exploring the ChestXray14 dataset: problems. 18 12 2017 [cited 26 Jan 2018]. In: Luke Oakden-Rayner [Internet]. Available from: https://lukeoakdenrayner.wordpress.com/2017/12/18/the-chestxray14-dataset-problems/.
Not in our library
29.
Mandell LA, Wunderink RG, Anzueto A, Bartlett JG, Campbell GD, Dean NC, et al Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community-acquired pneumonia in adults. Clin Infect Dis. 2007;44 Suppl 2: S27–72.
Not in our library10.1086/511159

Cited by papers we hold 28

The limits of fair medical imaging AI in real-world generalization · 2024
No notice
Self-supervised learning for medical image classification: a systematic review and implementation guidelines · 2023
No notice
Leakage and the reproducibility crisis in machine-learning-based science · 2023
No notice
AI pitfalls and what not to do: mitigating bias in AI · 2023
No notice
Artificial intelligence in diagnostic pathology · 2023
No notice
Fairness of artificial intelligence in healthcare: review and recommendations · 2023
No notice
Limited generalizability of single deep neural network for surgical instrument segmentation in different surgical environments · 2022
Not checked
Federated learning enables big data for rare cancer boundary detection · 2022
Corrected
Sources of bias in artificial intelligence that perpetuate healthcare disparities—A global review · 2022
No notice
Randomized Clinical Trials of Machine Learning Interventions in Health Care · 2022
No notice
Machine learning for medical imaging: methodological failures and recommendations for the future · 2022
No notice
Critical appraisal of a machine learning paper: A guide for the neurologist · 2021
Not checked
Second opinion needed: communicating uncertainty in medical machine learning · 2021
No notice
Diagnostic accuracy of deep learning in medical imaging: a systematic review and meta-analysis · 2021
No notice
Classification of optical coherence tomography images using a capsule network · 2020
Not checked
Extensions of the External Validation for Checking Learned Model Interpretability and Generalizability · 2020
Not checked
An Overview of Deep Learning Approaches in Chest Radiograph · 2020
Not checked
Predicting COVID-19 Pneumonia Severity on Chest X-ray With Deep Learning · 2020
No notice
Reporting guidelines for clinical trial reports for interventions involving artificial intelligence: the CONSORT-AI extension · 2020
Not checked
Automatic lung segmentation in routine imaging is primarily a data diversity problem, not a methodology problem · 2020
No notice

and 8 more

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.