No retraction, correction or concern found when last checked on Oct 15, 2026.

Scalable and accurate deep learning with electronic health records

Alvin Rajkomar, Eyal Oren, Kai Chen and 32 more

npj Digital Medicine · 2018 · arXiv licence (no reuse)

Abstract

Predictive modeling with electronic health record (EHR) data is anticipated to drive personalized medicine and improve healthcare quality. Constructing predictive statistical models typically requires extraction of curated predictor variables from normalized EHR data, a labor-intensive process that discards the vast majority of information in each patient’s record. We propose a representation of patients’ entire raw EHR records based on the Fast Healthcare Interoperability Resources (FHIR) format. We demonstrate that deep learning methods using this representation are capable of accurately predicting multiple medical events from multiple centers without site-specific data harmonization. We validated our approach using de-identified EHR data from two US academic medical centers with 216,221 adult patients hospitalized for at least 24 h. In the sequential format we propose, this volume of EHR data unrolled into a total of 46,864,534,945 data points, including clinical notes. Deep learning models achieved high accuracy for tasks such as predicting: in-hospital mortality (area under the receiver operator curve [AUROC] across sites 0.93–0.94), 30-day unplanned readmission (AUROC 0.75–0.76), prolonged length of stay (AUROC 0.85–0.86), and all of a patient’s final discharge diagnoses (frequency-weighted AUROC 0.90). These models outperformed traditional, clinically-used predictive models in all cases. We believe that this approach can be used to create accurate and scalable predictions for a variety of clinical scenarios. In a case study of a particular prediction, we demonstrate that neural networks can be used to identify relevant information from the patient’s chart.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 89

1.
The Digital Universe: Driving Data Growth in Healthcare. Available at: https://www.emc.com/analyst-report/digital-universe-healthcare-vertical-report-ar.pdf (Accessed 23 Feb 2017).
Not in our library
2.
Parikh RB, Schwartz JS, Navathe AS. Beyond genes and molecules - a precision delivery initiative for precision medicine. N. Engl. J. Med. 2017;376:1609-1612.
Not in our library10.1056/nejmp1613224
3.
Parikh RB, Kakad M, Bates DW. Integrating predictive analytics into high-value care: the dawn of precision delivery. JAMA. 2016;315:651-652.
Not in our library10.1001/jama.2015.19417
4.
Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Aff. 2014;33:1123-1131.
Not in our library10.1377/hlthaff.2014.0041
5.
Krumholz HM. Big data and new knowledge in medicine: the thinking, training, and tools needed for a learning health system. Health Aff. 2014;33:1163-1170.
Not in our library10.1377/hlthaff.2014.0053
6.
Jameson JL, Longo DL. Precision medicine--personalized, problematic, and promising. N. Engl. J. Med. 2015;372:2229-2234.
Not in our library10.1056/nejmsb1503104
7.
Goldstein BA, Navar AM, Pencina MJ, Ioannidis JPA. Opportunities and challenges in developing risk prediction models with electronic health records data: a systematic review. J. Am. Med. Inform. Assoc. 2017;24:198-208.
Not in our library10.1093/jamia/ocw042
8.
Press, G. Cleaning big data: most time-consuming, least enjoyable data science task, survey says. Forbes (2016). Available at: https://www.forbes.com/sites/gilpress/2016/03/23/data-preparation-most-time-consuming-least-enjoyable-data-science-task-survey-says/ (Accessed 22 Oct 2017).
Not in our library
9.
Lohr, S. For Big-Data Scientists, ‘Janitor Work’ Is Key Hurdle to Insights. (NY Times, 2014).
Not in our library
10.
Insights into the Problem of Alarm Fatigue with Physiologic Monitor Devices: A Comprehensive Observational Study of Consecutive Intensive Care Unit Patients · 2014
No notice
11.
Chopra V, McMahon LF. Redesigning hospital alarms for patient safety: alarmed and potentially dangerous. JAMA. 2014;311:1199-1200.
Not in our library10.1001/jama.2014.710
12.
Kaukonen KM, Bailey M, Pilcher D, Cooper DJ, Bellomo R. Systemic inflammatory response syndrome criteria in defining severe sepsis. N. Engl. J. Med. 2015;372:1629-1638.
Not in our library10.1056/nejmoa1415236
13.
Deep learning · 2015
No notice
14.
Frome, A. et al. DeViSE: a deep visual-semantic embedding model. In Advances in Neural Information Processing Systems 26 (eds Burges, C. J. C., Bottou, L., Welling, M., Ghahramani, Z. & Weinberger, K. Q.), pp 2121–2129 (Curran Associates, Inc. Red Hook, NY, 2013).
Not in our library
15.
Gulshan V, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA. 2016;316:2402-2410.
Not in our library10.1001/jama.2016.17216
16.
Wu, Y. et al. Google’s neural machine translation system: bridging the gap between human and machine translation. arXiv [cs.CL] (2016).
Not in our library
17.
Dai, A. M. & Le, Q. V. Semi-supervised sequence learning. In Advances in Neural Information Processing Systems 28 (eds Cortes, C., Lawrence, N. D., Lee, D. D., Sugiyama, M. & Garnett, R.), pp 3079–3087 (Curran Associates, Inc. Red Hook, NY, 2015).
Not in our library
18.
Bengio Y, Courville A, Vincent P. Representation learning: a review and new perspectives. IEEE. Trans. Pattern Anal. Mach. Intell. 2013;35:1798-1828.
Not in our library10.1109/tpami.2013.50
19.
Weed LL. Medical records that guide and teach. N. Engl. J. Med. 1968;278:652-657.
Not in our library10.1056/nejm196803212781204
20.
Adler-Milstein J, et al. Electronic health record adoption In US hospitals: progress continues, but challenges persist. Health Aff. 2015;34:2174-2180.
Not in our library10.1377/hlthaff.2015.0992
21.
Mandell LA, et al. Infectious Diseases Society of America/American Thoracic Society consensus guidelines on the management of community-acquired pneumonia in adults. Clin. Infect. Dis. 2007;44:S27-S72.
Not in our library10.1086/511159
22.
Lim WS, Smith DL, Wise MP, Welham SA. British Thoracic Society community acquired pneumonia guideline and the NICE pneumonia guideline: how they fit together. BMJ Open Respir. Res. 2015;2:e000091.
Not in our library10.1136/bmjresp-2015-000091
23.
Churpek MM, et al. Multicenter comparison of machine learning methods and conventional regression for predicting clinical deterioration on the wards. Crit. Care. Med. 2016;44:368-374.
Not in our library10.1097/ccm.0000000000001571
24.
Howell MD, et al. Sustained effectiveness of a primary-team-based rapid response system. Crit. Care. Med. 2012;40:2562-2568.
Not in our library10.1097/ccm.0b013e318259007b
25.
Sun H, et al. Semantic processing of EHR data for clinical research. J. Biomed. Inform. 2015;58:247-259.
Not in our library10.1016/j.jbi.2015.10.009
26.
Newton KM, et al. Validation of electronic medical record-based phenotyping algorithms: results and lessons learned from the eMERGE network. J. Am. Med. Inform. Assoc. 2013;20:e147-e154.
Not in our library10.1136/amiajnl-2012-000896
27.
OHDSI. OMOP common data model. Observational health data sciences and informatics. Available at: https://www.ohdsi.org/data-standardization/the-common-data-model/ (Accessed 23 Jan 2018).
Not in our library
28.
Mandel JC, Kreda DA, Mandl KD, Kohane IS, Ramoni RB. SMART on FHIR: a standards-based, interoperable apps platform for electronic health records. J. Am. Med. Inform. Assoc. 2016;23:899-908.
Not in our library10.1093/jamia/ocv189
29.
Deep Patient: An Unsupervised Representation to Predict the Future of Patients from the Electronic Health Records · 2016
No notice
30.
Lipton, Z. C., Kale, D. C., Elkan, C. & Wetzel, R. Learning to diagnose with LSTM recurrent neural networks. arXiv [cs.LG] (2015).
Not in our library
31.
Aczon, M. et al. Dynamic mortality risk predictions in pediatric critical care using recurrent neural networks. arXiv [stat.ML] (2017).
Not in our library
32.
Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F. & Sun, J. Doctor AI: predicting clinical events via recurrent neural networks. In Proceedings of the 1st Machine Learning for Healthcare Conference, vol 56 (eds F. Doshi-Velez, J. Fackler, D. Kale and B. Wallace, J. Wiens) 301–318 (PMLR, Los Angeles, CA, 2016).
Not in our library
33.
Suresh, H. et al. Clinical intervention prediction and understanding using deep networks. arXiv [cs.LG] (PMLR, Los Angeles, CA, USA, 2017).
Not in our library
34.
Razavian, N., Marcus, J. & Sontag, D. Multi-task prediction of disease onsets from longitudinal laboratory tests. In Proceedings of the 1st Machine Learning for Healthcare Conference, (eds F. Doshi-Velez, J. Fackler, D. Kale and B. Wallace, J. Wiens) Vol. 56, pp 73–100 (PMLR, Los Angeles, CA, 2016).
Not in our library
35.
Recurrent Neural Networks for Multivariate Time Series with Missing Values · 2018
No notice
36.
MIMIC-III, a freely accessible critical care database · 2016
No notice
37.
Multitask learning and benchmarking with clinical time series data · 2019
No notice
38.
Society of Critical Care Medicine. Critical care statistics. Available at: http://www.sccm.org/Communications/Pages/CriticalCareStats.aspx (Accessed 25 Jan 2018).
Not in our library
39.
American Hospital Association. Fast facts on U.S. Hospitals, 2018. Available at: https://www.aha.org/statistics/fast-facts-us-hospitals (Accessed 25 Jan 2018).
Not in our library
40.
Shickel, B., Tighe, P., Bihorac, A. & Rashidi, P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR) analysis. arXiv [cs.LG] (2017).
Not in our library10.1109/jbhi.2017.2767063
41.
Bergstrom N, Braden BJ, Laguzza A, Holman V. The braden scale for predicting pressure sore risk. Nurs. Res. 1987;36:205-210.
Not in our library10.1097/00006199-198707000-00002
42.
Tabak YP, Sun X, Nunez CM, Johannes RS. Using electronic health record data to develop inpatient mortality predictive model: Acute Laboratory Risk of Mortality Score (ALaRMS). J. Am. Med. Inform. Assoc. 2014;21:455-463.
Not in our library10.1136/amiajnl-2013-001790
43.
Nguyen OK, et al. Predicting all-cause readmissions using electronic health record data from the entire hospitalization: Model development and comparison. J. Hosp. Med. 2016;11:473-480.
Not in our library10.1002/jhm.2568
44.
Liu V, Kipnis P, Gould MK, Escobar GJ. Length of stay predictions: improvements through the use of automated laboratory and comorbidity variables. Med. Care. 2010;48:739-744.
Not in our library10.1097/mlr.0b013e3181e359f3
45.
Walsh C, Hripcsak G. The effects of data sources, cohort selection, and outcome definition on a predictive model of risk of thirty-day hospital readmissions. J. Biomed. Inform. 2014;52:418-426.
Not in our library10.1016/j.jbi.2014.08.006
46.
Kellett J, Kim A. Validation of an abbreviated VitalpacTM Early Warning Score (ViEWS) in 75,419 consecutive admissions to a Canadian regional hospital. Resuscitation. 2012;83:297-302.
Not in our library10.1016/j.resuscitation.2011.08.022
47.
Escobar GJ, et al. Risk-adjusting hospital inpatient mortality using automated inpatient, outpatient, and laboratory databases. Med. Care. 2008;46:232-239.
Not in our library10.1097/mlr.0b013e3181589bb6
48.
van Walraven C, et al. Derivation and validation of an index to predict early death or unplanned readmission after discharge from hospital to the community. CMAJ. 2010;182:551-557.
Not in our library10.1503/cmaj.091117
49.
Yamana H, Matsui H, Fushimi K, Yasunaga H. Procedure-based severity index for inpatients: development and validation using administrative database. BMC Health Serv. Res. 2015;15:261.
Not in our library10.1186/s12913-015-0889-x
50.
Pine M, et al. Modifying ICD-9-CM coding of secondary diagnoses to improve risk-adjustment of inpatient mortality rates. Med. Decis. Making. 2009;29:69-81.
Not in our library10.1177/0272989x08323297
51.
Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation. 2013;84:465-470.
Not in our library10.1016/j.resuscitation.2012.12.016
52.
Khurana HS, et al. Real-time automated sampling of electronic medical records predicts hospital mortality. Am. J. Med. 2016;129:688-698.e2.
Not in our library10.1016/j.amjmed.2016.02.037
53.
Rothman MJ, Rothman SI, Beals J. Development and validation of a continuous measure of patient condition using the electronic medical record. J. Biomed. Inform. 2013;46:837-848.
Not in our library10.1016/j.jbi.2013.06.011
54.
Finlay GD, Rothman MJ, Smith RA. Measuring the modified early warning score and the Rothman index: advantages of utilizing the electronic medical record in an early warning system. J. Hosp. Med. 2014;9:116-119.
Not in our library10.1002/jhm.2132
55.
Zapatero A, et al. Predictive model of readmission to internal medicine wards. Eur. J. Intern. Med. 2012;23:451-456.
Not in our library10.1016/j.ejim.2012.01.005
56.
Shams I, Ajorlou S, Yang K. A predictive analytics approach to reducing 30-day avoidable readmissions among patients with heart failure, acute myocardial infarction, pneumonia, or COPD. Health Care. Manag. Sci. 2015;18:19-34.
Not in our library10.1007/s10729-014-9278-y
57.
Tsui E, Au SY, Wong CP, Cheung A, Lam P. Development of an automated model to predict the risk of elderly emergency medical admissions within a month following an index hospital visit: A Hong Kong experience. Health Inform. J. 2013;21:46-56.
Not in our library10.1177/1460458213501095
58.
Choudhry SA, et al. A public-private partnership develops and externally validates a 30-day hospital readmission risk prediction model. Online J. Public Health Inform. 2013;5:219.
Not in our library10.5210/ojphi.v5i2.4726
59.
Caruana, R. et al. Intelligible models for healthcare: predicting pneumonia risk and hospital 30-day readmission. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp 1721–1730. http://doi.acm.org/10.1145/2783258.2788613 (ACM, Sydney, NSW, Australia, 2015).
Not in our library
60.
Tonkikh O, et al. Functional status before and during acute hospitalization and readmission risk identification. J. Hosp. Med. 2016;11:636-641.
Not in our library10.1002/jhm.2595
61.
Betihavas V, et al. An absolute risk prediction model to determine unplanned cardiovascular readmissions for adults with chronic heart failure. Heart Lung Circ. 2015;24:1068-1073.
Not in our library10.1016/j.hlc.2015.04.168
62.
Whitlock TL, et al. A scoring system to predict readmission of patients with acute pancreatitis to the hospital within thirty days of discharge. Clin. Gastroenterol. Hepatol. 2011;9:175-180.
Not in our library10.1016/j.cgh.2010.08.017
63.
Coleman EA, Min SJ, Chomiak A, Kramer AM. Posthospital care transitions: patterns, complications, and risk identification. Health Serv. Res. 2004;39:1449-1465.
Not in our library10.1111/j.1475-6773.2004.00298.x
64.
Graboyes EM, Liou TN, Kallogjeri D, Nussenbaum B, Diaz JA. Risk factors for unplanned hospital readmission in otolaryngology patients. Otolaryngol. Head Neck Surg. 2013;149:562-571.
Not in our library10.1177/0194599813500023
65.
He D, Mathews SC, Kalloo AN, Hutfless S. Mining high-dimensional administrative claims data to predict early hospital readmissions. J. Am. Med. Inform. Assoc. 2014;21:272-279.
Not in our library10.1136/amiajnl-2013-002151
66.
Futoma J, Morris J, Lucas J. A comparison of models for predicting early hospital readmissions. J. Biomed. Inform. 2015;56:229-238.
Not in our library10.1016/j.jbi.2015.05.016
67.
Donzé J, Aujesky D, Williams D, Schnipper JL. Potentially avoidable 30-day hospital readmissions in medical patients: derivation and validation of a prediction model. JAMA Intern. Med. 2013;173:632-638.
Not in our library10.1001/jamainternmed.2013.3023
68.
Perotte A, et al. Diagnosis code assignment: models and evaluation metrics. J. Am. Med. Inform. Assoc. 2014;21:231-237.
Not in our library10.1136/amiajnl-2013-002159
69.
Krumholz HM, Terry SF, Waldstreicher J. Data acquisition, curation, and use for a continuously learning health system. JAMA. 2016;316:1669-1670.
Not in our library10.1001/jama.2016.12537
70.
Grumbach K, Lucey CR, Claiborne Johnston S. Transforming from centers of learning to learning health systems: the challenge for academic health centers. JAMA. 2014;311:1109-1110.
Not in our library10.1001/jama.2014.705
71.
Halamka JD, Tripathi M. The HITECH era in retrospect. N. Engl. J. Med. 2017;377:907-909.
Not in our library10.1056/nejmp1709851
72.
Bates DW, et al. Ten commandments for effective clinical decision support: making the practice of evidence-based medicine a reality. J. Am. Med. Inform. Assoc. 2003;10:523-530.
Not in our library10.1197/jamia.m1370
73.
Obermeyer Z, Emanuel EJ. Predicting the future --- big data, machine learning, and clinical medicine. N. Engl. J. Med. 2016;375:1216-1219.
Not in our library10.1056/nejmp1606181
74.
Improving palliative care with deep learning · 2018
No notice
75.
Health Level 7. FHIR Specification Home Page (2017). Available at: http://hl7.org/fhir/ (Accessed 3 Aug 2017).
Not in our library
76.
Escobar GJ, et al. Nonelective rehospitalizations and postdischarge mortality: predictive models suitable for use in real time. Med. Care. 2015;53:916-923.
Not in our library10.1097/mlr.0000000000000435
77.
2016 Measure updates and specifications report: hospital-wide all-cause unplanned readmission --- version 5.0. Yale--New Haven Health Services Corporation/Center for Outcomes Research & Evaluation (New Haven, CT, 2016).
Not in our library
78.
Knaus WA, Draper EA, Wagner DP, Zimmerman JE. APACHE II: a severity of disease classification system. Crit. Care Med. 1985;13:818-829.
Not in our library10.1097/00003246-198510000-00009
79.
Kansagara D, et al. Risk prediction models for hospital readmission: a systematic review. JAMA. 2011;306:1688-1698.
Not in our library10.1001/jama.2011.1515
80.
Hochreiter S, Schmidhuber J. Long short-term memory. Neural Comput. 1997;9:1735-1780.
Not in our library10.1162/neco.1997.9.8.1735
81.
Rokach L. Ensemble-based Classifiers. Artif. Intell. Rev. 2010;33:1-39.
Not in our library10.1007/s10462-009-9124-7
82.
Cabitza F, Rasoini R, Gensini GF. Unintended consequences of machine learning in medicine. JAMA. 2017;18:517-518.
Not in our library10.1001/jama.2017.7797
83.
Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. arXiv [cs.CL] (2014).
Not in our library
84.
Pedregosa F, et al. Scikit-learn: machine learning in python. J. Mach. Learn. Res. 2011;12:2825-2830.
Not in our library
85.
Pencina MJ, D’Agostino RB. Evaluating discrimination of risk prediction models: the C statistic. JAMA. 2015;314:1063-1064.
Not in our library10.1001/jama.2015.11082
86.
Kramer AA, Zimmerman JE. Assessing the calibration of mortality benchmarks in critical care: the Hosmer-Lemeshow test revisited. Crit. Care Med. 2007;35:2052-2056.
Not in our library10.1097/01.ccm.0000275267.64078.b0
87.
Why the C-statistic is not informative to evaluate early warning scores and what metrics to use · 2015
No notice
88.
SciKit Learn. SciKit learn documentation on area under the curve scores. Available at: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html (Accessed 3 Aug 2017).
Not in our library
89.
SciKit Learn. SciKit learn documentation on F1 score. Available at: http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html (Accessed 3 Aug 2017).
Not in our library

Cited by papers we hold 34

Artificial intelligence in healthcare and medicine: clinical applications, therapeutic advances, and future perspectives · 2025
No notice
ChatGPT for shaping the future of dentistry: the potential of multi-modal large language model · 2023
No notice
Redefining Radiology: A Review of Artificial Intelligence Integration in Medical Imaging · 2023
No notice
The shaky foundations of large language models and foundation models for electronic health records · 2023
No notice
Predicting outcomes in patients with aortic stenosis using machine learning: the Aortic Stenosis Risk (ASteRisk) score · 2022
Not checked
Integrated multimodal artificial intelligence framework for healthcare applications · 2022
No notice
A systematic review of the prediction of hospital length of stay: Towards a unified framework · 2022
No notice
Machine learning model to predict mental health crises from electronic health records · 2022
No notice
Identifying subtypes of depression in clinician-annotated text: a retrospective cohort study · 2021
Not checked
Recurrent disease progression networks for modelling risk trajectory of heart failure. · 2021
Not checked
Neural network predicts need for red blood cell transfusion for patients with acute gastrointestinal bleeding admitted to the intensive care unit · 2021
Not checked
Med-BERT: pretrained contextualized embeddings on large-scale structured electronic health records for disease prediction · 2021
No notice
Prognostic Modeling of COVID-19 Using Artificial Intelligence in the United Kingdom: Model Development and Validation · 2020
Not checked
Machine learning and artificial intelligence research for patient benefit: 20 critical questions on transparency, replicability, ethics, and effectiveness · 2020
Corrected
The Application of Deep Learning in Cancer Prognosis Prediction · 2020
No notice
Explainable artificial intelligence model to predict acute critical illness from electronic health records · 2020
No notice
Deep representation learning of electronic health records to unlock patient stratification at scale · 2020
No notice
Combining structured and unstructured data for predictive models: a deep learning approach · 2020
No notice
AI in Medical Imaging Informatics: Current Challenges and Future Directions · 2020
No notice
Artificial intelligence with multi-functional machine learning platform development for better healthcare and precision medicine · 2020
No notice

and 14 more

Reads

Read by people on this page

None yet.

Checked or read by AI through naulon

None yet.

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.

Are you an author?

Is this your paper?

Claim it to add your note and be paid when agents cite it. Free.