Integrated multimodal artificial intelligence framework for healthcare applications

Abstract

Artificial intelligence (AI) systems hold great promise to improve healthcare over the next decades. Specifically, AI systems leveraging multiple data sources and input modalities are poised to become a viable method to deliver more accurate results and deployable pipelines across a wide range of applications. In this work, we propose and evaluate a unified Holistic AI in Medicine (HAIM) framework to facilitate the generation and testing of AI systems that leverage multimodal inputs. Our approach uses generalizable data pre-processing and machine learning modeling stages that can be readily adapted for research and deployment in healthcare environments. We evaluate our HAIM framework by training and characterizing 14,324 independent models based on HAIM-MIMIC-MM, a multimodal clinical database ( N = 34,537 samples) containing 7279 unique hospitalizations and 6485 patients, spanning all possible input combinations of 4 data modalities (i.e., tabular, time-series, text, and images), 11 unique data sources and 12 predictive tasks. We show that this framework can consistently and robustly produce models that outperform similar single-source approaches across various healthcare demonstrations (by 6–33%), including 10 distinct chest pathology diagnoses, along with length-of-stay and 48 h mortality predictions. We also quantify the contribution of each modality and data source using Shapley values, which demonstrates the heterogeneity in data modality importance and the necessity of multimodal inputs across different healthcare-relevant tasks. The generalizable properties and flexibility of our Holistic AI in Medicine (HAIM) framework could offer a promising pathway for future multimodal predictive systems in clinical and operational healthcare settings.

References 45

1. 1.

   Topol, E. Deep medicine: how artificial intelligence can make healthcare human again. (Hachette UK, 2019).

   Not in our library

2. 2.
   Fusion of medical imaging and electronic health records using deep learning: a systematic review and implementation guidelines · 2020

   No notice
3. 3.

   Gietzelt M, Löpprich M, Karmen C, Ganzinger M. Models and data sources used in systems medicine. Methods Inf. Med. 2016;55:107-113.

   Not in our library10.3414/me15-01-0151

4. 4.

   Boonn WW, Langlotz CP. Radiologist use of and perceived need for patient data access. J. Dig. imaging. 2009;22:357-362.

   Not in our library10.1007/s10278-008-9115-2

5. 5.

   Wang W, Krishnan E. Big data and clinicians: a review on the state of the science. JMIR Med. Inform. 2014;2:e1.

   Not in our library10.2196/medinform.2913

6. 6.

   Sun W, et al. Data processing and text mining technologies on electronic medical records: a review. J. Healthcare Eng. 2018;2018:4-7.

   Not in our library10.1155/2018/4302425

7. 7.

   Agrawal S, et al. Selection of 51 predictors from 13,782 candidate multimodal features using machine learning improves coronary artery disease prediction. Patterns. 2021;2:100364.

   Not in our library10.1016/j.patter.2021.100364

8. 8.

   Bagheri, A. et al. Multimodal learning for cardiovascular risk prediction using EHR data. arXiv preprint arXiv:2008.11979 (2020).

   Not in our library

9. 9.

   Li P, Hu Y, Liu Z-P. Prediction of cardiovascular diseases by integrating multi-modal features with machine learning methods. Biomed. Signal Process. Control. 2021;66:102474.

   Not in our library

10. 10.

    Liu Y, et al. A deep learning system for differential diagnosis of skin diseases. Nat. Med. 2020;26:900-908.

    Not in our library10.1038/s41591-020-0842-3

11. 11.

    Stidham RW. Artificial Intelligence for Understanding Imaging, Text, and Data in Gastroenterology. Gastroenterol. Hepatol. 2020;16:341.

    Not in our library

12. 12.

    Paquette AG, Hood L, Price ND, Sadovsky Y. Deep Phenotyping During Pregnancy for Delivery of Predictive and Preventive Medicine. Sci.Transl. Med. 2020;12:2-4.

    Not in our library10.1126/scitranslmed.aay1059

13. 13.

    Purwar S, Tripathi RK, Ranjan R, Saxena R. Detection of microcytic hypochromia using cbc and blood film features extracted from convolution neural network by different classifiers. Multimed. Tools Appl. 2020;79:4573-4595.

    Not in our library

14. 14.

    Hügle, M., Kalweit, G., Hügle, T. & Boedecker, J. In Explainable AI in Healthcare and Medicine 79–92 (Springer, 2021).

    Not in our library

15. 15.

    Tomašev N, et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature. 2019;572:116-119.

    Not in our library10.1038/s41586-019-1390-1

16. 16.

    Ieracitano C, Mammone N, Hussain A, Morabito FC. A novel multi-modal machine learning based approach for automatic classification of EEG recordings in dementia. Neural Netw. 2020;123:176-190.

    Not in our library10.1016/j.neunet.2019.12.006

17. 17.

    Prashanth R, Roy SD, Mandal PK, Ghosh S. High-accuracy detection of early Parkinson’s disease through multimodal features and machine learning. Int. J. Med. Inform. 2016;90:13-21.

    Not in our library10.1016/j.ijmedinf.2016.03.001

18. 18.

    Hyun SH, Ahn MS, Koh YW, Lee SJ. A machine-learning approach using PET-based radiomics to predict the histological subtypes of lung cancer. Clin. Nucl. Med. 2019;44:956-960.

    Not in our library10.1097/rlu.0000000000002810

19. 19.

    Yala A, Lehman C, Schuster T, Portnoi T, Barzilay R. A deep learning mammography-based model for improved breast cancer risk prediction. Radiology. 2019;292:60-66.

    Not in our library10.1148/radiol.2019182716

20. 20.

    Reda I, et al. Deep learning role in early diagnosis of prostate cancer. Technol. Cancer Res. Treat. 2018;17:1533034618775530.

    Not in our library10.1177/1533034618775530

21. 21.

    An G, et al. Comparison of machine-learning classification models for glaucoma management. J. Healthcare Eng. 2018;2018:2-7.

    Not in our library10.1155/2018/6874765

22. 22.

    Patel MJ, et al. Machine learning approaches for integrating clinical and imaging features in late‐life depression classification and response prediction. Int. J. Geriatr. Psychiatry. 2015;30:1056-1067.

    Not in our library10.1002/gps.4262

23. 23.
    Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case-study in pulmonary embolism detection · 2020

    No notice
24. 24.
    Multimodal Machine Learning-based Knee Osteoarthritis Progression Prediction from Plain Radiographs and Clinical Data · 2019

    No notice
25. 25.

    Wu J, et al. Radiological tumour classification across imaging modality and histology. Nat. Mach. Intell. 2021;3:787-798.

    Not in our library10.1038/s42256-021-00377-0

26. 26.

    Mei X, et al. Artificial intelligence–enabled rapid diagnosis of patients with COVID-19. Nat. Med. 2020;26:1224-1228.

    Not in our library10.1038/s41591-020-0931-3

27. 27.

    Bardak B, Tan M. Improving clinical outcome predictions using convolution over medical entities with multimodal learning. Artif. Intell. Med. 2021;117:102112.

    Not in our library10.1016/j.artmed.2021.102112

28. 28.

    Jin, M. et al. Improving hospital mortality prediction with medical named entities and multimodal learning. arXiv preprint arXiv:1811.12276 (2018).

    Not in our library

29. 29.
    Scalable and accurate deep learning with electronic health records · 2018

    No notice
30. 30.

    Li Y, et al. Inferring multimodal latent topics from electronic health records. Nat. Commun. 2020;11:1-17.

    Not in our library10.1038/s41467-020-16378-3

31. 31.

    Štrumbelj E, Kononenko I. Explaining prediction models and individual predictions with feature contributions. Knowl. Inf. Syst. 2014;41:647-665.

    Not in our library

32. 32.

    Johnson, A. et al. MIMIC-IV (version 1.0). PhysioNet, 10.13026/s6n6-xd98. (2021).

    Not in our library

33. 33.

    Goldberger AL, et al. PhysioBank, PhysioToolkit, and PhysioNet: components of a new research resource for complex physiologic signals. Circulation. 2000;101:e215-e220.

    Not in our library10.1161/01.cir.101.23.e215

34. 34.
    MIMIC-III, a freely accessible critical care database · 2016

    No notice
35. 35.

    Royalty, J. P. Machine Learning Time-to-Event Mortality Prediction in MIMIC-IV Critical Care Database (Doctoral dissertation). Undergraduate Research Scholars Program. Available electronically from https://hdl.handle.net/1969.1/194429 (2021).

    Not in our library

36. 36.

    Meng, C., Trinh, L., Xu, N. & Liu, Y. MIMIC-IF: Interpretability and Fairness Evaluation of Deep Learning Models on MIMIC-IV Dataset. arXiv preprint arXiv:2102.06761 (2021).

    Not in our library10.1038/s41598-022-11012-2

37. 37.

    Johnson, A. E. et al. MIMIC-CXR-JPG, a large publicly available database of labeled chest radiographs. arXiv preprint arXiv:1901.07042 (2019).

    Not in our library

38. 38.

    Soenksen, L. R. & Ma, Y. Code for generating the HAIM multimodal dataset of MIMIC-IV clinical data and x-rays (version 1.0.0). PhysioNet, 10.13026/dxcx-n572 (2022).

    Not in our library

39. 39.

    Alsentzer, E. et al. Publicly available clinical BERT embeddings. arXiv preprint arXiv:1904.03323 (2019).

    Not in our library

40. 40.

    Irvin J, et al. Chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. Proc. AAAI Conf. Artif. Intell. 2019;33:590-597.

    Not in our library

Cited by papers we hold 1

• The shaky foundations of large language models and foundation models for electronic health records · 2023
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.