Explainable artificial intelligence model to predict acute critical illness from electronic health records

Acute critical illness is often preceded by deterioration of routinely measured clinical parameters, e.g., blood pressure and heart rate. Early clinical prediction is typically based on manually calculated screening metrics that simply weigh these parameters, such as early warning scores (EWS). The predictive performance of EWSs yields a tradeoff between sensitivity and specificity that can lead to negative outcomes for the patient. Previous work on electronic health records (EHR) trained artificial intelligence (AI) systems offers promising results with high levels of predictive performance in relation to the early, real-time prediction of acute critical illness. However, without insight into the complex decisions by such system, clinical translation is hindered. Here, we present an explainable AI early warning score (xAI-EWS) system for early detection of acute critical illness. xAI-EWS potentiates clinical translation by accompanying a prediction with information on the EHR data explaining it.

Its licence does not allow us to show the full text. Read it at PubMed Central:

1. 1.

   Tomašev N, et al. A clinically applicable approach to continuous prediction of future acute kidney injury. Nature. 2019;572:116-119.

   Not in our library10.1038/s41586-019-1390-1 (opens in a new tab)

2. 2.

   Lauritsen SM, et al. Early detection of sepsis utilizing deep learning on electronic health record event sequences. Artif. Intell. Med. 2020;104:101820.

   Not in our library10.1016/j.artmed.2020.101820 (opens in a new tab)

3. 3.

   Barton C, et al. Evaluation of a machine learning algorithm for up to 48-hour advance prediction of sepsis using six vital signs. Comput. Biol. Med. 2019;109:79-84.

   Not in our library10.1016/j.compbiomed.2019.04.027 (opens in a new tab)

4. 4.

   Islam MM, et al. Prediction of sepsis patients using machine learning approach: a meta-analysis. Comput. Meth. Prog. Bio. 2019;170:1-9.

   Not in our library10.1016/j.cmpb.2018.12.027 (opens in a new tab)

5. 5.

   Moor, M., Horn, M., Rieck, B., Roqueiro, D. & Borgwardt, K. Temporal convolutional networks and dynamic time warping can drastically improve the early prediction of Sepsis. In Proceedings of the 4th Machine Learning for Healthcare Conference. PMLR 106 (2019).

   Not in our library

6. 6.

   Mao Q, et al. Multicentre validation of a sepsis prediction algorithm using only vital sign data in the emergency department, general ward and ICU. BMJ Open. 2018;8:e017833.

   Not in our library10.1136/bmjopen-2017-017833 (opens in a new tab)

7. 7.

   Shimabukuro DW, Barton CW, Feldman MD, Mataraso SJ, Das R. Effect of a machine learning-based severe sepsis prediction algorithm on patient survival and hospital length of stay: a randomised clinical trial. BMJ Open Respir. Res. 2017;4:e000234.

   Not in our library10.1136/bmjresp-2017-000234 (opens in a new tab)

8. 8.

   Kam HJ, Kim HY. Learning representations for the early detection of sepsis with deep neural networks. Comput. Biol. Med. 2017;89:248-255.

   Not in our library10.1016/j.compbiomed.2017.08.015 (opens in a new tab)

9. 9.

   Calvert JS, et al. A computational approach to early sepsis detection. Comput. Biol. Med. 2016;74:69-73.

   Not in our library10.1016/j.compbiomed.2016.05.003 (opens in a new tab)

10. 10.

    Futoma, J. et al. An improved multi-output gaussian process RNN with real-time validation for early Sepsis detection. In Proceedings of Machine Learning for Healthcare 2017. JMLR, 68 (2017).

    Not in our library

11. 11.

    Futoma, J., Hariharan, S. & Heller, K. Learning to detect sepsis with a multitask gaussian process RNN Classifier. In Proceedings of the 34th International Conference on Machine Learning. ICML vol. 70, 1174–1182 (2017).

    Not in our library

12. 12.

    Vellido, A., Ribas, V., Morales, C., Sanmartín, A. R. & Ruiz-Rodríguez, J. C. Machine learning for critical care: an overview and a Sepsis case study. In Bioinformatics and Biomedical Engineering 2017 (eds Rojas, I. & Ortuño, F.) (IWBBIO) vol. 10208, 15–30 (2017).

    Not in our library

13. 13.

    Shickel B, et al. DeepSOFA: a continuous acuity score for critically ill patients using clinically interpretable deep learning. Sci. Rep. 2019;9:1879.

    Not in our library10.1038/s41598-019-38491-0 (opens in a new tab)

14. 14.

    Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat. Med. 2019;25:44-56.

    Not in our library10.1038/s41591-018-0300-7 (opens in a new tab)

15. 15.

    Shickel B, Tighe PJ, Bihorac A, Rashidi P. Deep EHR: a survey of recent advances in deep learning techniques for electronic health record (EHR). Anal. IEEE J. Biomed. Health. 2017;22:1589-1604.

    Not in our library10.1109/jbhi.2017.2767063 (opens in a new tab)

16. 16.

    Rajkomar A, Dean J, Kohane I. Machine learning in medicine. N. Engl. J. Med. 2019;380:1347-1358.

    Not in our library10.1056/nejmra1814259 (opens in a new tab)

17. 17.

    Cabitza F, Rasoini R, Gensini GF. Unintended consequences of machine learning in medicine. JAMMA. 2017;318:517-518.

    Not in our library10.1001/jama.2017.7797 (opens in a new tab)

18. 18.

    Xiao C, Choi E, Sun J. Opportunities and challenges in developing deep learning models using electronic health records data: a systematic review. J. Am. Med. Inf. Assn. 2018;25:1419-1428.

    Not in our library10.1093/jamia/ocy068 (opens in a new tab)

19. 19.

    The Lancet Respiratory Medicine. Opening the black box of machine learning. Lancet Respir. Med.6, 837 10.1016/S2213-2600(18)30425-9 (2018).

    Not in our library10.1016/s2213-2600(18)30425-9 (opens in a new tab)

20. 20.

    Causability and explainability of artificial intelligence in medicine · 2019

21. 21.

    Kalchbrenner, N. et al. Neural machine translation in linear time. Preprint at https://arxiv.org/abs/1610.10099 (2016).

    Not in our library

22. 22.

    Bai, S., Kolter, J. Z. & Koltun, V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. Preprint at https://arxiv.org/abs/1803.01271 (2018).

    Not in our library

23. 23.

    Montavon G, Lapuschkin S, Binder A, Samek W, Müller K. Explaining nonlinear classification decisions with deep Taylor decomposition. Pattern Recogn. 2017;65:211-222.

    Not in our library10.1016/j.patcog.2016.11.008 (opens in a new tab)

24. 24.

    Montavon G, Samek W, Müller K-R. Methods for interpreting and understanding deep neural networks. Digit Signal Process. 2017;73:1-15.

    Not in our library10.1016/j.dsp.2017.10.011 (opens in a new tab)

25. 25.

    On Pixel-Wise Explanations for Non-Linear Classifier Decisions by Layer-Wise Relevance Propagation · 2015

26. 26.

    Samek, W., Wiegand, T. & Müller, K.-R. Explainable artificial intelligence: understanding, visualizing and interpreting deep learning models. The Impact of Artificial Intelligence (AI) on Communication Networks and Services1, 39–48 (2015)

    Not in our library

27. 27.

    Samek, W., Binder, A., Montavon, G., Lapuschkin, S. & Müller, K. Evaluating the visualization of what a deep neural network has learned. IEEE T. Neur. Net. Lear Syst.28, 2660–2673 (2017).

    Not in our library10.1109/tnnls.2016.2599820 (opens in a new tab)

28. 28.

    Lea, C., Vidal, R., Reiter, A. & Hager, G. D. Temporal convolutional networks: a unified approach to action segmentation. In Computer Vision – ECCV 2016 Workshops. 47–54 (Springer International Publishing, 2016).

    Not in our library

29. 29.

    Aksan, E. & Hilliges, O. STCN: Stochastic temporal convolutional networks. In International Conference on Learning Representations 2019, CoRR (2019).

    Not in our library

30. 30.

    Singer M, et al. The third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315:801-810.

    Not in our library10.1001/jama.2016.0287 (opens in a new tab)

31. 31.

    Khwaja A. KDIGO clinical practice guidelines for acute kidney injury. Nephron Clin. Pract. 2012;120:c179-c184.

    Not in our library10.1159/000339789 (opens in a new tab)

32. 32.

    Zhang J, Kowsari K, Harrison JH, Lobo JM, Barnes LE. patient2Vec: a personalized interpretable deep representation of the longitudinal electronic health record. IEEE Access. 2018;6:65333-65346.

    Not in our library10.1109/access.2018.2875677 (opens in a new tab)

33. 33.

    Scalable and accurate deep learning with electronic health records · 2018

34. 34.

    Lundberg SM, et al. From local explanations to global understanding with explainable AI for trees. Nat. Mach. Intell. 2020;2:56-67.

    Not in our library10.1038/s42256-019-0138-9 (opens in a new tab)

35. 35.

    Kaji DA, et al. An attention based deep learning model of clinical events in the intensive care unit. PLos ONE. 2019;14:e0211057.

    Not in our library10.1371/journal.pone.0211057 (opens in a new tab)

36. 36.

    Choi, E. et al. RETAIN: An interpretable predictive model for healthcare using reverse time attention mechanism. In 30th Annual Conference on Neural Information Processing Systems (NIPS 2016). Advances in Neural Information Processing Systems 3512–3520 (2016).

    Not in our library

37. 37.

    Holzinger A, Carrington A, Müller H. Measuring the quality of explanations: the system causability scale (SCS). Künstl Intell. 2020;34:193-198.

    Not in our library10.1007/s13218-020-00636-z (opens in a new tab)

38. 38.

    Selbst AD, Powles J. Meaningful information and the right to explanation. Int Data Priv. Law. 2017;7:233-242.

    Not in our library10.1093/idpl/ipx022 (opens in a new tab)

39. 39.

    Seymour CW, et al. Assessment of clinical criteria for sepsis: for the third international consensus definitions for sepsis and septic shock (Sepsis-3). JAMA. 2016;315:762-774.

    Not in our library10.1001/jama.2016.0288 (opens in a new tab)

40. 40.

    Vincent J-L, et al. The SOFA (Sepsis-related organ failure assessment) score to describe organ dysfunction/failure. Intens. Care Med. 2018;22:707-710.

    Not in our library10.1007/bf01709751 (opens in a new tab)

41. 41.

    Deep learning · 2015

42. 42.

    Crowley, J. L. & Sandersen, A. Multiple resolution representation and probabilistic matching of 2-D gray-scale shape. IEEE Trans. Pattern Anal. Mach. Intell.https://ieeexplore.ieee.org/document/4767876 (1984).

    Not in our library10.1109/tpami.1987.4767876 (opens in a new tab)

43. 43.

    Nair, V. & Hinton, G. E. Rectified Linear Units Improve Restricted Boltzmann Machines. In ICML'10: Proceedings of the 27th International Conference on International Conference on Machine Learning, 807–814 (Omnipress, 2010).

    Not in our library

44. 44.

    Ba, J. L., Kiros, J. R. & Hinton, G. E. Layer normalization. Preprint at https://arxiv.org/abs/1607.06450 (2016).

    Not in our library

45. 45.

    Srivastava N, Hinton G, Krizhevsky A, Sutskever I, Salakhutdinov R. Dropout: a simple way to prevent neural networks from overfitting. J. Mach. Learn. Res. 2014;15:1929-1958.

    Not in our library

46. 46.

    Lin, M., Chen, Q. & Yan, S. Network in network. 2nd International Conference on Learning Representations. 14–16 http://arxiv.org/abs/1312.4400 (2013).

    Not in our library

47. 47.

    Kingma, D. P. & Ba, J. Adam: a method for stochastic optimization. 3rd International Conference for Learning Representations, San Diego Preprint at https://arxiv.org/abs/1412.6980 (2014).

    Not in our library

48. 48.

    He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778 (2016).

    Not in our library

49. 49.

    Montavon, G., Binder, A., Lapuschkin, S., Samek, W. & Müller, K. R. In Explainable AI: Interpreting, Explaining and Visualizing Deep Learning (eds Samek, W., Montavon, G., Vedaldi, A., Hansen, L. & Müller, K. R.), Vol. 11700 (Springer, 2019).

    Not in our library

50. 50.

    Breiman L. Random forests. Mach. Learn. 2001;45:5-32.

    Not in our library10.1023/a:1010933404324 (opens in a new tab)

51. 51.

    Alber, M. et al. iNNvestigate neural networks! Preprint at https://arxiv.org/abs/1808.04260 (2018).

    Not in our library

52. 52.

    Simonyan, K., Vedaldi, A. & Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. In Workshop at International Conference on Learning Representations. CoRR (2013).

    Not in our library

53. 53.

    Strumbelj E, Kononenko I. Explaining prediction models and individual predictions with feature contributions. Knowl. Inf. Syst. 2013;41:647-665.

    Not in our library10.1007/s10115-013-0679-x (opens in a new tab)

54. 54.

    Zeiler, M. D. & Fergus, R. Visualizing and understanding convolutional networks. In Computer Vision—ECCV. 818–833 (Springer, Cham, 2014).

    Not in our library

55. 55.

    Springenberg, J. T., Dosovitskiy, A., Brox, T. & Riedmiller, M. Striving for Simplicity: The all convolutional net. In ICLR (workshop track) (2014).

    Not in our library

56. 56.

    Zintgraf, L. M., Cohen, T. S., Adel, T. & Welling, M. Visualizing deep neural network decisions: prediction difference analysis. In 5th International Conference on Learning Representations (2017).

    Not in our library

57. 57.

    Ribeiro, M. T., Singh, S. & Guestrin, C. ‘Why Should I Trust You?’: Explaining the predictions of any classifier. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) (2016).

    Not in our library

58. 58.

    Shrikumar, A., Greenside, P. & Kundaje, A. Not just a black box: learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning Vol. 70, 3145–3153 (2017).

    Not in our library

59. 59.

    Shrikumar, A., Greenside, P. & Kundaje, A. Learning important features through propagating activation differences. In Proceedings of the 34th International Conference on Machine Learning Vol. 70, 3145–3153 (2017).

    Not in our library

60. 60.

    Sundararajan, M., Taly, A. & Yan, Q. Axiomatic attribution for deep networks. In Proceedings of the 34th International Conference on Machine Learning Vol. 70, 3319–3328 (2017).

    Not in our library

61. 61.

    Datta, A., Sen, S. & Zick, Y. Algorithmic transparency via quantitative input influence: theory and experiments with learning systems. In 2016 IEEE Symposium on Security and Privacy (SP) 598–617 10.1109/sp.2016.42 (2016).

    Not in our library

62. 62.

    Ancona, M., Ceolini, E., Öztireli, C. & Gross, M. Towards better understanding of gradient-based attribution methods for Deep Neural Networks. In Sixth International Conference on Learning Representations (2018).

    Not in our library

63. 63.

    Lundberg SM, Lee S-I. A unified approach to interpreting model predictions. Adv. Neural Inf. Process. Syst. 2017;30:4765-4774.

    Not in our library

64. 64.

    Ancona, M., Öztireli, C. & Gross, M. Explaining deep neural networks with a polynomial time algorithm for shapley values approximation. In Thirty-sixth International Conference on Machine Learning. ICML (2019).

    Not in our library

65. 65.

    Shapley, L. S. A value for n-person games. Contributions to the Theory of Games2, 307–317 (1953).

    Not in our library

66. 66.

    Matsui, Y. & Matsui, T. NP-completeness for calculating power indices of weighted majority games. Theor. Compu. Sci263, 305–310 (1998).

    Not in our library

67. 67.

    Chen, T. & Guestrin, C. XGBoost: A scalable tree boosting system. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining 785–794 (2016).

    Not in our library

68. 68.

    Riis, A. H. CROSS-TRACKS—a population-based danish cross-sectorial cohort. Preprint at https://www.medrxiv.org/content/10.1101/2020.05.13.20100263v1 (2019).

    Not in our library

• Explainable AI in Clinical Decision Support Systems: A Meta-Analysis of Methods, Applications, and Usability Challenges · 2025

• Should AI models be explainable to clinicians? · 2024

• Unbox the black-box for the medical explainable AI via multi-modal and multi-centre data fusion: A mini-review, two showcases and beyond · 2022

• Machine Learning–Based Early Warning Systems for Clinical Deterioration: Systematic Scoping Review · 2021

  Not checked

• Accessing Artificial Intelligence for Clinical Decision-Making · 2021

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.