Multitask learning and benchmarking with clinical time series data

Abstract

Health care is one of the most exciting frontiers in data mining and machine learning. Successful adoption of electronic health records (EHRs) created an explosion in digital clinical data available for analysis, but progress in machine learning for healthcare research has been difficult to measure because of the absence of publicly available benchmark data sets. To address this problem, we propose four clinical prediction benchmarks using data derived from the publicly available Medical Information Mart for Intensive Care (MIMIC-III) database. These tasks cover a range of clinical problems including modeling risk of mortality, forecasting length of stay, detecting physiologic decline, and phenotype classification. We propose strong linear and neural baselines for all four tasks and evaluate the effect of deep supervision, multitask training and data-specific architectural modifications on the performance of neural models.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 78

1. 1.

   Introduction to the HCUP National Inpatient Sample (NIS) 2012. (Agency for Healthcare Research and Quality, 2014).

   Not in our library

2. 2.

   Henry, J., Pylypchuk, Y., Talisha Searcy, M. & Patel, V. Adoption of electronic health record systems among US non-federal acute care hospitals: 2008–2015. ONCData Brief35 (Office of the National Coordinator for Health Information Technology, Washington DC, USA, 2015).

   Not in our library

3. 3.

   Bates DW, Saria S, Ohno-Machado L, Shah A, Escobar G. Big data in health care: using analytics to identify and manage high-risk and high-cost patients. Health Affairs. 2014;33:1123-1131.

   Not in our library10.1377/hlthaff.2014.0041

4. 4.

   Zimmerman JE, Kramer AA, McNair DS, Malila FM. Acute physiology and chronic health evaluation (apache) iv: hospital mortality assessment for today’s critically ill patients. Crit. Care Med. 2006;34:1297-1310.

   Not in our library10.1097/01.ccm.0000215112.84523.f0

5. 5.

   Williams, B. et al. National Early Warning Score (NEWS): Standardising the assessment of acute-illness severity in the NHS. (London: The Royal College of Physicians, 2012).

   Not in our library

6. 6.

   Dahl D, et al. The high cost of low-acuity icu outliers. Journal of Healthcare Management. 2012;57:421-433.

   Not in our library10.1097/00115514-201211000-00009

7. 7.

   Saria S, Goldenberg A. Subtyping: What it is and its role in precision medicine. IEEE Intelligent Systems. 2015;30:70-75.

   Not in our library10.1109/mis.2015.60

8. 8.

   Iserson KV, Moskop JC. Triage in medicine, part i: concept, history, and types. Ann. Emerg. Med. 2007;49:275-281.

   Not in our library10.1016/j.annemergmed.2006.05.019

9. 9.

   Apgar V. A proposal for a new method of evaluation of the newborn. Curr. Res. Anesth. Analg. 1952;32:260-267.

   Not in our library

10. 10.

    Ferrucci D, Levas A, Bagchi S, Gondek D, Mueller ET. Watson: beyond jeopardy! Artificial Intelligence. 2013;199:93-105.

    Not in our library10.1016/j.artint.2012.06.009

11. 11.

    Silver D, et al. Mastering the game of go with deep neural networks and tree search. Nature. 2016;529:484-489.

    Not in our library10.1038/nature16961

12. 12.

    Caballero Barajas, K. L. & Akella, R. Dynamically modeling patient’s health state from electronic medical records: A time series approach. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 69–78 (ACM, Sydney, Australia, 2015).

    Not in our library

13. 13.

    Ghassemi, M. et al. A multivariate timeseries modeling approach to severity of illness assessment and forecasting in icu with sparse, heterogeneous clinical data. In Proceedings of the Twenty-Ninth AAAI Conference on Artificial Intelligence, 446–453 (AAAI Press, Austin, Texas, 2015).

    Not in our library

14. 14.

    Luo, Y., Xin, Y., Joshi, R., Celi, L. & Szolovits, P. Predicting icu mortality risk by grouping temporal trends from a multivariate panel of physiologic measurements. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, 42–50 (AAAI Press, Phoenix, Arizona, 2016).

    Not in our library

15. 15.

    Lee J, Maslove DM. Customization of a severity of illness score using local electronic medical record data. Journal of intensive care medicine. 2017;32:38-47.

    Not in our library10.1177/0885066615585951

16. 16.

    Johnson, A., Pollard, T. & Mark, R. Reproducibility in critical care: a mortality prediction case study. In Proceedings of the 2nd Machine Learning for Healthcare Conference, vol. 68, 361–376 (PMLR, Boston, Massachusetts, 2017).

    Not in our library

17. 17.

    Quinn JA, Williams CK, McIntosh N. Factorial switching linear dynamical systems applied to physiological condition monitoring. IEEE Trans. Pattern Anal. Mach. Intell. 2009;31:1537-1551.

    Not in our library10.1109/tpami.2008.191

18. 18.

    Laxmisan A, et al. The multitasking clinician: Decision-making and cognitive demand during and after team handoffs in emergency care. Int. J. Med. Inform. 2007;76:801-811.

    Not in our library10.1016/j.ijmedinf.2006.09.019

19. 19.

    Horn SD, et al. The relationship between severity of illness and hospital length of stay and mortality. Med. Care. 1991;29:305-317.

    Not in our library10.1097/00005650-199104000-00001

20. 20.
    MIMIC-III, a freely accessible critical care database · 2016

    No notice
21. 21.

    Laboratory For Computational Physiology, M. I. T. The MIMIC-III clinical database. PhysioNet., 10.13026/C2XW26 (2015).

    Not in our library

22. 22.

    Harutyunyan H, et al. MIMIC-III benchmark repository. Zenodo. 2018.

    Not in our library10.5281/zenodo.1306527

23. 23.

    Caruana, R., Baluja, S. & Mitchell, T. Using the future to “sort out” the present: Rankprop and multitask learning for medical risk evaluation. In Advances in Neural Information Processing Systems8, 959–965 (MIT Press, Denver, Colorado, 1996).

    Not in our library

24. 24.

    Clermont G, Angus DC, DiRusso SM, Griffin M, Linde-Zwirble WT. Predicting hospital mortality for patients in the intensive care unit: a comparison of artificial neural networks with logistic fmultion models. Crit. Care Med. 2001;29:291-296.

    Not in our library10.1097/00003246-200102000-00012

25. 25.

    Celi LA, et al. A database-driven decision support system: customized mortality prediction. J. Pers. Med. 2012;2:138-148.

    Not in our library10.3390/jpm2040138

26. 26.
    Scalable and accurate deep learning with electronic health records · 2018

    No notice
27. 27.

    Grigsby J, Kooken R, Hershberger J. Simulated neural networks to predict outcomes, costs, and length of stay among orthopedic rehabilitation patients. Arch. Phys. Med. Rehabil. 1994;75:1077-1081.

    Not in our library10.1016/0003-9993(94)90081-7

28. 28.

    Mobley BA, Leasure R, Davidson L. Artificial neural network predictions of lengths of stay on a post-coronary care unit. Heart & Lung: The Journal of Acute and Critical Care. 1995;24:251-256.

    Not in our library10.1016/s0147-9563(05)80045-7

29. 29.

    Buchman TG, Kubos KL, Seidler AJ, Siegforth MJ. A comparison of statistical and connectionist models for the prediction of chronicity in a surgical intensive care unit. Crit. Care Med. 1994;22:750-762.

    Not in our library10.1097/00003246-199405000-00008

30. 30.

    Yousefi, S., Song, C., Nauata, N. & Cooper, L. Learning genomic representations to predict clinical outcomes in cancer. Preprint at, https://arxiv.org/abs/1609.08663 (2016).

    Not in our library

31. 31.

    Ranganath, R., Perotte, A., Elhadad, N. & Blei, D. Deep survival analysis. In Proceedings of the 1st Machine Learning for Healthcare Conference, vol. 56 (PMLR, Los Angeles, California, USA, 2016).

    Not in our library

32. 32.
    Computational Phenotype Discovery Using Unsupervised Feature Learning over Noisy, Sparse, and Irregular Clinical Data · 2013

    Corrected
33. 33.

    Che, Z., Kale, D., Li, W., Bahadori, M. T. & Liu, Y. Deep computational phenotyping. In Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 507–516 (ACM, Sydney, Australia, 2015).

    Not in our library

34. 34.

    Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F. & Sun, J. Doctor AI: Predicting clinical events via recurrent neural networks. In Proceedings of the 1st Machine Learning for Healthcare Conference, vol. 56 (PMLR, Los Angeles, California, USA, 2016).

    Not in our library

35. 35.

    Razavian, N., Marcus, J. & Sontag, D. Multi-task prediction of disease onsets from longitudinal lab tests. In Proceedings of the 1st Machine Learning for Healthcare Conference, vol. 56 (PMLR, Los Angeles, California, USA, 2016).

    Not in our library

36. 36.

    Lipton, Z. C., Kale, D. C., Elkan, C. & Wetzel, R. Learning to diagnose with LSTM recurrent neural networks. In International Conference on Learning Representations (San Juan, Puerto Rico, 2016).

    Not in our library

37. 37.

    Ngufor, C., Upadhyaya, S., Murphree, D., Kor, D. & Pathak, J. Multi-task learning with selective cross-task transfer for predicting bleeding and other important patient outcomes. In IEEE International Conference on Data Science and Advanced Analytics (DSAA), 1–8 (IEEE, Paris, France, 2015).

    Not in our library

38. 38.

    Collobert, R. & Weston, J. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th international conference on Machine learning, 160–167 (ACM, Helsinki, Finland, 2008).

    Not in our library

39. 39.

    Gupta, P., Malhotra, P., Vig, L. & Shroff, G. Using features from pre-trained timenet for clinical predictions. In Proceedings of the 3rd International Workshop on Knowledge Discovery in Healthcare Data at IJCAI-ECAI, 38–44 (Stockholm, Sweden, 2018).

    Not in our library

40. 40.

    Gupta, P., Malhotra, P., Vig, L. & Shroff, G. Transfer learning for clinical time series analysis using recurrent neural networks. In Machine Learning for Medicine and Healthcare Workshop at ACM KDD 2018 Conference (London, United Kingdom, 2018).

    Not in our library

41. 41.

    Jin, M. et al. Improving hospital mortality prediction with medical named entities and multimodal learning. In Machine Learning for Health (ML4H) Workshop at NeurIPS (Montreal, Canada, 2018).

    Not in our library

42. 42.

    Oh, J., Wang, J. & Wiens, J. Learning to exploit invariances in clinical time-series data using sequence transformer networks. In Proceedings of the 3rd Machine Learning for Healthcare Conference, vol. 85, 332–347 (PMLR, Palo Alto, California, USA, 2018).

    Not in our library

43. 43.

    Malone, B., Garcia-Duran, A. & Niepert, M. Learning representations of missing data for predicting patient outcomes. Preprint at, https://arxiv.org/abs/1811.04752 (2018).

    Not in our library

44. 44.

    Chang, C.-H., Mai, M. & Goldenberg, A. Dynamic measurement scheduling for adverse event forecasting using deep RL. In Machine Learning for Health (ML4H) Workshop at NeurIPS (Montreal, Canada, 2018).

    Not in our library

45. 45.

    Xu, Y., Biswal, S., Deshpande, S. R., Maher, K. O. & Sun, J. Raim: Recurrent attentive and intensive model of multimodal patient monitoring data. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, 2565–2573 (ACM, London, United Kingdom, 2018).

    Not in our library

46. 46.

    Chung, I., Kim, S., Lee, J., Hwang, S. J. & Yang, E. Mixed effect composite RNN-GP: A personalized and reliable prediction model for healthcare. Preprint at, https://arxiv.org/abs/1806.01551 (2018).

    Not in our library

47. 47.

    Bahadori, M. T. Spectral capsule networks. In International Conference on Learning Representations Workshop Track (New Orleans, Louisiana, USA, 2018).

    Not in our library

48. 48.

    Rafi, P., Pakbin, A. & Pentyala, S. K. Interpretable deep learning framework for predicting all-cause 30-day ICU readmissions. Tech. Rep., (Texas A&M University, 2018).

    Not in our library

49. 49.

    Song, H., Rajan, D., Thiagarajan, J. J. & Spanias, A. Attend and diagnose: Clinical time series analysis using attention models. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence (AAAI Press, New Orleans, Louisiana, USA, 2018).

    Not in our library

50. 50.

    Purushotham S, Meng C, Che Z, Liu Y. Benchmarking deep learning models on large healthcare datasets. J. Biomed. Inform. 2018;83:112-134.

    Not in our library10.1016/j.jbi.2018.04.007

51. 51.

    Le Gall J-R, et al. A simplified acute physiology score for icu patients. Crit. Care Med. 1984;12:975-977.

    Not in our library10.1097/00003246-198411000-00012

52. 52.

    Silva, I., Moody, G., Scott, D. J., Celi, L. A. & Mark, R. G. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In 2012 Computing in Cardiology, 245–248 (IEEE, Krakow, Poland, 2012).

    Not in our library

53. 53.

    Marlin, B. M., Kale, D. C., Khemani, R. G. & Wetzel, R. C. Unsupervised pattern discovery in electronic health care data using probabilistic clustering models. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium, 389–398 (ACM, Miami, Florida, 2012).

    Not in our library

54. 54.

    Davis, J. & Goadrich, M. The relationship between precision-recall and roc curves. In Proceedings of the 23rd International Conference on Machine Learning, 233–240 (ACM, Pittsburgh, Pennsylvania, USA, 2006).

    Not in our library

55. 55.

    Subbe C, Kruger M, Rutherford P, Gemmel L. Validation of a modified early warning score in medical admissions. Qjm. 2001;94:521-526.

    Not in our library10.1093/qjmed/94.10.521

56. 56.

    Prytherch DR, Smith GB, Schmidt PE, Featherstone PI. Views – towards a national early warning score for detecting adult inpatient deterioration. Resuscitation. 2010;81:932-937.

    Not in our library10.1016/j.resuscitation.2010.04.014

57. 57.

    Aleks, N. et al. Probabilistic detection of short events, with application to critical care monitoring. In Advances in Neural Information Processing Systems21, 49–56 (Curran Associates, Inc., Vancouver, Canada, 2009).

    Not in our library

58. 58.

    Henry KE, Hager DN, Pronovost PJ, Saria S. A targeted real-time early warning score (trewscore) for septic shock. Sci. Transl. Med. 2015;7:299ra122-299ra122.

    Not in our library10.1126/scitranslmed.aab3719

59. 59.

    Clifton, L., Clifton, D. A., Pimentel, M. A., Watkinson, P. J. & Tarassenko, L. Gaussian process regression in vital-sign early warning systems. In 2012 Annual International Conference of the IEEE Engineering in Medicine and Biology Society, 6161–6164 (IEEE, San Diego, California, USA, 2012).

    Not in our library10.1109/embc.2012.6347400

60. 60.

    Romano, P., Hussey P. & Ritley, D. Selecting quality and resource use measures: A decision guide for community quality collaboratives. (Agency for Healthcare Research and Quality, 2014).

    Not in our library

61. 61.

    Higgins TL, et al. Early indicators of prolonged intensive care unit stay: Impact of illness severity, physician staffing, and pre–intensive care unit length of stay. Crit. Care Med. 2003;31:45-51.

    Not in our library10.1097/00003246-200301000-00007

62. 62.

    Osler TM, et al. Predicting survival, length of stay, and cost in the surgical intensive care unit: Apache ii versus iciss. Journal of Trauma and Acute Care Surgery. 1998;45:234-238.

    Not in our library10.1097/00005373-199808000-00006

63. 63.

    Paterson R, et al. Prediction of in-hospital mortality and length of stay using an early warning scoring system: clinical audit. Clinical Medicine. 2006;6:281-284.

    Not in our library10.7861/clinmedicine.6-3-281

64. 64.

    Pofahl WE, Walczak SM, Rhone E, Izenberg SD. Use of an artificial neural network to predict length of stay in acute pancreatitis. The American Surgeon. 1998;64:868.

    Not in our library

65. 65.

    Cohen J. A coefficient of agreement for nominal scales. Educational and Psychological Measurement. 1960;20:37-46.

    Not in our library10.1177/001316446002000104

66. 66.

    Brennan RL, Prediger DJ. Coefficient kappa: Some uses, misuses, and alternatives. Educ. Psychol. Meas. 1981;41:687-699.

    Not in our library10.1177/001316448104100307

67. 67.

    Oellrich A, et al. The digital revolution in phenotyping. Brief. Bioinform. 2015;17:819-830.

    Not in our library10.1093/bib/bbv083

68. 68.

    Agarwal V, et al. Learning statistical models of phenotypes using noisy labeled training data. Journal of the American Medical Informatics Association. 2016;23:1166.

    Not in our library10.1093/jamia/ocw028

69. 69.

    Halpern Y, Horng S, Choi Y, Sontag D. Electronic medical record phenotyping using the anchor and learn framework. Journal of the American Medical Informatics Association. 2016;23:731.

    Not in our library10.1093/jamia/ocw011

70. 70.

    Ho, J. C., Ghosh, J. & Sun, J. Marble: High-throughput phenotyping from electronic health records via sparse nonnegative tensor factorization. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 115–124 (ACM, New York, New York, USA, 2014).

    Not in our library

71. 71.

    Clinical Classifications Software (CCS) for ICD-9-CM fact sheet (Agency for Healthcare Research and Quality, 2012).

    Not in our library

72. 72.

    Hochreiter S, Schmidhuber J. Long short-term memory. Neur. Comp. 1997;9:1735-1780.

    Not in our library10.1162/neco.1997.9.8.1735

73. 73.

    Gers, F. A. & Schmidhuber, J. Recurrent nets that time and count. In Proceedings of the IEEE-INNS-ENNS International Joint Conference on Neural Networks, vol. 3, 189–194 (IEEE, Como, Italy, 2000).

    Not in our library

74. 74.

    Lipton, Z. C., Kale, D. C. & Wetzel, R. Modeling missing data in clinical time series with rnns. In Proceedings of the 1st Machine Learning for Healthcare Conference, vol. 56 (PMLR, Los Angeles, California, USA, 2016).

    Not in our library

75. 75.

    Kingma, D. & Ba, J. Adam: A method for stochastic optimization. Preprint at, https://arxiv.org/abs/1412.6980 (2014).

    Not in our library

76. 76.

    Choi, E. et al. Retain: An interpretable predictive model for healthcare using reverse time attention mechanism. In Advances in Neural Information Processing Systems29, 3504–3512 (Curran Associates, Inc., Barcelona, Spain, 2016).

    Not in our library

77. 77.
    Overview of BioCreative II gene mention recognition · 2008

    No notice
78. 78.

    Rajpurkar, P. et al. CheXNet: Radiologist-level pneumonia detection on chest X-rays with deep learning. Preprint at, https://arxiv.org/abs/1711.05225 (2017).

    Not in our library

Cited by papers we hold 7

• Bias in medical AI: Implications for clinical decision-making · 2024
  No notice
• A comparison of approaches to improve worst-case predictive model performance over patient subpopulations · 2022
  Not checked
• A systematic review of the prediction of hospital length of stay: Towards a unified framework · 2022
  No notice
• A New Hybrid Predictive Model to Predict the Early Mortality Risk in Intensive Care Units on a Highly Imbalanced Dataset · 2020
  Not checked
• Combining structured and unstructured data for predictive models: a deep learning approach · 2020
  No notice
• Deep learning and alternative learning strategies for retrospective real-world clinical data · 2019
  No notice
• Scalable and accurate deep learning with electronic health records · 2018
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.