Recurrent Neural Networks for Multivariate Time Series with Missing Values

Abstract

Multivariate time series data in practical applications, such as health care, geoscience, and biology, are characterized by a variety of missing values. In time series prediction and other related tasks, it has been noted that missing values and their missing patterns are often correlated with the target labels, a.k.a., informative missingness. There is very limited work on exploiting the missing patterns for effective imputation and improving prediction performance. In this paper, we develop novel deep learning models, namely GRU - D , as one of the early attempts. GRU-D is based on Gated Recurrent Unit (GRU), a state-of-the-art recurrent neural network. It takes two representations of missing patterns, i.e., masking and time interval , and effectively incorporates them into a deep model architecture so that it not only captures the long-term temporal dependencies in time series, but also utilizes the missing patterns to achieve better prediction results. Experiments of time series classification tasks on real-world clinical datasets (MIMIC-III, PhysioNet) and synthetic datasets demonstrate that our models achieve state-of-the-art performance and provide useful insights for better understanding and utilization of missing values in time series analysis.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 46

1. 1.

   Rubin DB. Inference and missing data. Biom. 1976;63:581-592.

   Not in our library

2. 2.
   MIMIC-III, a freely accessible critical care database · 2016

   No notice
3. 3.

   Schafer, J. L. & Graham, J. W. Missing data: our view of the state of the art. Psychol. methods (2002).

   Not in our library

4. 4.

   Kreindler, D. M. & Lumsden, C. J. The effects of the irregular sample and missing data in time series analysis. Nonlinear Dyn. Syst. Analysis for Behav. Sci. Using Real Data (2012).

   Not in our library

5. 5.

   De Boor C, De Boor C, Mathématicien E-U, De Boor C, De Boor C. A practical guide to splines. 1978.

   Not in our library

6. 6.

   Mondal D, Percival DB. Wavelet variance analysis for gappy time series. Annals Inst. Stat. Math. 2010;62:943-966.

   Not in our library10.1007/s10463-008-0195-z

7. 7.

   Rehfeld, K., Marwan, N., Heitzig, J. & Kurths, J. Comparison of correlation analysis techniques for irregularly sampled time series. Nonlinear Process. Geophys. 18 (2011).

   Not in our library

8. 8.

   Garca-Laencina, P. J., Sancho-Gómez, J.-L. & Figueiras-Vidal, A. R. Pattern classification with missing data: a review. Neural Comput. Appl. 19 (2010).

   Not in our library

9. 9.

   Mazumder R, Hastie T, Tibshirani R. Spectral regularization algorithms for learning large incomplete matrices. J. machine learning research. 2010;11:2287-2322.

   Not in our library

10. 10.

    Koren, Y., Bell, R. & Volinsky, C. Matrix factorization techniques for recommender systems. Comput. 42 (2009).

    Not in our library

11. 11.

    White IR, Royston P, Wood AM. Multiple imputation using chained equations: issues and guidance for practice. Stat. medicine. 2011;30:377-399.

    Not in our library10.1002/sim.4067

12. 12.

    Azur MJ, Stuart EA, Frangakis C, Leaf PJ. Multiple imputation by chained equations: what is it and how does it work? Int. journal methods psychiatric research. 2011;20:40-49.

    Not in our library10.1002/mpr.329

13. 13.

    Wells, B. J., Chagin, K. M., Nowacki, A. S. & Kattan, M. W. Strategies for handling missing data in electronic health record derived data. EGEMS1 (2013).

    Not in our library10.13063/2327-9214.1035

14. 14.

    Hochreiter S, Schmidhuber J. Long short-term memory. Neural computation. 1997;9:1735-1780.

    Not in our library10.1162/neco.1997.9.8.1735

15. 15.

    Cho, K. et al. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP, 1724–1734 (2014).

    Not in our library

16. 16.

    Bahdanau, D., Cho, K. & Bengio, Y. Neural machine translation by jointly learning to align and translate. ICLR (2015).

    Not in our library

17. 17.

    Sutskever, I., Vinyals, O. & Le, Q. V. Sequence to sequence learning with neural networks. In Advances in neural information processing systems, 3104–3112 (2014).

    Not in our library

18. 18.

    Hinton G, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. Signal Process. Mag. IEEE. 2012;29:82-97.

    Not in our library10.1109/msp.2012.2205597

19. 19.

    Bengio, Y. & Gingras, F. Recurrent neural networks for missing or asynchronous data. Adv. neural information processing systems 395–401 (1996).

    Not in our library

20. 20.

    Tresp, V. & Briegel, T. A solution for missing data in recurrent neural networks with an application to blood glucose prediction. NIPS 971–977 (1998).

    Not in our library

21. 21.

    Parveen, S. & Green, P. Speech recognition with missing data using recurrent neural nets. In Advances in Neural Information Processing Systems, 1189–1195 (2001).

    Not in our library

22. 22.

    Lipton, Z. C., Kale, D. & Wetzel, R. Directly modeling missing data in sequences with rnns: Improved classification of clinical time series. In Machine Learning for Healthcare Conference, 253–270 (2016).

    Not in our library

23. 23.

    Choi, E., Bahadori, M. T., Schuetz, A., Stewart, W. F. & Sun, J. Doctor ai: Predicting clinical events via recurrent neural networks. In Machine Learning for Healthcare Conference, 301–318 (2016).

    Not in our library

24. 24.

    Pham, T., Tran, T., Phung, D. & Venkatesh, S. Deepcare: A deep dynamic memory model for predictive medicine. In Advances in Knowledge Discovery and Data Mining, 30–41 (2016).

    Not in our library

25. 25.
    Recurrent Neural Networks for Multivariate Time Series with Missing Values · 2018

    No notice
26. 26.

    Vodovotz, Y., An, G. & Androulakis, I. P. A systems engineering perspective on homeostasis and disease. Front. bioengineering biotechnology1 (2013).

    Not in our library10.3389/fbioe.2013.00006

27. 27.

    Zhou L, Hripcsak G. Temporal reasoning with medical data—a review with emphasis on medical natural language processing. J. biomedical informatics. 2007;40:183-202.

    Not in our library10.1016/j.jbi.2006.12.009

28. 28.

    Batista GE, Monard MC, et al. A study of k-nearest neighbour as an imputation method. HIS. 2002;87:48.

    Not in our library

29. 29.

    Josse J, Husson F. Handling missing values in exploratory multivariate data analysis methods. J. de la Société Française de Stat. 2012;153:79-99.

    Not in our library

30. 30.

    Stekhoven DJ, Bühlmann P. Missforest—non-parametric missing value imputation for mixed-type data. Bioinforma. 2011;28:112-118.

    Not in our library10.1093/bioinformatics/btr597

31. 31.

    Alex Rubinsteyn, S. F. fancyimpute. https://github.com/hammerlab/fancyimpute (2015).

    Not in our library

32. 32.

    English, P. predictive_imputer. https://github.com/log0ymxm/predictive_imputer (2016).

    Not in our library

33. 33.

    Jones, E., Oliphant, T. & Peterson, P. Scipy: Open source scientific tools for python. http://www.scipy.org/ (2001).

    Not in our library

34. 34.

    Pedregosa F, et al. Scikit-learn: Machine learning in Python. J. Mach. Learn. Res. 2011;12:2825-2830.

    Not in our library

35. 35.

    Ioffe, S. & Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning, 448–456 (2015).

    Not in our library

36. 36.

    Srivastava, N., Hinton, G. E., Krizhevsky, A., Sutskever, I. & Salakhutdinov, R. Dropout: a simple way to prevent neural networks from overfitting. JMLR15 (2014).

    Not in our library

37. 37.

    Kingma, D. & Ba, J. Adam: A method for stochastic optimization. ICLR (2015).

    Not in our library

38. 38.

    Chollet, F. et al. Keras. https://github.com/keras-team/keras (2015).

    Not in our library

39. 39.

    Bergstra, J. et al. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy) (2010).

    Not in our library

40. 40.

    Madeo, R. C., Lima, C. A. & Peres, S. M. Gesture unit segmentation using support vector machines: segmenting gestures from rest positions. In SAC (2013).

    Not in our library

41. 41.

    Silva, I., Moody, G., Scott, D. J., Celi, L. A. & Mark, R. G. Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012. In CinC (2012).

    Not in our library

42. 42.

    Gal, Y. & Ghahramani, Z. A theoretically grounded application of dropout in recurrent neural networks. In Advances in Neural Information Processing Systems, 1019–1027 (2016).

    Not in our library

43. 43.

    Che, Z., Kale, D., Li, W., Bahadori, M. T. & Liu, Y. Deep computational phenotyping. In SIGKDD (2015).

    Not in our library

44. 44.

    Purushotham, S., Meng, C., Che, Z. & Liu, Y. Benchmark of deep learning models on large healthcare mimic datasets. arXiv preprint arXiv:1710.08531 (2017).

    Not in our library

45. 45.

    Johnson, A. E., Pollard, T. J. & Mark, R. G. Reproducibility in critical care: a mortality prediction case study. In Machine Learning for Healthcare Conference, 361–376 (2017).

    Not in our library

46. 46.

    Luo, Y.-F. & Rumshisky, A. Interpretable topic features for post-icu mortality prediction. In AMIA Annual Symposium Proceedings, 827 (2016).

    Not in our library

Cited by papers we hold 9

• Speech Emotion Recognition by Late Fusion for Bidirectional Reservoir Computing With Random Projection · 2021
  Not checked
• Real-World Integration of a Sepsis Deep Learning Technology Into Routine Clinical Care: Implementation Study · 2020
  No notice
• Influenza Screening via Deep Learning Using a Combination of Epidemiological and Patient-Generated Health Data: Development and Validation Study · 2020
  Not checked
• A New Hybrid Predictive Model to Predict the Early Mortality Risk in Intensive Care Units on a Highly Imbalanced Dataset · 2020
  Not checked
• Machine Learning‐Based Prediction of Spatiotemporal Uncertainties in Global Wind Velocity Reanalyses · 2020
  Not checked
• Assessment of a Deep Learning Model Based on Electronic Health Record Data to Forecast Clinical Outcomes in Patients With Rheumatoid Arthritis · 2019
  No notice
• Deep learning and alternative learning strategies for retrospective real-world clinical data · 2019
  No notice
• Opportunities and obstacles for deep learning in biology and medicine · 2018
  No notice
• Scalable and accurate deep learning with electronic health records · 2018
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.