Deep learning for universal linear embeddings of nonlinear dynamics

Abstract

Identifying coordinate transformations that make strongly nonlinear dynamics approximately linear has the potential to enable nonlinear prediction, estimation, and control using linear theory. The Koopman operator is a leading data-driven embedding, and its eigenfunctions provide intrinsic coordinates that globally linearize the dynamics. However, identifying and representing these eigenfunctions has proven challenging. This work leverages deep learning to discover representations of Koopman eigenfunctions from data. Our network is parsimonious and interpretable by construction, embedding the dynamics on a low-dimensional manifold. We identify nonlinear coordinates on which the dynamics are globally linear using a modified auto-encoder. We also generalize Koopman representations to include a ubiquitous class of systems with continuous spectra. Our framework parametrizes the continuous frequency using an auxiliary network, enabling a compact and efficient embedding, while connecting our models to decades of asymptotics. Thus, we benefit from the power of deep learning, while retaining the physical interpretability of Koopman embeddings.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 56

1. 1.

   Guckenheimer, J. & Holmes, P. Nonlinear Oscillations, Dynamical Systems, and Bifurcations of Vector Fields, Vol. 42. Applied Mathematical Sciences (Springer-Verlag New York 1983).

   Not in our library

2. 2.

   Cross MC, Hohenberg PC. Pattern formation outside of equilibrium. Rev. Mod. Phys. 1993;65:851-1112.

   Not in our library10.1103/revmodphys.65.851

3. 3.

   Dullerud GE, Paganini F. A Course in Robust Control Theory: A Cconvex Approach. Texts in Applied Mathematics. 2000.

   Not in our library

4. 4.

   Koopman BO. Hamiltonian systems and transformation in Hilbert space. Proc. Natl Acad. Sci. USA. 1931;17:315-318.

   Not in our library10.1073/pnas.17.5.315

5. 5.

   Koopman B, Neumann Jv. Dynamical systems of continuous spectra. Proc. Natl Acad. Sci. USA. 1932;18:255-263.

   Not in our library10.1073/pnas.18.3.255

6. 6.

   Mezić I, Banaszuk A. Comparison of systems with complex behavior. Phys. D. 2004;197:101-133.

   Not in our library10.1016/j.physd.2004.06.015

7. 7.

   Mezić I. Spectral properties of dynamical systems, model reduction and decompositions. Nonlinear Dyn. 2005;41:309-325.

   Not in our library10.1007/s11071-005-2824-x

8. 8.

   Budišić M, Mohr R, Mezić I. Applied Koopmanism a). Chaos. 2012;22:047510.

   Not in our library10.1063/1.4772195

9. 9.

   Mezic I. Analysis of fluid flows via spectral properties of the Koopman operator. Annu. Rev. Fluid. Mech. 2013;45:357-378.

   Not in our library10.1146/annurev-fluid-011212-140652

10. 10.

    Mezić, I. Spectral Operator Methods in Dynamical Systems: Theory and Applications (Springer, New York, NY 2017).

    Not in our library

11. 11.

    Schmid PJ. Dynamic mode decomposition of numerical and experimental data. J. Fluid Mech. 2010;656:5-28.

    Not in our library10.1017/s0022112010001217

12. 12.

    Rowley CW, Mezić I, Bagheri S, Schlatter P, Henningson D. Spectral analysis of nonlinear flows. J. Fluid Mech. 2009;645:115-127.

    Not in our library10.1017/s0022112009992059

13. 13.

    Kutz, J. N., Brunton, S. L., Brunton, B. W. & Proctor, J. L. Dynamic Mode Decomposition: Data-Driven Modeling of Complex Systems (Society for Industrial and Applied Mathematics, Philadelphia, PA 2016).

    Not in our library

14. 14.

    Hubel DH, Wiesel TN. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. J. Physiol. 1962;160:106-154.

    Not in our library10.1113/jphysiol.1962.sp006837

15. 15.

    Fukushima F. A self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biol. Cybern. 1980;36:193-202.

    Not in our library10.1007/bf00344251

16. 16.

    Cybenko G. Approximation by superpositions of a sigmoidal function. Math. Control Signals Syst. 1989;2:303-314.

    Not in our library10.1007/bf02551274

17. 17.

    Hornik K, Stinchcombe M, White H. Multilayer feedforward networks are universal approximators. Neural Netw. 1989;2:359-366.

    Not in our library10.1016/0893-6080(89)90020-8

18. 18.

    Hornik K, Stinchcombe M, White H. Universal approximation of an unknown mapping and its derivatives using multilayer feedforward networks. Neural Netw. 1990;3:551-560.

    Not in our library10.1016/0893-6080(90)90005-6

19. 19.

    Krizhevsky, A., Sutskever, I. & Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Advances inNeural Information Processing Systems, (eds F. Pereira and C. J. C. Burges and L. Bottou and K. Q. Weinberger) 1097–1105 (Curran Associates, Inc. 2012). https://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks

    Not in our library

20. 20.
    Deep learning · 2015

    No notice
21. 21.

    Goodfellow, I., Bengio, Y. & Courville, A. Deep Learning (MIT Press, Cambridge, MA 2016).

    Not in our library

22. 22.

    Yosinski, J., Clune, J., Bengio, Y. & Lipson, H. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems, (eds Z. Ghahramani and M. Welling and C. Cortes and N. D. Lawrence and K. Q. Weinberger) 3320–3328 (Curran Associates, Inc. 2014). https://papers.nips.cc/paper/5347-how-transferable-are-features-in-deep-neural-networks

    Not in our library

23. 23.

    Bongard J, Lipson H. Automated reverse engineering of nonlinear dynamical systems. Proc. Natl Acad. Sci. USA. 2007;104:9943-9948.

    Not in our library10.1073/pnas.0609476104

24. 24.

    Schmidt M, Lipson H. Distilling free-form natural laws from experimental data. Science. 2009;324:81-85.

    Not in our library10.1126/science.1165893

25. 25.

    Brunton SL, Proctor JL, Kutz JN. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl Acad. Sci. USA. 2016;113:3932-3937.

    Not in our library10.1073/pnas.1517384113

26. 26.

    Gonzalez-Garcia R, Rico-Martinez R, Kevrekidis I. Identification of distributed parameter systems: a neural net based approach. Comp. Chem. Eng. 1998;22:S965-S968.

    Not in our library10.1016/s0098-1354(98)00191-4

27. 27.

    Milano M, Koumoutsakos P. Neural network modeling for near wall turbulent flow. J. Comput. Phys. 2002;182:1-26.

    Not in our library10.1006/jcph.2002.7146

28. 28.

    Wehmeyer C, Noé F. Time-lagged autoencoders: deep learning of slow collective variables for molecular kinetics. J. Chem. Phys. 2018;148:241703.

    Not in our library10.1063/1.5011399

29. 29.
    VAMPnets for deep learning of molecular kinetics · 2018

    Corrected
30. 30.

    Takeishi, N., Kawahara, Y. & Yairi, T. Learning koopman invariant subspaces for dynamic mode decomposition. In Advances in Neural Information Processing Systems, (eds I. Guyon and U. V. Luxburg and S. Bengio and H. Wallach and R. Fergus and S. Vishwanathan and R. Garnett) 1130–1140 (Curran Associates, Inc. 2017). https://papers.nips.cc/paper/6713-learning-koopman-invariant-subspaces-for-dynamic-mode-decomposition

    Not in our library

31. 31.

    Yeung, E., Kundu, S. & Hodas, N. Learning deep neural network representations for Koopman operators of nonlinear dynamical systems. Preprint at http://arxiv.org/abs/1708.06850 (2017).

    Not in our library

32. 32.

    Otto, S. E. & Rowley, C. W. Linearly-recurrent autoencoder networks for learning dynamics. Preprint at http://arxiv.org/abs/1712.01378 (2017).

    Not in our library

33. 33.

    Li Q, Dietrich F, Bollt EM, Kevrekidis IG. Extended dynamic mode decomposition with dictionary learning: a data-driven adaptive spectral decomposition of the koopman operator. Chaos. 2017;27:103111.

    Not in our library10.1063/1.4993854

34. 34.

    Noé F, Nuske F. A variational approach to modeling slow processes in stochastic dynamical systems. Multiscale Model. Simul. 2013;11:635-655.

    Not in our library10.1137/110858616

35. 35.

    Nüske F, Keller BG, Pérez-Hernández G, Mey AS, Noé F. Variational approach to molecular kinetics. J. Chem. Theory Comput. 2014;10:1739-1752.

    Not in our library10.1021/ct4009156

36. 36.

    Nüske F, Schneider R, Vitalini F, Noé F. Variational tensor approach for approximating the rare-event kinetics of macromolecular systems. J. Chem. Phys. 2016;144:054105.

    Not in our library10.1063/1.4940774

37. 37.

    Williams MO, Kevrekidis IG, Rowley CW. A data-driven approximation of the Koopman operator: extending dynamic mode decomposition. J. Nonlin. Sci. 2015;6:1307-1346.

    Not in our library10.1007/s00332-015-9258-5

38. 38.

    Klus S, et al. Data-driven model reduction and transfer operator approximation. J. Nonlin. Sci. 2018;28:985-1010.

    Not in our library10.1007/s00332-017-9437-7

39. 39.

    Korda M, Mezić I. Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control. Automatica. 2018;93:149-160.

    Not in our library10.1016/j.automatica.2018.03.046

40. 40.

    Brunton SL, Brunton BW, Proctor JL, Kutz JN. Koopman invariant subspaces and finite linear representations of nonlinear dynamical systems for control. PLoS One. 2016;11:e0150171.

    Not in our library10.1371/journal.pone.0150171

41. 41.

    Williams MO, Rowley CW, Kevrekidis IG. A kernel approach to data-driven Koopman spectral analysis. J. Comput. Dyn. 2015;2:247-265.

    Not in our library

42. 42.

    Kaiser, E., Kutz, J. N. & Brunton, S. L. Data-driven discovery of Koopman eigenfunctions for control. Preprint at http://arxiv.org/abs/1707.01146 (2017).

    Not in our library

43. 43.

    Bender Carl M., Orszag Steven A. Perturbation Series. Advanced Mathematical Methods for Scientists and Engineers I. 1999;319-367.

    Not in our library

44. 44.

    Kevorkian J., Cole J. D. Perturbation Methods in Applied Mathematics. 1981.

    Not in our library

45. 45.

    Giannakis D, Majda AJ. Nonlinear laplacian spectral analysis for time series with intermittency and low-frequency variability. Proc. Natl Acad. Sci. USA. 2012;109:2222-2227.

    Not in our library10.1073/pnas.1118984109

46. 46.
    Chaos as an intermittently forced linear system · 2017

    No notice
47. 47.

    Arbabi H, Mezić I. Ergodic theory, dynamic mode decomposition and computation of spectral properties of the koopman operator. SIAM J. Appl. Dyn. Syst. 2017;16:2096-2126.

    Not in our library10.1137/17m1125236

48. 48.

    Tu JH, Rowley CW, Luchtenburg DM, Brunton SL, Kutz JN. On dynamic mode decomposition: theory and applications. J. Comput. Dyn. 2014;1:391-421.

    Not in our library10.3934/jcd.2014.1.391

49. 49.

    Mezic, I. Koopman operator spectrum and data analysis. Preprint at http://arxiv.org/abs/1702.07597 (2017).

    Not in our library

50. 50.

    Noack BR, Afanasiev K, Morzynski M, Tadmor G, Thiele F. A hierarchy of low-dimensional models for the transient and post-transient cylinder wake. J. Fluid Mech. 2003;497:335-363.

    Not in our library10.1017/s0022112003006694

51. 51.

    Loiseau JC, Brunton SL. Constrained sparse Galerkin regression. J. Fluid Mech. 2018;838:42-67.

    Not in our library10.1017/jfm.2017.823

52. 52.

    Bagheri S. Koopman-mode decomposition of the cylinder wake. J. Fluid Mech. 2013;726:596-623.

    Not in our library10.1017/jfm.2013.249

53. 53.

    Cui, H., Zhang, H., Ganger, G. R., Gibbons, P. B. & Xing, E. P. Geeps: Scalable deep learning on distributed gpus with a gpu-specialized parameter server. In Proc. 11th European Conference on Computer Systems, Vol. 4 (ACM New York, NY, USA 2016). https://dl.acm.org/citation.cfm?id=2901318

    Not in our library

54. 54.

    Abadi, M. et al. TensorFlow: Large-scale Machine Learning on Heterogeneous Systems. tensorflow.org (2015).

    Not in our library

55. 55.

    Mallat S. Understanding deep convolutional networks. Philos. Trans. R. Soc. A. 2016;374:20150203.

    Not in our library10.1098/rsta.2015.0203

56. 56.

    Kingma, D. & Ba, J. Adam: a method for stochastic optimization. Preprint at http://arxiv.org/abs/1412.6980 (2014).

    Not in our library

Cited by papers we hold 6

• Dynamic mode decomposition for analysis and prediction of metabolic oscillations from time-lapse imaging of cellular autofluorescence · 2025
  Not checked
• Machine Learning and Its Application to Reacting Flows · 2023
  Not checked
• Deep learning the structural determinants of protein biochemical properties by comparing structural ensembles with DiffNets · 2021
  Not checked
• Unsupervised Learning Methods for Molecular Simulation Data · 2021
  No notice
• A Unified Sparse Optimization Framework to Learn Parsimonious Physics-Informed Models From Data · 2020
  Not checked
• Data-driven discovery of coordinates and governing equations · 2019
  No notice

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.