Next generation reservoir computing

Abstract

Reservoir computing is a best-in-class machine learning algorithm for processing information generated by dynamical systems using observed time-series data. Importantly, it requires very small training data sets, uses linear optimization, and thus requires minimal computing resources. However, the algorithm uses randomly sampled matrices to define the underlying recurrent neural network and has a multitude of metaparameters that must be optimized. Recent results demonstrate the equivalence of reservoir computing to nonlinear vector autoregression, which requires no random matrices, fewer metaparameters, and provides interpretable results. Here, we demonstrate that nonlinear vector autoregression excels at reservoir computing benchmark tasks and requires even shorter training data sets and training time, heralding the next generation of reservoir computing.

Full text

Its licence does not allow us to show the full text. Read it at PubMed Central:

References 38

1. 1.

   Jaeger H, Haas H. Harnessing nonlinearity: predicting chaotic systems and saving energy in wireless communication. Science. 2004;304:78-80.

   Not in our library10.1126/science.1091277

2. 2.

   Maass W, Natschläger T, Markram H. Real-time computing without stable states: a new framework for neural computation based on perturbations. Neural Comput. 2002;14:2531-2560.

   Not in our library10.1162/089976602760407955

3. 3.

   Pathak J, Lu Z, Hunt BR, Girvan M, Ott E. Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data. Chaos. 2017;27:121102.

   Not in our library10.1063/1.5010300

4. 4.

   Pathak J, Hunt B, Girvan M, Lu Z, Ott E. Model-free prediction of large spatiotemporally chaotic systems from data: a reservoir computing approach. Phys. Rev. Lett. 2018;120:24102.

   Not in our library10.1103/physrevlett.120.024102

5. 5.

   Bengio, Y., Boulanger-Lewandowski, N. & Pascanu, R. Advances in optimizing recurrent networks. 2013 IEEE International Conference on Acoustics, Speech and Signal Processing, 2013, pp. 8624–8628 10.1109/ICASSP.2013.6639349 (2013).

   Not in our library

6. 6.

   Vogel, C. R. Computational Methods for Inverse Problems (Society for Industrial and Applied Mathematics, 2002).

   Not in our library

7. 7.

   Vlachas PR, et al. Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics. Neural Netw. 2020;126:191-217.

   Not in our library10.1016/j.neunet.2020.02.016

8. 8.

   Bompas S, Georgeot B, Guéry-Odelin D. Accuracy of neural networks for the simulation of chaotic dynamics: precision of training data vs precision of the algorithm. Chaos. 2020;30:113118.

   Not in our library10.1063/5.0021264

9. 9.

   Yperman, J. & Becker, T. Bayesian optimization of hyper-parameters in reservoir computing. Preprint at arXiv:1611.0519 (2016).

   Not in our library

10. 10.

    Livi L, Bianchi FM, Alippi C. Determination of the edge of criticality in echo state networks through fisher information maximization. IEEE Trans. Neural Netw. Learn. Syst. 2018;29:706-717.

    Not in our library10.1109/tnnls.2016.2644268

11. 11.

    Thiede LA, Parlitz U. Gradient based hyperparameter optimization in echo state networks. Neural Netw. 2019;115:23-29.

    Not in our library10.1016/j.neunet.2019.02.001

12. 12.

    Griffith A, Pomerance A, Gauthier DJ. Forecasting chaotic systems with very low connectivity reservoir computers. Chaos. 2019;29:123108.

    Not in our library10.1063/1.5120710

13. 13.

    Antonik P, Marsal N, Brunner D, Rontani D. Bayesian optimisation of large-scale photonic reservoir computers. Cogn. Comput. 2021;2021:1-9.

    Not in our library

14. 14.

    Lu Z, Hunt BR, Ott E. Attractor reconstruction by machine learning. Chaos. 2018;28:061104.

    Not in our library10.1063/1.5039508

15. 15.

    Platt, J. A., Wong, A. S., Clark, R., Penny, S. G. & Abarbanel, H. D. I. Robust forecasting through generalized synchronization in reservoir computing. Preprint at arXiv:2103.0036 (2021).

    Not in our library10.1063/5.0066013

16. 16.

    Gonon L, Ortega JP. Reservoir computing universality with stochastic inputs. IEEE Trans. Neural Netw. Learn. Syst. 2020;31:100-112.

    Not in our library10.1109/tnnls.2019.2899649

17. 17.

    Hart AG, Hook JL, Dawes JHP. Echo state networks trained by Tikhonov least squares are L2(μ) approximators of ergodic dynamical systems. Phys. D. Nonlinear Phenom. 2021;421:132882.

    Not in our library10.1016/j.physd.2021.132882

18. 18.

    Bollt E. On explaining the surprising success of reservoir computing forecaster of chaos? The universal machine learning dynamical system with contrast to VAR and DMD. Chaos. 2021;31:013108.

    Not in our library10.1063/5.0024890

19. 19.

    Gauthier, D. J. Reservoir computing: harnessing a universal dynamical system. SIAM News51, 12 (2018).

    Not in our library

20. 20.

    Franz MO, Schölkopf B. A unifying view of Wiener and Volterra theory and polynomial kernel regression. Neural. Comput. 2006;18:3097-3118.

    Not in our library10.1162/neco.2006.18.12.3097

21. 21.

    Billings, S. A. Nonlinear System Identification (John Wiley & Sons, Ltd., 2013).

    Not in our library

22. 22.

    Kim JZ, Lu Z, Nozari E, Papas GJ, Bassett DS. Teaching recurrent neural networks to infer global temporal structure from local examples. Nat. Mach. Intell. 2021;3:316-323.

    Not in our library10.1038/s42256-021-00321-2

23. 23.

    Lorenz EN. Deterministic nonperiodic flow. J. Atmos. Sci. 1963;20:130-141.

    Not in our library10.1175/1520-0469(1963)020<0130:dnf>2.0.co;2

24. 24.

    Chang, A., Bienfang, J. C., Hall, G. M., Gardner, J. R. & Gauthier, D. J. Stabilizing unstable steady states using extended time-delay autosynchronization. Chaos8, 782–790 (1998).

    Not in our library10.1063/1.166357

25. 25.

    Crutchfield JP, McNamara BS. Equations of motion from a data series. Complex Sys. 1987;1:417-452.

    Not in our library

26. 26.

    Wang W-X, Lai Y-C, Grebogi C, Ye J. Network reconstruction based on evolutionary-game data via compressive sensing. Phys. Rev. X. 2011;1:021021.

    Not in our library

27. 27.

    Brunton SL, Proctor JL, Kutz JN, Bialek W. Discovering governing equations from data by sparse identification of nonlinear dynamical systems. Proc. Natl Acad. Sci. USA. 2016;113:3932-3937.

    Not in our library10.1073/pnas.1517384113

28. 28.

    Lai, Y.-C. Finding nonlinear system equations and complex network structures from data: a sparse optimization approach. Chaos 31, 082101 (2021).

    Not in our library10.1063/5.0062042

29. 29.

    Takens, F. Detecting strange attractors in turbulence. In Dynamical Systems and Turbulence, Warwick 1980 (eds Rand, D. & Young, L. S.) 366–381 (Springer, 1981).

    Not in our library

30. 30.

    Pyle R, Jovanovic N, Subramanian D, Palem KV, Patel AB. Domain-driven models yield better predictions at lower cost than reservoir computers in Lorenz systems. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 2021;379:24102.

    Not in our library10.1098/rsta.2020.0246

31. 31.

    Carleman T. Application de la théorie des équations intégrales linéares aux d’équations différentielles non linéares. Acta Math. 1932;59:63-87.

    Not in our library10.1007/bf02546499

32. 32.

    Almomani AAR, Sun J, Bollt E. How entropic regression beats the outliers problem in nonlinear system identification. Chaos. 2020;30:013107.

    Not in our library10.1063/1.5133386

33. 33.

    Grieves, M. W. Virtually Intelligent Product Systems: Digital and Physical Twins. In Complex Systems Engineering: Theory and Practice (eds Flumerfelt, S., et al.) 175–200 (American Institute of Aeronautics and Astronautics, Inc., 2019).

    Not in our library

34. 34.

    Wikner A, et al. Combining machine learning with knowledge-based modeling for scalable forecasting and subgrid-scale closure of large, complex, spatiotemporal systems. Chaos. 2020;30:053111.

    Not in our library10.1063/5.0005541

35. 35.

    Wikner A, et al. Using data assimilation to train a hybrid forecast system that combines machine-learning and knowledge-based components. Chaos. 2021;31:053114.

    Not in our library10.1063/5.0048050

36. 36.

    Canaday, D., Pomerance, A. & Gauthier, D. J. Model-free control of dynamical systems with deep reservoir computing. to appear in J. Phys. Complex. http://iopscience.iop.org/article/10.1088/2632-072X/ac24f3 (2021).

    Not in our library

37. 37.

    Li, Z. et al. Fourier neural operator for parametric partial differential equations. Preprint at arXiv:2010.08895 (2020). In The International Conference on Learning Representations (ICLR 2021).

    Not in our library

38. 38.

    Gauthier, D. J., Griffith, A. & de sa Barbosa, W. ng-rc-paper-code repository. 10.5281/zenodo.5218954 (2021).

    Not in our library

Cited by papers we hold 2

• Modeling nonlinear oscillator networks using physics-informed hybrid reservoir computing · 2025
  Not checked
• Optical next generation reservoir computing · 2025
  Not checked

Reads

A person counts once a day per paper. We keep no cookie and no address: a reader is a hash that changes every day. Any month under five reads shows no number.