In recent years, the artificial intelligence community has seen a continuous interest in research aimed at investigating dynamical aspects of both training procedures and machine learning models. Of particular interest among recurrent neural networks, we have the Reservoir Computing (RC) paradigm characterized by conceptual simplicity and a fast training scheme. Yet, the guiding principles under which RC operates are only partially understood. In this work, we analyze the role played by Generalized Synchronization (GS) when training a RC to solve a generic task. In particular, we show how GS allows the reservoir to correctly encode the system generating the input signal into its dynamics. We also discuss necessary and sufficient conditions for the learning to be feasible in this approach. Moreover, we explore the role that ergodicity plays in this process, showing how its presence allows the learning outcome to apply to multiple input trajectories. Finally, we show that satisfaction of the GS can be measured by means of the mutual false nearest neighbors index, which makes effective to practitioners theoretical derivations.
REFERENCES
1.
G.-H.
Liu
and E. A.
Theodorou
, “Deep learning theory review: An optimal control and dynamical systems perspective,” arXiv:1908.10920 (2019).2.
F. M.
Bianchi
, L.
Livi
, and C.
Alippi
, “Investigating echo state networks dynamics by means of recurrence analysis
,” IEEE Trans. Neural Netw. Learn. Syst.
29
, 427
–439
(2018
). 3.
D.
Sussillo
and L. F.
Abbott
, “Generating coherent patterns of activity from chaotic neural networks
,” Neuron
63
, 544
–557
(2009
). 4.
Y.
Bengio
, P.
Simard
, and P.
Frasconi
, “Learning long-term dependencies with gradient descent is difficult
,” IEEE Trans. Neural Networks
5
, 157
–166
(1994
). 5.
J.
Bouvrie
and B.
Hamzi
, “Kernel methods for the approximation of nonlinear systems
,” SIAM J. Control Optim.
55
, 2460
–2492
(2017
). 6.
D.
Qi
and A. J.
Majda
, “Using machine learning to predict extreme events in complex systems
,” Proc. Natl. Acad. Sci. U.S.A.
117
, 52
–59
(2020
). 7.
W.
Gilpin
, “Deep learning of dynamical attractors from time series measurements,” arXiv:2002.05909 (2020).8.
J. H.
Tu
, C. W.
Rowley
, D. M.
Luchtenburg
, S. L.
Brunton
, and J. N.
Kutz
, “On dynamic mode decomposition: Theory and applications,” arXiv:1312.0041 (2013).9.
T.
Berry
, D.
Giannakis
, and J.
Harlim
, “Bridging data science and dynamical systems theory,” arXiv:2002.07928 (2020).10.
D.
Verstraeten
, B.
Schrauwen
, M.
d’Haene
, and D.
Stroobandt
, “An experimental unification of reservoir computing methods
,” Neural Netw.
20
, 391
–403
(2007
). 11.
Z.
Lu
, J.
Pathak
, B.
Hunt
, M.
Girvan
, R.
Brockett
, and E.
Ott
, “Reservoir observers: Model-free inference of unmeasured variables in chaotic systems
,” Chaos
27
, 041102
(2017
). 12.
Z.
Lu
, B. R.
Hunt
, and E.
Ott
, “Attractor reconstruction by machine learning
,” Chaos
28
, 061104
(2018
). 13.
A.
Chattopadhyay
, P.
Hassanzadeh
, and D.
Subramanian
, “Data-driven predictions of a multiscale lorenz 96 chaotic system using machine-learning methods: Reservoir computing, artificial neural network, and long short-term memory network
,” Nonlinear Process. Geophys.
27
, 373
–389
(2020
). 14.
P. R.
Vlachas
, J.
Pathak
, B. R.
Hunt
, T. P.
Sapsis
, M.
Girvan
, E.
Ott
, and P.
Koumoutsakos
, “Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics
,” Neural Netw.
126
, 191
–217
(2020
). 15.
S.
Bompas
, B.
Georgeot
, and D.
Guéry-Odelin
, “Accuracy of neural networks for the simulation of chaotic dynamics: Precision of training data vs precision of the algorithm,” arXiv:2008.04222 (2020).16.
H.
Jaeger
, “The ‘echo state’ approach to analysing and training recurrent neural networks—With an erratum note,” GMD Technical Report 148, Ger. Natl. Res. Center Inf. Technol., Bonn, Germany, 2001, p. 13.17.
W.
Maass
, T.
Natschläger
, and H.
Markram
, “Real-time computing without stable states: A new framework for neural computation based on perturbations
,” Neural Comput.
14
, 2531
–2560
(2002
). 18.
P.
Tiňo
and G.
Dorffner
, “Predicting the future of discrete sequences from fractal representations of the past
,” Mach. Learn.
45
, 187
–217
(2001
). 19.
L.
Grigoryeva
and J.-P.
Ortega
, “Echo state networks are universal
,” Neural Netw.
108
, 495
–508
(2018
). 20.
A.
Hart
, J.
Hook
, and J.
Dawes
, “Embedding and approximation theorems for echo state networks
,” Neural Netw.
128
, 234
–247
(2020
). 21.
A. G.
Hart
, J. L.
Hook
, and J. H.
Dawes
, “Echo state networks trained by Tikhonov least squares are l2() approximators of ergodic dynamical systems,” arXiv:2005.06967 (2020).22.
L.
Gonon
, L.
Grigoryeva
, and J.-P.
Ortega
, “Memory and forecasting capacities of nonlinear recurrent networks
,” Physica D
414
, 132721
(2020
). 23.
M.
Massar
and S.
Massar
, “Mean-field theory of echo state networks
,” Phys. Rev. E
87
, 042809
(2013
). 24.
F.
Mastrogiuseppe
and S.
Ostojic
, “A geometrical analysis of global stability in trained feedback networks
,” Neural Comput.
31
, 1139
–1182
(2019
). 25.
A.
Rivkind
and O.
Barak
, “Local dynamics in trained recurrent neural networks
,” Phys. Rev. Lett.
118
, 258101
(2017
). 26.
D.
Verstraeten
, J.
Dambre
, X.
Dutoit
, and B.
Schrauwen
, “Memory versus non-linearity in reservoirs,” in The 2010 International Joint Conference on Neural Networks (IJCNN) (IEEE, 2010), pp. 1–8.27.
A.
Goudarzi
, S.
Marzen
, P.
Banda
, G.
Feldman
, C.
Teuscher
, and D.
Stefanovic
, “Memory and information processing in recurrent neural networks,” arXiv:1604.06929 (2016).28.
S.
Marzen
, “Difference between memory and prediction in linear recurrent networks
,” Phys. Rev. E
96
, 032308
(2017
). 29.
P.
Tiňo
, “Dynamical systems as temporal feature spaces
,” J. Mach. Learn. Res.
21
, 1
–42
(2020
).30.
P.
Verzelli
, C.
Alippi
, L.
Livi
, and P.
Tino
, “Input representation in recurrent neural networks dynamics,” arXiv:2003.10585 (2020).31.
S.
Ganguli
, D.
Huh
, and H.
Sompolinsky
, “Memory traces in dynamical systems
,” Proc. Natl. Acad. Sci. U.S.A.
105
, 18970
–18975
(2008
). 32.
G.
Tanaka
, T.
Yamane
, J. B.
Héroux
, R.
Nakane
, N.
Kanazawa
, S.
Takeda
, H.
Numata
, D.
Nakano
, and A.
Hirose
, “Recent advances in physical reservoir computing: A review
,” Neural Netw.
115
, 100
–123
(2019
). 33.
I. B.
Yildiz
, H.
Jaeger
, and S. J.
Kiebel
, “Re-visiting the echo state property
,” Neural Netw.
35
, 1
–9
(2012
). 34.
B.
Zhang
, D. J.
Miller
, and Y.
Wang
, “Nonlinear system modeling with random matrices: Echo state networks revisited
,” IEEE Trans. Neural Netw. Learn. Syst.
23
, 175
–182
(2011
). 35.
S.
Basterrech
, “Empirical analysis of the necessary and sufficient conditions of the echo state property,” in 2017 International Joint Conference on Neural Networks (IJCNN) (IEEE, 2017), pp. 888–896.36.
G.
Manjunath
and H.
Jaeger
, “Echo state property linked to an input: Exploring a fundamental characteristic of recurrent neural networks
,” Neural Comput.
25
, 671
–696
(2013
). 37.
K.
Caluwaerts
, F.
Wyffels
, S.
Dieleman
, and B.
Schrauwen
, “The spectral radius remains a valid indicator of the echo state property for large reservoirs,” in The 2013 International Joint Conference on Neural Networks (IJCNN) (IEEE, 2013), pp. 1–6.38.
A.
Ceni
, P.
Ashwin
, L.
Livi
, and C.
Postlethwaite
, “The echo index and multistability in input-driven recurrent neural networks
,” Physica D
412
, 132609
(2020
). 39.
Z.
Lu
and D. S.
Bassett
, “Invertible generalized synchronization: A putative mechanism for implicit learning in neural systems
,” Chaos
30
, 063133
(2020
). 40.
T.
Weng
, H.
Yang
, C.
Gu
, J.
Zhang
, and M.
Small
, “Synchronization of chaotic systems and their machine-learning models
,” Phys. Rev. E
99
, 042203
(2019
). 41.
T.
Lymburn
, D. M.
Walker
, M.
Small
, and T.
Jüngling
, “The reservoir’s perspective on generalized synchronization
,” Chaos
29
, 093133
(2019
). 42.
L.
Grigoryeva
, A.
Hart
, and J.-P.
Ortega
, “Chaos on compact manifolds: Differentiable synchronizations beyond the Takens theorem,” arXiv:2010.03218 (2020).43.
V.
Afraimovich
, N.
Verichev
, and M. I.
Rabinovich
, “Stochastic synchronization of oscillation in dissipative systems
,” Radiophys. Quantum Electron.
29
, 795
–803
(1986
). 44.
N. F.
Rulkov
, M. M.
Sushchik
, L. S.
Tsimring
, and H. D.
Abarbanel
, “Generalized synchronization of chaos in directionally coupled chaotic systems
,” Phys. Rev. E
51
, 980
(1995
). 45.
L. M.
Pecora
, T. L.
Carroll
, G. A.
Johnson
, D. J.
Mar
, and J. F.
Heagy
, “Fundamentals of synchronization in chaotic systems, concepts, and applications
,” Chaos
7
, 520
–543
(1997
). 46.
U.
Parlitz
, “Detecting generalized synchronization
,” Nonlinear Theory Appl. IEICE
3
, 113
–127
(2012
). 47.
S.
Boccaletti
, J.
Kurths
, G.
Osipov
, D.
Valladares
, and C.
Zhou
, “The synchronization of chaotic systems
,” Phys. Rep.
366
, 1
–101
(2002
). 48.
We choose this term—following Ref. 11—to avoid the possible ambiguity between the testing and validation phases typically used in machine learning tasks, since this distinction is not well-defined in this context.
49.
In fact, the theory also applies to continuous-time models.
50.
We implicitly assume that is characterized by the same dynamics in both phases, implying some form a stationarity of the source system. Otherwise, the learning would be unfeasible without proper adaptation mechanisms to changes in the driving input. Namely, we are assuming that the source system (1) has reached its attractor in the listening phase and that it will continue to stay on it.
51.
This assumption is required just to simplify the exposition. For the case where the system has multiple attractors, see, for instance, Ref. 39.
52.
G.
Manjunath
, P.
Tino
, and H.
Jaeger
, “Theory of input driven dynamical systems,” in ESANN 2012 Proceedings, European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, Bruges, 25–27 April 2012 (2012
).53.
K.
Pyragas
, “Weak and strong synchronization of chaos
,” Phys. Rev. E
54
, R4508
(1996
). 54.
F.
Takens
, “Detecting strange attractors in turbulence,” in Dynamical Systems and Turbulence (Springer, 1981), pp. 366–381.56.
J.
Shawe-Taylor
and N.
Cristianini
, Kernel Methods for Pattern Analysis
(Cambridge University Press
, Cambridge
, 2004
).57.
Z.
Shi
and M.
Han
, “Support vector echo-state machine for chaotic time-series prediction
,” IEEE Trans. Neural Netw.
18
, 359
–372
(2007
). 58.
S.
Shalev-Shwartz
and S.
Ben-David
, Understanding Machine Learning: From Theory to Algorithms
(Cambridge University Press
, 2014
).59.
G. D.
Birkhoff
, “Proof of the ergodic theorem
,” Proc. Natl. Acad. Sci. U.S.A.
17
, 656
–660
(1931
). 60.
Note that different choices can be made for and the results do not depend on its particular form. We use the RMSE because it is the one we use in the Experimental results section.
61.
J.
Arnhold
, P.
Grassberger
, K.
Lehnertz
, and C. E.
Elger
, “A robust method for detecting interdependences: Application to intracranially recorded EEG
,” Physica D
134
, 419
–430
(1999
). 62.
A.
Čenys
, G.
Lasiene
, and K.
Pyragas
, “Estimation of interrelation between chaotic observables
,” Physica D
52
, 332
–337
(1991
). 63.
J. A.
Platt
, A. S.
Wong
, R.
Clark
, S. G.
Penny
, and H. D.
Abarbanel
, “Forecasting using reservoir computing: The role of generalized synchronization,” arXiv:2103.00362 (2021).64.
C.
Gallicchio
, A.
Micheli
, and L.
Pedrelli
, “Deep reservoir computing: A critical experimental analysis
,” Neurocomputing
268
, 87
–99
(2017
). 65.
S.
Løkse
, F. M.
Bianchi
, and R.
Jenssen
, “Training echo state networks with regularization through dimensionality reduction
,” Cognit. Comput.
9
, 364
–378
(2017
). 66.
T.
Lymburn
, A.
Khor
, T.
Stemler
, D. C.
Corrêa
, M.
Small
, and T.
Jüngling
, “Consistency in echo-state networks
,” Chaos
29
, 023118
(2019
). 67.
E.
Ott
, Chaos in Dynamical Systems
(Cambridge University Press
, 2002
).68.
L. M.
Pecora
and T. L.
Carroll
, “Synchronization in chaotic systems
,” Phys. Rev. Lett.
64
, 821
(1990
). 69.
L.
Kocarev
and U.
Parlitz
, “Generalized synchronization, predictability, and equivalence of unidirectionally coupled dynamical systems
,” Phys. Rev. Lett.
76
, 1816
(1996
). 70.
E. N.
Lorenz
, “Deterministic nonperiodic flow
,” J. Atmos. Sci.
20
, 130
–141
(1963
). 71.
O. E.
Rössler
, “An equation for continuous chaos
,” Phys. Lett. A
57
, 397
–398
(1976
). © 2021 Author(s). Published under an exclusive license by AIP Publishing.
2021
Author(s)
You do not currently have access to this content.
