Drawing on ergodic theory, we introduce a novel training method for machine learning based forecasting methods for chaotic dynamical systems. The training enforces dynamical invariants—such as the Lyapunov exponent spectrum and the fractal dimension—in the systems of interest, enabling longer and more stable forecasts when operating with limited data. The technique is demonstrated in detail using reservoir computing, a specific kind of recurrent neural network. Results are given for the Lorenz 1996 chaotic dynamical system and a spectral quasi-geostrophic model of the atmosphere, both typical test cases for numerical weather prediction.

1.
H.
Abarbanel
,
Predicting the Future: Completing Models of Complex Systems
(
Springer
,
New York
,
2013
).
2.
S. H.
Strogatz
,
Nonlinear Dynamics and Chaos: With Applications to Physics, Biology, Chemistry and Engineering
(
Westview Press
,
2000
).
3.
E.
Kalnay
,
Atmospheric Modeling, Data Assimilation and Predictability
(
Cambridge University Press
,
2002
).
4.
R. E.
Kalman
, “
A new approach to linear filtering and prediction problems
,”
J. Basic Eng.
82
(
1
),
35
45
(
1960
).
5.
J.
Mandel
, “A brief tutorial on the ensemble Kalman filter,” arXiv:0901.3725 (2009).
6.
S. G.
Penny
,
T. A.
Smith
,
T.-C.
Chen
,
J. A.
Platt
,
H.-Y.
Lin
,
M.
Goodliff
, and
H. D. I.
Abarbanel
, “
Integrating recurrent neural networks with data assimilation for scalable data-driven state estimation
,”
J. Adv. Modell. Earth Syst.
14
,
e2021MS002843
(
2022
).
7.
G.
Karniadakis
,
Y.
Kevrekidis
,
L.
Lu
,
P.
Perdikaris
,
S.
Wang
, and
L.
Yang
, “
Physics-informed machine learning
,”
Nat. Rev. Phys.
3
,
422
440
(
2021
).
8.
S.
Ha
and
H.
Jeong
, “Discovering conservation laws from trajectories via machine learning,” arXiv:2102.04008 (2021).
9.
F.
Alet
,
D. D.
Doblar
,
A.
Zhou
,
J. B.
Tenenbaum
,
K.
Kawaguchi
, and
C.
Finn
, “Noether networks: Meta-learning useful conserved quantities,” arXiv:2112.03321 (2021).
10.
Z.
Liu
and
M.
Tegmark
, “
Machine learning conservation laws from trajectories
,”
Phys. Rev. Lett.
126
,
180604
(
2021
).
11.
T.
Beucler
,
M.
Pritchard
,
S.
Rasp
,
J.
Ott
,
P.
Baldi
, and
P.
Gentine
, “
Enforcing analytic constraints in neural networks emulating physical systems
,”
Phys. Rev. Lett.
126
,
098302
(
2021
).
12.
Z.
Chen
,
J.
Zhang
,
M.
Arjovsky
, and
L.
Bottou
, “Symplectic recurrent neural networks,” in International Conference on Learning Representations (International Conference on Learning Representations, 2020). https://openreview.net/forum?id=BkgYPREtPr.
13.
S.
Greydanus
,
M.
Dzamba
, and
J.
Yosinski
, “Hamiltonian neural networks,” in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Curran Associates, Inc., 2019), Vol. 32.
14.
M.
Raissi
,
P.
Perdikaris
, and
G.
Karniadakis
, “
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
,”
J. Comput. Phys.
378
,
686
707
(
2019
).
15.
L.
Yang
,
X.
Meng
, and
G. E.
Karniadakis
, “
B-PINNs: Bayesian physics-informed neural networks for forward and inverse PDE problems with noisy data
,”
J. Comput. Phys.
425
,
109913
(
2021
).
16.
A.
Azizi
and
M.
Pleimling
, “
A cautionary tale for machine learning generated configurations in presence of a conserved quantity
,”
Sci. Rep.
11
(
1
),
6395
(
2021
).
17.
N.
Doan
,
W.
Polifke
, and
L.
Magri
, “
Physics-informed echo state networks
,”
J. Comput. Sci.
47
,
101237
(
2020
).
18.
N. A. K.
Doan
,
W.
Polifke
, and
L.
Magri
, “
Short- and long-term predictions of chaotic flows and extreme events: A physics-constrained reservoir computing approach
,”
Proc. R. Soc. A
477
(
2253
),
20210135
(
2021
).
19.
A.
Racca
and
L.
Magri
, “Automatic-differentiated physics-informed echo state network (API-ESN),” in Computational Science—ICCS 2021, edited by M. Paszynski, D. Kranzlmüller, V. V. Krzhizhanovskaya, J. J. Dongarra, and P. M. Sloot (Springer International Publishing, Cham, 2021), pp. 323–329.
20.
H.
Goldstein
,
C.
Poole
, and
J.
Safko
,
Classical Mechanics
,
3rd
ed. (
Addison-Wesley
,
2001
).
21.
I.
Ispolatov
,
V.
Madhok
,
S.
Allende
, and
M.
Doebeli
, “
Chaos in high-dimensional dissipative dynamical systems
,”
Sci. Rep.
5
,
12506
(
2015
).
22.
L.
Zanna
and
T.
Bolton
, “
Data-driven equation discovery of ocean mesoscale closures
,”
Geophys. Res. Lett.
47
(
17
),
e2020GL088376
, https://doi.org/10.1029/2020GL088376 (
2020
).
23.
T.
Beucler
,
M.
Pritchard
,
P.
Gentine
, and
S.
Rasp
, “Towards physically-consistent, data-driven models of convection,” in IGARSS 2020—2020 IEEE International Geoscience and Remote Sensing Symposium (IEEE, 2020), pp. 3987–3990, ISSN: 2153-7003.
24.
I.
Goodfellow
,
Y.
Bengio
, and
A.
Courville
,
Deep Learning
(
MIT Press
,
2016
).
25.
E. N.
Lorenz
, “Predictability: A problem partly solved,” in Predictability of Weather and Climate, edited by T. Palmer and R. Hagedorn (Cambridge University Press, Cambridge, 2006).
26.
B. B.
Reinhold
and
R. T.
Pierrehumbert
, “
Dynamics of weather regimes: Quasi-stationary waves and blocking
,”
Mon. Weather Rev.
110
(
9
),
1105
1145
(
1982
).
27.
G.
Datseris
and
U.
Parlitz
,
Nonlinear Dynamics: A Concise Introduction Interlaced with Code
,
1st
ed. (
Springer
,
2022
).
28.
J.
Demaeyer
,
L.
De Cruz
, and
S.
Vannitsem
, “
qgs: A flexible Python framework of reduced-order multiscale climate models
,”
J. Open Source Softw.
5
,
2597
(
2020
).
29.
H. D. I.
Abarbanel
,
The Analysis of Observed Chaotic Data
(
Springer-Verlag
,
New York
,
1996
).
30.
J. P.
Eckmann
and
D.
Ruelle
, “
Ergodic theory of chaos and strange attractors
,”
Rev. Mod. Phys.
57
,
617
656
(
1985
).
31.
J. M.
Ottino
,
S. R.
Wiggins
,
M. R.
Bringer
,
C. J.
Gerdts
,
H.
Song
,
J. D.
Tice
, and
R. F.
Ismagilov
, “
Microfluidic systems for chemical kinetics that rely on chaotic mixing in droplets
,”
Philos. Trans. R. Soc. London, Ser. A
362
(
1818
),
923
935
(
2004
).
32.
R. O.
Grigoriev
,
M. F.
Schatz
, and
V.
Sharma
, “
Chaotic mixing in microdroplets
,”
Lab Chip
6
,
1369
1372
(
2006
).
33.
F. T.
Arecchi
,
G.
Giacomelli
,
P. L.
Ramazza
, and
S.
Residori
, “
Experimental evidence of chaotic itinerancy and spatiotemporal chaos in optics
,”
Phys. Rev. Lett.
65
,
2531
2534
(
1990
).
34.
X.
Zang
,
S.
Iqbal
,
Y.
Zhu
,
X.
Liu
, and
J.
Zhao
, “
Applications of chaotic dynamics in robotics
,”
Int. J. Adv. Rob. Syst.
13
(
2
),
60
(
2016
).
35.
A. M.
Lyapunov
, “
The general problem of the stability of motion
,”
Int. J. Control
55
(
3
),
531
534
(
1992
).
36.
V. I.
Oseledec
, “
A multiplicative ergodic theorem. Lyapunov characteristic numbers for dynamical systems
,”
Trudy Mosk. Mat. Obsc.
19
,
197
(
1968
).
37.
W.
Maass
,
T.
Natschläger
, and
H.
Markram
, “
Real-time computing without stable states: A new framework for neural computation based on perturbations
,”
Neural Comput.
14
,
2531
2560
(
2002
).
38.
H.
Jaeger
, “The “echo state” approach to analysing and training recurrent neural networks—With an erratum note,” GMD Technical Report No. 148, German National Research Center for Information Technology, Bonn, Germany, 2010, pp. 1–47.
39.
M.
Lukosevicius
, A Practical Guide to Applying Echo State Networks in
Neural Networks: Tricks of the Trade. Lecture Notes in Computer Science
, edited by G. Montavon, G. B. Orr, and K. R. Müller (Springer, Berlin, Heidelberg, 2012), Vol. 7700.
40.
J. L.
Elman
, “
Finding structure in time
,”
Cogn. Sci.
14
(
2
),
179
211
(
1990
).
41.
H.
Jaeger
, “Short term memory in echo state networks,” GMD Report No. 152, German National Research Institute for Computer Science (GMD), 2002.
42.
H.
Jaeger
and
H.
Haas
, “
Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication
,”
Science
304
,
78
80
(
2004
).
43.
M.
Lukosevicius
and
H.
Jaeger
, “
Reservoir computing approaches to recurrent neural network training
,”
Comput. Sci. Rev.
3
,
127
149
(
2009
).
44.
H.
Jaeger
, “Long short-term memory in echo state networks: Details of a simulation study,” Constructor University Technical Reports 27 (2012).
45.
J. A.
Platt
,
S. G.
Penny
,
T. A.
Smith
,
T.-C.
Chen
, and
H. D. I.
Abarbanel
, “A systematic exploration of reservoir computing for forecasting complex spatiotemporal dynamics,” arXiv:2201.08910 (2022).
46.
J. A.
Platt
,
A. S.
Wong
,
R.
Clark
,
S. G.
Penny
, and
H. D. I.
Abarbanel
, “
Robust forecasting using predictive generalized synchronization in reservoir computing
,”
Chaos
31
,
123118
(
2021
).
47.
A.
Griffith
,
A.
Pomerance
, and
D. J.
Gauthier
, “
Forecasting chaotic systems with very low connectivity reservoir computers
,”
Chaos
29
(
12
),
123108
(
2019
).
48.
P.
Vlachas
,
J.
Pathak
,
B.
Hunt
,
T.
Sapsis
,
M.
Girvan
,
E.
Ott
, and
P.
Koumoutsakos
, “
Backpropagation algorithms and reservoir computing in recurrent neural networks for the forecasting of complex spatiotemporal dynamics
,”
Neural Netw.
126
,
191
217
(
2020
).
49.
T.
Beucler
,
M.
Pritchard
,
S.
Rasp
,
J.
Ott
,
P.
Baldi
, and
P.
Gentine
, “
Enforcing analytic constraints in neural networks emulating physical systems
,”
Phys. Rev. Lett.
126
(
9
),
098302
(
2021
).
50.
N.
Hansen
,
S. D.
Müller
, and
P.
Koumoutsakos
, “
Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (CMA-ES)
,”
Evol. Comput.
11
(
1
),
1
18
(
2003
).
51.
N.
Hansen
, “The CMA evolution strategy: A tutorial,” arXiv:1604.00772 (2023).
52.
M. T.
Rosenstein
,
J. J.
Collins
, and
C. J.
De Luca
, “
A practical method for calculating largest Lyapunov exponents from small data sets
,”
Physica D
65
(
1
),
117
134
(
1993
).
53.
H.
Kantz
, “
A robust method to estimate the maximal Lyapunov exponent of a time series
,”
Phys. Lett. A
185
(
1
),
77
87
(
1994
).
54.
J.
Theiler
, “
Estimating fractal dimension
,”
J. Opt. Soc. Am. A
7
,
1055
1073
(
1990
).
55.
Z.
Lu
,
B. R.
Hunt
, and
E.
Ott
, “
Attractor reconstruction by machine learning
,”
Chaos
28
(
6
),
061104
(
2018
).
56.
K.
Geist
,
U.
Parlitz
, and
W.
Lauterborn
, “
Comparison of different methods for computing Lyapunov exponents
,”
Prog. Theor. Phys.
83
(
5
),
875
893
(
1990
).
57.
J. L.
Kaplan
and
J. A.
Yorke
, “Chaotic behavior of multidimensional difference equations,” in Functional Differential Equations and Approximation of Fixed Points, edited by H.-O. Peitgen and H.-O. Walther (Springer, Berlin, 1979), pp. 204–227.
58.
P.
Frederickson
,
J. L.
Kaplan
,
E. D.
Yorke
, and
J. A.
Yorke
, “
The Liapunov dimension of strange attractors
,”
J. Differ. Equ.
49
(
2
),
185
207
(
1983
).
59.
G.
Benettin
,
L.
Galgani
, and
J.-M.
Strelcyn
, “
Kolmogorov entropy and numerical experiments
,”
Phys. Rev. A
14
,
2338
2345
(
1976
).
60.
J. G.
Charney
, “On the scale of atmospheric motions,” Geof. Publ. 17, 1–17 (1948).
61.
S.-C.
Yang
,
M.
Corazza
,
A.
Carrassi
,
E.
Kalnay
, and
T.
Miyoshi
, “
Comparison of local ensemble transform Kalman filter, 3DVAR, and 4DVAR in a quasigeostrophic model
,”
Mon. Weather Rev.
137
(
2
),
693
709
(
2009
).
62.
K.
Swanson
,
R.
Vautard
, and
C.
Pires
, “
Four-dimensional variational assimilation and predictability in a quasi-geostrophic model
,”
Tellus A: Dyn. Meteorol. Oceanogr.
50
(
4
),
369
390
(
2022
).
63.
J. G.
Charney
and
D. M.
Straus
, “
Form-drag instability, multiple equilibria and propagating planetary waves in baroclinic, orographically forced, planetary wave systems
,”
J. Atmos. Sci.
37
(
6
),
1157
1176
(
1980
).
64.
E. N.
Lorenz
, “
Deterministic nonperiodic flow
,”
J. Atmos. Sci.
20
(
2
),
130
141
(
1963
).
65.
E.
Kalnay
,
Atmospheric Modeling, Data Assimilation, and Predictability
(
Cambridge University Press
,
2003
).
66.
Z.
Toth
and
E.
Kalnay
, “
Ensemble forecasting at NMC: The generation of perturbations
,”
Bull. Am. Meteorol. Soc.
74
(
12
),
2317
2330
(
1993
).
67.
Z.
Toth
and
E.
Kalnay
, “
Ensemble forecasting at NCEP and the breeding method
,”
Mon. Weather Rev.
125
(
12
),
3297
3319
(
1997
).
68.
J.
Pathak
,
S.
Subramanian
,
P.
Harrington
,
S.
Raja
,
A.
Chattopadhyay
,
M.
Mardani
,
T.
Kurth
,
D.
Hall
,
Z.
Li
,
K.
Azizzadenesheli
,
P.
Hassanzadeh
,
K.
Kashinath
, and
A.
Anandkumar
, “FourCastNet: A global data-driven high-resolution weather model using adaptive Fourier neural operators,” arXiv:2202.11214 (2022).
69.
R.
Lam
,
A.
Sanchez-Gonzalez
,
M.
Willson
,
P.
Wirnsberger
,
M.
Fortunato
,
A.
Pritzel
,
S.
Ravuri
,
T.
Ewalds
,
F.
Alet
,
Z.
Eaton-Rosen
,
W.
Hu
,
A.
Merose
,
S.
Hoyer
,
G.
Holland
,
J.
Stott
,
O.
Vinyals
,
S.
Mohamed
, and
P.
Battaglia
, “GraphCast: Learning skillful medium-range global weather forecasting,” arXiv:2212.12794 (2022).
70.
K.
Bi
,
L.
Xie
,
H.
Zhang
,
X.
Chen
,
X.
Gu
, and
Q.
Tian
, “Pangu-Weather: A 3D high-resolution model for fast and accurate global weather forecast,” arXiv:2211.02556 (2022).
You do not currently have access to this content.