We develop and test machine learning techniques for successfully using past state time series data and knowledge of a time-dependent system parameter to predict the evolution of the “climate” associated with the long-term behavior of a non-stationary dynamical system, where the non-stationary dynamical system is itself unknown. By the term climate, we mean the statistical properties of orbits rather than their precise trajectories in time. By the term non-stationary, we refer to systems that are, themselves, varying with time. We show that our methods perform well on test systems predicting both continuous gradual climate evolution as well as relatively sudden climate changes (which we refer to as “regime transitions”). We consider not only noiseless (i.e., deterministic) non-stationary dynamical systems, but also climate prediction for non-stationary dynamical systems subject to stochastic forcing (i.e., dynamical noise), and we develop a method for handling this latter case. The main conclusion of this paper is that machine learning has great promise as a new and highly effective approach to accomplishing data driven prediction of non-stationary systems.

1
H.
Jaeger
and
H.
Haas
, “
Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication
,”
Science
304
,
78
(
2004
).
2
J.
Pathak
,
B. R.
Hunt
,
M.
Girvan
,
Z.
Lu
, and
E.
Ott
, “
Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach
,”
Phys. Rev. Lett.
120
,
024102
(
2018
).
3
J.
Pathak
,
A.
Wikner
,
R.
Fussel
,
S.
Chandra
,
B.
Hunt
,
M.
Girvan
, and
E.
Ott
, “
Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge based model
,”
Chaos
28
,
041101
(
2018
).
4
J.
Pathak
,
Z.
Lu
,
B. R.
Hunt
,
M.
Girvan
, and
E.
Ott
, “
Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data
,”
Chaos
27
,
121102
(
2017
).
5
Z.
Lu
,
B. R.
Hunt
, and
E.
Ott
, “
Attractor reconstruction by machine learning
,”
Chaos
28
,
061104
(
2018
).
6
Z.
Shi
and
M.
Han
, “
Support vector echo-state machine for chaotic time series prediction
,”
IEEE Trans. Neural Netw.
18
,
359
(
2007
).
7
S.
Mukherjee
,
E.
Osuna
, and
F.
Girosi
, “Nonlinear prediction of chaotic time series using support vector machines,” in Neural Networks for Signal Processing VII. Proceedings of the 1997 IEEE Signal Processing Society Workshop (IEEE, 1997) pp. 511–520.
8
Q.
Ma
,
Q.
Zheng
,
H.
Peng
,
T.
Zhong
, and
L.
Xu
, “Chaotic time series prediction based on evolving recurrent neural networks,” in 2007 International Conference on Machine Learning and Cybernetics (IEEE, 2007), Vol. 6, pp. 3496–3500.
9
R.
Chandra
,
Y.-S.
Ong
, and
C.-K.
Goh
, “
Co-evolutionary multi-task learning with predictive recurrence for multi-step chaotic time series prediction
,”
Neurocomputing
243
,
21
34
(
2017
).
10
D.
Li
,
M.
Han
, and
J.
Wang
, “
Chaotic time series prediction based on a novel robust echo state network
,”
IEEE Trans. Neural Netw. Learn. Syst.
23
,
787
799
(
2012
).
11
M.
Sangiorgio
and
F.
Dercole
, “
Robustness of LSTM neural networks for multi-step forecasting of chaotic time series
,”
Chaos, Solitons Fractals
139
,
110045
(
2020
).
12
Z.
Tian
, “
Echo state network based on improved fruit fly optimization algorithm for chaotic time series prediction
,”
J. Ambient Intell. Humaniz. Comput.
(published online).
13
A.
Vaughan
and
S. V.
Bohac
, “
Real-time, adaptive machine learning for non-stationary, near chaotic gasoline engine combustion time series
,”
Neural Netw.
70
,
18
26
(
2015
).
14
A.
Griffith
,
A.
Pomerance
, and
D. J.
Gauthier
, “
Forecasting chaotic systems with very low connectivity reservoir computers
,”
Chaos
29
,
123108
(
2019
).
15
G.
Shen
,
J.
Kurths
, and
Y.
Yuan
, “
Sequence-to-sequence prediction of spatiotemporal systems
,”
Chaos
30
,
023102
(
2020
).
16
R.
Follmann
and
J. E.
Rosa
, “
Predicting slow and fast neuronal dynamics with machine learning
,”
Chaos
29
,
113119
(
2019
).
17
S.
Herzog
,
F.
Wörgötter
, and
U.
Parlitz
, “
Convolutional autoencoder and conditional random fields hybrid for predicting spatial-temporal chaos
,”
Chaos
29
,
123116
(
2019
).
18
A.
Chattopadhyay
,
P.
Hassanzadeh
, and
D.
Subramanian
, “
Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: Reservoir computing, artificial neural network, and long short-term memory network
,”
Nonlinear Process. Geophys.
27
,
373
389
(
2020
).
19
D.
Chen
and
W.
Han
, “
Prediction of multivariate chaotic time series via radial basis function neural networks
,”
Complexity
18
,
55
(
2013
).
20
W.
Huang
,
Y.
Li
, and
Y.
Huang
, “
Deep hybrid neural network and improved differential neuroevolution for chaotic time series prediction
,”
IEEE Access
8
,
159552
159565
(
2020
).
21
H.
Fan
,
J.
Jiang
,
C.
Zhang
,
X.
Wang
, and
Y.-C.
Lai
, “
Long-term prediction of chaotic systems with machine learning
,”
Phys. Rev. Res.
2
,
012080
(
2020
).
22
A.
Faqih
,
A. P.
Lianto
, and
B.
Kusumoputro
, “Mackey-Glass chaotic time series prediction using modified RBF neural networks,” in Proceedings of the 2nd International Conference on Software Engineering and Information Management (ICSIM 2019) (Association for Computing Machinery, New York, 2019), pp. 7–11.
23
P.
Antonik
,
M.
Gulina
,
J.
Pauwels
, and
S.
Massar
, “
Using a reservoir computer to learn chaotic attractors, with applications in chaos synchronization and cryptography
,”
Phys. Rev. E
98
,
012215
(
2018
).
24
D.
Nguyen
,
S.
Ouala
,
L.
Drumetz
, and
R.
Fablet
, “EM-like learning chaotic dynamics from noisy and partial observations,” arXiv:1903.10335 (2019).
25
F. Z.
Xing
,
E.
Cambria
, and
X.
Zou
, “Predicting evolving chaotic time series with fuzzy neural networks,” in 2017 International Joint Conference on Neural Networks (IJCNN) (IEEE Press, 2017) pp. 3176–3183.
26
J. P.
Eckmann
and
D.
Ruelle
, “
Ergodic theory of chaos and strange attractors
,”
Rev. Mod. Phys.
57
,
617
(
1985
).
27
G.
Drótos
,
T.
Bódai
, and
T.
Tél
, “
Quantifying nonergodicity in nonautonomous dissipative dynamical systems: An application to climate change
,”
Phys. Rev. E
94
,
022214
(
2016
).
28
G.
Drótos
,
T.
Bódai
, and
T.
Tél
, “
Probabilistic concepts in a changing climate: A snapshot attractor picture
,”
J. Clim.
28
,
3275
3288
(
15 Apr. 2015
).
29
F. J.
Romeiras
,
C.
Grebogi
, and
E.
Ott
, “
Multifractal properties of snapshot attractors of random maps
,”
Phys. Rev. A
41
,
784
799
(
1990
).
30
M. D.
Chekroun
,
E.
Simonnet
, and
M.
Ghil
, “
Stochastic climate dynamics: Random attractors and time-dependent invariant measures
,”
Phys. D
240
,
1685
1700
(
2011
).
31
S. H.
Lim
,
L.
Theo Giorgini
,
W.
Moon
, and
J. S.
Wettlaufer
, “
Predicting critical transitions in multiscale dynamical systems using reservoir computing
,”
Chaos
30
,
123126
(
2020
).
32
M. J.
Feigenbaum
, “
Quantitative universality for a class of nonlinear transformations
,”
J. Stat. Phys.
19
,
25
52
(
1978
).
33
M. J.
Feigenbaum
, “
Universal behavior in nonlinear systems
,”
Phys. D
7
,
16–39
(
1983
).
34
Y.
Pomeau
and
P. M.
Manneville
, “
Intermittent transition to turbulence in dissipative dynamical systems
,”
Commun. Math. Phys.
74
,
189
197
(
1980
).
35
C.
Grebogi
,
E.
Ott
, and
J. A.
Yorke
, “
Crises, sudden changes in chaotic attractors, and transient chaos
,”
Phys. D
7
,
181
200
(
1983
).
36
C.
Grebogi
,
E.
Ott
, and
J. A.
Yorke
, “
Critical exponents of chaotic transients in nonlinear dynamical systems
,”
Phys. Rev. Lett.
57
,
1284
(
1986
).
37
L.-W.
Kong
,
H.-W.
Fan
,
C.
Grebogi
, and
Y.-C.
Lai
, “Machine learning prediction of critical transition and system collapse,” arXiv:2012.01545 (2019).
38
T. M.
Lenton
,
H.
Held
,
E.
Kriegler
,
J. W.
Wall
,
W.
Lucht
,
S.
Rahmstorf
, and
H. J.
Schellnhuber
, “
Tipping elements in the Earth’s climate system
,”
Proc. Natl. Acad. Sci. U.S.A.
105
,
1786
1793
(
2008
).
39
P.
Ashwin
and
A. S.
von der Heydt
, “
Extreme sensitivity and climate tipping points
,”
J. Stat. Phys.
179
,
1531
1552
(
2020
).
40
S. O.
Rasmussen
,
M.
Bigler
,
S. P.
Blockley
,
T.
Blunier
,
S. L.
Buchardt
,
H. B.
Clausen
,
I.
Cvijanovic
,
D.
Dahl-Jensen
,
S. J.
Johnsen
,
H.
Fischer
,
V.
Gkinis
,
M.
Guillevic
,
W. Z.
Hoek
,
J. J.
Lowe
,
J. B.
Pedro
,
T.
Popp
,
I. K.
Seierstad
,
J. P.
Steffensen
,
A. M.
Svensson
,
P.
Vallelonga
,
B. M.
Vinther
,
M. J.
Walker
,
J. J.
Wheatley
, and
M.
Winstrup
, “
A stratigraphic framework for abrupt climatic changes during the Last Glacial period based on three synchronized Greenland ice-core records: Refining and extending the INTIMATE event stratigraphy
,”
Quat. Sci. Rev.
106
,
14
28
(
2014
).
41
R. T.
Pierrehumbert
, “
Warming the world
,”
Nature
432
,
677
(
2004
).
42
A.
Zhang
,
G.
Knorr
,
G.
Lohmann
, and
S.
Barker
, “
Abrupt North Atlantic circulation changes in response to gradual CO2 forcing in a glacial climate state
,”
Nat. Geosci.
10
,
518
523
(
2017
).
43
Q.
Schiermeier
, “
A sea change
,”
Nature
439
,
256
260
(
2006
).
44
T. F.
Stocker
and
D. G.
Wright
, “
Rapid transitions of the ocean’s deep circulation induced by changes in surface water fluxes
,”
Nature
315
,
729
731
(
1991
).
45
K.
Zickfeld
,
B.
Knopf
,
V.
Petoukhov
, and
H. J.
Schellnhuber
, “
Is the Indian summer monsoon stable against global change?
Geophys. Res. Lett.
32
,
L15707
, https://doi.org/10.1029/2005GL022771 (
2005
).
46
Z.
Liu
and
M.
Alexander
, “
Atmospheric bridge, oceanic tunnel, and global climate teleconnections
,”
Rev. Geophys.
45
,
RG2005
, https://doi.org/10.1029/2005RG000172 (
2007
).
47
A.
Angstrom
, “
Teleconnections of climate changes in present times
,”
Geogr. Ann.
17
,
242
258
(
1935
).
48
J.
Kurths
,
A.
Agarwal
,
R.
Shukla
,
N.
Marwan
,
M.
Rathinasamy
,
L.
Caesar
,
R.
Krishnan
, and
B.
Merz
, “
Unravelling the spatial diversity of Indian precipitation teleconnections via a non-linear multi-scale approach
,”
Nonlinear Process. Geophys.
26
,
251
266
(
2019
).
49
T.
Lenton
and
J.
Ciscar
, “
Integrating tipping points into climate impact assessments
,”
Clim. Change
117
,
585
597
(
2013
).
50
D.
Lemoine
and
C. P.
Traeger
, “
Economics of tipping the climate dominoes
,”
Nat. Clim. Change
6
,
514
519
(
2016
).
51
A.
Hastings
and
D. B.
Wysham
, “
Regime shifts in ecological systems can occur with no warning
,”
Ecol. Lett.
13
,
464
472
(
2010
).
52
C.
Folke
,
S.
Carpenter
,
B.
Walker
,
M.
Scheffer
,
T.
Elmqvist
,
L.
Gunderson
, and
C.
Holling
, “
Regime shifts, resilience, and biodiversity in ecosystem management
,”
Annu. Rev. Ecol. Evol. Syst.
35
,
557
581
(
2004
).
53
T.
Amemiya
,
T.
Enomoto
,
A. G.
Rossberg
,
T.
Yamamoto
,
Y.
Inamori
, and
K.
Itoh
, “
Stability and dynamical behavior in a lake-model and implications for regime shifts in real lakes
,”
Ecol. Modell.
206
,
54
62
(
2007
).
54
M.
Scheffer
,
S. H.
Hosper
,
M.-L.
Meijer
,
B.
Moss
, and
E.
Jeppesen
, “
Alternative equilibria in shallow lakes
,”
Trends Ecol. Evol.
8
,
275
279
(
1993
).
55
B.
deYoung
,
M.
Barange
,
G.
Beaugrand
,
R.
Harris
,
R. I.
Perry
,
M.
Scheffer
, and
F.
Werner
, “
Regime shifts in marine ecosystems: Detection, prediction and management
,”
Trends Ecol. Evol.
23
,
402
409
(
2008
).
56
P. J.
Mumby
,
A.
Hastings
, and
H. J.
Edwards
, “
Thresholds and the resilience of Caribbean coral reefs
,”
Nature
450
,
98
101
(
2007
).
57
V.
Brovkin
,
M.
Claussen
,
V.
Petoukhov
, and
A.
Ganopolski
, “
On the stability of the atmosphere-vegetation system in the Sahara/Sahel region
,”
J. Geophys. Res.
103
,
31613
31624
, https://doi.org/10.1029/1998JD200006 (
1998
).
58
Y. R.
Zelnik
,
S.
Kinast
,
H.
Yizhaq
,
G.
Bel
, and
E.
Meron
, “
Regime shifts in models of dryland vegetation
,”
Philos. Trans. R. Soc. A
371
,
20120358
(
2013
).
59
S.
Ellner
and
P.
Turchin
, “
Chaos in a noisy world: New methods and evidence from time-series analysis
,”
Am. Nat.
45
,
343
347
(
1995
).
60
J. M.
Drake
and
B. D.
Griffen
, “
Early warning signals of extinction in deteriorating environment
,”
Nature
467
,
456
459
(
2010
).
61
M.
Krkošek
and
J. M.
Drake
, “
On signals of phase transitions in salmon population dynamics
,”
Proc. R. Soc. B: Biol. Sci.
281
,
20133221
(
2014
).
62
I. V.
Telesh
,
H.
Schubert
,
K. D.
Joehnk
,
R.
Heerkloss
,
R.
Schumann
,
M.
Feike
,
A.
Schoor
, and
S. O.
Skarlato
, “
Chaos theory discloses triggers and drivers of plankton dynamics in stable environment
,”
Sci. Rep.
9
,
20351
(
2019
).
63
C.
Trefois
,
P. M.
Antony
,
J.
Goncalves
,
A.
Skupin
, and
R.
Balling
, “
Critical transitions in chronic disease: Transferring concepts from ecology to systems medicine
,”
Curr. Opin. Biotechnol.
34
,
48
55
(
2015
).
64
B.
Litt
,
R.
Esteller
,
J.
Echauz
,
M.
D’Alessandro
,
R.
Shor
,
T.
Henry
,
P.
Pennell
,
C.
Epstein
,
R.
Bakay
,
M.
Dichter
, and
G.
Vachtsevanos
, “
Epileptic seizures may begin hours in advance of clinical onset: A report of five patients
,”
Neuron
30
,
51
64
(
2001
).
65
E.
Gopalakrishnan
,
Y.
Sharma
,
T.
John
et al., “
Early warning signals for critical transitions in a thermoacoustic system
,”
Sci. Rep.
6
,
35310
(
2016
).
66
I.
Dobson
, “
Observations on the geometry of saddle node bifurcation and voltage collapse in electrical power systems
,”
IEEE Trans. Circuits Syst. I Fundam. Theory Appl.
39
,
240
243
(
1992
).
67
C.
Kuehn
,
E. A.
Martens
, and
D. M.
Romero
, “
Critical transitions in social network activity
,”
J. Complex Netw.
2
,
141
152
(
2014
).
68
M.
Lukosevicius
and
H.
Jaeger
, “
Reservoir computing approaches to recurrent neural network training
,”
Comput. Sci. Rev.
3
,
127
149
(
2009
).
69
H.
Jaeger
, “The “echo state” approach to analyzing and training recurrent neural networks—With an erratum note,” GMD Technical Report No. 148, German National Research Center for Information Technology, Bonn, Germany, 2001.
70
W.
Maass
,
T.
Natschlager
, and
H.
Markram
, “
Real-time computing without stable states: A new framework for neural computation based on perturbations
,”
Neural Comput.
14
,
2531
2560
(
2002
).
71
I.
Goodfellow
,
Y.
Bengio
, and
A.
Courville
,
Deep Learning
(
MIT Press
,
2016
).
72
E.
Lorenz
, “
Deterministic nonperiodic flow
,”
J. Atmos. Sci.
20
,
130
(
1963
).
73
Y.
Kuramoto
, “
Diffusion-induced chaos in reaction systems
,”
Prog. Theor. Phys. Suppl.
64
,
346
367
(
1978
).
74
G. I.
Sivashinsky
, “
On flame propagation under conditions of stoichiometry
,”
SIAM J. Appl. Math.
39
,
67
82
(
1980
).
75
G.
Tanaka
,
T.
Yamane
,
J. B.
Héroux
,
R.
Nakane
,
N.
Kanazawa
,
S.
Takeda
,
H.
Numata
,
D.
Nakano
, and
A.
Hirose
, “
Recent advances in physical reservoir computing: A review
,”
Neural Netw.
115
,
100
123
(
2019
).
76
B.
Penkovsky
,
L.
Larger
, and
D.
Brunner
, “
Efficient design of hardware-enabled reservoir computing in FPGAs
,”
J. Appl. Phys.
124
,
162101
(
2018
).
77
Y. K.
Chembo
, “
Machine learning based on reservoir computing with time-delayed optoelectronic and photonic systems
,”
Chaos
30
,
013111
(
2020
).
78
C.
Fernando
and
S.
Sojakka
, “Pattern recognition in a bucket,” in Advances in Artificial Life, edited by W. Banzhaf, J. Ziegler, T. Christaller, P. Dittrich, and J. T. Kim (Springer, Berlin, 2003), pp. 588–597.
79
J. C.
Coulombe
,
M. C.
York
, and
J.
Sylvestre
, “
Computing with networks of nonlinear mechanical oscillators
,”
PLoS One
12
,
0178663
(
2017
).
80
K.
Harkhoe
and
G. Van
der Sande
, “
Delay-based reservoir computing using multimode semiconductor lasers: Exploiting the rich carrier dynamics
,”
IEEE J. Sel. Top. Quantum Electron.
25
,
1502909
(
2019
).
81
E.
Ott
,
Chaos in Dynamical Systems
(
Cambridge University Press
,
2002
).
82
C.
Grebogi
,
E.
Ott
, and
J.
Yorke
, “
Chaotic attractors in crisis
,”
Phys. Rev. Lett.
48
,
1507
(
1982
).
83
K.
Erguler
and
M.
Stumpf
, “
Statistical interpretations of the interplay between noise and chaos in the stochastic logistic map
,”
Math. Biosci.
216
,
90
99
(
2008
).
84
B.
Shraiman
,
C.
Wayne
, and
P.
Martin
, “
Scaling theory for noisy period-doubling transitions to chaos
,”
Phys. Rev. Lett.
46
,
935
(
1981
).
85
F.
Ledrappier
and
L. S.
Young
, “
Dimension formula for random transformations
,”
Commun. Math. Phys.
117
,
529
548
(
1988
).
86
L.
Yu
,
E.
Ott
, and
Q.
Chen
, “
Transition to chaos for random dynamical systems
,”
Phys. Rev. Lett.
65
,
2935
2938
(
1990
).
87
T.
Tél
,
T.
Bódai
,
G.
Drótos
,
T.
Haszpra
,
M.
Herein
,
B.
Kaszás
,
M.
Vincze
et al., “
The theory of parallel climate realizations
,”
J. Stat. Phys.
179
,
1496
1530
(
2020
).
88
S. S.
Vallender
, “
Calculation of the Wasserstein distance between probability distributions on the line
,”
Theory Probab. Appl.
18
,
784
786
(
1973
).
89
S.
Kullback
and
R. A.
Leibler
, “
On information and sufficiency
,”
Ann. Math. Stat.
22
,
79
86
(
1951
).
90
S.
Chandra
,
B. R.
Hunt
,
R.
Roy
, and
E.
Ott
, “Using machine learning to uncover the deterministic dynamics of systems subject to dynamical noise” (to be published).
91
F.
Christiansen
,
P.
Cvitanovic
, and
V.
Putkaradze
, “
Spatiotemporal chaos in terms of unstable recurrent patterns
,”
Nonlinearity
10
,
55
70
(
1997
).
You do not currently have access to this content.