The ability of machine learning (ML) models to “extrapolate” to situations outside of the range spanned by their training data is crucial for predicting the long-term behavior of non-stationary dynamical systems (e.g., prediction of terrestrial climate change), since the future trajectories of such systems may (perhaps after crossing a tipping point) explore regions of state space which were not explored in past time-series measurements used as training data. We investigate the extent to which ML methods can yield useful results by extrapolation of such training data in the task of forecasting non-stationary dynamics, as well as conditions under which such methods fail. In general, we find that ML can be surprisingly effective even in situations that might appear to be extremely challenging, but do (as one would expect) fail when “too much” extrapolation is required. For the latter case, we show that good results can potentially be obtained by combining the ML approach with an available inaccurate conventional model based on scientific knowledge.

1
B.
Kaszas
,
U.
Feudel
, and
T.
Tel
, “
Tipping phenomena in typical dynamical systems subjected to parameter drift
,”
Sci. Rep.
9
,
8654
(
2019
).
2
U.
Feudel
,
A. N.
Pisarchik
, and
K.
Showalter
, “
Multistability and tipping: From mathematics and physics to climate and brain—minireview and preface to the focus issue
,”
Chaos
28
,
033501
(
2018
).
3
H.
Jaeger
and
H.
Haas
, “
Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication
,”
Science
304
,
78
(
2004
).
4
J.
Pathak
,
B. R.
Hunt
,
M.
Girvan
,
Z.
Lu
, and
E.
Ott
, “
Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach
,”
Phys. Rev. Lett.
120
,
024102
(
2018
).
5
J.
Pathak
,
A.
Wikner
,
R.
Fussel
,
S.
Chandra
,
B.
Hunt
,
M.
Girvan
, and
E.
Ott
, “
Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge based model
,”
Chaos
28
,
041101
(
2018
).
6
A.
Vaughan
and
S. V.
Bohac
, “
Real-time, adaptive machine learning for non-stationary, near chaotic gasoline engine combustion time series
,”
Neural Netw.
70
,
18
26
(
2015
).
7
A.
Griffith
,
A.
Pomerance
, and
D. J.
Gauthier
, “
Forecasting chaotic systems with very low connectivity reservoir computers
,”
Chaos
29
,
123108
(
2019
).
8
G.
Shen
,
J.
Kurths
, and
Y.
Yuan
, “
Sequence-to-sequence prediction of spatiotemporal systems
,”
Chaos
30
,
023102
(
2020
).
9
R.
Follmann
and
J. E.
Rosa
, “
Predicting slow and fast neuronal dynamics with machine learning
,”
Chaos
29
,
113119
(
2019
).
10
S.
Herzog
,
F.
Wörgötter
, and
U.
Parlitz
, “
Convolutional autoencoder and conditional random fields hybrid for predicting spatial-temporal chaos
,”
Chaos
29
,
123116
(
2019
).
11
A.
Chattopadhyay
,
P.
Hassanzadeh
, and
D.
Subramanian
, “
Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: Reservoir computing, artificial neural network, and long short-term memory network
,”
Nonlin. Processes Geophys.
27
,
373
389
(
2020
).
12
D.
Chen
and
W.
Han
, “
Prediction of multivariate chaotic time series via radial basis function neural networks
,”
Complexity
18
,
55
(
2013
).
13
W.
Huang
,
Y.
Li
, and
Y.
Huang
, “
Deep hybrid neural network and improved differential neuroevolution for chaotic time series prediction
,”
IEEE Access
8
,
159552
159565
(
2020
).
14
H.
Fan
,
J.
Jiang
,
C.
Zhang
,
X.
Wang
, and
Y.-C.
Lai
, “
Long-term prediction of chaotic systems with machine learning
,”
Phys. Rev. Res.
2
,
012080
(
2020
).
15
A.
Faqih
,
A. P.
Lianto
, and
B.
Kusumoputro
, “Mackey-Glass chaotic time series prediction using modified RBF neural networks,” in Proceedings of the 2nd International Conference on Software Engineering and Information Management (ICSIM 2019), series and number ICSIM 2019 (Association for Computing Machinery, New York, NY, 2019), pp. 7–11.
16
D.
Canaday
,
A.
Griffith
, and
D. J.
Gauthier
, “
Rapid time series prediction with a hardware-based reservoir computer
,”
Chaos
28
,
123119
(
2018
).
17
F.
Borra
,
A.
Vulpiani
, and
M.
Cencini
, “
Effective models and predictability of chaotic multiscale systems via machine learning
,”
Phys. Rev. E
102
,
052203
(
2020
).
18
J.
Pathak
,
Z.
Lu
,
B. R.
Hunt
,
M.
Girvan
, and
E.
Ott
, “
Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data
,”
Chaos
27
,
121102
(
2017
).
19
Z.
Lu
,
B. R.
Hunt
, and
E.
Ott
, “
Attractor reconstruction by machine learning
,”
Chaos
28
,
061104
(
2018
).
20
P.
Antonik
,
M.
Gulina
,
J.
Pauwels
, and
S.
Massar
, “
Using a reservoir computer to learn chaotic attractors, with applications in chaos synchronization and cryptography
,”
Phys. Rev. E
98
,
012215
(
2018
).
21
D.
Nguyen
,
S.
Ouala
,
L.
Drumetz
, and
R.
Fablet
, “EM-like learning chaotic dynamics from noisy and partial observations,” arXiv:abs/1903.10335 (2019).
22
J. P.
Eckmann
and
D.
Ruelle
, “
Ergodic theory of chaos and strange attractors
,”
Rev. Mod. Phys.
57
,
617
(
1985
).
23
G.
Drótos
,
T.
Bódai
, and
T.
Tél
, “
Quantifying nonergodicity in nonautonomous dissipative dynamical systems: An application to climate change
,”
Phys. Rev. E
94
,
022214
(
2016
).
24
G.
Drótos
,
T.
Bódai
, and
T.
Tél
, “
Probabilistic concepts in a changing climate: A snapshot attractor picture
,”
J. Clim.
28
,
3275
3288
(
2015
).
25
F. J.
Romeiras
,
C.
Grebogi
, and
E.
Ott
, “
Multifractal properties of snapshot attractors of random maps
,”
Phys. Rev. A
41
,
784
799
(
1990
).
26
M. D.
Chekroun
,
E.
Simonnet
, and
M.
Ghil
, “
Stochastic climate dynamics: Random attractors and time-dependent invariant measures
,”
Physica D
240
,
1685
1700
(
2011
).
27
D.
Patel
,
D.
Canaday
,
M.
Girvan
,
A.
Pomerance
, and
E.
Ott
, “
Using machine learning to predict statistical properties of non-stationary dynamical processes: System climate, regime transitions, and the effect of stochasticity
,”
Chaos
31
,
033149
(
2021
).
28
S. H.
Lim
,
L.
Theo Giorgini
,
W.
Moon
, and
J. S.
Wettlaufer
, “
Predicting critical transitions in multiscale dynamical systems using reservoir computing
,”
Chaos
30
,
123126
(
2020
).
29
F. Z.
Xing
,
E.
Cambria
, and
X.
Zou
, “Predicting evolving chaotic time series with fuzzy neural networks,” in
2017 International Joint Conference on Neural Networks (IJCNN)
(IEEE, 2017), pp. 3176–3183.
30
Z.
Shi
,
Y.
Bai
,
X.
Jin
,
X.
Wang
,
T.
Su
, and
J.
Kong
, “
Parallel deep prediction with covariance intersection fusion on non-stationary time series
,”
Know.-Based Syst.
211
,
106523
(
2021
).
31
A.
Pershin
,
C.
Beaume
,
K.
Li
, and
S. M.
Tobias
, “Can neural networks predict dynamics they have never seen?” arXiv:abs/2111.06783 (2021).
32
T. M.
Bury
,
R. I.
Sujith
,
I.
Pavithran
,
M.
Scheffer
,
T. M.
Lenton
,
M.
Anand
, and
C. T.
Bauch
, “
Deep learning for early warning signals of tipping points
,”
Proc. Natl. Acad. Sci. U.S.A.
118
,
2106140118
(
2021
).
33
L.-W.
Kong
,
H.-W.
Fan
,
C.
Grebogi
, and
Y.-C.
Lai
, “
Machine learning prediction of critical transition and system collapse
,”
Phys. Rev. Res.
3
,
013090
(
2021
).
34
R.
Xiao
,
L.-W.
Kong
,
Z.-K.
Sun
, and
Y.-C.
Lai
, “
Predicting amplitude death with machine learning
,”
Phys. Rev. E
104
,
014205
(
2021
).
35
M.
Ghil
and
V.
Lucarini
, “
The physics of climate variability and climate change
,”
Rev. Mod. Phys.
92
,
035002
(
2020
).
36
M. M.
Dekker
,
A. S.
von der Heydt
, and
H. A.
Dijkstra
, “
Cascading transitions in the climate system
,”
Earth Syst. Dynam.
9
,
1243
1260
(
2018
).
37
A.
Tantet
,
V.
Lucarini
,
F.
Lunkeit
, and
H.
Dijkstra
, “
Crisis of the chaotic attractor of a climate model: A transfer operator approach
,”
Nonlinearity
31
,
2221
(
2018
).
38
T. M.
Lenton
,
H.
Held
,
E.
Kriegler
,
J. W.
Hall
,
W.
Lucht
,
S.
Rahmstorf
, and
H. J.
Schellnhuber
, “
Tipping elements in the earth’s climate system
,”
Proc. Natl. Acad. Sci. U.S.A.
105
,
1786
1793
(
2008
).
39
S. O.
Rasmussen
,
M.
Bigler
,
S. P.
Blockley
,
T.
Blunier
,
S. L.
Buchardt
,
H. B.
Clausen
,
I.
Cvijanovic
,
D.
Dahl-Jensen
,
S. J.
Johnsen
,
H.
Fischer
,
V.
Gkinis
,
M.
Guillevic
,
W. Z.
Hoek
,
J. J.
Lowe
,
J. B.
Pedro
,
T.
Popp
,
I. K.
Seierstad
,
J. P.
Steffensen
,
A. M.
Svensson
,
P.
Vallelonga
,
B. M.
Vinther
,
M. J.
Walker
,
J. J.
Wheatley
, and
M.
Winstrup
, “
A stratigraphic framework for abrupt climatic changes during the last glacial period based on three synchronized Greenland ice-core records: Refining and extending the INTIMATE event stratigraphy
,”
Q. Sci. Rev.
106
,
14
28
(
2014
).
40
J. C.
Stager
and
P. A.
Mayewski
, “
Abrupt early to mid-holocene climatic transition registered at the equator and the poles
,”
Science
276
,
1834
1836
(
1997
).
41
S.
Drijfhout
,
S.
Bathiany
,
C.
Beaulieu
,
V.
Brovkin
,
M.
Claussen
,
C.
Huntingford
,
M.
Scheffer
,
G.
Sgubin
, and
D.
Swingedouw
, “
Catalogue of abrupt shifts in intergovernmental panel on climate change climate models
,”
Proc. Natl. Acad. Sci. U.S.A.
112
,
E5777
E5786
(
2015
).
42
A.
Zhang
,
G.
Knorr
,
G.
Lohmann
, and
S.
Barker
, “
Abrupt North Atlantic circulation changes in response to gradual CO2 forcing in a glacial climate state
,”
Nat. Geosci.
10
,
518
523
(
2017
).
43
S.
Rahmstorf
, “
Bifurcations of the atlantic thermohaline circulation in response to changes in the hydrological cycle
,”
Nature
378
,
145
149
(
1995
).
44
P.
Ashwin
and
A. S.
von der Heydt
, “
Extreme sensitivity and climate tipping points
,”
J. Stat. Phys.
179
,
1531
1552
(
2020
).
45
K.
Zickfeld
,
B.
Knopf
,
V.
Petoukhov
, and
H. J.
Schellnhuber
, “
Is the Indian summer monsoon stable against global change?
,”
Geophys. Res. Lett.
32
,
L15707
, https://doi.org/10.1029/2005GL022771 (
2005
).
46
Y. R.
Zelnik
,
S.
Kinast
,
H.
Yizhaq
,
G.
Bel
, and
E.
Meron
, “
Regime shifts in models of dryland vegetation
,”
Phil. Trans. R. Soc. A.
371
,
20120358
(
2013
).
47
T.
Amemiya
,
T.
Enomoto
,
A. G.
Rossberg
,
T.
Yamamoto
,
Y.
Inamori
, and
K.
Itoh
, “
Stability and dynamical behavior in a lake-model and implications for regime shifts in real lakes
,”
Ecolog. Model.
206
,
54
62
(
2007
).
48
G.
Fussmann
,
S. P.
Ellner
,
K. W.
Shertzer
, and
N. G.
Hairston
, Jr.
, “
Crossing the Hopf bifurcation in a live predator-prey system
,”
Science
290
,
1358
1360
(
2000
).
49
P. J.
Mumby
,
A.
Hastings
, and
H. J.
Edwards
, “
Thresholds and the resilience of caribbean coral reefs
,”
Nature
450
,
98
101
(
2007
).
50
P. E.
O’Keefe
and
S.
Wieczorek
,
“Tipping phenomena and points of no return in ecosystems: Beyond classical bifurcations
,”
SIAM J. Appl. Dyn. Syst.
19
(
4
),
2371
2402
(
2020
).
51
C.
Folke
,
S.
Carpenter
,
B.
Walker
,
M.
Scheffer
,
T.
Elmqvist
,
L.
Gunderson
, and
C.
Holling
, “
Regime shifts, resilience, and biodiversity in ecosystem management
,”
Ann. Rev. Ecol. Evolut. Syst.
35
,
557
581
(
2004
).
52
T. P.
Hughes
,
C.
Linares
,
V.
Dakos
,
I. A.
van de Leemput
, and
E. H.
van Nes
, “
Living dangerously on borrowed time during slow, unrecognized regime shifts
,”
Trends Ecol. Evolut.
28
,
149
155
(
2013
).
53
M.
Scheffer
,
S.
Carpenter
,
J. A.
Foley
,
C.
Folke
, and
B.
Walker
, “
Catastrophic shifts in ecosystems
,”
Nature
413
,
591
596
(
2001
).
54
D.
Alonso
,
A.
Dobson
, and
M.
Pascual
, “
Critical transitions in malaria transmission models are consistently generated by superinfection
,”
Philos. Trans. R. Soc. B: Biolog. Sci.
374
,
20180275
(
2019
).
55
Z.
Mukandavire
,
S.
Liao
,
J.
Wang
,
H.
Gaff
,
D. L.
Smith
, and
J. G.
Morris
, “
Estimating the reproductive numbers for the 2008–2009 cholera outbreaks in Zimbabwe
,”
Proc. Natl. Acad. Sci. U.S.A.
108
,
8767
8772
(
2011
).
56
Y.
Pomeau
and
P. M.
Manneville
, “
Intermittent transition to turbulence in dissipative dynamical systems
,”
Commun. Math. Phys.
74
,
189
197
(
1980
).
57
C.
Grebogi
,
E.
Ott
, and
J. A.
Yorke
, “
Critical exponents of chaotic transients in nonlinear dynamical systems
,”
Phys. Rev. Lett.
57
,
1284
(
1986
).
58
S.
Chiriac
,
D. G.
Dimitriu
, and
M.
Sanduloviciu
, “
Type I intermittency related to the spatiotemporal dynamics of double layers and ion-acoustic instabilities in plasma
,”
Phys. Plasmas.
14
,
072309
(
2007
).
59
P. Y.
Cheung
,
S.
Donovan
, and
A. Y.
Wong
, “
Observations of intermittent chaos in plasmas
,”
Phys. Rev. Lett.
61
,
1360
1363
(
1988
).
60
P. A.
Cassak
,
M. A.
Shay
, and
J. F.
Drake
, “
A saddle-node bifurcation model of magnetic reconnection onset
,”
Phys. Plasmas.
17
,
062105
(
2010
).
61
A. C.-L.
Chian
,
W. M.
Santana
,
E. L.
Rempel
,
F. A.
Borotto
,
T.
Hada
, and
Y.
Kamide
, “
Chaos in driven Alfvén systems: Unstable periodic orbits and chaotic saddles
,”
Nonlinear Proc. Geophys.
14
,
17
29
(
2007
).
62
E. F.
Manffra
,
I. L.
Caldas
,
R. L.
Viana
, and
H. J.
Kalinowski
, “
Type-I intermittency and crisis-induced intermittency in a semiconductor laser under injection current modulation
,”
Nonlinear. Dyn.
27
,
185
195
(
2002
).
63
J.
San-Martín
and
J.
Antoranz
, “
Transition to chaos via type-II intermittency in a laser with saturable absorber externally excited
,”
Progr. Theor. Phys.
94
,
535
542
(
1995
).
64
I.
Dobson
, “
Observations on the geometry of saddle node bifurcation and voltage collapse in electrical power systems
,”
IEEE Trans. Circuits Syst. I: Fundamental Theory Appl.
39
,
240
243
(
1992
).
65
C.-M.
Kim
,
G.-S.
Yim
,
Y. S.
Kim
,
J.-M.
Kim
, and
H. W.
Lee
, “
Experimental evidence of characteristic relations of type-i intermittency in an electronic circuit
,”
Phys. Rev. E
56
,
2573
2577
(
1997
).
66
J.-Y.
Huang
and
J.-J.
Kim
, “
Type-II intermittency in a coupled nonlinear oscillator: Experimental observation
,”
Phys. Rev. A
36
,
1495
1497
(
1987
).
67
L. E.
Guerrero
and
M.
Octavio
, “
Spatiotemporal effects in long rf-biased Josephson junctions: Chaotic transitions and intermittencies between dynamical attractors
,”
Phys. Rev. A
40
,
3371
3380
(
1989
).
68
G.
Penelet
,
T.
Watanabe
, and
T.
Biwa
, “
Study of a thermoacoustic-Stirling engine connected to a piston-crank-flywheel assembly
,”
J. Acoust. Soc. Am.
149
,
1674
1684
(
2021
).
69
Y.
Guan
,
V.
Gupta
, and
L. K. B.
Li
, “
Intermittency route to self-excited chaotic thermoacoustic oscillations
,”
J. Fluid. Mech.
894
,
R3
(
2020
).
70
L.
Kabiraj
and
R. I.
Sujith
, “
Nonlinear self-excited thermoacoustic oscillations: Intermittency and flame blowout
,”
J. Fluid. Mech.
713
,
376
397
(
2012
).
71
P.
Subramanian
,
S.
Mariappan
,
R. I.
Sujith
, and
P.
Wahi
, “
Bifurcation analysis of thermoacoustic instability in a horizontal rijke tube
,”
Inter. J. Spray Combust. Dyn.
2
,
325
355
(
2010
).
72
E.
Ringuet
,
C.
Rozé
, and
G.
Gouesbet
, “
Experimental observation of type-II intermittency in a hydrodynamic system
,”
Phys. Rev. E
47
,
1405
1407
(
1993
).
73
S.
Suresha
,
R. I.
Sujith
,
B.
Emerson
, and
T.
Lieuwen
, “
Nonlinear dynamics and intermittency in a turbulent reacting wake with density ratio as bifurcation parameter
,”
Phys. Rev. E
94
,
042206
(
2016
).
74
K.
Krischer
,
M.
Lüke
,
W.
Wolf
,
M.
Eiswirth
, and
G.
Ertl
, “
Chaos and interior crisis in an electrochemical reaction
,”
Berichte der Bunsengesellschaft für physikalische Chemie
95
,
820
823
(
1991
).
75
M.
Lukosevicius
and
H.
Jaeger
, “
Reservoir computing approaches to recurrent neural network training
,”
Comput. Sci. Rev.
3
,
127
149
(
2009
).
76
H.
Jaeger
, “The ‘echo state’ approach to analyzing and training recurrent neural networks-with an erratum note,” Type GMD Technical Report No. 148 (institution German National Research Center for Information Technology, Bonn, 2001).
77
W.
Maass
,
T.
Natschlager
, and
H.
Markram
, “
Real-time computing without stable states: A new framework for neural computation based on perturbations
,”
Neural Comput.
14
,
2531
2560
(
2002
).
78
E.
Lorenz
, “
Deterministic nonperiodic flow
,”
J. Atmos. Sci.
20
,
130
(
1963
).
79
S. M.
Hammel
,
C. K. R.
T. Jones
, and
J. V.
Moloney
, “
Global dynamical behavior of the optical field in a ring cavity
,”
J. Opt. Soc. Am. B
2
,
552
564
(
1985
).
80
Y.
Kuramoto
, “
Diffusion-induced chaos in reaction systems
,”
Prog. Theor. Phys. Suppl.
64
,
346
367
(
1978
).
81
G. I.
Sivashinsky
, “
On flame propagation under conditions of stoichiometry
,”
SIAM J. Appl. Math.
39
,
67
82
(
1980
).
82
A.
Wikner
,
B.
Hunt
,
J.
Harvey
,
M.
Girvan
, and
E.
Ott
, “Stabilizing machine learning prediction of dynamics: Noise and noise-inspired regularization” (to be published).
83
S. S.
Vallender
, “
Calculation of the Wasserstein distance between probability distributions on the line
,”
Theory Probab. Appl.
18
,
784
786
(
1973
).
84
A.
Mishra
,
S.
Leo Kingston
,
C.
Hens
,
T.
Kapitaniak
,
U.
Feudel
, and
S. K.
Dana
, “
Routes to extreme events in dynamical systems: Dynamical and statistical characteristics
,”
Chaos
30
,
063114
(
2020
).
85
K. E.
Trenberth
,
A.
Dai
,
R. M.
Rasmussen
, and
D. B.
Parsons
, “
The changing character of precipitation
,”
Bull. Am. Meteorol. Soc.
84
,
1205
1218
(
2003
).
86
B. N.
Goswami
,
V.
Venugopal
,
D.
Sengupta
,
M. S.
Madhusoodanan
, and
P. K.
Xavier
, “
Increasing trend of extreme rain events over india in a warming environment
,”
Science
314
,
1442
1445
(
2006
).
87
A.
Ray
,
S.
Rakshit
,
D.
Ghosh
, and
S. K.
Dana
, “
Intermittent large deviation of chaotic trajectory in the Ikeda map: Signature of extreme events
,”
Chaos
29
,
043131
(
2019
).
88
A.-K.
Kassam
and
L. N.
Trefethen
, “
Fourth-order time-stepping for stiff PDEs
,”
SIAM J. Sci. Comput.
26
,
1214
1233
(
2005
).
89
R. A.
Edson
,
J. E.
Bunder
,
T. W.
Mattner
, and
A. J.
Roberts
, “
Lyapunov exponents of the Kuramoto–Sivashinsky PDE
,”
Anziam J.
61
,
270
285
(
2019
).
90
J. A.
Yorke
and
E. D.
Yorke
, “
Metastable chaos: The transition to sustained chaotic behavior in the lorenz model
,”
J. Stat. Phys.
21
,
263
277
(
1979
).
91
E.
Ott
,
Chaos in Dynamical Systems
, 2nd ed. (
Cambridge University Press
,
2002
).

Supplementary Material

You do not currently have access to this content.