The ability of machine learning (ML) models to “extrapolate” to situations outside of the range spanned by their training data is crucial for predicting the long-term behavior of non-stationary dynamical systems (e.g., prediction of terrestrial climate change), since the future trajectories of such systems may (perhaps after crossing a tipping point) explore regions of state space which were not explored in past time-series measurements used as training data. We investigate the extent to which ML methods can yield useful results by extrapolation of such training data in the task of forecasting non-stationary dynamics, as well as conditions under which such methods fail. In general, we find that ML can be surprisingly effective even in situations that might appear to be extremely challenging, but do (as one would expect) fail when “too much” extrapolation is required. For the latter case, we show that good results can potentially be obtained by combining the ML approach with an available inaccurate conventional model based on scientific knowledge.
REFERENCES
1
B.
Kaszas
, U.
Feudel
, and T.
Tel
, “Tipping phenomena in typical dynamical systems subjected to parameter drift
,” Sci. Rep.
9
, 8654
(2019
).2
U.
Feudel
, A. N.
Pisarchik
, and K.
Showalter
, “Multistability and tipping: From mathematics and physics to climate and brain—minireview and preface to the focus issue
,” Chaos
28
, 033501
(2018
). 3
H.
Jaeger
and H.
Haas
, “Harnessing nonlinearity: Predicting chaotic systems and saving energy in wireless communication
,” Science
304
, 78
(2004
). 4
J.
Pathak
, B. R.
Hunt
, M.
Girvan
, Z.
Lu
, and E.
Ott
, “Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach
,” Phys. Rev. Lett.
120
, 024102
(2018
). 5
J.
Pathak
, A.
Wikner
, R.
Fussel
, S.
Chandra
, B.
Hunt
, M.
Girvan
, and E.
Ott
, “Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge based model
,” Chaos
28
, 041101
(2018
). 6
A.
Vaughan
and S. V.
Bohac
, “Real-time, adaptive machine learning for non-stationary, near chaotic gasoline engine combustion time series
,” Neural Netw.
70
, 18
–26
(2015
). 7
A.
Griffith
, A.
Pomerance
, and D. J.
Gauthier
, “Forecasting chaotic systems with very low connectivity reservoir computers
,” Chaos
29
, 123108
(2019
). 8
G.
Shen
, J.
Kurths
, and Y.
Yuan
, “Sequence-to-sequence prediction of spatiotemporal systems
,” Chaos
30
, 023102
(2020
). 9
R.
Follmann
and J. E.
Rosa
, “Predicting slow and fast neuronal dynamics with machine learning
,” Chaos
29
, 113119
(2019
). 10
S.
Herzog
, F.
Wörgötter
, and U.
Parlitz
, “Convolutional autoencoder and conditional random fields hybrid for predicting spatial-temporal chaos
,” Chaos
29
, 123116
(2019
). 11
A.
Chattopadhyay
, P.
Hassanzadeh
, and D.
Subramanian
, “Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: Reservoir computing, artificial neural network, and long short-term memory network
,” Nonlin. Processes Geophys.
27
, 373
–389
(2020
). 12
D.
Chen
and W.
Han
, “Prediction of multivariate chaotic time series via radial basis function neural networks
,” Complexity
18
, 55
(2013
). 13
W.
Huang
, Y.
Li
, and Y.
Huang
, “Deep hybrid neural network and improved differential neuroevolution for chaotic time series prediction
,” IEEE Access
8
, 159552
–159565
(2020
). 14
H.
Fan
, J.
Jiang
, C.
Zhang
, X.
Wang
, and Y.-C.
Lai
, “Long-term prediction of chaotic systems with machine learning
,” Phys. Rev. Res.
2
, 012080
(2020
). 15
A.
Faqih
, A. P.
Lianto
, and B.
Kusumoputro
, “Mackey-Glass chaotic time series prediction using modified RBF neural networks,” in Proceedings of the 2nd International Conference on Software Engineering and Information Management (ICSIM 2019), series and number ICSIM 2019 (Association for Computing Machinery, New York, NY, 2019), pp. 7–11.16
D.
Canaday
, A.
Griffith
, and D. J.
Gauthier
, “Rapid time series prediction with a hardware-based reservoir computer
,” Chaos
28
, 123119
(2018
). 17
F.
Borra
, A.
Vulpiani
, and M.
Cencini
, “Effective models and predictability of chaotic multiscale systems via machine learning
,” Phys. Rev. E
102
, 052203
(2020
). 18
J.
Pathak
, Z.
Lu
, B. R.
Hunt
, M.
Girvan
, and E.
Ott
, “Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data
,” Chaos
27
, 121102
(2017
). 19
Z.
Lu
, B. R.
Hunt
, and E.
Ott
, “Attractor reconstruction by machine learning
,” Chaos
28
, 061104
(2018
). 20
P.
Antonik
, M.
Gulina
, J.
Pauwels
, and S.
Massar
, “Using a reservoir computer to learn chaotic attractors, with applications in chaos synchronization and cryptography
,” Phys. Rev. E
98
, 012215
(2018
). 21
D.
Nguyen
, S.
Ouala
, L.
Drumetz
, and R.
Fablet
, “EM-like learning chaotic dynamics from noisy and partial observations,” arXiv:abs/1903.10335 (2019).22
J. P.
Eckmann
and D.
Ruelle
, “Ergodic theory of chaos and strange attractors
,” Rev. Mod. Phys.
57
, 617
(1985
). 23
G.
Drótos
, T.
Bódai
, and T.
Tél
, “Quantifying nonergodicity in nonautonomous dissipative dynamical systems: An application to climate change
,” Phys. Rev. E
94
, 022214
(2016
).24
G.
Drótos
, T.
Bódai
, and T.
Tél
, “Probabilistic concepts in a changing climate: A snapshot attractor picture
,” J. Clim.
28
, 3275
–3288
(2015
).25
F. J.
Romeiras
, C.
Grebogi
, and E.
Ott
, “Multifractal properties of snapshot attractors of random maps
,” Phys. Rev. A
41
, 784
–799
(1990
). 26
M. D.
Chekroun
, E.
Simonnet
, and M.
Ghil
, “Stochastic climate dynamics: Random attractors and time-dependent invariant measures
,” Physica D
240
, 1685
–1700
(2011
). 27
D.
Patel
, D.
Canaday
, M.
Girvan
, A.
Pomerance
, and E.
Ott
, “Using machine learning to predict statistical properties of non-stationary dynamical processes: System climate, regime transitions, and the effect of stochasticity
,” Chaos
31
, 033149
(2021
). 28
S. H.
Lim
, L.
Theo Giorgini
, W.
Moon
, and J. S.
Wettlaufer
, “Predicting critical transitions in multiscale dynamical systems using reservoir computing
,” Chaos
30
, 123126
(2020
). 29
F. Z.
Xing
, E.
Cambria
, and X.
Zou
, “Predicting evolving chaotic time series with fuzzy neural networks,” in 2017 International Joint Conference on Neural Networks (IJCNN)
(IEEE, 2017), pp. 3176–3183.30
Z.
Shi
, Y.
Bai
, X.
Jin
, X.
Wang
, T.
Su
, and J.
Kong
, “Parallel deep prediction with covariance intersection fusion on non-stationary time series
,” Know.-Based Syst.
211
, 106523
(2021
). 31
A.
Pershin
, C.
Beaume
, K.
Li
, and S. M.
Tobias
, “Can neural networks predict dynamics they have never seen?” arXiv:abs/2111.06783 (2021).32
T. M.
Bury
, R. I.
Sujith
, I.
Pavithran
, M.
Scheffer
, T. M.
Lenton
, M.
Anand
, and C. T.
Bauch
, “Deep learning for early warning signals of tipping points
,” Proc. Natl. Acad. Sci. U.S.A.
118
, 2106140118
(2021
). 33
L.-W.
Kong
, H.-W.
Fan
, C.
Grebogi
, and Y.-C.
Lai
, “Machine learning prediction of critical transition and system collapse
,” Phys. Rev. Res.
3
, 013090
(2021
). 34
R.
Xiao
, L.-W.
Kong
, Z.-K.
Sun
, and Y.-C.
Lai
, “Predicting amplitude death with machine learning
,” Phys. Rev. E
104
, 014205
(2021
). 35
M.
Ghil
and V.
Lucarini
, “The physics of climate variability and climate change
,” Rev. Mod. Phys.
92
, 035002
(2020
). 36
M. M.
Dekker
, A. S.
von der Heydt
, and H. A.
Dijkstra
, “Cascading transitions in the climate system
,” Earth Syst. Dynam.
9
, 1243
–1260
(2018
). 37
A.
Tantet
, V.
Lucarini
, F.
Lunkeit
, and H.
Dijkstra
, “Crisis of the chaotic attractor of a climate model: A transfer operator approach
,” Nonlinearity
31
, 2221
(2018
). 38
T. M.
Lenton
, H.
Held
, E.
Kriegler
, J. W.
Hall
, W.
Lucht
, S.
Rahmstorf
, and H. J.
Schellnhuber
, “Tipping elements in the earth’s climate system
,” Proc. Natl. Acad. Sci. U.S.A.
105
, 1786
–1793
(2008
). 39
S. O.
Rasmussen
, M.
Bigler
, S. P.
Blockley
, T.
Blunier
, S. L.
Buchardt
, H. B.
Clausen
, I.
Cvijanovic
, D.
Dahl-Jensen
, S. J.
Johnsen
, H.
Fischer
, V.
Gkinis
, M.
Guillevic
, W. Z.
Hoek
, J. J.
Lowe
, J. B.
Pedro
, T.
Popp
, I. K.
Seierstad
, J. P.
Steffensen
, A. M.
Svensson
, P.
Vallelonga
, B. M.
Vinther
, M. J.
Walker
, J. J.
Wheatley
, and M.
Winstrup
, “A stratigraphic framework for abrupt climatic changes during the last glacial period based on three synchronized Greenland ice-core records: Refining and extending the INTIMATE event stratigraphy
,” Q. Sci. Rev.
106
, 14
–28
(2014
). 40
J. C.
Stager
and P. A.
Mayewski
, “Abrupt early to mid-holocene climatic transition registered at the equator and the poles
,” Science
276
, 1834
–1836
(1997
). 41
S.
Drijfhout
, S.
Bathiany
, C.
Beaulieu
, V.
Brovkin
, M.
Claussen
, C.
Huntingford
, M.
Scheffer
, G.
Sgubin
, and D.
Swingedouw
, “Catalogue of abrupt shifts in intergovernmental panel on climate change climate models
,” Proc. Natl. Acad. Sci. U.S.A.
112
, E5777
–E5786
(2015
). 42
A.
Zhang
, G.
Knorr
, G.
Lohmann
, and S.
Barker
, “Abrupt North Atlantic circulation changes in response to gradual forcing in a glacial climate state
,” Nat. Geosci.
10
, 518
–523
(2017
). 43
S.
Rahmstorf
, “Bifurcations of the atlantic thermohaline circulation in response to changes in the hydrological cycle
,” Nature
378
, 145
–149
(1995
). 44
P.
Ashwin
and A. S.
von der Heydt
, “Extreme sensitivity and climate tipping points
,” J. Stat. Phys.
179
, 1531
–1552
(2020
). 45
K.
Zickfeld
, B.
Knopf
, V.
Petoukhov
, and H. J.
Schellnhuber
, “Is the Indian summer monsoon stable against global change?
,” Geophys. Res. Lett.
32
, L15707
, https://doi.org/10.1029/2005GL022771 (2005
). 46
Y. R.
Zelnik
, S.
Kinast
, H.
Yizhaq
, G.
Bel
, and E.
Meron
, “Regime shifts in models of dryland vegetation
,” Phil. Trans. R. Soc. A.
371
, 20120358
(2013
). 47
T.
Amemiya
, T.
Enomoto
, A. G.
Rossberg
, T.
Yamamoto
, Y.
Inamori
, and K.
Itoh
, “Stability and dynamical behavior in a lake-model and implications for regime shifts in real lakes
,” Ecolog. Model.
206
, 54
–62
(2007
). 48
G.
Fussmann
, S. P.
Ellner
, K. W.
Shertzer
, and N. G.
Hairston
, Jr., “Crossing the Hopf bifurcation in a live predator-prey system
,” Science
290
, 1358
–1360
(2000
). 49
P. J.
Mumby
, A.
Hastings
, and H. J.
Edwards
, “Thresholds and the resilience of caribbean coral reefs
,” Nature
450
, 98
–101
(2007
). 50
P. E.
O’Keefe
and S.
Wieczorek
, “Tipping phenomena and points of no return in ecosystems: Beyond classical bifurcations
,” SIAM J. Appl. Dyn. Syst.
19
(4
), 2371
–2402
(2020
).51
C.
Folke
, S.
Carpenter
, B.
Walker
, M.
Scheffer
, T.
Elmqvist
, L.
Gunderson
, and C.
Holling
, “Regime shifts, resilience, and biodiversity in ecosystem management
,” Ann. Rev. Ecol. Evolut. Syst.
35
, 557
–581
(2004
). 52
T. P.
Hughes
, C.
Linares
, V.
Dakos
, I. A.
van de Leemput
, and E. H.
van Nes
, “Living dangerously on borrowed time during slow, unrecognized regime shifts
,” Trends Ecol. Evolut.
28
, 149
–155
(2013
). 53
M.
Scheffer
, S.
Carpenter
, J. A.
Foley
, C.
Folke
, and B.
Walker
, “Catastrophic shifts in ecosystems
,” Nature
413
, 591
–596
(2001
). 54
D.
Alonso
, A.
Dobson
, and M.
Pascual
, “Critical transitions in malaria transmission models are consistently generated by superinfection
,” Philos. Trans. R. Soc. B: Biolog. Sci.
374
, 20180275
(2019
). 55
Z.
Mukandavire
, S.
Liao
, J.
Wang
, H.
Gaff
, D. L.
Smith
, and J. G.
Morris
, “Estimating the reproductive numbers for the 2008–2009 cholera outbreaks in Zimbabwe
,” Proc. Natl. Acad. Sci. U.S.A.
108
, 8767
–8772
(2011
). 56
Y.
Pomeau
and P. M.
Manneville
, “Intermittent transition to turbulence in dissipative dynamical systems
,” Commun. Math. Phys.
74
, 189
–197
(1980
). 57
C.
Grebogi
, E.
Ott
, and J. A.
Yorke
, “Critical exponents of chaotic transients in nonlinear dynamical systems
,” Phys. Rev. Lett.
57
, 1284
(1986
). 58
S.
Chiriac
, D. G.
Dimitriu
, and M.
Sanduloviciu
, “Type I intermittency related to the spatiotemporal dynamics of double layers and ion-acoustic instabilities in plasma
,” Phys. Plasmas.
14
, 072309
(2007
). 59
P. Y.
Cheung
, S.
Donovan
, and A. Y.
Wong
, “Observations of intermittent chaos in plasmas
,” Phys. Rev. Lett.
61
, 1360
–1363
(1988
). 60
P. A.
Cassak
, M. A.
Shay
, and J. F.
Drake
, “A saddle-node bifurcation model of magnetic reconnection onset
,” Phys. Plasmas.
17
, 062105
(2010
). 61
A. C.-L.
Chian
, W. M.
Santana
, E. L.
Rempel
, F. A.
Borotto
, T.
Hada
, and Y.
Kamide
, “Chaos in driven Alfvén systems: Unstable periodic orbits and chaotic saddles
,” Nonlinear Proc. Geophys.
14
, 17
–29
(2007
). 62
E. F.
Manffra
, I. L.
Caldas
, R. L.
Viana
, and H. J.
Kalinowski
, “Type-I intermittency and crisis-induced intermittency in a semiconductor laser under injection current modulation
,” Nonlinear. Dyn.
27
, 185
–195
(2002
). 63
J.
San-Martín
and J.
Antoranz
, “Transition to chaos via type-II intermittency in a laser with saturable absorber externally excited
,” Progr. Theor. Phys.
94
, 535
–542
(1995
). 64
I.
Dobson
, “Observations on the geometry of saddle node bifurcation and voltage collapse in electrical power systems
,” IEEE Trans. Circuits Syst. I: Fundamental Theory Appl.
39
, 240
–243
(1992
). 65
C.-M.
Kim
, G.-S.
Yim
, Y. S.
Kim
, J.-M.
Kim
, and H. W.
Lee
, “Experimental evidence of characteristic relations of type-i intermittency in an electronic circuit
,” Phys. Rev. E
56
, 2573
–2577
(1997
). 66
J.-Y.
Huang
and J.-J.
Kim
, “Type-II intermittency in a coupled nonlinear oscillator: Experimental observation
,” Phys. Rev. A
36
, 1495
–1497
(1987
). 67
L. E.
Guerrero
and M.
Octavio
, “Spatiotemporal effects in long rf-biased Josephson junctions: Chaotic transitions and intermittencies between dynamical attractors
,” Phys. Rev. A
40
, 3371
–3380
(1989
). 68
G.
Penelet
, T.
Watanabe
, and T.
Biwa
, “Study of a thermoacoustic-Stirling engine connected to a piston-crank-flywheel assembly
,” J. Acoust. Soc. Am.
149
, 1674
–1684
(2021
). 69
Y.
Guan
, V.
Gupta
, and L. K. B.
Li
, “Intermittency route to self-excited chaotic thermoacoustic oscillations
,” J. Fluid. Mech.
894
, R3
(2020
). 70
L.
Kabiraj
and R. I.
Sujith
, “Nonlinear self-excited thermoacoustic oscillations: Intermittency and flame blowout
,” J. Fluid. Mech.
713
, 376
–397
(2012
). 71
P.
Subramanian
, S.
Mariappan
, R. I.
Sujith
, and P.
Wahi
, “Bifurcation analysis of thermoacoustic instability in a horizontal rijke tube
,” Inter. J. Spray Combust. Dyn.
2
, 325
–355
(2010
). 72
E.
Ringuet
, C.
Rozé
, and G.
Gouesbet
, “Experimental observation of type-II intermittency in a hydrodynamic system
,” Phys. Rev. E
47
, 1405
–1407
(1993
). 73
S.
Suresha
, R. I.
Sujith
, B.
Emerson
, and T.
Lieuwen
, “Nonlinear dynamics and intermittency in a turbulent reacting wake with density ratio as bifurcation parameter
,” Phys. Rev. E
94
, 042206
(2016
). 74
K.
Krischer
, M.
Lüke
, W.
Wolf
, M.
Eiswirth
, and G.
Ertl
, “Chaos and interior crisis in an electrochemical reaction
,” Berichte der Bunsengesellschaft für physikalische Chemie
95
, 820
–823
(1991
). 75
M.
Lukosevicius
and H.
Jaeger
, “Reservoir computing approaches to recurrent neural network training
,” Comput. Sci. Rev.
3
, 127
–149
(2009
).76
H.
Jaeger
, “The ‘echo state’ approach to analyzing and training recurrent neural networks-with an erratum note,” Type GMD Technical Report No. 148 (institution German National Research Center for Information Technology, Bonn, 2001).77
W.
Maass
, T.
Natschlager
, and H.
Markram
, “Real-time computing without stable states: A new framework for neural computation based on perturbations
,” Neural Comput.
14
, 2531
–2560
(2002
). 78
E.
Lorenz
, “Deterministic nonperiodic flow
,” J. Atmos. Sci.
20
, 130
(1963
). 79
S. M.
Hammel
, C. K. R.
T. Jones
, and J. V.
Moloney
, “Global dynamical behavior of the optical field in a ring cavity
,” J. Opt. Soc. Am. B
2
, 552
–564
(1985
). 80
Y.
Kuramoto
, “Diffusion-induced chaos in reaction systems
,” Prog. Theor. Phys. Suppl.
64
, 346
–367
(1978
). 81
G. I.
Sivashinsky
, “On flame propagation under conditions of stoichiometry
,” SIAM J. Appl. Math.
39
, 67
–82
(1980
). 82
A.
Wikner
, B.
Hunt
, J.
Harvey
, M.
Girvan
, and E.
Ott
, “Stabilizing machine learning prediction of dynamics: Noise and noise-inspired regularization” (to be published).83
S. S.
Vallender
, “Calculation of the Wasserstein distance between probability distributions on the line
,” Theory Probab. Appl.
18
, 784
–786
(1973
). 84
A.
Mishra
, S.
Leo Kingston
, C.
Hens
, T.
Kapitaniak
, U.
Feudel
, and S. K.
Dana
, “Routes to extreme events in dynamical systems: Dynamical and statistical characteristics
,” Chaos
30
, 063114
(2020
). 85
K. E.
Trenberth
, A.
Dai
, R. M.
Rasmussen
, and D. B.
Parsons
, “The changing character of precipitation
,” Bull. Am. Meteorol. Soc.
84
, 1205
–1218
(2003
). 86
B. N.
Goswami
, V.
Venugopal
, D.
Sengupta
, M. S.
Madhusoodanan
, and P. K.
Xavier
, “Increasing trend of extreme rain events over india in a warming environment
,” Science
314
, 1442
–1445
(2006
). 87
A.
Ray
, S.
Rakshit
, D.
Ghosh
, and S. K.
Dana
, “Intermittent large deviation of chaotic trajectory in the Ikeda map: Signature of extreme events
,” Chaos
29
, 043131
(2019
). 88
A.-K.
Kassam
and L. N.
Trefethen
, “Fourth-order time-stepping for stiff PDEs
,” SIAM J. Sci. Comput.
26
, 1214
–1233
(2005
). 89
R. A.
Edson
, J. E.
Bunder
, T. W.
Mattner
, and A. J.
Roberts
, “Lyapunov exponents of the Kuramoto–Sivashinsky PDE
,” Anziam J.
61
, 270
–285
(2019
). 90
J. A.
Yorke
and E. D.
Yorke
, “Metastable chaos: The transition to sustained chaotic behavior in the lorenz model
,” J. Stat. Phys.
21
, 263
–277
(1979
). 91
E.
Ott
, Chaos in Dynamical Systems
, 2nd ed. (Cambridge University Press
, 2002
).© 2023 Author(s). Published under an exclusive license by AIP Publishing.
2023
Author(s)
You do not currently have access to this content.