To predict rare extreme events using deep neural networks, one encounters the so-called small data problem because even long-term observations often contain few extreme events. Here, we investigate a model-assisted framework where the training data are obtained from numerical simulations, as opposed to observations, with adequate samples from extreme events. However, to ensure the trained networks are applicable in practice, the training is not performed on the full simulation data; instead, we only use a small subset of observable quantities, which can be measured in practice. We investigate the feasibility of this model-assisted framework on three different dynamical systems (Rössler attractor, FitzHugh–Nagumo model, and a turbulent fluid flow) and three different deep neural network architectures (feedforward, long short-term memory, and reservoir computing). In each case, we study the prediction accuracy, robustness to noise, reproducibility under repeated training, and sensitivity to the type of input data. In particular, we find long short-term memory networks to be most robust to noise and to yield relatively accurate predictions, while requiring minimal fine-tuning of the hyperparameters.
REFERENCES
1
L. K.
Comfort
, A.
Boin
, and C. C.
Demchak
, Designing Resilience: Preparing for Extreme Events
(University of Pittsburgh
, 2010
).2
M.
Farazmand
and T. P.
Sapsis
, “Extreme events: Mechanisms and prediction
,” Appl. Mech. Rev.
71
, 050801
(2019
). 3
L. E.
McPhillips
, H.
Chang
, M. V.
Chester
, Y.
Depietri
, E.
Friedman
, N. B.
Grimm
, J. S.
Kominoski
, T.
McPhearson
, P.
Méndez-Lázaro
, E. J.
Rosi
, and J.
Shafiei Shiva
, “Defining extreme events: A cross-disciplinary review
,” Earth’s Future
6
, 441
–455
(2018
). 4
J. L.
Callaham
, K.
Maeda
, and S. L.
Brunton
, “Robust flow reconstruction from limited measurements via sparse representation
,” Phys. Rev. Fluids
4
, 103907
(2019
). 5
B.
Chu
and M.
Farazmand
, “Data-driven prediction of multistable systems from sparse measurements
,” Chaos
31
, 063118
(2021
). 6
B.
Kramer
, P.
Grover
, P.
Boufounos
, S.
Nabi
, and M.
Benosman
, “Sparse sensing and DMD-based identification of flow regimes and bifurcations in complex flows
,” SIAM J. Appl. Dyn. Syst.
16
, 1164
–1196
(2017
). 7
M.
Farazmand
and T. P.
Sapsis
, “Reduced-order prediction of rogue waves in two-dimensional deep-water waves
,” J. Comput. Phys.
340
, 418
–434
(2017
). 8
A.
Mendez
and M.
Farazmand
, “Investigating climate tipping points under various emission reduction and carbon capture scenarios with a stochastic climate model,” Proc. R. Soc. London A
, 477
, 20210697
(2021
).9
J.
Bucklew
, Introduction to Rare Event Simulation
(Springer Science & Business Media
, 2004
).10
G.
Dematteis
, T.
Grafke
, and E.
Vanden-Eijnden
, “Rogue waves and large deviations in deep sea
,” Proc. Natl. Acad. Sci. U.S.A.
115
, 855
–860
(2018
). 11
M. A.
Mohamad
and T. P.
Sapsis
, “Sequential sampling strategy for extreme event statistics in nonlinear dynamical systems
,” Proc. Natl. Acad. Sci. U.S.A.
115
, 11138
–11143
(2018
). 12
S. L.
Brunton
, B. R.
Noack
, and P.
Koumoutsakos
, “Machine learning for fluid mechanics
,” Annu. Rev. Fluid Mech.
52
, 477
–508
(2020
). 13
H. I.
Fawaz
, G.
Forestier
, J.
Weber
, L.
Idoumghar
, and P.
Muller
, “Deep learning for time series classification: A review
,” Data Min. Knowl. Discov.
33
, 917
–963
(2019
). 14
G. E.
Karniadakis
, I. G.
Kevrekidis
, L.
Lu
, P.
Perdikaris
, S.
Wang
, and L.
Yang
, “Physics-informed machine learning
,” Nat. Rev. Phys.
3
, 422
–440
(2021
). 15
D.
Kochkov
, J. A.
Smith
, A.
Alieva
, Q.
Wang
, M. P.
Brenner
, and S.
Hoyer
, “Machine learning-accelerated computational fluid dynamics
,” Proc. Natl. Acad. Sci. U.S.A.
118
, e2101784118
(2021
). 16
J.
Pathak
, Z.
Lu
, B. R.
Hunt
, M.
Girvan
, and E.
Ott
, “Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data
,” Chaos
27
, 121102
(2017
). 17
D.
Ding
, M.
Zhang
, X.
Pan
, M.
Yang
, and X.
He
, “Modeling extreme events in time series prediction,” in Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD ’19 (Association for Computing Machinery, New York, NY, 2019), pp. 1114–1122.18
D.
Qi
and A. J.
Majda
, “Using machine learning to predict extreme events in complex systems
,” Proc. Natl. Acad. Sci. U.S.A.
117
, 52
–59
(2020
). 19
S. H.
Rudy
and T. P.
Sapsis
, “Output-weighted and relative entropy loss functions for deep learning precursors of extreme events,”arXiv:abs/2112.00825 (2021).20
J.
Meiyazhagan
, S.
Sudharsan
, and M.
Senthilvelan
, “Model-free prediction of emergence of extreme events in a parametrically driven nonlinear dynamical system by deep learning
,” Eur. Phys. J. B
94
, 1
–13
(2021
). 21
M.
Lellep
, J.
Prexl
, M.
Linkmann
, and B.
Eckhardt
, “Using machine learning to predict extreme events in the Hénon map
,” Chaos
30
, 013113
(2020
). 22
M.
Närhi
, L.
Salmela
, J.
Toivonen
, C.
Billet
, J. M.
Dudley
, and G.
Genty
, “Machine learning analysis of extreme events in optical fibre modulation instability
,” Nat. Commun.
9
, 1
–11
(2018
). 23
P. K.
Yeditha
, V.
Kasi
, M.
Rathinasamy
, and A.
Agarwal
, “Forecasting of extreme flood events using different satellite precipitation products and wavelet-based machine learning methods
,” Chaos
30
, 063115
(2020
). 24
Z. Y.
Wan
, P.
Vlachas
, P.
Koumoutsakos
, and T.
Sapsis
, “Data-assisted reduced-order modeling of extreme events in complex dynamical systems
,” PLoS One
13
, e0197704
(2018
).25
A.
Chattopadhyay
, E.
Nabizadeh
, and P.
Hassanzadeh
, “Analog forecasting of extreme-causing weather patterns using deep learning
,” J. Adv. Model. Earth Syst.
12
, e2019MS001958
(2020
). 26
S. H.
Rudy
and T. P.
Sapsis
, “Prediction of intermittent fluctuations from surface pressure measurements on a turbulent airfoil,” AIAA
(online 2022).27
A.
Chattopadhyay
, P.
Hassanzadeh
, and D.
Subramanian
, “Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: Reservoir computing, artificial neural network, and long short-term memory network
,” Nonlinear Process. Geophys.
27
, 373
–389
(2020
). 28
V.
Pyragas
and K.
Pyragas
, “Using reservoir computer to predict and prevent extreme events
,” Phys. Lett. A
384
, 126591
(2020
). 29
M.
Farazmand
and T. P.
Sapsis
, “Dynamical indicators for the prediction of bursting phenomena in high-dimensional systems
,” Phys. Rev. E
94
, 032212
(2016
). 30
P. J.
Blonigan
, M.
Farazmand
, and T. P.
Sapsis
, “Are extreme dissipation events predictable in turbulent fluid flows?
,” Phys. Rev. Fluids
4
, 044606
(2019
). 31
G.
Ansmann
, R.
Karnatak
, K.
Lehnertz
, and U.
Feudel
, “Extreme events in excitable systems and mechanisms of their generation
,” Phys. Rev. E
88
, 052911
(2013
). 32
M.
Farazmand
, “An adjoint-based approach for finding invariant solutions of Navier-Stokes equations
,” J. Fluid Mech.
795
, 278
–312
(2016
). 33
M.
Farazmand
and T. P.
Sapsis
, “A variational approach to probing extreme events in turbulent dynamical systems
,” Sci. Adv.
3
, e1701533
(2017
). 34
J. J.
Moré
, “The Levenberg-Marquardt algorithm: Implementation and theory,” in Numerical Analysis, edited by G. A. Watson (Springer, Berlin, 1978), pp. 105–116.35
F.
Takens
, “Detecting strange attractors in turbulence,” in Dynamical Systems and Turbulence, Warwick 1980, edited by D. Rand and L. Young (Springer, Berlin, 1981), pp. 366–381.36
S.
Hochreiter
and J.
Schmidhuber
, “Long short-term memory
,” Neural Comput.
9
, 1735
–1780
(1997
). 37
A.
Sherstinsky
, “Fundamentals of recurrent neural network (RNN) and long short-term memory (LSTM) network
,” Physica D
404
, 132306
(2020
). 38
B.
Schrauwen
, D.
Verstraeten
, and J.
Van Campenhout
, “An overview of reservoir computing: Theory, applications and implementations,” in Proceedings of the 15th European Symposium on Artificial Neural Networks (i6doc.com, 2007), pp. 471–482.39
A.
Asch
, E. J.
Brady
, H.
Gallardo
, J.
Hood
, B.
Chu
, and M.
Farazmand
, “GitHub repository: Deep learning for extreme events
,” (2021); available at https://github.com/mfarazmand/DeepLearningExtremeEvents.40
R.
Zimmermann
, see https://github.com/kalekiu/easyesn/ for “Easyesn: A Library for Recurrent Neural Networks Using Echo State Networks,” version 0.1.6.1 (2021).41
S.
Guth
and T. P.
Sapsis
, “Machine learning predictors of extreme events occurring in complex dynamical systems
,” Entropy
21
, 925
(2019
). 42
J.
Davis
and M.
Goadrich
, “The relationship between precision-recall and ROC curves,” in Proceedings of the 23rd International Conference on Machine Learning, ICML ’06 (Association for Computing Machinery, New York, NY, 2006), pp. 233–240.43
T.
Saito
and M.
Rehmsmeier
, “The precision-recall plot is more informative than the ROC plot when evaluating binary classifiers on imbalanced datasets
,” PLoS One
10
, e0118432
(2015
).44
G.
An
, “The effects of adding noise during backpropagation training on a generalization performance
,” Neural Comput.
8
, 643
–674
(1996
). 45
R. M.
Zur
, Y.
Jiang
, L. L.
Pesce
, and K.
Drukker
, “Noise injection for training artificial neural networks: A comparison with weight decay and early stopping
,” Med. Phys.
36
, 4810
–4818
(2009
). 46
S. R.
Young
, D. C.
Rose
, T. P.
Karnowski
, S.-H.
Lim
, and R. M.
Patton
, “Optimizing deep learning hyper-parameters through an evolutionary algorithm,” in Proceedings of the Workshop on Machine Learning in High-Performance Computing Environments, MLHPC ’15 (Association for Computing Machinery, 2015).47
R.
Karnatak
, G.
Ansmann
, U.
Feudel
, and K.
Lehnertz
, “Route to extreme events in excitable systems
,” Phys. Rev. E
90
, 022917
(2014
). © 2022 Author(s). Published under an exclusive license by AIP Publishing.
2022
Author(s)
You do not currently have access to this content.
