We suggest a new method for building data-driven dynamical models from observed multidimensional time series. The method is based on a recurrent neural network with specific structure, which allows for the joint reconstruction of both a low-dimensional embedding for dynamical components in the data and an evolution operator. The key link of the method is a Bayesian optimization of both model structure and the hypothesis about the data generating law, which is needed for constructing the cost function for model learning. First, the performance of the method is successfully tested in the situation when a signal from a low-dimensional dynamical system is hidden in noisy multidimensional observations. Second, the method is used for building the data-driven model of the low frequency variability (LFV) in the quasigeostrophic model of the Earth’s midlatitude atmosphere—a high-dimensional chaotic system. It is demonstrated that the key regimes of the atmospheric LFV are reproduced correctly in data simulations by means of the obtained model.

1.
I. T.
Jolliffe
,
Springer Series in Statistics
, 2nd ed. (
Springer, New York
,
1986
).
2.
A.
Hannachi
,
I. T.
Jolliffe
, and
D. B.
Stephenson
, “
Empirical orthogonal functions and related techniques in atmospheric science: A review
,”
Int. J. Climatol.
27
,
1119
1152
(
2007
).
3.
R. R.
Coifman
and
S.
Lafon
, “
Diffusion maps
,”
Appl. Comput. Harmon. Anal.
21
,
5
30
(
2006
).
4.
A. J.
Gámez
,
C. S.
Zhou
,
A.
Timmermann
, and
J.
Kurths
, “
Nonlinear dimensionality reduction in climate data
,”
Nonlinear Process. Geophys.
11
,
393
398
(
2004
).
5.
M. A.
Kramer
, “
Nonlinear principal component analysis using autoassociative neural networks
,”
AIChE J.
37
,
233
243
(
1991
).
6.
D.
Dong
and
T.
McAvoy
, “
Nonlinear principal component analysis—Based on principal curves and neural networks
,”
Comput. Chem. Eng.
20
,
65
78
(
1996
).
7.
S.
Tan
and
M. L.
Mayrovouniotis
, “
Reducing data dimensionality through optimizing neural network inputs
,”
AIChE J.
41
,
1471
1480
(
1995
).
8.
D.
Mukhin
,
A.
Gavrilov
,
A.
Feigin
,
E.
Loskutov
, and
J.
Kurths
, “
Principal nonlinear dynamical modes of climate variability
,”
Sci. Rep.
5
,
15510
(
2015
).
9.
A.
Gavrilov
,
D.
Mukhin
,
E.
Loskutov
,
E.
Volodin
,
A.
Feigin
, and
J.
Kurths
, “
Method for reconstructing nonlinear modes with adaptive structure from multidimensional data
,”
Chaos
26
,
123101
(
2016
).
10.
A. F.
Seleznev
,
A. S.
Gavrilov
,
D. N.
Mukhin
,
E. M.
Loskutov
, and
A. M.
Feigin
, “
Simulation of complex systems using the observed data based on recurrent artificial neural networks
,”
Radiophys. Quantum Electron.
61
,
893
907
(
2019
).
11.
A. P.
Trischler
and
G. M.
D’Eleuterio
, “
Synthesis of recurrent neural networks for dynamical system simulation
,”
Neural Netw.
80
,
67
78
(
2016
); e-print arXiv:1512.05702.
12.
P. R.
Vlachas
,
W.
Byeon
, and
Z. Y.
Wan
, “
Data-driven forecasting of high-dimensional chaotic systems with long-short term memory networks
,”
Proc. R. Soc. A: Math. Phys. Eng. Sci.
474
,
20170844
(
2018
); e-print arXiv:1802.07486v4.
13.
R. J.
Williams
and
D.
Zipser
, “
A learning algorithm for continually running fully recurrent neural networks
,”
Neural Comput.
1
,
270
280
(
1989
).
14.
L. A.
Feldkamp
,
D. V.
Prokhorov
,
C. F.
Eagen
, and
F.
Yuan
, “Enhanced multi-stream Kalman filter training for recurrent networks,” in Nonlinear Modeling: Advanced Black-Box Techniques, edited by J. A. K. Suykens and J. Vandewalle (Springer US, Boston, MA, 1998), pp. 29–53.
15.
M.
Lukoševičius
and
H.
Jaeger
, “
Reservoir computing approaches to recurrent neural network training
,”
Comput. Sci. Rev.
3
,
127
149
(
2009
).
16.
J.
Pathak
,
B.
Hunt
,
M.
Girvan
,
Z.
Lu
, and
E.
Ott
, “
Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach
,”
Phys. Rev. Lett.
120
,
24102
(
2018
).
17.
Z.
Lu
,
B. R.
Hunt
, and
E.
Ott
, “
Attractor reconstruction by machine learning through generalized synchronization
,”
Chaos
28
,
061104
(
2018
); e-print arXiv:1805.03362.
18.
F.
Tsung
and
G. W.
Cottrell
, “
Phase-space learning
,”
Adv. Neural Inf. Process. Syst.
7
,
481
488
(
1995
).
19.
D. N.
Mukhin
,
A. M.
Feigin
,
E. M.
Loskutov
, and
Y. I.
Molkov
, “
Modified Bayesian approach for the reconstruction of dynamical systems from time series
,”
Phys. Rev. E Stat. Nonlin. Soft Matter Phys.
73
,
036211
(
2006
).
20.
S.
Hochreiter
, “
The vanishing gradient problem during learning recurrent neural nets and problem solutions
,”
Int. J. Uncertain. Fuzziness Knowl.-Based Syst.
06
,
107
116
(
1998
).
21.
A.
Gavrilov
,
E.
Loskutov
, and
D.
Mukhin
, “
Bayesian optimization of empirical model with state-dependent stochastic forcing
,”
Chaos Solitons Fractals
104
,
327
337
(
2017
).
22.
A.
Gavrilov
,
A.
Seleznev
,
D.
Mukhin
,
E.
Loskutov
,
A.
Feigin
, and
J.
Kurths
, “
Linear dynamical modes as new variables for data-driven ENSO forecast
,”
Clim. Dyn.
52
,
2199
2216
(
2019
).
23.
D.
Mukhin
,
A.
Gavrilov
,
E.
Loskutov
,
J.
Kurths
, and
A.
Feigin
, “
Bayesian data analysis for revealing causes of the middle pleistocene transition
,”
Sci. Rep.
9
(
1
),
7328
(
2019
).
24.
D. J. C.
MacKay
,
Information Theory, Inference & Learning Algorithms
(
Cambridge University Press
,
New York, NY
,
2002
).
25.
D. P.
Kingma
and
J.
Ba
, “
Adam: A method for stochastic optimization
,” e-print arXiv:1412.6980.
26.
R.
Fletcher
,
Practical Methods of Optimization
, 2nd ed. (
John Wiley & Sons
,
New York, NY
,
1987
).
27.
M.
Hénon
, “
A two-dimensional mapping with a strange attractor
,”
Commun. Math. Phys.
50
,
69
77
(
1976
).
28.
B.
Silverman
, Density Estimation for Statistics and Data Analysis, Chapman & Hall/CRC Monographs on Statistics & Applied Probability (Taylor & Francis, 1986).
29.
K. C.
Mo
and
M.
Ghil
, “
Statistics and dynamics of persistent anomalies
,”
J. Atmos. Sci.
44
,
877
902
(
1987
).
30.
D.
Kondrashov
,
K.
Ide
, and
M.
Ghil
, “
Weather regimes and preferred transition paths in a three-level quasigeostrophic model
,”
J. Atmos. Sci.
61
,
568
587
(
2004
).
31.
M. R.
Allen
and
L. A.
Smith
, “
Monte Carlo SSA: Detecting irregular oscillations in the presence of coloured noise
,”
J. Clim.
9
,
3373
3404
(
1996
).
32.
J.
Marshall
and
F.
Molteni
, “
Toward a dynamical understanding of planetary-scale flow regimes
,”
J. Atmos. Sci.
50
,
1792
1818
(
1993
).
33.
S.
Vannitsem
and
C.
Nicolis
, “
Lyapunov vectors and error growth patterns in a T21L3 quasigeostrophic model
,”
J. Atmos. Sci.
54
,
347
361
(
1997
).
34.
S.
Corti
,
A.
Giannini
,
S.
Tibaldi
, and
F.
Molteni
, “
Patterns of low-frequency variability in a three-level quasi-geostrophic model
,”
Clim. Dyn.
13
,
883
904
(
1997
).
35.
A.
Hannachi
and
W.
Iqbal
, “
On the nonlinearity of winter northern hemisphere atmospheric variability
,”
J. Atmos. Sci.
76
,
333
356
(
2019
).
36.
A.
Gritsun
, “
Statistical characteristics, circulation regimes and unstable periodic orbits of a barotropic atmospheric model
,”
Philos. Trans. R. Soc. A: Math. Phys. Eng. Sci.
371
,
20120336
(
2013
).
37.
D.
Reynolds
, “Gaussian mixture models,” in Encyclopedia of Biometrics, edited by S. Z. Li and A. Jain (Springer US, Boston, MA, 2009), pp. 659–663.
38.
T. C.
Mills
, “
Is the North Atlantic oscillation a random walk? A comment with further results
,”
Int. J. Climatology
24
,
377
383
(
2004
).
39.
T.
Woollings
,
A.
Hannachi
,
B.
Hoskins
, and
A.
Turner
, “
A regime view of the North Atlantic oscillation and its response to anthropogenic forcing
,”
J. Clim.
23
,
1291
1307
(
2010
).
40.
J. M.
Wallace
and
D. W. J.
Thompson
, “
The Arctic oscillation signature in the wintertime geopotential height and temperature fields
,”
Geophys. Res. Lett.
25
,
1297
1300
, (
1998
).
41.
Z.
Chen
,
B.
Gan
,
L.
Wu
, and
F.
Jia
, “
Pacific-North American teleconnection and North Pacific oscillation: Historical simulation and future projection in CMIP5 models
,”
Clim. Dyn.
50
,
4379
4403
(
2018
).
42.
D.
Kondrashov
,
S.
Kravtsov
, and
M.
Ghil
, “
Empirical mode reduction in a model of extratropical low-frequency variability
,”
J. Atmos. Sci.
63
,
1859
1877
(
2006
).
43.
M.
Ghil
,
M. R.
Allen
,
M. D.
Dettinger
,
K.
Ide
,
D.
Kondrashov
,
M. E.
Mann
,
A. W.
Robertson
,
A.
Saunders
,
Y.
Tian
,
F.
Varadi
, and
P.
Yiou
, “
Advanced spectral methods for climatic time series
,”
Rev. Geophys.
40
,
3–1
3–41
, (
2002
).
44.
S.
Kravtsov
,
D.
Kondrashov
, and
M.
Ghil
, “
Multilevel regression modeling of nonlinear processes: Derivation and applications to climatic variability
,”
J. Clim.
18
,
4404
4424
(
2005
).
45.
S.-J.
Wu
and
M. T.
Chu
, “
Markov chains with memory, tensor formulation, and the dynamics of power iteration
,”
Appl. Math. Comput.
303
,
226
239
(
2017
).
46.
G.
Cybenko
, “
Approximations by superpositions of sigmoidal functions
,”
Approx. Theory Appl.
9
,
17
28
(
1989
).
47.
B.
Karlik
and
A.
Vehbi
, “
Performance analysis of various activation functions in generalized MLP architectures of neural networks
,”
Int. J. Artif. Intell. Expert Syst.
1
,
111
122
(
2011
).
48.
X.
Glorot
and
Y.
Bengio
, “
Understanding the difficulty of training deep feedforward neural networks
,”
Proc. Mach. Learn Res.
9
,
249
256
(
2010
). e-print arXiv:1011.1669v3.
You do not currently have access to this content.