Recently developed reduced-order modeling techniques aim to approximate nonlinear dynamical systems on low-dimensional manifolds learned from data. This is an effective approach for modeling dynamics in a post-transient regime where the effects of initial conditions and other disturbances have decayed. However, modeling transient dynamics near an underlying manifold, as needed for real-time control and forecasting applications, is complicated by the effects of fast dynamics and nonnormal sensitivity mechanisms. To begin to address these issues, we introduce a parametric class of nonlinear projections described by constrained autoencoder neural networks in which both the manifold and the projection fibers are learned from data. Our architecture uses invertible activation functions and biorthogonal weight matrices to ensure that the encoder is a left inverse of the decoder. We also introduce new dynamics-aware cost functions that promote learning of oblique projection fibers that account for fast dynamics and nonnormality. To demonstrate these methods and the specific challenges they address, we provide a detailed case study of a three-state model of vortex shedding in the wake of a bluff body immersed in a fluid, which has a two-dimensional slow manifold that can be computed analytically. In anticipation of future applications to high-dimensional systems, we also propose several techniques for constructing computationally efficient reduced-order models using our proposed nonlinear projection framework. This includes a novel sparsity-promoting penalty for the encoder that avoids detrimental weight matrix shrinkage via computation on the Grassmann manifold.

1.
A.
Ghadami
and
B. I.
Epureanu
, “
Data-driven prediction in dynamical systems: Recent developments
,”
Philos. Trans. R. Soc. A
380
,
20210213
(
2022
).
2.
C. W.
Rowley
and
S. T. M.
Dawson
, “
Model reduction for flow analysis and control
,”
Annu. Rev. Fluid Mech.
49
,
387
417
(
2017
).
3.
P.
Benner
,
M.
Ohlberger
,
A.
Cohen
, and
K.
Willcox
,
Model Reduction and Approximation: Theory and Algorithms
(
SIAM
,
2017
).
4.
G.
Rozza
,
G.
Stabile
, and
F.
Ballarin
,
Advanced Reduced Order Methods and Applications in Computational Fluid Dynamics
(
SIAM
,
2022
).
5.
M.
Ohlberger
and
S.
Rave
, “Reduced basis methods: Success, limitations and future challenges,” in Proceedings of Algoritmy (Publishing House of Slovak University of Technology in Bratislava, 2016), pp. 1–12.
6.
K.
Lee
and
K. T.
Carlberg
, “
Model reduction of dynamical systems on nonlinear manifolds using deep convolutional autoencoders
,”
J. Comput. Phys.
404
,
108973
(
2020
).
7.
F.
Romor
,
G.
Stabile
, and
G.
Rozza
, “
Non-linear manifold reduced-order models with convolutional autoencoders and reduced over-collocation method
,”
J. Sci. Comput.
94
,
74
(
2023
).
8.
W.
Anderson
and
M.
Farazmand
, “
Evolution of nonlinear reduced-order solutions for PDEs with conserved quantities
,”
SIAM J. Sci. Comput.
44
,
A176
A197
(
2022
).
9.
R.
Geelen
,
S.
Wright
, and
K.
Willcox
, “
Operator inference for non-intrusive model reduction with quadratic manifolds
,”
Comput. Methods Appl. Mech. Eng.
403
,
115717
(
2023
).
10.
P.
Benner
,
P.
Goyal
,
J.
Heiland
, and
I.
Pontes
, “A quadratic decoder approach to nonintrusive reduced-order modeling of nonlinear dynamical systems,” arXiv:2209.15412 (2022).
11.
M.
Cenedese
,
J.
Axås
,
B.
Bäuerlein
,
K.
Avila
, and
G.
Haller
, “
Data-driven modeling and prediction of non-linearizable dynamics via spectral submanifolds
,”
Nat. Commun.
13
,
1
13
(
2022
).
12.
S.
Fresca
,
L.
Dede’
, and
A.
Manzoni
, “
A comprehensive deep learning-based approach to reduced order modeling of nonlinear time-dependent parametrized PDEs
,”
J. Sci. Comput.
87
,
61
(
2021
).
13.
P.
Conti
,
G.
Gobat
,
S.
Fresca
,
A.
Manzoni
, and
A.
Frangi
, “
Reduced order modeling of parametrized systems through autoencoders and SINDy approach: Continuation of periodic solutions
,”
Comput. Methods Appl. Mech. Eng.
411
,
116072
(
2023
).
14.
K.
Champion
,
B.
Lusch
,
J. N.
Kutz
, and
S. L.
Brunton
, “
Data-driven discovery of coordinates and governing equations
,”
Proc. Natl. Acad. Sci. U.S.A.-
116
,
22445
22451
(
2019
).
15.
N.
Fenichel
, “
Asymptotic stability with rate conditions
,”
Indiana Univ. Math. J.
23
,
1109
1137
(
1974
).
16.
N.
Fenichel
, “
Asymptotic stability with rate conditions, II
,”
Indiana Univ. Math. J.
26
,
81
93
(
1977
).
17.
C.
Kuehn
,
Multiple Time Scale Dynamics
(
Springer International
,
2015
).
18.
A. J.
Roberts
, “
Appropriate initial conditions for asymptotic descriptions of the long term evolution of dynamical systems
,”
ANZIAM J.
31
,
48
75
(
1989
).
19.
A. J.
Roberts
, “
Computer algebra derives correct initial conditions for low-dimensional dynamical models
,”
Comput. Phys. Commun.
126
,
187
206
(
2000
).
20.
L. N.
Trefethen
,
A. E.
Trefethen
,
S. C.
Reddy
, and
T. A.
Driscoll
, “
Hydrodynamic stability without eigenvalues
,”
Science
261
,
578
584
(
1993
).
21.
P. J.
Schmid
and
D. S.
Henningson
,
Stability and Transition in Shear Flows
(
Springer-Verlag
,
New York
,
2001
), Vol. 142.
22.
M.
Embree
and
L. N.
Trefethen
,
Spectra and Pseudospectra: The Behavior of Nonnormal Matrices and Operators
(
Princeton University Press
,
Princeton, NJ
,
2005
).
23.
A. C.
Antoulas
,
Approximation of Large-Scale Dynamical Systems
(
SIAM
,
2005
).
24.
S.
Gugercin
,
A. C.
Antoulas
, and
C.
Beattie
, “
H 2 model reduction for large-scale linear dynamical systems
,”
SIAM J. Matrix Anal. Appl.
30
,
609
638
(
2008
).
25.
C. W.
Rowley
, “
Model reduction for fluids using balanced proper orthogonal decomposition
,”
Int. J. Bifurcation Chaos
15
,
997
1013
(
2005
).
26.
B.
Moore
, “
Principal component analysis in linear systems: Controllability, observability, and model reduction
,”
IEEE Trans. Autom. Control
26
,
17
32
(
1981
).
27.
P.
Benner
,
P.
Goyal
, and
S.
Gugercin
, “
H 2-quasi-optimal model order reduction for quadratic-bilinear control systems
,”
SIAM J. Matrix Anal. Appl.
39
,
983
1032
(
2018
).
28.
P.
Benner
and
P.
Goyal
, “Balanced truncation model order reduction for quadratic-bilinear control systems,” arXiv:1705.00160 (2017).
29.
S.
Ahuja
and
C. W.
Rowley
, “
Feedback control of unstable steady states of flow past a flat plate using reduced-order estimators
,”
J. Fluid Mech.
645
,
447
478
(
2010
).
30.
A.
Barbagallo
,
D.
Sipp
, and
P. J.
Schmid
, “
Closed-loop control of an open cavity flow using reduced-order models
,”
J. Fluid Mech.
641
,
1
(
2009
).
31.
M.
Ilak
,
S.
Bagheri
,
L.
Brandt
,
C. W.
Rowley
, and
D. S.
Henningson
, “
Model reduction of the nonlinear complex Ginzburg–Landau equation
,”
SIAM J. Appl. Dyn. Syst.
9
,
1284
1302
(
2010
).
32.
S. J.
Illingworth
,
A. S.
Morgans
, and
C. W.
Rowley
, “
Feedback control of flow resonances using balanced reduced-order models
,”
J. Sound Vib.
330
,
1567
1581
(
2011
).
33.
S. E.
Otto
,
A.
Padovan
, and
C. W.
Rowley
, “
Optimizing oblique projections for nonlinear systems using trajectories
,”
SIAM J. Sci. Comput.
44
,
A1681
A1702
(
2022
).
34.
S. E.
Otto
,
A.
Padovan
, and
C. W.
Rowley
, “
Model reduction for nonlinear systems by balanced truncation of state and gradient covariance
,”
SIAM J. Sci. Comput.
45
,
A2325
A2355
(
2023
).
35.
J. M. A.
Scherpen
, “
Balancing for nonlinear systems
,”
Syst. Control Lett.
21
,
143
153
(
1993
).
36.
B.
Kramer
,
S.
Gugercin
, and
J.
Borggaard
, “Nonlinear balanced truncation: Part 1—Computing energy functions,” arXiv:2209.07645 (2022).
37.
B.
Kramer
,
S.
Gugercin
, and
J.
Borggaard
, “Nonlinear balanced truncation: Part 2—Model reduction on manifolds,” arXiv:2302.02036 (2023).
38.
I.
Goodfellow
,
Y.
Bengio
, and
A.
Courville
,
Deep Learning
(
MIT Press
,
2016
).
39.
M.
Lezcano-Casado
and
D.
Martınez-Rubio
, “Cheap orthogonal constraints in neural networks: A simple parametrization of the orthogonal and unitary group,” in International Conference on Machine Learning (PMLR, 2019), pp. 3794–3803.
40.
L.
Dinh
,
D.
Krueger
, and
Y.
Bengio
, “NICE: Non-linear independent components estimation,” arXiv:1410.8516 (2014).
41.
L.
Dinh
,
J.
Sohl-Dickstein
, and
S.
Bengio
, “Density estimation using real NVP,” arXiv:1605.08803 (2016).
42.
D. P.
Kingma
and
P.
Dhariwal
, “
Glow: Generative flow with invertible 1 × 1 convolutions
,” in
Advances in Neural Information Processing Systems 31 (NeurIPS 2018)
, edited by S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Curran Associates, 2019).
43.
L.
Ardizzone
,
J.
Kruse
,
S.
Wirkert
,
D.
Rahner
,
E. W.
Pellegrini
,
R. S.
Klessen
,
L.
Maier-Hein
,
C.
Rother
, and
U.
Köthe
, “Analyzing inverse problems with invertible neural networks,” arXiv:1808.04730 (2018).
44.
S.
Chaturantabut
and
D. C.
Sorensen
, “
Nonlinear model reduction via discrete empirical interpolation
,”
SIAM J. Sci. Comput.
32
,
2737
2764
(
2010
).
45.
P. W.
Michor
,
Topics in Differential Geometry
(
American Mathematical Society
,
2008
), Vol. 93.
46.
J. M.
Lee
,
Introduction to Smooth Manifolds
, 2nd ed. (
Springer
,
New York
,
2013
).
47.
K.
He
,
X.
Zhang
,
S.
Ren
, and
J.
Sun
, “Delving deep into rectifiers: Surpassing human-level performance on ImageNet classification,” in Proceedings of the IEEE International Conference on Computer Vision (IEEE, 2015), pp. 1026–1034.
48.
D. P.
Kingma
and
J. L.
Ba
, “Adam: A method for stochastic optimization,” arXiv:1412.6980 (2014).
49.
P.-A.
Absil
,
R.
Mahony
, and
R.
Sepulchre
,
Optimization Algorithms on Matrix Manifolds
(
Princeton University Press
,
2009
).
50.
P.-A.
Absil
and
J.
Malick
, “
Projection-like retractions on matrix manifolds
,”
SIAM J. Optim.
22
,
135
158
(
2012
).
51.
S. E.
Otto
, “Advances in data-driven modeling and sensing for high-dimensional nonlinear systems,” Ph.D. thesis (Princeton University, 2022).
52.
A.
Paszke
,
S.
Gross
,
F.
Massa
,
A.
Lerer
,
J.
Bradbury
,
G.
Chanan
,
T.
Killeen
,
Z.
Lin
,
N.
Gimelshein
,
L.
Antiga
,
A.
Desmaison
,
A.
Kopf
,
E.
Yang
,
Z.
DeVito
,
M.
Raison
,
A.
Tejani
,
S.
Chilamkurthy
,
B.
Steiner
,
L.
Fang
,
J.
Bai
, and
S.
Chintala
, “PyTorch: An imperative style, high-performance deep learning library,” in Advances in Neural Information Processing Systems, edited by H. Wallach, H. Larochelle, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and R. Garnett (Neural Information Processing Systems Foundation, Inc., 2019), Vol. 32.
53.
M.
Abadi
,
A.
Agarwal
,
P.
Barham
,
E.
Brevdo
,
Z.
Chen
,
C.
Citro
,
G. S.
Corrado
,
A.
Davis
,
J.
Dean
,
M.
Devin
,
S.
Ghemawat
,
I.
Goodfellow
,
A.
Harp
,
G.
Irving
,
M.
Isard
,
Y.
Jia
,
R.
Jozefowicz
,
L.
Kaiser
,
M.
Kudlur
,
J.
Levenberg
,
D.
Mané
,
R.
Monga
,
S.
Moore
,
D.
Murray
,
C.
Olah
,
M.
Schuster
,
J.
Shlens
,
B.
Steiner
,
I.
Sutskever
,
K.
Talwar
,
P.
Tucker
,
V.
Vanhoucke
,
V.
Vasudevan
,
F.
Viégas
,
O.
Vinyals
,
P.
Warden
,
M.
Wattenberg
,
M.
Wicke
,
Y.
Yu
, and
X.
Zheng
, “see tensorflow.org for “TensorFlow: Large-Scale Machine Learning on Heterogeneous Systems” (2015).
54.
V.
Guillemin
and
A.
Pollack
,
Differential Topology
(
AMS Chelsea Publishing
,
Providence, RI
,
1974
).
55.
P.
Holmes
,
J. L.
Lumley
,
G.
Berkooz
, and
C. W.
Rowley
,
Turbulence, Coherent Structures, Dynamical Systems and Symmetry
(
Cambridge University Press
,
2012
).
56.
J. R.
Singler
, “
Optimality of balanced proper orthogonal decomposition for data reconstruction
,”
Numer. Funct. Anal. Optim.
31
,
852
869
(
2010
).
57.
J. R.
Singler
, “
Optimality of balanced proper orthogonal decomposition for data reconstruction II: Further approximation results
,”
J. Math. Anal. Appl.
421
,
1006
1020
(
2015
).
58.
S. N.
Afriat
, “Orthogonal and oblique projectors and the characteristics of pairs of vector spaces,” in Mathematical Proceedings of the Cambridge Philosophical Society (Cambridge University Press, 1957), Vol. 53, pp. 800–816.
59.
R.
Björck
and
G. H.
Golub
, “
Numerical methods for computing angles between linear subspaces
,”
Math. Comput.
27
,
579
594
(
1973
).
60.
B. R.
Noack
,
K.
Afanasiev
,
M.
Morzyński
,
G.
Tadmor
, and
F.
Thiele
, “
A hierarchy of low-dimensional models for the transient and post-transient cylinder wake
,”
J. Fluid Mech.
497
,
335
363
(
2003
).
61.
D.
Hendrycks
and
K.
Gimpel
, “Gaussian error linear units (gelus),” arXiv:1606.08415 (2016).
62.
D.
Hendrycks
and
K.
Gimpel
, “Adjusting for dropout variance in batch normalization and weight initialization,” arXiv:1607.02488 (2016).
63.
A.
Bychkov
,
O.
Issan
,
B.
Kramer
, and
G.
Pogudin
, “Exact and optimal quadratization of nonlinear finite-dimensional non-autonomous dynamical systems,” arXiv:2303.10285 (2023).
64.
C.
Gu
, “
QLMOR: A projection-based nonlinear model order reduction approach using quadratic-linear representation of nonlinear systems
,”
IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst.
30
,
1307
1320
(
2011
).
65.
B.
Kramer
and
K.
Willcox
, “Balanced truncation model reduction for lifted nonlinear systems,” in Realization and Model Reduction of Dynamical Systems: A Festschrift in Honor of the 70th Birthday of Thanos Antoulas (Springer, 2022), pp. 157–174.
66.
Z.
Drmac
and
S.
Gugercin
, “
A new selection operator for the discrete empirical interpolation method—Improved a priori error bound and extensions
,”
SIAM J. Sci. Comput.
38
,
A631
A648
(
2016
).
67.
R.
Tibshirani
, “
Regression shrinkage and selection via the Lasso
,”
J. R. Stat. Soc. Ser. B
58
,
267
288
(
1996
).
68.
M.
Yuan
and
Y.
Lin
, “
Model selection and estimation in regression with grouped variables
,”
J. R. Stat. Soc. Ser. B
68
,
49
67
(
2006
).
69.
S.
Scardapane
,
D.
Comminiello
,
A.
Hussain
, and
A.
Uncini
, “
Group sparse regularization for deep neural networks
,”
Neurocomputing
241
,
81
89
(
2017
).
70.
B. N. G.
Koneru
and
V.
Vasudevan
, “
Sparse artificial neural networks using a novel smoothed LASSO penalization
,”
IEEE Trans. Circuits Syst. II: Express Briefs
66
,
848
852
(
2019
).
71.
J.
Wang
,
C.
Xu
,
X.
Yang
, and
J. M.
Zurada
, “
A novel pruning algorithm for smoothing feedforward neural networks based on group lasso method
,”
IEEE Trans. Neural Networks Learn. Syst.
29
,
2012
2024
(
2017
).
72.
T.
Bendokat
,
R.
Zimmermann
, and
P.-A.
Absil
, “A Grassmann manifold handbook: Basic geometry and computational aspects,” arXiv:2011.13699 (2020).
73.
P.-A.
Absil
,
R.
Mahony
, and
R.
Sepulchre
, “
Riemannian geometry of Grassmann manifolds with a view on algorithmic computation
,”
Acta Appl. Math.
80
,
199
220
(
2004
).
74.
Y.-C.
Wong
, “
Differential geometry of Grassmann manifolds
,”
Proc. Natl. Acad. Sci.
57
,
589
594
(
1967
).
75.
M.
Raissi
,
P.
Perdikaris
, and
G. E.
Karniadakis
, “
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
,”
J. Comput. Phys.
378
,
686
707
(
2019
).
76.
H. K.
Khalil
,
Nonlinear Systems
, 3rd ed. (
Prentice Hall
,
2002
).
77.
W. G.
Kelly
and
A. C.
Peterson
,
The Theory of Differential Equations, Classical and Qualitative
(
Pearson Prentice Hall
,
2004
).
78.
R. A.
Horn
and
C. R.
Johnson
,
Topics in Matrix Analysis
(
Cambridge University Press
,
1991
).
79.
J. R.
Magnus
and
H.
Neudecker
,
Matrix Differential Calculus with Applications in Statistics and Econometrics
, 3rd ed. (
John Wiley and Sons
,
2007
).
80.
C. D.
Meyer
,
Matrix Analysis and Applied Linear Algebra
(
SIAM
,
2000
), Vol. 71.
81.
F.
Clarke
,
Functional Analysis, Calculus of Variations, and Optimal Control
(
Springer
,
2013
).
82.
A.
Edelman
,
T. A.
Arias
, and
S. T.
Smith
, “
The geometry of algorithms with orthogonality constraints
,”
SIAM J. Matrix Anal. Appl.
20
,
303
353
(
1998
).
83.
G.
Macchio
,
C.
Rowley
, and
S.
Otto
(2023). “Learning nonlinear projections for reduced-order modeling of dynamical systems using constrained autoencoders,” version 0.0.1,
Zenodo
.
You do not currently have access to this content.