Time-dependent flow fields are typically generated by a computational fluid dynamics method, which is an extremely time-consuming process. However, the latent relationship between the flow fields is governed by the Navier–Stokes equations and can be described by an operator. We therefore train a deep operator network (DeepONet) to learn the temporal evolution between flow snapshots. Once properly trained, given a few consecutive snapshots as input, the network has a great potential to generate the next snapshot accurately and quickly. Using the output as a new input, the network iterates the process, generating a series of successive snapshots with little wall time. Specifically, we consider two-dimensional flow around a circular cylinder at Reynolds number 1000 and prepare a set of high-fidelity data using a high-order spectral/hp element method as ground truth. Although the flow fields are periodic, there are many small-scale features in the wake flow that are difficult to generate accurately. Furthermore, any discrepancy between the prediction and the ground truth for the first snapshots can easily accumulate during the iterative process, which eventually amplifies the overall deviations. Therefore, we propose two alternative techniques to improve the training of DeepONet. The first one enhances the feature extraction of the network by harnessing the “multi-head non-local block.” The second one refines the network parameters by leveraging the local smooth optimization technique. Both techniques prove to be highly effective in reducing the cumulative errors, and our results outperform those of the dynamic mode decomposition method.

1.
X.
Zhang
,
F.
Xie
,
T.
Ji
,
Z.
Zhu
, and
Y.
Zheng
, “
Multi-fidelity deep neural network surrogate model for aerodynamic shape optimization
,”
Comput. Methods Appl. Mech. Eng.
373
,
113485
(
2021
).
2.
J.
Viquerat
,
J.
Rabault
,
A.
Kuhnle
,
H.
Ghraieb
,
A.
Larcher
, and
E.
Hachem
, “
Direct shape optimization through deep reinforcement learning
,”
J. Comput. Phys.
428
,
110080
(
2021
).
3.
J.
Rabault
,
M.
Kuchta
,
A.
Jensen
,
U.
Réglade
, and
N.
Cerardi
, “
Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
,”
J. Fluid Mech.
865
,
281
302
(
2019
).
4.
C.
Zheng
,
T.
Ji
,
F.
Xie
,
X.
Zhang
,
H.
Zheng
, and
Y.
Zheng
, “
From active learning to deep reinforcement learning: Intelligent active flow control in suppressing vortex-induced vibration
,”
Phys. Fluids
33
,
063607
(
2021
).
5.
Z.
Wang
,
D.
Fan
,
X.
Jiang
,
M. S.
Triantafyllou
, and
G. E.
Karniadakis
, “
Deep reinforcement transfer learning of active control for bluff body flows at high Reynolds number
,”
J. Fluid Mech.
973
,
A32
(
2023
).
6.
C.
Mo
,
Q.
Fu
, and
X.
Bian
, “
Chemotaxis of an elastic flagellated microrobot
,”
Phys. Rev. E
108
,
044408
(
2023
).
7.
Z.
Ma
and
W.
Pan
, “
Data-driven nonintrusive reduced order modeling for dynamical systems with moving boundaries using Gaussian process regression
,”
Comput. Methods Appl. Mech. Eng.
373
,
113495
(
2021
).
8.
L.
Pegolotti
,
M. R.
Pfaller
,
A. L.
Marsden
, and
S.
Deparis
, “
Model order reduction of flow based on a modular geometrical approximation of blood vessels
,”
Comput. Methods Appl. Mech. Eng.
380
,
113762
(
2021
).
9.
L.
Sirovich
, “
Turbulence and the dynamics of coherent structures. I. Coherent structures
,”
Q. Appl. Math.
45
,
561
571
(
1987
).
10.
P. J.
Schmid
, “
Dynamic mode decomposition of numerical and experimental data
,”
J. Fluid Mech.
656
,
5
28
(
2010
).
11.
J. N.
Kutz
, “
Deep learning in fluid dynamics
,”
J. Fluid Mech.
814
,
1
4
(
2017
).
12.
T.
Mikolov
,
K.
Chen
,
G.
Corrado
, and
J.
Dean
, “
Efficient estimation of word representations in vector space
,” arXiv:1301.3781 (
2013
).
13.
J.
Pennington
,
R.
Socher
, and
C. D.
Manning
, “
Glove: Global vectors for word representation
,” in
Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing
(
Association for Computational Linguistics
,
2014
), pp.
1532
1543
.
14.
K.
He
,
X.
Zhang
,
S.
Ren
, and
J.
Sun
, “
Deep residual learning for image recognition
,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
(
IEEE
,
2016
), pp.
770
778
.
15.
K.
Simonyan
and
A.
Zisserman
, “
Very deep convolutional networks for large-scale image recognition
,” arXiv:1409.1556 (
2014
).
16.
M.
Raissi
,
P.
Perdikaris
, and
G. E.
Karniadakis
, “
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
,”
J. Comput. Phys.
378
,
686
707
(
2019
).
17.
Y.
Zhu
,
W.
Kong
,
J.
Deng
, and
X.
Bian
, “
Physics-informed neural networks for incompressible flows with moving boundaries
,”
Phys. Fluids
36
,
013617
(
2024
).
18.
J.
Song
,
W.
Cao
,
F.
Liao
, and
W.
Zhang
, “
VW-PINNs: A volume weighting method for PDE residuals in physics-informed neural networks
,” arXiv:2401.06196 (
2024
).
19.
W. E. B.
Yu
, “
The Deep Ritz Method: A deep learning-based numerical algorithm for solving variational problems
,”
Commun. Math. Stat.
6
,
1
12
(
2018
).
20.
J.
Sirignano
and
K.
Spiliopoulos
, “
DGM: A deep learning algorithm for solving partial differential equations
,”
J. Comput. Phys.
375
,
1339
1364
(
2018
).
21.
P.
Karnakov
,
S.
Litvinov
, and
P.
Koumoutsakos
, “
Optimizing a discrete loss (ODIL) to solve forward and inverse problems for partial differential equations using machine learning tools
,” arXiv:2205.04611 (
2022
).
22.
X.
Jin
,
S.
Cai
,
H.
Li
, and
G. E.
Karniadakis
, “
NSFnets (Navier–Stokes flow nets): Physics-informed neural networks for the incompressible Navier–Stokes equations
,”
J. Comput. Phys.
426
,
109951
(
2021
).
23.
Z.
Mao
,
A. D.
Jagtap
, and
G. E.
Karniadakis
, “
Physics-informed neural networks for high-speed flows
,”
Comput. Methods Appl. Mech. Eng.
360
,
112789
(
2020
).
24.
S.
Cai
,
Z.
Wang
,
F.
Fuest
,
Y. J.
Jeon
,
C.
Gray
, and
G. E.
Karniadakis
, “
Flow over an espresso cup: Inferring 3D velocity and pressure fields from tomographic background oriented schlieren via physics-informed neural networks
,”
J. Fluid Mech.
915
,
A102
(
2021
).
25.
S. L.
Brunton
and
J. N.
Kutz
,
Data-Driven Science and Engineering
, 2nd ed. (
Cambridge University Press
,
2018
).
26.
S.
Hochreiter
and
J.
Schmidhuber
, “
Long short-term memory
,”
Neural Comput.
9
,
1735
1780
(
1997
).
27.
R.
Han
,
Y.
Wang
,
Y.
Zhang
, and
G.
Chen
, “
A novel spatial-temporal prediction method for unsteady wake flows based on hybrid deep neural network
,”
Phys. Fluids
31
,
127101
(
2019
).
28.
T.
Nakamura
,
K.
Fukami
,
K.
Hasegawa
,
Y.
Nabae
, and
K.
Fukagata
, “
Convolutional neural network and long short-term memory based reduced order surrogate for minimal turbulent channel flow
,”
Phys. Fluids
33
,
025116
(
2021
).
29.
P.
Wu
,
S.
Gong
,
K.
Pan
,
F.
Qiu
,
W.
Feng
, and
C.
Pain
, “
Reduced order model using convolutional auto-encoder with self-attention
,”
Phys. Fluids
33
,
077107
(
2021
).
30.
X.
Zhang
,
T.
Ji
,
F.
Xie
,
H.
Zheng
, and
Y.
Zheng
, “
Unsteady flow prediction from sparse measurements by compressed sensing reduced order modeling
,”
Comput. Methods Appl. Mech. Eng.
393
,
114800
(
2022
).
31.
L.
Lu
,
P.
Jin
,
G.
Pang
,
Z.
Zhang
, and
G. E.
Karniadakis
, “
Learning nonlinear operators via deeponet based on the universal approximation theorem of operators
,”
Nat. Mach. Intell.
3
,
218
229
(
2021
).
32.
Z.
Li
,
N.
Kovachki
,
K.
Azizzadenesheli
,
B.
Liu
,
K.
Bhattacharya
,
A.
Stuart
, and
A.
Anandkumar
, “
Fourier neural operator for parametric partial differential equations
,” arXiv:2010.08895 (
2020
).
33.
Z.
Li
,
K.
Meidani
, and
A. B.
Farimani
, “
Transformer for partial differential equations' operator learning
,” arXiv:2205.13671 (
2022
).
34.
B.
Xu
,
Y.
Zhou
, and
X.
Bian
, “
Self-supervised learning based on transformer for flow reconstruction and prediction
,”
Phys. Fluids
36
,
023607
(
2024
).
35.
T.
Chen
and
H.
Chen
, “
Universal approximation to nonlinear operators by neural networks with arbitrary activation functions and its application to dynamical systems
,”
IEEE Trans. Neural Network
6
,
911
917
(
1995
).
36.
C.
Lin
,
M.
Maxey
,
Z.
Li
, and
G. E.
Karniadakis
, “
A seamless multiscale operator neural network for inferring bubble dynamics
,”
J. Fluid Mech.
929
,
A18
(
2021
).
37.
V.
Oommen
,
K.
Shukla
,
S.
Goswami
,
R.
Dingreville
, and
G. E.
Karniadakis
, “
Learning two-phase microstructure evolution using neural operators and autoencoder architectures
,”
npj Comput. Mater.
8
,
190
(
2022
).
38.
Z.
Mao
,
L.
Lu
,
O.
Marxen
,
T. A.
Zaki
, and
G. E.
Karniadakis
, “
DeepM&Mnet for hypersonics: Predicting the coupled flow and finite-rate chemistry behind a normal shock using neural-network approximation of operators
,”
J. Comput. Phys.
447
,
110698
(
2021
).
39.
S.
Wang
and
P.
Perdikaris
, “
Long-time integration of parametric evolution equations with physics-informed deeponets
,”
J. Comput. Phys.
475
,
111855
(
2023
).
40.
K.
Michałowska
,
S.
Goswami
,
G. E.
Karniadakis
, and
S.
Riemer-Sørensen
, “
Neural operator learning for long-time integration in dynamical systems with recurrent neural networks
,” arXiv:2303.02243 (
2023
).
41.
G.
Karniadakis
and
S.
Sherwin
,
Spectral/hp Element Methods for Computational Fluid Dynamics
, 2nd ed. (
Oxford University Press
,
Oxford, UK
,
2005
).
42.
L.
Lu
,
X.
Meng
,
S.
Cai
,
Z.
Mao
,
S.
Goswami
,
Z.
Zhang
, and
G. E.
Karniadakis
, “
A comprehensive and fair comparison of two neural operators (with practical extensions) based on fair data
,”
Comput. Methods Appl. Mech. Eng.
393
,
114778
(
2022
).
43.
C.
Szegedy
,
W.
Liu
,
Y.
Jia
,
P.
Sermanet
,
S.
Reed
,
D.
Anguelov
,
D.
Erhan
,
V.
Vanhoucke
, and
A.
Rabinovich
, “
Going deeper with convolutions
,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
(
IEEE
,
2015
), pp.
1
9
.
44.
N.
Wandel
,
M.
Weinmann
,
M.
Neidlin
, and
R.
Klein
, “
Spline-PINN: Approaching PDES without data using fast, physics-informed Hermite-spline CNNS
,” in
Proceedings of the AAAI Conference on Artificial Intelligence
(
AAAI
,
2022
), Vol.
36
, pp.
8529
8538
.
45.
X.
Wang
,
R.
Girshick
,
A.
Gupta
, and
K.
He
, “
Non-local neural networks
,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
(
IEEE
,
2018
), pp.
7794
7803
.
46.
A.
Buades
,
B.
Coll
, and
J.-M.
Morel
, “
A non-local algorithm for image denoising
,” in
2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition
(
IEEE
,
2005
), Vol.
2
, pp.
60
65
.
47.
A.
Vaswani
,
N.
Shazeer
,
N.
Parmar
,
J.
Uszkoreit
,
L.
Jones
,
A. N.
Gomez
,
Ł.
Kaiser
, and
I.
Polosukhin
, “
Attention is all you need
,” in
Advances in Neural Information Processing Systems
(
Curran Associates, Inc.
,
2017
), Vol.
36
.
48.
N.
Park
and
S.
Kim
, “
How do vision transformers work?
,” arXiv:2202.06709 (
2022
).
49.
A.
Arnab
,
M.
Dehghani
,
G.
Heigold
,
C.
Sun
,
M.
Lučić
, and
C.
Schmid
, “
Vivit: A video vision transformer
,” in
Proceedings of the IEEE/CVF International Conference on Computer Vision
(
IEEE
,
2021
), pp.
6836
6846
.
50.
N. S.
Keskar
,
D.
Mudigere
,
J.
Nocedal
,
M.
Smelyanskiy
, and
P. T. P.
Tang
, “
On large-batch training for deep learning: Generalization gap and sharp minima
,” arXiv:1609.04836 (
2016
).
51.
G. K.
Dziugaite
and
D. M.
Roy
, “
Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data
,” arXiv:1703.11008 (
2017
).
52.
Y.
Jiang
,
B.
Neyshabur
,
H.
Mobahi
,
D.
Krishnan
, and
S.
Bengio
, “
Fantastic generalization measures and where to find them
,” arXiv:1912.02178 (
2019
).
53.
P.
Foret
,
A.
Kleiner
,
H.
Mobahi
, and
B.
Neyshabur
, “
Sharpness-aware minimization for efficiently improving generalization
,” arXiv:2010.01412 (
2020
).
54.
D. P.
Kingma
and
J.
Ba
, “
Adam: A method for stochastic optimization
,” arXiv:1412.6980 (
2014
).
55.
J.
Zhuang
,
B.
Gong
,
L.
Yuan
,
Y.
Cui
,
H.
Adam
,
N.
Dvornek
,
S.
Tatikonda
,
J.
Duncan
, and
T.
Liu
, “
Surrogate gap minimization improves sharpness-aware training
,” arXiv:2203.08065 (
2022
).
56.
Z.-Q. J.
Xu
,
Y.
Zhang
,
T.
Luo
,
Y.
Xiao
, and
Z.
Ma
, “
Frequency principle: Fourier analysis sheds light on deep neural networks
,” arXiv:1901.06523 (
2019
).
57.
R. D.
Henderson
, “
Details of the drag curve near the onset of vortex shedding
,”
Phys. Fluids
7
,
2102
2104
(
1995
).
You do not currently have access to this content.