The development of a reliable subgrid-scale (SGS) model for large-eddy simulation (LES) is of great importance for many scientific and engineering applications. Recently, deep learning approaches have been tested for this purpose using high-fidelity data such as direct numerical simulation (DNS) in a supervised learning process. However, such data are generally not available in practice. Deep reinforcement learning (DRL) using only limited target statistics can be an alternative algorithm in which the training and testing of the model are conducted in the same LES environment. The DRL of turbulence modeling remains challenging owing to its chaotic nature, high dimensionality of the action space, and large computational cost. In this study, we propose a physics-constrained DRL framework that can develop a deep neural network-based SGS model for LES of turbulent channel flow. The DRL models that produce the SGS stress were trained based on the local gradient of the filtered velocities. The developed SGS model automatically satisfies the reflectional invariance and wall boundary conditions without an extra training process so that DRL can quickly find the optimal policy. Furthermore, direct accumulation of reward, spatially and temporally correlated exploration, and the pre-training process are applied for efficient and effective learning. In various environments, our DRL could discover SGS models that produce the viscous and Reynolds stress statistics perfectly consistent with the filtered DNS. By comparing various statistics obtained by the trained models and conventional SGS models, we present a possible interpretation of better performance of the DRL model.

1.
P.
Moin
and
K.
Mahesh
, “
Direct numerical simulation: A tool in turbulence research
,”
Annu. Rev. Fluid Mech.
30
,
539
578
(
1998
).
2.
M.
Lesieur
and
O.
Metais
, “
New trends in large-eddy simulations of turbulence
,”
Annu. Rev. Fluid Mech.
28
,
45
82
(
1996
).
3.
C.
Meneveau
and
J.
Katz
, “
Scale-invariance and turbulence models for large-eddy simulation
,”
Annu. Rev. Fluid Mech.
32
,
1
32
(
2000
).
4.
U.
Piomelli
and
E.
Balaras
, “
Wall-layer models for large-eddy simulations
,”
Annu. Rev. Fluid Mech.
34
,
349
374
(
2002
).
5.
S. B.
Pope
, “
Ten questions concerning the large-eddy simulation of turbulent flows
,”
New J. Phys.
6
,
35
35
(
2004
).
6.
P. R.
Spalart
, “
Detached-eddy simulation
,”
Annu. Rev. Fluid Mech.
41
,
181
202
(
2009
).
7.
S. T.
Bose
and
G. I.
Park
, “
Wall-modeled large-eddy simulation for complex turbulent flows
,”
Annu. Rev. Fluid Mech.
50
,
535
561
(
2018
).
8.
R. D.
Moser
,
S. W.
Haering
, and
G. R.
Yalla
, “
Statistical properties of subgrid-scale turbulence models
,”
Annu. Rev. Fluid Mech.
53
,
255
286
(
2021
).
9.
J.
Smagorinsky
, “
General circulation experiments with primitive equations
,”
Mon. Weather Rev.
91
,
99
164
(
1963
).
10.
M.
Germano
,
U.
Piomelli
,
P.
Moin
, and
W. H.
Cabot
, “
A dynamic subgrid-scale eddy viscosity model
,”
Phys. Fluids A
3
,
1760
1765
(
1991
).
11.
D. K.
Lilly
, “
A proposed modification of the Germano subgrid-scale closure method
,”
Phys. Fluids A
4
,
633
635
(
1992
).
12.
A. W.
Vreman
, “
An eddy-viscosity subgrid-scale model for turbulent shear flow: Algebraic theory and applications
,”
Phys. Fluids
16
,
3670
3681
(
2004
).
13.
N.
Park
,
S.
Lee
,
J.
Lee
, and
H.
Choi
, “
A dynamic subgrid-scale eddy viscosity model with a global model coefficient
,”
Phys. Fluids
18
,
125109
(
2006
).
14.
D.
You
and
P.
Moin
, “
A dynamic global-coefficient subgrid-scale eddy-viscosity model for large-eddy simulation in complex geometries
,”
Phys. Fluids
19
,
065110
(
2007
).
15.
J.
Bardina
,
J.
Ferziger
, and
W.
Reynolds
, “
Improved subgrid-scale models for large-eddy simulation
,” in
13th Fluid and Plasma Dynamics Conference
(
American Institute of Aeronautics and Astronautics
,
1980
).
16.
S.
Liu
,
C.
Meneveau
, and
J.
Katz
, “
On the properties of similarity subgrid-scale models as deduced from measurements in a turbulent jet
,”
J. Fluid Mech.
275
,
83
119
(
1994
).
17.
A.
Leonard
, “
Energy Cascade in large-eddy simulations of turbulent fluid flows
,” in
Advances in Geophysics
(
Elsevier
,
1974
), Vol.
18
, pp.
237
248
.
18.
R. A.
Clark
,
J. H.
F
, and
W. C.
Reynolds
, “
Evaluation of subgrid-scale models using an accurately simulated turbulent flow
,”
J. Fluid Mech.
91
,
1
16
(
1979
).
19.
Y.
Zang
,
R. L.
Street
, and
J. R.
Koseff
, “
A dynamic mixed subgrid-scale model and its application to turbulent recirculating flows
,”
Phys. Fluids A
5
,
3186
3196
(
1993
).
20.
M. V.
Salvetti
and
S.
Banerjee
, “
A priori tests of a new dynamic subgrid-scale model for finite-difference large-eddy simulations
,”
Phys. Fluids
7
,
2831
2847
(
1995
).
21.
K.
Horiuti
, “
A new dynamic two-parameter mixed model for large-eddy simulation
,”
Phys. Fluids
9
,
3443
3464
(
1997
).
22.
J. N.
Kutz
, “
Deep learning in fluid dynamics
,”
J. Fluid Mech.
814
,
1
4
(
2017
).
23.
M. P.
Brenner
,
J. D.
Eldredge
, and
J. B.
Freund
, “
Perspective on machine learning for advancing fluid mechanics
,”
Phys. Rev. Fluids
4
,
100501
(
2019
).
24.
K.
Duraisamy
,
G.
Iaccarino
, and
H.
Xiao
, “
Turbulence modeling in the age of data
,”
Annu. Rev. Fluid Mech.
51
,
357
377
(
2019
).
25.
S. L.
Brunton
,
B. R.
Noack
, and
P.
Koumoutsakos
, “
Machine learning for fluid mechanics
,”
Annu. Rev. Fluid Mech.
52
,
477
508
(
2020
).
26.
K.
Duraisamy
, “
Perspectives on machine learning-augmented Reynolds-averaged and large eddy simulation models of turbulence
,”
Phys. Rev. Fluids
6
,
050504
(
2021
).
27.
R.
Maulik
and
O.
San
, “
A neural network approach for the blind deconvolution of turbulent flows
,”
J. Fluid Mech.
831
,
151
181
(
2017
).
28.
R.
Maulik
,
O.
San
,
A.
Rasheed
, and
P.
Vedula
, “
Subgrid modelling for two-dimensional turbulence using neural networks
,”
J. Fluid Mech.
858
,
122
144
(
2019
).
29.
Y.
Guan
,
A.
Chattopadhyay
,
A.
Subel
, and
P.
Hassanzadeh
, “
Stable a posteriori les of 2D turbulence using convolutional neural networks: Backscattering analysis and generalization to higher re via transfer learning
,” arXiv:2102.11400v1 (
2021
).
30.
D.
Kochkov
,
J. A.
Smith
,
A.
Alieva
,
Q.
Wang
,
M. P.
Brenner
, and
S.
Hoyer
, “
Machine learning–accelerated computational fluid dynamics
,”
Proc. Natl. Acad. Sci. U. S. A.
118
,
e2101784118
(
2021
).
31.
A.
Vollant
,
G.
Balarac
, and
C.
Corre
, “
Subgrid-scale scalar flux modelling based on optimal estimation theory and machine-learning procedures
,”
J. Turbul.
18
,
854
878
(
2017
).
32.
Z.
Zhou
,
G.
He
,
S.
Wang
, and
G.
Jin
, “
Subgrid-scale model for large-eddy simulation of isotropic turbulent flows using an artificial neural network
,”
Comput. Fluids
195
,
104319
(
2019
).
33.
C.
Xie
,
J.
Wang
, and
W.
E
, “
Modeling subgrid-scale forces by spatial artificial neural networks in large eddy simulation of turbulence
,”
Phys. Rev. Fluids
5
,
054606
(
2020
).
34.
G. D.
Portwood
,
B. T.
Nadiga
,
J. A.
Saenz
, and
D.
Livescu
, “
Interpreting neural network models of residual scalar flux
,”
J. Fluid Mech.
907
,
A23
(
2021
).
35.
H.
Frezat
,
G.
Balarac
,
J. L.
Sommer
,
R.
Fablet
, and
R.
Lguensat
, “
Physical invariance in neural networks for subgrid-scale scalar flux modeling
,”
Phys. Rev. Fluids
6
,
024607
(
2021
).
36.
A.
Prakash
,
K. E.
Jansen
, and
J. A.
Evans
, “
Invariant data-driven subgrid stress modeling in the strain-rate eigenframe for large eddy simulation
,” arXiv:2106.13410v1 (
2021
).
37.
Z.
Wang
,
K.
Luo
,
D.
Li
,
J.
Tan
, and
J.
Fan
, “
Investigations of data-driven closure for subgrid-scale stress in large-eddy simulation
,”
Phys. Fluids
30
,
125101
(
2018
).
38.
A.
Beck
,
D.
Flad
, and
C.-D.
Munz
, “
Deep neural networks for data-driven LES closure models
,”
J. Comput. Phys.
398
,
108910
(
2019
).
39.
C.
Xie
,
K.
Li
,
C.
Ma
, and
J.
Wang
, “
Modeling subgrid-scale force and divergence of heat flux of compressible isotropic turbulence by artificial neural network
,”
Phys. Rev. Fluids
4
,
104605
(
2019
).
40.
C.
Xie
,
J.
Wang
,
K.
Li
, and
C.
Ma
, “
Artificial neural network approach to large-eddy simulation of compressible isotropic turbulence
,”
Phys. Rev. E
99
,
053113
(
2019
).
41.
C.
Wollblad
and
L.
Davidson
, “
POD based reconstruction of subgrid stresses for wall bounded flows using neural networks
,”
Flow Turbul. Combust.
81
,
77
96
(
2008
).
42.
M.
Gamahara
and
Y.
Hattori
, “
Searching for turbulence models by artificial neural network
,”
Phys. Rev. Fluids
2
,
054604
(
2017
).
43.
J.
Park
and
H.
Choi
, “
Toward neural-network-based large eddy simulation: Application to turbulent channel flow
,”
J. Fluid Mech.
914
,
A16
(
2021
).
44.
B.
Font
,
G. D.
Weymouth
,
V.-T.
Nguyen
, and
O. R.
Tutty
, “
Deep learning of the spanwise-averaged Navier–Stokes equations
,”
J. Comput. Phys.
434
,
110199
(
2021
).
45.
S.
Colabrese
,
K.
Gustavsson
,
A.
Celani
, and
L.
Biferale
, “
Flow navigation by smart microswimmers via reinforcement learning
,”
Phys. Rev. Lett.
118
,
158004
(
2017
).
46.
S.
Verma
,
G.
Novati
, and
P.
Koumoutsakos
, “
Efficient collective swimming by harnessing vortices through deep reinforcement learning
,”
Proc. Natl. Acad. Sci. U. S. A.
115
,
5849
5854
(
2018
).
47.
J.
Rabault
,
M.
Kuchta
,
A.
Jensen
,
U.
Réglade
, and
N.
Cerardi
, “
Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
,”
J. Fluid Mech.
865
,
281
302
(
2019
).
48.
J.
Rabault
and
A.
Kuhnle
, “
Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach
,”
Phys. Fluids
31
,
094105
(
2019
).
49.
H.
Tang
,
J.
Rabault
,
A.
Kuhnle
,
Y.
Wang
, and
T.
Wang
, “
Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning
,”
Phys. Fluids
32
,
053605
(
2020
).
50.
D.
Fan
,
L.
Yang
,
Z.
Wang
,
M. S.
Triantafyllou
, and
G.
Karniadakis
, “
Reinforcement learning for bluff body active flow control in experiments and simulations
,”
Proc. Natl. Acad. Sci. U. S. A.
117
,
26091
26098
(
2020
).
51.
R.
Paris
,
S.
Beneddine
, and
J.
Dandois
, “
Robust flow control and optimal sensor placement using deep reinforcement learning
,”
J. Fluid Mech.
913
,
A25
(
2021
).
52.
G.
Beintema
,
A.
Corbetta
,
L.
Biferale
, and
F.
Toschi
, “
Controlling Rayleigh–Bénard convection via reinforcement learning
,”
J. Turbul.
21
,
585
605
(
2020
).
53.
E.
Hachem
,
H.
Ghraieb
,
J.
Viquerat
,
A.
Larcher
, and
P.
Meliga
, “
Deep reinforcement learning for the control of conjugate heat transfer
,”
J. Comput. Phys.
436
,
110317
(
2021
).
54.
M. A.
Bucci
,
O.
Semeraro
,
A.
Allauzen
,
G.
Wisniewski
,
L.
Cordier
, and
L.
Mathelin
, “
Control of chaotic systems by deep reinforcement learning
,”
Proc. R. Soc. A
475
,
20190351
(
2019
).
55.
K.
Zeng
and
M. D.
Graham
, “
Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics
,”
Phys. Rev. E
104
,
014210
(
2021
).
56.
J.
Viquerat
,
J.
Rabault
,
A.
Kuhnle
,
H.
Ghraieb
,
A.
Larcher
, and
E.
Hachem
, “
Direct shape optimization through deep reinforcement learning
,”
J. Comput. Phys.
428
,
110080
(
2021
).
57.
J.
Rabault
,
F.
Ren
,
W.
Zhang
,
H.
Tang
, and
H.
Xu
, “
Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization
,”
J. Hydrodyn.
32
,
234
246
(
2020
).
58.
F.
Ren
,
H.
Hu
, and
H.
Tang
, “
Active flow control using machine learning: A brief review
,”
J. Hydrodyn.
32
,
247
253
(
2020
).
59.
P.
Garnier
,
J.
Viquerat
,
J.
Rabault
,
A.
Larcher
,
A.
Kuhnle
, and
E.
Hachem
, “
A review on deep reinforcement learning for fluid mechanics
,”
Comput. Fluids
225
,
104973
(
2021
).
60.
G.
Novati
,
H.
de Laroussilhe
, and
P.
Koumoutsakos
, “
Automating turbulence modelling by multi-agent reinforcement learning
,”
Nat. Mach. Intell.
3
,
87
96
(
2021
).
61.
J.
Sirignano
,
J. F.
MacArt
, and
J. B.
Freund
, “
DPM: A deep learning PDE augmentation method with application to large-eddy simulation
,”
J. Comput. Phys.
423
,
109811
(
2020
).
62.
J. F.
MacArt
,
J.
Sirignano
, and
J. B.
Freund
, “
Embedded training of neural-network subgrid-scale turbulence models
,”
Phys. Rev. Fluids
6
,
050502
(
2021
).
63.
T. P.
Lillicrap
,
J. J.
Hunt
,
A.
Pritzel
,
N.
Heess
,
T.
Erez
,
Y.
Tassa
,
D.
Silver
, and
D.
Wierstra
, “
Continuous control with deep reinforcement learning
,” arXiv:1509.02971v6 (
2015
).
64.
V.
Mnih
,
K.
Kavukcuoglu
,
D.
Silver
,
A.
Graves
,
I.
Antonoglou
,
D.
Wierstra
, and
M.
Riedmiller
, “
Playing Atari with deep reinforcement learning
,” arXiv:1312.5602v1 (
2013
).
65.
V.
Mnih
,
K.
Kavukcuoglu
,
D.
Silver
 et al, “
Human-level control through deep reinforcement learning
,”
Nature
518
,
529
533
(
2015
).
66.
S.
Fujimoto
,
H.
van Hoof
, and
D.
Meger
, “
Addressing function approximation error in actor-critic methods
,” arXiv:1802.09477 (
2018
).
67.
G.
Barth-Maron
,
M. W.
Hoffman
,
D.
Budden
,
W.
Dabney
,
D.
Horgan
,
D.
TB
,
A.
Muldal
,
N.
Heess
, and
T.
Lillicrap
, “
Distributed distributional deterministic policy gradients
,” arXiv:1804.08617 (
2018
).
68.
M.
Plappert
,
R.
Houthooft
,
P.
Dhariwal
,
S.
Sidor
,
R. Y.
Chen
,
X.
Chen
,
T.
Asfour
,
P.
Abbeel
, and
M.
Andrychowicz
, “
Parameter space noise for exploration
,” arXiv:1706.01905v2 (
2017
).
69.
J.
Ling
,
A.
Kurzawski
, and
J.
Templeton
, “
Reynolds averaged turbulence modelling using deep neural networks with embedded invariance
,”
J. Fluid Mech.
807
,
155
166
(
2016
).
70.
J.
Kim
and
C.
Lee
, “
Deep unsupervised learning of turbulence for inflow generation at various Reynolds numbers
,”
J. Comput. Phys.
406
,
109216
(
2020
).
71.
J.
Kim
and
C.
Lee
, “
Prediction of turbulent heat transfer using convolutional neural networks
,”
J. Fluid Mech.
882
,
A18
(
2020
).
72.
M.
Raissi
,
P.
Perdikaris
, and
G.
Karniadakis
, “
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
,”
J. Comput. Phys.
378
,
686
707
(
2019
).
73.
S.
Lee
and
D.
You
, “
Data-driven prediction of unsteady flow over a circular cylinder using deep learning
,”
J. Fluid Mech.
879
,
217
254
(
2019
).
74.
K.
He
,
X.
Zhang
,
S.
Ren
, and
J.
Sun
, “
Deep residual learning for image recognition
,” arXiv:1512.03385v1 (
2015
).
75.
H.
Kim
,
J.
Kim
,
S.
Won
, and
C.
Lee
, “
Unsupervised deep learning for super-resolution reconstruction of turbulence
,”
J. Fluid Mech.
910
,
A29
(
2021
).
76.
D. P.
Kingma
and
J. L.
Ba
, “
Adam: A method for stochastic optimization
,” arXiv:1412.6980 (
2014
).
77.
M.
Abadi
 et al, www.tensorflow.org for “
Tensorflow: Large-scale machine learning on heterogeneous systems
.”
78.
J.
Kim
, “
Modeling and prediction of wall-bounded turbulence using deep learning
,” Ph.D. thesis (
Yonsei University
,
2022
).
79.
M.
Quadrio
and
P.
Luchini
, “
Integral space–time scales in turbulent wall flows
,”
Phys. Fluids
15
,
2219
2227
(
2003
).
80.
J.
Kim
,
P.
Moin
, and
R.
Moser
, “
Turbulence statistics in fully developed channel flow at low Reynolds number
,”
J. Fluid Mech.
177
,
133
166
(
1987
).
81.
U.
Piomelli
,
W. H.
Cabot
,
P.
Moin
, and
S.
Lee
, “
Subgrid-scale backscatter in turbulent and transitional flows
,”
Phys. Fluids A
3
,
1766
1771
(
1991
).
82.
U.
Piomelli
,
Y.
Yu
, and
R. J.
Adrian
, “
Subgrid-scale energy transfer and near-wall turbulence structure
,”
Phys. Fluids
8
,
215
224
(
1996
).
83.
S.
Völker
,
R. D.
Moser
, and
P.
Venugopal
, “
Optimal large eddy simulation of turbulent channel flow based on direct numerical simulation statistical data
,”
Phys. Fluids
14
,
3675
3691
(
2002
).
84.
J.
Jeong
and
F.
Hussain
, “
On the identification of a vortex
,”
J. Fluid Mech.
285
,
69
94
(
1995
).
85.
R. D.
Moser
,
J.
Kim
, and
N. N.
Mansour
, “
Direct numerical simulation of turbulent channel flow up to Reτ=590
,”
Phys. Fluids
11
,
943
945
(
1999
).
86.
X. I. A.
Yang
,
G. I.
Park
, and
P.
Moin
, “
Log-layer mismatch and modeling of the fluctuating wall stress in wall-modeled large-eddy simulations
,”
Phys. Rev. Fluids
2
,
104601
(
2017
).
87.
S. B.
Pope
, “
A more general effective-viscosity hypothesis
,”
J. Fluid Mech.
72
,
331
(
1975
).
88.
P. Y.
Lu
,
S.
Kim
, and
M.
Soljačić
, “
Extracting interpretable physical parameters from spatiotemporal systems using unsupervised learning
,”
Phys. Rev. X
10
,
031056
(
2020
).
89.
T. N.
Kipf
and
M.
Welling
, “
Semi-supervised classification with graph convolutional networks
,” arXiv:1609.02907v4 (
2016
).
90.
C. R.
Qi
,
H.
Su
,
K.
Mo
, and
L. J.
Guibas
, “
PointNet: Deep learning on point sets for 3D classification and segmentation
,” in
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
,
2017
.
91.
A.
Kashefi
,
D.
Rempe
, and
L. J.
Guibas
, “
A point-cloud deep learning framework for prediction of fluid flow fields on irregular geometries
,”
Phys. Fluids
33
,
027104
(
2021
).
92.
C. A. M.
Ströfer
and
H.
Xiao
, “
End-to-end differentiable learning of turbulence models from indirect observations
,”
Theor. Appl. Mech. Lett.
11
,
100280
(
2021
).
93.
B.
List
,
L.-W.
Chen
, and
N.
Thuerey
, “
Learned turbulence modelling with differentiable fluid solvers
,” arXiv:2202.06988 (
2022
).
94.
M.
Ma
,
W.-X.
Huang
, and
C.-X.
Xu
, “
A dynamic wall model for large eddy simulation of turbulent flow over complex/moving boundaries based on the immersed boundary method
,”
Phys. Fluids
31
,
115101
(
2019
).
95.
D.-G.
Caprace
,
P.
Chatelain
, and
G.
Winckelmans
, “
Wakes of rotorcraft in advancing flight: A large-eddy simulation study
,”
Phys. Fluids
32
,
087107
(
2020
).
96.
X. I. A.
Yang
,
S.
Zafar
,
J.-X.
Wang
, and
H.
Xiao
, “
Predictive large-eddy-simulation wall modeling via physics-informed neural networks
,”
Phys. Rev. Fluids
4
,
034602
(
2019
).
97.
J.
Kim
,
H.
Kim
,
J.
Kim
, and
C.
Lee
(
2022
). “Deep reinforcement learning for large-eddy simulation modeling in wall-bounded turbulence,”
GitHub
. https://github.com/junhyuk6/DRL-LES.
You do not currently have access to this content.