The development of a reliable subgrid-scale (SGS) model for large-eddy simulation (LES) is of great importance for many scientific and engineering applications. Recently, deep learning approaches have been tested for this purpose using high-fidelity data such as direct numerical simulation (DNS) in a supervised learning process. However, such data are generally not available in practice. Deep reinforcement learning (DRL) using only limited target statistics can be an alternative algorithm in which the training and testing of the model are conducted in the same LES environment. The DRL of turbulence modeling remains challenging owing to its chaotic nature, high dimensionality of the action space, and large computational cost. In this study, we propose a physics-constrained DRL framework that can develop a deep neural network-based SGS model for LES of turbulent channel flow. The DRL models that produce the SGS stress were trained based on the local gradient of the filtered velocities. The developed SGS model automatically satisfies the reflectional invariance and wall boundary conditions without an extra training process so that DRL can quickly find the optimal policy. Furthermore, direct accumulation of reward, spatially and temporally correlated exploration, and the pre-training process are applied for efficient and effective learning. In various environments, our DRL could discover SGS models that produce the viscous and Reynolds stress statistics perfectly consistent with the filtered DNS. By comparing various statistics obtained by the trained models and conventional SGS models, we present a possible interpretation of better performance of the DRL model.
References
1.
P.
Moin
and K.
Mahesh
, “Direct numerical simulation: A tool in turbulence research
,” Annu. Rev. Fluid Mech.
30
, 539
–578
(1998
).2.
M.
Lesieur
and O.
Metais
, “New trends in large-eddy simulations of turbulence
,” Annu. Rev. Fluid Mech.
28
, 45
–82
(1996
).3.
C.
Meneveau
and J.
Katz
, “Scale-invariance and turbulence models for large-eddy simulation
,” Annu. Rev. Fluid Mech.
32
, 1
–32
(2000
).4.
U.
Piomelli
and E.
Balaras
, “Wall-layer models for large-eddy simulations
,” Annu. Rev. Fluid Mech.
34
, 349
–374
(2002
).5.
S. B.
Pope
, “Ten questions concerning the large-eddy simulation of turbulent flows
,” New J. Phys.
6
, 35
–35
(2004
).6.
P. R.
Spalart
, “Detached-eddy simulation
,” Annu. Rev. Fluid Mech.
41
, 181
–202
(2009
).7.
S. T.
Bose
and G. I.
Park
, “Wall-modeled large-eddy simulation for complex turbulent flows
,” Annu. Rev. Fluid Mech.
50
, 535
–561
(2018
).8.
R. D.
Moser
, S. W.
Haering
, and G. R.
Yalla
, “Statistical properties of subgrid-scale turbulence models
,” Annu. Rev. Fluid Mech.
53
, 255
–286
(2021
).9.
J.
Smagorinsky
, “General circulation experiments with primitive equations
,” Mon. Weather Rev.
91
, 99
–164
(1963
).10.
M.
Germano
, U.
Piomelli
, P.
Moin
, and W. H.
Cabot
, “A dynamic subgrid-scale eddy viscosity model
,” Phys. Fluids A
3
, 1760
–1765
(1991
).11.
D. K.
Lilly
, “A proposed modification of the Germano subgrid-scale closure method
,” Phys. Fluids A
4
, 633
–635
(1992
).12.
A. W.
Vreman
, “An eddy-viscosity subgrid-scale model for turbulent shear flow: Algebraic theory and applications
,” Phys. Fluids
16
, 3670
–3681
(2004
).13.
N.
Park
, S.
Lee
, J.
Lee
, and H.
Choi
, “A dynamic subgrid-scale eddy viscosity model with a global model coefficient
,” Phys. Fluids
18
, 125109
(2006
).14.
D.
You
and P.
Moin
, “A dynamic global-coefficient subgrid-scale eddy-viscosity model for large-eddy simulation in complex geometries
,” Phys. Fluids
19
, 065110
(2007
).15.
J.
Bardina
, J.
Ferziger
, and W.
Reynolds
, “Improved subgrid-scale models for large-eddy simulation
,” in 13th Fluid and Plasma Dynamics Conference
(American Institute of Aeronautics and Astronautics
, 1980
).16.
S.
Liu
, C.
Meneveau
, and J.
Katz
, “On the properties of similarity subgrid-scale models as deduced from measurements in a turbulent jet
,” J. Fluid Mech.
275
, 83
–119
(1994
).17.
A.
Leonard
, “Energy Cascade in large-eddy simulations of turbulent fluid flows
,” in Advances in Geophysics
(Elsevier
, 1974
), Vol. 18
, pp. 237
–248
.18.
R. A.
Clark
, J. H.
F
, and W. C.
Reynolds
, “Evaluation of subgrid-scale models using an accurately simulated turbulent flow
,” J. Fluid Mech.
91
, 1
–16
(1979
).19.
Y.
Zang
, R. L.
Street
, and J. R.
Koseff
, “A dynamic mixed subgrid-scale model and its application to turbulent recirculating flows
,” Phys. Fluids A
5
, 3186
–3196
(1993
).20.
M. V.
Salvetti
and S.
Banerjee
, “A priori tests of a new dynamic subgrid-scale model for finite-difference large-eddy simulations
,” Phys. Fluids
7
, 2831
–2847
(1995
).21.
K.
Horiuti
, “A new dynamic two-parameter mixed model for large-eddy simulation
,” Phys. Fluids
9
, 3443
–3464
(1997
).22.
J. N.
Kutz
, “Deep learning in fluid dynamics
,” J. Fluid Mech.
814
, 1
–4
(2017
).23.
M. P.
Brenner
, J. D.
Eldredge
, and J. B.
Freund
, “Perspective on machine learning for advancing fluid mechanics
,” Phys. Rev. Fluids
4
, 100501
(2019
).24.
K.
Duraisamy
, G.
Iaccarino
, and H.
Xiao
, “Turbulence modeling in the age of data
,” Annu. Rev. Fluid Mech.
51
, 357
–377
(2019
).25.
S. L.
Brunton
, B. R.
Noack
, and P.
Koumoutsakos
, “Machine learning for fluid mechanics
,” Annu. Rev. Fluid Mech.
52
, 477
–508
(2020
).26.
K.
Duraisamy
, “Perspectives on machine learning-augmented Reynolds-averaged and large eddy simulation models of turbulence
,” Phys. Rev. Fluids
6
, 050504
(2021
).27.
R.
Maulik
and O.
San
, “A neural network approach for the blind deconvolution of turbulent flows
,” J. Fluid Mech.
831
, 151
–181
(2017
).28.
R.
Maulik
, O.
San
, A.
Rasheed
, and P.
Vedula
, “Subgrid modelling for two-dimensional turbulence using neural networks
,” J. Fluid Mech.
858
, 122
–144
(2019
).29.
Y.
Guan
, A.
Chattopadhyay
, A.
Subel
, and P.
Hassanzadeh
, “Stable a posteriori les of 2D turbulence using convolutional neural networks: Backscattering analysis and generalization to higher re via transfer learning
,” arXiv:2102.11400v1 (2021
).30.
D.
Kochkov
, J. A.
Smith
, A.
Alieva
, Q.
Wang
, M. P.
Brenner
, and S.
Hoyer
, “Machine learning–accelerated computational fluid dynamics
,” Proc. Natl. Acad. Sci. U. S. A.
118
, e2101784118
(2021
).31.
A.
Vollant
, G.
Balarac
, and C.
Corre
, “Subgrid-scale scalar flux modelling based on optimal estimation theory and machine-learning procedures
,” J. Turbul.
18
, 854
–878
(2017
).32.
Z.
Zhou
, G.
He
, S.
Wang
, and G.
Jin
, “Subgrid-scale model for large-eddy simulation of isotropic turbulent flows using an artificial neural network
,” Comput. Fluids
195
, 104319
(2019
).33.
C.
Xie
, J.
Wang
, and W.
E
, “Modeling subgrid-scale forces by spatial artificial neural networks in large eddy simulation of turbulence
,” Phys. Rev. Fluids
5
, 054606
(2020
).34.
G. D.
Portwood
, B. T.
Nadiga
, J. A.
Saenz
, and D.
Livescu
, “Interpreting neural network models of residual scalar flux
,” J. Fluid Mech.
907
, A23
(2021
).35.
H.
Frezat
, G.
Balarac
, J. L.
Sommer
, R.
Fablet
, and R.
Lguensat
, “Physical invariance in neural networks for subgrid-scale scalar flux modeling
,” Phys. Rev. Fluids
6
, 024607
(2021
).36.
A.
Prakash
, K. E.
Jansen
, and J. A.
Evans
, “Invariant data-driven subgrid stress modeling in the strain-rate eigenframe for large eddy simulation
,” arXiv:2106.13410v1 (2021
).37.
Z.
Wang
, K.
Luo
, D.
Li
, J.
Tan
, and J.
Fan
, “Investigations of data-driven closure for subgrid-scale stress in large-eddy simulation
,” Phys. Fluids
30
, 125101
(2018
).38.
A.
Beck
, D.
Flad
, and C.-D.
Munz
, “Deep neural networks for data-driven LES closure models
,” J. Comput. Phys.
398
, 108910
(2019
).39.
C.
Xie
, K.
Li
, C.
Ma
, and J.
Wang
, “Modeling subgrid-scale force and divergence of heat flux of compressible isotropic turbulence by artificial neural network
,” Phys. Rev. Fluids
4
, 104605
(2019
).40.
C.
Xie
, J.
Wang
, K.
Li
, and C.
Ma
, “Artificial neural network approach to large-eddy simulation of compressible isotropic turbulence
,” Phys. Rev. E
99
, 053113
(2019
).41.
C.
Wollblad
and L.
Davidson
, “POD based reconstruction of subgrid stresses for wall bounded flows using neural networks
,” Flow Turbul. Combust.
81
, 77
–96
(2008
).42.
M.
Gamahara
and Y.
Hattori
, “Searching for turbulence models by artificial neural network
,” Phys. Rev. Fluids
2
, 054604
(2017
).43.
J.
Park
and H.
Choi
, “Toward neural-network-based large eddy simulation: Application to turbulent channel flow
,” J. Fluid Mech.
914
, A16
(2021
).44.
B.
Font
, G. D.
Weymouth
, V.-T.
Nguyen
, and O. R.
Tutty
, “Deep learning of the spanwise-averaged Navier–Stokes equations
,” J. Comput. Phys.
434
, 110199
(2021
).45.
S.
Colabrese
, K.
Gustavsson
, A.
Celani
, and L.
Biferale
, “Flow navigation by smart microswimmers via reinforcement learning
,” Phys. Rev. Lett.
118
, 158004
(2017
).46.
S.
Verma
, G.
Novati
, and P.
Koumoutsakos
, “Efficient collective swimming by harnessing vortices through deep reinforcement learning
,” Proc. Natl. Acad. Sci. U. S. A.
115
, 5849
–5854
(2018
).47.
J.
Rabault
, M.
Kuchta
, A.
Jensen
, U.
Réglade
, and N.
Cerardi
, “Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
,” J. Fluid Mech.
865
, 281
–302
(2019
).48.
J.
Rabault
and A.
Kuhnle
, “Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach
,” Phys. Fluids
31
, 094105
(2019
).49.
H.
Tang
, J.
Rabault
, A.
Kuhnle
, Y.
Wang
, and T.
Wang
, “Robust active flow control over a range of Reynolds numbers using an artificial neural network trained through deep reinforcement learning
,” Phys. Fluids
32
, 053605
(2020
).50.
D.
Fan
, L.
Yang
, Z.
Wang
, M. S.
Triantafyllou
, and G.
Karniadakis
, “Reinforcement learning for bluff body active flow control in experiments and simulations
,” Proc. Natl. Acad. Sci. U. S. A.
117
, 26091
–26098
(2020
).51.
R.
Paris
, S.
Beneddine
, and J.
Dandois
, “Robust flow control and optimal sensor placement using deep reinforcement learning
,” J. Fluid Mech.
913
, A25
(2021
).52.
G.
Beintema
, A.
Corbetta
, L.
Biferale
, and F.
Toschi
, “Controlling Rayleigh–Bénard convection via reinforcement learning
,” J. Turbul.
21
, 585
–605
(2020
).53.
E.
Hachem
, H.
Ghraieb
, J.
Viquerat
, A.
Larcher
, and P.
Meliga
, “Deep reinforcement learning for the control of conjugate heat transfer
,” J. Comput. Phys.
436
, 110317
(2021
).54.
M. A.
Bucci
, O.
Semeraro
, A.
Allauzen
, G.
Wisniewski
, L.
Cordier
, and L.
Mathelin
, “Control of chaotic systems by deep reinforcement learning
,” Proc. R. Soc. A
475
, 20190351
(2019
).55.
K.
Zeng
and M. D.
Graham
, “Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics
,” Phys. Rev. E
104
, 014210
(2021
).56.
J.
Viquerat
, J.
Rabault
, A.
Kuhnle
, H.
Ghraieb
, A.
Larcher
, and E.
Hachem
, “Direct shape optimization through deep reinforcement learning
,” J. Comput. Phys.
428
, 110080
(2021
).57.
J.
Rabault
, F.
Ren
, W.
Zhang
, H.
Tang
, and H.
Xu
, “Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization
,” J. Hydrodyn.
32
, 234
–246
(2020
).58.
F.
Ren
, H.
Hu
, and H.
Tang
, “Active flow control using machine learning: A brief review
,” J. Hydrodyn.
32
, 247
–253
(2020
).59.
P.
Garnier
, J.
Viquerat
, J.
Rabault
, A.
Larcher
, A.
Kuhnle
, and E.
Hachem
, “A review on deep reinforcement learning for fluid mechanics
,” Comput. Fluids
225
, 104973
(2021
).60.
G.
Novati
, H.
de Laroussilhe
, and P.
Koumoutsakos
, “Automating turbulence modelling by multi-agent reinforcement learning
,” Nat. Mach. Intell.
3
, 87
–96
(2021
).61.
J.
Sirignano
, J. F.
MacArt
, and J. B.
Freund
, “DPM: A deep learning PDE augmentation method with application to large-eddy simulation
,” J. Comput. Phys.
423
, 109811
(2020
).62.
J. F.
MacArt
, J.
Sirignano
, and J. B.
Freund
, “Embedded training of neural-network subgrid-scale turbulence models
,” Phys. Rev. Fluids
6
, 050502
(2021
).63.
T. P.
Lillicrap
, J. J.
Hunt
, A.
Pritzel
, N.
Heess
, T.
Erez
, Y.
Tassa
, D.
Silver
, and D.
Wierstra
, “Continuous control with deep reinforcement learning
,” arXiv:1509.02971v6 (2015
).64.
V.
Mnih
, K.
Kavukcuoglu
, D.
Silver
, A.
Graves
, I.
Antonoglou
, D.
Wierstra
, and M.
Riedmiller
, “Playing Atari with deep reinforcement learning
,” arXiv:1312.5602v1 (2013
).65.
V.
Mnih
, K.
Kavukcuoglu
, D.
Silver
et al., “Human-level control through deep reinforcement learning
,” Nature
518
, 529
–533
(2015
).66.
S.
Fujimoto
, H.
van Hoof
, and D.
Meger
, “Addressing function approximation error in actor-critic methods
,” arXiv:1802.09477 (2018
).67.
G.
Barth-Maron
, M. W.
Hoffman
, D.
Budden
, W.
Dabney
, D.
Horgan
, D.
TB
, A.
Muldal
, N.
Heess
, and T.
Lillicrap
, “Distributed distributional deterministic policy gradients
,” arXiv:1804.08617 (2018
).68.
M.
Plappert
, R.
Houthooft
, P.
Dhariwal
, S.
Sidor
, R. Y.
Chen
, X.
Chen
, T.
Asfour
, P.
Abbeel
, and M.
Andrychowicz
, “Parameter space noise for exploration
,” arXiv:1706.01905v2 (2017
).69.
J.
Ling
, A.
Kurzawski
, and J.
Templeton
, “Reynolds averaged turbulence modelling using deep neural networks with embedded invariance
,” J. Fluid Mech.
807
, 155
–166
(2016
).70.
J.
Kim
and C.
Lee
, “Deep unsupervised learning of turbulence for inflow generation at various Reynolds numbers
,” J. Comput. Phys.
406
, 109216
(2020
).71.
J.
Kim
and C.
Lee
, “Prediction of turbulent heat transfer using convolutional neural networks
,” J. Fluid Mech.
882
, A18
(2020
).72.
M.
Raissi
, P.
Perdikaris
, and G.
Karniadakis
, “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
,” J. Comput. Phys.
378
, 686
–707
(2019
).73.
S.
Lee
and D.
You
, “Data-driven prediction of unsteady flow over a circular cylinder using deep learning
,” J. Fluid Mech.
879
, 217
–254
(2019
).74.
K.
He
, X.
Zhang
, S.
Ren
, and J.
Sun
, “Deep residual learning for image recognition
,” arXiv:1512.03385v1 (2015
).75.
H.
Kim
, J.
Kim
, S.
Won
, and C.
Lee
, “Unsupervised deep learning for super-resolution reconstruction of turbulence
,” J. Fluid Mech.
910
, A29
(2021
).76.
77.
M.
Abadi
et al., www.tensorflow.org for “Tensorflow: Large-scale machine learning on heterogeneous systems
.”78.
J.
Kim
, “Modeling and prediction of wall-bounded turbulence using deep learning
,” Ph.D. thesis (Yonsei University
, 2022
).79.
M.
Quadrio
and P.
Luchini
, “Integral space–time scales in turbulent wall flows
,” Phys. Fluids
15
, 2219
–2227
(2003
).80.
J.
Kim
, P.
Moin
, and R.
Moser
, “Turbulence statistics in fully developed channel flow at low Reynolds number
,” J. Fluid Mech.
177
, 133
–166
(1987
).81.
U.
Piomelli
, W. H.
Cabot
, P.
Moin
, and S.
Lee
, “Subgrid-scale backscatter in turbulent and transitional flows
,” Phys. Fluids A
3
, 1766
–1771
(1991
).82.
U.
Piomelli
, Y.
Yu
, and R. J.
Adrian
, “Subgrid-scale energy transfer and near-wall turbulence structure
,” Phys. Fluids
8
, 215
–224
(1996
).83.
S.
Völker
, R. D.
Moser
, and P.
Venugopal
, “Optimal large eddy simulation of turbulent channel flow based on direct numerical simulation statistical data
,” Phys. Fluids
14
, 3675
–3691
(2002
).84.
J.
Jeong
and F.
Hussain
, “On the identification of a vortex
,” J. Fluid Mech.
285
, 69
–94
(1995
).85.
R. D.
Moser
, J.
Kim
, and N. N.
Mansour
, “Direct numerical simulation of turbulent channel flow up to =590
,” Phys. Fluids
11
, 943
–945
(1999
).86.
X. I. A.
Yang
, G. I.
Park
, and P.
Moin
, “Log-layer mismatch and modeling of the fluctuating wall stress in wall-modeled large-eddy simulations
,” Phys. Rev. Fluids
2
, 104601
(2017
).87.
S. B.
Pope
, “A more general effective-viscosity hypothesis
,” J. Fluid Mech.
72
, 331
(1975
).88.
P. Y.
Lu
, S.
Kim
, and M.
Soljačić
, “Extracting interpretable physical parameters from spatiotemporal systems using unsupervised learning
,” Phys. Rev. X
10
, 031056
(2020
).89.
T. N.
Kipf
and M.
Welling
, “Semi-supervised classification with graph convolutional networks
,” arXiv:1609.02907v4 (2016
).90.
C. R.
Qi
, H.
Su
, K.
Mo
, and L. J.
Guibas
, “PointNet: Deep learning on point sets for 3D classification and segmentation
,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)
, 2017
.91.
A.
Kashefi
, D.
Rempe
, and L. J.
Guibas
, “A point-cloud deep learning framework for prediction of fluid flow fields on irregular geometries
,” Phys. Fluids
33
, 027104
(2021
).92.
C. A. M.
Ströfer
and H.
Xiao
, “End-to-end differentiable learning of turbulence models from indirect observations
,” Theor. Appl. Mech. Lett.
11
, 100280
(2021
).93.
B.
List
, L.-W.
Chen
, and N.
Thuerey
, “Learned turbulence modelling with differentiable fluid solvers
,” arXiv:2202.06988 (2022
).94.
M.
Ma
, W.-X.
Huang
, and C.-X.
Xu
, “A dynamic wall model for large eddy simulation of turbulent flow over complex/moving boundaries based on the immersed boundary method
,” Phys. Fluids
31
, 115101
(2019
).95.
D.-G.
Caprace
, P.
Chatelain
, and G.
Winckelmans
, “Wakes of rotorcraft in advancing flight: A large-eddy simulation study
,” Phys. Fluids
32
, 087107
(2020
).96.
X. I. A.
Yang
, S.
Zafar
, J.-X.
Wang
, and H.
Xiao
, “Predictive large-eddy-simulation wall modeling via physics-informed neural networks
,” Phys. Rev. Fluids
4
, 034602
(2019
).97.
J.
Kim
, H.
Kim
, J.
Kim
, and C.
Lee
(2022
). “Deep reinforcement learning for large-eddy simulation modeling in wall-bounded turbulence,” GitHub
. https://github.com/junhyuk6/DRL-LES.© 2022 Author(s). Published under an exclusive license by AIP Publishing.
2022
Author(s)
You do not currently have access to this content.
