This paper focuses on the use of reinforcement learning (RL) as a machine-learning (ML) modeling tool for near-wall turbulence. RL has demonstrated its effectiveness in solving high-dimensional problems, especially in domains such as games. Despite its potential, RL is still not widely used for turbulence modeling and is primarily used for flow control and optimization purposes. A new RL wall model (WM) called VYBA23 is developed in this work, which uses agents dispersed in the flow near the wall. The model is trained on a single Reynolds number ( R e τ = 10 4) and does not rely on high-fidelity data, as the backpropagation process is based on a reward rather than an output error. The states of the RLWM, which are the representation of the environment by the agents, are normalized to remove dependence on the Reynolds number. The model is tested and compared to another RLWM (BK22) and to an equilibrium wall model, in a half-channel flow at eleven different Reynolds numbers { R e τ [ 180 ; 10 10 ]}. The effects of varying agents' parameters, such as actions range, time step, and spacing, are also studied. The results are promising, showing little effect on the average flow field but some effect on wall-shear stress fluctuations and velocity fluctuations. This work offers positive prospects for developing RLWMs that can recover physical laws and for extending this type of ML models to more complex flows in the future.

1.
J.
Kober
,
J. A.
Bagnell
, and
J.
Peters
, “
Reinforcement learning in robotics: A survey
,”
Int. J. Rob. Res.
32
,
1238
1274
(
2013
).
2.
T. T.
Nguyen
and
V. J.
Reddi
, “
Deep reinforcement learning for cyber security
,”
IEEE Trans. Neural Networks Learn. Syst.
(published online 2021).
3.
J.
Jumper
,
R.
Evans
,
A.
Pritzel
,
T.
Green
,
M.
Figurnov
,
O.
Ronneberger
,
K.
Tunyasuvunakool
,
R.
Bates
,
A.
Žídek
,
A.
Potapenko
et al, “
Highly accurate protein structure prediction with AlphaFold
,”
Nature
596
,
583
589
(
2021
).
4.
D.
Silver
,
T.
Hubert
,
J.
Schrittwieser
,
I.
Antonoglou
,
M.
Lai
,
A.
Guez
,
M.
Lanctot
,
L.
Sifre
,
D.
Kumaran
,
T.
Graepel
et al, “
A general reinforcement learning algorithm that masters chess, shogi, and go through self-play
,”
Science
362
,
1140
1144
(
2018
).
5.
X.
Yang
,
S.
Zafar
,
J.-X.
Wang
, and
H.
Xiao
, “
Predictive large-eddy-simulation wall modeling via physics-informed neural networks
,”
Phys. Rev. Fluids
4
,
034602
(
2019
).
6.
R.
Maulik
,
O.
San
,
J. D.
Jacob
, and
C.
Crick
, “
Sub-grid scale model classification and blending through deep learning
,”
J. Fluid Mech.
870
,
784
812
(
2019
).
7.
A.
Eidi
,
N.
Zehtabiyan-Rezaie
,
R.
Ghiassi
,
X.
Yang
, and
M.
Abkar
, “
Data-driven quantification of model-form uncertainty in Reynolds-averaged simulations of wind farms
,”
Phys. Fluids
34
,
085135
(
2022
).
8.
F.
Ren
,
H-b
Hu
, and
H.
Tang
, “
Active flow control using machine learning: A brief review
,”
J. Hydrodyn.
32
,
247
253
(
2020
).
9.
L.
Guastoni
,
J.
Rabault
,
P.
Schlatter
,
H.
Azizpour
, and
R.
Vinuesa
, “
Deep reinforcement learning for turbulent drag reduction in channel flows
,” arXiv:2301.09889 (
2023
).
10.
K.
Hasegawa
,
K.
Fukami
,
T.
Murata
, and
K.
Fukagata
, “
Machine-learning-based reduced-order modeling for unsteady flows around bluff bodies of various shapes
,”
Theor. Comput. Fluid Dyn.
34
,
367
383
(
2020
).
11.
J.
Viquerat
,
J.
Rabault
,
A.
Kuhnle
,
H.
Ghraieb
,
A.
Larcher
, and
E.
Hachem
, “
Direct shape optimization through deep reinforcement learning
,”
J. Comput. Phys.
428
,
110080
(
2021
).
12.
B. S.
Mekki
,
J.
Langer
, and
S.
Lynch
, “
Genetic algorithm based topology optimization of heat exchanger fins used in aerospace applications
,”
Int. J. Heat Mass Transfer
170
,
121002
(
2021
).
13.
M. J.
Rincón
,
M.
Reclari
,
X. I.
Yang
, and
M.
Abkar
, “
Validating the design optimisation of ultrasonic flow meters using computational fluid dynamics and surrogate modelling
,”
Int. J. Heat Fluid Flow
100
,
109112
(
2023
).
14.
J. L.
Callaham
,
J. V.
Koch
,
B. W.
Brunton
,
J. N.
Kutz
, and
S. L.
Brunton
, “
Learning dominant physical processes with data-driven balance models
,”
Nat. Commun.
12
,
1016
(
2021
).
15.
J.
Bakarji
,
J.
Callaham
,
S. L.
Brunton
, and
J. N.
Kutz
, “
Dimensionally consistent learning with buckingham Pi
,”
Nat. Comput. Sci.
2
,
834
844
(
2022
).
16.
K.
Duraisamy
,
G.
Iaccarino
, and
H.
Xiao
, “
Turbulence modeling in the age of data
,”
Annu. Rev. Fluid Mech.
51
,
357
377
(
2019
).
17.
S. L.
Brunton
,
B. R.
Noack
, and
P.
Koumoutsakos
, “
Machine learning for fluid mechanics
,”
Annu. Rev. Fluid Mech.
52
,
477
508
(
2020
).
18.
R.
Vinuesa
and
S. L.
Brunton
, “
Enhancing computational fluid dynamics with machine learning
,”
Nat. Comput. Sci.
2
,
358
366
(
2022
).
19.
N.
Zehtabiyan-Rezaie
,
A.
Iosifidis
, and
M.
Abkar
, “
Data-driven fluid mechanics of wind farms: A review
,”
J. Renewable Sustainable Energy
14
,
032703
(
2022
).
20.
C. M.
Legaard
,
T.
Schranz
,
G.
Schweiger
,
J.
Drgoňa
,
B.
Falay
,
C.
Gomes
,
A.
Iosifidis
,
M.
Abkar
, and
P. G.
Larsen
, “
Constructing neural network-based models for simulating dynamical systems
,”
ACM Comput. Surv.
55
,
1
34
(
2021
).
21.
P. A.
Durbin
, “
Some recent developments in turbulence closure modeling
,”
Annu. Rev. Fluid Mech.
50
,
77
103
(
2018
).
22.
H. J.
Bae
and
P.
Koumoutsakos
, “
Scientific multi-agent reinforcement learning for wall-models of turbulent flows
,”
Nat. Commun.
13
,
1443
(
2022
).
23.
J.-L.
Wu
,
H.
Xiao
, and
E.
Paterson
, “
Physics-informed machine learning approach for augmenting turbulence models: A comprehensive framework
,”
Phys. Rev. Fluids
3
,
074602
(
2018
).
24.
X.
Zhang
,
J.
Wu
,
O.
Coutier-Delgosha
, and
H.
Xiao
, “
Recent progress in augmenting turbulence models with physics-informed machine learning
,”
J. Hydrodyn.
31
,
1153
1158
(
2019
).
25.
M.
Kurz
,
P.
Offenhäuser
, and
A.
Beck
, “
Deep reinforcement learning for turbulence modeling in large eddy simulations
,”
Int. J. Heat Fluid Flow
99
,
109094
(
2023
).
26.
Z.
Deng
,
C.
He
,
Y.
Liu
, and
K. C.
Kim
, “
Super-resolution reconstruction of turbulent velocity fields using a generative adversarial network-based artificial intelligence framework
,”
Phys. Fluids
31
,
125111
(
2019
).
27.
C. J.
Lapeyre
,
A.
Misdariis
,
N.
Cazard
,
D.
Veynante
, and
T.
Poinsot
, “
Training convolutional neural networks to estimate turbulent sub-grid scale reaction rates
,”
Combust. Flame
203
,
255
264
(
2019
).
28.
J.
Duvall
,
K.
Duraisamy
, and
S.
Pan
, “
Discretization-independent surrogate modeling over complex geometries using hypernetworks and implicit representations
,” arXiv:2109.07018 (
2021
).
29.
G.
Novati
,
H. L.
de Laroussilhe
, and
P.
Koumoutsakos
, “
Automating turbulence modelling by multi-agent reinforcement learning
,”
Nat. Mach. Intell.
3
,
87
96
(
2021
).
30.
J.
Kim
,
H.
Kim
,
J.
Kim
, and
C.
Lee
, “
Deep reinforcement learning for large-eddy simulation modeling in wall-bounded turbulence
,”
Phys. Fluids
34
,
105132
(
2022
).
31.
S.
Verma
,
G.
Novati
, and
P.
Koumoutsakos
, “
Efficient collective swimming by harnessing vortices through deep reinforcement learning
,”
Proc. Natl. Acad. Sci. U. S. A.
115
,
5849
5854
(
2018
).
32.
T.-R.
Xiang
,
X.
Yang
, and
Y.-P.
Shi
, “
Neuroevolution-enabled adaptation of the Jacobi method for Poisson's equation with density discontinuities
,”
Theor. Appl. Mech. Lett.
11
,
100252
(
2021
).
33.
K.
Duraisamy
, “
Perspectives on machine learning-augmented Reynolds-averaged and large eddy simulation models of turbulence
,”
Phys. Rev. Fluids
6
,
050504
(
2021
).
34.
A.
Vadrot
,
X. I.
Yang
, and
M.
Abkar
, “
A survey of machine learning wall models for large eddy simulation
,” arXiv:2211.03614 (
2022
).
35.
X. I. A.
Yang
,
G. I.
Park
, and
P.
Moin
, “
Log-layer mismatch and modeling of the fluctuating wall stress in wall-modeled large-eddy simulations
,”
Phys. Rev. Fluids
2
,
104601
(
2017
).
36.
S.
Kawai
and
J.
Larsson
, “
Wall-modeling in large eddy simulation: Length scales, grid resolution, and accuracy
,”
Phys. Fluids
24
,
015105
(
2012
).
37.
U.
Schumann
, “
Subgrid scale model for finite difference simulations of turbulent flows in plane channels and annuli
,”
J. Comput. Phys.
18
,
376
404
(
1975
).
38.
X. I. A.
Yang
and
Y.
Lv
, “
A semi-locally scaled eddy viscosity formulation for LES wall models and flows at high speeds
,”
Theor. Comput. Fluid Dyn.
32
,
617
627
(
2018
).
39.
P. E.
Chen
,
Y.
Lv
,
H. H.
Xu
,
Y.
Shi
, and
X. I. A.
Yang
, “
LES wall modeling for heat transfer at high speeds
,”
Phys. Rev. Fluids
7
,
014608
(
2022
).
40.
H. H.
Xu
,
X. I. A.
Yang
, and
P. M.
Milani
, “
Assessing wall-modeled large-eddy simulation for low-speed flows with heat transfer
,”
AIAA J.
59
,
2060
2069
(
2021
).
41.
J.
Larsson
,
S.
Kawai
,
J.
Bodart
, and
I.
Bermejo-Moreno
, “
Large eddy simulation with modeled wall-stress: Recent progress and future directions
,”
Mech. Eng. Rev.
3
,
15–00418
00418
(
2016
).
42.
C.
Hansen
,
X. I.
Yang
, and
M.
Abkar
, “
POD-mode-augmented wall model and its applications to flows at non-equilibrium conditions
,” arXiv:2301.06803 (
2023
).
43.
M.
Fowler
,
T. A.
Zaki
, and
C.
Meneveau
, “
A Lagrangian relaxation towards equilibrium wall model for large eddy simulation
,”
J. Fluid Mech.
934
,
A44
(
2022
).
44.
X.
Yang
,
J.
Sadique
,
R.
Mittal
, and
C.
Meneveau
, “
Integral wall model for large eddy simulations of wall-bounded turbulent flows
,”
Phys. Fluids
27
,
025112
(
2015
).
45.
G. I.
Park
and
P.
Moin
, “
An improved dynamic non-equilibrium wall-model for large eddy simulation
,”
Phys. Fluids
26
,
015108
015148
(
2014
).
46.
S. T.
Bose
and
P.
Moin
, “
A dynamic slip boundary condition for wall-modeled large-eddy simulation
,”
Phys. Fluids
26
,
015104
(
2014
).
47.
H. J.
Bae
,
A.
Lozano-Durán
,
S. T.
Bose
, and
P.
Moin
, “
Dynamic slip wall model for large-eddy simulation
,”
J. Fluid Mech.
859
,
400
432
(
2019
).
48.
E.
Perlman
,
R.
Burns
,
Y.
Li
, and
C.
Meneveau
, “
Data exploration of turbulence simulations using a database cluster
,” in
Proceedings of the ACM/IEEE Conference on Supercomputing
(IEEE,
2007
), pp. 
1
11
.
49.
J.
Graham
,
K.
Kanov
,
X.
Yang
,
M.
Lee
,
N.
Malaya
,
C.
Lalescu
,
R.
Burns
,
G.
Eyink
,
A.
Szalay
,
R.
Moser
et al, “
A web services accessible database of turbulent channel flow and its use for testing a new integral wall model for LES
,”
J. Turbul.
17
,
181
215
(
2016
).
50.
M.
Lee
and
R. D.
Moser
, “
Direct numerical simulation of turbulent channel flow up to
,”
J. Fluid Mech.
774
,
395
415
(
2015
).
51.
X. L.
Huang
,
X. I. A.
Yang
, and
R. F.
Kunz
, “
Wall-modeled large-eddy simulations of spanwise rotating turbulent channels: Comparing a physics-based approach and a data-based approach
,”
Phys. Fluids
31
,
125105
(
2019
).
52.
X. L.
Huang
and
X. I. A.
Yang
, “
A Bayesian approach to the mean flow in a channel with small but arbitrarily directional system rotation
,”
Phys. Fluids
33
,
015103
(
2021
).
53.
Z.
Zhou
,
G.
He
, and
X.
Yang
, “
Wall model based on neural networks for LES of turbulent flows over periodic hills
,”
Phys. Rev. Fluids
6
,
054610
(
2021
).
54.
Z.
Zhou
,
X. I. A.
Yang
,
F.
Zhang
, and
X.
Yang
, “
A wall model trained using the periodic hill data and the law of the wall
,” arXiv:2211.03614 (
2022
).
55.
D.
Zhou
,
M.
Whitmore
,
K.
Griffin
, and
J.
Bae
, “
Multi-agent reinforcement learning for wall models in LES of flow over periodic hills
,” in
Center for Turbulence Research Summer Program Proceeding
,
2022
.
56.
A.
Lozano-Durán
and
H. J.
Bae
, “
Self-critical machine-learning wall-modeled LES for external aerodynamics
,” arXiv:2012.10005 (
2020
).
57.
R.
Bhaskaran
,
R.
Kannan
,
B.
Barr
, and
S.
Priebe
, “
Science-guided machine learning for wall-modeled large eddy simulation
,” in
IEEE International Conference on Big Data
(
IEEE
,
2021
), pp.
1809
1816
.
58.
S.
Radhakrishnan
,
L. A.
Gyamfi
,
A.
Miró
,
B.
Font
,
J.
Calafell
, and
O.
Lehmkuhl
, “
A data-driven wall-shear stress model for LES using gradient boosted decision trees
,” in
International Conference on High Performance Computing
(
Springer
,
2021
), pp.
105
121
.
59.
N.
Moriya
,
K.
Fukami
,
Y.
Nabae
,
M.
Morimoto
,
T.
Nakamura
, and
K.
Fukagata
, “
Inserting machine-learned virtual wall velocity for large-eddy simulation of turbulent channel flows
,” arXiv:2106.09271 (
2021
).
60.
Y.
Bin
,
L.
Chen
,
G.
Huang
, and
X. I. A.
Yang
, “
Progressive, extrapolative machine learning for near-wall turbulence modeling
,”
Phys. Rev. Fluids
7
,
084610
(
2022
).
61.
See https://lesgo.me.jhu.edu for “
LESGO: A Parallel Pseudo-Spectral Large-Eddy Simulation Code
.”
62.
F.
Porté-Agel
,
C.
Meneveau
, and
M. B.
Parlange
, “
A scale-dependent dynamic model for large-eddy simulation: Application to a neutral atmospheric boundary layer
,”
J. Fluid Mech.
415
,
261
284
(
2000
).
63.
M.
Abkar
and
F.
Porté-Agel
, “
A new boundary condition for large-eddy simulation of boundary-layer flow over surface roughness transitions
,”
J. Turbul.
13
,
N23
(
2012
).
64.
M.
Abkar
and
F.
Porté-Agel
, “
Influence of atmospheric stability on wind-turbine wakes: A large-eddy simulation study
,”
Phys. Fluids
27
,
035104
(
2015
).
65.
X. I. A.
Yang
and
M.
Abkar
, “
A hierarchical random additive model for passive scalars in wall-bounded flows at high Reynolds numbers
,”
J. Fluid Mech.
842
,
354
380
(
2018
).
66.
X. I.
Yang
,
S.
Pirozzoli
, and
M.
Abkar
, “
Scaling of velocity fluctuations in statistically unstable boundary-layer flows
,”
J. Fluid Mech.
886
,
A3
(
2020
).
67.
X. I. A.
Yang
,
P. E.
Chen
,
R.
Hu
, and
M.
Abkar
, “
Logarithmic-linear law of the streamwise velocity variance in stably stratified boundary layers
,”
Boundary-Layer Meteorol.
183
,
199
213
(
2022
).
68.
E.
Bou-Zeid
,
C.
Meneveau
, and
M.
Parlange
, “
A scale-dependent Lagrangian dynamic model for large eddy simulation of complex turbulent flows
,”
Phys. Fluids
17
,
025105
(
2005
).
69.
R.
Stoll
and
F.
Porté-Agel
, “
Dynamic subgrid-scale models for momentum and scalar fluxes in large-eddy simulations of neutrally stratified atmospheric boundary layers over heterogeneous terrain
,”
Water Resour. Res.
42
,
W01409
, https://doi.org/10.1029/2005WR003989 (
2006
).
70.
M.
Abkar
,
H. J.
Bae
, and
P.
Moin
, “
Minimum-dissipation scalar transport model for large-eddy simulation of turbulent flows
,”
Phys. Rev. Fluids
1
,
041701
(
2016
).
71.
M.
Abkar
and
P.
Moin
, “
Large-eddy simulation of thermally stratified atmospheric boundary-layer flow using a minimum dissipation model
,”
Boundary-Layer Meteorol.
165
,
405
419
(
2017
).
72.
J.
Smagorinsky
, “
General circulation experiments with the primitive equations: I. The basic experiment
,”
Mon. Weather Rev.
91
,
99
164
(
1963
).
73.
M.
Germano
,
U.
Piomelli
,
P.
Moin
, and
W. H.
Cabot
, “
A dynamic subgrid-scale eddy viscosity model
,”
Phys. Fluids
3
,
1760
1765
(
1991
).
74.
C.
Meneveau
,
T. S.
Lund
, and
W. H.
Cabot
, “
A Lagrangian dynamic subgrid-scale model of turbulence
,”
J. Fluid Mech.
319
,
353
385
(
1996
).
75.
W.
Rozema
,
H. J.
Bae
,
P.
Moin
, and
R.
Verstappen
, “
Minimum-dissipation models for large-eddy simulation
,”
Phys. Fluids
27
,
085107
(
2015
).
76.
X.
Yang
,
S.
Bose
, and
P.
Moin
, “
A physics-based interpretation of the slip-wall LES model
,”
Center for Turbulence Research, Annual Briefs
(
Stanford University
,
2016
), pp.
65
74
.
77.
C.
Hansen
,
M.
Whitmore
,
M.
Abkar
, and
X. I. A.
Yang
, “
POD-mode-augmented wall model and its applications to flows at non-equilibrium conditions
,” in
Center for Turbulence Research Summer Program Proceeding
,
2022
.
78.
C.-H.
Moeng
, “
A large-eddy-simulation model for the study of planetary boundary-layer turbulence
,”
J. Atmos. Sci.
41
,
2052
2062
(
1984
).
79.
I.
Marusic
,
J. P.
Monty
,
M.
Hultmark
, and
A. J.
Smits
, “
On the logarithmic region in wall turbulence
,”
J. Fluid Mech.
716
,
R3
(
2013
).
80.
G.
Novati
and
P.
Koumoutsakos
, “
Remember and forget for experience replay
,” in
Proceedings of the 36th International Conference on Machine Learning
,
2019
.
81.
H. M.
Nagib
and
K. A.
Chauhan
, “
Variations of von Kármán coefficient in canonical flows
,”
Phys. Fluids
20
,
101518
(
2008
).
82.
M.
Hultmark
,
M.
Vallikivi
,
S. C. C.
Bailey
, and
A.
Smits
, “
Turbulent pipe flow at extreme Reynolds numbers
,”
Phys. Rev. Lett.
108
,
094501
(
2012
).
83.
C.
Diaz-Daniel
,
S.
Laizet
, and
J. C.
Vassilicos
, “
Wall shear stress fluctuations: Mixed scaling and their effects on velocity fluctuations in a turbulent boundary layer
,”
Phys. Fluids
29
,
055102
(
2017
).
84.
C. J.
Maddison
,
A.
Huang
,
I.
Sutskever
, and
D.
Silver
, “
Move evaluation in go using deep convolutional neural networks
,” arXiv:1412.6564 (
2014
).
You do not currently have access to this content.