This paper focuses on the active flow control of a computational fluid dynamics simulation over a range of Reynolds numbers using deep reinforcement learning (DRL). More precisely, the proximal policy optimization (PPO) method is used to control the mass flow rate of four synthetic jets symmetrically located on the upper and lower sides of a cylinder immersed in a two-dimensional flow domain. The learning environment supports four flow configurations with Reynolds numbers 100, 200, 300, and 400, respectively. A new smoothing interpolation function is proposed to help the PPO algorithm learn to set continuous actions, which is of great importance to effectively suppress problematic jumps in lift and allow a better convergence for the training process. It is shown that the DRL controller is able to significantly reduce the lift and drag fluctuations and actively reduce the drag by ∼5.7%, 21.6%, 32.7%, and 38.7%, at Re = 100, 200, 300, and 400, respectively. More importantly, it can also effectively reduce drag for any previously unseen value of the Reynolds number between 60 and 400. This highlights the generalization ability of deep neural networks and is an important milestone toward the development of practical applications of DRL to active flow control.

1.
M.
Gad-el Hak
,
Flow Control: Passive, Active, and Reactive Flow Management
(
Cambridge University Press
,
2000
).
2.
P.
Ludwig
, “
Über flüssigkeitsbewegung bei sehr kleiner reibung
,” in
Proceedings of the 3rd International Mathematics Congress
(
Springer
,
Heidelberg, Germany
,
1904
), pp.
484
491
.
3.
R.
Vinuesa
,
H.
Azizpour
,
I.
Leite
,
M.
Balaam
,
V.
Dignum
,
S.
Domisch
,
F.
Anna
,
S.
Langhans
,
M.
Tegmark
, and
F.
Nerini
, “
The role of artificial intelligence in achieving the sustainable development goals
,”
Nat. Commun.
11
,
233
(
2020
).
4.
A. F.
Shahrabi
, “
The control of flow separation: Study of optimal open loop parameters
,”
Phys. Fluids
31
(
3
),
035104
(
2019
).
5.
D.
Dolgopyat
and
A.
Seifert
, “
Active flow control virtual maneuvering system applied to conventional airfoil
,”
AIAA J.
57
(
1
),
72
89
(
2019
).
6.
H.
Zhu
,
T.
Tang
,
H.
Zhao
, and
Y.
Gao
, “
Control of vortex-induced vibration of a circular cylinder using a pair of air jets at low Reynolds number
,”
Phys. Fluids
31
(
4
),
043603
(
2019
).
7.
C.
Wang
,
H.
Tang
,
S. C. M.
Yu
, and
F.
Duan
, “
Control of vortex-induced vibration using a pair of synthetic jets: Influence of active lock-on
,”
Phys. Fluids
29
(
8
),
083602
(
2017
).
8.
B.
Plumejeau
,
S.
Delprat
,
L.
Keirsbulck
,
M.
Lippert
, and
W.
Abassi
, “
Ultra-local model-based control of the square-back Ahmed body wake flow
,”
Phys. Fluids
31
(
8
),
085103
(
2019
).
9.
C.
Wang
,
H.
Tang
,
F.
Duan
, and
S. C. M.
Yu
, “
Control of wakes and vortex-induced vibrations of a single circular cylinder using synthetic jets
,”
J. Fluids Struct.
60
,
160
179
(
2016
).
10.
C.
Raibaudo
,
P.
Zhong
,
B. R.
Noack
, and
R. J.
Martinuzzi
, “
Machine learning strategies applied to the control of a fluidic pinball
,”
Phys. Fluids
32
(
1
),
015108
(
2020
).
11.
S.
Aubrun
,
A.
Leroy
, and
P.
Devinant
, “
A review of wind turbine-oriented active flow control strategies
,”
Exp. Fluids
58
(
10
),
134
(
2017
).
12.
R.
Pereira
,
W. A.
Timmer
,
G.
de Oliveira
, and
G. J. W.
van Bussel
, “
Design of HAWT airfoils tailored for active flow control
,”
Wind Energy
20
(
9
),
1569
1583
(
2017
).
13.
A.
Wolf
,
T.
Lutz
,
W.
Würz
,
E.
Krämer
,
O.
Stalnov
, and
A.
Seifert
, “
Trailing edge noise reduction of wind turbine blades by active flow control
,”
Wind Energy
18
(
5
),
909
923
(
2015
).
14.
J.
Bons
,
S.
Benton
,
C.
Bernardini
, and
M.
Bloxham
, “
Active flow control for low-pressure turbines
,”
AIAA J.
56
(
7
),
2687
2698
(
2018
).
15.
S. L.
Brunton
,
B. R.
Noack
, and
P.
Koumoutsakos
, “
Machine learning for fluid mechanics
,”
Annu. Rev. Fluid Mech.
52
(
1
),
477
508
(
2020
).
16.
S.
Collis
,
R. D.
Joslin
,
A.
Seifert
, and
V.
Theofilis
, “
Issues in active flow control: Theory, control, simulation, and experiment
,”
Prog. Aerosp. Sci.
40
(
4
),
237
289
(
2004
).
17.
T. L. B.
Flinois
and
T.
Colonius
, “
Optimal control of circular cylinder wakes using long control horizons
,”
Phys. Fluids
27
(
8
),
087105
(
2015
).
18.
C.
Leclercq
,
F.
Demourant
,
C.
Poussot-Vassal
, and
D.
Sipp
, “
Linear iterative method for closed-loop control of quasiperiodic flows
,”
J. Fluid Mech.
868
,
26
65
(
2019
).
19.
M.
Bergmann
,
L.
Cordier
, and
J.-P.
Brancher
, “
Optimal rotary control of the cylinder wake using proper orthogonal decomposition reduced-order model
,”
Phys. Fluids
17
(
9
),
097101
(
2005
).
20.
R. D.
Brackston
,
J. M.
García de la Cruz
,
A.
Wynn
,
G.
Rigas
, and
J. F.
Morrison
, “
Stochastic modelling and feedback control of bistability in a turbulent bluff body wake
,”
J. Fluid Mech.
802
,
726
749
(
2016
).
21.
S. L.
Brunton
and
B. R.
Noack
, “
Closed-loop turbulence control: Progress and challenges
,”
Appl. Mech. Rev.
67
(
5
),
050801
(
2015
).
22.
W.
Schoppa
and
F.
Hussain
, “
A large-scale control strategy for drag reduction in turbulent boundary layers
,”
Phys. Fluids
10
(
5
),
1049
1051
(
1998
).
23.
N.
Gautier
,
J.-L.
Aider
,
T.
Duriez
,
B. R.
Noack
,
M.
Segond
, and
M.
Abel
, “
Closed-loop separation control using machine learning
,”
J. Fluid Mech.
770
,
442
457
(
2015
).
24.
T.
Duriez
,
S. L.
Brunton
, and
B. R.
Noack
,
Machine Learning Control-Taming Nonlinear Dynamics and Turbulence
(
Springer
,
2017
).
25.
F.
Ren
,
C.
Wang
, and
H.
Tang
, “
Active control of vortex-induced vibration of a circular cylinder using machine learning
,”
Phys. Fluids
31
(
9
),
093601
(
2019
).
26.
D.
Antoine
,
K. A. F. F.
Von Krbek
,
N.
Mazellier
,
T.
Duriez
,
L.
Cordier
,
B. R.
Noack
,
M. W.
Abel
, and
A.
Kourta
, “
Closed-loop separation control over a sharp edge ramp using genetic programming
,”
Exp. Fluids
57
(
3
),
40
(
2016
).
27.
V.
Mnih
,
K.
Kavukcuoglu
,
D.
Silver
,
A. A.
Rusu
,
J.
Veness
,
M. G.
Bellemare
,
A.
Graves
,
M.
Riedmiller
,
A. K.
Fidjeland
,
G.
Ostrovski
,
S.
Petersen
,
A. C.
Beattie
,
I.
Sadik
,
I.
Antonoglou
,
D. H.
King
,
D.
Kumaran
,
D.
Wierstra
,
D.
Legg
, and
D.
Hassabis
, “
Human-level control through deep reinforcement learning
,”
Nature
518
(
7540
),
529
533
(
2015
).
28.
Y.
Duan
,
X.
Chen
,
R.
Houthooft
,
J.
Schulman
, and
P.
Abbeel
, “
Benchmarking deep reinforcement learning for continuous control
,” in
International Conference on Machine Learning
(
ICML
,
2016
), pp.
1329
1338
.
29.
S.
Gu
,
T.
Lillicrap
,
I.
Sutskever
, and
S.
Levine
, “
Continuous deep Q-learning with model-based acceleration
,” in
International Conference on Machine Learning
(
ICML
,
2016
), pp.
2829
2838
.
30.
M.
Hessel
,
J.
Modayil
,
H.
Van Hasselt
,
T.
Schaul
,
G.
Ostrovski
,
W.
Dabney
,
D.
Horgan
,
B.
Piot
,
M.
Azar
, and
D.
Silver
, “
Rainbow: Combining improvements in deep reinforcement learning
,” in
Thirty-Second AAAI Conference on Artificial Intelligence
,
2018
.
31.
M.
Volodymyr
,
K.
Koray
,
D.
Silver
,
A.
Graves
,
A.
Ioannis
,
W.
Daan
, and
R.
Martin
, “
Playing Atari with deep reinforcement learning
,” in
NIPS Deep Learning Workshop
,
2013
.
32.
J.
Li
,
W.
Monroe
,
A.
Ritter
,
D.
Jurafsky
,
M.
Galley
, and
J.
Gao
, “
Deep reinforcement learning for dialogue generation
,” in
Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
(
Association for Computational Linguistics
,
2016
), pp.
1192
1202
.
33.
S.
Gu
,
E.
Holly
,
T.
Lillicrap
, and
S.
Levine
, “
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
,” in
IEEE International Conference on Robotics and Automation (ICRA)
(
IEEE
,
2017
), pp.
3389
3396
.
34.
J.
Rabault
,
J.
Kolaas
, and
A.
Jensen
, “
Performing particle image velocimetry using artificial neural networks: A proof-of-concept
,”
Meas. Sci. Technol.
28
(
12
),
125301
(
2017
).
35.
M. A.
Mendez
,
M. T.
Scelzo
, and
J.-M.
Buchlin
, “
Multiscale modal analysis of an oscillating impinging gas jet
,”
Exp. Therm. Fluid Sci.
91
,
256
276
(
2018
).
36.
M. A.
Mendez
,
A.
Gosset
, and
J.-M.
Buchlin
, “
Experimental analysis of the stability of the jet wiping process, part II: Multiscale modal analysis of the gas jet-liquid film interaction
,”
Exp. Therm. Fluid Sci.
106
,
48
67
(
2019
).
37.
M. A.
Mendez
,
M.
Balabane
, and
J.-M.
Buchlin
, “
Multi-scale proper orthogonal decomposition of complex fluid flows
,”
J. Fluid Mech.
870
,
988
1036
(
2019
).
38.
M.
Tatar
and
M. H.
Sabour
, “
Reduced-order modeling of dynamic stall using neuro-fuzzy inference system and orthogonal functions
,”
Phys. Fluids
32
(
4
),
045101
(
2020
).
39.
P. A.
Srinivasan
,
L.
Guastoni
,
H.
Azizpour
,
P.
Schlatter
, and
R.
Vinuesa
, “
Predictions of turbulent shear flows using deep neural networks
,”
Phys. Rev. Fluids
4
,
054603
(
2019
).
40.
V.
Sekar
,
B. C.
Khoo
,
S.
Chang
, and
B.
Cheong Khoo
, “
Fast flow field prediction over airfoils using deep learning approach
,”
Phys. Fluids
31
(
5
),
057103
(
2019
).
41.
S. Z.
Islami rad
,
R.
Gholipour Peyvandi
, and
S.
Sadrzadeh
, “
Determination of the volume fraction in (water-gasoil-air) multiphase flows using a simple and low-cost technique: Artificial neural networks
,”
Phys. Fluids
31
(
9
),
093301
(
2019
).
42.
G.
Novati
,
L.
Mahadevan
, and
P.
Koumoutsakos
, “
Controlled gliding and perching through deep-reinforcement-learning
,”
Phys. Rev. Fluids
4
,
093902
(
2019
).
43.
M.
Gazzola
,
B.
Hejazialhosseini
, and
P.
Koumoutsakos
, “
Reinforcement learning and wavelet adapted vortex methods for simulations of self-propelled swimmers
,”
SIAM J. Sci. Comput.
36
(
3
),
B622
B639
(
2014
).
44.
M.
Gazzola
,
A. A.
Tchieu
,
D.
Alexeev
,
A.
de Brauer
, and
P.
Koumoutsakos
, “
Learning to school in the presence of hydrodynamic interactions
,”
J. Fluid Mech.
789
,
726
749
(
2016
).
45.
S.
Verma
,
G.
Novati
, and
P.
Koumoutsakos
, “
Efficient collective swimming by harnessing vortices through deep reinforcement learning
,”
Proc. Natl. Acad. Sci. U. S. A.
115
(
23
),
5849
5854
(
2018
).
46.
G.
Reddy
,
J.
Wong-Ng
,
A.
Celani
,
T. J.
Sejnowski
, and
M.
Vergassola
, “
Glider soaring via reinforcement learning in the field
,”
Nature
562
(
7726
),
236
(
2018
).
47.
S.
Colabrese
,
K.
Gustavsson
,
A.
Celani
, and
L.
Biferale
, “
Flow navigation by smart microswimmers via reinforcement learning
,”
Phys. Rev. Lett.
118
,
158004
(
2017
).
48.
J.
Rabault
,
M.
Kuchta
,
A.
Jensen
,
U.
Réglade
, and
N.
Cerardi
, “
Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
,”
J. Fluid Mech.
865
,
281
302
(
2019
).
49.
J.
Rabault
and
A.
Kuhnle
, “
Accelerating deep reinforcement learning strategies of flow control through a multi-environment approach
,”
Phys. Fluids
31
(
9
),
094105
(
2019
).
50.
M.
Schäfer
,
S.
Turek
,
F.
Durst
,
E.
Krause
, and
R.
Rannacher
, “
Benchmark computations of laminar flow around a cylinder
,” in
Flow Simulation with High-Performance Computers II: DFG Priority Research Programme Results 1993–1995
(
Vieweg Teubner Verlag
,
Wiesbaden
,
1996
), pp.
547
566
.
51.
K.
Goda
, “
A multistep technique with implicit difference schemes for calculating two- or three-dimensional cavity flows
,”
J. Comput. Phys.
30
(
1
),
76
95
(
1979
).
52.
L.
Anders
,
K.-A.
Mardal
, and
G.
Wells
,
Automated Solution of Differential Equations by the Finite Element Method: The FEniCS Book
(
Springer Science & Business Media
,
2012
), Vol. 84.
53.
T. A.
Davis
and
I. S.
Duff
, “
An unsymmetric-pattern multifrontal method for sparse LU factorization
,”
SIAM J. Matrix Anal. Appl.
18
(
1
),
140
158
(
1997
).
54.
J.
Nathan Kutz
, “
Deep learning in fluid dynamics
,”
J. Fluid Mech.
814
,
1
4
(
2017
).
55.
N.
Thuerey
,
K.
Weißenow
,
L.
Prantl
, and
X.
Hu
, “
Deep learning methods for Reynolds-averaged Navier–Stokes simulations of airfoil flows
,”
AIAA J.
58
,
25
36
(
2019
).
56.
A.
Beck
,
D.
Flad
, and
C.-D.
Munz
, “
Deep neural networks for data-driven LES closure models
,”
J. Comput. Phys.
398
,
108910
(
2019
).
57.
J.
Sirignano
and
K.
Spiliopoulos
, “
DGM: A deep learning algorithm for solving partial differential equations
,”
J. Comput. Phys.
375
,
1339
(
2018
).
58.
M.
Raissi
,
P.
Perdikaris
, and
G. E.
Karniadakis
, “
Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations
,”
J. Comput. Phys.
378
,
686
707
(
2019
).
59.
X.
Yan
,
J.
Zhu
,
M.
Kuang
, and
X.
Wang
, “
Aerodynamic shape optimization using a novel optimizer based on machine learning techniques
,”
Aerosp. Sci. Technol.
86
,
826
835
(
2019
).
60.
K.
Yonekura
and
H.
Hattori
, “
Framework for design optimization using deep reinforcement learning
,”
Struct. Multidiscip. Optim.
60
,
1709
1713
(
2019
).
61.
D.
Silver
,
A.
Huang
,
C. J.
Maddison
,
A.
Guez
,
L.
Sifre
,
G.
van den Driessche
,
J.
Schrittwieser
,
I.
Antonoglou
,
V.
Panneershelvam
,
M.
Lanctot
,
S.
Dieleman
,
D.
Grewe
,
J.
Nham
,
N.
Kalchbrenner
,
I.
Sutskever
,
T.
Lillicrap
,
M.
Leach
,
K.
Kavukcuoglu
,
T.
Graepel
, and
D.
Hassabis
, “
Mastering the game of Go with deep neural networks and tree search
,”
Nature
529
(
7587
),
484
489
(
2016
).
62.
E.
Bøhn
,
E. M.
Coates
,
S.
Moe
, and
T. A.
Johansen
, “
Deep reinforcement learning attitude control of fixed-wing UAVs using proximal policy optimization
,” in
International Conference on Unmanned Aircraft Systems (ICUAS)
(
ICUAS
,
2019
), pp.
523
533
.
63.
R. S.
Sutton
and
A. G.
Barto
,
Reinforcement Learning: An Introduction
(
MIT Press
,
2018
).
64.
B.
Recht
, “
A tour of reinforcement learning: The view from continuous control
,”
Annu. Rev. Control, Rob., Auton. Syst.
2
(
1
),
253
279
(
2019
).
65.
A.
Braylan
,
M.
Hollenbeck
,
E.
Meyerson
, and
R.
Miikkulainen
, “
Frame skip is a powerful parameter for learning to play Atari
,” in
AAAI-15 Workshop on Learning for General Competency in Video Games
,
2015
.
66.
A.
Neitz
,
G.
Parascandolo
,
S.
Bauer
, and
B.
Schölkopf
, “
Adaptive skip intervals: Temporal abstraction for recurrent dynamical models
,” in
Advances in Neural Information Processing Systems
(
NeurIPS
,
2018
), pp.
9816
9826
.
67.
B.
Protas
and
J. E.
Wesfreid
, “
Drag force in the open-loop control of the cylinder wake in the laminar regime
,”
Phys. Fluids
14
(
2
),
810
826
(
2002
).
68.
M. E.
Taylor
and
P.
Stone
, “
Transfer learning for reinforcement learning domains: A survey
,”
J. Mach. Learn. Res.
10
,
1633
1685
(
2009
).
69.
H.
Xu
,
S. M.
Mughal
,
E. R.
Gowree
,
C. J.
Atkin
, and
S. J.
Sherwin
, “
Destabilisation and modification of Tollmien-Schlichting disturbances by a three-dimensional surface indentation
,”
J. Fluid Mech.
819
,
592
620
(
2017
).
70.
H.
Xu
,
J.-E. W.
Lombard
, and
S. J.
Sherwin
, “
Influence of localised smooth steps on the instability of a boundary layer
,”
J. Fluid Mech.
817
,
138
170
(
2017
).
71.
V.
Belus
,
J.
Rabault
,
J.
Viquerat
,
Z.
Che
,
E.
Hachem
, and
U.
Reglade
, “
Exploiting locality and translational invariance to design effective deep reinforcement learning control of the 1-dimensional unstable falling liquid film
,”
AIP Adv.
9
(
12
),
125014
(
2019
).
72.
B.
Anderson
,
H.
Truong Son
, and
R.
Kondor
, “
Cormorant: Covariant molecular neural networks
,” in
Advances in Neural Information Processing Systems
(
NeurIPS
,
2019
), pp.
14510
14519
.
73.
S.
Pawar
,
S. E.
Ahmed
,
O.
San
, and
A.
Rasheed
, “
Data-driven recovery of hidden physics in reduced order modeling of fluid flows
,”
Phys. Fluids
32
(
3
),
036602
(
2020
).
74.
M.
Raissi
,
A.
Yazdani
, and
G. E.
Karniadakis
, “
Hidden fluid mechanics: Learning velocity and pressure fields from flow visualizations
,”
Science
367
(
6481
),
1026
1030
(
2020
).
75.
A.
Kuhnle
,
M.
Schaarschmidt
, and
K.
Fricke
, Tensorforce: A tensorflow library for applied reinforcement learning, https://github.com/tensorforce/tensorforce.
You do not currently have access to this content.