Deep Reinforcement Learning (DRL) has recently been proposed as a methodology to discover complex active flow control strategies [Rabault et al., “Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control,” J. Fluid Mech. 865, 281–302 (2019)]. However, while promising results were obtained on a simple 2-dimensional benchmark flow at a moderate Reynolds number, considerable speedups will be required to investigate more challenging flow configurations. In the case of DRL trained with Computational Fluid Dynamics (CFD) data, it was found that the CFD part, rather than training the artificial neural network, was the limiting factor for speed of execution. Therefore, speedups should be obtained through a combination of two approaches. The first one, which is well documented in the literature, is to parallelize the numerical simulation itself. The second one is to adapt the DRL algorithm for parallelization. Here, a simple strategy is to use several independent simulations running in parallel to collect experiences faster. In the present work, we discuss this solution for parallelization. We illustrate that perfect speedups can be obtained up to the batch size of the DRL agent, and slightly suboptimal scaling still takes place for an even larger number of simulations. This is, therefore, an important step toward enabling the study of more sophisticated fluid mechanics problems through DRL.

1.
M.
Pastoor
,
L.
Henning
,
B. R.
Noack
,
R.
King
, and
G.
Tadmor
, “
Feedback shear layer control for bluff body drag reduction
,”
J. Fluid Mech.
608
,
161
196
(
2008
).
2.
D.
You
and
P.
Moin
, “
Active control of flow separation over an airfoil using synthetic jets
,”
J. Fluids Struct.
24
,
1349
1357
(
2008
), unsteady separated flows and their control.
3.
R.
Li
,
J.
Borée
,
B. R.
Noack
,
L.
Cordier
, and
F.
Harambat
, “
Drag reduction mechanisms of a car model at moderate yaw by bi-frequency forcing
,”
Phys. Rev. Fluids
4
,
034604
(
2019
).
4.
Y.
Li
,
W.
Cui
,
Q.
Jia
,
Q.
Li
,
Z.
Yang
, and
B. R.
Noack
, “
Optimization of active drag reduction for a slanted Ahmed body in a high-dimensional parameter space
,” preprint arXiv:1905.12036 (
2019
).
5.
Z.
Wu
,
D.
Fan
,
Y.
Zhou
,
R.
Li
, and
B. R.
Noack
, “
Jet mixing optimization using machine learning control
,”
Exp. Fluids
59
,
131
(
2018
).
6.
S. L.
Brunton
and
B. R.
Noack
, “
Closed-loop turbulence control: Progress and challenges
,”
Appl. Mech. Rev.
67
,
050801
(
2015
).
7.
T.
Duriez
,
S. L.
Brunton
, and
B. R.
Noack
,
Machine Learning Control-Taming Nonlinear Dynamics and Turbulence
(
Springer
,
2016
).
8.
M.
Bergmann
,
L.
Cordier
, and
J.-P.
Brancher
, “
Optimal rotary control of the cylinder wake using proper orthogonal decomposition reduced-order model
,”
Phys. Fluids
17
,
097101
(
2005
).
9.
T. L. B.
Flinois
and
T.
Colonius
, “
Optimal control of circular cylinder wakes using long control horizons
,”
Phys. Fluids
27
,
087105
(
2015
).
10.
K. H.
Lee
,
L.
Cortelezzi
,
J.
Kim
, and
J.
Speyer
, “
Application of reduced-order controller to turbulent flows for drag reduction
,”
Phys. Fluids
13
,
1321
1330
(
2001
).
11.
M.
Queguineur
,
L. Y. M.
Gicquel
,
F.
Dupuy
,
A.
Misdariis
, and
G.
Staffelbach
, “
Dynamic mode tracking and control with a relaxation method
,”
Phys. Fluids
31
,
034101
(
2019
).
12.
M.
Sato
,
T.
Nonomura
,
K.
Okada
,
K.
Asada
,
H.
Aono
,
A.
Yakeno
,
Y.
Abe
, and
K.
Fujii
, “
Mechanisms for laminar separated-flow control using dielectric-barrier-discharge plasma actuator at low Reynolds number
,”
Phys. Fluids
27
,
117101
(
2015
).
13.
C.
Wang
,
H.
Tang
,
S. C. M.
Yu
, and
F.
Duan
, “
Active control of vortex-induced vibrations of a circular cylinder using windward-suction-leeward-blowing actuation
,”
Phys. Fluids
28
,
053601
(
2016
).
14.
F.
Li
and
N.
Aubry
, “
Feedback control of a flow past a cylinder via transverse motion
,”
Phys. Fluids
15
,
2163
2176
(
2003
).
15.
L.
Lu
,
J.-M.
Qin
,
B.
Teng
, and
Y.-C.
Li
, “
Numerical investigations of lift suppression by feedback rotary oscillation of circular cylinder at low Reynolds number
,”
Phys. Fluids
23
,
033601
(
2011
).
16.
Y.
Bao
and
J.
Tao
, “
Active control of a cylinder wake flow by using a streamwise oscillating foil
,”
Phys. Fluids
25
,
053601
(
2013
).
17.
H.
Zhu
,
T.
Tang
,
H.
Zhao
, and
Y.
Gao
, “
Control of vortex-induced vibration of a circular cylinder using a pair of air jets at low Reynolds number
,”
Phys. Fluids
31
,
043603
(
2019
).
18.
K. S.
Breuer
,
J.
Park
, and
C.
Henoch
, “
Actuation and control of a turbulent channel flow using Lorentz forces
,”
Phys. Fluids
16
,
897
907
(
2004
).
19.
B. A.
Belson
,
O.
Semeraro
,
C. W.
Rowley
, and
D. S.
Henningson
, “
Feedback control of instabilities in the two-dimensional Blasius boundary layer: The role of sensors and actuators
,”
Phys. Fluids
25
,
054106
(
2013
).
20.
B.
Che
,
N.
Chu
,
L.
Cao
,
S. J.
Schmidt
,
D.
Likhachev
, and
D.
Wu
, “
Control effect of micro vortex generators on attached cavitation instability
,”
Phys. Fluids
31
,
064102
(
2019
).
21.
X.
Meng
,
Y.
Long
,
J.
Wang
,
F.
Liu
, and
S.
Luo
, “
Dynamics and control of the vortex flow behind a slender conical forebody by a pair of plasma actuators
,”
Phys. Fluids
30
,
024101
(
2018
).
22.
T. N.
Jukes
and
K.-S.
Choi
, “
Control of unsteady flow separation over a circular cylinder using dielectric-barrier-discharge surface plasma
,”
Phys. Fluids
21
,
094106
(
2009
).
23.
T. N.
Jukes
and
K.-S.
Choi
, “
Flow control around a circular cylinder using pulsed dielectric barrier discharge surface plasma
,”
Phys. Fluids
21
,
084103
(
2009
).
24.
P.
Meliga
,
D.
Sipp
, and
J.-M.
Chomaz
, “
Open-loop control of compressible afterbody flows using adjoint methods
,”
Phys. Fluids
22
,
054109
(
2010
).
25.
A. F.
Shahrabi
, “
The control of flow separation: Study of optimal open loop parameters
,”
Phys. Fluids
31
,
035104
(
2019
).
26.
F.
Jelinek
,
Statistical Methods for Speech Recognition
(
MIT Press
,
1997
).
27.
M. J.
Tarr
and
H. H.
Bülthoff
, “
Image-based object recognition in man, monkey and machine
,”
Cognition
67
,
1
20
(
1998
).
28.
A.
Graves
,
A.-R.
Mohamed
, and
G.
Hinton
, “
Speech recognition with deep recurrent neural networks
,” in
IEEE International Conference on Acoustics, Speech and Signal Processing
(
IEEE
,
2013
), pp.
6645
6649
.
29.
A.
Krizhevsky
,
I.
Sutskever
, and
G. E.
Hinton
, “
Imagenet classification with deep convolutional neural networks
,” in
Advances in Neural Information Processing Systems
(
ACM
,
2012
), pp.
1097
1105
.
30.
D.
Silver
,
J.
Schrittwieser
,
K.
Simonyan
,
I.
Antonoglou
,
A.
Huang
,
A.
Guez
,
T.
Hubert
,
L.
Baker
,
M.
Lai
,
A.
Bolton
 et al, “
Mastering the game of Go without human knowledge
,”
Nature
550
,
354
(
2017
).
31.
N.
Brown
and
T.
Sandholm
, “
Superhuman AI for multiplayer poker
,”
Science
365
,
885
(
2019
).
32.
N.
Gautier
,
J.-L.
Aider
,
T.
Duriez
,
B.
Noack
,
M.
Segond
, and
M.
Abel
, “
Closed-loop separation control using machine learning
,”
J. Fluid Mech.
770
,
442
457
(
2015
).
33.
S.
Verma
,
G.
Novati
, and
P.
Koumoutsakos
, “
Efficient collective swimming by harnessing vortices through deep reinforcement learning
,”
Proc. Natl. Acad. Sci. U. S. A.
115
,
5849
(
2018
).
34.
J.
Rabault
,
U.
Reglade
,
N.
Cerardi
,
M.
Kuchta
, and
A.
Jensen
, “
Deep reinforcement learning achieves flow control of the 2D Karman vortex street
,” preprint arXiv:1808.10754 (
2018
).
35.
J.
Rabault
,
M.
Kuchta
,
A.
Jensen
,
U.
Réglade
, and
N.
Cerardi
, “
Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
,”
J. Fluid Mech.
865
,
281
302
(
2019
).
36.
M.
Schäfer
,
S.
Turek
,
F.
Durst
,
E.
Krause
, and
R.
Rannacher
, “
Benchmark computations of laminar flow around a cylinder
,” in
Flow Simulation with High-Performance Computers II: DFG Priority Research Programme Results 1993–1995
, edited by
E. H.
Hirschel
(
Vieweg+Teubner Verlag
,
Wiesbaden
,
1996
), pp.
547
566
.
37.
K.
Valen-Sendstad
,
A.
Logg
,
K.-A.
Mardal
,
H.
Narayanan
, and
M.
Mortensen
, “
A comparison of finite element schemes for the incompressible Navier–Stokes equations
,” in
Automated Solution of Differential Equations by the Finite Element Method
(
Springer
,
2012
), pp.
399
420
.
38.
K.
Goda
, “
A multistep technique with implicit difference schemes for calculating two- or three-dimensional cavity flows
,”
J. Comput. Phys.
30
,
76
95
(
1979
).
39.
A.
Logg
,
K.-A.
Mardal
, and
G.
Wells
,
Automated Solution of Differential Equations by the Finite Element Method: The FEniCS Book
(
Springer Science & Business Media
,
2012
), Vol. 84.
40.
Y.
LeCun
,
Y.
Bengio
, and
G.
Hinton
, “
Deep learning
,”
Nature
521
,
436
(
2015
).
41.
S.
Gu
,
E.
Holly
,
T.
Lillicrap
, and
S.
Levine
, “
Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates
,” in
IEEE International Conference on Robotics and Automation (ICRA)
(
IEEE
,
2017
), pp.
3389
3396
.
42.
J. N.
Kutz
, “
Deep learning in fluid dynamics
,”
J. Fluid Mech.
814
,
1
4
(
2017
).
43.
S.
Brunton
,
B.
Noack
, and
P.
Koumoutsakos
, “
Machine learning for fluid mechanics
,” preprint arXiv:1905.11075 (
2019
).
44.
J.
Rabault
,
J.
Kolaas
, and
A.
Jensen
, “
Performing particle image velocimetry using artificial neural networks: A proof-of-concept
,”
Meas. Sci. Technol.
28
,
125301
(
2017
).
45.
P.
Srinivasan
,
L.
Guastoni
,
H.
Azizpour
,
P.
Schlatter
, and
R.
Vinuesa
, “
Predictions of turbulent shear flows using deep neural networks
,”
Phys. Rev. Fluids
4
,
054603
(
2019
).
46.
M. A.
Bucci
,
O.
Semeraro
,
A.
Allauzen
,
G.
Wisniewski
,
L.
Cordier
, and
L.
Mathelin
, “
Control of chaotic systems by deep reinforcement learning
,” preprint arXiv:1906.07672 (
2019
).
47.
J.
Schulman
,
F.
Wolski
,
P.
Dhariwal
,
A.
Radford
, and
O.
Klimov
, “
Proximal policy optimization algorithms
,” preprint arXiv:1707.06347 (
2017
).
48.
A.
Braylan
,
M.
Hollenbeck
,
E.
Meyerson
, and
R.
Miikkulainen
, “
Frame skip is a powerful parameter for learning to play Atari
,” in
Workshops at the Twenty-Ninth AAAI Conference on Artificial Intelligence
,
2015
.
49.
A.
Neitz
,
G.
Parascandolo
,
S.
Bauer
, and
B.
Schölkopf
, “
Adaptive skip intervals: Temporal abstraction for recurrent dynamical models
,” in
Advances in Neural Information Processing Systems
(
ACM
,
2018
), pp.
9816
9826
.
50.
H. D.
Simon
, “
Parallel computational fluid dynamics-implementations and results
,” NASA STI/Recon Technical Report A 94,
1992
.
51.
W. D.
Gropp
and
E. B.
Smith
, “
Computational fluid dynamics on parallel processors
,”
Comput. Fluids
18
,
289
304
(
1990
).
52.
W. D.
Gropp
,
D. K.
Kaushik
,
D. E.
Keyes
, and
B. F.
Smith
, “
High-performance parallel implicit CFD
,”
Parallel Comput.
27
,
337
362
(
2001
), parallel computing in aerospace.
53.
A.
Kuhnle
,
M.
Schaarschmidt
, and
K.
Fricke
, “
Tensorforce: A tensorflow library for applied reinforcement learning
,” Web page,
2017
.
54.
D.
Horgan
,
J.
Quan
,
D.
Budden
,
G.
Barth-Maron
,
M.
Hessel
,
H.
van Hasselt
, and
D.
Silver
, “
Distributed prioritized experience replay
,” in
6th International Conference on Learning Representations, ICLR 2018, Conference Track Proceedings
,
Vancouver, BC, Canada
,
30 April–3 May 2018
.
55.
L.
Espeholt
,
H.
Soyer
,
R.
Munos
,
K.
Simonyan
,
V.
Mnih
,
T.
Ward
,
Y.
Doron
,
V.
Firoiu
,
T.
Harley
,
I.
Dunning
,
S.
Legg
, and
K.
Kavukcuoglu
, “
IMPALA: Scalable distributed deep-RL with importance weighted actor-learner architectures
,” in
Proceedings of the 35th International Conference on Machine Learning
, Proceedings of Machine Learning Research (PMLR) Vol. 80, edited by
J.
Dy
and
A.
Krause
(
Stockholmsmässan
,
Stockholm, Sweden
,
2018
), pp.
1407
1416
.
56.
B.
Recht
,
C.
Re
,
S.
Wright
, and
F.
Niu
, “
Hogwild: A lock-free approach to parallelizing stochastic gradient descent
,” in
Advances in Neural Information Processing Systems 24
, edited by
J.
Shawe-Taylor
,
R. S.
Zemel
,
P. L.
Bartlett
,
F.
Pereira
, and
K. Q.
Weinberger
(
Curran Associates, Inc.
,
2011
), pp.
693
701
.
57.
M.
Abadi
,
P.
Barham
,
J.
Chen
,
Z.
Chen
,
A.
Davis
,
J.
Dean
,
M.
Devin
,
S.
Ghemawat
,
G.
Irving
,
M.
Isard
 et al, “
Tensorflow: A system for large-scale machine learning
,” in
Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation
(
ACM
,
2016
), Vol. 16, pp.
265
283
.
You do not currently have access to this content.