Deep reinforcement learning (DRL) for fluidic pinball, three individually rotating cylinders in the uniform flow arranged in an equilaterally triangular configuration, can learn the efficient flow control strategies due to the validity of self-learning and data-driven state estimation for complex fluid dynamic problems. In this work, we present a DRL-based real-time feedback strategy to control the hydrodynamic force on fluidic pinball, i.e., force extremum and tracking, from cylinders' rotation. By adequately designing reward functions and encoding historical observations, and after automatic learning of thousands of iterations, the DRL-based control was shown to make reasonable and valid control decisions in nonparametric control parameter space, which is comparable to and even better than the optimal policy found through lengthy brute-force searching. Subsequently, one of these results was analyzed by a machine learning model that enabled us to shed light on the basis of decision-making and physical mechanisms of the force tracking process. The finding from this work can control hydrodynamic force on the operation of fluidic pinball system and potentially pave the way for exploring efficient active flow control strategies in other complex fluid dynamic problems.
REFERENCES
1.
R.
Kerswell
, “
Nonlinear nonmodal stability theory
,” Annu. Rev. Fluid Mech.
50
, 319
–345
(2018
).2.
C. W.
Rowley
and
S. T.
Dawson
, “
Model reduction for flow analysis and control
,” Annu. Rev. Fluid Mech.
49
, 387
–417
(2017
).3.
P.
Ashill
,
J.
Fulker
, and
K.
Hackett
, “
A review of recent developments in flow control
,” Aeronaut. J.
109
, 205
–232
(2005
).4.
H.
Choi
,
W.-P.
Jeon
, and
J.
Kim
, “
Control of flow over a bluff body
,” Annu. Rev. Fluid Mech.
40
, 113
–139
(2008
).5.
D.
Xu
and
M.
Zhang
, “
Reinforcement-learning-based control of convectively unstable flows
,” J. Fluid Mech.
954
, A37
(2023
).6.
D.
Greenblatt
and
D. R.
Williams
, “
Flow control for unmanned air vehicles
,” Annu. Rev. Fluid Mech.
54
, 383
–412
(2022
).7.
M. R.
Jovanović
, “
From bypass transition to flow control and data-driven turbulence modeling: An input–output viewpoint
,” Annu. Rev. Fluid Mech.
53
, 311
–345
(2021
).8.
C.-A.
Yeh
,
M. G.
Meena
, and
K.
Taira
, “
Network broadcast analysis and control of turbulent flows
,” J. Fluid Mech.
910
, A15
(2021
).9.
C. S.
Greco
,
G.
Paolillo
,
T.
Astarita
, and
G.
Cardone
, “
The von Kármán street behind a circular cylinder: Flow control through synthetic jet placed at the rear stagnation point
,” J. Fluid Mech.
901
, A39
(2020
).10.
E. J.
Gutmark
and
F. F.
Grinstein
, “
Flow control with noncircular jets
,” Annu. Rev. Fluid Mech.
31
, 239
–272
(1999
).11.
J.
Kim
and
T. R.
Bewley
, “
A linear systems approach to flow control
,” Annu. Rev. Fluid Mech.
39
, 383
–417
(2007
).12.
L. N.
Cattafesta
III
and
M.
Sheplak
, “
Actuators for active flow control
,” Annu. Rev. Fluid Mech.
43
, 247
–272
(2011
).13.
T.
Shaqarin
,
B. R.
Noack
, and
M.
Morzyński
, “
The need for prediction in feedback control of a mixing layer
,” Fluid Dyn. Res.
50
, 065514
(2018
).14.
U. K.
Kaul
, “
An active flow control approach for spatially growing mixing layer
,” J. Fluids Eng.
144
, 061110
(2022
).15.
M.
Linkmann
,
F.
Knierim
,
S.
Zammert
, and
B.
Eckhardt
, “
Linear feedback control of invariant solutions in channel flow
,” J. Fluid Mech.
900
, A10
(2020
).16.
O. M.
Aamo
and
M.
Krstic
, Flow Control by Feedback: Stabilization and Mixing
(
Springer Science & Business Media
, 2003
).17.
R.
Ishar
,
E.
Kaiser
,
M.
Morzyński
,
D.
Fernex
,
R.
Semaan
,
M.
Albers
,
P. S.
Meysonnat
,
W.
Schröder
, and
B. R.
Noack
, “
Metric for attractor overlap
,” J. Fluid Mech.
874
, 720
–755
(2019
).18.
N.
Deng
,
B. R.
Noack
,
M.
Morzyński
, and
L. R.
Pastur
, “
Low-order model for successive bifurcations of the fluidic pinball
,” J. Fluid Mech.
884
, A37
(2020
).19.
S.
Li
,
W.
Li
, and
B. R.
Noack
, “
Machine-learned control-oriented flow estimation for multiactuator multi-sensor systems exemplified for the fluidic pinball
,” J. Fluid Mech.
952, A36
(2022
).20.
N.
Deng
,
L. R.
Pastur
,
M.
Morzyński
, and
B. R.
Noack
, “
Route to chaos in the fluidic pinball
,” in Fluids Engineering Division Summer Meeting
(
American Society of Mechanical Engineers
, 2018
), Vol.
51555
, p. V001T01A005
.21.
L. R.
Pastur
,
N.
Deng
,
M.
Morzyński
, and
B. R.
Noack
, “
Reduced-order modeling of the fluidic pinball
,” in Chaotic Modeling and Simulation International Conference
(
Springer
, 2018
), pp. 205
–213
.22.
G. Y. C.
Maceda
,
B. R.
Noack
,
F.
Lusseyran
,
N.
Deng
,
L.
Pastur
, and
M.
Morzynski
, “
Artificial intelligence control applied to drag reduction of the fluidic pinball
,” in Proceedings of Applied Mathematics and Mechanics (PAMM)
(Wiley Online Library, 2019
), Vol.
19
, p. e201900268
.23.
S.
Peitz
,
S. E.
Otto
, and
C. W.
Rowley
, “
Data-driven model predictive control using interpolated Koopman generators
,” SIAM J. Appl. Dyn. Syst.
19
, 2162
–2193
(2020
).24.
C.
Raibaudo
,
P.
Zhong
,
B. R.
Noack
, and
R. J.
Martinuzzi
, “
Machine learning strategies applied to the control of a fluidic pinball
,” Phys. Fluids
32
, 015108
(2020
).25.
C.
Raibaudo
and
R.
Martinuzzi
, “
Unsteady actuation and feedback control of the experimental fluidic pinball using genetic programming
,” Exp. Fluids
62
, 219
(2021
).26.
A. B.
Blanchard
,
G. Y. C.
Maceda
,
D.
Fan
,
Y.
Li
,
Y.
Zhou
,
B. R.
Noack
, and
T. P.
Sapsis
, “
Bayesian optimization for active flow control
,” Acta Mech. Sin.
37
, 1786
–1798
(2021
).27.
H.
Ghraieb
,
J.
Viquerat
,
A.
Larcher
,
P.
Meliga
, and
E.
Hachem
, “
Single-step deep reinforcement learning for open-loop control of laminar and turbulent flows
,” Phys. Rev. Fluids
6
, 053902
(2021
).28.
G. Y. C.
Maceda
,
Y.
Li
,
F.
Lusseyran
,
M.
Morzyński
, and
B. R.
Noack
, “
Stabilization of the fluidic pinball with gradient-enriched machine learning control
,” J. Fluid Mech.
917
, A42
(2021
).29.
Y.
Li
,
W.
Cui
,
Q.
Jia
,
Q.
Li
,
Z.
Yang
,
M.
Morzyński
, and
B. R.
Noack
, “
Explorative gradient method for active drag reduction of the fluidic pinball and slanted Ahmed body
,” J. Fluid Mech.
932
, A7
(2022
).30.
M.
Brenner
,
J.
Eldredge
, and
J.
Freund
, “
Perspective on machine learning for advancing fluid mechanics
,” Phys. Rev. Fluids
4
, 100501
(2019
).31.
S. L.
Brunton
,
B. R.
Noack
, and
P.
Koumoutsakos
, “
Machine learning for fluid mechanics
,” Annu. Rev. Fluid Mech.
52
, 477
–508
(2020
).32.
S. L.
Brunton
, “
Applying machine learning to study fluid mechanics
,” Acta Mech. Sin.
37
, 1718
–1726
(2022
).33.
S.
Lee
and
D.
You
, “
Data-driven prediction of unsteady flow over a circular cylinder using deep learning
,” J. Fluid Mech.
879
, 217
–254
(2019
).34.
E.
Kharazmi
,
D.
Fan
,
Z.
Wang
, and
M. S.
Triantafyllou
, “
Inferring vortex induced vibrations of flexible cylinders using physics-informed neural networks
,” J. Fluids Struct.
107
, 103367
(2021
).35.
R. S.
Sutton
and
A. G.
Barto
, Reinforcement Learning: An Introduction
(
MIT Press
, 2018
).36.
R. S.
Sutton
,
D.
Precup
, and
S.
Singh
, “
Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
,” Artif. Intell.
112
, 181
–211
(1999
).37.
D.
Silver
,
T.
Hubert
,
J.
Schrittwieser
,
I.
Antonoglou
,
M.
Lai
,
A.
Guez
,
M.
Lanctot
,
L.
Sifre
,
D.
Kumaran
,
T.
Graepel
et al, “
A general reinforcement learning algorithm that masters chess, shogi, and go through self-play
,” Science
362
, 1140
–1144
(2018
).38.
S.
Aradi
, “
Survey of deep reinforcement learning for motion planning of autonomous vehicles
,” IEEE Trans. Intelligent Transp. Syst.
23
(3
), 740
–759
(2020
).39.
J.
Wang
,
J.
Cao
,
S.
Wang
,
Z.
Yao
, and
W.
Li
, “
IRDA: Incremental reinforcement learning for dynamic resource allocation
,” IEEE Trans. Big Data
8
, 770
–783
(2020
).40.
B.
Singh
,
R.
Kumar
, and
V. P.
Singh
, “
Reinforcement learning in robotic applications: A comprehensive survey
,” Artif. Intell. Rev.
55
, 945
–990
(2022
).41.
H.
Feng
,
L.
Yu
, and
Y.
Chen
, “
Optimal control and reinforcement learning for robot: A survey
,” in International Conference on Collaborative Computing: Networking, Applications and Worksharing
(
Springer
, 2021
), pp. 54
–66
.42.
W.-L.
Chen
,
Y.
Huang
,
C.
Chen
,
H.
Yu
, and
D.
Gao
, “
Review of active control of circular cylinder flow
,” Ocean Eng.
258
, 111840
(2022
).43.
J.
Rabault
,
F.
Ren
,
W.
Zhang
,
H.
Tang
, and
H.
Xu
, “
Deep reinforcement learning in fluid mechanics: A promising method for both active flow control and shape optimization
,” J. Hydrodyn.
32
, 234
–246
(2020
).44.
J.
Viquerat
,
P.
Meliga
,
A.
Larcher
, and
E.
Hachem
, “
A review on deep reinforcement learning for fluid mechanics: An update
,” Phys. Fluids
34
, 111301
(2022
).45.
J.
Rabault
,
M.
Kuchta
,
A.
Jensen
,
U.
Réglade
, and
N.
Cerardi
, “
Artificial neural networks trained through deep reinforcement learning discover control strategies for active flow control
,” J. Fluid Mech.
865
, 281
–302
(2019
).46.
H.
Xu
,
W.
Zhang
,
J.
Deng
, and
J.
Rabault
, “
Active flow control with rotating cylinders by an artificial neural network trained by deep reinforcement learning
,” J. Hydrodyn.
32
, 254
–258
(2020
).47.
D.
Fan
,
L.
Yang
,
Z.
Wang
,
M. S.
Triantafyllou
, and
G. E.
Karniadakis
, “
Reinforcement learning for bluff body active flow control in experiments and simulations
,” Proc. Natl. Acad. Sci. U.S.A.
117
, 26091
–26098
(2020
).48.
C.
Zheng
,
T.
Ji
,
F.
Xie
,
X.
Zhang
,
H.
Zheng
, and
Y.
Zheng
, “
From active learning to deep reinforcement learning: Intelligent active flow control in suppressing vortex-induced vibration
,” Phys. Fluids
33
, 063607
(2021
).49.
Q.
Wang
,
L.
Yan
,
G.
Hu
,
C.
Li
,
Y.
Xiao
,
H.
Xiong
,
J.
Rabault
, and
B. R.
Noack
, “
DRLinfluids—An open-source python platform of coupling deep reinforcement learning and openfoam
,” Phys. Fluids
34
, 081801
(2022
).50.
B.-Z.
Han
,
W.-X.
Huang
, and
C.-X.
Xu
, “
Deep reinforcement learning for active control of flow over a circular cylinder with rotational oscillations
,” Int. J. Heat Fluid Flow
96
, 109008
(2022
).51.
E.
Amico
,
G.
Cafiero
, and
G.
Iuso
, “
Deep reinforcement learning for active control of a three-dimensional bluff body wake
,” Phys. Fluids
34
, 105126
(2022
).52.
Y.-F.
Mei
,
C.
Zheng
,
Y.
Hua
,
Q.
Zhao
,
P.
Wu
, and
W.-T.
Wu
, “
Active control for the flow around various geometries through deep reinforcement learning
,” Fluid Dyn. Res.
54
, 015510
(2022
).53.
Y.-Z.
Wang
,
Y.
Hua
,
N.
Aubry
,
Z.-H.
Chen
,
W.-T.
Wu
, and
J.
Cui
, “
Accelerating and improving deep reinforcement learning-based active flow control: Transfer training of policy network
,” Phys. Fluids
34
, 073609
(2022
).54.
C.
Zheng
,
F.
Xie
,
T.
Ji
,
X.
Zhang
,
Y.
Lu
,
H.
Zhou
, and
Y.
Zheng
, “
Data-efficient deep reinforcement learning with expert demonstration for active flow control
,” Phys. Fluids
34
, 113603
(2022
).55.
F.
Ren
,
J.
Rabault
, and
H.
Tang
, “
Applying deep reinforcement learning to active flow control in weakly turbulent conditions
,” Phys. Fluids
33
, 037121
(2021
).56.
F.
Ren
,
C.
Wang
, and
H.
Tang
, “
Bluff body uses deep-reinforcement-learning trained active flow control to achieve hydrodynamic stealth
,” Phys. Fluids
33
, 093602
(2021
).57.
Y.
Xie
,
X.
Zhao
, and
M.
Luo
, “
An active-controlled heaving plate breakwater trained by an intelligent framework based on deep reinforcement learning
,” Ocean Eng.
244
, 110357
(2022
).58.
J.
Li
and
M.
Zhang
, “
Reinforcement-learning-based control of confined cylinder wakes with stability analyses
,” J. Fluid Mech.
932
, A44
(2022
).59.
F.
Pino
,
L.
Schena
,
J.
Rabault
, and
M. A.
Mendez
, “
Comparative analysis of machine learning methods for active flow control
,” J. Fluid Mech.
958
, A39
(2022
).60.
R.
Paris
,
S.
Beneddine
, and
J.
Dandois
, “
Robust flow control and optimal sensor placement using deep reinforcement learning
,” J. Fluid Mech.
913
, A25
(2021
).61.
P.
Varela
,
P.
Suárez
,
F.
Alcántara-Ávila
,
A.
Miró
,
J.
Rabault
,
B.
Font
,
L. M.
García-Cuevas
,
O.
Lehmkuhl
, and
R.
Vinuesa
, “
Deep reinforcement learning for flow control exploits different physics for increasing Reynolds number regimes
,” in Actuators
(
Multidisciplinary Digital Publishing Institute
, 2022
), Vol.
11
, p. 359
.62.
L.
Guastoni
,
J.
Rabault
,
P.
Schlatter
,
H.
Azizpour
, and
R.
Vinuesa
, “
Deep reinforcement learning for turbulent drag reduction in channel flows
,” arXiv:2301.09889 (2023
).63.
G. Y. C.
Maceda
,
F.
Lusseyran
, and
B. R.
Noack
, “
xMLC—A toolkit for machine learning control
,” arXiv:2208.13172 (2022
).64.
G. D.
Weymouth
and
D. K.
Yue
, “
Boundary data immersion method for Cartesian-grid simulations of fluid-body interaction problems
,” J. Comput. Phys.
230
, 6233
–6247
(2011
).65.
S. C.
Schlanderer
,
G. D.
Weymouth
, and
R. D.
Sandberg
, “
The boundary data immersion method for compressible flows with application to aeroacoustics
,” J. Comput. Phys.
333
, 440
–461
(2017
).66.
A.
Li
,
S.
Shi
, and
D.
Fan
, “
Fluid forces and vortex patterns of an oscillating cylinder pair in still water with both fixed side-by-side and tandem configurations
,” J. Offshore Mech. Arct. Eng.
144
, 021903
(2022
).67.
S.
Fujimoto
,
H.
Hoof
, and
D.
Meger
, “
Addressing function approximation error in actor-critic methods
,” in International Conference on Machine Learning
(
PMLR
, 2018
), pp. 1587
–1596
.68.
W.-Y.
Loh
, “
Classification and regression trees
,” Wiley Interdiscip. Rev.: Data Min. Knowl. Discovery
1
, 14
–23
(2011
).69.
R. H.
Alsagheer
,
A. F.
Alharan
, and
A. S.
Al-Haboobi
, “
Popular decision tree algorithms of data mining techniques: A review
,” Int. J. Comput. Sci. Mobile Comput.
6
, 133
–142
(2017
).70.
D.
Gunning
,
M.
Stefik
,
J.
Choi
,
T.
Miller
,
S.
Stumpf
, and
G.-Z.
Yang
, “
XAI—Explainable artificial intelligence
,” Sci. Robotics
4
, eaay7120
(2019
).71.
B.
Mahbooba
,
M.
Timilsina
,
R.
Sahal
, and
M.
Serrano
, “
Explainable artificial intelligence (XAI) to enhance trust management in intrusion detection systems using decision tree model
,” Complexity
2021
, 6634811
.72.
O.
Bastani
,
Y.
Pu
, and
A.
Solar-Lezama
, “
Verifiable reinforcement learning via policy extraction
,” in Proceedings of the Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems
, 2018
.73.
E.
Puiutta
and
E.
Veith
, “
Explainable reinforcement learning: A survey
,” in International Cross-Domain Conference for Machine Learning and Knowledge Extraction
(
Springer
, 2020
), pp. 77
–95
.74.
G. A.
Vouros
, “
Explainable deep reinforcement learning: State of the art and challenges
,” ACM Comput. Surv.
55
, 1
(2023
).75.
Dataset:
H. D.
Feng
, (2023
). “How-to-control-hydrodynamic-force-on-fluidic-pinball-via-deep-reinforcement-learning
,” Github, https://github.com/HDFengChina/How-to-Control-Hydrodynamic-Force.© 2023 Crown.
2023
Crown
Published under a nonexclusive license by AIP Publishing.
You do not currently have access to this content.
