This paper presents a semi-analytical method of suppressing acoustic scattering using reinforcement learning (RL) algorithms. We give a RL agent control over design parameters of a planar configuration of cylindrical scatterers in water. These design parameters control the position and radius of the scatterers. As these cylinders encounter an incident acoustic wave, the scattering pattern is described by a function called total scattering cross section (TSCS). Through evaluating the gradients of TSCS and other information about the state of the configuration, the RL agent perturbatively adjusts design parameters, considering multiple scattering between the scatterers. As each adjustment is made, the RL agent receives a reward negatively proportional to the root mean square of the TSCS across a range of wavenumbers. Through maximizing its reward per episode, the agent discovers designs with low scattering. Specifically, the double deep Q-learning network and the deep deterministic policy gradient algorithms are employed in our models. Designs discovered by the RL algorithms performed well when compared to a state-of-the-art optimization algorithm using fmincon.
References
1.
A. N.
Norris
, “Acoustic cloaking theory
,” Proc. R. Soc. A
464
, 2411
–2434
(2008
).2.
A.
Climente
, D.
Torrent
, and J.
Sánchez-Dehesa
, “Sound focusing by gradient index sonic lenses
,” Appl. Phys. Lett.
97
, 104103
(2010
).3.
P.
Packo
, A.
Norris
, and D.
Torrent
, “Metaclusters for the full control of mechanical waves
,” arXiv:2009.13376 (2020
).4.
V.
Popov
, F.
Boust
, and S. N.
Burokur
, “Beamforming with metagratings at microwave frequencies: Design procedure and experimental demonstration
,” IEEE Trans. Antennas Propag.
68
(3
), 1533
–1541
(2020
).5.
K.
Butler
, D.
Davies
, H.
Cartwright
, O.
Isayev
, and A.
Walsh
, “Machine learning for molecular and materials science
,” Nature
559
(7715
), 547
–555
(2018
).6.
D.
Elton
, Z.
Boukouvalas
, M.
Fuge
, and P.
Chung
, “Deep learning for molecular design—A review of the state of the art
,” Mol. Syst. Des. Eng.
4
(4
), 828
–849
(2019
).7.
J.
Noh
, G. H.
Gu
, S.
Kim
, and Y.
Jung
, “Machine-enabled inverse design of inorganic solid materials: Promises and challenges
,” Chem. Sci.
11
(19
), 4871
–4881
(2020
).8.
R. S.
Hegde
, “Deep learning: A new tool for photonic nanostructure design
,” Nanoscale Adv.
2
(3
), 1007
–1023
(2020
).9.
S.
So
, T.
Badloe
, J.
Noh
, J.
Rho
, and J.
Bravo-Abad
, “Deep learning enabled inverse design in nanophotonics
,” Nanophotonics
9
(5
), 1041
–1057
(2020
).10.
S.
Campbell
, D.
Sell
, E.
Jenkins
, R.
Whiting
, J.
Fan
, and D.
Werner
, “Review of numerical optimization techniques for meta-device design
,” Opt. Mater. Express
9
(4
), 1842
–1863
(2019
).11.
D.
Silver
, T.
Hubert
, J.
Schrittwieser
, I.
Antonoglou
, M.
Lai
, A.
Guez
, M.
Lanctot
, L.
Sifre
, D.
Kumaran
, T.
Graepel
, T.
Lillicrap
, K.
Simonyan
, and D.
Hassabis
, “Mastering chess and shogi by self-play with a general reinforcement learning algorithm
,” arXiv:1712.01815 (2017
).12.
V.
Mnih
, K.
Kavukcuoglu
, D.
Silver
, A.
Graves
, I.
Antonoglou
, D.
Wierstra
, and M.
Riedmiller
, “Playing Atari with deep reinforcement learning
,” arXiv:1312.5602v1 (2013
).13.
T.
Badloe
, I.
Kim
, and J.
Rho
, “Biomimetic ultra-broadband perfect absorbers optimised with reinforcement learning
,” Phys. Chem. Chem. Phys.
22
(4
), 2337
–2342
(2020
).14.
I.
Goodfellow
, Y.
Bengio
, and, and A.
Courville
, Deep Learning
(MIT
, Cambridge, MA
, 2016
).15.
16.
R.
Jenison
, “A spherical basis function neural network for approximating acoustic scatter
,” J. Acoust. Soc. Am.
99
(5
), 3242
–3245
(1996
).17.
M. M.
Morgan
, I.
Bhattacharya
, R. J.
Radke
, and J.
Braasch
, “Classifying the emotional speech content of participants in group meetings using convolutional long short-term memory network
,” J. Acoust. Soc. Am.
149
(2
), 885
–894
(2021
).18.
N.
Liu
, H.
Chen
, K.
Songgong
, and Y.
Li
, “Deep learning assisted sound source localization using two orthogonal first-order differential microphone arrays
,” J. Acoust. Soc. Am.
149
(2
), 1069
–1084
(2021
).19.
R. L.
Jenison
, “Models of direction estimation with spherical-function approximated cortical receptive fields
,” in Central Auditory Processing and Neural Modeling
, edited by P.
Poon
and J.
Brugge
(Plenum
, New York
, 1998
), pp. 161
–174
.20.
M.
Hesham
and M.
El-Gamal
, “Neural network model for solving integral equation of acoustic scattering using wavelet basis
,” Commun. Numer. Methods Eng.
24
, 183
–194
(2006
).21.
M.
Bianco
, P.
Gerstoft
, J.
Traer
, E.
Ozanich
, M. A.
Roch
, S.
Gannot
, and C.-A.
Deledalle
, “Machine learning in acoustics: Theory and applications
,” J. Acoust. Soc. Am.
146
(5
), 3590
–3628
(2019
).22.
C.
Cueto
and B.
Hadithi
, “Cancelling out skull-induced aberrations: Analysis of acoustic metamaterials using neural networks
,” IEEE Latin Am. Trans.
15
(10
), 1948
–1959
(2017
).23.
P.
Meng
, L.
Su
, W.
Yin
, and S.
Zhang
, “Solving a kind of inverse scattering problem of acoustic waves based on linear sampling method and neural network
,” Alex. Eng. J.
59
(3
), 1451
–1462
(2020
).24.
Z.
Fan
, V.
Vineet
, H.
Gamper
, and N.
Raghuvanshi
, “Fast acoustic scattering using convolutional neural networks
,” in Proceedings of ICASSP 2020—2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)
, Barcelona, Spain (May 4–8, 2020
).25.
Z.
Fan
, V.
Vineet
, C.
Lu
, T. W.
Wu
, and K.
McMullen
, “Prediction of object geometry from acoustic scattering using convolutional neural networks
,” arXiv:2010.10691 (2021
).26.
D.
Komen
, T. B.
Neilsen
, D. B.
Mortenson
, M. C.
Acree
, D. P.
Knobles
, M.
Badiey
, and W. S.
Hodgkiss
, “Seabed type and source parameters predictions using ship spectrograms in convolutional neural networks
,” J. Acoust. Soc. Am.
149
(2
), 1198
–1210
(2021
).27.
S.
Kumar
, S.
Tan
, L.
Zheng
, and D. M.
Kochmann
, “Inverse-designed spinodoid metamaterials
,” NPJ Comput. Mater.
6
(1
), 73
–83
(2020
).28.
H.
Gao
and J.
Zhu
, “Inverse design method for acoustic metamaterials
,” J. Acoust. Soc. Am.
146
(4
), 2828
–2828
(2019
).29.
D.
Finol
, Y.
Lu
, V.
Mahadevan
, and A.
Srivastava
, “Deep convolutional neural networks for eigenvalue problems in mechanics
,” Numer. Methods Eng.
118
(5
), 258
–275
(2019
).30.
D.
Pornaras
, W.-C.
Wang
, Y.
Chen
, G.
Kwak
, F.
Amirkulova
, and E.
Khatami
, “Broadband suppression of total multiple scattering cross section using neural networks
,” J. Acoust. Soc. Am.
146
(4
), 2876
–2877
(2019
).31.
L.
Wu
, L.
Liu
, Y.
Wang
, Z.
Zhai
, H.
Zhuang
, D.
Krishnaraju
, Q.
Wang
, and H.
Jiang
, “A machine learning-based method to design modular metamaterials
,” Extreme Mech. Lett.
36
, 100657
(2020
).32.
C.
Gurbuz
, F.
Kronowetter
, C.
Dietz
, M.
Eser
, J.
Schmid
, and S.
Marburg
, “Generative adversarial networks for the design of acoustic metamaterials
,” J. Acoust. Soc. Am.
149
(2
), 1162
–1174
(2021
).33.
Y.
Peurifoy
, J.
Shen
, L.
Jing
, Y.
Yang
, F.
Cano-Renteria
, B.
DeLacy
, J.
Joannopoulos
, M.
Tegmark
, and M.
Soljačić
, “Nanophotonic particle simulation and inverse design using artificial neural networks
,” Sci. Adv.
4
(6
), eaar4206
(2018
).34.
E.
Bor
, O.
Alparslan
, M.
Turduev
, Y. S.
Hanay
, H.
Kurt
, S.
Arakawa
, and M.
Murata
, “Integrated silicon photonic device design by attractor selection mechanism based on artificial neural networks: Optical coupler and asymmetric light transmitter
,” Opt. Express
26
(22
), 29032
(2018
).35.
Z.
Liu
, Y.
Tan
, E.
Khoram
, and Z.
Yu
, “Training deep neural networks for the inverse design of nanophotonic structures
,” ACS Photonics
5
, 1365
–1369
(2018
).36.
T.
Zhang
, J.
Wang
, Q.
Liu
, J.
Zhou
, J.
Dai
, X.
Han
, Y.
Zhou
, and K.
Xu
, “Efficient spectrum prediction and inverse design for plasmonic waveguide systems based on artificial neural networks
,” Photonics Res.
7
(3
), 368
(2019
).37.
J.
Jiang
and J. A.
Fan
, “Global optimization of dielectric metasurfaces using a physics-driven neural network
,” Nano Lett.
19
(8
), 5366
–5372
(2019
).38.
S.
Inampudi
and H.
Mosallaei
, “Neural network based design of metagratings
,” Appl. Phys. Lett.
112
(24
), 241102
(2018
).39.
G.
Oliveri
and J. T.
Overvelde
, “Inverse design of mechanical metamaterials that undergo buckling
,” Adv. Funct. Mater.
30
(12
), 1909033
(2020
).40.
R.
Khodayi
and M.
Zavlanos
, “Deep learning for robotic mass transport cloaking
,” IEEE Trans. Robot.
36
(3
), 967
–974
(2020
).41.
R.
Barrett
, M.
Chakraborty
, D.
Amirkulova
, H.
Gandhi
, and A.
White
, “A GPU-accelerated machine learning framework for molecular simulation: HOOMD-blue with TensorFlow
,” ChemRxiv:8019527 (2019
).42.
B.
Sanchez-Lengeling
and A.
Aspuru-Guzik
, “Inverse molecular design using machine learning: Generative models for matter engineering
,” Science
361
(6400
), 360
–365
(2018
).43.
E.
Putin
, A.
Asadulaev
, Y.
Ivanenkov
, V.
Aladinskiy
, B.
Sanchez-Lengeling
, A.
Aspuru-Guzik
, and A.
Zhavoronkov
, “Reinforced adversarial neural computer for de novo molecular design
,” J. Chem. Inf. Model.
58
(6
), 1194
–1204
(2018
).44.
Y.
Dan
, Y.
Zhao
, X.
Li
, S.
Li
, M.
Hu
, and J.
Hu
, “Generative adversarial networks (GAN) based efficient sampling of chemical composition space for inverse design of inorganic materials
,” NPJ Comput. Mater.
6
(1
), 84
–91
(2020
).45.
J.
Fan
, “Freeform metasurface design based on topology optimization
,” MRS Bull.
45
(3
), 196
–201
(2020
).46.
J.
Jiang
and J.
Fan
, “Simulator-based training of generative neural networks for the inverse design of metasurfaces
,” Nanophotonics
9
(5
), 1059
–1069
(2019
).47.
J.
Jiang
, D.
Sell
, S.
Hoyer
, J.
Hickey
, J.
Yang
, and J.
Fan
, “Free-form diffractive metagrating design based on generative adversarial networks
,” ACS Nano
13
(8
), 8872
–8878
(2019
).48.
F.
Wen
, J.
Jiang
, and J. A.
Fan
, “Robust freeform metasurface design based on progressively growing generative networks
,” ACS Photonics
7
, 2098
–2104
(2020
).49.
A.
Blanchard-Dionne
and O.
Martin
, “Successive training of a generative adversarial network for the design of an optical cloak
,” OSA Contin.
4
(1
), 87
–95
(2021
).50.
X.
Han
, Z.
Fan
, Z.
Liu
, C.
Li
, and L. J.
Guo
, “Inverse design of metasurface optical filters using deep neural network with high degrees of freedom
,” InfoMat
3
, 432
–442
(2021
).51.
Y.
Tang
, K.
Kojima
, T.
Koike-Akino
, Y.
Wang
, P.
Wu
, M.
Tahersima
, D.
Jha
, K.
Parsons
, and M.
Qi
, “Generative deep learning model for a multi-level nano-optic broadband power splitter
,” in Proceedings of Optical Fiber Communication Conference 2020
, San Diego, CA (March 8–12, 2020
).52.
R.
Tan
, N.
Zhang
, and W.
Ye
, “A deep learning–based method for the design of microstructural materials
,” Struct. Multidis. Optim.
61
(4
), 1417
–1438
(2020
).53.
I.
Sajedian
, T.
Badloe
, and J.
Rho
, “Optimisation of colour generation from dielectric nanostructures using reinforcement learning
,” Opt. Expr.
27
(4
), 5874
–5883
(2019
).54.
A.
Singh
, L.
Yang
, K.
Hartikainen
, C.
Finn
, and S.
Levine
, “End-to-end robotic reinforcement earning without Reward engineering
,” arXiv:1904.07854 (2019
).55.
B.
Abdulhai
and L.
Kattan
, “Reinforcement learning: Introduction to theory and potential for transport applications
,” Can. J. Civil Eng.
30
(6
), 981
–991
(2003
).56.
R.
Lowe
, Y.
Wu
, A.
Tamar
, J.
Harb
, P.
Abbeel
, and I.
Mordatch
, “Multi-agent actor-critic for mixed cooperative-competitive environments
,” arXiv:1706.02275 (2017
).57.
J.
Yao
, M.
Bukov
, and L.
Lin
, “Policy gradient based quantum approximate optimization algorithm
,” Proc. Machine Learning Res.
107
, 605
–634
(2020
).58.
F. P.
Such
, V.
Madhavan
, E.
Conti
, J.
Lehman
, K. O.
Stanley
, and J.
Clune
, “Deep neuroevolution: Genetic algorithms are a competitive alternative for training deep neural networks for reinforcement learning
,” arXiv:1712.06567 (2017
).59.
G. L.
Guimaraes
, B.
Sanchez-Lengeling
, C.
Outeiral
, P. L. C.
Farias
, and A.
Aspuru-Guzik
, “Objective-reinforced generative adversarial networks (organ) for sequence generation models
,” arXiv:1705.10843v3 (2017
).60.
I.
Sajedian
, H.
Lee
, and J.
Rho
, “Double-deep Q-learning to increase the efficiency of metasurface holograms
,” Sci. Rep.
9
(1
), 10899
(2019
).61.
T. P.
Lillicrap
, J. J.
Hunt
, A.
Pritzel
, N.
Heess
, T.
Erez
, Y.
Tassa
, D.
Silver
, and D.
Wierstra
, “Continuous control with deep reinforcement learning
,” arXiv:1509.02971 (2015
).62.
F. A.
Amirkulova
and A. N.
Norris
, “The gradient of total multiple scattering cross-section and its application to acoustic cloaking
,” J. Theor. Comput. Acoust.
28
, 1950016
(2020
).63.
A. N.
Norris
, “Acoustic integrated extinction
,” Proc. R. Soc. A
471
(2177
), 20150008
(2015
).64.
65.
T.
Schaul
, J.
Quan
, I.
Antonoglou
, and D.
Silver
, “Prioritized experience replay
,” arXiv:1511.05952 (2015
).66.
M.
Hessel
, J.
Modayil
, H.
van Hasselt
, T.
Schaul
, G.
Ostrovski
, W.
Dabney
, D.
Horgan
, B.
Piot
, M.
Azar
, and D.
Silver
, “Rainbow: Combining improvements in deep reinforcement learning
,” arXiv:1710.02298 (2017
).67.
P. J.
Huber
, “Robust estimation of a location parameter
,” Ann. Math. Statist.
35
(1
), 73
–101
(1964
).68.
“
The College of Engineering high performance computing system
,” San Jose State University, http://coe-hpc-web.sjsu.edu (Last viewed 4 July 2021).69.
“
When the solver fails. MathWorks MATLAB documentation
,” https://www.mathworks.com/help/optim/ug/when-the-solver-fails.html (Last viewed 4 July 2021).70.
71.
L.
Espeholt
, R.
Marinier
, P.
Stanczyk
, K.
Wang
, and M.
Michalski
, “SEED RL: Scalable and efficient deep-RL with accelerated central inference
,” arXiv:1910.06591 (2019
).72.
M.
Abramowitz
and I.
Stegun
, Handbook of Mathematical Functions with Formulas, Graphs, and Mathematical Tables
(Dover
, New York
, 1974
).73.
F. A.
Amirkulova
, “Acoustic and elastic multiple scattering and radiation from cylindrical structures
,” Ph.D. thesis, Rutgers University
, Piscataway, NJ
, 2014
.74.
V. K.
Varadan
and V. V.
Varadan
, eds., Acoustic, Electromagnetic and Elastic Wave Scattering - Focus on the T-Matrix Approach
(Pergamon
, New York
, 1980
).75.
F.
Amirkulova
and A.
Norris
, “Acoustic multiple scattering using fast iterative techniques
,” in Proceedings of 2017 ASME International Mechanical Engineering Congress and Exposition (IMECE 2017)
, Tampa, FL (November 3–9, 2017
), Paper No. IMECE2017/72249.© 2021 Acoustical Society of America.
2021
Acoustical Society of America
You do not currently have access to this content.