There has been a rapid advancement in deep learning models for diverse research fields and, more recently, in fluid dynamics. This study presents self-supervised transformers' deep learning for complex turbulent flow signals across various test problems. Self-supervision aims to leverage the ability to extract meaningful representations from sparse flow time-series data to improve the transformer model accuracy and computational efficiency. Two high-speed flow cases are considered: a supersonic compression ramp and shock-boundary layer interaction over a statically deformed surface. Several training scenarios are investigated across the two different supersonic configurations. The training data concern wall pressure fluctuations due to their importance in aerodynamics, aeroelasticity, noise, and acoustic fatigue. The results provide insight into transformers, self-supervision, and deep learning with application to complex time series. The architecture is extendable to other research domains where time series data are essential.

1.
A.
Vaswani
,
N.
Shazeer
,
N.
Parmar
,
J.
Uszkoreit
,
L.
Jones
,
A. N.
Gomez
,
L. U.
Kaiser
, and
I.
Polosukhin
, “
Attention is all you need
,” in
Advances in Neural Information Processing Systems
, edited by
I.
Guyon
,
U. V.
Luxburg
,
S.
Bengio
,
H.
Wallach
,
R.
Fergus
,
S.
Vishwanathan
, and
R.
Garnett
(
Curran Associates, Inc
.,
2017
), Vol.
30
.
2.
J.
Sousa
and
R.
Henriques
, “
Intersecting reinforcement learning and deep factor methods for optimizing locality and globality in forecasting: A review
,”
Eng. Appl. Artif. Intell.
133
,
108082
(
2024
).
3.
T.
Lin
,
Y.
Wang
,
X.
Liu
, and
X.
Qiu
, “
A survey of transformers
,”
AI Open
3
,
111
132
(
2022
).
4.
B.
Cottier
, “
Trends in the dollar training cost of machine learning systems
” (accessed 26 January
2024
), see https://epochai.org/blog/trends-in-the-dollar-training-cost-of-machine-learning-systems.
5.
D.
Hendrycks
,
M.
Mazeika
,
S.
Kadavath
, and
D.
Song
, “
Using self-supervised learning can improve model robustness and uncertainty
,” arXiv:1906.12340 (
2019
).
6.
H. A.
Gündüz
,
M.
Binder
,
X.-Y.
To
,
R.
Mreches
,
B.
Bischl
,
A. C.
McHardy
,
P. C.
Münch
, and
M.
Rezaei
, “
A self-supervised deep learning method for data-efficient training in genomics
,”
Commun. Biol.
6
,
928
(
2023
).
7.
S.
Schneider
,
A.
Baevski
,
R.
Collobert
, and
M.
Auli
, “
Speech recognition
,” in
The SAGE Encyclopedia of Human Communication Sciences and Disorders
(
Sage
,
2019
).
8.
J.
Devlin
,
M.-W.
Chang
,
K.
Lee
, and
K.
Toutanova
, “
Bert: Pre-training of deep bidirectional transformers for language understanding
,” in
North American Chapter of the Association for Computational Linguistics
(
Association for Computational Linguistics
2019
).
9.
J.
Jiang
,
G.
Li
,
Y.
Jiang
,
L.
Zhang
, and
X.
Deng
, “
Transcfd: A transformer-based decoder for flow field prediction
,”
Eng. Appl. Artif. Intell.
123
,
106340
(
2023
).
10.
W.
Peng
,
Z.
Yuan
, and
J.
Wang
, “
Attention-enhanced neural network models for turbulence simulation
,”
Phys. Fluids
34
,
025111
(
2022
).
11.
P.
Wu
,
S.
Gong
,
K.
Pan
,
F.
Qiu
,
W.
Feng
, and
C.
Pain
, “
Reduced order model using convolutional auto-encoder with self-attention
,”
Phys. Fluids
33
,
077107
(
2021
).
12.
B.
Xu
,
Y.
Zhou
, and
X.
Bian
, “
Self-supervised learning based on transformer for flow reconstruction and prediction
,” arXiv:2311.15232 (
2023
).
13.
D.
Drikakis
,
I. W.
Kokkinakis
,
D.
Fung
, and
S. M.
Spottswood
, “
Generalizability of transformer-based deep learning for multidimensional turbulent flow data
,”
Phys. Fluids
36
,
026102
(
2024
).
14.
R.
Wiebe
and
S. M.
Spottswood
, “
On the dimension of complex responses in nonlinear structural vibrations
,”
J. Sound Vib.
373
,
192
204
(
2016
).
15.
K. R.
Brouwer
,
R.
Perez
,
T. J.
Beberniss
, and
S. M.
Spottswood
, “
Aeroelastic experiments and companion computations assessing the impact of impinging shock sweep
,” in
AIAA SCITECH 2023 Forum, 23–27 January 2023, National Harbor, MD
(
2023
).
16.
K. R.
Brouwer
,
R.
Perez
,
T. J.
Beberniss
, and
S. M.
Spottswood
, “
Surface pressure measurements and predictions in shock-dominated flows
,” in
AIAA AVIATION 2023 Forum, 12–16 June 2023, San Diego, CA
(
2023
).
17.
S. J.
Peltier
,
K. R.
Brouwer
,
R.
Perez
,
S. M.
Spottswood
, and
S.
Hammack
, “
Boundary-layer measurements for FTSI systems: Influence of panel flutter on a Mach 2 turbulent boundary-layer
,” in
AIAA SCITECH 2023 Forum, 23–27 January 2023, National Harbor, MD
(
2023
).
18.
A.
D'Aguanno
,
P.
Quesada Allerhand
,
F. F. J.
Schrijer
, and
B. W.
van Oudheusden
, “
Characterization of shock-induced panel flutter with simultaneous use of DIC and PIV
,”
Exp. Fluids
64
,
15
(
2023
).
19.
L.
Piccolo Serafim
,
M.
Freydin
, and
E. H.
Dowell
, “
Correlation of supersonic wind tunnel measurements with a nonlinear aeroelastic theoretical/computational model of a thin plate
,”
J. Fluids Struct.
122
,
103981
(
2023
).
20.
J. D.
Thayer
,
J. J.
McNamara
, and
D. V.
Gaitonde
, “
Unsteady aerodynamic response of a high-speed, separated flow to a deforming cantilever plate
,” in
AIAA SCITECH 2022 Forum, January 3–7, 2022, San Diego, CA
(
2022
).
21.
V.
Shinde
,
J.
McNamara
, and
D.
Gaitonde
, “
Dynamic interaction between shock wave turbulent boundary layer and flexible panel
,”
J. Fluids Struct.
113
,
103660
(
2022
).
22.
I. W.
Kokkinakis
,
D.
Drikakis
,
S. M.
Spottswood
,
K. R.
Brouwer
, and
Z. B.
Riley
, “
High-speed shock-boundary-layer interaction over deformed surfaces
,”
Phys. Fluids
35
,
106109
(
2023
).
23.
I. W.
Kokkinakis
,
D.
Drikakis
,
S.
Michael Spottswood
,
K. R.
Brouwer
, and
Z. B.
Riley
, “
Aeroacoustic loading of impinging supersonic boundary-layer interaction on statically deformed surfaces
,”
AIAA J.
(published online).
24.
K.
Yamamoto
,
R.
Hiromatsu
, and
T.
Ohtsuki
, “
ECG signal reconstruction via doppler sensor by hybrid deep learning model with CNN and LSTM
,”
IEEE Access
8
,
130551
130560
(
2020
).
25.
R. B.
Lund
,
C.
Beaulieu
,
R.
Killick
,
Q.
Lu
, and
X.
Shi
, “
Good practices and common pitfalls in climate time series changepoint techniques: A review
,”
J. Clim.
36
,
8041
8057
(
2023
).
26.
S. M.
Spottswood
,
T. J.
Beberniss
,
T. G.
Eason
,
R. A.
Perez
,
J. M.
Donbar
,
D. A.
Ehrhardt
, and
Z. B.
Riley
, “
Exploring the response of a thin, flexible panel to shock-turbulent boundary-layer interactions
,”
J. Sound Vib.
443
,
74
89
(
2019
).
27.
K. R.
Brouwer
,
R. A.
Perez
,
T. J.
Beberniss
,
S. M.
Spottswood
,
D. A.
Ehrhardt
, and
R.
Wiebe
, “
Investigation of aeroelastic instabilities for a thin panel in turbulent flow
,”
Nonlinear Dyn.
104
,
3323
3346
(
2021
).
28.
K. R.
Brouwer
,
R. A.
Perez
,
T. J.
Beberniss
,
S. M.
Spottswood
, and
D. A.
Ehrhardt
, “
Experiments on a thin panel excited by turbulent flow and shock/boundary-layer interactions
,”
AIAA J.
59
,
2737
2752
(
2021
).
29.
I. W.
Kokkinakis
,
D.
Drikakis
,
K.
Ritos
, and
S. M.
Spottswood
, “
Direct numerical simulation of supersonic flow and acoustics over a compression ramp
,”
Phys. Fluids
32
,
066107
(
2020
).
30.
I.
Kokkinakis
and
D.
Drikakis
, “
Implicit large eddy simulation of weakly-compressible turbulent channel flow
,”
Comput. Methods Appl. Mech. Eng.
287
,
229
261
(
2015
).
31.
X.-D.
Liu
,
S.
Osher
, and
T.
Chan
, “
Weighted essentially non-oscillatory schemes
,”
J. Comput. Phys.
115
,
200
212
(
1994
).
32.
G.-S.
Jiang
and
C.-W.
Shu
, “
Efficient implementation of weighted ENO schemes
,”
J. Comput. Phys.
126
,
202
228
(
1996
).
33.
D. S.
Balsara
and
C.-W.
Shu
, “
Monotonicity preserving weighted essentially non-oscillatory schemes with increasingly high order of accuracy
,”
J. Comput. Phys.
160
,
405
452
(
2000
).
34.
E. F.
Toro
,
M.
Spruce
, and
W.
Speares
, “
Restoration of the contact surface in the HLL-Riemann solver
,”
Shock Waves
4
,
25
34
(
1994
).
35.
R.
Spiteri
and
S.
Ruuth
, “
A new class of optimal high-order strong-stability-preserving time discretization methods
,”
SIAM J. Numer. Anal.
40
,
469
491
(
2002
).
36.
M.
Klein
,
A.
Sadiki
, and
J.
Janicka
, “
A digital filter based generation of inflow data for spatially developing direct numerical or large eddy simulations
,”
J. Comput. Phys.
186
,
652
665
(
2003
).
37.
E.
Touber
and
N. D.
Sandham
, “
Large-eddy simulation of low-frequency unsteadiness in a turbulent shock-induced separation bubble
,”
Theor. Comput. Fluid Dyn.
23
,
79
107
(
2009
).
38.
Z.
Rana
,
B.
Thornber
, and
D.
Drikakis
, “
Transverse jet injection into a supersonic turbulent cross-flow
,”
Phys. Fluids
23
,
046103
(
2011
).
39.
Z. A.
Rana
,
B.
Thornber
, and
D.
Drikakis
, “
On the importance of generating accurate turbulent boundary condition for unsteady simulations
,”
J. Turbul.
12
,
N35
(
2011
).
40.
K.
Fukushima
, “
Cognitron: A self-organizing multilayered neural network
,”
Biol. Cybern.
20
,
121
136
(
1975
).
41.
S.
Amari
, “
A theory of adaptive pattern classifiers
,”
IEEE Trans. Electron. Comput.
EC-16
,
299
307
(
1967
).
42.
D. E.
Rumelhart
,
G. E.
Hinton
, and
R. J.
Williams
, “
Learning internal representations by error propagation
,” in
Parallel Distributed Processing: Explorations in the Microstructure of Cognition: Foundations
(
IEEE
,
1986
).
43.
S.
Cuomo
,
V. S.
Di Cola
,
F.
Giampaolo
,
G.
Rozza
,
M.
Raissi
, and
F.
Piccialli
, “
Scientific machine learning through physics-informed neural networks: Where we are and what's next
,”
J. Sci. Comput.
92
,
88
(
2022
).
You do not currently have access to this content.