Studies examining sensorimotor adaptation of speech to changing sensory conditions have demonstrated a central role for both auditory and somatosensory feedback in speech motor learning. The potential influence of visual feedback of oral articulators, which is not typically available during speech production but may nonetheless enhance oral motor control, remains poorly understood. The present study explores the influence of ultrasound visual feedback of the tongue on adaptation of speech production (focusing on the sound /s/) to a physical perturbation of the oral articulators (prosthesis altering the shape of the hard palate). Two visual feedback groups were tested that differed in the two-dimensional plane being imaged (coronal or sagittal) during practice producing /s/ words, along with a no-visual-feedback control group. Participants in the coronal condition were found to adapt their speech production across a broader range of acoustic spectral moments and syllable contexts than the no-feedback controls. In contrast, the sagittal group showed reduced adaptation compared to no-feedback controls. The results indicate that real-time visual feedback of the tongue is spontaneously integrated during speech motor adaptation, with effects that can enhance or interfere with oral motor learning depending on compatibility of the visual articulatory information with requirements of the speaking task.

1.
Aasland
,
W. A.
,
Baum
,
S. R.
, and
McFarland
,
D. H.
(
2006
). “
Electropalatographic, acoustic, and perceptual data on adaptation to a palatal perturbation
,”
J. Acoust. Soc. Am.
119
,
2372
2381
.
2.
Adler-Bock
,
M.
,
Bernhardt
,
B. M.
,
Gick
,
B.
, and
Bacsfalvi
,
P.
(
2007
). “
The use of ultrasound in remediation of North American English /r/ in 2 adolescents
,”
Am. J. Speech. Lang. Pathol.
16
,
128
139
.
3.
Arnold
,
P.
, and
Hill
,
F.
(
2001
). “
Bisensory augmentation: A speechreading advantage when speech is clearly audible and intact
,”
Br. J. Psychol.
92
,
339
355
.
4.
Avery
,
J. D.
, and
Liss
,
J. M.
(
1996
). “
Acoustic characteristics of less‐masculine‐sounding male speech
,”
J. Acoust. Soc. Am.
99
,
3738
3748
.
5.
Bacsfalvi
,
P.
(
2010
). “
Attaining the lingual components of /r/ with ultrasound for three adolescents with cochlear implants
,”
Can. J. Speech Lang. Pathol. Audiol.
34
,
206
217
.
6.
Bacsfalvi
,
P.
,
Bernhardt
,
B. M.
, and
Gick
,
B.
(
2007
). “
Electropalatography and ultrasound in vowel remediation for adolescents with hearing impairment
,”
Adv. Speech Lang. Pathol.
9
,
36
45
.
7.
Barbier
,
G.
,
Baum
,
S. R.
,
Ménard
,
L.
, and
Shiller
,
D. M.
(
2020
). “
Sensorimotor adaptation across the speech production workspace in response to a palatal perturbation
,”
J. Acoust. Soc. Am.
147
,
1163
1178
.
8.
Bates
,
D.
,
Mächler
,
M.
,
Bolker
,
B.
, and
Walker
,
S.
(
2015
). “
Fitting linear mixed-effects models using lme4
,”
J. Stat. Softw.
67
,
1
48
.
9.
Baum
,
S. R.
, and
McFarland
,
D. H.
(
1997
). “
The development of speech adaptation to an artificial palate
,”
J. Acoust. Soc. Am.
102
,
2353
2359
.
10.
Baum
,
S. R.
, and
McFarland
,
D. H.
(
2000
). “
Individual differences in speech adaptation to an artificial palate
,”
J. Acoust. Soc. Am.
107
,
3572
3575
.
11.
Bernhardt
,
B. M.
,
Bacsfalvi
,
P.
,
Adler-Bock
,
M.
,
Shimizu
,
R.
,
Cheney
,
A.
,
Giesbrecht
,
N.
,
O'connell
,
M.
,
Sirianni
,
J.
, and
Radanov
,
B.
(
2008
). “
Ultrasound as visual feedback in speech habilitation: Exploring consultative use in rural British Columbia, Canada
,”
Clin. Linguist. Phon.
22
,
149
162
.
12.
Bernhardt
,
B. M.
,
Gick
,
B.
,
Bacsfalvi
,
P.
, and
Ashdown
,
J.
(
2003
). “
Speech habilitation of hard of hearing adolescents using electropalatography and ultrasound as evaluated by trained listeners
,”
Clin. Linguist. Phon.
17
,
199
216
.
13.
Bliss
,
H.
,
Abel
,
J.
, and
Gick
,
B.
(
2018
). “
Computer-assisted visual articulation feedback in L2 pronunciation instruction: A review
,”
J. Second Lang. Pronunciation
4
,
129
153
.
14.
Bressmann
,
T.
,
Harper
,
S.
,
Zhylich
,
I.
, and
Kulkarni
,
G. V.
(
2016
). “
Perceptual, durational and tongue displacement measures following articulation therapy for rhotic sound errors
,”
Clin. Linguist. Phon.
30
,
345
362
.
15.
Brunner
,
J.
(
2009
). “
Perturbed speech: How compensation mechanisms can inform us about 1154 phonemic targets
,”
Sudwestdeutscher Verlag Fur Hochschulschrifte
196
,
hal-00372151
; available at https://hal.archives-ouvertes.fr/hal-00372151.
16.
Brunner
,
J.
,
Ghosh
,
S.
,
Hoole
,
P.
,
Matthies
,
M.
,
Tiede
,
M.
, and
Perkell
,
J.
(
2011
). “
The influence of auditory acuity on acoustic variability and the use of motor equivalence during adaptation to a perturbation
,”
J. Speech Lang. Hear. Res.
54
,
727
739
.
17.
Carter
,
P.
, and
Edwards
,
S.
(
2004
). “
EPG therapy for children with long‐standing speech disorders: Predictions and outcomes
,”
Clin. Linguist. Phon.
18
,
359
372
.
18.
Cleland
,
J.
,
Scobbie
,
J. M.
,
Roxburgh
,
Z.
,
Heyde
,
C.
, and
Wrench
,
A.
(
2019
). “
Enabling new articulatory gestures in children with persistent speech sound disorders using ultrasound visual biofeedback
,”
J. Speech Lang. Hear. Res.
62
,
229
246
.
19.
Cleland
,
J.
,
Scobbie
,
J. M.
, and
Wrench
,
A. A.
(
2015
). “
Using ultrasound visual biofeedback to treat persistent primary speech sound disorders
,”
Clin. Linguist. Phon.
29
,
575
597
.
20.
Dagenais
,
P. A.
,
Critz-Crosby
,
P.
, and
Adams
,
J. B.
(
1994
). “
Defining and remediating persistent lateral lisps in children using electropalatography
,”
Am. J. Speech. Lang. Pathol.
3
,
67
76
.
21.
Duffy
,
J. R.
(
2019
).
Motor Speech Disorders: Substrates, Differential Diagnosis, and Management
, 4th ed. (
Elsevier Health Sciences
,
Philadelphia, PA
).
22.
Erber
,
N. P.
(
1975
). “
Auditory-visual perception of speech
,”
J. Speech Hear. Disord.
40
,
481
492
.
23.
Forrest
,
K.
,
Weismer
,
G.
,
Milenkovic
,
P.
, and
Dougall
,
R. N.
(
1988
). “
Statistical analysis of word-initial voiceless obstruents: Preliminary data
,”
J. Acoust. Soc. Am.
84
,
115
123
.
24.
Freedman
,
S. E.
,
Maas
,
E.
,
Caligiuri
,
M. P.
,
Wulf
,
G.
, and
Robin
,
D. A.
(
2007
). “
Internal versus external: Oral-motor performance as a function of attentional focus
,”
J. Speech Lang. Hear. Res.
50
,
131
136
.
25.
Gibbon
,
F.
, and
Hardcastle
,
W.
(
1987
). “
Articulatory description and treatment of ‘lateral/s/’ using electropalatography: A case study
,”
Int. J. Lang. Commun. Disord.
22
,
203
217
.
26.
Gibbon
,
F.
, and
Hardcastle
,
W.
(
1989
). “
Deviant articulation in a cleft palate child following late repair of the hard palate: A description and remediation procedure using electropalatography (EPG)
,”
Clin. Linguist. Phon.
3
,
93
110
.
27.
Gibbon
,
F.
,
McNeill
,
A. M.
,
Wood
,
S. E.
, and
Watson
,
J. M. M.
(
2003
). “
Changes in linguapalatal contact patterns during therapy for velar fronting in a 10-year-old with Down's syndrome
,”
Int. J. Lang. Commun. Disord.
38
,
47
64
.
28.
Gibbon
,
F.
, and
Wood
,
S. E.
(
2003
). “
Using electropalatography (EPG) to diagnose and treat articulation disorders associated with mild cerebral palsy: A case study
,”
Clin. Linguist. Phon.
17
,
365
374
.
29.
Gick
,
B.
,
Bernhardt
,
B. M.
,
Bacsfalvi
,
P.
, and
Wilson
,
I.
(
2008
). “
Ultrasound imaging applications in second language acquisition
,” in
Phonology and Second Language Acquisition
, edited by
J.
Hansen
and
M.
Zampini
(
John Benjamins
,
Amsterdam
), Chap. 11, pp.
309
322
.
30.
Hamlet
,
S.
,
Geoffrey
,
V. C.
, and
Bartlett
,
D. M.
(
1976
). “
Effect of a dental prosthesis on speaker-specific characteristics of voice
,”
J. Speech Hear. Res.
19
,
639
650
.
31.
Hamlet
,
S.
,
Stone
,
M.
, and
McCarty
,
T.
(
1978
). “
Conditioning prostheses viewed from the standpoint of speech adaptation
,”
J. Prosthet. Dent.
40
,
60
66
.
32.
Hardcastle
,
W. J.
,
Morgan Barry
,
R. A.
, and
Clark
,
C. J.
(
1987
). “
An instrumental phonetic study of lingual activity in articulation-disordered children
,”
J. Speech. Hear. Res.
30
,
171
184
.
33.
Hitchcock
,
E. R.
, and
Byun
,
T. M.
(
2015
). “
Enhancing generalisation in biofeedback intervention using the challenge point framework: A case study
,”
Clin. Linguist. Phon.
29
,
59
75
.
34.
Hitchcock
,
E. R.
,
Byun
,
T. M.
,
Swartz
,
M.
, and
Lazarus
,
R.
(
2017
). “
Efficacy of electropalatography for treating misarticulation of /r
/,”
Am. J. Speech. Lang. Pathol.
26
,
1141
1158
.
35.
Houde
,
J. F.
, and
Jordan
,
M. I.
(
1998
). “
Sensorimotor adaptation in speech production
,”
Science
279
,
1213
1216
.
36.
Jongman
,
A.
,
Wayland
,
R.
, and
Wong
,
S.
(
2000
). “
Acoustic characteristics of English fricatives
,”
J. Acoust. Soc. Am.
108
,
1252
1263
.
37.
Katz
,
W.
,
Lawn
,
A.
, and
Kumar
,
H.
(
2020
). “
Opti-Speech: A real-time tongue model for research and speech training
,”
Proceedings of the 12th International Seminar on Speech Production (ISSP)
, December 14–18.
38.
Katz
,
W. F.
, and
Mehta
,
S.
(
2015
). “
Visual feedback of tongue movement for novel speech sound learning
,”
Front. Hum. Neurosci.
9
,
612
.
39.
Koenig
,
L. L.
,
Shadle
,
C. H.
,
Preston
,
J. L.
, and
Mooshammer
,
C. R.
(
2013
). “
Toward improved spectral measures of /s/: Results from adolescents
,”
J. Speech Lang. Hear. Res.
56
,
1175
1189
.
40.
Krakauer
,
J. W.
, and
Mazzoni
,
P.
(
2011
). “
Human sensorimotor learning: Adaptation, skill, and beyond
,”
Curr. Opin. Neurobiol.
21
,
636
644
.
41.
Kröger
,
B. J.
,
Kannampuzha
,
J.
, and
Neuschaefer-Rube
,
C.
(
2009
). “
Towards a neurocomputational model of speech production and perception
,”
Speech Communication
51
(
9
),
793
809
.
42.
Kroger
,
B.
,
Miller
,
N.
,
Lowit
,
A., Lowit
, and
Neuschafer-Rube
,
C.
(
2011
). “
Defective neural motor speech mappings as a source for apraxia of speech: Evidence from a quantitative neural model of speech processing (ACT)
,” in
Assessment of Motor Speech Disorders
(
Plural Publishing
), pp.
325
346
.
43.
Ladefoged
,
P.
, and
Johnson
,
K.
(
2014
).
A Course in Phonetics
(
Nelson Education
,
Toronto, Canada
).
44.
Lametti
,
D. R.
,
Smith
,
H. J.
,
Watkins
,
K. E.
, and
Shiller
,
D. M.
(
2018
). “
Robust sensorimotor learning during variable sentence-level speech
,”
Curr. Biol.
28
,
3106
3113.e2
.
45.
Lee
,
A. S.-Y.
,
Law
,
J.
, and
Gibbon
,
F. E.
(
2009
). “
Electropalatography for articulation disorders associated with cleft palate
,”
Cochrane Database Syst. Rev.
3
,
CD006854
.
46.
McAllister Byun
,
T. M.
,
Hitchcock
,
E. R.
, and
Swartz
,
M. T.
(
2014
). “
Retroflex versus bunched in treatment for rhotic misarticulation: Evidence from ultrasound biofeedback intervention
,”
J. Speech Lang. Hear. Res.
57
,
2116
2130
.
47.
McAuliffe
,
M. J.
, and
Cornwell
,
P. L.
(
2008
). “
Intervention for lateral /s/ using electropalatography (EPG) biofeedback and an intensive motor learning approach: A case report
,”
Int. J. Lang. Commun. Disord.
43
,
219
229
.
48.
McFarland
,
D. H.
,
Baum
,
S. R.
, and
Chabot
,
C.
(
1996
). “
Speech compensation to structural modifications of the oral cavity
,”
J. Acoust. Soc. Am.
100
,
1093
1104
.
49.
McGurk
,
H.
, and
MacDonald
,
J.
(
1976
). “
Hearing lips and seeing voices
,”
Nature
264
,
746
748
.
50.
McLeod
,
S.
, and
Searl
,
J.
(
2006
). “
Adaptation to an electropalatograph palate: Acoustic, impressionistic, and perceptual data
,”
Am. J. Speech Lang. Pathol.
15
,
192
206
.
51.
Ménard
,
L.
,
Côté
,
D.
, and
Trudeau-Fisette
,
P.
(
2016a
). “
Maintaining distinctiveness at increased speaking rates: A comparison between congenitally blind and sighted speakers
,”
Folia Phoniatr. Logop.
68
,
232
238
.
52.
Ménard
,
L.
,
Dupont
,
S.
,
Baum
,
S. R.
, and
Aubin
,
J.
(
2009
). “
Production and perception of French vowels by congenitally blind adults and sighted adults
,”
J. Acoust. Soc. Am.
126
,
1406
1414
.
53.
Ménard
,
L.
,
Trudeau-Fisette
,
P.
,
Côté
,
D.
, and
Turgeon
,
C.
(
2016b
). “
Speaking clearly for the blind: Acoustic and articulatory correlates of speaking conditions in sighted and congenitally blind speakers
,”
PLoS One
11
,
e0160088
.
54.
Ménard
,
L.
,
Turgeon
,
C.
,
Trudeau-Fisette
,
P.
, and
Bellavance-Courtemanche
,
M.
(
2016c
). “
Effects of blindness on production-perception relationships: Compensation strategies for a lip-tube perturbation of the French [u]
,”
Clin. Linguist. Phon.
30
,
227
248
.
55.
Michi
,
K.
,
Suzuki
,
N.
,
Yamashita
,
Y.
, and
Imai
,
S.
(
1986
). “
Visual training and correction of articulation disorders by use of dynamic palatography: Serial observation in a case of cleft palate
,”
J. Speech Hear. Disord.
51
,
226
238
.
56.
Mozaffari
,
M. H.
,
Guan
,
S.
,
Wen
,
S.
,
Wang
,
N.
, and
Lee
,
W.
(
2018
). “
Guided learning of pronunciation by visualizing tongue articulation in ultrasound image sequences
,”
Proceedings of the 2018 IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA)
, June 12–13, Ottawa, Canada.
57.
Nissen
,
S. L.
, and
Fox
,
R. A.
(
2005
). “
Acoustic and spectral characteristics of young children's fricative productions: A developmental perspective
,”
J. Acoust. Soc. Am.
118
,
2570
2578
.
58.
Nittrouer
,
S.
(
1995
). “
Children learn separate aspects of speech production at different rates: Evidence from spectral moments
,”
J. Acoust. Soc. Am.
97
,
520
530
.
59.
Nittrouer
,
S.
,
Studdert-Kennedy
,
M.
, and
McGowan
,
R. S.
(
1989
). “
The emergence of phonetic segments: Evidence from the spectral structure of fricative-vowel syllables spoken by children and adults
,”
J. Speech Hear. Res.
32
,
120
132
.
60.
Niziolek
,
C. A.
,
Nagarajan
,
S. S.
, and
Houde
,
J. F.
(
2013
). “
What does motor efference copy represent? Evidence from speech production
,”
J. Neurosci.
33
,
16110
16116
.
61.
Parrell
,
B.
,
Lammert
,
A. C.
,
Ciccarelli
,
G.
, and
Quatieri
,
T. F.
(
2019
). “
Current models of speech motor control: A control-theoretic overview of architectures and properties
,”
J. Acoust. Soc. Am.
145
,
1456
1481
.
62.
Perkell
,
J. S.
,
Matthies
,
M. L.
,
Tiede
,
M.
,
Lane
,
H.
,
Zandipour
,
M.
,
Marrone
,
N.
,
Stockmann
,
E.
, and
Guenther
,
F. H.
(
2004
). “
The distinctness of speakers' /s/—/∫/ contrast is related to their auditory discrimination and use of an articulatory saturation effect
,”
J. Speech. Lang. Hear. Res.
47
,
1259
1269
.
63.
Preston
,
J. L.
,
Brick
,
N.
, and
Landi
,
N.
(
2013
). “
Ultrasound biofeedback treatment for persisting childhood apraxia of speech
,”
Am. J. Speech. Lang. Pathol.
22
,
627
643
.
64.
Preston
,
J. L.
,
Maas
,
E.
,
Whittle
,
J.
,
Leece
,
M. C.
, and
McCabe
,
P.
(
2016
). “
Limited acquisition and generalisation of rhotics with ultrasound visual feedback in childhood apraxia
,”
Clin. Linguist. Phon.
30
,
363
381
.
65.
Preston
,
J. L.
,
McAllister
,
T.
,
Boyce
,
S. E.
,
Hamilton
,
S.
,
Tiede
,
M.
,
Phillips
,
E.
,
Rivera-Campos
,
A.
, and
Whalen
,
D. H.
(
2017
). “
Ultrasound images of the tongue: A tutorial for assessment and remediation of speech sound errors
,”
J. Vis. Exp.
119
,
e55123
.
66.
Preston
,
J. L.
,
McAllister
,
T.
,
Phillips
,
E.
,
Boyce
,
S.
,
Tiede
,
M.
,
Kim
,
J. S.
, and
Whalen
,
D. H.
(
2019
). “
Remediating residual rhotic errors with traditional and ultrasound-enhanced treatment: A single-case experimental study
,”
Am. J. Speech. Lang. Pathol.
28
,
1167
1183
.
67.
R Core Team
(
2020
). “
R: A language and environment for statistical computing
,” version 4.0.1 (
R Foundation for Statistical Computing
,
Vienna, Austria
), https://www.R-project.org/ (Last viewed September 1, 2020).
68.
Roxburgh
,
Z.
,
Cleland
,
J.
, and
Scobbie
,
J. M.
(
2016
). “
Multiple phonetically trained-listener comparisons of speech before and after articulatory intervention in two children with repaired submucous cleft palate
,”
Clin. Linguist. Phon.
30
,
398
415
.
69.
Rvachew
,
S.
, and
Brosseau-Lapré
,
F.
(
2016
).
Developmental Phonological Disorders: Foundations of Clinical Practice
, 2nd ed. (
Plural Publishing
,
San Diego, CA
).
70.
Shadle
,
C. H.
(
1990
). “
Articulatory-Acoustic relationships in fricative consonants
,” in
Speech Production and Speech Modelling
, edited by
W. J.
Hardcastle
and
A.
Marchal
(
Springer
,
Dordrecht, Netherlands
), pp.
187
209
.
71.
Shadle
,
C. H.
, and
Mair
,
S. J.
(
1996
). “
Quantifying spectral characteristics of fricatives
,” in
Proceedings of the Fourth International Conference on Spoken Language Processing: ICSLP '96
, October 3–6, Philadelphia, PA, Vol.
3
, pp.
1521
1524
.
72.
Sjolie
,
G. M.
,
Leece
,
M. C.
, and
Preston
,
J. L.
(
2016
). “
Acquisition, retention, and generalization of rhotics with and without ultrasound visual feedback
,”
J. Commun. Disord.
64
,
62
77
.
73.
Stevens
,
K. N.
(
1998
).
Acoustic Phonetics
(
MIT
,
Cambridge, MA
).
74.
Stone
,
M.
, and
Lundberg
,
A.
(
1996
). “
Three‐dimensional tongue surface shapes of English consonants and vowels
,”
J. Acoust. Soc. Am.
99
,
3728
3737
.
75.
Suemitsu
,
A.
,
Dang
,
J.
,
Ito
,
T.
, and
Tiede
,
M.
(
2015
). “
A real-time articulatory visual feedback approach with target presentation for second language pronunciation learning
,”
J. Acoust. Soc. Am.
138
,
EL382
EL387
.
76.
Sugden
,
E.
,
Lloyd
,
S.
,
Lam
,
J.
, and
Cleland
,
J.
(
2019
). “
Systematic review of ultrasound visual biofeedback in intervention for speech sound disorders
,”
Int. J. Lang. Commun. Disord.
54
,
705
728
.
77.
Sumby
,
W. H.
, and
Pollack
,
I.
(
1954
). “
Visual contribution to speech intelligibility in noise
,”
J. Acoust. Soc. Am.
26
,
212
215
.
78.
Thibeault
,
M.
,
Ménard
,
L.
,
Baum
,
S. R.
,
Richard
,
G.
, and
McFarland
,
D. H.
(
2011
). “
Articulatory and acoustic adaptation to palatal perturbation
,”
J. Acoust. Soc. Am.
129
,
2112
2120
.
79.
Thomson
,
D. J.
(
1982
). “
Spectrum estimation and harmonic analysis
,”
Proc. IEEE
70
(
9
),
1055
1096
.
80.
Tjaden
,
K.
, and
Turner
,
G. S.
(
1997
). “
Spectral properties of fricatives in amyotrophic lateral sclerosis
,”
J. Speech Lang. Hear. Res.
40
,
1358
1372
.
81.
Todd
,
A. E.
,
Edwards
,
J. R.
, and
Litovsky
,
R. Y.
(
2011
). “
Production of contrast between sibilant fricatives by children with cochlear implants
,”
J. Acoust. Soc. Am.
130
,
3969
3979
.
82.
Tomiak
,
G. R.
(
1990
). “
An acoustic and perceptual analysis of the spectral moments invariant with voiceless fricative obstruents
,” Ph.D. thesis,
SUNY Buffalo
,
Buffalo, NY
.
83.
Tremblay
,
S.
,
Shiller
,
D. M.
, and
Ostry
,
D. J.
(
2003
). “
Somatosensory basis of speech production
,”
Nature
423
,
866
869
.
84.
Trudeau-Fisette
,
P.
,
Tiede
,
M.
, and
Ménard
,
L.
(
2017
). “
Compensations to auditory feedback perturbations in congenitally blind and sighted speakers: Acoustic and articulatory data
,”
PLoS One
12
,
e0180300
.
85.
Turgeon
,
C.
,
Trudeau-Fisette
,
P.
,
Lepore
,
F.
,
Lippé
,
S.
, and
Ménard
,
L.
(
2020
). “
Impact of visual and auditory deprivation on speech perception and production in adults
,”
Clin. Linguist. Phon.
34
,
1061
1087
.
86.
Whitehill
,
T. L.
,
Stokes
,
S. F.
, and
Yonnie
,
M. Y.
(
1996
). “
Electropalatography treatment in an adult with late repair of cleft palate
,”
Cleft Palate Craniofac. J.
33
,
160
168
.

Supplementary Material

You do not currently have access to this content.