Speech perception is a multi-sensory experience. Visual information enhances [Sumby and Pollack (1954). J. Acoust. Soc. Am. 25, 212–215] and interferes [McGurk and MacDonald (1976). Nature 264, 746–748] with speech perception. Similarly, tactile information, transmitted by puffs of air arriving at the skin and aligned with speech audio, alters [Gick and Derrick (2009). Nature 462, 502–504] auditory speech perception in noise. It has also been shown that aero-tactile information influences visual speech perception when an auditory signal is absent [Derrick, Bicevskis, and Gick (2019a). Front. Commun. Lang. Sci. 3(61), 1–11]. However, researchers have not yet identified the combined influence of aero-tactile, visual, and auditory information on speech perception. The effects of matching and mismatching visual and tactile speech on two-way forced-choice auditory syllable-in-noise classification tasks were tested. The results showed that both visual and tactile information altered the signal-to-noise threshold for accurate identification of auditory signals. Similar to previous studies, the visual component has a strong influence on auditory syllable-in-noise identification, as evidenced by a 28.04 dB improvement in SNR between matching and mismatching visual stimulus presentations. In comparison, the tactile component had a small influence resulting in a 1.58 dB SNR match-mismatch range. The effects of both the audio and tactile information were shown to be additive.
References
1.
2.
Araya-Salas
, M.
, and Smith-Vidaurre
, G.
(2017
). “warbleR: An R package to streamline analysis of animal acoustic signals
,” Methods Ecol. Evol.
8
(2
), 184
.3.
Baron-Cohen
, S.
, Wheelwright
, S.
, Skinner
, R.
, Martin
, J.
, and Clubley
, E.
(2001
). “The autism-spectrum quotient (AQ): Evidence from Asperger syndrome/high-functioning autism, males and females, scientists and mathematicians
,” J. Autism Dev. Disorders
31
(1
), 5
–17
.4.
Bates
, D.
, Mächler
, M.
, Bolker
, B.
, and Walker
, S.
(2015
). “Fitting linear mixed-effects models using lme4
,” J. Stat. Softw.
67
(1
), 1
–48
.5.
Bernstein
, L. E.
, Demorest
, M. E.
, Coulter
, D. C.
, and O'Connell
, M. P.
(1991
). “Lipreading sentences with vibrotactile vocoders: Performance of normal-hearing and hearing-impaired subjects
,” J. Acoust. Soc. Am.
90
(6
), 2971
–2984
.6.
Bicevskis
, K.
, Derrick
, D.
, and Gick
, B.
(2016
). “Visual-tactile integration in speech perception: Evidence for modality neutral speech primitives
,” J. Acoust. Soc. Am.
140
(5
), 3531
–3539
.7.
Boersma
, P.
, and Weenink
, D.
(2019
). “Praat: Doing phonetics by computer (version 6.0.52) [computer program]
,” http://www.praat.org (Last viewed May 2, 2019).8.
Brainard
, D. H.
(1997
). “The psychophysics toolbox
,” Spatial Vis.
10
, 433
–436
.9.
Broadbent
, D.
(1970
). “Televised visual contribution to speech recognition
,” IEEE Trans. Educ.
13
(2
), 79
–82
.10.
Derrick
, D.
, Bicevskis
, K.
, and Gick
, B.
(2019a
). “Visual-tactile speech perception and the autism quotient
,” Front. Commun. Lang. Sci.
3
(61
), 1
–11
.11.
Derrick
, D.
, and De Rybel
, T.
(2015
). “System for audio analysis and perception enhancement
,” PCT patent no. WO 2015/122785 A1.12.
Derrick
, D.
, De Rybel
, T.
, and Fiasson
, R.
(2015
). “Recording and reproducing speech airflow outside the mouth
,” Can. Acoust.
43
(3
), 108
–109
, availabe at https://jcaa.caa-aca.ca/index.php/jcaa/issue/view/268/pdf_198.13.
Derrick
, D.
, and Gick
, B.
(2013
). “Aerotactile integration from distal skin stimuli
,” Multisens. Res.
26
, 405
–416
.14.
Derrick
, D.
, Heyne
, M.
, O'Beirne
, G.
, and Hay
, J.
(2019b
). “Aero-tactile integration in Mandarin
,” in Proceedings of the 19th International Congress of Phonetic Sciences
, August 5–9, Melbourne, Australia, pp. 3508
–3512
.15.
Derrick
, D.
, Madappallimattam
, J.
, and Theys
, C.
(2019c
). “Aero-tactile integration during speech perception: Effects of an open-choice task
,” J. Acoust. Soc. Am.
146
(3
), 1605
.16.
Derrick
, D.
, O'Beirne
, G. A.
, De Rybel
, T.
, and Hay
, J.
(2014
). “Aero-tactile integration in fricatives: Converting audio to air flow information for speech perception enhancement
,” in Proceedings of the 15th Annual Conference of the International Speech Communication Association (INTERSPEECH 2014)
, September 14–18, Singapore, pp. 2580
–2584
.17.
Derrick
, D.
, O'Beirne
, G. A.
, Gordon
, M.
, De Rybel
, T.
, Fiasson
, R.
, and Hay
, J.
(2016
). “Effects of aero-tactile stimuli on continuous speech perception
,” in 5th Joint Meeting, Acoustical Society of America and Acoustical Society of Japan
, November 28–December 2, Honolulu, HI.18.
Desai
, S.
, and Zen
, G. S. F.-G.
(2008
). “Auditory-visual speech perception in normal-hearing and cochlear-implant listeners
,” J. Acoust. Soc. Am.
123
(1
), 428
–440
.19.
Dixon
, N.
, and Spitz
, L.
(1980
). “The detection of audiovisual desynchrony
,” Perception
9
, 719
–721
.20.
Draper
, M. H.
, Ladefoged
, P.
, and Whitteridge
, D.
(1960
). “Expiratory pressures and air flow during speech
,” Br. Med. J.
18
(5189
), 1837
–1843
.21.
Erber
, N.
(1969
). “Interaction of audition and vision in the recognition of oral speech stimuli
,” J. Speech Hear. Res.
12
, 423
–425
.22.
Ewertsen
, H.
, and Nielsen
, H. B.
(1971
). “A comparative analysis of the audiovisual, auditive and visual perception of speech
,” Acta Otolaryngol.
72
, 201
–205
.23.
Feldman
, J. I.
, Dunham
, K.
, Cassidy
, M.
, Wallace
, M. T.
, Liu
, Y.
, and Woynaroski
, T. G.
(2018
). “Audiovisual multisensory integration in individuals with autism spectrum disorder: A systematic review and meta-analysis
,” Neurosci. Biobehav. Rev.
95
, 220
–234
.24.
25.
Gick
, B.
, and Derrick
, D.
(2009
). “Aero-tactile integration in speech perception
,” Nature
462
, 502
–504
.26.
Gick
, B.
, Ikegami
, Y.
, and Derrick
, D.
(2010
). “The temporal window of audio-tactile integration in speech perception
,” J. Acoust. Soc. Am.
128
(5
), EL342
–EL346
.27.
Gick
, B.
, Jóhannsdóttir
, K. M.
, Gibraiel
, D.
, and Mühlbauer
, J.
(2008
). “Tactile enhancement of auditory and visual speech perception in untrained perceivers
,” J. Acoust. Soc. Am.
123
(4
), EL72
–EL76
.28.
Goldenberg
, D.
, Tiede
, M.
, and Whalen
, D. H.
(2018
). “Concurrent aero-tactile stimulation does not bias perception of vot for non-initial stops
,” J. Acoust. Soc. Am.
144
(3
), 1801
–1801
.29.
Goldenberg
, D.
, Tiede
, M. K.
, and Whalen
, D. H.
(2015
). “Aero-tactile influence on speech perception of voicing continua
,” in Proceedings of the 18th International Congress of the Phonetic Sciences (ICPhS2015)
, August 10–14, Glasgow, Scotland.30.
Huyse
, A.
, Leybaert
, J.
, and Berthommier
, F.
(2014
). “Effects of aging on audio-visual speech integration
,” J. Acoust. Soc. Am.
136
(4
), 1918
–1931
.31.
Jansen
, S.
, Luts
, H.
, Wagener
, K. C.
, Frachet
, B.
, and Wouters
, J.
(2010
). “The French digit triplet test: A hearing screening tool for speech intelligibility in noise
,” Int. J. Audiol.
49
(5
), 378
–387
.32.
Kaganovich
, N.
, Schumaker
, J.
, and Rowland
, C.
(2016
). “Matching heard and seen speech: An ERP study of audiovisual word recognition
,” Brain Lang.
157–158
, 14
–24
.33.
Kleiner
, M.
, Brainard
, D.
, and Pelli
, D.
(2007
). “What's new in psychtoolbox-3?
,” in Perception Thirtieth European Conference on Visual Perception Abstract Supplement
, August 27–31, Arezzo, Italy.34.
Kuznetsova
, A.
, Brockhoff
, P. B.
, and Christensen
, R. H. B.
(2017
). “lmerTest package: Tests in linear mixed effects models
,” J. Stat. Softw.
82
(13
), 1
–26
.35.
Lisker
, L.
(1957
). “Closure duration and the intervocalic voiced-voiceless distinction in English
,” Ling. Soc. Am.
33
(1
), 42
–49
.36.
Lisker
, L.
, and Abramson
, A. S.
(1966
). “Some effects of context on voice onset time in English stops
,” Lang. Speech
10
, 1
–28
.37.
Mallick
, D. B.
, Magnotti
, J. F.
, and Beauchamp
, M. S.
(2015
). “Variability and stability in the McGurk effect: Contributions of participants, stimuli, time, and response type
,” Psychonom. Bull. Rev.
22
(5
), 1299
–1300
.38.
Massaro
, D. W.
(1998
). Perceiving Talking Faces: From Speech Perception to a Behavioural Principle
(MIT Press
, Cambridge, MA
), Vol. 1.39.
McGurk
, H.
, and MacDonald
, J.
(1976
). “Hearing lips and seeing voices
,” Nature
264
, 746
–748
.40.
Munhall
, K. G.
, Gribble
, P.
, Sacco
, L.
, and Ward
, M.
(1996
). “Temporal constraints on the McGurk effect
,” Atten. Percept. Psychophys.
58
(3
), 351
–362
.41.
Peelle
, J. E.
, and Sommers
, M. S.
(2015
). “Prediction and constraint in audiovisual speech perception
,” Cortex
68
, 169
–181
.42.
Pelli
, D. G.
(1987
). “The ideal psychometric procedure
,” Investig. Opthalmol. Visual Sci.
20
, 366
.43.
Pelli
, D. G.
(1997
). “The videotoolbox software for visual psychophysics: Transforming numbers into movies
,” Spatial Vis.
10
, 437
–442
.44.
R Development Core Team
(2018
). R: A Language and Environment for Statistical Computing
(R Foundation for Statistical Computing
, Vienna, Austria
).45.
Ross
, L. A.
, Molholm
, S.
, Blanco
, D.
, Gomez-Ramirez
, M.
, Saint-Amour
, D.
, and Foxe
, J. J.
(2011
). “The development of multisensory speech perception continues into the late childhood years
,” Eur. J. Neurosci.
33
, 2329
–2337
.46.
Rothenberg
, M.
, Verillo
, R. T.
, Zahorian
, S. A.
, Brachman
, M. L.
, and Bolanowski
, S. J.
, Jr. (1977
). “Vibrotactile frequency for encoding a speech parameter
,” J. Acoust. Soc. Am.
62
(4
), 1003
–1012
.47.
Sekiyama
, K.
, Kanno
, I.
, Miura
, S.
, and Sugita
, Y.
(2003
). “Auditory-visual speech perception examined by fMRI and PET
,” Neurosci. Res.
47
, 277
–287
.48.
Sekiyama
, K.
, Soshi
, T.
, and Sakamoto
, S.
(2014
). “Enhanced audiovisual integration with aging in speech perception: A heightened mcgurk effect in older adults
,” Front. Psychol.
5
(323
), 1
–12
.49.
Smeele
, P. M. T.
, Sittig
, A. C.
, and Heuven
, V. J. V.
(1992
). “Intelligibility of audio-visually desynchronized speech: Asymmetrical effect of phoneme position
,” in Proceedings of the International Conference on Spoken Language Processing
, October 13–16, Alberta, Canada, pp. 65
–68
.50.
Smits
, C.
, Kapteyn
, T. S.
, and Houtgast
, T.
(2004
). “Development and validation of an automatic speech-in-noise screening test by telephone
,” Int. J. Audiol.
43
(1
), 15
–28
.51.
Stathopoulos
, E. T.
, and Weismer
, G.
(1985
). “Oral airflow and air pressure during speech production: A comparative study of children, youths and adults
,” Folia Phon. Logopaed.
37
(3–4
), 152
–159
.52.
Sumby
, W. H.
, and Pollack
, I.
(1954
). “Visual contribution to speech intelligibility in noise
,” J. Acoust. Soc. Am.
26
, 212
–215
.53.
Summerfield
, Q.
(1992
). “Lipreading and audio-visual speech perception
,” Philos. Trans. R. Soc. Lond. Ser. B
335
, 71
–78
.54.
Tathan
, M.
, and Morton
, K.
(1973
). “Electromyographic and intraoral air pressure studies of bilabial stops
,” Lang. Speech
16
(4
), 336
–350
.55.
The MathWorks, Inc
. (2014
). MATLAB and Statistics Toolbox Release 2014b
(The MathWorks, Inc
., Natick, MA)
.56.
Tremblay
, C.
, Champoux
, F.
, Voss
, P.
, Bacon
, B. A.
, Lepore
, F.
, and Théoret
, H.
(2007
). “Speech and non-speech audio-visual illusions: A developmental study
,” PLoS ONE
8
, e742
.57.
van Wassenhove
, V.
(2013
). “Speech through the ears and eyes: Interfacing the senses with the supramodal brain
,” Front. Psychol.
4
(368
), 1
–17
.58.
van Wassenhove
, V.
, Grant
, K. W.
, and Poeppel
, D.
(2007
). “Temporal window of integration in auditory-visual speech perception
,” Neuropsychologia
45
, 598
–607
.59.
Venezia
, J. H.
, Thurman
, S. M.
, Matchin
, W.
, George
, S. E.
, and Hickok
, G.
(2016
). “Timing in audiovisual speech perception: A mini review and new psychophysical data
,” Atten. Percept. Psychophys.
78
(2
), 583
–601
.60.
Watson
, A. B.
(1983
). “QUEST: A Bayesian adaptive psychometric method
,” Percept. Psychophys.
33
(2
), 113
–120
.© 2019 Acoustical Society of America.
2019
Acoustical Society of America
You do not currently have access to this content.