Audio-visual [McGurk and MacDonald (1976). Nature 264, 746–748] and audio-tactile [Gick and Derrick (2009). Nature 462(7272), 502–504] speech stimuli enhance speech perception over audio stimuli alone. In addition, multimodal speech stimuli form an asymmetric window of integration that is consistent with the relative speeds of the various signals [Munhall, Gribble, Sacco, and Ward (1996). Percept. Psychophys. 58(3), 351–362; Gick, Ikegami, and Derrick (2010). J. Acoust. Soc. Am. 128(5), EL342–EL346]. In this experiment, participants were presented video of faces producing /pa/ and /ba/ syllables, both alone and with air puffs occurring synchronously and at different timings up to 300 ms before and after the stop release. Perceivers were asked to identify the syllable they perceived, and were more likely to respond that they perceived /pa/ when air puffs were present, with asymmetrical preference for puffs following the video signal—consistent with the relative speeds of visual and air puff signals. The results demonstrate that visual-tactile integration of speech perception occurs much as it does with audio-visual and audio-tactile stimuli. This finding contributes to the understanding of multimodal speech perception, lending support to the idea that speech is not perceived as an audio signal that is supplemented by information from other modes, but rather that primitives of speech perception are, in principle, modality neutral.

1.
Abel
,
J.
,
Barbosa
,
A. V.
,
Mayer
,
C.
, and
Vatikiotis-Bateson
,
E.
(
2011
). “
The labial visemereconsidered: Evidencce from production and perception
,” in
9th International Seminar on Speech Production (ISSP)
, edited by
Y.
Laprie
and
I.
Steiner
,
Montreal, Quebec
, pp.
337
344
.
2.
Alcorn
,
S.
(
1932
). “
The Tadoma method
,”
Volta Rev.
34
,
195
198
.
3.
Auer
,
E. T.
, Jr.
, and
Bernstein
,
L. E.
(
1997
). “
Speechreading and the structure of the lexicon: Computationally modeling the effects of reduced phonetic distinctiveness on lexical uniqueness
,”
J. Acoust. Soc. Am.
102
,
3704
3710
.
4.
Bates
,
D.
,
Maechler
,
M.
,
Bolker
,
B.
, and
Walker
,
S.
(
2014
). “
From _lme4: Linear mixed-effects models using Eigen and S4_
,” R package version 1.1-7, http://CRAN.R-project.org/package=lme4 (Last viewed 2/13/2016).
5.
Bernstein
,
L. E.
, and
Liebenthal
,
E.
(
2014
). “
Neural pathways for visual speech perception
,”
Front. Neurosci.
8
(
386
),
1
18
.
6.
Boersma
,
P.
, and
Weenink
,
D.
(
2015
). “
Praat: Doing phonetics by computer
” [computer program], version 5.4.08, http://www.praat.org/ (Last viewed 3/24/2015).
7.
Calvert
,
G. A.
,
Bullmore
,
E. T.
,
Brammer
,
M. J.
,
Campbell
,
R.
,
Williams
,
S. C.
,
McGuire
,
P. K.
,
Woodruff
,
P. W. R.
,
Iversen
,
S. D.
, and
David
,
A. S.
(
1997
). “
Activation of auditory cortex during silent lipreading
,”
Science
276
(
5312
),
593
596
.
8.
Derrick
,
D.
,
Anderson
,
P.
,
Gick
,
B.
, and
Green
,
S.
(
2009
). “
Characteristics of air puffs produced in English “pa”: Experiments and simulations
,”
J. Acoust. Soc. Am.
125
(
4
),
2272
2281
.
9.
Derrick
,
D.
, and
Gick
,
B.
(
2013
). “
Aerotactile integration from distal skin stimuli
,”
Multisens. Res.
26
,
405
416
.
10.
Diehl
,
R. L.
,
Lotto
,
A. J.
, and
Holt
,
L. L.
(
2004
). “
Speech perception
,”
Annu. Rev. Psychol.
55
,
149
179
.
11.
Fisher
,
C.
(
1968
). “
Confusion among visually perceived consonants
,”
J. Speech Hear. Res.
11
,
796
804
.
12.
Fowler
,
C.
, and
Dekle
,
D.
(
1991
). “
Listening with eye and hand: Cross-modal contributions to speech perception
,”
J. Exp. Psychol.: Hum. Percept. Perform.
17
(
3
),
816
828
.
13.
Fowler
,
C. A.
(
1986
). “
An event approach to the study of speech perception from a direct-realist perspective
,” in
Status Report on Speech Research
, edited by
I. G.
Mattingly
and
N.
O'Brien
,
Haskins Laboratories
,
New Haven, CT
, pp.
139
169
.
14.
Fowler
,
C. A.
(
2004
). “
Speech as a supramodal or amodal phenomenon
,” in
The Handbook of Multisensory Processes
, edited by
G. A.
Calvert
,
C.
Spence
, and
B. E.
Stein
(
MIT Press
,
Cambridge, MA
), pp.
189
201
.
15.
Gick
,
B.
, and
Derrick
,
D.
(
2009
). “
Aero-tactile integration in speech perception
,”
Nature
462
(
7272
),
502
504
.
16.
Gick
,
B.
,
Ikegami
,
Y.
, and
Derrick
,
D.
(
2010
). “
The temporal window of audio-tactile integration in speech perception
,”
J. Acoust. Soc. Am.
128
(
5
),
EL342
EL346
.
17.
Gick
,
B.
,
Jóhannsdóttir
,
K.
,
Gibraiel
,
D.
, and
Mühlbauer
,
J.
(
2008
). “
Tactile enhancement of auditory and visual speech perception in untrained perceivers
,”
J. Acoust. Soc. Am.
123
(
4
),
EL72
EL76
.
18.
Hayden
,
R. E.
(
1950
). “
The relative frequency of phonemes in general-American English
,”
Word
6
(
3
),
217
223
.
19.
Mayer
,
C.
,
Gick
,
B.
,
Weigel
,
T.
, and
Whalen
,
D.
(
2013
). “
Perceptual integration of visual evidence of the airstream from aspirated stops
,”
Can. Acoust.
41
(
3
),
23
27
.
20.
McGurk
,
H.
, and
MacDonald
,
J.
(
1976
). “
Hearing lips and seeing voices
,”
Nature
264
,
746
748
.
21.
Munhall
,
K.
,
Gribble
,
P.
,
Sacco
,
L.
, and
Ward
,
M.
(
1996
). “
Temporal constraints on the McGurk effect
,”
Percept. Psychophys.
58
(
3
),
351
362
.
22.
Peirce
,
J.
(
2007
). “
PsychoPy—Psychophysics software in Python
,”
J. Neurosci. Meth.
162
(
1-2
),
8
13
.
23.
R Core Team
. (
2014
). “
R: A language and environment for statistical computing
,” R Foundation for Statistical Computing, Vienna, Austria, http://www.R-project.org/ (Last viewed 2/13/2016).
24.
Rosenblum
,
L. D.
(
2008
). “
Speech perception as a multimodal phenomenon
,”
Curr. Direct. Psychol. Sci.
17
(
6
),
405
409
.
25.
Rosenblum
,
L. D.
,
Pisoni
,
D. B.
, and
Remez
,
R.
(
2005
). “
Primacy of multimodal speech perception
,” in
The Handbook of Speech Perception
(
Blackwell
,
Malden, MA
), pp.
51
78
.
26.
Sumby
,
W.
, and
Pollack
,
I.
(
1954
). “
Visual contribution to speech intelligibility in noise
,”
J. Acoust. Soc. Am.
26
(
2
),
212
215
.
27.
van Wassenhove
,
V.
,
Grant
,
K.
, and
Poeppel
,
D.
(
2007
). “
Temporal window of integration in auditory-visual speech perception
,”
Neuropsychologia
45
(
4
),
598
607
.
You do not currently have access to this content.