Although many audio-visual speech experiments have focused on situations where the presence of an incongruent visual speech signal influences the perceived utterance heard by an observer, there are also documented examples of a related effect in which the presence of an incongruent audio speech signal influences the perceived utterance seen by an observer. This study examined the effects that different distracting audio signals had on performance in a color and number keyword speechreading task. When the distracting sound was noise, time-reversed speech, or continuous speech, it had no effect on speechreading. However, when the distracting audio signal consisted of speech that started at the same time as the visual stimulus, speechreading performance was substantially degraded. This degradation did not depend on the semantic similarity between the target and masker speech, but it was substantially reduced when the onset of the audio speech was shifted relative to that of the visual stimulus. Overall, these results suggest that visual speech perception is impaired by the presence of a simultaneous mismatched audio speech signal, but that other types of audio distracters have little effect on speechreading performance.
REFERENCES
1.
Burnham
, D.
, and Dodd
, B.
(1996
). “Audio-visual speech perception as a direct process: the McGurk effect in infants and across languages
,” in Speechreading by Man and Machine: Data, Models, and Systems
, edited by D.
Stork
and M.
Hennecke
(Springer-Verlag
, New York), pp. 103
–114
.2.
Conrey
, B.
, and Pisoni
, D.
(2003
). “Audiovisual asynchrony detection for speech and nonspeech signals
,” in Proceedings of the 2003 Audio-Visual Speech Processing Workshop
.3.
4.
Dekle
, D.
, Fowler
, C.
, and Funnell
, M.
(1992
). “Audiovisual integration in the perception of real words
,” Percept. Psychophys.
51
, 335
–362
.5.
Dixon
, N.
, and Spitz
, L.
(1980
). “The detection of auditory visual desynchrony
,” Perception
9
, 719
–721
.6.
Easton
, R.
, and Basala
, M.
(1982
). “Perceptual dominance during lipreading
,” Percept. Psychophys.
32
, 562
–570
.7.
Ernst
, M.
, and Bülthoff
, H.
(2004
). “Merging the senses into a robust precept
,” Trends in Cognitive Science
8
, 162
–169
.8.
Fuster-Duran
, A.
(1996
). “Perception of conflicting audio-visual speech: an examination across Spanish and German
,” in Speechreading by Man and Machine: Data, Models, and Systems
, edited by D.
Stork
and M.
Hennecke
(Springer-Verlag
, New York), pp. 135
–143
.9.
Gelder
, B.
, and Bertlson
, P.
(2003
). “Multisensory integration, perception and ecological validity
,” Trends in Cognitive Science
7
, 460
–467
.10.
Grant
, K.
, and Walden
, B.
(1996
). “Evaluating the articulation index for auditory-visual consonant recognition
,” J. Acoust. Soc. Am.
100
, 2415
–2424
.11.
Grant
, K. W.
, Greenberg
, S.
, Poeppel
, D.
, and van Wassenhove
, V.
(2004a
). “Effects of spectrotemporal asynchrony in auditory and auditory-visual speech processing
,” in Seminars in Hearing 25
, pp. 241
–255
.12.
Grant
, K. W.
, van Wassenhove
, V.
, and Poeppel
, D.
(2004b
). “Detection of auditory (crossspectral) and auditory-visual (cross-modal) synchrony
,” Speech Commun.
Special Issue on Audio Visual Speech Processing, edited by J.-L.
Schwartz
, F.
Bertommier
, M.-A.
Cathiard
, and R.
de Mori
44
, pp. 43
–53
.13.
Green
, K.
(1996
). “The use of auditory an visual information in phonetic perception
,” Speechreading by Man and Machine: Data, Models, and Systems
, edited by D.
Stork
and M.
Hennecke
(Springer-Verlag
, New York), pp. 55
–77
.14.
Jones
, J.
, and Munhall
, K.
(1997
). “The effects of separating auditory and visual sources on audiovisual integration of speech
,” Can. Acoust.
25
, 13
–19
.15.
Lyxell
, B.
, and Ronnberg
, J.
(1993
). “The effects of background noise and working memory capacity on speechreading performance
,” Scand. Audiol.
22
, 67
–70
.16.
Markides
, A.
(1989
). “Background noise and lip-reading ability
,” Br. J. Audiol.
23
, 251
–253
.17.
Massaro
, D.
(1987
). Hearing by Eye: A paradigm for Psychological Inquiry
(Erlbaum
, Mahwah, NJ).18.
Massaro
, D.
(1998
). Perceiving Talking Faces: From speech perception to a Behavioral Principle
(MIT
, Cambridge, MA).19.
Massaro
, D.
, Cohen
, M.
, and Smeele
, P.
(1996
). “Evaluating the articulation index for auditory-visual consonant recognition
,” J. Acoust. Soc. Am.
100
, 1777
–1786
.20.
McAdams
, S.
, and Bregman
, A.
(1979
). “Hearing musical streams
,” J. Comput. Music
3
, 26
–44
.21.
McGurk
, H.
, and McDonald
, J.
(1976
). “Hearing lips and seeing voices
,” Nature (London)
264
, 746
–748
.22.
Moore
, T.
(1981
). “Voice communication jamming research
,” in AGARD Conference Proceedings 331: Aural Communication in Aviation
, Neuilly-Sur-Seine
, France, pp. 2
–1
2
–6
.23.
Munhall
, K.
, Gribble
, P.
, Sacco
, L.
, and Ward
, M.
(1996
). “Temporal constraints on the McGurk effect
,” Percept. Psychophys.
58
, 351
–362
.24.
Oxenham
, A.
, and Dau
, T.
(2001
). “Modulation detection interference: Effects of concurrent and sequential streaming
,” J. Acoust. Soc. Am.
110
, 402
–408
.25.
Pandey
, P.
, Kunov
, H.
, and Abel
, S.
(1986
). “Disruptive effects of auditory signal delay on speechreading
,” J. Aud Res.
26
, 27
–41
.26.
Reisberg
, D.
(1978
). “Looking where you listen: Visual cues and auditory attention
,” Acta Psychol.
42
, 331
–341
.27.
Schwartz
, J.-L.
, Robert-Ribes
, J.
, and Escudier
, P.
(1998
). “Ten years after Summerfield: a taxonomy of models for audio-visual fusion in speech perception
,” in Hearing by Eye (II): The Psychology of Speechreading and Auditory-visual Speech
, edited by B. D. R.
Campbell
and D.
Burnham
(Psychology
, East Sussex, UK), pp. 85
–108
.28.
Sumby
, W.
, and Pollack
, I.
(1954
). “Visual contribution to speech intelligibility in noise
,” J. Acoust. Soc. Am.
26
, 212
–215
.29.
Summerfield
, Q.
(1987
). “Some preliminaries to a comprehensive account of audio-visual speech perception
,” in Hearing by Eye: The Psychology of Lipreading
, edited by B.
Dodd
and R.
Campbell
(Erlbaum
, New York), pp. 3
–51
.30.
Summerfield
, Q.
(1992
). “Lipreading and audio-visual speech perception
,” Philos. Trans. R. Soc. London
335
, 71
–78
.31.
Tiippana
, K.
, Sams
, M.
, and Andersen
, T. S.
(2001
). Visual Attention Influences Audiovisual Speech Perception, in Proceedings of the 2001 Audio-Visual Speech Processing Workshop
, pp. 167
–171
.32.
Walker
, S.
, Bruce
, V.
, and O’Malley
, C.
(1995
). “Facial identity and facial speech processing: Familiar faces and voices in the McGurk effect
,” Percept. Psychophys.
57
, 1124
–1133
.2005
U.S. Government
You do not currently have access to this content.
