This study examines the influence of the position of prosodic heads (accented syllables) and prosodic edges (prosodic word and intonational phrase boundaries) on the timing of head movements. Gesture movements and prosodic events tend to be temporally aligned in the discourse, the most prominent part of gestures typically being aligned with prosodically prominent syllables in speech. However, little is known about the impact of the position of intonational phrase boundaries on gesture-speech alignment patterns. Twenty-four Catalan speakers produced spontaneous (experiment 1) and semi-spontaneous head gestures with a confirmatory function (experiment 2), along with phrase-final focused words in different prosodic conditions (stress-initial, stress-medial, and stress-final). Results showed (a) that the scope of head movements is the associated focused prosodic word, (b) that the left edge of the focused prosodic word determines where the interval of gesture prominence starts, and (c) that the speech-anchoring site for the gesture peak (or apex) depends both on the location of the accented syllable and the distance to the upcoming intonational phrase boundary. These results demonstrate that prosodic heads and edges have an impact on the timing of head movements, and therefore that prosodic structure plays a central role in the timing of co-speech gestures.

1.
Ahmad
,
M. I.
,
Tariq
,
H.
,
Saeed
,
M.
,
Shahid
,
S.
, and
Krahmer
,
E.
(
2011
). “
Guess who? An interactive and entertaining game-like platform for investigating human emotions
,” in
Human Computer Interaction. Towards Mobile and Intelligent Interaction Environments
, Lecture Notes in Computer Science 6763, edited by
J. A.
Jacko
(
Springer
,
Berlin, Germany
), Vol.
3
, pp.
543
551
.
2.
Alexanderson
,
S.
,
House
,
D.
, and
Beskow
,
J.
(
2013
). “
Aspects of co-occurring syllables and head nods in spontaneous dialogue
,” in
Proceedings of 12th International Conference on Auditory-Visual Speech Processing (AVSP2013)
.
3.
Ambrazaitis
,
G.
,
Svensson Lundmark
,
M.
, and
House
,
D.
(
2015
). “
Head movements, eyebrows, and phonological prosodic prominence levels in Stockholm Swedish news broadcasts
,” in
FAAVSP - The 1st Joint Conference on Facial Analysis, Animation, and Auditory-Visual Speech Processing
,
Vienna, Austria
, pp.
42
42
.
4.
Barkhuysen
,
P.
,
Krahmer
,
E.
, and
Swerts
,
M.
(
2008
). “
The interplay between the auditory and visual modality for end-of-utterance detection
,”
J. Acoust. Soc. Am.
123
,
354
365
.
5.
Bates
,
D.
,
Maechler
,
M.
, and
Bolker
,
B.
(
2011
). “
lme4: Linear mixed-effects models using S4 classes [R package version 0.99375-39]
,” http://CRAN.R-project.org/package-lme4 (Last viewed January 27, 2017).
6.
Bergmann
,
K.
,
Aksu
,
V.
, and
Kopp
,
S.
(
2011
). “
The relation of speech and gestures: Temporal synchrony follows semantic synchrony
,” in
Proceedings of the 2nd Workshop on Gesture and Speech in Interaction
, pp.
1
6
.
7.
Bergmann
,
K.
,
Kahl
,
S.
, and
Kopp
,
S.
(
2014
). “
How is information distributed across speech and gesture? A cognitive modeling approach
,”
Cognit. Processing
15
(
1
),
S84
S87
.
8.
Billmyer
,
K.
, and
Varghese
,
M.
(
2000
). “
Investigating instrument-based pragmatic variability: Effects of enhancing discourse completion tests
,”
Appl. Linguist.
21
(
4
),
517
552
.
9.
Blum-Kulka
,
S.
,
House
,
J.
, and
Kasper
,
G.
(
1989
). “
Investigating cross- cultural pragmatics: An introductory overview
,” in
Cross-Cultural Pragmatics: Requests and Apologies
, edited by
S.
Blum-Kulka
,
J.
House
, and
G.
Kasper
(
Ablex
,
Norwood, NJ
), pp.
1
34
.
10.
Boersma
,
P.
, and
Weenink
,
D.
(
2012
). “
Praat: Doing phonetics by computer
,” http://www.praat.org/ (Last viewed July 25, 2016).
11.
Butterworth
,
B.
, and
Beattie
,
G.
(
1978
). “
Gesture and silence as indicators of planning in speech
,” in
Recent Advances in the Psychology of Language: Formal and Experimental Approaches
, edited by
R.
Campbell
and
G. T.
Smith
(
Plenum Press
,
New York
), pp.
347
360
.
12.
De Ruiter
,
J. P.
(
1998
). “
Gesture and speech production
,” doctoral dissertation,
Katholieke Universiteit
,
Nijmegen, the Netherlands
.
13.
Dohen
,
M.
, and
Loevenbruck
,
H.
(
2004
). “
Pre-focal rephrasing, focal enhancement and post-focal deaccentuation in French
,” in
Proceedings of the 8th International Conference on Spoken Language Processing
, pp.
2
5
.
14.
Esteve-Gibert
,
N.
,
Pons
,
F.
,
Bosch
,
L.
, and
Prieto
,
P.
(
2014
). “
Are gesture and prosodic prominences always coordinated? Evidence from perception and production
,” in
Proceedings of the Speech Prosody Conference
, edited by
N.
Campbell
,
D.
Gibbon
, and
D.
Hirst
, pp.
222
226
.
15.
Esteve-Gibert
,
N.
, and
Prieto
,
P.
(
2013
). “
Prosodic structure shapes the temporal realization of intonation and manual gesture movements
,”
J. Speech Language Hear. Res.
56
,
850
864
.
16.
Fernández-Baena
,
A.
,
Montaño
,
R.
,
Antonijoan
,
M.
,
Roversi
,
A.
,
Miralles
,
D.
, and
Alías
,
F.
(
2014
). “
Gesture synthesis adapted to speech emphasis
,”
Speech Commun.
57
,
331
350
.
17.
Ferré
,
G.
(
2014
). “
A multimodal approach to markedness in spoken French
,”
Speech Commun.
57
,
268
282
.
18.
Goldenberg
,
D.
,
Tiede
,
M.
,
Honorof
,
D. N.
, and
Mooshammer
,
C.
(
2014
). “
Temporal alignment between head gesture and prosodic prominence in naturally occurring conversation: An electromagnetic articulometry study
,”
J. Acoust. Soc. Am.
135
,
2294
.
19.
Graf
,
H. P.
,
Cosatto
,
E.
,
Strom
,
V.
, and
Huang
,
F. J.
(
2002
). “
Visual prosody: Facial movements accompanying speech
,” in
Proceedings of the 5th IEEE International Conference on Automatic Face Gesture Recognition
, pp.
396
401
.
20.
Guellaï
,
B.
,
Langus
,
A.
, and
Nespor
,
M.
(
2014
). “
Prosody in the hands of the speaker
,”
Front. Psychol.
5
,
1
8
.
21.
Hadar
,
U.
,
Steiner
,
T. J.
,
Grant
,
E. C.
, and
Rose
,
F. C.
(
1983
). “
Kinematics of head movements accompanying speech during conversation
,”
Human Movement Sci.
2
(
1–2
),
35
46
.
22.
House
,
D.
,
Beskow
,
J.
, and
Granström
,
B.
(
2001
). “
Timing and interaction of visual cues for prominence in audiovisual speech perception
,” in
Proceedings of Eurospeech
, pp.
387
390
.
23.
Ishi
,
C. T.
,
Ishiguro
,
H.
, and
Hagita
,
N.
(
2014
). “
Analysis of relationship between head motion events and speech in dialogue conversations
,”
Speech Commun.
57
,
233
243
.
24.
Jannedy
,
S.
, and
Mendoza-Denton
,
N.
(
2005
). “
Structuring Information through Gesture and Intonation
,”
Interdisciplinary Stud. Inf. Struct.
3
,
199
244
.
25.
Kelly
,
S. D.
,
Ozyürek
,
A.
, and
Maris
,
E.
(
2010
). “
Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension
,”
Psychol. Sci.
21
(
2
),
260
267
.
26.
Kendon
,
A.
(
1980
). “
Gesticulation and speech: Two aspects of the process of utterance
,” in
The Relationship of Verbal and Nonverbal Communication
, edited by
M. R.
Key
(
Mouton
,
the Hague, the Netherlands
), pp.
207
227
.
27.
Kim
,
J.
,
Cvejic
,
E.
, and
Davis
,
C.
(
2014
). “
Tracking eyebrows and head gestures associated with spoken prosody
,”
Speech Commun.
57
,
317
330
.
28.
Krahmer
,
E.
, and
Swerts
,
M.
(
2007
). “
The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception
,”
J. Mem. Language
57
(
3
),
396
414
.
29.
Krivokapić
,
J.
(
2014
). “
Gestural coordination at prosodic boundaries and its role for prosodic structure and speech planning processes
,”
Philos. Trans. R. Soc. London Ser. B Biol. Sci.
369
(
1658
),
20130397
.
30.
Krivokapić
,
J.
,
Tiede
,
M. K.
, and
Tyrone
,
M. E.
(
2015
). “
A kinematic analysis of prosodic structure in speech and manual gestures
,” in
Proceedings of the 18th International Congress of Phonetic Sciences
.
31.
Krivokapić
,
J.
,
Tiede
,
M. K.
,
Tyrone
,
M. E.
, and
Goldenberg
,
D.
(
2016
). “
Speech and manual gesture coordination in a pointing task
,” in
Proceedings of the 8th International Conference on Speech Prosody
, pp.
1240
1244
.
32.
Lausberg
,
H.
, and
Sloetjes
,
H.
(
2009
). “
Coding gestural behavior with the NEUROGES-ELAN system
,”
Behav. Res. Methods Instrum. Comput.
41
(
3
),
841
849
.
33.
Leonard
,
T.
, and
Cummins
,
F.
(
2011
). “
The temporal relation between beat gestures and speech
,”
Lang. Cognit. Processes
26
(
10
),
1457
1471
.
34.
Levelt
,
W. J. M.
,
Richardson
,
G.
, and
La Heij
,
W.
(
1985
). “
Pointing and voicing in deictic expressions
,”
J. Mem. Language
24
,
133
164
.
35.
Loehr
,
D. P.
(
2012
). “
Temporal, structural, and pragmatic synchrony between intonation and gesture
,”
Lab. Phonol.
3
,
71
89
.
36.
McClave
,
E. Z.
(
2000
). “
Linguistic functions of head movements in the context of speech
,”
J. Pragmatics
32
,
855
878
.
37.
McNeill
,
D.
(
1992
).
Hand and Mind: What Gestures Reveal About Thought
(
University of Chicago Press
,
Chicago, IL
).
38.
Nobe
,
S.
(
2000
). “
Where to most spontaneous representational gestures actually occur with respect to speech?
,” in
Language and Gesture
, edited by
D.
McNeill
(
Cambridge University Press
,
Cambridge, UK
), pp.
186
198
.
39.
Özyürek
,
A.
,
Willems
,
R. M.
,
Kita
,
S.
, and
Hagoort
,
P.
(
2007
). “
On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials
,”
J. Cognit. Neurosci.
19
(
4
),
605
616
.
40.
Poggi
,
I.
,
D'Errico
,
F.
, and
Vincze
,
L.
(
2011
). “
68 Nods. But not only of agreement
,” in
68 Zeichen Für Roland Posner. Ein Semiotisches Mosaik. (68 Signs for Roland Posner. A Semiotic Mosaic
) (
Stauffenburg Verlag
,
Tübingen, Germany
).
41.
Poggi
,
I.
,
D'Errico
,
F.
,
Vincze
,
L.
, and
Milazzo
,
V.
(
2010
). “
Types of nods. The polysemy of a social signal
,” in
Proceedings of the Seventh conference on International Language Resources and Evaluation (LREC'10)
, Malta.
42.
Prieto
,
P.
, and
Ortega-Llebaria
,
M.
(
2009
). “
Do complex pitch gestures induce syllable lengthening in Catalan and Spanish?
,” in
Phonetics and Phonology: Interactions and Interrelations
, edited by
M.
Vigário
,
S.
Frota
, and
M. J.
Freitas
(
John Benjamins
,
Philadelphia, PA
), pp.
51
70
.
43.
Prieto
,
P.
,
Puglesi
,
C.
,
Borràs-Comes
,
J.
,
Arroyo
,
E.
, and
Blat
,
J.
(
2015
). “
Exploring the contribution of prosody and gesture to the perception of focus using an animated agent
,”
J. Phonetics
49
,
41
54
.
44.
Rochet-Capellan
,
A.
,
Laboissière
,
R.
,
Galván
,
A.
, and
Schwartz
,
J.
(
2008
). “
The speech focus position effect on jaw-finger coordination in a pointing task
,”
J. Speech Language Hearing Res.
51
(
6
),
1507
1521
.
45.
Roustan
,
B.
, and
Dohen
,
M.
(
2010
). “
Gesture and speech coordination: The influence of the relationship between manual gesture and speech
,” in
Proceedings of 11th Annual Conference of the International Speech Communication Association (INTERSPEECH 2010)
, Makuhari, Japan.
46.
Rusiewicz
,
H. L.
,
Shaiman
,
S.
,
Iverson
,
J. M.
, and
Szuminsky
,
N.
(
2013
). “
Effects of prosody and position on the timing of deictic gestures
,”
J. Speech Language Hear. Res.
56
(
2
),
458
470
.
47.
Schegloff
,
E. A.
(
1984
). “
On some gestures' relation to talk
,” in
Structures of Social Action
, edited by
J. M.
Atkinson
and
J.
Heritage
(
Cambridge University Press
,
Cambridge, UK
), pp.
266
298
.
48.
Shattuck-Hufnagel
,
S.
,
Ren
,
P. L.
, and
Tauscher
,
E.
(
2010
). “
Are torso movements during speech timed with intonational phrases?
,” in
Proceedings of the Speech Prosody 2010
, Chicago, IL.
49.
Swerts
,
M.
, and
Krahmer
,
E.
(
2010
). “
Visual prosody of newsreaders: Effects of information structure, emotional content and intended audience on facial expressions
,”
J. Phonetics
38
,
197
206
.
50.
Treffner
,
P.
,
Peter
,
M.
, and
Kleidon
,
M.
(
2008
). “
Gestures and phases: The dynamics of speech-hand communication
,”
Ecol. Psychol.
20
(
1
),
32
64
.
51.
Wagner
,
P.
,
Malisz
,
Z.
, and
Kopp
,
S.
(
2014
). “
Gesture and speech in interaction: An overview
,”
Speech Commun.
57
,
209
232
.
52.
Yasinnik
,
Y.
,
Renwick
,
M.
, and
Shattuck-Hufnagel
,
S.
(
2004
). “
The timing of speech-accompanying gestures with respect to prosody
,” in
Proceedings From Sound to Sense: 50+ Years of Discoveries in Speech Communication
(
MIT
,
Cambridge, MA
), pp.
C97
C102
.
You do not currently have access to this content.