Expressive moments in communicative hand gestures often align with emphatic stress in speech. It has recently been found that acoustic markers of emphatic stress arise naturally during steady-state phonation when upper-limb movements impart physical impulses on the body, most likely affecting acoustics via respiratory activity. In this confirmatory study, participants (N = 29) repeatedly uttered consonant-vowel (/pa/) mono-syllables while moving in particular phase relations with speech, or not moving the upper limbs. This study shows that respiration-related activity is affected by (especially high-impulse) gesturing when vocalizations occur near peaks in physical impulse. This study further shows that gesture-induced moments of bodily impulses increase the amplitude envelope of speech, while not similarly affecting the Fundamental Frequency (F0). Finally, tight relations between respiration-related activity and vocalization were observed, even in the absence of movement, but even more so when upper-limb movement is present. The current findings expand a developing line of research showing that speech is modulated by functional biomechanical linkages between hand gestures and the respiratory system. This identification of gesture-speech biomechanics promises to provide an alternative phylogenetic, ontogenetic, and mechanistic explanatory route of why communicative upper limb movements co-occur with speech in humans.
References
1.
Alexanderson
, S.
, Henter
, G. E.
, Kucherenko
, T.
, and Beskow
, J.
(2020
). “Style-controllable speech-driven gesture synthesis using normalising flows
,” Comput. Graphics Forum
39
(2
), 487
–496
.2.
Arnold
, D.
, Wagner
, P.
, and Baayen
, R. H.
(2013
). “Using generalized additive models and random forests to model prosodic prominence in German
,” in Proceedings of Interspeech 2013
, August 25–29, Lyon, France, pp. 272
–276
.3.
Aruin
, A. S.
, and Latash
, M. L.
(1995
). “Directional specificity of postural muscles in feed-forward postural reactions during fast voluntary arm movements
,” Exp. Brain Res.
103
, 323
–332
.4.
Baer
, T.
(1979
). “Reflex activation of laryngeal muscles by sudden induced subglottal pressure changes
,” J. Acoust. Soc. Am.
65
, 1271
–1275
.5.
Basmajian
, J. V.
, and De Luca
, C. J.
(1985
). Muscles Alive: Their Functions Revealed By Electromyography
, 5th ed. (Williams and Wilkins
, Baltimore, MD
).6.
Bernstein
, N.
(1966
). The Co-Ordination and Regulation of Movements
(Pergamon Press
, London, UK
).7.
Blasi
, D. E.
, Moran
, S.
, Moisik
, S. R.
, Widmer
, P.
, Dediu
, D.
, and Bickel
, B.
(2019
). “Human sound systems are shaped by post-Neolithic changes in bite configuration
,” Science
363
(6432
), eaav3218
.8.
Bombien
, L.
, Winkelmann
, R.
, and Scheffers
, M.
(2020
). “wrassp: An R wrapper to the ASSP Library
,” R package version 0.1.9, https://cran.r-project.org/web/packages/wrassp/index.html (Last viewed August 18, 2020).9.
Bouisset
, S.
, and Do
, M. C.
(2008
). “Posture, dynamic stability, and voluntary movement
,” Clin. Neurophysiol.
38
, 345
–362
.10.
Chandrasekaran
, C.
, Trubanova
, A.
, Stillittano
, S.
, Caplier
, A.
, and Ghazanfar
, A. A.
(2009
). “The natural statistics of audiovisual speech
,” PLoS Comput. Biol.
5
(7
), e1000436
.11.
Chang
, P.
, and Hammond
, G. R.
(1987
). “Mutual interactions between speech and finger movements
,” J. Motor Behav.
19
(2
), 265
–274
.12.
Chu
, M.
, and Hagoort
, P.
(2014
). “Synchronization of speech and gesture: Evidence for interaction in action
,” J. Exp. Psychol. General
143
(3
), 1726
–1741
.13.
Cooperrider
, K.
(2017
). “Foreground gesture, background gesture
,” Gesture
16
(2
), 176
–202
.14.
Cordo
, P. J.
, and Nashner
, L. M.
(1982
). “Properties of postural adjustments associated with rapid arm movements
,” J. Neurophysiol.
47
(2
), 287
–302
.15.
Cravotta
, A.
, Busà
, M. G.
, and Prieto
, P.
(2019
). “Effects of encouraging the use of gestures on speech
,” J. Speech Lang. Hear. Res.
62
, 3204
–3219
.16.
Ćwiek
, A.
, and Fuchs
S.
(2019
). “Iconic prosody is rooted in sensori-motor properties: Fundamental frequency and the vertical space
,” in Proceedings of the 41st Annual Meeting of the Cognitive Science Society
, July 24–27, Montreal, Canada, pp. 1572
–1578
.17.
Danner
, S. G.
, Barbosa
, A. V.
, and Goldstein
, L.
(2018
). “Quantitative analysis of multimodal speech data
,” J. Phon.
71
, 268
–283
.18.
Dediu
, D.
, Janssen
, R.
, and Moisik
, S. R.
(2019
). “Weak biases emerging from vocal tract anatomy shape the repeated transmission of vowels
,” Nat. Hum. Behav.
3
, 1107
.19.
de Boer
, B.
, Wich
, S. A.
, Hardus
, M. E.
, and Lameira
, A. R.
(2015
). “Acoustic models of orangutan hand-assisted alarm calls
,” J. Exp. Biol.
218
(6
), 907
–914
.20.
Dimensions.Guide (2019). “
Male (side) dimensions & drawings
,” https://www.dimensions.com/element/sitting-male-side-1 (Last viewed May 1, 2019).21.
Efron
, D.
, Efron
, J. M.
, and Veen
, S. V.
(1972
). Gesture, Race and Culture: A Tentative Study of the Spatio-Temporal and “Linguistic” Aspects of the Gestural Behavior of Eastern Jews and Southern Italians in New York City, Living Under Similar as Well as Different Environmental Conditions
(Mouton
, the Hague, the Netherlands
).22.
Ejiri
, K.
(1998
). “Relationship between rhythmic behavior and canonical babbling in infant vocal development
,” Phonetica
55
(4
), 226
–237
.23.
Esteve-Gibert
, N.
, and Prieto
, P.
(2013
). “Prosodic structure shapes the temporal realization of intonation and manual gesture movements
,” J. Speech Lang. Hear. Res.
56
(3
), 850
–864
.24.
Finnegan
, E. M.
, Luschei
, E. S.
, and Hoffman
, H. T.
(2000
). “Modulations in respiratory and laryngeal activity associated with changes in vocal intensity during speech
,” J. Speech Lang. Hear. Res.
43
(4
), 934
–950
.25.
Fuchs
, S.
(2019
). “Vocal tract variations affect vowel sounds
,” Nat. Hum. Behav.
3
, 1043
.26.
Fuchs
, S.
, Petrone
, C.
, Rochet-Capellan
, A.
, Reichel
, W. D.
, and Koenig
, L. L.
(2015
). “Assessing respiratory contributions to F0 declination in German across varying speech tasks and respiratory demands
,” J. Phon.
52
, 35
–45
.27.
Gibson
, J.
(1966
). The Senses Considered as Perceptual Systems
(Houghton–Mifflin
, Boston, MA
).28.
Ginosar
, S.
, Bar
, A.
, Kohavi
, G.
, Chan
, C.
, Owens
, A.
, and Malik
, J.
(2019
). “Learning individual styles of conversational gesture
,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition
, June 16–20, Long Beach, CA, pp. 3497
–3506
.29.
Hanson
, H. M.
(2009
). “Effects of obstruent consonants on fundamental frequency at vowel onset in English
,” J. Acoust. Soc. Am.
125
(1
), 425
–441
.30.
Hardus
, M. E.
, Lameira
, A. R.
, Schaik
, C. S.
, and Wich
, S. A.
(2009
). “Tool use in wild orang-utans modifies sound production: A functionally deceptive innovation?
,” Proc. R. Soc. B: Biol. Sci.
276
(1673
), 3689
–3694
.31.
Hastie
, T.
, and Hastie
, M. T.
(2018
). “Package ‘GAM
,’ ” GAM Package CRAN, https://cran.r-project.org/web/packages/gam/ (Last viewed August 18, 2020).32.
He
, L.
, and Dellwo
, V.
(2017
). “Amplitude envelope kinematics of speech: Parameter extraction and applications
,” J. Acoust. Soc. Am.
141
(5
), 3582
.33.
Hodges
, P. W.
, and Richardson
, C. A.
(1997
). “Feedforward contraction of transversus abdominis is not influenced by the direction of arm movement
,” Exp. Brain Res.
114
(2
), 362
–370
.34.
Hübscher
, I.
, and Prieto
, P.
(2019
). “Gestural and prosodic development act as sister systems and jointly pave the way for children's sociopragmatic development
,” Front. Psychol.
10
, 1259
.35.
Ingber
, D. W.
(2008
). “Tensegrity and mechanotransduction
,” J. Bodywork Move. Ther.
12
(3
), 198
–200
.36.
Iverson
, J. M.
, and Thelen
, E.
(2005
). “Hand, mouth and brain: The dynamic emergence of speech and gesture
,” J. Conscious. Studies
22
, 19
–40
.37.
Kelso
, S.
, and Tuller
, B.
(1984
). “Converging evidence in support of common dynamical principles for speech and movement coordination
,” Am. J. Physiol.
246
, 928
–935
.38.
Kelso
, J. A. S.
, Tuller
, B.
, and Harris
, K.
(1983
). “A ‘dynamic Pattern’ perspective on the control and coordination of movement
,” in The Production of Speech
, edited by P. F.
McNeilage
(Springer
, New York)
, pp. 137
–173
.39.
Kleiman
, E. M.
(2017
). “EMAtools: Data management tools for real-time monitoring/ecological momentary assessment data
,” https://CRAN.R-project.org/package=EMAtools (Last viewed August 18, 2020).40.
Krahmer
, E.
, and Swerts
, M.
(2007
). “The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception
,” J. Mem. Lang.
57
(3
), 396
–414
.41.
Krivokapić
, J.
(2014
). “Gestural coordination at prosodic boundaries and its role for prosodic structure and speech planning processes
,” Philos. Trans. R. Soc. B
369
(1658
), 20130397
.42.
Krivokapic
, J.
, Tiede
, M. K.
, Tyrone
, M. E.
, and Goldenberg
, D.
(2016
). “Speech and manual gesture coordination in a pointing task
,” in Proceedings of Speech Prosody 2016
, May 31–June 3, Boston, MA.43.
Kucherenko
, T.
, Hasegawa
, D.
, Henter
, G. E.
, Kaneko
, N.
, and Kjellström
, H.
(2019
). “Analyzing input and output representations for speech-driven gesture generation
,” in Proceedings of the 19th ACM International Conference on Intelligent Virtual Agents—IVA '19
, July 2–5, Paris, France, pp. 97
–104
.44.
Kugler
, P. N.
, and Turvey
, M. T.
(1987
). Information, Natural Law, and the Self-Assembly of Rhythmic Movement
(L. Erlbaum Associates
, Hillsdale, NJ
).45.
Ladefagod
, P.
(1968
). “Linguistic aspects of respiratory phenomena
,” in Sound Production in Man
, edited by A.
Bouhuys
(New York Academy of Sciences
, New York
), pp. 141
–151
.46.
Lancaster
, W. C.
, Henson
, O. W.
, and Keating
, A. W.
(1995
). “Respiratory muscle activity in relation to vocalization in flying bats
,” J. Exp. Biol.
198
(1
), 175
–191
.47.
48.
Leonard
, T.
, and Cummins
, F.
(2011
). “The temporal relation between beat gestures and speech
,” Lang. Cogn. Process.
26
(10
), 1457
–1471
.49.
Levin
, S. M.
(2006
). “Tensegrity: The new biomechanics
,” in Textbook of Muscularskeletal Medicine
, edited by M.
Hutson
and R.
Ellis
(Oxford University Press
, Oxford, UK
), pp. 69
–80
.50.
Lieberman
, P.
(1996
). “Some biological constraints on the analysis of prosody
,” in Signal to Syntax
, edited by J. L.
Morgan
and K.
Demuth
(Erlbaum
, Mahwah, NJ
), pp. 67
–78
.51.
Loehr
, D. P.
(2012
). “Temporal, structural, and pragmatic synchrony between intonation and gesture
,” Lab. Phonol.
3
(1
), 71
–89
.52.
Löfqvist
, A.
, Baer
, T.
, McGarr
, N. S.
, and Seider Story
, R.
(1989
). “The cricothyroid muscle in voicing control
,” J. Acoust. Soc. Am.
85
, 1314
–1321
.53.
MacLarnon
, A. M.
, and Hewitt
, G. P.
(1999
). “The evolution of human speech: The role of enhanced breathing control
,” Am. J. Phys. Anthropol.
109
(3
), 341
–363
.54.
MacNeilage
, P. F.
(1998
). “The frame/content theory of evolution of speech production
,” Behav. Brain Sci.
21
(4
), 499
–511
.55.
McClave
, E.
(1998
). “Pitch and manual gestures
,” J. Psycholing. Res.
27
(2
), 69
–89
.56.
McNeill
, D.
(1992
). Hand and Mind: What Gestures Reveal About Thought
(University of Chicago Press
, Chicago, IL
).57.
McNeill
, D.
(2005
). Gesture and Thought
(University of Chicago Press
, Chicago, IL
).58.
Ohala
, J. J.
(1990
). “Respiratory activity in speech
,” in Speech Production and Speech Modeling
, edited by W. J.
Hardcastle
and A.
Marchal
(Kluwer
, Dordrecht, the Netherlands
), pp. 22
–53
.59.
Parrell
, B.
, Goldstein
, L.
, Lee
, S.
, and Byrd
, D.
(2014
). “Spatiotemporal coupling between speech and manual motor actions
,” J. Phon.
42
, 1
–11
.60.
Perrier
, P.
, and Fuchs
, S.
(2015
). “Motor equivalence in speech production
,” in The Handbook of Speech Production
, edited by M. A.
Redford
(Wiley
, New York
), pp. 223
–247
.61.
Petrone
, C.
, Fuchs
, S.
, and Koenig
, L. L.
(2017
). “Relations among subglottal pressure, breathing, and acoustic parameters of sentence-level prominence in German
,” J. Acoust. Soc. Am.
141
(3
), 1715
–1725
.62.
Pinheiro
, J.
, Bates
, D.
, DebRoy
, S.
, Sarkar
, D.
, and R Team
(2019
). “nlme: Linear and nonlinear mixed effects models
,” https://cran.r-project.org/web/packages/nlme/index.html (Last viewed August 18, 2020).63.
Pisanski
, K.
, Cartei
, V.
, McGettigan
, C.
, Raine
, J.
, and Reby
, D.
(2016
). “Voice modulation: Window into the origins of human vocal control?
,” Trends Cogn. Sci.
20
(4
), 304
–318
.64.
Pouw
, W.
, de Jonge-Hoekstra
, L.
, Harrison
, S. J.
, Paxton
, A.
, and Dixon
, J. A.
(2020a
). “Gesture-speech physics in fluent speech and rhythmic upper limb movements
,” Psyarxiv, .65.
Pouw
, W.
, and Dixon
, J. A.
(2019a
). “Entrainment and modulation of gesture—Speech synchrony under delayed auditory feedback
,” Cogn. Sci.
43
(3
), e12721
.66.
Pouw
, W.
, and Dixon
, J. A.
(2019b
). “Quantifying gesture-speech synchrony
,” in Proceedings of the 6th Meeting of Gesture and Speech in Interaction
, September 11–13, Paderborn, Germany
.67.
Pouw
, W.
, Harrison
, S. H.
, and Dixon
, J.
(2019a
). “Gesture-speech physics: The biomechanical basis of the emergence of gesture-speech synchrony
,” J. Exp. Psychol. General
149
, 391
–404
.68.
Pouw
, W.
, Harrison
, S. J.
, Esteve-Gibert
, N.
, and Dixon
, J. A.
(2019b
). “Energy Flows in Gesture-Speech Physics: Exploratory Findings and Pre-Registration of Confirmatory Analysis
,” Open Science Framework, and https://osf.io/x7zdc/ (Last viewed August 18, 2020).69.
Pouw
, W.
, Paxton
, A.
, Harrison
, S. J.
, and Dixon
, J. A.
(2019c
). “Acoustic specification of upper limb movement in voicing
,” in Proceedings of the 6th Meeting of Gesture and Speech in Interaction
, September 11–13, Paderborn, Germany, available at .70.
Pouw
, W.
, Paxton
, A.
, Harrison
, S. J.
, and Dixon
, J. A.
(2020b
). “Multimodal origins of the human voice: Acoustic information about upper limb movement in voicing
,” Proc. Natl. Acad. Sci.
117
(21
), 11364
–11367
.71.
Pouw
, W.
, and Trujillo
, J. P.
(2019
). “Tutorial Gespin2019—Using video-based motion tracking to quantify speech-gesture synchrony
,” (Last viewed August 18, 2020).72.
Pouw
, W.
, Trujillo
, J. P.
, and Dixon
, J. A.
(2020c
). “The quantification of gesture-speech synchrony: An overview and validation of video-based motion tracking
,” Behav. Res. Methods
52
, 723
–740
.73.
Profeta
, V. L.
, and Turvey
, M. T.
(2018
). “Bernstein's levels of movement construction: A contemporary perspective
,” Hum. Move. Sci.
57
, 111
–133
.74.
Raja
, V.
(2020
). “Resonance and radical embodiment
,” Synthese, .75.
Richardson
, M.
(2009
). “Polhemus applications and example code
,” http://xkiwilabs.com/software-toolboxes/ (November 1, 2018).76.
Rochet-Capellan
, A.
, and Fuchs
, S.
(2014
). “Take a breath and take the turn: How breathing meets turns in spontaneous dialogue
,” Philos. Trans. R. Soc. B: Biol. Sci.
369
(1658
), 20130399
.77.
Rochet-Capellan
, A.
, Laboissière
, R.
, Galván
, A.
, and Schwartz
, J.
(2008
). “The speech focus position effect on jaw–finger coordination in a pointing Task
,” J. Speech Lang. Hear. Res.
51
(6
), 1507
–1521
.78.
Rosch
, A.
, and Schmidbauer
, H.
(2014
). “WaveletComp 1.1: A guided tour through the R package 59
,” http://www.hs-stat.com/projects/WaveletComp/WaveletComp_guided_tour.pdf (Last viewed August 18, 2020).79.
Rusiewicz
, H. L.
, and Esteve-Gibert
, N.
(2018
). “Set in time: Temporal coordination of prosody and gesture in the development of spoken language production
,” in The Development of Prosody in First Language Acquisition
, edited by P.
Prieto
and N.
Esteve-Gibert
(John Benjamins Publishing Company
, Amsterdam, the Netherlands)
, pp. 103
–124
.80.
Rusiewicz
, H. L.
, Shaiman
, S.
, Iverson
, J.
, and Szuminsky
, N.
(2013
). “Effects of prosody and position on the timing of deictic gestures
,” J. Speech Lang. Hear. Res.
56
(2
), 458
–470
.81.
Shattuck-Hufnagel
, S.
, and Prieto
, P.
(2019
). “Dimensionalizing co-speech gestures
,” in Proceedings of the International Congress of Phonetic Sciences 2019
, August 5–9, Melbourne, Australia
.82.
Silva
, P.
, Moreno
, M.
, Mancini
, M.
, Fonseca
, S.
, and Turvey
, M. T.
(2007
). “Steady-state stress at one hand magnifies the amplitude, stiffness, and non-linearity of oscillatory behavior at the other hand
,” Neurosci. Lett.
429
(1
), 64
–68
.83.
Smith
, R.
, Nyquist-Battie
, C.
, Clark
, M.
, and Rains
, J.
(2003
). “Anatomical characteristics of the upper serratus anterior: Cadaver dissection
,” J. Orthopaed. Sports Phys. Therapy
33
(8
), 449
–454
.84.
Stoltmann
, K.
, and Fuchs
, S.
(2017
). “Syllable-pointing gesture coordination in Polish counting out rhymes: The effect of speech rate
,” J. Multimodal Commun. Stud.
4
, 63
–68
.85.
Tilsen
, S.
, and Arvaniti
, A.
(2013
). “Speech rhythm analysis with decomposition of the amplitude envelope: Characterizing rhythmic patterns within and across languages
,” J. Acoust. Soc. Am.
134
(1
), 628
–639
.86.
Treffner
, P.
, and Peter
, M.
(2002
). “Intentional and attentional dynamics of speech–hand coordination
,” Hum. Move. Sci.
21
(5–6
), 641
–697
.87.
Turvey
, M. T.
, and Fonseca
, S. T.
(2014
). “The medium of haptic perception: A tensegrity hypothesis
,” J. Motor Behav.
46
(3
), 143
–187
.88.
Wagner
, P.
, Malisz
, Z.
, and Kopp
, S.
(2014
). “Gesture and speech in interaction: An overview
,” Speech Commun.
57
, 209
–232
.89.
Wieling
, M.
(2018
). “Analyzing dynamic phonetic data using generalized additive mixed modeling: A tutorial focusing on articulatory differences between L1 and L2 speakers of English
,” J. Phon.
70
, 86
–116
.90.
Wittenburg
, P.
, Brugman
, H.
, Russel
, A.
, Klassmann
, A.
, and Sloetjes
, H.
(2006
). “ELAN: A professional framework for multimodality research
,” in Proceedings of the Fifth International Conference on Language Resources and Evaluation (LREC'06)
, May 22–28, Genoa, Italy
.91.
Wood
, S. N.
(2017
). Generalized Additive Models: An Introduction with R, Second Edition
(CRC Press
, Boca Raton, FL
).92.
Zelic
, G.
, Kim
, J.
, and Davis
, C.
(2015
). “Articulatory constraints on spontaneous entrainment between speech and manual gesture
,” Hum. Move. Sci.
42
, 232
–245
.© 2020 Acoustical Society of America.
2020
Acoustical Society of America
You do not currently have access to this content.
