Skip to main content
SpringerLink
Log in

Hand Gestures Alert Auditory Cortices

Possible Impacts of Learning on Foreign Language Processing

  • Chapter
  • First Online:
Positive Learning in the Age of Information

  • 1043 Accesses

Abstract

When acquiring a foreign language, the first challenge is to break into the speech stream to identify basic linguistic units. The present study tested the hypothesis that hand gestures facilitate this process by alerting auditory cortices to attend to and identify meaningful phonemic information. During fMRI data acquisition, participants watched videos of an actor speaking in Russian under three conditions. Sentences were produced with just speech alone or were accompanied by two types of hand gestures: 1) metaphoric gesture and 2) free gesture. The main finding was that there was increased auditory cortex activation when both types of gestures accompanied speech compared to speech alone, but there were no differences between the two speech + gesture conditions (or gesture alone conditions). These results suggest that hand gestures may play a role in focusing attention to auditory processing to increase capacity when listening to novel speech in a foreign language.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
eBook
USD 16.99
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 16.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Bibliography

  • Calvert, G. A., Brammer, M. J., Bullmore, E. T., Campbell, R., Iversen, S. D., & David, A. S. (1999). Response amplification in sensory-specific cortices during crossmodal binding. Neuroreport 10(12), 2619–2623.

    Google Scholar 

  • Clark, H. H. (1996). Using language. Cambridge: University Press Cambridge.

    Google Scholar 

  • Colin, C., Radeau, M., Soquet, A, Demolin, D., Colin, F., & Deltenre, P. (2002). Mismatch negativity evoked by the McGurk-MacDonald effect: A phonetic representation within short-term memory. Clinical Neurophysiology 113(4), 495–506.

    Google Scholar 

  • Dick, A. S., Goldin-Meadow, S., Hasson, U., Skipper, J. I., & Small, S. L. (2009). Co-speech gestures infl uence neural activity in brain regions associated with processing semantic information. Hum Brain Mapp 30(11), 3509–3526.

    Google Scholar 

  • Feyereisen, P., Van de Wiele, M., & Dubois, F. (1988). The meaning of gestures: What can be understood without speech? Curr Psychol Cogn 8, 3–25.

    Google Scholar 

  • Friston, K. J., Fletcher, P., Josephs, O., Holmes, A., Rugg, M. D., & Turner, R. (1998). Event-related fMRI: Characterizing differential responses. Neuroimage 7(1), 30–40.

    Google Scholar 

  • Green, A., Straube, B., Weis, S., Jansen, A., Willmes, K., Konrad, K., & Kircher, T. (2009). Neural integration of iconic and unrelated coverbal gestures: A functional MRI study. Hum Brain Mapp 30(10), 3309–3324.

    Google Scholar 

  • Gullberg, M., Roberts, L., Dimroth, C., Veroude, K., & Indefrey, P. (2010). Adult language learning after minimal exposure to an unknown natural language. Language Learning 60, 5–24.

    Google Scholar 

  • Habets, B., Kita, S., Shao, Z., Ozyurek, A., & Hagoort, P. (2011). The Role of Synchrony and Ambiguity in Speech-Gesture Integration during Comprehension. J Cogn Neurosci 23(8), 1845–1854.

    Google Scholar 

  • Holle, H., & Gunter, T. C. (2007): The role of iconic gestures in speech disambiguation: ERP evidence. J Cogn Neurosci 19(7), 1175–1192.

    Google Scholar 

  • Holle, H., Gunter, T. C., Rueschemeyer, S.-A., Hennenlotter, A, & Iacoboni, M. (2008). Neural correlates of the processing of co-speech gestures. Neuroimage 39(4), 2010–2024.

    Google Scholar 

  • Holle, H., Obleser, J., Rueschemeyer, S.-A., Gunter, T. C. (2010). Integration of iconic gestures and speech in left superior temporal areas boosts speech comprehension under adverse listening conditions. Neuroimage 49(1), 875–884.

    Google Scholar 

  • Hubbard, A. L., Wilson, S. M., Callan, D. E., Dapretto, M. (2009). Giving speech a hand: Gesture modulates activity in auditory cortex during speech perception. Hum Brain Mapp 30(3), 1028–1037.

    Google Scholar 

  • Ibáñez A., Manes, F., Escobar, J., Trujillo, N., Andreucci, P., & Hurtado, E. (2010). Gesture influences the processing of figurative language in non-native speakers: ERP evidence. Neurosci Lett 471(1), 48–52.

    Google Scholar 

  • Ibáñez, A., Toro, P., Cornejo, C., Urquina, H., Hurquina, H., Manes, F., Weisbrod, M., & Schröder, J. (2011). High contextual sensitivity of metaphorical expressions and gesture blending: A video event-related potential design. Psychiatry Res 191(1), 68–75.

    Google Scholar 

  • Kelly, S. D., Barr, D. J., Church, R. B., & Lynch, K. (1999). Offering a Hand to Pragmatic Understanding: The Role of Speech and Gesture in Comprehension and Memory. Journal of Memory and Language 40, 577–592.

    Google Scholar 

  • Kelly, S. D., Creigh, P., & Bartolotti, J. (2010a). Integrating speech and iconic gestures in a Stroop-like task: Evidence for automatic processing. J Cogn Neurosci 22(4), 683–694.

    Google Scholar 

  • Kelly, S. D., Kravitz, C., & Hopkins, M. (2004). Neural correlates of bimodal speech and gesture comprehension. Brain Lang 89(1), 253–260.

    Google Scholar 

  • Kelly, S. D., McDevitt, T., & Esch, M. (2009). Brief training with co-speech gesture lends a hand to word learning in a foreign language. Language and cognitive processes 24(2), 313–334.

    Google Scholar 

  • Kelly, S. D., Ozyürek, A., & Maris, E. (2010b): Two sides of the same coin: Speech and gesture mutually interact to enhance comprehension. Psychol Sci 21(2), 260–267.

    Google Scholar 

  • Kelly, S. D., Ward, S., Creigh, P., & Bartolotti, J. (2007). An intentional stance modulates the integration of gesture and speech during comprehension. Brain Lang 101(3), 222–233.

    Google Scholar 

  • Kircher, T., Straube, B., Leube, D., Weis, S., Sachs, O., Willmes, K., Konrad, K., & Green, A. (2009). Neural interaction of speech and gesture: Differential activations of metaphoric co-verbal gestures. Neuropsychologia 47(1), 169-179.

    Google Scholar 

  • Krahmer, E., & Swerts, M. (2007). The effects of visual beats on prosodic prominence: Acoustic analyses, auditory perception and visual perception. Journal of Memory and Language 57(3), 396–414.

    Google Scholar 

  • Macedonia, M., Müller, K., & Friederici, A. D. (2011). The impact of iconic gestures on foreign language word learning and its neural substrate. Hum Brain Mapp 32(6), 982–998.

    Google Scholar 

  • MacSweeney, M., Woll, B., Campbell, R., McGuire, P. K., David, A. S., Williams, S. C., Suckling, J., Calvert, G. A., & Brammer, M. J. (2002). Neural systems underlying British Sign Language and audio-visual English processing in native users. Brain 125(7), 1583–1593.

    Google Scholar 

  • McNeill, D. (1992). Hand and mind: What gestures reveal about thought. Chicago: Univ. of Chicago Press.

    Google Scholar 

  • Oldfield, R. C. (1971). The assessment and analysis of handedness: The Edinburgh inventory. Neuropsychologia 9(1), 97–113.

    Google Scholar 

  • Ozyürek, A., Willems, R. M., Kita, S., & Hagoort, P. (2007). On-line integration of semantic information from speech and gesture: Insights from event-related brain potentials. J Cogn Neurosci 19(4), 605–616.

    Google Scholar 

  • Sams, M., Aulanko, R., Hämäläinen, M., Hari, R., Lounasmaa, O. V., Lu, S. T., & Simola, J. (1991): Seeing speech: Visual information from lip movements modifies activity in the human auditory cortex. Neurosci Lett 127(1), 141–145.

    Google Scholar 

  • Skipper, J. I., Goldin-Meadow, S., Nusbaum, H. C., & Small, S. L. (2007). Speech-associated gestures, Broca’s area, and the human mirror system. Brain Lang 101(3), 260-277.

    Google Scholar 

  • Skipper, J. I., Goldin-Meadow, S., Nusbaum, H. C., & Small, S. L. (2009). Gestures orchestrate brain networks for language understanding. Curr Biol 19(8), 661-667.

    Google Scholar 

  • Slotnick, S. D., Moo, L. R., Segal, J. B., & Hart, J., Jr. (2003). Distinct prefrontal cortex activity associated with item memory and source memory for visual shapes. Brain Res Cogn Brain Res 17(1), 75–82.

    Google Scholar 

  • Stein, B. E., & Meredith, M. A. (1993). The merging of the senses. Cambridge: MIT Press.

    Google Scholar 

  • Straube, B., Green, A., Bromberger, B., & Kircher, T. (2011a). The differentiation of iconic and metaphoric gestures: Common and unique integration processes. Hum Brain Mapp 32(4), 520–533.

    Google Scholar 

  • Straube, B., Green, A., Chatterjee, A., & Kircher, T. (2011b). Encoding social interactions: The neural correlates of true and false memories. J Cogn Neurosci 23(2), 306–324.

    Google Scholar 

  • Straube, B., Green, A., Jansen, A., Chatterjee, A., & Kircher, T. (2010). Social cues, mentalizing and the neural processing of speech accompanied by gestures. Neuropsychologia 48(2), 382–393.

    Google Scholar 

  • Straube, B., Green, A., Weis, S., Chatterjee, A., & Kircher, T. (2009). Memory effects of speech and gesture binding: Cortical and hippocampal activation in relation to subsequent memory performance. J Cogn Neurosci 21(4), 821–836.

    Google Scholar 

  • Willems, R. M., Ozyurek, A., & Hagoort, P. (2007). When language meets action: the neural integration of gesture and speech. Cereb Cortex 17(10), 2322–2333.

    Google Scholar 

  • Willems, R. M., Ozyurek, A., & Hagoort, P. (2009). Differential roles for left inferior frontal and superior temporal cortex in multimodal integration of action and language. Neuroimage 47(4), 1992–2004.

    Google Scholar 

  • Wu, Y. C., Coulson, S. (2005). Meaningful gestures: Electrophysiological indices of iconic gesture comprehension. Psychophysiology 42(6), 654–667.

    Google Scholar 

  • Wu, Y. C., & Coulson, S. (2007a). How iconic gestures enhance communication: An ERP study. Brain Lang 101(3), 234–245.

    Google Scholar 

  • Wu, Y. C., & Coulson, S. (2007b). Iconic gestures prime related concepts: An ERP study. Psychon Bull Rev 14(1), 57–63.

    Google Scholar 

  • Wu, Y. C., & Coulson, S. (2010). Gestures modulate speech processing early in utterances. Neuroreport 21(7), 522–526.

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Johannes Gutenberg University Mainz, Mainz, Germany

    Arne Nagels

  2. Colgate University Hamilton, Hamilton, USA

    Spencer D. Kelly

  3. Philipps-University Marburg, Marburg, Germany

    Tilo Kircher & Benjamin Straube

Authors
  1. Arne Nagels
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Spencer D. Kelly
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Tilo Kircher
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Benjamin Straube
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arne Nagels .

Editor information

Editors and Affiliations

  1. Lehrstuhl für Wirtschaftspädagogik, Johannes Gutenberg-University Mainz, Mainz, Rheinland-Pfalz, Germany

    Olga Zlatkin-Troitschanskaia

  2. Goethe-University, Frankfurt am Main, Germany

    Gabriel Wittum

  3. German Research Center for Artificial Intelligence, Kaiserslautern, Germany

    Andreas Dengel

Rights and permissions

Reprints and permissions

Copyright information

© 2018 Springer Fachmedien Wiesbaden GmbH

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Nagels, A., Kelly, S.D., Kircher, T., Straube, B. (2018). Hand Gestures Alert Auditory Cortices. In: Zlatkin-Troitschanskaia, O., Wittum, G., Dengel, A. (eds) Positive Learning in the Age of Information. Springer VS, Wiesbaden. https://doi.org/10.1007/978-3-658-19567-0_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-658-19567-0_5

  • Published:

  • Publisher Name: Springer VS, Wiesbaden

  • Print ISBN: 978-3-658-19566-3

  • Online ISBN: 978-3-658-19567-0

  • eBook Packages: EducationEducation (R0)

Publish with us

Policies and ethics

Search

Navigation