Elsevier

Cognition

Volume 116, Issue 3, September 2010, Pages 394-408
Cognition

On the nature of hand-action representations evoked during written sentence comprehension

https://doi.org/10.1016/j.cognition.2010.06.001Get rights and content

Abstract

We examine the nature of motor representations evoked during comprehension of written sentences describing hand actions. We distinguish between two kinds of hand actions: a functional action, applied when using the object for its intended purpose, and a volumetric action, applied when picking up or holding the object. In Experiment 1, initial activation of both action representations was followed by selection of the functional action, regardless of sentence context. Experiment 2 showed that when the sentence was followed by a picture of the object, clear context-specific effects on evoked action representations were obtained. Experiment 3 established that when a picture of an object was presented alone, the time course of both functional and volumetric actions was the same. These results provide evidence that representations of object-related hand actions are evoked as part of sentence processing. In addition, we discuss the conditions that elicit context-specific evocation of motor representations.

Introduction

Sentences that describe physical actions on objects are commonplace in everyday language. Despite the apparent ordinariness of such sentences, a number of provocative questions arise regarding the mental processes that take place during their comprehension. These questions concern the interaction between language and other cognitive systems, in particular the visual and motor systems that deal with objects and the actions they afford. It seems quite reasonable to assume that knowing the full meaning of a word denoting a physical object must in part include a representation of what such an object looks like. The same requirement applies to many actions denoted by verbs. Jackendoff (1987) notes that the difference between verbs such as walk, run, jog, lope and sprint cannot adequately be captured by a set of binary algebraic features. Instead, to understand the difference, we require a representation of three-dimensional form that includes distinctions in gate and speed as part of the verbs’ lexical entries.

The meaning of certain words, then, incorporates details of an object’s physical appearance and movement. To what extent, however, do we understand a garden-variety sentence describing an action, such as John grasped the cell phone, by directly appealing to our experience of what a cell phone looks like, and to the actions we make when picking it up? A substantial amount of evidence shows that motor or perceptual representations are automatically evoked during or shortly after the processing of sentences describing actions or objects (e.g., Borreggine and Kaschak, 2006, Glenberg and Kaschak, 2002, Kaschak and Borreggine, 2008, Zwaan et al., 2002). It is clear from these demonstrations that sentence comprehension does indeed yield motor and/or perceptual “resonance” (Zwaan & Taylor, 2006). Much less clear, though, is how these motor-based representations stand in relationship to the computation of sentence meaning.

In this article, we consider the nature of motor representations evoked by written sentences that describe simple hand actions with objects (e.g., Mary picked up the calculator). We begin by introducing and commenting on two relatively neglected questions that are fundamental to an adequate understanding of motor resonance as part of sentence processing. Objects evoke multiple possible actions, only a subset of which is selected on the basis of intentions or context. The evocation of action representations in sentence comprehension will presumably involve the same or a similar form of competition. A word like cup, for example, indicating a manipulable object, evokes multiple possible motor affordances. To what extent does selection of the contextually-relevant affordance occur in constructing the meaning of a sentence? Our second question concerns language comprehension as a dynamic process that unfolds in real time. How does the phenomenon of motor resonance stand in relationship to the dynamics of sentence comprehension?

These two questions serve as background to motivate a set of experiments on the nature of motor representations evoked by sentences describing hand actions applied to manipulable objects. We seek to better understand the relationship between motor resonance and sentence comprehension. One possibility is that the phenomenon of motor resonance is driven by a literal interpretation of the sentence. In other words, a sentence referring to picking up a calculator entails one kind of action, but a sentence about using that object entails another. The resulting action representations, then, are part and parcel of the literal meaning of the sentence.

There are, however, more complex possibilities. First, motor resonance may enable predictions about changes in real-world states implied by the event structures described in the sentence. This predictive aspect of motor resonance is consistent with the notion of mental simulation favored by Altmann and Mirkovic (2009). On this account, a sentence like John picked up the pencil may evoke an anticipation of writing as the final goal of the action rather than more immediate effects produced on the way to that goal (see Searle, 1983, for the distinction between “prior intentions” and “intentions in action”, referring to the distinction between an intention that requires a number of steps to completion and an intention directed toward an immediately accessible goal). In that case, motor resonance will reflect functional hand actions at the end of the sentence, even though the literal meaning simply refers to a precision grip for lifting.

A second alternative is that action representations evoked during sentence comprehension may play only a peripheral role in the construction of meaning. Jackendoff (2002) argues that modality-specific representations that include information about the shapes of objects and the nature of the actions applied to them serve to capture details that cannot easily be represented by an abstract conceptual structure of word meaning. For example, the distinction between the sentences John swaggered into the room and John marched into the room is best captured by referring to our embodied knowledge of different styles of walking. At another level (the interface between meaning and syntax), the sentences are equivalent in that both refer to John entering the room by walking. Motor resonance may reflect fleeting activation of knowledge associated with the details of an action (e.g., the form of walking implied by the verb) that do not contribute to the final representation of the sentence in memory unless the particular action is embellished by subsequent text. Alternatively, resonance effects may occur as a by-product of processing the core aspects of word meaning divorced from sentence context (e.g., a calculator refers to a device for computing numbers and we usually operate it by pressing its keys). Thus, a sentence that refers to picking up a calculator may evoke a functional action representation (pressing keys) because the word calculator denotes an object that is used in a particular way regardless of the verb accompanying it.

Section snippets

Motor resonance and the selection of contextually-relevant affordances

We interact with objects in a variety of ways, depending on our intentions. Consider a common object like a cell phone. If we wish to pick up the phone and move it to a new location, one set of actions is required; if we intend instead to make a call, another set of movements must occur. Intuition suggests that we program the actions to an object after we decide on what to do. The cell phone evokes an inverted closed-grasp once we decide to lift and move it. The same object affords a set of key

Motor resonance and the dynamics of sentence comprehension

Studies looking for motor resonance in comprehension have demonstrated the evocation of motor representations at various time points (early as well as late) in sentence processing. Zwaan and Taylor (2006) required subjects to read a sentence one frame at time, with each frame comprising between one and three words. Subjects made each frame appear by rotating a small knob (1 in. in diameter) mounted on a keyboard, clockwise or counterclockwise (direction varied between the first and second half

Measurement of action representations

We have introduced two questions that serve to motivate the present enquiry into the nature of motor resonance during sentence comprehension. One question we raised concerns the dynamics of sentence processing. We have argued that if motor resonance reflects the processing of meaning in real time, then the nature of the motor representations evoked after a verb-noun combination in a simple declarative sentence is of particular interest. For example, the word lift in isolation is consistent with

Experiment 1

In Experiment 1 we wished to examine the effect of two kinds of sentence context on the evocation of hand-action representations. One type of context emphasized manual interactions based on the shape of the object by describing activities such as picking up, moving, or giving an object to another person. On the assumption that a literal interpretation of these contexts determines motor resonance, such sentences will ultimately evoke volumetric action representations of object concepts. The

Experiment 2

Written sentences that describe hand actions involving manipulable objects ultimately refer to physical events taking place in a world external to the reader. What kind of relationship exists between the mental world, in which the senses of individual words are combined, and the world where the actions referred to are actually taking place? The question prompts us to look beyond tasks that require only the processing of written sentences, to conditions that would require a linkage between the

Experiment 3

We turn now to the question of which action representations are evoked by the sight of an object alone, passively viewed in the absence of a sentence context or other intentional cue. We do not know whether activation of action representations during passive viewing of an object is sufficiently precise to enable priming of volumetric and/or functional representations. Furthermore, we need to establish whether an object on its own differentially evokes functional versus volumetric action

General discussion

Motor resonance is an undeniable aspect of sentence comprehension. Reach and grasp responses are primed by sentences that reference actions describing lifting or using manipulable objects. What though, is the relationship between the evocation of hand-action representations and the processing of sentence meaning? We introduced, as background, the idea that a word referring to a manipulable object is associated by past experience with multiple action representations (e.g., we pick up a thumbtack

Acknowledgements

This research was supported by discovery grants from the Natural Sciences and Engineering Research Council of Canada to Daniel Bub and to Michael Masson, and by a grant from the J.S. McDonnell Foundation, and National Science Foundation Grant #SBE-0542013 to the Temporal Dynamics of Learning Center, an NSF Science of Learning Center. We are grateful to Marnie Jedynak and Lisa Worth for assistance in conducting the experiments.

References (46)

  • L. Smith et al.

    Naming in young children: A dumb mechanism?

    Cognition

    (1996)
  • P. Tabossi

    Accessing lexical ambiguity in different types of sentential context

    Journal of Memory and Language

    (1988)
  • E. Yee et al.

    fMRI-adaptation evidence of overlapping neural representations for objects related in function or manipulation

    NeuroImage

    (2010)
  • R.A. Zwaan et al.

    Moving words: Dynamic mental representations in language comprehension

    Cognitive Science

    (2004)
  • G.T.M. Altmann et al.

    Incrementality and prediction in human sentence processing

    Cognitive Science

    (2009)
  • Bailey, D. (1997). When push comes to shove: A computational model of the role of motor control in the acquisition of...
  • M.A. Baumann et al.

    Context-specific grasp movement representation in the macaque anterior intraparietal area

    Journal of Neuroscience

    (2009)
  • K.L. Borreggine et al.

    The action–sentence compatibility effect: It’s all in the timing

    Cognitive Science

    (2006)
  • P. Cisek

    Neural representations of motor plans, desired trajectories, and controlled objects

    Cognitive Processing

    (2005)
  • P. Cisek

    Cortical mechanisms of action selection: The affordance competition hypothesis

    Philosophical Transactions of the Royal Society B

    (2007)
  • P. Cisek et al.

    Simultaneous encoding of multiple potential reach directions in dorsal premotor cortex

    Journal of Neurophysiology

    (2002)
  • A. Gail et al.

    Neural dynamics in monkey parietal reach region reflect context-specific sensorimotor transformations

    Journal of Neuroscience

    (2006)
  • V. Gallese et al.

    The brain’s concepts: The role of the sensory-motor system in conceptual knowledge

    Cognitive Neuropsychology

    (2005)
  • Cited by (42)

    • The action-sentence compatibility effect (ACE): Meta-analysis of a benchmark finding for embodiment

      2022, Acta Psychologica
      Citation Excerpt :

      For responding, participants pressed a button with the hand or tapped a foot pedal with the foot (Ahlberg et al., 2013; Dalla Volta et al., 2009; Klepp et al., 2017; Miller & Kaup, 2020). Other examples of variation in the dimension involve backward vs. forward-leaning body postures (Zwaan et al., 2012), nodding vs. shaking head movements (Moretti & Greco, 2018), open vs. closed hands (Cardona et al., 2014; Melloni et al., 2015), different types of functional and volumetric gestures (Bub & Masson, 2010, 2012), pressing vs. turning movements (van Dam et al., 2013), and many others. In addition to these studies looking at the interaction between language and action, there are studies investigating interactions between language and perception.

    • The Role of Motor Action in Memory for Objects and Words

      2016, Psychology of Learning and Motivation - Advances in Research and Theory
    • Learning verbs more effectively through meaning congruent action animations

      2015, Learning and Instruction
      Citation Excerpt :

      Depending upon whether sentences put initial focus on the conceptual level or perceptual-motor level information, grasping was executed more rapidly (Bub & Masson, 2010; Masson, et al. 2013). In both Ondobaka et al. (2012) and Bub and Masson (2010), the manipulation of perceptual-motor and conceptual level information is not exactly the same as our implementation. They examined the effect of perceptual-motor and conceptual level information on performing actions.

    • Verb gapping: An action-gap compatibility study

      2015, Acta Psychologica
      Citation Excerpt :

      One important line of evidence – which also is of immediate relevance to the study to be presented in the present paper – stems from studies addressing the interaction between language comprehension and action and indicating that processing linguistic input can affect motor actions. In particular, many studies revealed an action–sentence compatibility effect (Glenberg & Kaschak, 2002), i.e., response times for sentences describing an action in a particular direction were found to be affected by whether the direction of the hand movement required for responding was compatible or incompatible to the direction implied by the sentence (e.g., Bergen & Wheeler, 2005; de Vega, Moreno, & Castillo, 2013; Glenberg & Kaschak, 2002; Zwaan & Taylor, 2006; see also Bub & Masson, 2010). By now, the embodied-simulation view has gained considerable importance in research on language comprehension.

    View all citing articles on Scopus
    View full text