Elsevier

NeuroImage

Volume 61, Issue 4, 16 July 2012, Pages 1195-1205
NeuroImage

The fraction of an action is more than a movement: Neural signatures of event segmentation in fMRI

https://doi.org/10.1016/j.neuroimage.2012.04.008Get rights and content

Abstract

When we observe an action, we recognize meaningful action steps that help us to predict probable upcoming action steps. This segmentation of observed actions, or more generally events, has been proposed to rely in part on changes in motion features. However, segmentation of actions, in contrast to meaningless movements, may exploit additional information such as action knowledge. The present fMRI study sought to tear apart the neural signatures of processing two sources of information that observers may exploit at action boundaries: change in motion dynamics and action knowledge. To this end, subjects performed a segmentation task on both actions (that can be segmented based on motion and action knowledge) as well as tai chi movements (that can be segmented only based on motion) and two further control conditions that implemented point-light walker like displays of the same videos. Behavioral tests showed that motion features played a critical role in boundary detection in all conditions. Consistent with this finding, activity in area MT was enhanced during boundary detection in all conditions, but importantly, this effect was not stronger for actions. In contrast, only action boundary detection was reflected by specific activation in the superior frontal sulcus, parietal angular gyrus and the parahippocampal cortex. Based on these findings, we propose that during action observation, motion features trigger a top–down modulation of the attentional focus and the incitement of retrieving long-term memory place-action associations. While action perception entails activity common to processing of all motion stimuli, it is at the same time unique as it allows long-term memory based predictions of succeeding steps.

Introduction

A considerable amount of what we perceive consists of dynamic change. Motion, language, music or actions can all be conceived of as providing an ongoing stream of structured perceivable change. From a biological perspective, events entail change, and catch our particular attention because they often imply the need to adapt one's behavior; therefore, perceiving events leads to the build-up of implicit or explicit expectations (Kurby and Zacks, 2008, Schiffer et al., 2012, Schubotz, 2007, Schütz-Bosbach and Prinz, 2007).

However, some fractions of such events are more predictive of their respective next instance than others, and thus, to an observer, prediction of the event undergoes fluctuations of more certain and less certain phases. For actions, this has been demonstrated in a series of behavioral studies using the unit marking procedure developed by Newtson (1973). In this paradigm, subjects were asked to press a response button when they judged one meaningful action part to end and the next to start. The time points indicated by these judgments are called event boundaries. Using this approach, Newtson and Engquist (1976) showed (a) that perceivers are better in detecting manipulations of the stimulus material that occur at boundaries, (b) that they rely on boundaries when reconstructing an action's meaning, and (c) that recognition memory is enhanced for event boundaries compared to the remaining components of an event. The authors argued that the objective quality of boundaries is that within the continuous sequence boundaries convey a higher amount of information than the remainder of the event, called non-boundaries hereafter (Newtson and Engquist, 1976). In this regard, action boundaries compare to chunk boundaries in general. Both can be described as the first elements to sequentially grouped bits of information, which are considered to be charged with information about all upcoming (chunk) elements (Koch and Hoffmann, 2000, Restle and Brown, 1970, Rosenbaum et al., 1983).

While action observation has attracted considerable attention in human imaging research (Decety and Grezes, 1999, Rizzolatti et al., 2001), most studies have focused on very short actions of limited complexity, i.e., single segments or chunks. Thus, action has been treated as one homogeneous bit of information, not as a structured event. Hence, we are largely ignorant of the neural processes that occur at or around action boundaries. Recently, Zacks and co-workers (Zacks and Sargent, 2009, Zacks et al., 2007) have put forward the Event Segmentation Theory, proposing that a set of event models bias processing in the perceptual stream. The authors propose that prediction undergoes fluctuations of certainty: when an event boundary is approached, prediction certainty declines leading to a gating mechanism and thus an update of the current event model (Zacks et al., 2007). Zacks and co-workers (Speer et al., 2003, Zacks et al., 2001) report motion selective area MT and the junction of the left inferior frontal and precentral sulcus to show enhanced activity when subjects perceive boundaries in everyday action. The finding was interpreted as indicating that event segmentation could be driven by prominent features of the observed trajectories, i.e., by perceptual rather than top–down cognitive processing (Zacks et al., 2001). Greater motion changes at event boundaries, as an example of prominent motion features, may thus directly trigger MT activation, which could in turn propagate to prefrontal areas (Speer et al., 2003).

While it is conceivable that the sight of objects or characteristic movement patterns of the body or the manipulating hands triggers the recognition of the upcoming action step more or less directly, there is usually a set of possible next action steps, and hence competition among these options has to be resolved. Since probabilities of these different action options differ, the selection process may be realized by a Bayesian function of probabilistically weighted forward models (Körding and Wolpert, 2006). Moreover, purely stimulus-triggered recognition implies a process following the stimulus, whereas the notion of predictive perception (Bubic et al., 2009, Grush, 2004, Schiffer and Schubotz, 2011, Zacks et al., 2011) means preparation for early selection among available cues. Here, a frontal top–down signal of transiently enhanced control or selection mechanisms would be expected (Miller and Cohen, 2001, O'Reilly et al., 2002, Ridderinkhof et al., 2004a, Ridderinkhof et al., 2004b). The capacity to predict ongoing action relies on learnt event models (Friston et al., 2011, Neal and Kilner, 2010, Schiffer et al., 2012), therefore activity would be expected in long-term memory-related areas. Further evidence for this long-term memory influence lies in differences between adults' and children's segmentation performances (Baird and Baldwin, 2001).

We agree that action boundaries are relevant nodes for an observer of a continuous action stream, and at these nodes, an update of the internal or event model occurs, probably triggered in part by changes in motion features (Zacks et al., 2001). However, we took a somewhat different view in that we explicitly wished to control for the influence or usage of movement features as cues for action boundaries. While movement information probably serves as a good cue for action segmentation we propose that action script knowledge has to be a further source of information that is exploited here. Changes in movement features may also occur in other kinds of events, including biological and non-biological motion, and we aimed to find out whether area MT is activated specifically for boundaries in goal-directed action or not.

Moreover, we expected the segmentation of action, but not segmentation in a non-action baseline condition (see below), to draw on prefrontal sites. This hypothesis was based mainly on patient studies showing that chunking action segments into longer, more complex actions relies predominantly on the anterior frontal lobes (Allain et al., 1999, Sirigu et al., 1995, Zalla et al., 2001). Thus, the prefrontal cortex should contribute to the selection among competing alternative action steps on the basis of action scripts. Moreover, prefrontal cortex has been associated with the suppression of dominant but currently unwarranted decisions/actions (Frank and Claus, 2006, Ghahremani et al., 2010, Kuhl et al., 2007, Miller and Cohen, 2001, Ridderinkhof et al., 2004a), a function that would be required when preceding action steps bias or restrict the expectation of the upcoming action steps.

Participants were presented with movie clips showing an actress performing everyday actions such as ironing a shirt or hanging out the laundry. Actions were of a duration of about 1 min and comprised of several action steps; thus, ironing a shirt included for example carrying the washing basket to the ironing board, placing the shirt on the board, switching on the iron, starting to move the iron over the shirt's sleeves and so forth. We used the unit marking procedure developed by Newtson (1973). That is, participants were asked to press a response button whenever they felt that the action proceeded to its next step. No feedback was provided for this purely subjective judgment.

In order to identify action-specific neural correlates of event segmentation, we implemented a control condition where subjects were asked to perform the same segmentation task as in the action observation while watching movie clips that showed an actor or actress performing whole-body tai chi movements. Action-specific boundary detection was tackled by calculating the interaction contrast [(action boundary vs. action non-boundary) vs. (tai chi boundary vs. tai chi non-boundary)]. Importantly, the tai chi condition allowed us to control for the perception of a human being in motion, as well as for button presses. As the segmentation of tai chi movements based on changes in movement patterns at event boundaries, this condition was moreover used to control for those portions of boundary judgments that merely relied on changes of spatiotemporal dynamics of the body.

To investigate whether area MT is particularly sensitive towards action boundaries as compared to movement boundaries, we implemented a region of interest (ROI) analysis that was based on the area MT coordinates reported by Zacks et al. (2001). We tested whether we could replicate the finding that action boundaries yielded a higher response than non-boundaries in area MT, as reported by Zacks et al. (2001), corresponding to a main effect of factor PART (boundary > non-boundary). Moreover, we wanted to test whether this effect was disproportionally large for actions, corresponding to an interaction between the factors CONDITION (action, tai chi) and PART (boundary, non-boundary) area MT.

Regarding behavioral analyses, we aimed to assess that subjects' response behavior was meaningful and consistent, not random button pressing. This was particularly important because the task we implemented was a subjective judgment; therefore, the recorded responses could be neither right nor wrong. Accordingly, we assessed test–retest reliability of response patterns and comparability of conditions on the single subject as well as on the group level.

Section snippets

Methods

The present study consisted of a pilot study that included two sessions and one fMRI session. The task (as will be described below) was identical in all sessions. The main idea of the pilot study was to assess comparability between the four conditions that were implemented (see below) and test–retest-reliability. As we employed a subjective judgment task, the pilot studies' tests for comparability and retest reliability were important to establish whether participants responded according to an

Behavioral data, first pilot session

As described in the Methods, boundaries were determined as frames with a frame value two standard deviations above average bin value. As explained above (Fig. 2), frame value is an index of the sum of all participants' responses (aggregated responses) within the next 25-frames bin. An example of a typical movie can be seen in Table 1 and Fig. 3.

Discussion

The current fMRI study investigated whether the detection of meaningful steps in actions relies on more than the detection of change in motion, i.e., dynamic information. In particular, we tested whether motion area (MT) enhancement at action boundaries is specific to action segmentation as compared to segmentation of movements, and whether selection- and long-term-memory related brain areas are particularly elevated at action boundaries, as opposed to movement boundaries.

The behavioral results

Predictive attentional control

Shifts of visual or mnemonic attention are reported to be associated with transient control signals in a fronto-parietal network (for references, see Tamber-Rosenau et al., 2011). Both the middle frontal gyrus and SFS are suggested to play a part in shifts of perceptual and mnemonic attention, but their respective specific functional role remains undescribed so far. Perceptual and mnemonic attention is relevant at event boundaries: Kurby and Zacks (2008) have argued that at event boundaries, a

Adaptation of spatial attention

In addition to aSFS, we found the left posterior angular gyrus and the parahippocampal gyrus bilaterally to be specifically enhanced by boundary detection in actions. Rushworth and colleagues (Rushworth et al., 2006) showed that the posterior angular gyrus can be distinguished from adjacent parietal fields by its distinct connection (via inferior longitudinal fascicle) with the parahippocampal gyrus. This study strongly corroborates a functional connection between angular gyrus and

Relevant action models supplied by long term memory

The medial temporal lobe, including the hippocampal formation, is established for long-term episodic and semantic memory (Kim and Baxter, 2001). The hippocampus proper is a core structure for encoding spatial environment and relations in long-term memory (Rosenbaum et al., 2004). However, mental navigation in old environments was found to rather rely on the parahippocampal gyrus (Maguire et al., 1997, Rosenbaum et al., 2004 Shelton and Gabrieli, 2002). Thus, parahippocampal gyrus seems to

Concluding remarks

Actions and movements provide a continuous stream of input to observers. In this study we found that individuals are very consistent when judging upon meaningful segments or steps, albeit high inter-individual variability. Imaging results indicate that motion serves as a bottom–up cue for boundary detection in different types of dynamic stimuli, but that only for actions, this information also triggers a long-term memory search that guides further expectations. The emerging picture is that

Acknowledgments

The authors wish to cordially thank David Horbank, Marcel Mücke, Stephan Liebig, Sarah Schräder, Christiane Ahlheim and Andreas Johnen for their valuable contributions.

References (85)

  • I.R. Olson et al.

    Some surprising findings on the involvement of the parietal lobe in human memory

    Neurobiol. Learn. Mem.

    (2009)
  • K.R. Ridderinkhof et al.

    Neurocognitive mechanisms of cognitive control: the role of prefrontal cortex in action selection, response inhibition, performance monitoring, and reward-based learning

    Brain Cogn.

    (2004)
  • R.S. Rosenbaum et al.

    New views on old memories: re-evaluating the role of the hippocampal complex

    Behav. Brain Res.

    (2001)
  • M.F.S. Rushworth et al.

    The left parietal and premotor cortices: motor attention and selection

    Neuroimage

    (2003)
  • R.I. Schubotz

    Prediction of external events with our motor system: towards a new framework

    TICS

    (2007)
  • A. Sirigu et al.

    Selective impairments in managerial knowledge following pre-frontal cortex damage

    Cortex

    (1995)
  • B.A. Strange et al.

    Information theory, novelty and hippocampal responses: unpredicted or unpredictable?

    Neural Netw.

    (2005)
  • C.M. Thiel et al.

    Cerebral correlates of alerting, orienting and reorienting of visuospatial attention: an event-related fMRI study

    Neuroimage

    (2004)
  • K.J. Worsley et al.

    Analysis of FMRI time series revisited — again

    Neuroimage

    (1995)
  • T. Zalla et al.

    Action planning in a virtual context after prefrontal cortex damage

    Neuropsychologia

    (2001)
  • G.K. Aguirre et al.

    The parahippocampus subserves topographical learning in man

    Cereb. Cortex

    (1996)
  • P. Allain et al.

    Mental representation of knowledge following frontal-lobe lesion: dissociations on tasks using scripts

    J. Clin. Exp. Neuropsychol.

    (1999)
  • E. Aminoff et al.

    The parahippocampal cortex mediates spatial and nonspatial associations

    Cereb. Cortex

    (2007)
  • J.A. Baird et al.

    Making sense of human behavior: action parsing and intentional inference

  • M. Bar et al.

    Scenes unseen: the parahippocampal cortex intrinsically Subserves contextual associations, not scenes or places per se

    J. Neurosci.

    (2008)
  • C. Bledowski et al.

    What “works” in working memory? Separate systems for selection and updating of critical information

    J. Neurosci.

    (2009)
  • A. Bubic et al.

    Prediction, cognition and the brain

    Front. Hum. Neurosci.

    (2009)
  • C. Cavada et al.

    Posterior parietal cortex in rhesus-monkey. 2. Evidence for segregated corticocortical networks linking sensory and limbic areas with the frontal-lobe

    J. Comp. Neurol.

    (1989)
  • M. Corbetta et al.

    Control of goal-directed and stimulus-driven attention in the brain

    Nat. Rev. Neurosci.

    (2002)
  • K. Christoffersen et al.

    Discovering the events expert practitioners extract from dynamic data streams: the mUMP technique

    Cogn. Technol. Work

    (2007)
  • E. Düzel et al.

    Human hippocampal and parahippocampal activity during visual associative recognition memory for spatial and nonspatial stimulus configurations

    J. Neurosci.

    (2003)
  • L.L. Eldridge et al.

    Remembering episodes: a selective role for the hippocampus during retrieval

    Nat. Neurosci.

    (2000)
  • R.A. Epstein et al.

    A cortical representation of the local visual environment

    Nature

    (1998)
  • R.A. Epstein et al.

    How reliable are visual context effects in the parahippocampal place area?

    Cereb. Cortex

    (2010)
  • M.J. Frank et al.

    Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal

    Psychol. Rev.

    (2006)
  • K.J. Friston et al.

    Action understanding and active inference

    Biol. Cybern.

    (2011)
  • D.G. Ghahremani et al.

    Neural components underlying behavioral flexibility in human reversal learning

    Cereb. Cortex

    (2010)
  • R. Grush

    The emulation theory of representation: motor control, imagery, and perception

    Behav. Brain Sci.

    (2004)
  • S.M. Hayes et al.

    The effect of scene context on episodic object recognition: parahippocampal cortex mediates memory encoding and retrieval success

    Hippocampus

    (2007)
  • K. Henke et al.

    Human hippocampus establishes associations in memory

    Hippocampus

    (1997)
  • K. Henke et al.

    Human hippocampus associates information in memory

    PNAS

    (1999)
  • J.M. Kincade et al.

    An event-related functional magnetic resonance imaging study of voluntary and stimulus-driven orienting of attention

    J. Neurosci.

    (2005)
  • Cited by (27)

    • Touching events predict human action segmentation in brain and behavior

      2021, NeuroImage
      Citation Excerpt :

      On the one hand, activity increased in the anterior dorsal insula (dAI) alerting to a behaviorally important event (Han et al., 2019; Tamber-Rosenau et al., 2018), dorsal anterior cingulate cortex (dACC), which is engaged in saliency detection and attention switching (Han et al., 2019), and finally the inferior frontal junction (IFJ) proposed to subserve transient, dynamic attentional reconfiguration (Sundermann and Pfleiderer, 2012; Xu, 2014). On the other hand, two components that we formerly identified for action segmentation (Schubotz et al., 2012), superior frontal sulcus (SFS) and parahippocampal gyrus (PHG), could now be objectively attributed to the processing of untouching. SFS/PMd serve the selection between alternative competing motor acts based on conditional operations (Petrides, 2005; Tamber-Rosenau et al., 2011).

    • Intact action segmentation in Parkinson's disease: Hypothesis testing using a novel computational approach

      2015, Neuropsychologia
      Citation Excerpt :

      Pseudo-randomisation ensured that each of the 12 movies appeared in all possible conditions across participants: first sessions ON medication, first sessions OFF medication, second sessions ON medication and second sessions OFF medication. This setup allowed us to measure reliability scores and movie specific segmentation independent of order and medication effects (please refer to Schubotz et al., 2012 for a comparable design in a study with young healthy volunteers). Within the segmentation task, participants were instructed to indicate with a button press whenever a new action step began.

    View all citing articles on Scopus
    View full text