The fraction of an action is more than a movement: Neural signatures of event segmentation in fMRI
Introduction
A considerable amount of what we perceive consists of dynamic change. Motion, language, music or actions can all be conceived of as providing an ongoing stream of structured perceivable change. From a biological perspective, events entail change, and catch our particular attention because they often imply the need to adapt one's behavior; therefore, perceiving events leads to the build-up of implicit or explicit expectations (Kurby and Zacks, 2008, Schiffer et al., 2012, Schubotz, 2007, Schütz-Bosbach and Prinz, 2007).
However, some fractions of such events are more predictive of their respective next instance than others, and thus, to an observer, prediction of the event undergoes fluctuations of more certain and less certain phases. For actions, this has been demonstrated in a series of behavioral studies using the unit marking procedure developed by Newtson (1973). In this paradigm, subjects were asked to press a response button when they judged one meaningful action part to end and the next to start. The time points indicated by these judgments are called event boundaries. Using this approach, Newtson and Engquist (1976) showed (a) that perceivers are better in detecting manipulations of the stimulus material that occur at boundaries, (b) that they rely on boundaries when reconstructing an action's meaning, and (c) that recognition memory is enhanced for event boundaries compared to the remaining components of an event. The authors argued that the objective quality of boundaries is that within the continuous sequence boundaries convey a higher amount of information than the remainder of the event, called non-boundaries hereafter (Newtson and Engquist, 1976). In this regard, action boundaries compare to chunk boundaries in general. Both can be described as the first elements to sequentially grouped bits of information, which are considered to be charged with information about all upcoming (chunk) elements (Koch and Hoffmann, 2000, Restle and Brown, 1970, Rosenbaum et al., 1983).
While action observation has attracted considerable attention in human imaging research (Decety and Grezes, 1999, Rizzolatti et al., 2001), most studies have focused on very short actions of limited complexity, i.e., single segments or chunks. Thus, action has been treated as one homogeneous bit of information, not as a structured event. Hence, we are largely ignorant of the neural processes that occur at or around action boundaries. Recently, Zacks and co-workers (Zacks and Sargent, 2009, Zacks et al., 2007) have put forward the Event Segmentation Theory, proposing that a set of event models bias processing in the perceptual stream. The authors propose that prediction undergoes fluctuations of certainty: when an event boundary is approached, prediction certainty declines leading to a gating mechanism and thus an update of the current event model (Zacks et al., 2007). Zacks and co-workers (Speer et al., 2003, Zacks et al., 2001) report motion selective area MT and the junction of the left inferior frontal and precentral sulcus to show enhanced activity when subjects perceive boundaries in everyday action. The finding was interpreted as indicating that event segmentation could be driven by prominent features of the observed trajectories, i.e., by perceptual rather than top–down cognitive processing (Zacks et al., 2001). Greater motion changes at event boundaries, as an example of prominent motion features, may thus directly trigger MT activation, which could in turn propagate to prefrontal areas (Speer et al., 2003).
While it is conceivable that the sight of objects or characteristic movement patterns of the body or the manipulating hands triggers the recognition of the upcoming action step more or less directly, there is usually a set of possible next action steps, and hence competition among these options has to be resolved. Since probabilities of these different action options differ, the selection process may be realized by a Bayesian function of probabilistically weighted forward models (Körding and Wolpert, 2006). Moreover, purely stimulus-triggered recognition implies a process following the stimulus, whereas the notion of predictive perception (Bubic et al., 2009, Grush, 2004, Schiffer and Schubotz, 2011, Zacks et al., 2011) means preparation for early selection among available cues. Here, a frontal top–down signal of transiently enhanced control or selection mechanisms would be expected (Miller and Cohen, 2001, O'Reilly et al., 2002, Ridderinkhof et al., 2004a, Ridderinkhof et al., 2004b). The capacity to predict ongoing action relies on learnt event models (Friston et al., 2011, Neal and Kilner, 2010, Schiffer et al., 2012), therefore activity would be expected in long-term memory-related areas. Further evidence for this long-term memory influence lies in differences between adults' and children's segmentation performances (Baird and Baldwin, 2001).
We agree that action boundaries are relevant nodes for an observer of a continuous action stream, and at these nodes, an update of the internal or event model occurs, probably triggered in part by changes in motion features (Zacks et al., 2001). However, we took a somewhat different view in that we explicitly wished to control for the influence or usage of movement features as cues for action boundaries. While movement information probably serves as a good cue for action segmentation we propose that action script knowledge has to be a further source of information that is exploited here. Changes in movement features may also occur in other kinds of events, including biological and non-biological motion, and we aimed to find out whether area MT is activated specifically for boundaries in goal-directed action or not.
Moreover, we expected the segmentation of action, but not segmentation in a non-action baseline condition (see below), to draw on prefrontal sites. This hypothesis was based mainly on patient studies showing that chunking action segments into longer, more complex actions relies predominantly on the anterior frontal lobes (Allain et al., 1999, Sirigu et al., 1995, Zalla et al., 2001). Thus, the prefrontal cortex should contribute to the selection among competing alternative action steps on the basis of action scripts. Moreover, prefrontal cortex has been associated with the suppression of dominant but currently unwarranted decisions/actions (Frank and Claus, 2006, Ghahremani et al., 2010, Kuhl et al., 2007, Miller and Cohen, 2001, Ridderinkhof et al., 2004a), a function that would be required when preceding action steps bias or restrict the expectation of the upcoming action steps.
Participants were presented with movie clips showing an actress performing everyday actions such as ironing a shirt or hanging out the laundry. Actions were of a duration of about 1 min and comprised of several action steps; thus, ironing a shirt included for example carrying the washing basket to the ironing board, placing the shirt on the board, switching on the iron, starting to move the iron over the shirt's sleeves and so forth. We used the unit marking procedure developed by Newtson (1973). That is, participants were asked to press a response button whenever they felt that the action proceeded to its next step. No feedback was provided for this purely subjective judgment.
In order to identify action-specific neural correlates of event segmentation, we implemented a control condition where subjects were asked to perform the same segmentation task as in the action observation while watching movie clips that showed an actor or actress performing whole-body tai chi movements. Action-specific boundary detection was tackled by calculating the interaction contrast [(action boundary vs. action non-boundary) vs. (tai chi boundary vs. tai chi non-boundary)]. Importantly, the tai chi condition allowed us to control for the perception of a human being in motion, as well as for button presses. As the segmentation of tai chi movements based on changes in movement patterns at event boundaries, this condition was moreover used to control for those portions of boundary judgments that merely relied on changes of spatiotemporal dynamics of the body.
To investigate whether area MT is particularly sensitive towards action boundaries as compared to movement boundaries, we implemented a region of interest (ROI) analysis that was based on the area MT coordinates reported by Zacks et al. (2001). We tested whether we could replicate the finding that action boundaries yielded a higher response than non-boundaries in area MT, as reported by Zacks et al. (2001), corresponding to a main effect of factor PART (boundary > non-boundary). Moreover, we wanted to test whether this effect was disproportionally large for actions, corresponding to an interaction between the factors CONDITION (action, tai chi) and PART (boundary, non-boundary) area MT.
Regarding behavioral analyses, we aimed to assess that subjects' response behavior was meaningful and consistent, not random button pressing. This was particularly important because the task we implemented was a subjective judgment; therefore, the recorded responses could be neither right nor wrong. Accordingly, we assessed test–retest reliability of response patterns and comparability of conditions on the single subject as well as on the group level.
Section snippets
Methods
The present study consisted of a pilot study that included two sessions and one fMRI session. The task (as will be described below) was identical in all sessions. The main idea of the pilot study was to assess comparability between the four conditions that were implemented (see below) and test–retest-reliability. As we employed a subjective judgment task, the pilot studies' tests for comparability and retest reliability were important to establish whether participants responded according to an
Behavioral data, first pilot session
As described in the Methods, boundaries were determined as frames with a frame value two standard deviations above average bin value. As explained above (Fig. 2), frame value is an index of the sum of all participants' responses (aggregated responses) within the next 25-frames bin. An example of a typical movie can be seen in Table 1 and Fig. 3.
Discussion
The current fMRI study investigated whether the detection of meaningful steps in actions relies on more than the detection of change in motion, i.e., dynamic information. In particular, we tested whether motion area (MT) enhancement at action boundaries is specific to action segmentation as compared to segmentation of movements, and whether selection- and long-term-memory related brain areas are particularly elevated at action boundaries, as opposed to movement boundaries.
The behavioral results
Predictive attentional control
Shifts of visual or mnemonic attention are reported to be associated with transient control signals in a fronto-parietal network (for references, see Tamber-Rosenau et al., 2011). Both the middle frontal gyrus and SFS are suggested to play a part in shifts of perceptual and mnemonic attention, but their respective specific functional role remains undescribed so far. Perceptual and mnemonic attention is relevant at event boundaries: Kurby and Zacks (2008) have argued that at event boundaries, a
Adaptation of spatial attention
In addition to aSFS, we found the left posterior angular gyrus and the parahippocampal gyrus bilaterally to be specifically enhanced by boundary detection in actions. Rushworth and colleagues (Rushworth et al., 2006) showed that the posterior angular gyrus can be distinguished from adjacent parietal fields by its distinct connection (via inferior longitudinal fascicle) with the parahippocampal gyrus. This study strongly corroborates a functional connection between angular gyrus and
Relevant action models supplied by long term memory
The medial temporal lobe, including the hippocampal formation, is established for long-term episodic and semantic memory (Kim and Baxter, 2001). The hippocampus proper is a core structure for encoding spatial environment and relations in long-term memory (Rosenbaum et al., 2004). However, mental navigation in old environments was found to rather rely on the parahippocampal gyrus (Maguire et al., 1997, Rosenbaum et al., 2004 Shelton and Gabrieli, 2002). Thus, parahippocampal gyrus seems to
Concluding remarks
Actions and movements provide a continuous stream of input to observers. In this study we found that individuals are very consistent when judging upon meaningful segments or steps, albeit high inter-individual variability. Imaging results indicate that motion serves as a bottom–up cue for boundary detection in different types of dynamic stimuli, but that only for actions, this information also triggers a long-term memory search that guides further expectations. The emerging picture is that
Acknowledgments
The authors wish to cordially thank David Horbank, Marcel Mücke, Stephan Liebig, Sarah Schräder, Christiane Ahlheim and Andreas Johnen for their valuable contributions.
References (85)
- et al.
Contributions of the hippocampus and the striatum to simple association and frequency-based learning
Neuroimage
(2005) - et al.
Spatial memory deficits in patients with lesions to the right hippocampus and to the right parahippocampal cortex
Neuropsychologia
(1998) - et al.
Neural mechanisms subserving the perception of human actions
TICS
(1999) - et al.
Nonconscious formation and reactivation of semantic associations by way of the medial temporal lobe
Neuropsychologia
(2003) - et al.
Multiple brain-memory systems: the whole does not equal the sum of its parts
Trends Neurosci.
(2001) - et al.
Bayesian decision theory in sensorimotor control
TICS
(2006) - et al.
Segmentation in the perception and memory of events
TICS
(2008) - et al.
Lipsia — a new software system for the evaluation of functional magnetic resonance images of the human brain
Comput. Med. Imaging Graph.
(2001) - et al.
Differential cortical activation during voluntary and reflexive saccades in man
Neuroimage
(2003) - et al.
The perceptual organization of ongoing behavior
J. Exp. Soc. Psychol.
(1976)
Some surprising findings on the involvement of the parietal lobe in human memory
Neurobiol. Learn. Mem.
Neurocognitive mechanisms of cognitive control: the role of prefrontal cortex in action selection, response inhibition, performance monitoring, and reward-based learning
Brain Cogn.
New views on old memories: re-evaluating the role of the hippocampal complex
Behav. Brain Res.
The left parietal and premotor cortices: motor attention and selection
Neuroimage
Prediction of external events with our motor system: towards a new framework
TICS
Selective impairments in managerial knowledge following pre-frontal cortex damage
Cortex
Information theory, novelty and hippocampal responses: unpredicted or unpredictable?
Neural Netw.
Cerebral correlates of alerting, orienting and reorienting of visuospatial attention: an event-related fMRI study
Neuroimage
Analysis of FMRI time series revisited — again
Neuroimage
Action planning in a virtual context after prefrontal cortex damage
Neuropsychologia
The parahippocampus subserves topographical learning in man
Cereb. Cortex
Mental representation of knowledge following frontal-lobe lesion: dissociations on tasks using scripts
J. Clin. Exp. Neuropsychol.
The parahippocampal cortex mediates spatial and nonspatial associations
Cereb. Cortex
Making sense of human behavior: action parsing and intentional inference
Scenes unseen: the parahippocampal cortex intrinsically Subserves contextual associations, not scenes or places per se
J. Neurosci.
What “works” in working memory? Separate systems for selection and updating of critical information
J. Neurosci.
Prediction, cognition and the brain
Front. Hum. Neurosci.
Posterior parietal cortex in rhesus-monkey. 2. Evidence for segregated corticocortical networks linking sensory and limbic areas with the frontal-lobe
J. Comp. Neurol.
Control of goal-directed and stimulus-driven attention in the brain
Nat. Rev. Neurosci.
Discovering the events expert practitioners extract from dynamic data streams: the mUMP technique
Cogn. Technol. Work
Human hippocampal and parahippocampal activity during visual associative recognition memory for spatial and nonspatial stimulus configurations
J. Neurosci.
Remembering episodes: a selective role for the hippocampus during retrieval
Nat. Neurosci.
A cortical representation of the local visual environment
Nature
How reliable are visual context effects in the parahippocampal place area?
Cereb. Cortex
Anatomy of a decision: striato-orbitofrontal interactions in reinforcement learning, decision making, and reversal
Psychol. Rev.
Action understanding and active inference
Biol. Cybern.
Neural components underlying behavioral flexibility in human reversal learning
Cereb. Cortex
The emulation theory of representation: motor control, imagery, and perception
Behav. Brain Sci.
The effect of scene context on episodic object recognition: parahippocampal cortex mediates memory encoding and retrieval success
Hippocampus
Human hippocampus establishes associations in memory
Hippocampus
Human hippocampus associates information in memory
PNAS
An event-related functional magnetic resonance imaging study of voluntary and stimulus-driven orienting of attention
J. Neurosci.
Cited by (27)
Relation between event segmentation and memory dysfunction in Parkinson's disease
2022, Brain and CognitionTouching events predict human action segmentation in brain and behavior
2021, NeuroImageCitation Excerpt :On the one hand, activity increased in the anterior dorsal insula (dAI) alerting to a behaviorally important event (Han et al., 2019; Tamber-Rosenau et al., 2018), dorsal anterior cingulate cortex (dACC), which is engaged in saliency detection and attention switching (Han et al., 2019), and finally the inferior frontal junction (IFJ) proposed to subserve transient, dynamic attentional reconfiguration (Sundermann and Pfleiderer, 2012; Xu, 2014). On the other hand, two components that we formerly identified for action segmentation (Schubotz et al., 2012), superior frontal sulcus (SFS) and parahippocampal gyrus (PHG), could now be objectively attributed to the processing of untouching. SFS/PMd serve the selection between alternative competing motor acts based on conditional operations (Petrides, 2005; Tamber-Rosenau et al., 2011).
Affordance processing in segregated parieto-frontal dorsal stream sub-pathways
2016, Neuroscience and Biobehavioral ReviewsIntact action segmentation in Parkinson's disease: Hypothesis testing using a novel computational approach
2015, NeuropsychologiaCitation Excerpt :Pseudo-randomisation ensured that each of the 12 movies appeared in all possible conditions across participants: first sessions ON medication, first sessions OFF medication, second sessions ON medication and second sessions OFF medication. This setup allowed us to measure reliability scores and movie specific segmentation independent of order and medication effects (please refer to Schubotz et al., 2012 for a comparable design in a study with young healthy volunteers). Within the segmentation task, participants were instructed to indicate with a button press whenever a new action step began.