Elsevier

NeuroImage

Volume 36, Issue 3, 1 July 2007, Pages 933-942
NeuroImage

Interactions between working memory and visual perception: An ERP/EEG study

https://doi.org/10.1016/j.neuroimage.2007.04.014Get rights and content

Abstract

How do working memory and perception interact with each other? Recent theories of working memory suggest that they are closely linked, and in fact share certain brain mechanisms. We used a sequential motion imitation task in combination with EEG and ERP techniques for a direct, online examination of memory load's influence on the processing of visual stimuli. Using a paradigm in which subjects tried to reproduce random motion sequences from memory, we found a systematic decrease in ERP amplitude with each additional motion segment that was viewed and memorized for later imitation. High-frequency (> 20 Hz) oscillatory activity exhibited a similar position-dependent decrease. When trials were sorted according to the accuracy of subsequent imitation, the amplitude of the ERPs during stimulus presentation correlated with behavioral performance: the larger the amplitude, the more accurate the subsequent imitation. These findings imply that visual processing of sequential stimuli is not uniform. Rather, earlier information elicits stronger neural activity. We discuss possible explanations for this observation, among them competition for attention between memory and perception and encoding of serial order by means of differential activation strengths.

Introduction

The interplay between working memory and visual perception has been the subject of considerable debate. Even though working memory has received much interest from researchers in psychology and cognitive neuroscience, remarkably little is known about the mechanisms on which it depends, or about the factors that set its limits. At the theoretical level, two general classes of models posit different interactions between memory and perception. One approach conceptualizes working memory as a set of specialized buffers for the storage of information (Baddeley, 2003), which are controlled by an attention-based structure, the central executive. The multi-component, modular model emphasizes limited amounts of activation and temporal decay in the buffers as the sources of capacity limitations in working memory (Baddeley and Logie, 1999). Physiological demonstrations of activity in the prefrontal cortex (PFC) during the memory retention period have led to the identification of the PFC as the neural substrate for the proposed storage buffers (Goldman-Rakic, 1987, Postle, 2006). An alternative view treats working memory not as a separate module, but as an emergent property, which harnesses existing neural mechanisms specialized for sensory perception and long-term memory representations (Cowan, 1999, Cowan, 2000, Jonides et al., 2005, Pasternak and Greenlee, 2005, Postle, 2006). By this account, after the visual stimulus has disappeared from sight, visuospatial working memory is achieved by prolonged activations, via attention, of the same occipital and parietal regions that are thought to mediate visual perception (Druzgal and D'Esposito, 2001, Postle et al., 2003, Todd and Marois, 2004, Vogel and Machizawa, 2004). In the “embedded-process”, or “emergent property” framework, the prefrontal cortex does not provide the actual substrate for memory storage, but rather mediates attentional control of the sensory reactivation process (Curtis and D'Esposito, 2003, Lebedev et al., 2004, Postle, 2005). Consequently, this approach emphasizes attentional control as a limiting factor in working memory capacity (Cowan, 1999, Cowan, 2000).

Attempts to choose between these theoretical accounts have produced mixed results. The key issues that distinguish the two accounts are (i) the degree to which short-term storage of visual information overlaps with early stages of visual processing, and (ii) whether memory's capacity limit is dictated by some limit on attentional selection of visual input. Several groups used visual search paradigms in conjunction with a working memory task, measuring the degree to which the content of working memory affects search efficiency. Woodman et al. (2001) found no detrimental effect on visual search when subjects had to concurrently remember a visual object. More recently, however, they found that, when some location in space had to be remembered, the slope of the function relating reaction time to search array size increased, indicating reduced search efficiency (Woodman and Luck, 2004). A similar result was reported by Oh and Kim (2004). Olivers et al. (2006) explored the relation between the content of working memory and the items in the search array and reported that loading working memory with information increased interference from singleton distractors (see also Lavie and de Fockert, 2005), especially when those distractors were identical to, or shared features with, the remembered material. This is inconsistent with studies by Downing and Dodds (2004) and Woodman and Luck (2007), which found no such interference effects. A second line of inquiry examined how perceptual processing is affected by a concurrent working memory task. Awh and colleagues, using functional imaging (Awh et al., 1999), ERPs (Awh et al., 2000) and behavioral measures (Awh et al., 1998), showed that remembering spatial locations has the same perceptual consequences as spatial attention, i.e., increased sensitivity to visual stimuli at those locations, as measured by BOLD signal changes, ERP amplitude and reaction times. Theeuwes et al. (2005) demonstrated the same principle using eye movements: remembering a location caused the subjects' gaze to deviate, just as if they were directing their attention towards that location. Downing (2000) showed that, when subjects remembered a face at one of two locations, perceptual judgments about stimuli subsequently shown in that location were faster than in the other location. In the verbal domain, Shulman and Greenberg (1971) observed a perceptual deficit, as measured by response time, in recognizing a digit while remembering a list of consonants, a deficit that depended on list length. Fougnie and Marois (2006) used a dual-task study that joined multiple object tracking and working memory. Somewhat in contradiction with other studies, they found that, although the two tasks interfered with each other, working memory capacity cannot be explained solely by limits on attention, and working memory likely involves its own distinct, capacity-limited subprocesses.

We believe one useful approach, which has not been used so far, is to directly measure neural responses to visual stimuli at the same time they are being loaded into working memory, rather than use indirect measures, such as visual search, or task-irrelevant stimuli, like those used in the spatial memory studies. We also sought to use a single-task paradigm, which would eliminate the complexities and added cost of handling two tasks at the same time (Olivers et al., 2006, Lavie et al., 2004). We reasoned this could be achieved by means of a procedure in which items of visual information are presented sequentially, so that subjects have to keep early items in working memory while continuing to encode each subsequently presented item. Under these conditions, perceptual responses can be gauged in the presence of a steadily increasing load on working memory. By recording neural responses to the presentation of each additional item, we could measure the consequences of the growth in the amount of stored information. Scalp EEG recordings provide a good basis for such an analysis: first, they provide excellent temporal resolution, which is essential for evaluating differences between responses to closely spaced stimuli. Second, ERP and EEG markers provide valuable information about visual perception: ERP amplitude is known to correlate with attentive visual processing (Hillyard and Münte, 1984, Hillyard et al., 1998, Luck et al., 2000, Awh et al., 2000), as does activity in the high-frequency (beta and gamma) bands of the EEG (Gruber et al., 1999, Müller et al., 2000, Tallon-Baudry et al., 2005). Thus, we used those electrophysiological markers to track changes in subjects' processing of incoming visual motion information while they were attempting to hold previously seen motion in working memory.

We recorded scalp EEG from human adults who performed a sequential imitation task (Agam et al., 2005, Agam et al., 2007). Fig. 1A shows a schematic diagram of the experimental paradigm. On each trial, subjects viewed a moving disc whose trajectory comprised five randomly oriented, connected linear segments. Then, several seconds later, subjects used a stylus and a graphic tablet to reproduce the trajectory from memory (see also Supplementary video clips). We focused on the period during which subjects were viewing the moving disc, the idea being that, as the disc progresses, there is more that the subject has to hold in memory of what he or she has already seen, so we would be measuring responses to the disc's motion under conditions of varying load in working memory.

Section snippets

Subjects and procedure

Seventeen right-handed subjects (8 male, 9 female, age range 18–26) participated after providing written informed consent. Each of the observers performed between 200 and 240 trials of the imitation task (memory condition). Each motion stimulus was generated by the steady movement of a yellow disc (1° visual angle in diameter) against a black background on a computer screen, which subjects viewed from a distance of 57 cm. Each model comprised a novel set of five directed motion segments, each

Results

As explained earlier, to score the accuracy of subjects' reproductions in the memory condition, we defined the error for each segment as the absolute difference in orientation between the reproduced segment and the corresponding segment in the stimulus. Behaviorally, the results demonstrated a pronounced primacy effect and a modest, one-item recency effect (Fig. 1B), confirming previous findings with this paradigm (Agam et al., 2005).

Fig. 1C shows event-related potentials (ERPs) at five midline

Discussion

The results reported here provide evidence that working memory load modulates neural responses to visual motion stimuli. As more segments had to be held in memory, ERP amplitude decreased. The same pattern was observed for high-frequency oscillations.

Using a perceptual control task with identical stimuli, we demonstrated that the observed drop in amplitude is indeed due to the requirement to remember the motion sequences.

One important question is what the ERPs represent: do they really index

Acknowledgments

We thank Jessica Maryott for collecting eye-tracking data. Supported by NSF grant SBE-0354378 and NIH grant R01MH068404.

References (63)

  • B.R. Postle

    Working memory as an emergent property of the mind and brain

    Neuroscience

    (2006)
  • B.R. Postle et al.

    Seeking the neural substrates of visual working memory storage

    Cortex

    (2003)
  • B. Rolke et al.

    Topography of brain electrical activity dissociates the sequential order transformation of verbal versus spatial information in humans

    Neurosci. Lett.

    (2000)
  • D.S. Ruchkin et al.

    Short-term memory storage and retention: an event-related brain potential study

    Electroencephalogr. Clin. Neurophysiol.

    (1990)
  • P.B. Sederberg et al.

    Oscillatory correlates of the primacy effect in episodic memory

    NeuroImage

    (2006)
  • N.K. Squires et al.

    Two varieties of long-latency positive waves evoked by unpredictable auditory stimuli

    Electroencephalogr. Clin. Neurophysiol.

    (1975)
  • Y. Agam et al.

    Imitating unfamiliar sequences of connected linear motions

    J. Neurophysiol.

    (2005)
  • Y. Agam et al.

    Learning to imitate novel motion sequences

    J. Vis.

    (2007)
  • M.A. Arbib et al.

    The mirror system hypothesis: from a macaque-like mirror system to imitation

  • E. Awh et al.

    Rehearsal in spatial working memory

    J. Exp. Psychol. Hum. Percept. Perform.

    (1998)
  • E. Awh et al.

    Rehearsal in spatial working memory: evidence from neuroimaging

    Psychol. Sci.

    (1999)
  • E. Awh et al.

    The role of spatial selective attention in working memory for locations: evidence from event-related potentials

    J. Cogn. Neurosci.

    (2000)
  • A. Baddeley

    Working memory: looking back and looking forward

    Nat. Rev., Neurosci.

    (2003)
  • A.D. Baddeley et al.

    Working memory: the multiple component model

  • M.M. Botvinick et al.

    Short-term memory for serial order: a recurrent neural network model

    Psychol. Rev.

    (2006)
  • J.F. Brown et al.

    The path of seen movement as a function of the vector-field

    Am. J. Psychol.

    (1937)
  • D. Bullock et al.

    Competitive queuing for serial planning and performance

  • N. Cowan

    An embedded-process model of working memory

  • N. Cowan

    The magical number 4 in short-term memory: a reconsideration of mental storage capacity

    Behav. Brain Sci.

    (2000)
  • P.E. Downing

    Interactions between visual working memory and selective attention

    Psychol. Sci.

    (2000)
  • P.E. Downing et al.

    Competition in visual working memory for control of search

    Vis. Cogn.

    (2004)
  • Cited by (36)

    • Oscillatory delta and theta frequencies differentially support multiple items encoding to optimize memory performance during the digit span task

      2022, NeuroImage
      Citation Excerpt :

      And as we expected, in line with Cowan's suggested mental storage capacity (3-4 items) and Nogueira et al. (2015)’s ERP study, event-related theta oscillations showed a decreasing pattern starting with the 3rd or 4th item. Similarly, Agam & Sekuler (2007) showed that EEG response declined as more chunks had to be stored, and they discussed it as the “interactions between working memory and visual perception” (Agam & Sekuler, 2007). They suggested memory load influences how the brain reacts to visual inputs.

    • Ensemble coding of average length and average orientation are correlated

      2021, Vision Research
      Citation Excerpt :

      The visual world is structured and contains groups of similar objects, like individual faces in a crowd or blades of grass in a field. Faced with limitations due to eye movements (Inamdar & Pomplun, 2003), working memory capacity (Agam & Sekuler, 2007; Cohen, Dennett, & Kanwisher, 2016; Luck & Vogel, 2013), attentional resources (Ghorashi et al., 2007; Wolfe, Reinecke, & Brawn, 2006) and visual crowding (Levi, 2008; Millin, Arman, Chung, & Tjan, 2014), one way to simplify complex scenes is to summarize redundant information into an ensemble percept (Whitney & Yamanashi Leib, 2018). When the features of individual items are not important, ensemble perception can be a fast (Ariely, 2001; Haberman & Whitney, 2007) and efficient (Alvarez, 2011) way to deal with a large number of items.

    • Prefrontal lesions disrupt oscillatory signatures of spatiotemporal integration in working memory

      2021, Cortex
      Citation Excerpt :

      Still, the lesion group performed well above chance (one-tailed t-test, p < 10−11), demonstrating that WM for spatiotemporal relations is only partially dependent on PFC. These results replicate the behavioral PFC lesion effects observed across all WM conditions, including identity trials (Johnson et al., 2017), suggesting that PFC supports spatiotemporal integration as part of a domain-general role. We used ERPs to quantify the time-course of spatiotemporal integration across the whole brain (Agam & Sekuler, 2007; Schupp et al., 2004; Woodman, 2010). Previous analysis of this dataset indicated that PFC lesions did not significantly impact pre-cue maintenance ERPs across all WM conditions (Johnson et al., 2017), suggesting that PFC does not govern maintenance-related ERPs independent of spatiotemporal integration.

    • Temporal multimodal data synchronisation for the analysis of a game driving task using EEG

      2014, Entertainment Computing
      Citation Excerpt :

      Event locking is a common methodology in EEG studies of signals before and after the event. Thereafter the signals will be averaged over many events and/or various participants [35,39–41]. The difficulty with this approach is isolating and distinguishing a discrete cognitive event in a driving task.

    View all citing articles on Scopus
    View full text