Elsevier

Acta Psychologica

Volume 152, October 2014, Pages 75-83
Acta Psychologica

Attentional cueing by cross-modal congruency produces both facilitation and inhibition on short-term visual recognition

https://doi.org/10.1016/j.actpsy.2014.07.008Get rights and content

Highlights

  • Cross-modal congruency captures attention and affects visual working memory (VWM).

  • Recognition of the shape cued by a congruent sound is facilitated after 150 ms.

  • After 1150 ms recognition suffers from inhibition of return (IOR).

  • The amount of IOR is proportional to the amount of initial facilitation.

  • The inverse effectiveness rule of multisensory integration holds also in VWM.

Abstract

The attentional modulation of performance in a memory task, comparable to the one obtained in a perceptual task, is at the focus of contemporary research. We hypothesized that a biphasic effect (namely, facilitation followed by inhibition) can be obtained in visual working memory when attention is cued towards one item of the memorandum and participants must recognize a delayed probe as being identical to any item of the memorandum. In every trial, a delayed spiky/curvy probe appeared centrally, to be matched with the same-category shape maintained in visual working memory which could be either physically identical (positive trials) or only categorically similar (negative trials). To orient the participant's attention towards a selected portion of a two-item memorandum, a (tzk/wow) sound was played simultaneously with two lateral visual shapes (one spiky and one curved). Our results indicate that an exogenous attentional shift during perception of the memorandum, induced by a congruent audio–visual pairing, first facilitates and then inhibits the recognition of a cued item (but not of a non-cued item) stored in visual working memory. A coherent pattern of individual differences emerged, indicating that the amount of early facilitation in congruent-sound trials was negatively correlated with recognition sensitivity in no-sound trials (suggesting that the inverse effectiveness rule may also apply to memory) and positively correlated with later inhibition, as well as with the self-reported susceptibility to memory failures.

Introduction

Competition for representational resources makes attention a crucial factor in both perceptual and reflective domains, supporting comparisons and analogies between external and internal attention that are particularly relevant for research on working memory, which lies at their intersection (Chun, Golomb, & Turk-Browne, 2011). Attention affects encoding (Eger, Henson, Driver, & Dolan, 2004) as well as memory retrieval (Guerin et al., 2012, Wolfe et al., 2006). Selection in the domains of visual perception and visual memory appears to be supported by similar neural mechanisms (Astle et al., 2009, Kuo et al., 2009, Nobre et al., 2004). Spatial memory is distorted by a task-irrelevant exogenous cue (Van der Stigchel, Merten, Meeter, & Theeuwes, 2007), and object memory deteriorates when attention shifts away from object location during retention (Williams, Pouget, Boucher, & Woodman, 2013).1

When observers are required to detect a target following the presentation of an exogenous cue that attracts spatial attention towards its location, the manipulation of cue–target onset asynchrony (CTOA) reveals a characteristic biphasic effect of cueing (Posner & Cohen, 1984). Detection is facilitated at short CTOAs (typically, less than 300 ms), while it deteriorates at longer CTOAs, generating an effect labeled as inhibition of return (IOR).

Classic IOR – that is, the detection loss for visual targets displayed in a previously attended location after a critical CTOA – can be explained by the reorienting hypothesis, which states that attention is automatically attracted towards the location of the peripheral cue (i.e., a lateral flash), but is subsequently disengaged from that particular location, because of a compensatory mechanism that inhibits the return of attention to previously attended locations, to maximize efficiency of visual search in a normally complex environment (Danziger et al., 1998, Klein, 2000). For an alternative, motor-based, view of IOR, as well as for its possible occurrence in a perceptual discrimination task, see Taylor and Donnelly (2002).

Attention can be attracted towards locations, objects, and features (Carrasco, 2011). Accordingly, IOR is not unique to spatial attention, having been found also when attention is object-based (i.e., oriented to targets that belong to a previously attended object; List & Robertson, 2007) and feature-based (i.e., oriented to targets that possess a previously attended feature; Busse, Katzner, & Treue, 2006). Outside the visual domain, IOR also occurs in auditory and audio–visual domains, provided that a second cue redirects attention back to the center (Reuter-Lorenz & Rosenquist, 1996).

Based on the idea that external and internal types of attention may share similar mechanisms, attentional cueing effects have been recently investigated in memory. Besides the well-known facilitatory effect of cueing on encoding (Uncapher, Hutchinson, & Wagner, 2011), attention can improve the maintenance of objects in memory and increase the probability of their recall (Murray, Nobre, Clark, Cravo, & Stokes, 2013). Johnson et al. (2013) manipulated participants' internal attention by presenting two items followed by a cue that required them to selectively think back to (i.e., refresh) only one item. Slower responses for refreshed than unrefreshed items revealed an IOR-like effect within working memory. In this case, internal attention was directed towards the semantic meaning of a word, which was independent of the memory of its spatial location. This again suggests that the memory-based IOR-like effect is not exclusively linked to spatial attention, but can arise from semantic cueing, as described in the perceptual domain (Fuentes, Vivas, & Humphreys, 1999).

Cross-modal congruency refers to the correspondence between inputs to different modalities that can make sounds and visual shapes perceptually similar (Köhler, 1929). Following previous demonstrations that cross-modal congruency can improve recognition (Murray et al., 2004), speed up cross-modal event detection (Makovac & Gerbino, 2010), and modulate attention (Chiou & Rich, 2012), here we asked whether the exogenous cueing of a perceptual event (i.e., the automatic orienting of external attention towards one of two visual shapes) can influence recognition. We expected that cross-modal congruency, by promoting multisensory integration and affecting the deployment of attention, would generate a biphasic effect of cueing on recognition performance. In particular, we explored the possibility that visual working memory (VWM) displays a biphasic effect of attentional cueing (facilitation followed by inhibition). Participants in our experiment were shown one spiky and one curvy shape on the left/right of the fixation point and maintained them in VWM until a central probe prompted for an old/new response based on physical identity. A new response (negative trials) was required when the match between the probe and the item of the memorandum with the same contour type was categorically similar (but not identical), whereas an old response was required when an identical probe was presented.

Our paradigm included multisensory and unisensory conditions. In multisensory conditions the memorandum was presented together with a simultaneous sound whose auditory features were congruent with the features of one of the two visual shapes, giving rise to cued trials. Cued trials were either valid (if sound and probe were congruent) or invalid (if sound and probe were incongruent). The unisensory condition included uncued trials (neutral; no sound was presented). Importantly, the sound in the multisensory conditions was task-irrelevant, thus making cued and uncued (neutral) trials formally equivalent in terms of task demand. Expected effects of cueing (facilitation followed by inhibition) should be dependent on automatic cross-modal binding.

We assumed that: (a) cross-modal congruency promotes the automatic binding of the sound with only one visual shape; (b) attention is exogenously oriented towards the cued shape; (c) multisensory integration enhances encoding (Lehmann and Murray, 2005, Nyberg et al., 2000, Wheeler et al., 2000) by driving exogenous attention towards the multisensory event comprising congruent sound/cued shape combination (Spence et al., 2004, Talsma et al., 2010); and (d) the central probe produces a disengagement of attention from the peripheral cued shape, and gives rise to IOR (Posner & Cohen, 1984).

We expected the following two results:

  • 1)

    Facilitation; when the probe is displayed immediately after the memorandum, recognition in valid trials should be facilitated, because the exogenously cued shape should benefit from enhanced encoding, at the expense of poorer encoding of the uncued shape;

  • 2)

    Inhibition; at a longer probe delay (around 1 s), recognition in valid trials is inhibited, as revealed by a reduction in recognition performance for the exogenously cued shape (Lupiáñez et al., 1997, Massen and Stegt, 2007).

We also explored the correlation between individual differences in performance and the self-reported frequency of cognitive mistakes, measured by the Memory and Distractibility subscales of the Cognitive Failures Questionnaire (CFQ; Broadbent et al., 1982, Wallace et al., 2002). We hypothesized that the amounts of the expected facilitation and inhibition effects (which should reveal how attention operates in VWM) could be positively correlated with the awareness of the propensity to attention and memory errors in everyday life, as measured by the relevant CFQ items. The reliability and validity of CFQ in quantifying the propensity for making mistakes have been extensively studied (Forster and Lavie, 2007, Kanai et al., 2011, Martin and Jones, 1983, Tipper and Baylis, 1987). However, CFQ scores might also reflect metacognitive worries (Mecacci & Righi, 2006) and the tendency to pessimistic self-evaluations (van Doorn, Lang, & Weijters, 2010).

Section snippets

Participants

Twenty right-handed undergraduates (14 females, mean age = 24 years, age range 19–29, SD = 3.5) participated in the experiment. All participants had normal hearing and normal/corrected-to-normal visual acuity. All gave their prior informed consent, were tested individually, and received course credit.

Stimuli, apparatus, and procedure

As regards visual stimuli, inspired by Köhler (1929, Figs. 18 and 19) we generated 40 unfamiliar shapes — 20 with spiky contours containing abrupt and frequent (median = 28 in the 14–44 range)

RTs

Fig. 2 shows mean RTs and s.e.m. in different conditions. Since exploratory 3-way ANOVAs on RT and transformed 1/RT values were consistent, only the outcome for transformed data is reported here. Responses were faster for curvy than spiky probes, 908 vs. 935 ms, F(1, 19) = 7.34, p < .05, ηp2= .28. The main effect of Delay was also significant, F(2, 38) = 27.79, p < .001, ηp2= .73, due to longer RTs at Delay6150 (980 ms) than at Delay150 (904 ms) and Delay1150 (881 ms). Neither the main effect of Validity, F

A biphasic effect of multisensory cueing in memory

We demonstrated a biphasic effect of the multisensory exogenous cueing of attention in a recognition paradigm. A memorandum made of two abstract shapes, one spiky and one curvy, was presented either alone (neutral trials) or together with a simultaneous sound congruent with one shape (valid/invalid trials). After a variable delay, a spiky/curvy probe appeared centrally, either physically identical (positive trials) or only categorically similar (negative trials) to one of the two shapes

References (66)

  • L. Mecacci et al.

    Cognitive failures, metacognitive beliefs and aging

    Personality and Individual Differences

    (2006)
  • M.M. Murray et al.

    Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging

    NeuroImage

    (2004)
  • B.E. Stein et al.

    Neurons and behavior: The same rules of multisensory integration apply

    Brain Research

    (1988)
  • D. Talsma et al.

    The multifaceted interplay between attention and multisensory integration

    Trends in Cognitive Science

    (2010)
  • S.P. Tipper et al.

    Individual differences in selective attention: The relation of priming and interference to cognitive failure

    Personality and Individual Differences

    (1987)
  • K.C. Aberg et al.

    Different types of feedback change decision criterion and sensitivity differently in perceptual learning

    Journal of Vision

    (2012)
  • D.E. Astle et al.

    Spatial selection of features within perceived and remembered objects

    Frontiers in Human Neuroscience

    (2009)
  • P.M. Bays et al.

    Dynamic shifts of limited working memory resources in human vision

    Science

    (2008)
  • G.C. Baylis et al.

    Shape-coding in IT cells generalizes over contrast and mirror reversal, but not figure-ground reversal

    Nature Neuroscience

    (2001)
  • A. Berger et al.

    Competition between endogenous and exogenous orienting of visual attention

    Journal of Experimental Psychology: General

    (2005)
  • P. Bertelson et al.

    The ventriloquist effect does not depend on the direction of deliberate visual attention

    Perception & Psychophysics

    (2000)
  • S. Bona et al.

    Metacognition of visual short-term memory: Dissociation between objective and subjective components of VSWM

    Frontiers in Psychology

    (2013)
  • D.E. Broadbent et al.

    The Cognitive Failures Questionnaire (CFQ) and its correlates

    British Journal of Clinical Psychology

    (1982)
  • R. Chiou et al.

    Cross-modality correspondence between pitch and spatial location modulates attentional orienting

    Perception

    (2012)
  • M.M. Chun et al.

    A taxonomy of external and internal attention

    Annual Review of Psychology

    (2011)
  • S. Danziger et al.

    Inhibition of return to successively stimulated locations in a sequential visual search paradigm

    Journal of Experimental Psychology: Human Perception and Performance

    (1998)
  • J. Driver

    Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading

    Nature

    (1996)
  • E. Eger et al.

    BOLD repetition decreases in object-responsive ventral visual areas depend on spatial attention

    Journal of Neurophysiology

    (2004)
  • S. Forster et al.

    High perceptual load makes everybody equal: Eliminating individual differences in distractibility with load

    Psychological Science

    (2007)
  • L.J. Fuentes et al.

    Inhibitory mechanisms of attentional networks: Spatial and semantic inhibitory processing

    Journal of Experimental Psychology: Human Perception and Performance

    (1999)
  • S.G. Gosling et al.

    Directing spatial attention to locations within remembered and imagined mental representations

    Frontiers in Human Neuroscience

    (2013)
  • S. Hotta et al.

    Visual attention model involving feature-based inhibition of return

    Artificial Life and Robotics

    (2010)
  • M.R. Johnson et al.

    Foraging for thought: An inhibition-of-return-like effect resulting from directing attention within working memory

    Psychological Science

    (2013)
  • Cited by (5)

    • A biphasic effect of cross-modal priming on visual shape recognition

      2018, Acta Psychologica
      Citation Excerpt :

      As regards acoustic stimuli (Fig. 1B), we used Audacity® 2.0 to generate two 200-ms sounds: tzk, a spiky sound with abrupt and frequent changes of intensity, and upo, a soft sound with smooth and infrequent changes of intensity. As in previous studies (Makovac et al., 2014; Makovac & Gerbino, 2010), the tzk sound was taken as congruent with the spiky shape category and incongruent with the curvy shape category, while the upo sound was taken as congruent with the curvy shape category and incongruent with the spiky shape category. Individual values of the da index, used to measure recognition sensitivity in our sequential shape matching task followed by confidence rating, were calculated according to the unequal-variance model of Signal Detection Theory (SDT; Macmillan & Creelman, 2004).

    • The impact of cross-modal correspondences on working memory performance

      2017, Journal of Experimental Psychology: Human Perception and Performance

    Authors thank David Pearson for comments on an earlier version of the manuscript. Supported by Miur-Cofin 2008 (PRIN200879EB93) and EcoAutobus-Industria (MS01_00006) 2015 grants.

    View full text