Attentional cueing by cross-modal congruency produces both facilitation and inhibition on short-term visual recognition☆
Introduction
Competition for representational resources makes attention a crucial factor in both perceptual and reflective domains, supporting comparisons and analogies between external and internal attention that are particularly relevant for research on working memory, which lies at their intersection (Chun, Golomb, & Turk-Browne, 2011). Attention affects encoding (Eger, Henson, Driver, & Dolan, 2004) as well as memory retrieval (Guerin et al., 2012, Wolfe et al., 2006). Selection in the domains of visual perception and visual memory appears to be supported by similar neural mechanisms (Astle et al., 2009, Kuo et al., 2009, Nobre et al., 2004). Spatial memory is distorted by a task-irrelevant exogenous cue (Van der Stigchel, Merten, Meeter, & Theeuwes, 2007), and object memory deteriorates when attention shifts away from object location during retention (Williams, Pouget, Boucher, & Woodman, 2013).1
When observers are required to detect a target following the presentation of an exogenous cue that attracts spatial attention towards its location, the manipulation of cue–target onset asynchrony (CTOA) reveals a characteristic biphasic effect of cueing (Posner & Cohen, 1984). Detection is facilitated at short CTOAs (typically, less than 300 ms), while it deteriorates at longer CTOAs, generating an effect labeled as inhibition of return (IOR).
Classic IOR – that is, the detection loss for visual targets displayed in a previously attended location after a critical CTOA – can be explained by the reorienting hypothesis, which states that attention is automatically attracted towards the location of the peripheral cue (i.e., a lateral flash), but is subsequently disengaged from that particular location, because of a compensatory mechanism that inhibits the return of attention to previously attended locations, to maximize efficiency of visual search in a normally complex environment (Danziger et al., 1998, Klein, 2000). For an alternative, motor-based, view of IOR, as well as for its possible occurrence in a perceptual discrimination task, see Taylor and Donnelly (2002).
Attention can be attracted towards locations, objects, and features (Carrasco, 2011). Accordingly, IOR is not unique to spatial attention, having been found also when attention is object-based (i.e., oriented to targets that belong to a previously attended object; List & Robertson, 2007) and feature-based (i.e., oriented to targets that possess a previously attended feature; Busse, Katzner, & Treue, 2006). Outside the visual domain, IOR also occurs in auditory and audio–visual domains, provided that a second cue redirects attention back to the center (Reuter-Lorenz & Rosenquist, 1996).
Based on the idea that external and internal types of attention may share similar mechanisms, attentional cueing effects have been recently investigated in memory. Besides the well-known facilitatory effect of cueing on encoding (Uncapher, Hutchinson, & Wagner, 2011), attention can improve the maintenance of objects in memory and increase the probability of their recall (Murray, Nobre, Clark, Cravo, & Stokes, 2013). Johnson et al. (2013) manipulated participants' internal attention by presenting two items followed by a cue that required them to selectively think back to (i.e., refresh) only one item. Slower responses for refreshed than unrefreshed items revealed an IOR-like effect within working memory. In this case, internal attention was directed towards the semantic meaning of a word, which was independent of the memory of its spatial location. This again suggests that the memory-based IOR-like effect is not exclusively linked to spatial attention, but can arise from semantic cueing, as described in the perceptual domain (Fuentes, Vivas, & Humphreys, 1999).
Cross-modal congruency refers to the correspondence between inputs to different modalities that can make sounds and visual shapes perceptually similar (Köhler, 1929). Following previous demonstrations that cross-modal congruency can improve recognition (Murray et al., 2004), speed up cross-modal event detection (Makovac & Gerbino, 2010), and modulate attention (Chiou & Rich, 2012), here we asked whether the exogenous cueing of a perceptual event (i.e., the automatic orienting of external attention towards one of two visual shapes) can influence recognition. We expected that cross-modal congruency, by promoting multisensory integration and affecting the deployment of attention, would generate a biphasic effect of cueing on recognition performance. In particular, we explored the possibility that visual working memory (VWM) displays a biphasic effect of attentional cueing (facilitation followed by inhibition). Participants in our experiment were shown one spiky and one curvy shape on the left/right of the fixation point and maintained them in VWM until a central probe prompted for an old/new response based on physical identity. A new response (negative trials) was required when the match between the probe and the item of the memorandum with the same contour type was categorically similar (but not identical), whereas an old response was required when an identical probe was presented.
Our paradigm included multisensory and unisensory conditions. In multisensory conditions the memorandum was presented together with a simultaneous sound whose auditory features were congruent with the features of one of the two visual shapes, giving rise to cued trials. Cued trials were either valid (if sound and probe were congruent) or invalid (if sound and probe were incongruent). The unisensory condition included uncued trials (neutral; no sound was presented). Importantly, the sound in the multisensory conditions was task-irrelevant, thus making cued and uncued (neutral) trials formally equivalent in terms of task demand. Expected effects of cueing (facilitation followed by inhibition) should be dependent on automatic cross-modal binding.
We assumed that: (a) cross-modal congruency promotes the automatic binding of the sound with only one visual shape; (b) attention is exogenously oriented towards the cued shape; (c) multisensory integration enhances encoding (Lehmann and Murray, 2005, Nyberg et al., 2000, Wheeler et al., 2000) by driving exogenous attention towards the multisensory event comprising congruent sound/cued shape combination (Spence et al., 2004, Talsma et al., 2010); and (d) the central probe produces a disengagement of attention from the peripheral cued shape, and gives rise to IOR (Posner & Cohen, 1984).
We expected the following two results:
- 1)
Facilitation; when the probe is displayed immediately after the memorandum, recognition in valid trials should be facilitated, because the exogenously cued shape should benefit from enhanced encoding, at the expense of poorer encoding of the uncued shape;
- 2)
Inhibition; at a longer probe delay (around 1 s), recognition in valid trials is inhibited, as revealed by a reduction in recognition performance for the exogenously cued shape (Lupiáñez et al., 1997, Massen and Stegt, 2007).
We also explored the correlation between individual differences in performance and the self-reported frequency of cognitive mistakes, measured by the Memory and Distractibility subscales of the Cognitive Failures Questionnaire (CFQ; Broadbent et al., 1982, Wallace et al., 2002). We hypothesized that the amounts of the expected facilitation and inhibition effects (which should reveal how attention operates in VWM) could be positively correlated with the awareness of the propensity to attention and memory errors in everyday life, as measured by the relevant CFQ items. The reliability and validity of CFQ in quantifying the propensity for making mistakes have been extensively studied (Forster and Lavie, 2007, Kanai et al., 2011, Martin and Jones, 1983, Tipper and Baylis, 1987). However, CFQ scores might also reflect metacognitive worries (Mecacci & Righi, 2006) and the tendency to pessimistic self-evaluations (van Doorn, Lang, & Weijters, 2010).
Section snippets
Participants
Twenty right-handed undergraduates (14 females, mean age = 24 years, age range 19–29, SD = 3.5) participated in the experiment. All participants had normal hearing and normal/corrected-to-normal visual acuity. All gave their prior informed consent, were tested individually, and received course credit.
Stimuli, apparatus, and procedure
As regards visual stimuli, inspired by Köhler (1929, Figs. 18 and 19) we generated 40 unfamiliar shapes — 20 with spiky contours containing abrupt and frequent (median = 28 in the 14–44 range)
RTs
Fig. 2 shows mean RTs and s.e.m. in different conditions. Since exploratory 3-way ANOVAs on RT and transformed 1/RT values were consistent, only the outcome for transformed data is reported here. Responses were faster for curvy than spiky probes, 908 vs. 935 ms, F(1, 19) = 7.34, p < .05, ηp2= .28. The main effect of Delay was also significant, F(2, 38) = 27.79, p < .001, ηp2= .73, due to longer RTs at Delay6150 (980 ms) than at Delay150 (904 ms) and Delay1150 (881 ms). Neither the main effect of Validity, F
A biphasic effect of multisensory cueing in memory
We demonstrated a biphasic effect of the multisensory exogenous cueing of attention in a recognition paradigm. A memorandum made of two abstract shapes, one spiky and one curvy, was presented either alone (neutral trials) or together with a simultaneous sound congruent with one shape (valid/invalid trials). After a variable delay, a spiky/curvy probe appeared centrally, either physically identical (positive trials) or only categorically similar (negative trials) to one of the two shapes
References (66)
- et al.
No direction-specific bimodal facilitation for audiovisual motion detection
Brain Research. Cognitive Brain Research
(2004) - et al.
Overlapping mechanisms of attention and spatial working memory
Trends in Cognitive Science
(2001) - et al.
Selection within visual memory representations activates the oculomotor system
Neuropsychologia
(2011) Ventriloquism: A case of cross-modal perceptual grouping
- et al.
Spatial and feature-based effects of exogenous cueing on visual motion processing
Vision Research
(2006) Visual attention: The past 25 years
Vision Research
(2011)- et al.
Interactions between visual attention and episodic retrieval: Dissociable contributions of parietal regions during gist-based false recognition
Neuron
(2012) Inhibition of return
Trends in Cognitive Sciences
(2000)- et al.
The role of multisensory memories in unisensory object discrimination
Cognitive Brain Research
(2005) Spatial attention, feature-based attention, and saccades: Three sides of one coin?
Biological Psychiatry
(2011)
Cognitive failures, metacognitive beliefs and aging
Personality and Individual Differences
Rapid discrimination of visual and multisensory memories revealed by electrical neuroimaging
NeuroImage
Neurons and behavior: The same rules of multisensory integration apply
Brain Research
The multifaceted interplay between attention and multisensory integration
Trends in Cognitive Science
Individual differences in selective attention: The relation of priming and interference to cognitive failure
Personality and Individual Differences
Different types of feedback change decision criterion and sensitivity differently in perceptual learning
Journal of Vision
Spatial selection of features within perceived and remembered objects
Frontiers in Human Neuroscience
Dynamic shifts of limited working memory resources in human vision
Science
Shape-coding in IT cells generalizes over contrast and mirror reversal, but not figure-ground reversal
Nature Neuroscience
Competition between endogenous and exogenous orienting of visual attention
Journal of Experimental Psychology: General
The ventriloquist effect does not depend on the direction of deliberate visual attention
Perception & Psychophysics
Metacognition of visual short-term memory: Dissociation between objective and subjective components of VSWM
Frontiers in Psychology
The Cognitive Failures Questionnaire (CFQ) and its correlates
British Journal of Clinical Psychology
Cross-modality correspondence between pitch and spatial location modulates attentional orienting
Perception
A taxonomy of external and internal attention
Annual Review of Psychology
Inhibition of return to successively stimulated locations in a sequential visual search paradigm
Journal of Experimental Psychology: Human Perception and Performance
Enhancement of selective listening by illusory mislocation of speech sounds due to lip-reading
Nature
BOLD repetition decreases in object-responsive ventral visual areas depend on spatial attention
Journal of Neurophysiology
High perceptual load makes everybody equal: Eliminating individual differences in distractibility with load
Psychological Science
Inhibitory mechanisms of attentional networks: Spatial and semantic inhibitory processing
Journal of Experimental Psychology: Human Perception and Performance
Directing spatial attention to locations within remembered and imagined mental representations
Frontiers in Human Neuroscience
Visual attention model involving feature-based inhibition of return
Artificial Life and Robotics
Foraging for thought: An inhibition-of-return-like effect resulting from directing attention within working memory
Psychological Science
Cited by (5)
Influence of virtual color on taste: Multisensory integration between virtual and real worlds
2019, Computers in Human BehaviorA biphasic effect of cross-modal priming on visual shape recognition
2018, Acta PsychologicaCitation Excerpt :As regards acoustic stimuli (Fig. 1B), we used Audacity® 2.0 to generate two 200-ms sounds: tzk, a spiky sound with abrupt and frequent changes of intensity, and upo, a soft sound with smooth and infrequent changes of intensity. As in previous studies (Makovac et al., 2014; Makovac & Gerbino, 2010), the tzk sound was taken as congruent with the spiky shape category and incongruent with the curvy shape category, while the upo sound was taken as congruent with the curvy shape category and incongruent with the spiky shape category. Individual values of the da index, used to measure recognition sensitivity in our sequential shape matching task followed by confidence rating, were calculated according to the unequal-variance model of Signal Detection Theory (SDT; Macmillan & Creelman, 2004).
Augmented reality flavor: cross-modal mapping across gustation, olfaction, and vision
2021, Multimedia Tools and ApplicationsThe impact of cross-modal correspondences on working memory performance
2017, Journal of Experimental Psychology: Human Perception and Performance
- ☆
Authors thank David Pearson for comments on an earlier version of the manuscript. Supported by Miur-Cofin 2008 (PRIN200879EB93) and EcoAutobus-Industria (MS01_00006) 2015 grants.