Skip to main content
SpringerLink
Log in

Accumulation and decay of visual capture and the ventriloquism aftereffect caused by brief audio-visual disparities

  • Research Article
  • Published:
Experimental Brain Research Aims and scope Submit manuscript

Abstract

Visual capture and the ventriloquism aftereffect resolve spatial disparities of incongruent auditory visual (AV) objects by shifting auditory spatial perception to align with vision. Here, we demonstrated the distinct temporal characteristics of visual capture and the ventriloquism aftereffect in response to brief AV disparities. In a set of experiments, subjects localized either the auditory component of AV targets (A within AV) or a second sound presented at varying delays (1–20 s) after AV exposure (A2 after AV). AV targets were trains of brief presentations (1 or 20), covering a ±30° azimuthal range, and with ±8° (R or L) disparity. We found that the magnitude of visual capture generally reached its peak within a single AV pair and did not dissipate with time, while the ventriloquism aftereffect accumulated with repetitions of AV pairs and dissipated with time. Additionally, the magnitude of the auditory shift induced by each phenomenon was uncorrelated across listeners and visual capture was unaffected by subsequent auditory targets, indicating that visual capture and the ventriloquism aftereffect are separate mechanisms with distinct effects on auditory spatial perception. Our results indicate that visual capture is a ‘sample-and-hold’ process that binds related objects and stores the combined percept in memory, whereas the ventriloquism aftereffect is a ‘leaky integrator’ process that accumulates with experience and decays with time to compensate for cross-modal disparities.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  • Angelaki DE, Gu Y, DeAngelis GC (2009) Multisensory integration. Curr Opin Neurobiol 19(4):452–458

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Battaglia PW, Jacobs RA, Aslin RN (2003) Bayesian integration of visual and auditory signals for spatial localization. J Opt Soc Am A: 20(7):1391–1397

    Article  Google Scholar 

  • Bertelson P, Radeau M (1981) Cross-modal bias and perceptual fusion with auditory-visual spatial discordance. Percept Psychophys 29(6):578–584

    Article  CAS  PubMed  Google Scholar 

  • Bertelson P, Frissen I, Vroomen J, de Gelder B (2006) The aftereffects of ventriloquism: patterns of spatial generalization. Percept Psychophys 68(3):428–436

    Article  PubMed  Google Scholar 

  • Bonath B, Noesselt T, Martinez A, Mishra J, Schwiecker K, Heinze HJ, Hillyard SA (2007) Neural basis of the ventriloquist illusion. Curr Biol 17:1697–1703

    Article  CAS  PubMed  Google Scholar 

  • Bonath B, Noesselt T, Krauel K, Tyll S, Tempelmann C, Hillyard SA (2014) Audio-visual synchrony modulates the ventriloquist illusion and its neural/spatial representation in the auditory cortex. NeuroImage 98:425–434

    Article  PubMed  Google Scholar 

  • Bruns P, Röder B (2015) Sensory recalibration integrates information from the immediate and the cumulative past. Sci Rep 5:12739. doi:10.1038/srep12739

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Chen L, Vroomen J (2013) Intersensory binding across space and time: a tutorial review. Atten Percept Psychophys 75(5):790–811

    Article  PubMed  Google Scholar 

  • Cui Q, Razavi B (2010) Perception of auditory, visual, and egocentric spatial alignment adapts differently to changes in eye position. J Neurophysiol 103(2):1020–1035

    Article  PubMed  Google Scholar 

  • Cui QN, Bachus L, Knoth E, O’Neill WE, Paige GD (2008) Eye position and cross-sensory learning both contribute to prism adaptation of auditory space. Prog Brain Res 171(08):265–270

    Article  PubMed  Google Scholar 

  • Dobreva MS, O’Neill WE, Paige GD (2012) Influence of age, spatial memory, and ocular fixation on localization of auditory, visual, and bimodal targets by human subjects. Exp Brain Res 223(4):441–455

    Article  PubMed  Google Scholar 

  • Eramudugolla R, Kamke MR, Soto-Faraco S, Mattingley JB (2011) Perceptual load influences auditory space perception in the ventriloquist aftereffect. Cognition 118(1):62–74

    Article  PubMed  Google Scholar 

  • Frissen I, Vroomen J, de Gelder B, Bertelson P (2003) The aftereffects of ventriloquism: are they sound-frequency specific? Acta Psychol 113(3):315–327

    Article  Google Scholar 

  • Frissen I, Vroomen J, de Gelder B, Bertelson P (2005) The aftereffects of ventriloquism: generalization across sound-frequencies. Acta Psychol 118(1–2):93–100

    Article  Google Scholar 

  • Frissen I, Vroomen J, de Gelder B (2012) The aftereffects of ventriloquism: the time course of the visual recalibration of auditory localization. See Perceiving 25(1):1–14

    Article  Google Scholar 

  • Gruters KG, Groh JM (2012) Sounds and beyond: multisensory and other non-auditory signals in the inferior colliculus. Front Neural Circuits 6:96

    Article  PubMed  PubMed Central  Google Scholar 

  • Hairston WD, Wallace MT, Vaughan JW, Stein BE, Norris JL, Schirillo JA (2003) Visual localization ability influences cross-modal bias. J Cogn Neurosci 15(1):20–29

    Article  CAS  PubMed  Google Scholar 

  • Howard I, Templeton W (1966) Human spatial orientation. Wiley, New York

    Google Scholar 

  • Jack CE, Thurlow WR (1973) Effects of degree of visual association and angle of displacement on the “ventriloquism” effect. Percept Motor Skills 37(3):967–979

    Article  CAS  PubMed  Google Scholar 

  • Kopco N, Lin IF, Shinn-Cunningham BG, Groh JM (2009) Reference frame of the ventriloquism aftereffect. J Neurosci 29(44):13809–13814

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Körding KP, Beierholm U, Ma WJ, Quartz S, Tenenbaum JB, Shams L (2007) Causal inference in multisensory perception. PLoS ONE 2(9):e943

    Article  PubMed  PubMed Central  Google Scholar 

  • Lewald J (2002) Rapid adaptation to auditory-visual spatial disparity. Learn Memory 9(5):268–278

    Article  Google Scholar 

  • Ma WJ, Beck JM, Latham PE, Pouget A (2006) Bayesian inference with probabilistic population codes. Nat Neurosci 9(11):1432–1438

    Article  CAS  PubMed  Google Scholar 

  • Magosso E, Cuppini C, Ursino M (2012) A neural network model of ventriloquism effect and aftereffect. PLoS ONE 7(8):e42503

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Mendonça C, Escher A, Van de Par S, Colonius H (2015) Predicting auditory space calibration from recent multisensory experience. Exp Brain Res 233(7):1983–1991

    Article  PubMed  PubMed Central  Google Scholar 

  • Odegaard B, Shams L (2016) The brain’s tendency to bind audiovisual signals is stable but not general. Psychol Sci 27(4):583–591

    Article  PubMed  Google Scholar 

  • Odegaard B, Wozny DR, Shams L (2015) Biases in visual, auditory, and audiovisual perception of space. PLoS Comput Biol 11(12):1–23

    Article  Google Scholar 

  • Odegaard B, Wozny DR, Shams L (2016) The effects of selective and divided attention on sensory precision and integration. Neurosci Lett 614:24–28

    Article  CAS  PubMed  Google Scholar 

  • Passamonti C, Frissen I, Làdavas E (2009) Visual recalibration of auditory spatial perception: two separate neural circuits for perceptual learning. Eur J Neurosci 30(6):1141–1150

    Article  PubMed  Google Scholar 

  • Radeau M, Bertelson P (1974) The after-effects of ventriloquism. Q J Exp Psychol 26(1):63–71

    Article  CAS  PubMed  Google Scholar 

  • Radeau M, Bertelson P (1978) Cognitive factors and adaptation to auditory-visual discordance. Percept Psychophys 23(4):341–343

    Article  CAS  PubMed  Google Scholar 

  • Razavi B, O’Neill WE, Paige GD (2007) Auditory spatial perception dynamically realigns with changing eye position. J Neurosci 27(38):10249–10258

    Article  CAS  PubMed  Google Scholar 

  • Recanzone GH (1998) Rapidly induced auditory plasticity: the ventriloquism aftereffect. Proc Natl Acad Sci USA 95(3):869–875

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Rohe T, Noppeney U (2015) Cortical hierarchies perform Bayesian causal inference in multisensory perception. PLoS Biol 13(2):1–18

    Article  Google Scholar 

  • Sato Y, Toyoizumi T, Aihara K (2007) Bayesian inference explains perception of unity and ventriloquism aftereffect: identification of common sources of audiovisual stimuli. Neural Comput 19(12):3335–3355

    Article  PubMed  Google Scholar 

  • Shams L, Beierholm UR (2010) Causal inference in perception. Trends Cogn Sci 14(9):425–432

    Article  PubMed  Google Scholar 

  • Thurlow WR, Jack CE (1973) Certain determinants of the “ventriloquism effect”. Percept Motor Skills 36(3):1171–1184

    Article  CAS  PubMed  Google Scholar 

  • Van Wanrooij MM, Bremen P, Van Opstal JA (2010) Acquired prior knowledge modulates audiovisual integration. Eur J Neurosci 31(10):1763–1771

    Article  PubMed  Google Scholar 

  • Wallace MT, Roberson GE, Hairston WD, Stein BE, Vaughan JW, Schirillo JA (2004) Unifying multisensory signals across time and space. Exp Brain Res 158(2):252–258

    Article  CAS  PubMed  Google Scholar 

  • Wozny DR, Shams L (2011a) Recalibration of auditory space following milliseconds of cross-modal discrepancy. J Neurosci 31(12):4607–4612

    Article  CAS  PubMed  PubMed Central  Google Scholar 

  • Wozny DR, Shams L (2011b) Computational characterization of visually induced auditory spatial adaptation. Front Integr Neurosci 5:75

    Article  PubMed  PubMed Central  Google Scholar 

  • Wozny DR, Beierholm UR, Shams L (2010) Probability matching as a computational strategy used in perception. PLoS Comput Biol 6(8):e1000871

    Article  PubMed  PubMed Central  Google Scholar 

  • Zwiers MP, Van Opstal AJ, Paige GD (2003) Plasticity in human sound localization induced by compressed spatial vision. Nat Neurosci 6(2):175–181

    Article  CAS  PubMed  Google Scholar 

Download references

Acknowledgements

We thank Martin Gira and Robert Schor for their technical assistance. Research was supported by NIDCD Grants P30 DC-05409 and T32 DC-009974-04 (Center for Navigation and Communication Sciences), NEI Grants P30-EY01319 and T32 EY-007125-25 (Center for Visual Science), and an endowment by the Schmitt Foundation.

Author information

Authors and Affiliations

  1. Department of Biomedical Engineering, University of Rochester, Rochester, NY, USA

    Adam K. Bosen

  2. Department of Neuroscience, University of Rochester, Rochester, NY, USA

    Justin T. Fleming, Paul D. Allen, William E. O‘Neill & Gary D. Paige

Authors
  1. Adam K. Bosen
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Justin T. Fleming
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul D. Allen
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. William E. O‘Neill
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Gary D. Paige
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Adam K. Bosen.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Bosen, A.K., Fleming, J.T., Allen, P.D. et al. Accumulation and decay of visual capture and the ventriloquism aftereffect caused by brief audio-visual disparities. Exp Brain Res 235, 585–595 (2017). https://doi.org/10.1007/s00221-016-4820-4

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00221-016-4820-4

Keywords

Search

Navigation