ReviewMultisensory contributions to the perception of vibrotactile events
Introduction
The feeling of skin dryness or moistness that arises when we rub our hands against each other is subjectively referred to the friction forces at the epidermis. Yet, it has been demonstrated that acoustic information also participates in this bodily sensation, because altering the sound arising from the hand rubbing action changes our sensation dryness/moistness at the skin. Above and beyond the mere curiosity of this phenomenon, dubbed the parchment-skin illusion [53], this illustration touches on the very fundamental issue of how the interplay between different sensory systems contributes to our perception of the world and of ourselves. There is growing consensus around the idea that a satisfactory account of multisensory processes will be integral to any comprehensive theory of perception [13], [22], [80]. Indeed, there are multiple examples of how interactions between the senses can have a dramatic impact on sensory and perceptual processes, both measured behaviourally as well as in terms of physiological correlates of neural activity [11], [101], [107]. Although the amount of research effort devoted to understand multisensory interactions has been remarkable over the last decade, the stress has been primarily placed on a few particular modality pairings (i.e., audio-visual being the most prominent one, possibly followed by visuo-tactile) and perceptual domains (i.e., spatial perception, speech perception, object recognition, and more recently, temporal perception). Here we focus on the specific case of auditory–tactile interactions in the perception of vibrotactile events. We think that this case is of particular interest in its own right because of the close relationship that exists between the senses of hearing and touch. In fact, it is surprising that despite the relatively good knowledge that currently exists regarding the physiology and psychophysics of vibrotactile processing at a unisensory level and the close relationship between the senses of hearing and touch (see next section), the multisensory aspects of vibrotactile perception have received little attention in the literature.
In our view, understanding the neuronal and cortical mechanisms underlying multisensory perception does not only require empirical observations of behavioural or neural responses, but also theoretical and computational analyses of the processes putatively involved. Computational models explicitly link neurophysiological and behavioural experimental observations by the construction and simulation of microscopic models based on local networks with large numbers of neurons and synapses that lead to the desired global behaviour of the whole system. Biophysically realistic microscopic models are expressed by a dynamical system due to the fact that processing does not operate in a completely feed-forward fashion, since recurrent feedback is also present and there are good grounds for supposing that these feedback connections have a functional role in most aspects of brain processing [28]. To understand information processing, we need to understand the dynamic contributions from feed-forward and feed-back processes over time, and this involves more than a one-sweep feed-forward computation [43], [67], [86]. Many recent examples in the neuroscience literature (e.g., working memory, attention, decision-making) advocate for a dynamical system approach [8], [18], [86], [113] that allows us to discover the neuronal computation underlying specific brain function by solving the neurodynamical inverse problem of finding the connectivity structure from which the measured neuronal correlates emerges.
In the present article, we will briefly review some relevant aspects about the perception of vibrotactile patterns in the senses of hearing and touch, with a special stress on studies addressing audio-tactile interactions. Then we focus on one particular methodological approach based on vibrotactile discrimination task which we have recently applied to address somatosensory perception using a combination of psychophysical and computational analysis tools. We argue that this type of methodological approach can be adopted to help address several important aspects of audio-tactile interactions and multisensory perception in general. Finally, we will point out several research questions which we think are relevant, and potentially addressable within the proposed context.
Section snippets
Some similarities and differences between hearing and feeling
The case of auditory–tactile interactions in the processing of vibratory stimulation is one of the few types of multisensory interaction in which the two sensory modalities involved are sensitive to the very same kind of physical property (mechanical pressure in the form of oscillations). The tight correlation between the information content (oscillatory patterns) being conveyed in the two senses can potentially support interactions of an integrative nature at a variety of levels along the
Interactions between audition and touch
Given the similarities between hearing and touch in terms of the type of physical energy they are sensitive to, their functional properties and, their possibly related phylogenetic origins, one would expect to find a close interplay between these two senses during perception. In fact, the opportunity for cross-talk between hearing and touch is quite prevalent in everyday behaviours, given that most activities that produce tactile sensations also produce sound ([34], [35], [45]; for a discussion
Vibrotactile discrimination and the somatosensory system
Despite the compelling nature of the phenomenological demonstrations discussed at the beginning of previous section, and that these interactions are already being used in applied contexts, the fact is that the principles and mechanisms underlying auditory influences on tactile perception remain largely unknown. For instance, it is illustrative that in the original report of the parchment skin illusion [53], enhancing high frequencies of sounds inflated the ratings of roughness whereas dampening
Some relevant issues for future research
In the light of the accumulating knowledge about the unisensory processing of hearing and touch (specially the perception of vibrotactile events), there are a number of questions regarding audio-tactile interactions that would seem important and feasible to address within the context of the psychophysical, physiological and computational framework discussed above.
Concluding remarks
In this article we have attempted to persuade the reader about the relevance of studying auditory contributions to vibrotactile perception as a mean to address several relevant aspects of multisensory integration. First, we have pointed out the close relationship between the type of encoding of vibrotactile events at the skin and the ear. In the first place, the sensitive ranges for vibrations in both senses overlap, and in consequence, there is opportunity for interactions of an integrative
Acknowledgements
This research was supported by the Ministerio de Educación y Ciencia (Spain) grants SEJ2007-64103/PSIC, BFU2007-61710/BFI and CDS2007-00012 Consolider-Ingenio programme. We thank Ranulfo Romo and Massimiliano Zampini for comments on an earlier version of this manuscript. Correspondence can be addressed to SS-F (email: [email protected]).
References (123)
- et al.
Evidence of vibrotactile input to human auditory cortex
Neuroimage
(2006) - et al.
Multisensory integration, perception and ecological validity
Trends Cogn Sci
(2003) - et al.
Multisensory interplay reveals crossmodal influences on ‘sensory-specific’ brain regions, neural responses, and judgments
Neuron
(2008) - et al.
Multisensory perception: beyond modularity and convergence
Curr Biol
(2000) - et al.
Attention and the crossmodal construction of space
Trends Cogn Sci
(1998) - et al.
Merging the senses into a robust percept
Trends Cogn Sci
(2004) - et al.
Multisensory auditory-somatosensory interactions in early cortical processing revealed by high-density electrical mapping
Brain Res Cogn Brain Res
(2000) Competitive learning: from interactive activation to adaptive resonance
Cogn Sci
(1987)- et al.
Temporal evolution of a decision-making process in medial premotor cortex
Neuron
(2002) - et al.
Parchment-skin illusion: sound-biased touch
Curr Biol
(1998)
Integration of touch and sound in auditory cortex
Neuron
Neuronal oscillations and multisensory interaction in primary auditory cortex
Neuron
Feedforward, horizontal, and feedback processing in the visual cortex
Curr Opin Neurobiol
Feeling vibrations: enhanced tactile sensitivity in congenitally deaf humans
Neurosci Lett
Vibration-induced auditory-cortex activation in a congenitally deaf adult
Curr Biol
Magnetoencephalographic correlates of audiotactile interaction
NeuroImage
Audiotactile temporal order judgments in sighted and blind individuals
Neuropsychologia
The metamodal organization of the brain
Prog Brain Res
Multisensory convergence in calcarine visual areas in macaque monkey
Int J Psychophysiol
Neuronal correlates of a perceptual decision in ventral premotor cortex
Neuron
Correlated neuronal discharges that increase coding efficiency during perceptual discrimination
Neuron
Perceptual and decisional contributions to audiovisual interactions in the perception of apparent motion: a signal detection study
Cognition
The timing and laminar profile of converging inputs to multisensory areas of the macaque neocortex
Brain Res Cogn Brain Res
Touch activates human auditory cortex
Neuroimage
A unified model for perceptual learning
Trends Cogn Sci
Vibrotactile activation of the auditory cortices in deaf versus hearing adults
Neuroreport
Differential neural coding of acoustic flutter within primate auditory cortex
Nat Neurosci
Automatic visual bias of perceived auditory location
Psychon Bull Rev
Starting from the ventriloquist: the perception of multimodal events
Signal reliability modulates auditory-tactile integration for event counting
Neuroreport
Feeling what you hear: auditory signals can modulate tactile tap perception
Exp Brain Res
Nonauditory events of a behavioral procedure activate auditory cortex of highly trained monkeys
J Neurosci
Effects of neuromodulation in a cortical network model of object working memory dominated by recurrent inhibition
J Comput Neurosci
Tactile “capture” of audition
Percept Psychophys
The ‘ventriloquist effect’: visual dominance or response bias?
Perc Psychophys
A critique of pure vision
Sensation and perception
Luminance discrimination of brief flashes under various conditions of adaptation
J Physiol
Neuronal correlates of subjective sensory experience
Nat Neurosci
Object-based visual neglect: a computational hypothesis
Eur J Neurosci
Decision-making and Weber’s law: a neurophysiological model
Eur J Neurosci
Weber’s law in decision making: integrating behavioral data in humans with a neurophysiological model
J Neurosci
Anatomical evidence of multimodal integration in primate striate cortex
J Neurosci
Auditory peripersonal space in humans
J Cogn Neurosci
Elements of psychophysics
Distributed hierarchical processing in the primate cerebral cortex
Cereb Cortex
The case for feedforward multisensory convergence during early cortical processing
Neuroreport
Auditory-somatosensory multisensory processing in auditory association cortex: an fMRI study
J Neurophysiol
Enhancement of visual perception by crossmodal visuo-auditory interaction
Exp Brain Res
Cited by (59)
Unisensory and Multisensory Responses in Fetal Alcohol Spectrum Disorders (FASD): Effects of Spatial Congruence
2020, NeuroscienceCitation Excerpt :Also, inter-stimulus interval (ISI) has been shown to be a large moderator of the amplitude of later somatosensory components that are dominated by SII activity in both adults (Hari and Forss, 1999) and children (Uppal et al., 2016). The ISI of the current study of 1 s differed slightly from Lutkenhoner and colleagues’ ISI of 1.3 s. Differences in results have also been attributed to differences in task design across prior multisensory auditory/somatosensory studies (Soto-Faraco and Deco, 2009; Wu et al., 2015). The current design with a focus on viewing a silent movie and short ISI will inhibit SII activation based on previous studies (Mima et al., 1998; Hari and Forss, 1999).
Audio-tactile enhancement of cortical speech-envelope tracking
2019, NeuroImageDevelopmental changes in the perception of audiotactile simultaneity
2019, Journal of Experimental Child PsychologySelective Attention Gates the Interactive Crossmodal Coupling between Perceptual Systems
2018, Current BiologyCitation Excerpt :Co-activation of these sensory cortical systems may reflect network recruitment supporting representations encoded by both touch and audition [8, 15]. This distributed activity may underlie the interactions observed between touch and audition in the temporal frequency domain [10, 16, 17]. Connectivity within frequency-processing networks conceivably fluctuates systematically with attention state to enable flexible behavior.
Background matters: Minor vibratory stimulation during motor skill acquisition selectively reduces off-line memory consolidation
2017, Neurobiology of Learning and MemoryCitation Excerpt :Whole body vibration is even suggested as an adjunct in enhancing motor training (Cardinale & Bosco, 2003) and rehabilitation (Madou & Cronin, 2008) (but see (Lau et al., 2011)), presumably as an alerting intervention for improving vigilance (Dolny & Reyes, 2008). Vibratory stimulation is inherently multi-modal and includes both auditory and vibrotactile inputs, tuned to the same kind of physical property - mechanical pressure in the form of oscillations: the very same vibratory stimulus is experienced simultaneously by the peripheral receptor organs of both sensory modalities in the cochlea and the skin (Soto-Faraco & Deco, 2009). In the audio-vibrotactile domain there are demonstrations of tactile influences on the perception of sound (Bresciani & Ernst, 2007; Caclin, Soto-Faraco, Kingstone, & Spence, 2002; Murray et al., 2005), as well as of auditory influences in the perception of touch (Bresciani et al., 2005; Soto-Faraco, Spence, & Kingstone, 2004).
An apparatus for sound, vibration and friction measurements of soft materials in aqueous environments
2016, BiotribologyCitation Excerpt :In texture perception, there is no solid consensus on whether vision and touch are an integrated system [23,24]. However, there is general agreement that auditory inputs are important for judgment of texture [25,26,27,28]. Since mechanoreceptors under the skin surface are very sensitive to vibratory stimuli and contribute to tactile perception [29,30,31,32,33,34,35,36,37,38,39,40,41,42], vibration induced by sliding friction is clearly a key to understanding tactile perception [43,44,45].
- 1
Hospital de Sant Joan de Déu (Edifici Docent), C/ Santa Rosa, 39-57, planta 4ª, 08950 Esplugues de Llobregat – Barcelona, Spain. Tel.: +34 93 600 97 69; fax: +34 93 600 97 68.