The emergence of multisensory systems through perceptual narrowing

doi:10.1016/j.tics.2009.08.004

Trends in Cognitive Sciences

Volume 13, Issue 11, November 2009, Pages 470-478

https://doi.org/10.1016/j.tics.2009.08.004 Get rights and content

According to conventional wisdom, multisensory development is a progressive process that results in the growth and proliferation of perceptual skills. We review new findings indicating that a regressive process – perceptual narrowing – also contributes in critical ways to perceptual development. These new data reveal that young infants are able to integrate non-native faces and vocalizations, that this broad multisensory perceptual tuning is present at birth, and that this tuning narrows by the end of the first year of life, leaving infants with the ability to integrate only socio-ecologically-relevant multisensory signals. This narrowing process forces us to reconsider the traditional progressive theories of multisensory development and opens up several new evolutionary questions as well.

Section snippets

The world is multisensory

The objects and events that make up our everyday experience provide us with a constant flow of sensory signals in multiple modalities. Although such inputs can potentially create confusion, our ability to integrate multisensory information enables us to have coherent and meaningful perceptual experiences. Talking faces, for example, are typically specified by various spatiotemporally congruent and modality-specific attributes as well as a host of invariant amodal attributes. Modality-specific

The developmental problem and the progressive framework

Because the world is multisensory, a developing infant's task is to discover the multisensory coherence of the objects and events that constitute the infant's normal ecology. This task is difficult because the infant has an immature nervous system and is perceptually inexperienced. Nonetheless, infants gradually overcome these limitations and become capable of detecting multisensory coherence 8, 9. They are able to overcome these limitations for two reasons. First, multisensory coherence is

The progressive framework is not the whole story: perceptual narrowing

Although the evidence consistent with the progressive theoretical framework is unquestionable, other evidence suggests that various human perceptual functions undergo developmental narrowing in early life and that this is crucial for the eventual development of species-specific patterns of perceptual expertise (see Box 1 for historical note). This evidence comes from studies of speech, face and music perception, and shows that, initially, perceptual tuning is so broad that it allows young

Multisensory perceptual narrowing

If multisensory perception is the default mode of perception 40, 41 then perceptual narrowing might be a pan-sensory process. If it is, then the progressive developmental theoretical framework does not adequately account for the development of multisensory perception. The first hint that this might be so came from a study that pre-dates the various findings of narrowing in the unisensory domain. This study showed that whereas 3-week-old infants made spontaneous intensity-based audiovisual (A-V)

Is perceptual narrowing unique to humans?

The evidence for multisensory perceptual narrowing raises questions about the evolution of this developmental mechanism. Naturally, the only way to get at this issue is through comparative studies with non-human primates (hereafter, primates). Given that both humans and other extant primates use both facial and vocal expressions as communication signals, it is perhaps not surprising that many primates recognize the correspondence between the visual and auditory components of vocal signals.

Conclusions

Experience can have complementary effects on the development of perceptual functions. On the one hand, it can induce and facilitate the emergence of a particular perceptual function and it can maintain that function through continued exposure to specific sensory inputs. On the other, it also can have the seemingly opposite but, in reality, complementary effect. That is, as experience with native sensory input accumulates, the scope of initially broad perceptual abilities is narrowed to best

Acknowledgements

This work was supported by National Science Foundation grants to DJL (BCS-0751888) and AAG (BCS-0547760 CAREER award). We thank Robert Seyfarth for pointing out examples of developmental narrowing in monkeys.

References (81)

L. Krubitzer
The magnificent compromise: Cortical field evolution in mammals
Neuron
(2007)
D.J. Lewkowicz
Heterogeneity and heterochrony in the development of intersensory perception
Cogn. Brain Res.
(2002)
G.E. Walton et al.
Amodal representations of speech in infants
Infant Behav. & Deve.
(1993)
M.L. Patterson et al.
Infants’ ability to match dynamic phonetic and gender information in the face and voice
J. Exp. Child Psychol.
(2002)
J.F. Werker et al.
Cross-language speech perception: Evidence for perceptual reorganization during the first year of life
Infant Behav. & Dev.
(1984)
C.T. Best
Divergent developmental patterns for infants’ perception of two nonnative consonant contrasts
Infant Behav. & Dev.
(1995)
D.J. Kelly
Development of the other-race effect during infancy: evidence toward universality?
J. Exp. Child Psychol.
(2009)
D.J. Lewkowicz
The decline of cross-species intersensory perception in human infants: Underlying mechanisms and its developmental persistence
Brain Res.
(2008)
K. Jordan
Monkeys match the number of voices they hear to the number of faces they see
Curr. Biol.
(2005)
A. Ghazanfar
Vocal-tract resonances as indexical cues in rhesus monkeys
Curr. Biol.
(2007)

R.M. Seyfarth et al.

Vocal development in vervet monkeys

Anim. Behav.

(1986)

V. Dufour

Face processing limitation to own species in primates: A comparative study in brown capuchins, Tonkean macaques and humans

Behav. Processes

(2006)

C. Chandrasekaran

The natural statistics of audiovisual speech

PLoS Comput. Biol.

(2009)

K.G. Munhall et al.

Spatial and Temporal Constraints on Audiovisual Speech Perception

L.E. Bahrick

Intersensory Redundancy Guides the Development of Selective Attention, Perception, and Cognition in Infancy

Curr. Directions Psychol. Sci.

(2004)

D.J. Lewkowicz et al.

The value of multimodal redundancy in the development of intersensory perception

M.A. Kingsbury et al.

The cortex in multidimensional space: where do cortical areas come from? Commentary

Dev. Sci.

(2001)

S. Oyama

The ontogeny of information

(2000)

R. Lickliter et al.

The development of infant intersensory perception: advantages of a comparative convergent-operations approach

Psychol. Bull.

(2000)

E.J. Gibson

Principles of perceptual learning and development

(1969)

G. Turkewitz et al.

Limitations on input as a basis for neural organization and perceptual development: A preliminary theoretical statement

Dev. Psychobiol.

(1982)

H.G. Birch et al.

Visual Differentiation, Intersensory Integration, and Voluntary Motor Control

Monogr. Soc. Res. Child Dev.

(1967)

J. Piaget

The origins of intelligence in children

(1952)

E.J. Gibson

Perceptual development from the ecological approach

P.K. Kuhl et al.

The bimodal perception of speech in infancy

Science

(1982)

M.L. Patterson et al.

Two-month-old infants match phonetic information in lips and voice

Dev. Sci.

(2003)

R. Kahana-Kalman et al.

The role of person familiarity in young infants’ perception of emotional expressions

Child Dev.

(2001)

A.S. Walker-Andrews

Intermodal perception of expressive behaviors: Relation of eye and voice?

Dev. Psy.

(1986)

D. Maurer

Sleeper effects

Dev. Sci.

(2007)

S.H. Horng et al.

Visual activity and cortical rewiring: activity-dependent plasticity of cortical networks

Prog. Brain Res.

(2006)

M.T. Wallace et al.

Early experience determines how the senses will interact

J. Neurophysiol.

(2007)

A.J. King

Visual influences on auditory spatial learning

Philos. Trans. R. Soc. Lond. B Biol. Sci

(2008)

P.K. Kuhl

Linguistic experience alters phonetic perception in infants by 6 months of age

Science

(1992)

M. Cheour

Development of language-specific phoneme representations in the infant brain

Nature Neurosci.

(1998)

P.K. Kuhl

Foreign-language experience in infancy: effects of short-term exposure and social interaction on phonetic learning

Proc. Natl. Acad. Sci. U.S.A.

(2003)

O. Pascalis

Is face processing species-specific during the first year of life?

Science

(2002)

O. Pascalis

Plasticity of face processing in infancy

Proc. Natl. Acad. Sci. U.S.A.

(2005)

P. Chiroro et al.

An investigation of the contact hypothesis of the own-race bias in face recognition

Q. J. Exp. Psychol. A: Human Experimental Psychology

(1995)

D.J. Kelly

The Other-Race Effect Develops During Infancy: Evidence of Perceptual Narrowing

Psychol. Sci

(2007)

S. Sangrigoli et al.

Recognition of own-race and other-race faces by three-month-old infants

J. Child Psychol. Psychiatr.

(2004)

Cited by (221)

A multisensory Interaction Framework for Human-Cyber–Physical System based on Graph Convolutional Networks
2024, Advanced Engineering Informatics
Human-Cyber-Physical Systems (HCPS), as an emerging paradigm centered around humans, provide a promising direction for the advancement of various domains, such as intelligent manufacturing and aerospace. In contrast to Cyber-Physical Systems (CPS), the development of HCPS emphasizes the expansion of human capabilities. Humans no longer solely function as operators or agents working in collaboration with computers and machines but extend their roles to include system design and innovation management. This paper proposes a Multisensory Interaction Framework for HCPS (MS-HCPS) that leverages human senses to facilitate system creation and management. Additionally, the introduced Multisensory Graph Convolutional Network (MS-GCN) model calculates recommendation values for multiple senses, elucidating their relevance to system development. Furthermore, the effectiveness of the proposed framework and model is validated through three practical engineering scenarios. This study explores the research on multisensory interaction in HCPS from a human sensory perspective, aiming to facilitate the progress and development of HCPS across various domains.
The emergence of the multisensory brain: From the womb to the first steps
2024, iScience
The becoming of the human being is a multisensory process that starts in the womb. By integrating spontaneous neuronal activity with inputs from the external world, the developing brain learns to make sense of itself through multiple sensory experiences. Over the past ten years, advances in neuroimaging and electrophysiological techniques have allowed the exploration of the neural correlates of multisensory processing in the newborn and infant brain, thus adding an important piece of information to behavioral evidence of early sensitivity to multisensory events. Here, we review recent behavioral and neuroimaging findings to document the origins and early development of multisensory processing, particularly showing that the human brain appears naturally tuned to multisensory events at birth, which requires multisensory experience to fully mature. We conclude the review by highlighting the potential uses and benefits of multisensory interventions in promoting healthy development by discussing emerging studies in preterm infants.
The development of the relationship between auditory and visual neural sensitivity and autonomic arousal from 6 m to 12 m
2023, Developmental Cognitive Neuroscience
The differential sensitivity hypothesis argues that environmental sensitivity has the bivalent effect of predisposing individuals to both the risk-inducing and development-enhancing influences of early social environments. However, the hypothesis requires that this variation in environmental sensitivity be general across domains. In this study, we focused on neural sensitivity and autonomic arousal to test domain generality. Neural sensitivity can be assessed by correlating measures of perceptual sensitivity, as indexed by event-related potentials (ERP) in electrophysiology. The sensitivity of autonomic arousal can be tested via heart rate changes. Domain generality was tested by comparing associations in perceptual sensitivity across auditory and visual domains, and associations between sensitivity in sensory domains and heart rate. We contrasted ERP components in auditory (P3) and visual (P1, N290 and P4) detection-of-difference tasks for N = 68 infants longitudinally at 6 and 12 months of age. Domain generality should produce correlated individual differences in sensitivity across the two modalities, with higher levels of autonomic arousal associating with increased perceptual sensitivity. Having controlled for multiple comparisons, at 6 months of age, the difference in amplitude of the P3 component evoked in response to standard and deviant tones correlated with the difference in amplitude of the P1 N290 and P4 face-sensitive components evoked in response to fearful and neutral faces. However, this correlation was not found at 12 months of age. Similarly, autonomic arousal correlated with neural sensitivity at 6 months but not at 12 months. The results suggest bottom-up neural perceptual sensitivity is domain-general across auditory and visual domains and is related to autonomic arousal at 6 months but not at 12 months of age. We interpret the development of the association of these markers of ES within a neuroconstructivist framework and with respect to the concept of interactive specialisation. By 12 months of age, more experience of visual processing may have led to top-down endogenous attention mechanisms that process visual information in a way that no longer associates with automatic auditory perceptual sensitivity.
Development and experience-dependence of multisensory spatial processing
2023, Trends in Cognitive Sciences
Multisensory spatial processes are fundamental for efficient interaction with the world. They include not only the integration of spatial cues across sensory modalities, but also the adjustment or recalibration of spatial representations to changing cue reliabilities, crossmodal correspondences, and causal structures. Yet how multisensory spatial functions emerge during ontogeny is poorly understood. New results suggest that temporal synchrony and enhanced multisensory associative learning capabilities first guide causal inference and initiate early coarse multisensory integration capabilities. These multisensory percepts are crucial for the alignment of spatial maps across sensory systems, and are used to derive more stable biases for adult crossmodal recalibration. The refinement of multisensory spatial integration with increasing age is further promoted by the inclusion of higher-order knowledge.
The discrimination of expressions in facial movements by infants: A study with point-light displays
2023, Journal of Experimental Child Psychology
Perceiving facial expressions is an essential ability for infants. Although previous studies indicated that infants could perceive emotion from expressive facial movements, the developmental change of this ability remains largely unknown. To exclusively examine infants’ processing of facial movements, we used point-light displays (PLDs) to present emotionally expressive facial movements. Specifically, we used a habituation and visual paired comparison (VPC) paradigm to investigate whether 3-, 6-, and 9-month-olds could discriminate between happy and fear PLDs after being habituated with a happy PLD (happy-habituation condition) or a fear PLD (fear-habituation condition). The 3-month-olds discriminated between the happy and fear PLDs in both the happy- and fear-habituation conditions. The 6- and 9-month-olds showed discrimination only in the happy-habituation condition but not in the fear-habituation condition. These results indicated a developmental change in processing expressive facial movements. Younger infants tended to process low-level motion signals regardless of the depicted emotions, and older infants tended to process expressions, which emerged in familiar facial expressions (e.g., happy). Additional analyses of individual difference and eye movement patterns supported this conclusion. In Experiment 2, we concluded that the findings of Experiment 1 were not due to a spontaneous preference for fear PLDs. Using inverted PLDs, Experiment 3 further suggested that 3-month-olds have already perceived PLDs as face-like stimuli.
The multisensory cocktail party problem in children: Synchrony-based segregation of multiple talking faces improves in early childhood
2022, Cognition
Extraction of meaningful information from multiple talkers relies on perceptual segregation. The temporal synchrony statistics inherent in everyday audiovisual (AV) speech offer a powerful basis for perceptual segregation. We investigated the developmental emergence of synchrony-based perceptual segregation of multiple talkers in 3–7-year-old children. Children either saw four identical or four different faces articulating temporally jittered versions of the same utterance and heard the audible version of the same utterance either synchronized with one of the talkers or desynchronized with all of them. Eye tracking revealed that selective attention to the temporally synchronized talking face increased while attention to the desynchronized faces decreased with age and that attention to the talkers' mouth primarily drove responsiveness. These findings demonstrate that the temporal synchrony statistics inherent in fluent AV speech assume an increasingly greater role in perceptual segregation of the multisensory clutter created by multiple talking faces in early childhood.

View all citing articles on Scopus

View full text

Trends in Cognitive Sciences

ReviewThe emergence of multisensory systems through perceptual narrowing

Section snippets

The world is multisensory

The developmental problem and the progressive framework

The progressive framework is not the whole story: perceptual narrowing

Multisensory perceptual narrowing

Is perceptual narrowing unique to humans?

Conclusions

Acknowledgements

Neuron

Cogn. Brain Res.

Infant Behav. & Deve.

J. Exp. Child Psychol.

Infant Behav. & Dev.

Infant Behav. & Dev.

J. Exp. Child Psychol.

Brain Res.

Curr. Biol.

Curr. Biol.

Anim. Behav.

Behav. Processes

The natural statistics of audiovisual speech

PLoS Comput. Biol.

Spatial and Temporal Constraints on Audiovisual Speech Perception

Intersensory Redundancy Guides the Development of Selective Attention, Perception, and Cognition in Infancy

Curr. Directions Psychol. Sci.

The value of multimodal redundancy in the development of intersensory perception

The cortex in multidimensional space: where do cortical areas come from? Commentary

Dev. Sci.

The ontogeny of information

The development of infant intersensory perception: advantages of a comparative convergent-operations approach

Psychol. Bull.

Principles of perceptual learning and development

Limitations on input as a basis for neural organization and perceptual development: A preliminary theoretical statement

Dev. Psychobiol.

Visual Differentiation, Intersensory Integration, and Voluntary Motor Control

Monogr. Soc. Res. Child Dev.

The origins of intelligence in children

Perceptual development from the ecological approach

The bimodal perception of speech in infancy

Science

Two-month-old infants match phonetic information in lips and voice

Dev. Sci.

The role of person familiarity in young infants’ perception of emotional expressions

Child Dev.

Intermodal perception of expressive behaviors: Relation of eye and voice?

Dev. Psy.

Sleeper effects

Dev. Sci.

Visual activity and cortical rewiring: activity-dependent plasticity of cortical networks

Prog. Brain Res.

Early experience determines how the senses will interact

J. Neurophysiol.

Visual influences on auditory spatial learning

Philos. Trans. R. Soc. Lond. B Biol. Sci

Linguistic experience alters phonetic perception in infants by 6 months of age

Science

Development of language-specific phoneme representations in the infant brain

Nature Neurosci.

Foreign-language experience in infancy: effects of short-term exposure and social interaction on phonetic learning

Proc. Natl. Acad. Sci. U.S.A.

Is face processing species-specific during the first year of life?

Science

Plasticity of face processing in infancy

Proc. Natl. Acad. Sci. U.S.A.

An investigation of the contact hypothesis of the own-race bias in face recognition

Q. J. Exp. Psychol. A: Human Experimental Psychology

The Other-Race Effect Develops During Infancy: Evidence of Perceptual Narrowing

Psychol. Sci

Recognition of own-race and other-race faces by three-month-old infants

J. Child Psychol. Psychiatr.

Review
The emergence of multisensory systems through perceptual narrowing