Sensory experience during early sensitive periods shapes cross-modal temporal biases

In every moment, a multitude of information reaches our brain through the different senses. These sensory inputs need to be separated, ordered in space and time to derive a coherent representation of the environment. Yet, the perception of temporal order is seldom veridical. Reports illustrating the subjectivity of cross-modal temporal perception date back to 18^th and 19^th century astronomers; small but stable individual biases in the perceived timing of visual and auditory events caused significant differences in the measurements of stellar transit times which in turn resulted in bitter scientific disputes. These early reports inspired the pioneering work of Wilhelm Wundt and colleagues on cross-modal temporal perception (Sanford, 1888; Canales, 2010) and, hence, are seen as the origin of experimental psychology (Mollon and Perkins, 1996). Early psychologists like William James transferred the personal equation, developed by Bessel to solve the astronomers’ problem of stable cross-modal temporal biases by adding a fixed amount to an observer’s measurements, to cognitive processes. Yet, even though the relative timing of events is crucial for the perception of time and causality (Hume, 2012; Pöppel, 1988; Pöppel, 1997; Spence and Squire, 2003; Aghdaee et al., 2014), it has yet remained unsolved why humans consistently misjudge temporal order across different sensory modalities.

Determining the spatio-temporal order of events across sensory modalities poses an especially difficult challenge as information arriving through different senses travels at different speeds – in the environment and within the nervous system (Hirsh and Sherrick, 1961). Light travels faster than sound, but within the nervous system, auditory information reaches the brain faster than visual information (Fain, 2019). At a distance of approximately 10 m from the observer, the environmental and the physiological differences presumably cancel each other out (Pöppel, 1988). However, both the relative time of arrival at the sensors and neural transmission times vary with the physical properties of the stimulus (King, 2005). Curiously, the brain seems to be able to adjust the perception of temporal order for variations in signal strength (Kopinska and Harris, 2004) and physical distance (Engel and Dougherty, 1971; Alais and Carlile, 2005), yet, within their peripersonal space humans typically perceive visual events as delayed compared to sounds (Keetels and Vroomen, 2012; Zampini et al., 2003; Grabot and van Wassenhove, 2017).

Cross-modal temporal biases exhibit a paradoxical characteristic, they are remarkably stable across longer periods of time (Sanford, 1888; Grabot and van Wassenhove, 2017) while being malleable within short time periods. After humans are exposed to a series of asynchronous stimulus pairs with a constant lag between vision and audition (Fujisaki et al., 2004; Vroomen et al., 2004) (or vision and touch; Keetels and Vroomen, 2008), they perceive the temporal order of subsequent visual-auditory events as shifted in the lag-reversed direction. Such temporal recalibration effects typically last for several minutes. Moreover, on a moment-by-moment basis, the perceived relative timing of events in different modalities changes with the focus of attention. Titchener’s law of prior entry (Titchener, 1908) states that attention towards one sensory input channel is capable of speeding up perceptual processing of that sensory information resulting in a demonstrable shift of temporal order perception towards events in the attended sensory modality (Spence et al., 2001; Zampini et al., 2005; Vibell et al., 2007; Spence, 2010). The co-existence of short-term plasticity and long-term stability re-emphasizes the question that has troubled scientists for 250 years: why do cross-modal temporal biases persist despite the brain’s capability for recalibration?

A potential reason for the high stability of cross-modal temporal biases is that they are shaped during a sensitive period of development. Sensitive periods are limited periods during brain development in which the influence of sensory experience on the brain is particularly strong (Knudsen, 2004). During a sensitive period, the architecture of a neural circuit is shaped to meet an individual’s environment. The long-term result is a preference of the neural circuits for certain states of activity even if the environment dramatically changes (Röder et al., 2007; Röder et al., 2004; Sourav et al., 2019), without prohibiting short-term changes in the neural circuits’ activity patterns (Knudsen, 2004; Knudsen, 2002). The role of early sensory experience for the genesis of perceptual biases is extremely difficult to address in humans; only individuals whose early sensory environments were atypical due to natural causes open a window into the role of experience in perceptual development. We tested the hypothesis of a sensitive period for cross-modal perceptual biases by asking individuals born with dense, bilateral cataracts whose sight was restored 6 to 168 months after birth (Table 1) to temporally order spatially separated events across vision, audition and touch.

Thirteen individuals with a history of congenital bilateral, dense cataracts which had been surgically removed (CC-group) participated in two spatial temporal order judgement tasks, ten in a visual-auditory task (Expt. 1) and ten in a visual-tactile task (Expt. 2). To test for the role of vision during infancy as well as to control for effects related to having had eye surgery and persisting visual impairments, sixteen individuals, nine per experiment, who had undergone surgery for cataracts which had developed during childhood served as controls (DC-group). As both cataract-reversal groups differed in age (Table 1) and age is known to influence cross-modal temporal perception (Röder et al., 2013; Noel et al., 2016), their performance was not directly compared but each group was contrasted separately against age-matched typically sighted individuals (MCC- and MDC-groups). In every trial, two successive stimuli were presented, one in each hemifield (Figure 1A). Stimulus order and the time interval between the two stimuli (stimulus onset asynchrony; SOA) varied randomly across trials. Visual-auditory and visual-tactile stimulus pairs were randomly interleaved with unimodal stimulus pairs. Participants reported the side of the first stimulus irrespective of its modality (Zampini et al., 2003). For the CC-group, we predicted that due to their history of visual loss and persisting visual impairments participants would give auditory and tactile stimuli a higher preference than visual stimuli. Such a preference would result in a shift of the point of perceived simultaneity (PSS) –the SOA at which auditory-visual or tactile-visual pairs are perceived as simultaneous– towards SOAs with greater auditory or tactile lags, and an overall lower proportion of ‘visual first’-responses compared to their controls. Additionally, we predicted a lower visual spatio-temporal resolution resulting in a lower proportion of correct temporal order judgments for the CC-group.

Figure 1 with 3 supplements see all

Download asset Open asset

Most strikingly and contrary to our predictions, CC-individuals showed a bias toward perceiving visual stimuli as occurring earlier than auditory (Figure 1D,E; CC-group, Expt. 1, PSS, t(9) = −1.51, p=0.033, r_equiv = 0.57; ‘visual first’-response probability, χ² (1)=4.31, p=0.038, r_equiv = 0.55; see Supplementary file 1 for full statistical models) and tactile stimuli (CC-group, Expt. 2, PSS, t(9) = −3.04, p=0.009, r_equiv = 0.69; ‘visual first’-response probability, χ²(1)=12.46, p<0.001, r_equiv = 0.85), respectively. CC-individuals’ bias toward perceiving visual stimuli as earlier stood in contrast to the bias observed for their matched controls (CC- vs. MCC-group, Expt. 1, visual-auditory, PSS, t(18) = −2.34, p=0.005, r_equiv = 0.56; ‘visual first’-response probability, χ² (1)=8.48, p=0.004, r_equiv = 0.57; Expt. 2, visual-tactile, PSS, t(18) = −2.47, p=0.008, r_equiv = 0.53; ‘visual first’-response probability, χ² (1)=5.31, p=0.021, r_equiv = 0.45) who perceived auditory stimuli as occurring earlier than visual stimuli (MCC-group, Expt. 1, visual-auditory, PSS, t(9) = 2.30, p=0.009, r_equiv = 0.69; ‘visual first’-response probability, χ² (1)=4.18, p=0.041, r_equiv = 0.55) and did not show a significant bias for visual-tactile comparisons (MCC-group, Expt. 2, visual-tactile, PSS, t(9) = −0.48, p=0.321, r_equiv = 0.16; ‘visual first’-response probability, χ² (1)=0.15, p=0.701, r_equiv = 0.17). In contrast to the CC-group, DC-individuals showed the typical bias toward perceiving visual stimuli as occurring later than auditory and tactile stimuli (DC-group, Expt. 1, visual-auditory, PSS, t(8) = 1.42, p=0.017, r_equiv = 0.67; ‘visual first’-response probability, χ² (1)=7.14, p=0.008, r_equiv = 0.74; Expt. 2, visual-tactile, PSS, t(8) = 2.31, p=0.017, r_equiv = 0.67; ‘visual first’-response probability, χ² (1)=8.86, p=0.003, r_equiv = 0.79). In fact, their spatio-temporal bias toward perceiving visual stimuli as later than tactile stimuli was stronger than the bias of their matched controls (DC- vs. MDC-group, Expt. 1, visual-auditory, PSS, t(18) = 1.40, p=0.123, r_equiv = 0.29; ‘visual first’-response probability, χ² (1)=1.84, p=0.174, r_equiv = 0.23; Expt. 2, visual-tactile, PSS, t(18) = 2.17, p=0.019, r_equiv = 0.49; ‘visual first’-response probability, χ² (1)=8.08, p=0.004, r_equiv = 0.60; MDC-group, Expt. 1, visual-auditory, PSS, t(8) = 1.32, p=0.152, r_equiv = 0.36; ‘visual first’-response probability, χ² (1)=0.83, p=0.362, r_equiv = 0.13; Expt. 2, visual-tactile, PSS, t(8) = −0.36, p=0.361, r_equiv = 0.13; ‘visual first’-response probability, χ²=0.85, p=0.356, r_equiv = 0.15).

CC-individuals’ proportion of correct temporal order judgments for visual and cross-modal stimulus pairs (Figure 1F) was reduced compared to their controls’ temporal resolution (CC-group vs. MCC-group, Expt. 1, visual, χ² (1)=9.68, p=0.002, r_equiv = 0.62; Expt. 1, visual-auditory, χ² (1)=17.88, p<0.001, r_equiv = 0.78; Expt. 2, visual, χ² (1)=4.72, p=0.030, r_equiv = 0.42; Expt. 2, visual-tactile, χ² (1)=4.03, p=0.045, r_equiv = 0.39; see Supplementary file 1 for full statistical models), but no significant difference between the CC-group and their controls emerged for auditory and tactile stimulus pairs, (CC-group vs. MCC-group, Expt. 1, auditory, χ² (1)=1.42, p=0.233, r_equiv = 0.17; Expt. 2, tactile, χ² (1)=0.11, p=0.742, r_equiv = 0.15). DC-individuals’ spatio-temporal resolution was lower than the resolution of their controls independent of modality condition (DC-group vs. MDC-group, Expt. 1, all modalities, χ² (1)=4.98, p=0.026, r_equiv = 0.46; Expt. 2, all modalities, χ² (1)=11.99, p=0.001, r_equiv = 0.68).

Across participants, the size of the bias in the proportion of ‘visual first’-responses in bimodal trials correlated negatively with the proportion of correct temporal order judgements in these trials (Figure 1G; Expt. 1, visual-auditory, r = −0.39, p=0.018; Expt. 2, visual-tactile, r = −0.49, p=0.002). The distribution of reaction times in bimodal trials peaked at different SOAs across groups (Figure 1H; CC- vs. MCC-group, Expt. 1, visual-auditory, t(16) = 2.11, p=0.041, r_equiv = 0.40; Expt. 2, visual-tactile, t(18) = 1.56, p=0.067, r_equiv = 0.35) with peaks being shifted in the same direction as the PSS.

Here, we investigated whether biases in cross-modal temporal perception are shaped during a sensitive period in early development by testing sight-recovery individuals with a history of congenital loss of pattern vision. In two spatio-temporal order judgement tasks, individuals with reversed congenital cataracts disproportionally often reported visual stimuli as occurring earlier than auditory and tactile stimuli (Figure 1B–E), exhibiting a reversed cross-modal bias compared to typically sighted controls and sight-recovery individuals whose cataracts had developed during childhood. These results, for the first time, demonstrate a sensitive period for cross-modal temporal biases. Moreover, the ability to correctly determine the spatio-temporal order of separate events across different sensory modalities and within vision was reduced after a transient phase of visual loss both after birth and later in childhood (Figure 1F) suggesting a persistent impairment of visual and cross-modal temporal resolution following transient visual deprivation.

At first glance, CC-individuals’ bias to perceive visual events as earlier than auditory and tactile events seems to indicate that visual stimuli were processed faster than auditory and tactile stimuli following transient, congenital visual deprivation. However, CC-individuals’ response accuracy (Figure 1F) indicated temporal processing impairments for vision but not for audition and touch. Consistently, studies measuring event-related potentials have provided no evidence for a shorter latency of the first visual cortical response in CC-individuals (Sourav et al., 2018; Mondloch et al., 2013; McCulloch and Skarf, 1994) and behavioral studies have demonstrated no advantage in reaction times to simple visual stimuli (Putzar et al., 2012; de Heering et al., 2016). Moreover, reduced visual contrast – typical for cataract-reversal individuals – is associated with delayed responses of the visual system and lower visual temporal sensitivity (Watson, 1986; Stromeyer and Martini, 2003). In sum, there is currently no evidence indicating accelerated processing of visual information following congenital visual deprivation.

During sensitive periods, experience shapes neural circuits customizing them to an individual’s body and environment (Knudsen, 2004). CC-individuals’ reversed cross-modal bias towards perceiving visual stimuli as occurring earlier than auditory and tactile stimuli might be rooted in consistent exposure to lagging visual input during an early sensitive period. Residual light perception, which exists even in the presence of dense cataracts, is typically sluggish and retinal transduction rates have been reported to be reduced in patients with dense, untreated cataracts (Yamauchi et al., 2016). Moreover, a suppression of visual cortex activity has recently been observed in CC-individuals in the context of cross-modal stimulation (Guerreiro et al., 2016). Thus, CC-individuals likely were exposed to consistently delayed visual signals (with respect to auditory and tactile signals) before the cataracts were removed. Recalibration studies have shown that exposure to a consistent visual delay results in a bias towards perceiving visual input as earlier (Fujisaki et al., 2004; Vroomen et al., 2004), consistent with the present results in the CC-group. Thus, we suggest that the reversed bias exhibited by CC-individuals reflects adaptation to their atypical sensory environment during infancy, which results in structural differences in the neural circuits processing temporal order across modalities.

Individual cross-modal temporal biases in typically sighted humans might reflect the relative timing of sensory information across modalities during infancy, too. One-month old infants have been found to already show rudimentary cross-modal temporal biases for audition and vision (Lewkowicz, 1996). Slight differences in the relative rate of brain development across individuals, for example, in the myelination of neural circuits in different sensory areas, could give rise to the characteristic but stable (Sanford, 1888; Grabot and van Wassenhove, 2017) inter-individual differences in cross-modal temporal biases in humans. In turn, the variation of individual cross-modal temporal biases as a function of task (Freeman et al., 2013; Ipser et al., 2018; Ipser et al., 2017) might arise from differences in the relative rate of development across cortical areas. In sum, we suggest that individual cross-modal temporal biases (Zampini et al., 2003; Grabot and van Wassenhove, 2017) show a high stability because the brain had optimized cross-modal temporal perception (Roseboom et al., 2015) during a sensitive period which constitutes a setpoint for all future recalibration.

Additionally, the congenital absence of pattern vision might have increased CC-individuals’ attention towards this modality after sight restoration, resulting in a permanent prior entry effect for vision (Titchener, 1908; Spence et al., 2001; Vibell et al., 2007; Shore et al., 2001). However, a previous study found longer switch times from audition to vision but not from vision to audition in CC-individuals whose cataracts had been removed in the first months of life than in controls, which was interpreted as indicating an attentional bias towards audition over vision (de Heering et al., 2016). Moreover, an attentional bias towards vision would have been expected to result in relatively shorter visual reaction times and early cortical potentials, which, as argued at the beginning of the Discussion is incompatible with the literature (Sourav et al., 2018; Mondloch et al., 2013; McCulloch and Skarf, 1994; Putzar et al., 2012; de Heering et al., 2016). Thus, while scarce existing data points towards a reduced dominance of vision in CC-individuals, the consequences of congenital loss of pattern vision for modality-specific attention are not yet understood.

The finding, that CC-individuals but not DC-individuals showed a reversed visual-auditory and visual-tactile temporal bias strongly suggests that perceptual biases are shaped by sensory experience during the initial period of life. In contrast to CC-individuals, DC-individuals had encountered unimpaired cross-modal stimulation after birth, which we suggest had enabled them to establish a typical bias. An early sensitive period would have prevented DC-individuals –who lost their vision later– from adapting this typical bias to their changed sensory environment. As a consequence, their partially severe, persisting visual impairments might have caused an increase of the typical bias of perceiving vision as delayed. However, the bias of some DC-individuals seems too large to be accounted for by delays in visual processing alone. The very high temporal uncertainty of some DC-individuals might have resulted in an overestimation of their cross-modal biases (Figure 1G): In agreement with Bayes’ law, temporal perceptual judgments tend to rely more on prior information such as existing biases when the sensory information is uncertain (Shi and Burr, 2016; Vercillo et al., 2015). Consistent with this idea, the current data indicated a correlation between the size of the bias and the degree of sensory uncertainty. The sensitive period for cross-modal temporal perception might culminate during the first six months of life, the minimal duration of deprived vision in our CC-group. A previous study tested CC-individuals whose vision was restored within early infancy (4 months of age on average) in an audio-visual simultaneity judgment task with spatially aligned visual-auditory and visual-tactile stimulus pairs and did not report a reversed cross-modal bias (Chen et al., 2017). However, simultaneity judgments of spatially aligned cross-modal stimuli are simpler and less sensitive to biases than the spatial temporal order judgements we used (van Eijk et al., 2008; Linares and Holcombe, 2014). Moreover, both tasks rely on different neuronal processes (Basharat et al., 2018) and even one-month old infants seem to be able to evaluate simultaneity (Lewkowicz, 1996). Thus, it is questionable whether the previous and our experimental design addressed the same multisensory processes. In sum, our results provide strong evidence for a sensitive period for cross-modal temporal perception, which leads to the consolidation of cross-modal spatio-temporal biases during early human ontogeny.

Typically, reaction times increase with increasing perceptual uncertainty. In temporal order judgements, uncertainty is highest for simultaneously perceived cross-modal stimuli. Thus, reaction times are expected to be longest at the PSS, if the PSS reflects biases in the perceived relative timing of the modalities, but not if the PSS reflects a response bias towards one modality (Spence, 2010; Shore et al., 2001). Here, CC-individuals’ reaction times and thus sensory uncertainty peaked when the visual stimulus slightly lagged behind the auditory or the tactile stimulus (Figure 1H), whereas all other participants responded slowest when the visual stimulus led by a short amount of time. Thus, even though response speed was not stressed in the instructions, reaction times of the present study suggest that the observed biases reflect perceptual shifts in the relative timing of cross-modal sensory information.

Both cataract-reversal groups exhibited lower spatio-temporal accuracy (Figure 1F) –indicative of a decreased temporal resolution and an increased lapse rate (Figure 1B)– than their controls in the visual and the cross-modal tasks. Persisting visual impairments might have contributed to the reduced visual spatio-temporal resolution of both cataract-reversal groups given that visual contrast affects visual temporal perception (Watson, 1986; Stromeyer and Martini, 2003). The observation that a lower temporal resolution was found not only for visual but additionally for cross-modal temporal order judgments suggests a strong dependence of cross-modal temporal ordering on the visual sense. This might be related to the spatial nature of our task; training of non-spatial, purely visual temporal order judgments did not lead to an improved visual-auditory temporal resolution (Alais and Cass, 2010). The finding that both cataract groups exhibited a lower temporal resolution but only the CC-group an altered cross-modal bias strongly suggests that cross-modal spatio-temporal biases and resolution are dissociable processes. The conjunction of CC- and DC-individuals’ reduced resolution might point towards a long sensitive period for the development of spatio-temporal sensory resolution which would be compatible with the protracted developmental time course of cross-modal temporal ordering of spatially separate events (Röder et al., 2013).

Furthermore, the present finding of increased cross-modal temporal uncertainty could explain why recent studies have found altered multisensory integration for some but not all functions following congenital, transient periods of visual deprivation (Chen et al., 2017; Putzar et al., 2007; Guerreiro et al., 2015). An increased spatio-temporal uncertainty hinders the detection of temporal correlations (Parise and Ernst, 2016) between complex signals such as speech stimuli (Putzar et al., 2007; Guerreiro et al., 2015). At the same time, higher temporal uncertainty predicts wider temporal integration windows for simple, spatially-aligned stimuli (Chen et al., 2017), which in turn might have enabled typical multisensory redundancy gains for such cross-modal stimuli (Putzar et al., 2012; de Heering et al., 2016). It is interesting to note, that this pattern mirrors the typical development of multisensory temporal perception. Very young infants integrate simple visual and auditory stimuli across a wide window of asynchronies but do not integrate more complex stimuli based on other features than stimulus onset (Lewkowicz, 1996; Lewkowicz, 2012). The multisensory integration window narrows during childhood (Noel et al., 2016; Lewkowicz and Flom, 2014; Hillock et al., 2011), in parallel to the improvement of unisensory temporal perception (Brannon et al., 2007). Some developmental theories suggest that more complex temporal multisensory functions build on previously acquired temporal multisensory skills (Lewkowicz, 2000) predicting that the impact of transient loss of pattern vision on multisensory perception should as suggested for the visual system (Hyvärinen et al., 1981; Maurer et al., 2005), increase with increasing task difficulty.

In conclusion, congenital but not late transient visual deprivation was associated with a bias towards perceiving visual events as earlier than auditory or tactile events in sight-recovery individuals, suggesting that cross-modal temporal biases depend on sensory experiences during an early sensitive period.

All data have been deposited on Open Science Framework under CQN48.

1. 1. Badde S
   2. Ley P
   3. Rajendran SS
   4. Kekunnaya R
   5. Shareef I
   6. Röder B

   (2019) Open Science Framework

   Cross-modal temporal biases emerge during early sensitive periods.

   https://doi.org/10.17605/OSF.IO/CQN48

1. 1. Aghdaee SM
   2. Battelli L
   3. Assad JA

   (2014) Relative timing: from behaviour to neurons

   Philosophical Transactions of the Royal Society B: Biological Sciences 369:20120472.

   https://doi.org/10.1098/rstb.2012.0472

   • PubMed
   • Google Scholar
2. 1. Alais D
   2. Carlile S

   (2005) Synchronizing to real events: subjective audiovisual alignment scales with perceived auditory depth and speed of sound

   PNAS 102:2244–2247.

   https://doi.org/10.1073/pnas.0407034102

   • PubMed
   • Google Scholar
3. 1. Alais D
   2. Cass J

   (2010) Multisensory perceptual learning of temporal order: audiovisual learning transfers to vision but not audition

   PLOS ONE 5:e11283.

   https://doi.org/10.1371/journal.pone.0011283

   • PubMed
   • Google Scholar
4. 1. Badde S
   2. Ley P
   3. Rajendran SS
   4. Shareef I
   5. Kekunnaya R
   6. Röder B

   (2019) Resources of sensory experience during early sensitive periods shapes cross-modal temporal biases

   Open Science Framework 1:CQN48.

   https://doi.org/10.17605/OSF.IO/CQN48

   • Google Scholar
5. 1. Basharat A
   2. Adams MS
   3. Staines WR
   4. Barnett-Cowan M

   (2018) Simultaneity and temporal order judgments are coded differently and change with age: an event-related potential study

   Frontiers in Integrative Neuroscience 12:15.

   https://doi.org/10.3389/fnint.2018.00015

   • PubMed
   • Google Scholar
6. 1. Brannon EM
   2. Suanda S
   3. Libertus K

   (2007) Temporal discrimination increases in precision over development and parallels the development of numerosity discrimination

   Developmental Science 10:770–777.

   https://doi.org/10.1111/j.1467-7687.2007.00635.x

   • PubMed
   • Google Scholar
7. Book

   1. Canales J

   (2010) A Tenth of a Second: A History

   University of Chicago Press.

   https://doi.org/10.2979/victorianstudies.54.2.314

   • Google Scholar
8. 1. Chen YC
   2. Lewis TL
   3. Shore DI
   4. Maurer D

   (2017) Early binocular input is critical for development of audiovisual but not visuotactile simultaneity perception

   Current Biology 27:583–589.

   https://doi.org/10.1016/j.cub.2017.01.009

   • PubMed
   • Google Scholar
9. 1. de Heering A
   2. Dormal G
   3. Pelland M
   4. Lewis T
   5. Maurer D
   6. Collignon O

   (2016) A brief period of postnatal visual deprivation alters the balance between auditory and visual attention

   Current Biology 26:3101–3105.

   https://doi.org/10.1016/j.cub.2016.10.014

   • PubMed
   • Google Scholar
10. 1. Engel GR
    2. Dougherty WG

    (1971) Visual-auditory distance constancy

    Nature 234:308.

    https://doi.org/10.1038/234308a0

    • PubMed
    • Google Scholar
11. Book

    1. Fain GL

    (2019) Sensory Transduction

    Oxford University Press.

    https://doi.org/10.1007/978-1-4684-6760-4_13

    • Google Scholar
12. 1. Freeman ED
    2. Ipser A
    3. Palmbaha A
    4. Paunoiu D
    5. Brown P
    6. Lambert C
    7. Leff A
    8. Driver J

    (2013) Sight and sound out of synch: fragmentation and renormalisation of audiovisual integration and subjective timing

    Cortex 49:2875–2887.

    https://doi.org/10.1016/j.cortex.2013.03.006

    • PubMed
    • Google Scholar
13. 1. Fujisaki W
    2. Shimojo S
    3. Kashino M
    4. Nishida S

    (2004) Recalibration of audiovisual simultaneity

    Nature Neuroscience 7:773–778.

    https://doi.org/10.1038/nn1268

    • PubMed
    • Google Scholar
14. 1. Grabot L
    2. van Wassenhove V

    (2017) Time order as psychological bias

    Psychological Science 28:670–678.

    https://doi.org/10.1177/0956797616689369

    • PubMed
    • Google Scholar
15. 1. Guerreiro MJ
    2. Putzar L
    3. Röder B

    (2015) The effect of early visual deprivation on the neural bases of multisensory processing

    Brain 138:1499–1504.

    https://doi.org/10.1093/brain/awv076

    • PubMed
    • Google Scholar
16. 1. Guerreiro MJ
    2. Putzar L
    3. Röder B

    (2016) The effect of early visual deprivation on the neural bases of auditory processing

    The Journal of Neuroscience 36:1620–1630.

    https://doi.org/10.1523/JNEUROSCI.2559-15.2016

    • PubMed
    • Google Scholar
17. 1. Hillock AR
    2. Powers AR
    3. Wallace MT

    (2011) Binding of sights and sounds: age-related changes in multisensory temporal processing

    Neuropsychologia 49:461–467.

    https://doi.org/10.1016/j.neuropsychologia.2010.11.041

    • PubMed
    • Google Scholar
18. 1. Hirsh IJ
    2. Sherrick CE

    (1961) Perceived order in different sense modalities

    Journal of Experimental Psychology 62:423–432.

    https://doi.org/10.1037/h0045283

    • PubMed
    • Google Scholar
19. Book

    1. Hume D

    (2012)

    A Treatise of Human Nature

    Courier Corporation.

    • Google Scholar
20. 1. Hyvärinen J
    2. Carlson S
    3. Hyvärinen L

    (1981) Early visual deprivation alters modality of neuronal responses in area 19 of monkey cortex

    Neuroscience Letters 26:239–243.

    https://doi.org/10.1016/0304-3940(81)90139-7

    • PubMed
    • Google Scholar
21. 1. Ipser A
    2. Agolli V
    3. Bajraktari A
    4. Al-Alawi F
    5. Djaafara N
    6. Freeman ED

    (2017) Sight and sound persistently out of synch: stable individual differences in audiovisual synchronisation revealed by implicit measures of lip-voice integration

    Scientific Reports 7:46413.

    https://doi.org/10.1038/srep46413

    • PubMed
    • Google Scholar
22. 1. Ipser A
    2. Karlinski M
    3. Freeman ED

    (2018) Correlation of individual differences in audiovisual asynchrony across stimuli and tasks: new constraints on temporal renormalization theory

    Journal of Experimental Psychology: Human Perception and Performance 44:1283–1293.

    https://doi.org/10.1037/xhp0000535

    • PubMed
    • Google Scholar
23. 1. Keetels M
    2. Vroomen J

    (2008) Temporal recalibration to tactile-visual asynchronous stimuli

    Neuroscience Letters 430:130–134.

    https://doi.org/10.1016/j.neulet.2007.10.044

    • PubMed
    • Google Scholar
24. Book

    1. Keetels M
    2. Vroomen J

    (2012)

    Perception of synchrony between the senses

    In: Murray MM, Wallace MT, editors. The Neural Bases of Multisensory Processes. CRC Press/Taylor & Francis. pp. 147–177.

    • Google Scholar
25. 1. King AJ

    (2005) Multisensory integration: strategies for synchronization

    Current Biology 15:R339–R341.

    https://doi.org/10.1016/j.cub.2005.04.022

    • PubMed
    • Google Scholar
26. 1. Knudsen EI

    (2002) Instructed learning in the auditory localization pathway of the barn owl

    Nature 417:322–328.

    https://doi.org/10.1038/417322a

    • PubMed
    • Google Scholar
27. 1. Knudsen EI

    (2004) Sensitive periods in the development of the brain and behavior

    Journal of Cognitive Neuroscience 16:1412–1425.

    https://doi.org/10.1162/0898929042304796

    • PubMed
    • Google Scholar
28. 1. Kopinska A
    2. Harris LR

    (2004) Simultaneity constancy

    Perception 33:1049–1060.

    https://doi.org/10.1068/p5169

    • PubMed
    • Google Scholar
29. 1. Lewkowicz DJ

    (1996) Perception of auditory-visual temporal synchrony in human infants

    Journal of Experimental Psychology: Human Perception and Performance 22:1094–1106.

    https://doi.org/10.1037/0096-1523.22.5.1094

    • PubMed
    • Google Scholar
30. 1. Lewkowicz DJ

    (2000) The development of intersensory temporal perception: an epigenetic systems/limitations view

    Psychological Bulletin 126:281–308.

    https://doi.org/10.1037/0033-2909.126.2.281

    • PubMed
    • Google Scholar
31. Book

    1. Lewkowicz DJ

    (2012)

    Development of Multisensory Temporal Perception

    In: Murray MM, Wallace MT, editors. The Neural Bases of Multisensory Processes. CRC Press/Taylor & Francis. pp. 325–343.

    • Google Scholar
32. 1. Lewkowicz DJ
    2. Flom R

    (2014) The audiovisual temporal binding window narrows in early childhood

    Child Development 85:685–694.

    https://doi.org/10.1111/cdev.12142

    • PubMed
    • Google Scholar
33. 1. Linares D
    2. Holcombe AO

    (2014) Differences in perceptual latency estimated from judgments of temporal order, simultaneity and duration are inconsistent

    I-Perception 5:559–571.

    https://doi.org/10.1068/i0675

    • PubMed
    • Google Scholar
34. 1. Maurer D
    2. Lewis TL
    3. Mondloch CJ

    (2005) Missing sights: consequences for visual cognitive development

    Trends in Cognitive Sciences 9:144–151.

    https://doi.org/10.1016/j.tics.2005.01.006

    • PubMed
    • Google Scholar
35. 1. McCulloch DL
    2. Skarf B

    (1994) Pattern reversal visual evoked potentials following early treatment of unilateral, congenital cataract

    Archives of Ophthalmology 112:510–518.

    https://doi.org/10.1001/archopht.1994.01090160086026

    • PubMed
    • Google Scholar
36. 1. Mollon JD
    2. Perkins AJ

    (1996) Errors of judgement at Greenwich in 1796

    Nature 380:101–102.

    https://doi.org/10.1038/380101a0

    • PubMed
    • Google Scholar
37. 1. Mondloch CJ
    2. Segalowitz SJ
    3. Lewis TL
    4. Dywan J
    5. Le Grand R
    6. Maurer D

    (2013) The effect of early visual deprivation on the development of face detection

    Developmental Science 16:728–742.

    https://doi.org/10.1111/desc.12065

    • PubMed
    • Google Scholar
38. 1. Noel JP
    2. De Niear M
    3. Van der Burg E
    4. Wallace MT

    (2016) Audiovisual simultaneity judgment and rapid recalibration throughout the lifespan

    PLOS ONE 11:e0161698.

    https://doi.org/10.1371/journal.pone.0161698

    • PubMed
    • Google Scholar
39. 1. Parise CV
    2. Ernst MO

    (2016) Correlation detection as a general mechanism for multisensory integration

    Nature Communications 7:11543.

    https://doi.org/10.1038/ncomms11543

    • PubMed
    • Google Scholar
40. Book

    1. Pöppel E

    (1988)

    Mindworks: Time and Conscious Experience

    Harcourt Brace Jovanovich.

    • Google Scholar
41. 1. Pöppel E

    (1997) A hierarchical model of temporal perception

    Trends in Cognitive Sciences 1:56–61.

    https://doi.org/10.1016/S1364-6613(97)01008-5

    • PubMed
    • Google Scholar
42. 1. Putzar L
    2. Goerendt I
    3. Lange K
    4. Rösler F
    5. Röder B

    (2007) Early visual deprivation impairs multisensory interactions in humans

    Nature Neuroscience 10:1243–1245.

    https://doi.org/10.1038/nn1978

    • PubMed
    • Google Scholar
43. 1. Putzar L
    2. Gondan M
    3. Röder B

    (2012) Basic multisensory functions can be acquired after congenital visual pattern deprivation in humans

    Developmental Neuropsychology 37:697–711.

    https://doi.org/10.1080/87565641.2012.696756

    • PubMed
    • Google Scholar
44. 1. Röder B
    2. Rösler F
    3. Spence C

    (2004) Early vision impairs tactile perception in the blind

    Current Biology 14:121–124.

    https://doi.org/10.1016/j.cub.2003.12.054

    • PubMed
    • Google Scholar
45. 1. Röder B
    2. Kusmierek A
    3. Spence C
    4. Schicke T

    (2007) Developmental vision determines the reference frame for the multisensory control of action

    PNAS 104:4753–4758.

    https://doi.org/10.1073/pnas.0607158104

    • PubMed
    • Google Scholar
46. 1. Röder B
    2. Pagel B
    3. Heed T

    (2013) The implicit use of spatial information develops later for crossmodal than for intramodal temporal processing

    Cognition 126:301–306.

    https://doi.org/10.1016/j.cognition.2012.09.009

    • PubMed
    • Google Scholar
47. 1. Roseboom W
    2. Linares D
    3. Nishida S

    (2015) Sensory adaptation for timing perception

    Proceedings of the Royal Society B: Biological Sciences 282:20142833.

    https://doi.org/10.1098/rspb.2014.2833

    • Google Scholar
48. 1. Rosenthal R
    2. Rubin DB

    (2003) R equivalent: a simple effect size Indicator

    Psychological Methods 8:492–496.

    https://doi.org/10.1037/1082-989X.8.4.492

    • PubMed
    • Google Scholar
49. 1. Sanford EC

    (1888) Personal equation

    The American Journal of Psychology 2:3–38.

    https://doi.org/10.2307/1411405

    • Google Scholar
50. 1. Shi Z
    2. Burr D

    (2016) Predictive coding of multisensory timing

    Current Opinion in Behavioral Sciences 8:200–206.

    https://doi.org/10.1016/j.cobeha.2016.02.014

    • PubMed
    • Google Scholar
51. 1. Shore DI
    2. Spence C
    3. Klein RM

    (2001) Visual prior entry

    Psychological Science 12:205–212.

    https://doi.org/10.1111/1467-9280.00337

    • PubMed
    • Google Scholar
52. 1. Sourav S
    2. Bottari D
    3. Kekunnaya R
    4. Röder B

    (2018) Evidence of a retinotopic organization of early visual cortex but impaired extrastriate processing in sight recovery individuals

    Journal of Vision 18:22.

    https://doi.org/10.1167/18.3.22

    • PubMed
    • Google Scholar
53. 1. Sourav S
    2. Kekunnaya R
    3. Shareef I
    4. Banerjee S
    5. Bottari D
    6. Röder B

    (2019) A protracted sensitive period regulates the development of cross-modal sound-shape associations in humans

    Psychological Science 30:1473–1482.

    https://doi.org/10.1177/0956797619866625

    • PubMed
    • Google Scholar
54. 1. Spence C
    2. Shore DI
    3. Klein RM

    (2001) Multisensory prior entry

    Journal of Experimental Psychology: General 130:799–832.

    https://doi.org/10.1037/0096-3445.130.4.799

    • PubMed
    • Google Scholar
55. Book

    1. Spence C

    (2010)

    Prior entry: attention and temporal perception

    In: Nobre A. C, Coull J. T, editors. Attention and Time. Oxford University Press. pp. 89–104.

    • Google Scholar
56. 1. Spence C
    2. Squire S

    (2003) Multisensory integration: maintaining the perception of synchrony

    Current Biology 13:R519–R521.

    https://doi.org/10.1016/S0960-9822(03)00445-7

    • PubMed
    • Google Scholar
57. 1. Stromeyer CF
    2. Martini P

    (2003) Human temporal impulse response speeds up with increased stimulus contrast

    Vision Research 43:285–298.

    https://doi.org/10.1016/S0042-6989(02)00412-1

    • PubMed
    • Google Scholar
58. Book

    1. Titchener EB

    (1908)

    Lectures on the elementary psychology of feeling and attention

    Macmillan.

    • Google Scholar
59. 1. van Eijk RL
    2. Kohlrausch A
    3. Juola JF
    4. van de Par S

    (2008) Audiovisual synchrony and temporal order judgments: effects of experimental method and stimulus type

    Perception & Psychophysics 70:955–968.

    https://doi.org/10.3758/PP.70.6.955

    • PubMed
    • Google Scholar
60. 1. Vercillo T
    2. Burr D
    3. Sandini G
    4. Gori M

    (2015) Children do not recalibrate motor-sensory temporal order after exposure to delayed sensory feedback

    Developmental Science 18:703–712.

    https://doi.org/10.1111/desc.12247

    • PubMed
    • Google Scholar
61. 1. Vibell J
    2. Klinge C
    3. Zampini M
    4. Spence C
    5. Nobre AC

    (2007) Temporal order is coded temporally in the brain: early event-related potential latency shifts underlying prior entry in a cross-modal temporal order judgment task

    Journal of Cognitive Neuroscience 19:109–120.

    https://doi.org/10.1162/jocn.2007.19.1.109

    • PubMed
    • Google Scholar
62. 1. Vroomen J
    2. Keetels M
    3. de Gelder B
    4. Bertelson P

    (2004) Recalibration of temporal order perception by exposure to audio-visual asynchrony

    Cognitive Brain Research 22:32–35.

    https://doi.org/10.1016/j.cogbrainres.2004.07.003

    • PubMed
    • Google Scholar
63. Book

    1. Watson AB

    (1986)

    Temporal sensitivity

    In: Boff K, Thomas J, editors. Handbook of Perception and Human Performance. Wiley. pp. 6.1–6.6.

    • Google Scholar
64. 1. Yamauchi Y
    2. Mochizuki JI
    3. Hirakata A
    4. Uda S

    (2016) Single flash electroretinograms of mature cataractous and fellow eyes

    Clinical Ophthalmology 10:2031–2034.

    https://doi.org/10.2147/OPTH.S118677

    • PubMed
    • Google Scholar
65. 1. Zampini M
    2. Shore DI
    3. Spence C

    (2003) Audiovisual temporal order judgments

    Experimental Brain Research 152:198–210.

    https://doi.org/10.1007/s00221-003-1536-z

    • PubMed
    • Google Scholar
66. 1. Zampini M
    2. Guest S
    3. Shore DI
    4. Spence C

    (2005) Audio-visual simultaneity judgments

    Perception & Psychophysics 67:531–544.

    https://doi.org/10.3758/BF03193329

    • PubMed
    • Google Scholar

Author details

1. Stephanie Badde

   1. Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany
   2. Department of Psychology and Center of Neural Science, New York University, New York, United States

   Contribution

   Data curation, Software, Formal analysis, Supervision, Visualization, Methodology, Writing - original draft, Writing - review and editing

   For correspondence

   stephanie.badde@nyu.edu

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0002-4005-5503

2. Pia Ley

   Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany

   Contribution

   Software, Investigation, Methodology, Writing - review and editing

   Competing interests

   No competing interests declared

3. Siddhart S Rajendran

   1. Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany
   2. Child Sight Institute, Jasti V Ramanamma Children's Eye Care Center, LV Prasad Eye Institute, Hyderabad, India

   Contribution

   Investigation, Writing - review and editing

   Competing interests

   No competing interests declared

4. Idris Shareef

   1. Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany
   2. Child Sight Institute, Jasti V Ramanamma Children's Eye Care Center, LV Prasad Eye Institute, Hyderabad, India

   Contribution

   Data curation, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-9258-2199

5. Ramesh Kekunnaya

   1. Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany
   2. Child Sight Institute, Jasti V Ramanamma Children's Eye Care Center, LV Prasad Eye Institute, Hyderabad, India

   Contribution

   Resources, Investigation, Writing - review and editing

   Competing interests

   No competing interests declared

   "This ORCID iD identifies the author of this article:" 0000-0001-5789-2300

6. Brigitte Röder

   Biological Psychology and Neuropsychology, University of Hamburg, Hamburg, Germany

   Contribution

   Conceptualization, Resources, Supervision, Funding acquisition, Investigation, Methodology, Writing - original draft, Writing - review and editing

   Competing interests

   No competing interests declared

• 1,556

  Page views

• 233

  Downloads

• 7

  Citations

Article citation count generated by polling the highest count across the following sources: Crossref, PubMed Central, Scopus.