Reliability-based cross-modal recalibration.

Which modality serves as the “teaching signal” in cross-modal recalibration might be determined based on cue reliability. Consistent with this hypothesis, more robust evidence for recalibration of visual perception was found in studies that examined aftereffects for properties for which the visual signal was not as reliable as the other signal [55–60]. Burge and colleagues directly tested the relationship between cue reliability and the amount of recalibration by manipulating the reliability of a visual stimulus for slant to be either smaller, almost equal, or greater than that of a haptic cue to slant [61]. As the visual cue became less reliable, greater recalibration of vision and less recalibration of haptics was observed. The authors concluded, in agreement with others [53, 59], that each sense is recalibrated proportional to its relative reliability.

Fixed-ratio cross-modal recalibration.

In contradiction to the results just discussed, a recent study investigating the mechanism underlying visual-vestibular recalibration found evidence that the two modalities are recalibrated in a fixed ratio regardless of cue reliability [62]. More specifically, after exposing humans and monkeys to systematically discrepant visual and vestibular heading directions, both cues significantly shifted in the direction required to reduce cue conflict, but the amount of recalibration in either modality did not depend on the measured relative cue reliabilities [62]. A model assuming a fixed ratio between the degree of recalibration of each sense, independent of relative reliability, best captured the data.

Causal inference and cross-modal recalibration.

Both reliability-based and fixed-ratio models of cross-modal recalibration assume that the brain acts upon the discrepancy between the two sensory cues. However, according to the principles of cross-modal integration, two discrepant cues can lead to very different percepts, depending on each cue’s reliability and inference about whether they have common or separate origins. Cross-modal recalibration might take this perceptual inference into account. Indeed, causal-inference models of cross-modal recalibration have successfully predicted visual-auditory [63] and visual-tactile [32] ventriloquism aftereffects. In these models, recalibration is not based on a mere comparison of two sensory cues, but rather relates the cues to the perceptual estimates and by doing so incorporates causal inference and cue reliability. According to this model, conflicting findings regarding the influence of cue reliability on recalibration might reflect differences in the perceptual estimates rather than diverging underlying mechanisms.

The reliability-based model.

According to this model, the brain determines the amount of recalibration (i.e., the measurement shift) for each modality based on the reliabilities of both measurements. The shift for one modality is updated by a fraction of the difference between the two measurements that is proportional to the other modality’s relative reliability. Across trials, the amount of recalibration depends not only on stimulus reliability, but also on a common learning rate α for both modalities. This model predicts increasing visual and decreasing auditory recalibration effects with decreasing visual stimulus reliability (Fig 7, dot-dashed line).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 7. Simulated auditory recalibration effects based on three candidate models of cross-modal recalibration.

The effect of visual (horizontal axis) and auditory (panels) stimulus reliability in bimodal trials on the amount of auditory recalibration by vision, based on causal-inference (solid line), fixed-ratio (dashed line), and reliability-based (dot-dashed line) models for two different learning rates (top row: slow; bottom row: fast).

https://doi.org/10.1371/journal.pcbi.1008877.g007

The fixed-ratio model.

According to this model, the brain determines the amount of recalibration based on the modalities of both sensory measurements. The measurement shift for each sense is updated by a fraction of the difference between the two measurements. This fraction is determined only by modality-specific learning rates. Therefore, this model predicts modality-specific recalibration effects that are not influenced by stimulus reliability (Fig 7, dashed line).

The causal-inference model.

According to this model, the brain determines the amount of recalibration for each modality based on the difference between a measurement and the corresponding location estimate. The measurement shifts are updated by a fraction of this difference, either implemented as modality-specific learning rates or as a supra-modal learning rate. In this model, cross-modal recalibration depends on stimulus reliability, modality-specific localization biases, and inference about a common cause through their influences on the location estimates. The model can predict various effects of visual reliability on auditory recalibration (Fig 7, solid line).

Model predictions.

The causal-inference model is the only model that can capture the idiosyncratic influence of visual stimulus reliability on auditory recalibration across participants (Fig 8A). Data from 5 out of 6 participants were best fitted by the causal-inference model, either with a supra-modal learning rate (see Section S3 in S1 Appendix for model predictions) or modality-specific ones. For most participants, neither the reliability-based nor the fixed-ratio model can reproduce the diverse influence of visual stimulus reliability on auditory recalibration (Fig 8B; Sections S4-S5 in S1 Appendix). The three models do not differ in terms of predicting visual recalibration and modality-specific biases (Section S6 in S1 Appendix).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 8. Model predictions and model comparison.

(A) Observed (circles) and predicted (diamonds) auditory (blue) and visual (pink) final measurement shifts after the recalibration phase based on 1,000 runs using the best-fitting parameters given the causal-inference model (diamonds) as a function of visual reliability for all participants (panels). Error bars: 95% bootstrapped confidence intervals. (B) Model predictions by the fixed-ratio model (triangles in left panel) and the reliability-based model (inverted triangles in right panel) for participant S1. (C) Model comparison indices, smaller values indicate more evidence (RB = reliability-based, FR = fixed ratio, CI = causal inference, = causal inference with supramodal learning rate).

https://doi.org/10.1371/journal.pcbi.1008877.g008

Model comparison.

To compare model performance quantitatively, we computed the Akaike information criterion (AIC) for all three models [64] and then calculated relative model-comparison scores, Δ_AIC, which relate the AIC value of the best-fitting model (the model with the lowest AIC) to that of each of the other models (a high value of Δ_AIC indicates stronger evidence for the best-fitting model; Fig 8C). Δ_AIC values comparing both the reliability-based and the fixed-ratio to the causal-inference models were large for the majority of participants, revealing substantial evidence for the causal-inference model of cross-modal recalibration.

The role of cue reliability for cross-modal recalibration.

Cue reliability influences the degree of cross-modal recalibration by influencing the posterior probability that both cues arose from a common cause as well as the integrated location estimate for the common-cause scenario. Both of them determine the final location estimate and in turn the amount of recalibration. When the visual cue is extremely reliable, the posterior probability that two discrepant cues originated from the same source is low (Fig 9B). If the common-cause scenario is unlikely, the auditory location estimate is mostly based on the estimate given separate sources and therefore located close to the auditory cue (Fig 9A, top panel), which leads to small recalibration effects.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 9. Cue reliability and the amount of recalibration.

(A) Non-linear effect of visual stimulus reliability in bimodal trials (; different panels) on the auditory spatial estimate (green vertical lines). Blue and red dashed vertical lines and curves: auditory and visual measurements ( and in perceptual space) and likelihood functions, respectively. Blue: when separate causes are assumed, the auditory measurement and likelihood equal the auditory location estimate and the posterior distribution over auditory locations (a flat prior over stimulus location is assumed). Grey dashed vertical lines and curves: audiovisual location estimates and posterior distributions of audiovisual locations conditioned on a common cause. (B-D) The effect of visual reliability on the posterior probability of a common cause, , the integrated location estimate, i.e., the estimate conditioned on a common audiovisual source , and the distance between auditory measurement and location estimate , which directly sets the amount of recalibration.

https://doi.org/10.1371/journal.pcbi.1008877.g009

However, the amount of auditory recalibration does not increase monotonically with decreasing visual reliability. With decreasing visual reliability, the integrated location estimate for the common-cause scenario is increasingly dominated by the auditory measurement (Fig 9C). The closer the integrated estimate is to the auditory measurement, the closer the final location estimate is to the auditory measurement, and the smaller the auditory recalibration effect will be.

Importantly, the effect of variation in reliability depends on the tested reliability range (Fig 9D). Thus, studies that use different types of stimuli will likely obtain different results regarding the influence of stimulus reliability on cross-modal recalibration. The contradictions that emerged between previous studies [61, 62] can be explained by the causal-inference model of recalibration.

The role of common cause prior assumptions for cross-modal recalibration.

The common-cause prior impacts the posterior probability of a common cause and hence the amount of recalibration independent of stimulus reliability and discrepancy (Fig 10). As the common-cause prior increases, the posterior probability of a common cause increases, leading the final auditory location estimate to be farther away from the auditory measurement and hence yielding a larger measurement-shift update. We used a post-response cue in the bimodal localization task during the recalibration phase to foster audiovisual recalibration as the common-cause prior is strengthened when participants attend to both modalities [32].

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 10. Determinants of recalibration.

The joint effects of visual reliability in bimodal trials, , the distance between auditory and visual measurements in perceptual space, , and the common-cause prior, p(C = 1) on the posterior probability of a common cause, (panel A), and the distance between the final location estimate and the auditory measurement, (panel B), which is proportional to the amount of recalibration.

https://doi.org/10.1371/journal.pcbi.1008877.g010

Modality-specific spatial biases.

Our model differs from previous causal-inference models of cross-modal perception in that we assumed the existence of modality-specific biases but not modality-specific spatial priors over stimulus location [32, 42, 66]. Both can account for the typically observed biases in localization, but modality-specific priors exert a stronger influence when visual reliability is reduced. In contrast, the influence of modality-specific biases does not vary as stimulus reliability changes. Thus, we conducted a control experiment examining whether the tendency to perceive visual stimuli as shifted towards the central fixation increased with decreasing visual reliability (Section S7 in S1 Appendix). The results showed no systematic influence of visual reliability. Thus, to avoid overfitting the model, we omitted modality-specific priors in our models and fitted only modality-specific biases.

The existence of fixed biases in the spatial perception of visual and auditory stimuli seems to be at odds with the concept of cross-modal recalibration. If discrepancies between the senses consistently lead to recalibration, why do these biases persist? Perceptual biases could reflect adjustments for sensory discrepancies during an early sensitive period of development as has recently been shown for cross-modal biases in temporal perception [31]. After the sensitive period, it might become impossible to fully compensate for newly developing differences between the senses. In terms of our model, this would mean that a limitation to the shift-updates is set during early infancy.

Our study differs from previous spatial-recalibration studies in that we adjusted for perceptual biases in auditory relative to visual spatial perception by selecting visual locations perceptually aligned with pre-selected auditory locations. During piloting we presented stimuli with a constant physical spatial audiovisual discrepancy during the recalibration phase, and did not find significant recalibration effects. We attributed this to the modality-specific biases we had observed combined with the small spatial discrepancy used in our study. Indeed, our simulations with the causal-inference model (Fig 11, top panel) show that there are many combinations of proportional and constant biases that lead to minuscule or even negative recalibration effects when these biases are not adjusted for. Additionally, our simulations reveal a complex interaction between these biases and the spatial discrepancy. Recalibration through exposure to a constant and relatively large physical discrepancy, as done in most previous studies [43–45, 47–50, 67], would produce positive recalibration effects on average but not necessarily in all participants given that humans differ in their spatial biases. In contrast, keeping the discrepancy constant in perceptual space makes the recalibration effects less prone to individual differences in modality-specific biases and works better with smaller spatial discrepancies (Fig 11, bottom panel).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 11. The influence of spatial discrepancy and modality-specific biases on the amount of auditory recalibration.

Top row: The auditory recalibration effects (color key) as a function of proportional and constant shift of visual relative to auditory location when spatial discrepancy (panels) is constant in physical space (i.e., s_V = s_A + spatial discrepancy). Bottom row: The auditory recalibration effects when spatial discrepancy is constant in perceptual space (i.e., visual stimulus locations are selected to adjust the perceptual biases in auditory relative to visual spatial perception, s_V = proportional shift × (s_A + spatialdiscrepancy) + constant shift).

https://doi.org/10.1371/journal.pcbi.1008877.g011

Unimodal spatial-discrimination task.

Participants’ spatial-discrimination thresholds were measured using a 2-interval, forced-choice (2IFC) procedure. In each trial, a standard and a test stimulus were presented in random order. Each stimulus was preceded by a fixation cross, presented at the center of the screen for 1,000 ms, followed by a 2,000 ms-long period of blank screen in which the loudspeaker moved to its position. The actual stimulus lasted 100 ms, followed by either a 900 ms-long backward masker (visual stimuli) or a blank screen for 900 ms (auditory stimuli). After the second stimulus period was over, a response probe was displayed and participants indicated by button press which interval contained the stimulus that was farther to the right (Fig 1B). Visual feedback was provided for 500 ms immediately after the response was given. The inter-trial interval was 500 ms.

The standard stimulus was located straight-ahead at the center of the screen; the location of the test stimulus was controlled by four interleaved staircases, two of which had the test stimulus start at 12.5° (to the right of straight-ahead), and the other two at −12.5° (12.5° to the left of straight-ahead). For the two staircases starting at one side, one followed the two-down-one-up rule (with down being defined as moving the test stimulus leftwards) converging to a probability of 71% [75] of perceiving the test stimulus as farther to the right than the standard stimulus; the other staircase followed the one-down-two-up rule converging to a probability of 29%. The initial step size was 1.9°, decreased to 1.0° after the first staircase reversal and to 0.5° after the third reversal. Each staircase consisted of 40 trials, resulting in a total of 160 trials. To improve the estimation of the lapse rate, a test stimulus was presented distant from the center (±12.5°) once every 10 trials. A total of 176 trials was evenly split into 4 blocks.

Participants completed the spatial-discrimination task for each of the three levels of visual stimulus reliability in random order, the auditory stimulus was always tested last. Participants took about an hour to complete one stimulus condition; typically, they spread all four stimulus conditions across two days.

Bimodal spatial-discrimination task.

Participants’ biases in auditory relative to visual spatial perception were measured using a spatial 2IFC procedure. An auditory and a high-reliability visual stimulus were presented in random order and participants indicated by button press whether the visual stimulus was located to the left or right of the auditory stimulus. The procedure was otherwise identical to that of the unimodal spatial-discrimination task (Fig 3A). No feedback was provided.

The auditory stimulus was presented at one of four locations (±2.5 or ±7.5°). We used the adaptive staircase procedure to effectively sample the visual stimulus space for each participant as the range of meaningful stimulus locations varied considerably across participants due to individual perceptual biases. Specifically, the location of the visual stimulus was controlled by eight interleaved staircases, two for each of the four auditory locations. Of the two staircases per auditory location, one started the test stimulus at 15° relative to the auditory location, and the other one at −15°. Staircases with the visual stimulus starting from the right of the auditory stimulus followed the one-down-two-up rule, converging to a probability of 29% of choosing the visual as to the right of the auditory stimulus. Staircases with the visual stimulus starting from the left of the auditory stimulus followed the two-down-one-up rule, converging to a probability of 71%. Staircase step size was updated as described above. Each staircase consisted of 36 trials. Trials with the visual stimulus being located at ±15° relative to the auditory stimulus were inserted once every nine trials, resulting in a total of 320 trials. The session was divided into six blocks. Usually, participants took about two hours to complete all trials.

Pointing practice task.

Participants’ localization precision independent of spatial perception was measured using a localization task with visual stimuli of maximal spatial reliability. In each trial, a white square (8 × 8 pixels ≈ 0.6° × 0.6°) was displayed on the screen for 100 ms. The stimulus was followed by a 900 ms-long backward masker and 500 ms of blank screen. Subsequently, the response cursor, a green square of the same size as the white square, appeared on the screen. Participants used the pointing device to move the cursor to the stimulus position, and confirmed their response with the footpedal. Response times were unrestricted. The cursor location was shown during adjustment, but error feedback was not provided. There were eight possible horizontal positions for the stimulus, evenly spaced from -17.5 to 17.5° in steps of 5°. Each stimulus location was visited 30 times in random order, resulting in a total of 240 trials. The inter-trial interval was 500 ms. This experiment took 30 minutes to complete and was typically administered after the bimodal spatial-discrimination task on the same day.

Unimodal localization task (pre- and post-recalibration phase).

Participants’ baseline and post-recalibration spatial perception were measured using a unimodal spatial-localization task. In each trial, either an auditory or a visual stimulus was presented, again preceded by a fixation cross and a blank screen. The inter-trial interval was 100 ms, otherwise timing was identical to that of the discrimination tasks. Auditory stimulus locations were the same as in the bimodal spatial-discrimination task; visual stimulus locations were the four locations that were identified as perceptually co-located with those four auditory locations using the bimodal spatial-discrimination task. Participants again responded by moving a visual cursor to the stimulus location (Fig 5A). There was no time limit for the response. The location of the visual cursor was shown during the adjustment, but feedback about the localization error was not provided. Each of the four target locations per modality was tested 12 times, resulting in a total of 96 trials administered in pseudorandom order. These trials were split into four blocks. Usually participants took about 25 min to complete all 96 trials; they did so once at the beginning of the session (pre-recalibration phase) and once again after the recalibration phase (post-recalibration phase).

Bimodal localization task (recalibration phase).

During the audiovisual recalibration phase, participants were presented with temporally synchronous but spatially discrepant audiovisual stimuli. We asked them to localize either the auditory or the visual component, with the localization modality cued after stimulus presentation (Fig 5B), a procedure that has been associated with larger recalibration effects than non-spatial tasks [32]. All other trial parameters were identical to the unimodal localization task. Three audiovisual stimulus pairs were chosen such that an auditory stimulus location was paired with the visual location perceived as aligned with the auditory stimulus location to its left in the visual-left-of-auditory condition and the auditory stimulus location to its right in the visual-right-of-auditory condition (Fig 5C). Each of the three audiovisual pairs was repeated 40 times in random order, resulting in a total of 120 trials. These trials were split into four blocks. Usually, participants took about 30 min to complete all 120 trials. A full session (pre-recalibration, recalibration, and post-recalibration phase) took 80 min. Participants completed the six sessions on separate days.

Unimodal spatial-discrimination task.

For the unimodal spatial-discrimination task, the data were coded as the probability of identifying the test stimulus as located farther to the right than the standard stimulus as a function of test stimulus location separately for each stimulus type (Fig 2A). These data were fitted with a cumulative Gaussian distribution centered at 0 and with a lapse rate constrained to be less than or equal to 6% [76]. The JND was calculated as half the distance between the stimulus locations corresponding to probabilities of 0.25 and 0.75 according to the fitted cumulative Gaussian distribution (unscaled by the lapse rate). To measure goodness of fit, we computed adjusted R² values based on the binned data (bin size = 1.8°). To derive error bars, we randomly resampled the raw data with replacement 1,000 times, fitted psychometric functions to each resampled dataset, calculated the JND, and took the 2.5 and 97.5 percentiles of the 1,000 JNDs as the bootstrapped confidence interval.

Bimodal spatial-discrimination task.

Data from the bimodal spatial-discrimination task were coded as the probability of identifying the visual stimulus as located to the right of the auditory stimulus as a function of visual stimulus location. We fitted four cumulative Gaussian distributions to these data, one for each auditory standard stimulus. Again, we included a lapse rate, constrained to be less than or equal to 6% [76]. Adjusted R² values were calculated based on binned data (bin size = 3°). The point of subjective equality (PSE) was defined as the visual stimulus location corresponding to a probability of 0.5 according to the unscaled psychometric function. The four PSEs were modeled as a linear function of auditory stimulus location. From this linear regression of the PSEs, we computed the locations of the visual stimulus perceived as co-located with the four auditory locations. In the subsequent unimodal and bimodal spatial-localization tasks, we presented visual stimuli at these locations rather than those directly indicated by the PSEs to reduce effects of random noise during the bimodal spatial-discrimination task. 95% confidence intervals for each parameter were obtained as before.

Pointing practice task.

Data from the pointing practice task, used to measure localization response precision, were not statistically analyzed but were filtered before the model fitting. For each stimulus location, we z-transformed the data by subtracting the mean localization response per stimulus location and then dividing by the standard deviation of all demeaned responses. Localization responses with a z-score outside of [−3, 3] were identified as outliers (0–1.67% of trials) and excluded from the model fitting.

Unimodal localization task (pre- and post-recalibration phase).

Data from the unimodal localization task were filtered separately for each modality, stimulus location and phase. To compute the means for the z-transformation, auditory and visual localization responses in the pre-recalibration phase were pooled across all six sessions, as the day of testing should not influence localization performance. In contrast, the means of auditory and visual localization responses in the post-recalibration phase were calculated separately for each of the six conditions as each recalibration condition should influence localization differently. Then, we computed the standard deviation of the demeaned auditory localization responses pooled across all six session and both phases. The standard deviation of demeaned visual localization responses was calculated separately for each visual-reliability condition given that stimulus reliability should influence localization precision. Localization responses identified as outliers (z-scores outside of [−3, 3]; 0.78–1.82% of trials) were excluded from all further analyses.

For statistical analysis and display, we summarized localization responses as a linear function of stimulus location, separately for each modality and phase. For each modality, localization responses in the pre-recalibration phase were pooled across all six sessions (visual reliability should influence the precision but not the accuracy of the localization responses; Section S7 in S1 Appendix). Responses from the post-recalibration phase were regressed separately for each condition, because each condition should influence localization differently. The seven regression lines per modality were fit with the constraint that all have the same slope. The amount of recalibration of one modality by the other was defined as the distance between the intercepts of the regression lines for pre- and post-recalibration localization responses. It was coded as positive if localization responses in the post-recalibration phase were shifted to compensate for the audiovisual discrepancy in the preceding recalibration phase. For the statistical analysis, we calculated the amount of recalibration for each modality and recalibration condition (2 recalibration directions × 3 reliability levels). To derive confidence intervals, localization responses were resampled, separately for each location, task, session, and modality.

Data from the bimodal spatial-localization task conducted during the audiovisual recalibration phase were not analyzed. Data analysis was done using Python 3.7, R 4.0.2, and MATLAB 2019a.

The reliability-based model of cross-modal recalibration.

According to this model, each modality should be recalibrated in the direction of the other modality by an amount that is proportional to the other modality’s relative reliability [53]. In other words, after every trial, the measurement shifts, and , are updated in the direction of the discrepancy between the visual and auditory measurements by an amount proportional to the two modalities’ relative reliabilities as follows: (1) where (2) and analogously (3) where l_A and l_V index the auditory and visual locations (l_A, l_V ∈ {1, 2, 3, 4}), α denotes a supra-modal learning rate, and t denotes trial number.

The fixed-ratio model of cross-modal recalibration.

According to this model, after every trial, and are updated in the direction of the discrepancy between the visual and auditory measurements by a fixed ratio of this discrepancy. The ratio of the update depends solely on the identity of the modality and thus is independent of stimulus reliability [62]. and are updated according to the following equations: (4) and (5) where α_A and α_V are modality-specific learning rates.

The causal-inference model of recalibration.

In this model, the shift updates are determined by the discrepancy between a measurement and the corresponding perceptual estimate, and [63] for each modality. The spatial discrepancy between auditory and visual measurements and the relative reliabilities of both stimuli have indirect influence on the shift updates by means of their influence on the location estimates, and . Additionally, the location estimates and thus the shift updates are contingent on the degree to which the brain infers a common cause or separate causes for the two measurements [30, 63].

The location estimates are a mixture of two conditional location estimates, one for each causal scenario (common audiovisual source, C = 1, or different auditory and visual sources, C = 2). In the case of a common source, the location estimate of the audiovisual stimulus pair, , equals the reliability-weighted average of the measurements , and the mean of an internal, Gaussian-shaped, supra-modal prior across stimulus locations with variance : (6)

In the case of two separate sources, the location estimates of the auditory and the visual stimulus, and , are equal to the reliability-weighted averages of and for the auditory estimate, and and for the visual estimate, respectively: (7) and (8)

The final location estimates are derived by model averaging (see alternative decision strategy in Section S3 in S1 Appendix). Specifically, the final location estimate is the average of the conditional location estimates, and , with each estimate weighted by the posterior probability of its causal structure: (9) and analogously for the visual location estimate: (10)

The posterior probability of a common source for the auditory and visual measurements in trial t, , is proportional to the product of the likelihood of a common source for these measurements in trial t and the prior probability of a common source for visual and auditory measurements in general, P(C = 1): (11)

The posterior probability of two separate sources, one for the auditory and one for the visual measurement, is .

The likelihood of a common source of the visual and auditory measurements in trial t, , is the product of the likelihood of the internally represented audiovisual stimulus location given the auditory measurement, , and the visual measurement , and the supra-modal prior, integrated over all possible remapped audiovisual stimulus locations [30]: (12)

The likelihood of different sources for the visual and auditory measurements in trial t, is the product of the likelihood of internally represented auditory and visual stimulus locations and given the auditory measurement, , and the visual measurement , and the supra-modal prior. Given that the measurements in this causal scenario stem from different sources, the product is integrated over all possible, remapped visual and auditory stimulus locations, and : (13)

The updates of the shifts are scaled by two modality-specific learning rates, α_A and α_V [63]: (14) and (15)

We also tested a version of the model with one supra-modal learning rate (α = α_A = α_V).

Unimodal spatial-discrimination task.

The unimodal spatial-discrimination task was conducted to estimate the one auditory and three visual stimulus reliabilities under unimodal presentation conditions as well as to constrain the estimates of the variable bias a_A introduced by the remapping process. We begin by describing the auditory version. The standard stimulus was presented straight ahead, at location s_A,0, and the test stimulus was presented at one of N_A locations, s_A,n, determined by an adaptive procedure. For each pair, the probability, p_A,n, of estimating the test stimulus to be located to the right of the standard stimulus is a function of the physical distance between the two stimuli.

We assume that the observer makes the decision by comparing the internal location estimates and , (16)

To further specify p_A,n, we have to derive the probability distributions of the internal location estimates. Each physical stimulus at location s_A,n results in an internal measurement, . The measurement distribution is Gaussian () and for a given measurement the estimate of the remapped location of the stimulus is the average of the measurement and the mean of the spatial prior, , each weighted by their relative reliabilities (). Thus, the probability distribution of the location estimates of a test stimulus is (17) where (18)

The probability distribution of the difference between the two location estimates and is (19) where . Taken together, the probability of perceiving an auditory test stimulus at location s_A,n to the right of an auditory standard stimulus at location s_A,0 is (20) where Φ(x;μ, σ²) is the cumulative Gaussian distribution.

However, as experimenters we only have access to response probabilities as a function of the stimulus locations in physical space. Given that the remapped location of is a function of the physical stimulus location s_A, we can rewrite Eq 20 as (21) and analogously, (22)

Finally, the model includes occasional response lapses (i.e., random button presses) at rate λ, so that the probability of reporting the test stimulus as located farther to the right than the standard (r_A,n = 1) is (23)

Bimodal spatial-discrimination task.

The bimodal spatial-discrimination task was conducted to estimate the relative bias of auditory compared to visual spatial perception, i.e., to estimate a_A and b_A. Auditory stimuli were presented at four different locations in physical space s_A,l, where l indexes the auditory location. Guided by a staircase procedure, on each trial t, an auditory stimulus at location s_A,l was paired with a visual stimulus of high spatial reliability (i = 1) at one of N test locations s_V,n, where n indexes the finer grid of locations of visual stimuli that were presented during the task. For each pair, the model predicts p_l,n, the probability of judging the visual stimulus at location s_V,n as to the right of the auditory stimulus at location s_A,l.

Analogous to the unimodal spatial-discrimination task, we specify the probability distributions of the internal auditory and visual location estimates as (24) where (25) (26) (27)

The probability of the observer perceiving a visual stimulus at physical location s_V,n as located to the right of an auditory stimulus at physical location s_A,l is thus: (28)

The distribution of this difference is: (29)

The probability of perceiving the visual stimulus to the right of the auditory can then be expressed as (30)

As in the unimodal spatial-discrimination task, the model includes occasional lapses at rate λ_AV. Therefore, the probability of reporting a visual stimulus at s_V,n as located to the right of an auditory stimulus at location s_A,l (r_l,n = 1) is equal to (31)

Pointing practice task.

The pointing practice task was used to estimate localization response variability, , due to sources unrelated to the spatial perception of the stimuli. Visual stimuli were presented at eight different locations s_V,o, where o indexes the stimulus location (o ∈ {1, 2, …, 8}). Localization responses (i.e., confirmed cursor positions) in each trial were modeled as perturbed by Gaussian-distributed noise and centered on the physical stimulus location: (32)

By doing so, we assume that 1) the location of the visual cursor in physical space maps directly to its location in perceptual space and 2) the stimulus location estimate is unbiased. This is based on our general assumption of identity remapping for visual stimuli as well as on the high spatial reliability of the visual cursor and the visual stimulus, which should safeguard their estimates against the influence of spatial priors. See Section S2 in S1 Appendix for a model that does not have these assumptions.

Unimodal localization task—Pre-recalibration phase.

The unimodal localization task was conducted before and after the recalibration phase to measure shifts in auditory and visual localization responses as a consequence of exposure to spatially discrepant audiovisual stimuli during the recalibration phase. Stimuli were presented at four locations for each modality, s_A,l and s_V,l.

As for the pointing practice task, we assumed that the probability distributions of the localization responses in physical space, r_A,l and , are centered on the location estimates and in perceptual space, and corrupted by additional unbiased noise. As in the spatial-discrimination tasks (and unlike in the pointing practice task that used a different, maximally reliable stimulus), the stimulus location estimates are assumed to be biased due to the remapping process and the incorporation of the supra-modal spatial prior. It follows that the probability distributions of the localization responses are (33) where the terms are defined in Eqs 24–27.

Unimodal localization task—Post-recalibration phase.

In the post-recalibration phase, the remapping from physical to perceptual space had been updated so that additional shifts and , accumulated after 120 exposures to discrepant audiovisual stimuli, were incorporated: and . This change in the measurement distributions affects the centers of the location estimates’ probability distributions as follows: (34)

We assumed that the probability distributions of the localization responses are centered on these updated values, that is, we did not implement the updated remapping for the location estimates of the cursor. In sum, localization responses to unimodally presented visual and auditory stimuli have a Gaussian probability distribution that, after the audiovisual recalibration task, additionally depends on the final shift updates and .

Model log-likelihood—Unimodal spatial-discrimination task.

In the unimodal spatial-discrimination task (auditory session), participants indicated whether the test stimulus was located to the left, r_A,n(t) = 0, or to the right of the standard stimulus, r_A,n(t) = 1. For each such trial, the likelihood of model parameters given the response r_A,n(t) is (36) where is defined in Eq 23. Thus, the log likelihood given responses across all T_1,A trials is (37)

Analogously, (38)

The log likelihood across all four sessions is (39)

The set of free parameters that were constrained by the binary responses in this task is .

Model log-likelihood—Pointing practice task.

For each trial, the likelihood of the model parameters given a visual stimulus at location s_V,o(t) and a subsequent response (cursor setting) r_V,o(t) is where φ refers to the Gaussian probability density. The only free parameter that was constrained by this task is Θ₂ = {σ_r}. The maximum-likelihood estimate of σ_r is (40) where T₂ and the sum do not include outlier trials.

Model log-likelihood—Bimodal spatial-discrimination task.

In the bimodal spatial-discrimination task, for each trial t, participants indicated whether the visual test stimulus at location was located to the left, r_l(t),n(t) = 0, or to the right, r_l(t),n(t) = 1, of the auditory standard stimulus presented at s_A,l(t). For each such trial, the likelihood of model parameters given the response r_l(t),n(t) is (41) where is defined in Eq 31. Thus, the log likelihood given the responses across all trials is (42) is a function of p_l(t),n(t), which in turn depends on the bias parameters a_A and b_A, the parameters of the supra-modal prior over locations and , as well as the measurement variances and (see Eqs 24–27). Fitting both the bias parameters and the supra-modal prior at once was impossible as they effectively traded off. Thus, we implemented a non-informative supra-modal prior over stimulus locations by setting to 100 and to 0. , , , and were estimated based on the forced-choice responses from the unimodal spatial-discrimination task. The final set of free parameters that were constrained by the binary responses in this task was . The bias parameters, a_A and b_A, were jointly estimated using the data from this task as well as pre- and post-recalibration responses from the unimodal localization task.

Model log-likelihood—Unimodal localization task—Pre-recalibration phase.

In this task, each localization results in cursor location settings r_A,i,j,l(t) and on trial t of session (i, j) where i indicates the visual-reliability condition and j the recalibration direction in the subsequent recalibration phase. The localization responses from this task were modeled as Gaussian-distributed. From these distributions, we can compute the likelihood of a model M and the parameter set as the Gaussian probability density function in Eq 33 evaluated at the observed localization responses r_A,i,j,l(t) and : (43)

The log likelihood is the sum of the log likelihoods across the trials of all six sessions: (44)

The log-likelihood depends on and , which in turn depend on the bias parameters a_A and b_A, the parameters of the supra-modal prior and , as well as the measurement variances and (see Eq 34), and the response noise σ_r. We chose a flat prior over stimulus locations, the (scaled) measurement variances ( and ) were estimated based on the unimodal spatial-discrimination task, and σ_r was estimated based on the pointing practice task. Consequently, the actual set of parameters constrained by localization responses from the pre-recalibration task was . Here, the values of the T variables and the sums do not include outlier trials.

Model log-likelihood—Unimodal localization task—Post-recalibration phase.

Localization responses in the post-recalibration phase additionally depend on the updates for the visual and auditory shifts accumulated after 120 trials during the recalibration phase, and (Eq 34). Since these accumulated shift updates are not accessible to the experimenter, we marginalized over these shift updates to calculate the log-likelihood. For each of the six experimental sessions (i, j), the log likelihood of a model M and its parameter set is the integral over and of the likelihood of the final shift updates given the observed data , the model M, and the parameter set , , multiplied by the joint probability of the auditory and visual shift updates, , summed across all six sessions (45)

We will describe in the following sections how the joint probability and the log-likelihood were derived for each of the three models of cross-modal recalibration.

Reliability-based model of cross-modal recalibration. In this model, auditory and visual shift updates have a constant ratio of , the ratio of the measurement noise variances. Therefore, can be rewritten as , and we can express the likelihood given a single auditory localization response as (46) where (47)

The visual response likelihoods and means are defined analogously (Eq 34). Thus, the joint likelihood can be written as . Given that the likelihood depends only on , we only need to integrate over and the log likelihood simplifies to (48)

The shift updates are stochastic because the visual and auditory measurements in each trial of the audiovisual recalibration task are stochastic. We cannot derive their probability distribution in closed form. Instead, we used Monte Carlo simulation to approximate this probability distribution. Given the reliability-based model, for each candidate set of parameters , visual-reliability condition i, and recalibration direction j, we simulated 120 recalibration trials analogous to the audiovisual recalibration task. We repeated this simulation 1,000 times, resulting in a sample of 1,000 shift updates () and checked whether the distribution of the 1,000 samples was well fit by a Gaussian with mean and standard deviation equal to the corresponding empirical parameters of the sampled distribution. To do so, we binned the simulated shift updates into 100 bins of equal size and computed the correlation between the observed and predicted number of samples per bin. The resulting value of R² was greater than 0.925 in all cases (Section S8 in S1 Appendix). The approximated probability distribution of the shift updates is denoted as .

We approximated the integral in Eq 48 by numerical integration over a region discretized into 100 bins. To ensure that we include enough of the tails of the probability distribution of the shift updates, we set the integration region to be three times larger than the range of the samples, and centered the integration region on that range. Thus, the lower bound, lb, is defined as lb = Δ_min − (Δ_max − Δ_min) and the upper bound is ub = Δ_max + (Δ_max − Δ_min). The numerical integration region was derived separately for each session. The log likelihood is: (49) where (50) with defined in Eq 47 and defined in Eq 34. and depend on the bias parameters a_A and b_A, as well as on , which depends on the measurement variances and given bimodal presentation and the common learning rate α. Note that and are not directly constrained by data from bimodal trials (because these trials were not included in the model fitting), but estimated based on their influence on the shift updates. Specifically, and affect the spread of the measurements, and as a consequence they influence the width of the predicted probability distribution of measurement-shift updates, which in turn affect the log likelihood of the model. The set of free parameters for this model is . was constrained to be a non-decreasing function of visual-reliability condition i, and and were constrained to be no greater than five times the average values of and across participants (Section S14 in S1 Appendix). The values of the T variables and the sums do not include outlier trials.

Log-likelihood—fixed-ratio model of cross-modal recalibration. In this model, auditory and visual shift updates have a fixed ratio of (Section S15 in S1 Appendix). Thus, we can express the likelihood for the fixed-ratio model and parameter set given an auditory localization response in a similar form to Eq 47: (51)

The approximation was generated in the same way as for the reliability-based model. The set of free parameters for this model is . Note that even though the shift updates in the fixed-ratio model do not depend on the stimulus reliabilities, the log-likelihood does due to the influence of stimulus reliability on the estimates in the localization task (Eq 51) and due to the influence of the spread of the simulated measurements on the spread of the estimated distribution of . As in the reliability-based model, was constrained to be a non-decreasing function of visual-reliability condition i, and and were constrained to be no greater than five times the average values of and across participants.

Log-likelihood—causal-inference model of cross-modal recalibration. For this model, the joint likelihood was truly two-dimensional. Thus, we approximated the joint probability of the auditory and visual shift updates, by drawing 1000 samples of shift-update pairs and compared the set of sample pairs to a 2-d Gaussian with the sample mean and covariance as parameters. We again tested whether the two-dimensional Gaussian distribution provided a good fit to the simulated density (defined as R² > 0.925). If the Gaussian fit was poor, we used a kernel density estimate (Gaussian kernel smoother with σ chosen automatically) of the distribution based on the 2-d density of the samples [77, 78]. Overall, the simulated auditory and visual shift updates were very well fit by a bivariate Gaussian, and we rarely used a kernel density estimate (Section S8 in S1 Appendix). We additionally used simulations to verify that our estimates of the partial model log-likelihood () had reasonably small bias (Section S8 in S1 Appendix).

For the causal-inference model, we approximate the log likelihood by numerical integration over a 2-dimensional region of Δ_A, Δ_V space discretized into 100x100 bins. The upper and lower bounds were determined for both dimensions in the same way as before. The log likelihood is: (52) where is defined analogously to the reliability-based model (see Eq 50) with the exception that and are defined in Eq 34. The set of free parameters used to fit the causal-inference model to the localization responses in the post-recalibration task is or .

Parameter estimation.

For each model, we approximated the set of parameters Θ₁ and Θ₂ that maximized the likelihood using the MATLAB function fmincon and Python SciPy.optimize [79], and approximated Θ₃ using the BADS toolbox [80]. To deal with the possibility that the returned parameter values might correspond to a local minimum, we ran BADS multiple times with different starting points, randomly chosen from a D-dimensional grid, where D is the number of free parameters in Θ₃ (see Table 1 for a summary of the free parameters for each model) and with three evenly spaced values chosen for each dimension. The final parameter estimates were those with the maximum likelihood across all runs of the fitting procedure.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Summary of model parameters in Θ₃.

https://doi.org/10.1371/journal.pcbi.1008877.t001

Model comparison.

To compare model performance quantitatively, we computed the Akaike information criterion (AIC) for all four models [64] and calculated relative model-comparison scores, Δ_AIC, which relate the AIC value of the best-fit model to that of each of the other models (a higher Δ_AIC value indicates stronger evidence for the best-fit model). Models with 0 < Δ_AIC < 2 are weakly supported; models with 4 < Δ_AIC < 7 have considerably less support; models with Δ_AIC > 10 have essentially no support [81].