Invariance of surface color representations across illuminant changes in the human cortex
Introduction
Color constitutes a fundamental quality of visual experience, and supports a large variety of behavioral tasks (Mollon, 1989). However, the fact that the light reflected from surfaces depends both on the surface color (i.e. its reflectance) and the color of the incident light (Land and McCann, 1971) poses a challenging problem to the visual system. It is therefore impossible to know the reflectance of a surface without any knowledge of the illumination. As numerous psychophysical studies have documented, however, the perception of surface color is fairly robust even in the face of changes in illumination. This property of the visual system is referred to as “color constancy”. It is unclear how the human brain transforms the highly ambiguous incoming color signals to create surface color representations that are stable across illuminants. What factors at the neural level are involved in color constancy?
Early investigations demonstrated the involvement of area V4 in color perception in monkeys (Wild et al., 1985, Zeki, 1983) and inspired neuroimaging studies suggesting a similar role for human V4 (Bartels and Zeki, 2000, Beauchamp et al., 1999, Lueck et al., 1989). Also the functional organization of color responses in this area was shown to reflect perceptually relevant stimulus dimensions in both non-human (Conway and Tsao, 2009, Kusunoki et al., 2006, Li et al., 2014) and human primates (Brouwer and Heeger, 2013, Brouwer and Heeger, 2009).
Human lesion studies have accordingly implicated a connection between area V4 and achromatopsia (but also form vision deficits) (Bouvier and Engel, 2005). As for selective color constancy impairments, evidence appears less conclusive with some work suggesting a link with V4 (Clarke et al., 1998, Kennard et al., 1995) while different research highlights the involvement of other areas (Rüttiger et al., 1999), including V1 (Kentridge et al., 2007).
Human neuroimaging indeed shows increased responses to color already in early areas V1 and V2 (Bartels and Zeki, 2000, Beauchamp et al., 1999, Engel et al., 1997). But also in non-human primates, the chromatic context modulation of neural color tuning (Wachtler et al., 2003) and the double opponency of neurons (Conway, 2001, Conway and Livingstone, 2006, Johnson et al., 2008, Johnson et al., 2001) outline possible early color constancy mechanisms in V1.
While prior monkey studies shed some light on neural responses encoding perceptual (i.e. color constant) versus physical color properties in isolated regions, to our knowledge no prior study manipulated reflectance and illumination spectra to examine regional encoding of perceptually constant colors or of illuminant systematically across the whole ventral visual pathway, neither in monkey nor human brains.
We used multi-voxel pattern analysis as a test for color constancy: if a neural surface color representation is invariant across illumination changes, distinctions between representations of different surface colors should generalize across these changes. Using physically realistic rendering methods we showed subjects 3D scenes rendered in three conditions that simulated three different illuminants. Surfaces that differed in their surface color were embedded in these scenes. We designed our experiment in this way to achieve higher ecological validity: color vision in real life occurs in 3D environments, and it is well established that some surface color cues depend on 3D scene structure (Maloney, 1999, Radonjic et al., 2015). Participants performed an attention task (that was independent of illuminant or color) on these surfaces during fMRI recording.
To test our hypotheses, we trained classifiers to discriminate BOLD responses to two surfaces (“blue” and “yellow”) under two out of three illuminations (e.g. “neutral”, and “blue”) and tested them on new BOLD responses measured in the third illumination condition (e.g. “yellow”), which was not part of the training set. This analysis showed that activity patterns in V1 and V4α encoded surface colors in a way that generalized across illumination conditions, i.e. in a color-constant way.
Furthermore, we tested a prediction from equivalent illuminant models of color constancy. We hypothesized that the neural accuracy of encoding the illuminant of a scene predicts the behavioral accuracy of constant color perception. We collected behavioral color constancy indices, including a cue conflict manipulation that abolishes behavioral color constancy. We collected fMRI data for the same stimuli. In accord with the equivalent illuminant model, we found that the behavioral effect of the cue conflict manipulation on color constancy could be predicted from neural decoding of the illuminant in hV4.
Lastly, we examined how visual areas interpret two different surfaces that reflect the same light because they are presented under different illuminations. These surfaces were perceived as having distinct colors, but emitted the same light. These stimuli can be discriminated on the basis of their surface reflectance or illumination. Our analysis revealed that higher visual regions hV4, VO1, V4α weighted the difference in surface reflectance more strongly than earlier visual areas.
In sum, the results provide a detailed account of the contributions of different visual areas to color constancy.
Section snippets
Participants
Our sample consisted of 20 healthy observers (15 female, 5 male) from the Tübingen University community between the ages of 19–35 (mean age: 24.5). All participants had normal color vision as ascertained with Ishihara plates (Ishihara, 2011). They provided written informed consent to participation in the experiment prior to the first session. The local ethics committee of the University Hospital Tübingen approved the study. Data from the fMRI main experiment of subject 12 could not be analyzed
Results
We rendered a complex scene containing four surfaces that appeared either blue or yellow. Three different illuminations were simulated: neutral D65, a blue illumination, and a yellow illumination (Fig. 1a, b, c). Additionally, we introduced a reduced-cue condition for the blue (Fig. 1d) and yellow (Fig. 1e) illumination conditions: in these conditions the background square on which the colored surfaces appeared was replaced with a surface that was chosen such that the light reflected from it
Discussion
The present study is the first, to our knowledge, to investigate fMRI brain responses to surface color that was perceived as constant during illuminant changes, and to relate behavioral color constancy to neural estimates of illumination. We found that the earliest cortical stage, V1, as well as one of the most anterior color-responsive regions in the fusiform gyrus, pV4α, encode color invariantly with respect to the illuminant. We also found that there is a gradient from early cortex to
Conclusion
The present study adds an important new piece to the puzzle of human color vision. Experimental approaches seeking to discover isomorphic mappings between perceptual and neural color spaces have found area V4 to be involved in color perception (Brouwer and Heeger, 2009, Li et al., 2014). In the present study we examined two central components of color constancy in the human brain, namely the robustness of neural encoding of surface reflectance during changes in illumination, and the neural
Acknowledgements
This work was supported by a grant from the German Federal Ministry for Education and Research (BMBF), grant number: FKZ 01GQ1002, by the German Excellence Initiative of the German Research Foundation (DFG) grant number EXC307, and by the Max Planck Society, Germany. The authors declare no competing financial interests.
References (80)
- et al.
Decoding the yellow of a gray banana
Curr. Biol.
(2013) - et al.
Colour constancy and conscious perception of changes of illuminant
Neuropsychologia
(2008) - et al.
fMRI and its interpretations: an illustration on directional selectivity in area V5/MT
Trends Neurosci.
(2008) - et al.
Attention to stimulus features shifts spectral tuning of V4 neurons during natural vision
Neuron
(2008) - et al.
The many colours of ‘the dress’
Curr. Biol.
(2015) Colour vision: primary visual cortex shows its influence
Curr. Biol.
(2003)- et al.
Attraction of position preference by spatial attention throughout human visual cortex
Neuron
(2014) - et al.
Feature-based attention in visual cortex
Trends Neurosci.
(2006) - et al.
Network interactions: non-geniculate input to V1
Curr. Opin. Neurobiol.
(2013) - et al.
Perception matches selectivity in the human anterior color center
Curr. Biol.
(2008)
The role of attention in figure-ground segregation in areas V1 and V4 of the visual cortex
Neuron
Toward a unified theory of visual area V4
Neuron
The coding of color, motion, and their conjunction in the human visual cortex
Curr. Biol.
Color in the cortex: single- and double-opponent cells
Vis. Res.
Representation of color stimuli in awake macaque primary visual cortex
Neuron
Colour coding in the cerebral cortex: the reaction of cells in monkey visual cortex to wavelengths and colours
Neuroscience
The architecture of the colour centre in the human visual brain: new results and a review
Eur. J. Neurosci.
An fMRI version of the Farnsworth-Munsell 100-Hue test reveals multiple color-selective areas in human ventral occipitotemporal cortex
Cereb. Cortex
An alternative method for significance testing of waveform difference potentials
Psychophysiology
Activity in visual area V4 correlates with surface perception
J. Vis.
Behavioral deficits and cortical damage loci in cerebral achromatopsia
Cereb. Cortex
Color constancy in the nearly natural image. 2. Achromatic loci
J. Opt. Soc. Am. A. Opt. Image Sci. Vis.
Surface color perception and equivalent illumination models
J. Vis.
Counterbalancing for serial order carryover effects in experimental condition orders
Psychol. Methods
Categorical clustering of the neural representation of color
J. Neurosci.
Decoding and reconstructing color from responses in human visual cortex
J. Neurosci.
Colour constancy impairments in patients with lesions of the prestriate cortex
Exp. Brain Res.
Color signals through dorsal and ventral visual pathways
Vis. Neurosci.
Spatial structure of cone inputs to color cells in alert macaque primary visual cortex (V-1)
J. Neurosci.
Spatial and temporal properties of cone signals in alert macaque primary visual cortex
J. Neurosci.
Color-tuned neurons are spatially clustered according to color preference within alert macaque posterior inferior temporal cortex
Proc. Natl. Acad. Sci.
Receptive field focus of visual area V4 neurons determines responses to illusory surfaces
Proc. Natl. Acad. Sci.
Does human color constancy incorporate the statistical regularity of natural daylight?
J. Vis.
Colour tuning in human visual cortex measured with functional magnetic resonance imaging
Nature
LIBLINEAR: a library for large linear classification
J. Mach. Learn. Res.
Distributed hierarchical processing in the primate cerebral cortex
Cereb. Cortex
The coding of uniform colour figures in monkey visual cortex
J. Physiol.
Eye-specific effects of binocular rivalry in the human lateral geniculate nucleus
Nature
RenderToolbox3: MATLAB tools that facilitate physically based stimulus rendering for vision research
J. Vis.
“Brain-reading” of perceived colors reveals a feature mixing mechanism underlying perceptual filling-in in cortical area V1
Hum. Brain Mapp.
Cited by (12)
The neural coding of face and body orientation in occipitotemporal cortex
2022, NeuroImageCitation Excerpt :Each block lasted 6 s followed by 2 s fixation before the next block began. This block length was used to increase fMRI signal by blocking the stimuli whilst ensuring runs did not become too long, and has been successfully used in previous fMRI MVPA paradigms (e.g. Bannert and Bartels, 2017, 2013; Foster et al., 2019). Each run contained 54 blocks (3 repetitions per condition).
fMRI representational similarity analysis reveals graded preferences for chromatic and achromatic stimulus contrast across human visual cortex
2020, NeuroImageCitation Excerpt :RSA, like classifier accuracy, will be adversely affected by low signal strength, but since responses in each region are correlated with multiple models the relative performance of these models can be compared across ROIs. RSA was first developed in vision for the understanding of object processing (Kriegeskorte et al., 2008), and has been applied in a range of predominantly high-level visual and cognitive functions, including memory, semantics and emotion (Borghesani et al., 2016; Skerry and Saxe, 2015), but has been little used for lower-level vision or colour vision (Bannert and Bartels, 2017, 2013; Bird et al., 2014; Goddard et al., 2017; Goddard and Mullen, 2019; Salmela et al., 2016; Wardle et al., 2016). Furthermore, in the visual object literature, it is increasingly clear that stimulus differences along lower-level feature dimensions may contribute to effects that have been attributed to ‘high-level’ feature coding (Andrews et al., 2015).
The response to colour in the human visual cortex: the fMRI approach
2019, Current Opinion in Behavioral SciencesCitation Excerpt :Goddard and Mullen [61] have used this approach to compare the relevance of different low-level stimulus attributes (colour, achromatic form, flicker) in driving discrimination across the visual hierarchies and found that while the voxel patterns in the early visual areas had more differential information about achromatic form, voxel patterns in areas V4, VO1 and VO2 of the ventral pathway had relatively more information about colour. Bannert and Bartels [60] applied an RSA to colour representation and the problem of colour constancy, comparing two models, and found evidence that the voxels patterns in higher ventral areas, including V4 and VO1, carry more information about surface colour than the illuminant, interpreted as a shift towards colour constancy in these areas. Taken together, fMRI studies provide strong evidence for a specialization for colour processing developing in V4 and progressing through to VO1 and VO2 of the human ventral visual cortex.
A tour of contemporary color vision research
2018, Vision ResearchExploring the functional nature of synaesthetic colour: Dissociations from colour perception and imagery
2018, CognitionCitation Excerpt :This resemblance suggests that either synaesthetic colour is represented in retinotopic visual cortices but additionally involves cortical sections outside the inducer’s corresponding retinotopic area or, perhaps more plausibly, that the core neural substrates of synaesthetic colour are beyond the realm of retinotopic cortex, perhaps in high-level regions for semantic knowledge about object colour (e.g., the anterior temporal lobe, see Chiou & Lambon Ralph, 2016; Chiou & Rich, 2014; Chiou, Sowman, Etchell, & Rich, 2014). While there are multiple cortical subregions of the ventral occipitotemporal cortex that are colour-sensitive, evidence from both fMRI and single-unit recording has shown that area V4 is particularly crucial for maintaining colour constancy across various illuminant conditions, making colour perception immune to changes of local spectral composition (Bannert & Bartels, 2017; Foster, 2011). Its pivotal role in normal colour perception has led synaesthesia researchers to examine if it similarly underlies synaesthetic colour.
Mental imagery: Philosophy, psychology, neuroscience
2023, Mental Imagery: Philosophy, Psychology, Neuroscience