Elsevier

NeuroImage

Volume 158, September 2017, Pages 356-370
NeuroImage

Invariance of surface color representations across illuminant changes in the human cortex

https://doi.org/10.1016/j.neuroimage.2017.06.079Get rights and content

Highlights

  • Physically realistic 3D renderings of surfaces under various illuminations.

  • Surface color representations in V1 and V4α are constant across illumination changes.

  • Increasing bias for encoding surface vs. illuminant color from early to higher areas.

Abstract

A central problem in color vision is that the light reaching the eye from a given surface can vary dramatically depending on the illumination. Despite this, our color percept, the brain's estimate of surface reflectance, remains remarkably stable. This phenomenon is called color constancy. Here we investigated which human brain regions represent surface color in a way that is invariant with respect to illuminant changes. We used physically realistic rendering methods to display natural yet abstract 3D scenes that were displayed under three distinct illuminants. The scenes embedded, in different conditions, surfaces that differed in their surface color (i.e. in their reflectance property). We used multivariate fMRI pattern analysis to probe neural coding of surface reflectance and illuminant, respectively. While all visual regions encoded surface color when viewed under the same illuminant, we found that only in V1 and V4α surface color representations were invariant to illumination changes. Along the visual hierarchy there was a gradient from V1 to V4α to increasingly encode surface color rather than illumination. Finally, effects of a stimulus manipulation on individual behavioral color constancy indices correlated with neural encoding of the illuminant in hV4. This provides neural evidence for the Equivalent Illuminant Model. Our results provide a principled characterization of color constancy mechanisms across the visual hierarchy, and demonstrate complementary contributions in early and late processing stages.

Introduction

Color constitutes a fundamental quality of visual experience, and supports a large variety of behavioral tasks (Mollon, 1989). However, the fact that the light reflected from surfaces depends both on the surface color (i.e. its reflectance) and the color of the incident light (Land and McCann, 1971) poses a challenging problem to the visual system. It is therefore impossible to know the reflectance of a surface without any knowledge of the illumination. As numerous psychophysical studies have documented, however, the perception of surface color is fairly robust even in the face of changes in illumination. This property of the visual system is referred to as “color constancy”. It is unclear how the human brain transforms the highly ambiguous incoming color signals to create surface color representations that are stable across illuminants. What factors at the neural level are involved in color constancy?

Early investigations demonstrated the involvement of area V4 in color perception in monkeys (Wild et al., 1985, Zeki, 1983) and inspired neuroimaging studies suggesting a similar role for human V4 (Bartels and Zeki, 2000, Beauchamp et al., 1999, Lueck et al., 1989). Also the functional organization of color responses in this area was shown to reflect perceptually relevant stimulus dimensions in both non-human (Conway and Tsao, 2009, Kusunoki et al., 2006, Li et al., 2014) and human primates (Brouwer and Heeger, 2013, Brouwer and Heeger, 2009).

Human lesion studies have accordingly implicated a connection between area V4 and achromatopsia (but also form vision deficits) (Bouvier and Engel, 2005). As for selective color constancy impairments, evidence appears less conclusive with some work suggesting a link with V4 (Clarke et al., 1998, Kennard et al., 1995) while different research highlights the involvement of other areas (Rüttiger et al., 1999), including V1 (Kentridge et al., 2007).

Human neuroimaging indeed shows increased responses to color already in early areas V1 and V2 (Bartels and Zeki, 2000, Beauchamp et al., 1999, Engel et al., 1997). But also in non-human primates, the chromatic context modulation of neural color tuning (Wachtler et al., 2003) and the double opponency of neurons (Conway, 2001, Conway and Livingstone, 2006, Johnson et al., 2008, Johnson et al., 2001) outline possible early color constancy mechanisms in V1.

While prior monkey studies shed some light on neural responses encoding perceptual (i.e. color constant) versus physical color properties in isolated regions, to our knowledge no prior study manipulated reflectance and illumination spectra to examine regional encoding of perceptually constant colors or of illuminant systematically across the whole ventral visual pathway, neither in monkey nor human brains.

We used multi-voxel pattern analysis as a test for color constancy: if a neural surface color representation is invariant across illumination changes, distinctions between representations of different surface colors should generalize across these changes. Using physically realistic rendering methods we showed subjects 3D scenes rendered in three conditions that simulated three different illuminants. Surfaces that differed in their surface color were embedded in these scenes. We designed our experiment in this way to achieve higher ecological validity: color vision in real life occurs in 3D environments, and it is well established that some surface color cues depend on 3D scene structure (Maloney, 1999, Radonjic et al., 2015). Participants performed an attention task (that was independent of illuminant or color) on these surfaces during fMRI recording.

To test our hypotheses, we trained classifiers to discriminate BOLD responses to two surfaces (“blue” and “yellow”) under two out of three illuminations (e.g. “neutral”, and “blue”) and tested them on new BOLD responses measured in the third illumination condition (e.g. “yellow”), which was not part of the training set. This analysis showed that activity patterns in V1 and V4α encoded surface colors in a way that generalized across illumination conditions, i.e. in a color-constant way.

Furthermore, we tested a prediction from equivalent illuminant models of color constancy. We hypothesized that the neural accuracy of encoding the illuminant of a scene predicts the behavioral accuracy of constant color perception. We collected behavioral color constancy indices, including a cue conflict manipulation that abolishes behavioral color constancy. We collected fMRI data for the same stimuli. In accord with the equivalent illuminant model, we found that the behavioral effect of the cue conflict manipulation on color constancy could be predicted from neural decoding of the illuminant in hV4.

Lastly, we examined how visual areas interpret two different surfaces that reflect the same light because they are presented under different illuminations. These surfaces were perceived as having distinct colors, but emitted the same light. These stimuli can be discriminated on the basis of their surface reflectance or illumination. Our analysis revealed that higher visual regions hV4, VO1, V4α weighted the difference in surface reflectance more strongly than earlier visual areas.

In sum, the results provide a detailed account of the contributions of different visual areas to color constancy.

Section snippets

Participants

Our sample consisted of 20 healthy observers (15 female, 5 male) from the Tübingen University community between the ages of 19–35 (mean age: 24.5). All participants had normal color vision as ascertained with Ishihara plates (Ishihara, 2011). They provided written informed consent to participation in the experiment prior to the first session. The local ethics committee of the University Hospital Tübingen approved the study. Data from the fMRI main experiment of subject 12 could not be analyzed

Results

We rendered a complex scene containing four surfaces that appeared either blue or yellow. Three different illuminations were simulated: neutral D65, a blue illumination, and a yellow illumination (Fig. 1a, b, c). Additionally, we introduced a reduced-cue condition for the blue (Fig. 1d) and yellow (Fig. 1e) illumination conditions: in these conditions the background square on which the colored surfaces appeared was replaced with a surface that was chosen such that the light reflected from it

Discussion

The present study is the first, to our knowledge, to investigate fMRI brain responses to surface color that was perceived as constant during illuminant changes, and to relate behavioral color constancy to neural estimates of illumination. We found that the earliest cortical stage, V1, as well as one of the most anterior color-responsive regions in the fusiform gyrus, pV4α, encode color invariantly with respect to the illuminant. We also found that there is a gradient from early cortex to

Conclusion

The present study adds an important new piece to the puzzle of human color vision. Experimental approaches seeking to discover isomorphic mappings between perceptual and neural color spaces have found area V4 to be involved in color perception (Brouwer and Heeger, 2009, Li et al., 2014). In the present study we examined two central components of color constancy in the human brain, namely the robustness of neural encoding of surface reflectance during changes in illumination, and the neural

Acknowledgements

This work was supported by a grant from the German Federal Ministry for Education and Research (BMBF), grant number: FKZ 01GQ1002, by the German Excellence Initiative of the German Research Foundation (DFG) grant number EXC307, and by the Max Planck Society, Germany. The authors declare no competing financial interests.

References (80)

  • J. Poort et al.

    The role of attention in figure-ground segregation in areas V1 and V4 of the visual cortex

    Neuron

    (2012)
  • A.W. Roe et al.

    Toward a unified theory of visual area V4

    Neuron

    (2012)
  • K.J. Seymour et al.

    The coding of color, motion, and their conjunction in the human visual cortex

    Curr. Biol.

    (2009)
  • R. Shapley et al.

    Color in the cortex: single- and double-opponent cells

    Vis. Res.

    (2011)
  • T. Wachtler et al.

    Representation of color stimuli in awake macaque primary visual cortex

    Neuron

    (2003)
  • S. Zeki

    Colour coding in the cerebral cortex: the reaction of cells in monkey visual cortex to wavelengths and colours

    Neuroscience

    (1983)
  • A. Bartels et al.

    The architecture of the colour centre in the human visual brain: new results and a review

    Eur. J. Neurosci.

    (2000)
  • M.S. Beauchamp et al.

    An fMRI version of the Farnsworth-Munsell 100-Hue test reveals multiple color-selective areas in human ventral occipitotemporal cortex

    Cereb. Cortex

    (1999)
  • R.C. Blair et al.

    An alternative method for significance testing of waveform difference potentials

    Psychophysiology

    (1993)
  • S.E. Bouvier et al.

    Activity in visual area V4 correlates with surface perception

    J. Vis.

    (2008)
  • S.E. Bouvier et al.

    Behavioral deficits and cortical damage loci in cerebral achromatopsia

    Cereb. Cortex

    (2005)
  • D.H. Brainard

    Color constancy in the nearly natural image. 2. Achromatic loci

    J. Opt. Soc. Am. A. Opt. Image Sci. Vis.

    (1998)
  • D.H. Brainard et al.

    Surface color perception and equivalent illumination models

    J. Vis.

    (2011)
  • J. Brooks

    Counterbalancing for serial order carryover effects in experimental condition orders

    Psychol. Methods

    (2012)
  • G.J. Brouwer et al.

    Categorical clustering of the neural representation of color

    J. Neurosci.

    (2013)
  • G.J. Brouwer et al.

    Decoding and reconstructing color from responses in human visual cortex

    J. Neurosci.

    (2009)
  • S. Clarke et al.

    Colour constancy impairments in patients with lesions of the prestriate cortex

    Exp. Brain Res.

    (1998)
  • B.R. Conway

    Color signals through dorsal and ventral visual pathways

    Vis. Neurosci.

    (2013)
  • B.R. Conway

    Spatial structure of cone inputs to color cells in alert macaque primary visual cortex (V-1)

    J. Neurosci.

    (2001)
  • B.R. Conway et al.

    Spatial and temporal properties of cone signals in alert macaque primary visual cortex

    J. Neurosci.

    (2006)
  • B.R. Conway et al.

    Color-tuned neurons are spatially clustered according to color preference within alert macaque posterior inferior temporal cortex

    Proc. Natl. Acad. Sci.

    (2009)
  • M.A. Cox et al.

    Receptive field focus of visual area V4 neurons determines responses to illusory surfaces

    Proc. Natl. Acad. Sci.

    (2013)
  • P.B. Delahunt et al.

    Does human color constancy incorporate the statistical regularity of natural daylight?

    J. Vis.

    (2004)
  • S. Engel et al.

    Colour tuning in human visual cortex measured with functional magnetic resonance imaging

    Nature

    (1997)
  • R.-E. Fan et al.

    LIBLINEAR: a library for large linear classification

    J. Mach. Learn. Res.

    (2008)
  • D.J. Felleman et al.

    Distributed hierarchical processing in the primate cerebral cortex

    Cereb. Cortex

    (1991)
  • H.S. Friedman et al.

    The coding of uniform colour figures in monkey visual cortex

    J. Physiol.

    (2003)
  • J.-D. Haynes et al.

    Eye-specific effects of binocular rivalry in the human lateral geniculate nucleus

    Nature

    (2005)
  • B.S. Heasly et al.

    RenderToolbox3: MATLAB tools that facilitate physically based stimulus rendering for vision research

    J. Vis.

    (2014)
  • P.-J. Hsieh et al.

    “Brain-reading” of perceived colors reveals a feature mixing mechanism underlying perceptual filling-in in cortical area V1

    Hum. Brain Mapp.

    (2010)
  • Cited by (12)

    • The neural coding of face and body orientation in occipitotemporal cortex

      2022, NeuroImage
      Citation Excerpt :

      Each block lasted 6 s followed by 2 s fixation before the next block began. This block length was used to increase fMRI signal by blocking the stimuli whilst ensuring runs did not become too long, and has been successfully used in previous fMRI MVPA paradigms (e.g. Bannert and Bartels, 2017, 2013; Foster et al., 2019). Each run contained 54 blocks (3 repetitions per condition).

    • fMRI representational similarity analysis reveals graded preferences for chromatic and achromatic stimulus contrast across human visual cortex

      2020, NeuroImage
      Citation Excerpt :

      RSA, like classifier accuracy, will be adversely affected by low signal strength, but since responses in each region are correlated with multiple models the relative performance of these models can be compared across ROIs. RSA was first developed in vision for the understanding of object processing (Kriegeskorte et al., 2008), and has been applied in a range of predominantly high-level visual and cognitive functions, including memory, semantics and emotion (Borghesani et al., 2016; Skerry and Saxe, 2015), but has been little used for lower-level vision or colour vision (Bannert and Bartels, 2017, 2013; Bird et al., 2014; Goddard et al., 2017; Goddard and Mullen, 2019; Salmela et al., 2016; Wardle et al., 2016). Furthermore, in the visual object literature, it is increasingly clear that stimulus differences along lower-level feature dimensions may contribute to effects that have been attributed to ‘high-level’ feature coding (Andrews et al., 2015).

    • The response to colour in the human visual cortex: the fMRI approach

      2019, Current Opinion in Behavioral Sciences
      Citation Excerpt :

      Goddard and Mullen [61] have used this approach to compare the relevance of different low-level stimulus attributes (colour, achromatic form, flicker) in driving discrimination across the visual hierarchies and found that while the voxel patterns in the early visual areas had more differential information about achromatic form, voxel patterns in areas V4, VO1 and VO2 of the ventral pathway had relatively more information about colour. Bannert and Bartels [60] applied an RSA to colour representation and the problem of colour constancy, comparing two models, and found evidence that the voxels patterns in higher ventral areas, including V4 and VO1, carry more information about surface colour than the illuminant, interpreted as a shift towards colour constancy in these areas. Taken together, fMRI studies provide strong evidence for a specialization for colour processing developing in V4 and progressing through to VO1 and VO2 of the human ventral visual cortex.

    • Exploring the functional nature of synaesthetic colour: Dissociations from colour perception and imagery

      2018, Cognition
      Citation Excerpt :

      This resemblance suggests that either synaesthetic colour is represented in retinotopic visual cortices but additionally involves cortical sections outside the inducer’s corresponding retinotopic area or, perhaps more plausibly, that the core neural substrates of synaesthetic colour are beyond the realm of retinotopic cortex, perhaps in high-level regions for semantic knowledge about object colour (e.g., the anterior temporal lobe, see Chiou & Lambon Ralph, 2016; Chiou & Rich, 2014; Chiou, Sowman, Etchell, & Rich, 2014). While there are multiple cortical subregions of the ventral occipitotemporal cortex that are colour-sensitive, evidence from both fMRI and single-unit recording has shown that area V4 is particularly crucial for maintaining colour constancy across various illuminant conditions, making colour perception immune to changes of local spectral composition (Bannert & Bartels, 2017; Foster, 2011). Its pivotal role in normal colour perception has led synaesthesia researchers to examine if it similarly underlies synaesthetic colour.

    • Mental imagery: Philosophy, psychology, neuroscience

      2023, Mental Imagery: Philosophy, Psychology, Neuroscience
    View all citing articles on Scopus
    View full text