Chapter Seven - What do neurons really want? The role of semantics in cortical representations
Section snippets
Assumptions and definitions
I focus here on triggering activity in the sense of firing rates defined as spike counts in short windows spanning tens of milliseconds. This is by no means the only or agreed upon relevant property of cortical neurons, there has been extensive discussion about neural codes (see for example, Kreiman, 2004). A neuron might contribute to representing information by firing only a few spikes at a precise time in concert with other spikes in the network. A neuron may also represent information by not
Neuronal responses in visual cortex, the classical view
The introduction of techniques to record the activity of neurons in the beginning of the last century led to decades of experiments interrogating neuronal responses to visual stimulation. The history of studying neuronal tuning properties in visual cortex is the history of visual stimuli. How do we investigate the feature preferences of a neuron in visual cortex? We need to decide which stimuli to use in the experiments. The central challenge in answering this question is that it is essentially
Computational models of ventral visual cortex
Inspired by neuroanatomy and neurophysiology, many investigators have developed computational models that capture the basic principles that progressively transform a pixel-like representation of inputs into complex features that can be linearly decoded to recognize objects (Deco & Rolls, 2004; DiCarlo et al., 2012; Fukushima, 1980; Mel, 1997; Olshausen, Anderson, & Van Essen, 1993; Riesenhuber & Poggio, 1999; Serre et al., 2007; Wallis & Rolls, 1997). More recently, this family of models has
Category-selective responses do not imply semantic encoding
In many typical neuroscience experiments, investigators may present images containing objects belonging to different categories (Desimone et al., 1984; Freedman, Riesenhuber, Poggio, & Miller, 2001; Hung et al., 2005; Kiani, Esteky, Mirpour, & Tanaka, 2007; Kourtzi & Connor, 2011; Kreiman, Koch, & Fried, 2000b; Liu et al., 2009; Logothetis & Sheinberg, 1996; Meyers, Freedman, Kreiman, Miller, & Poggio, 2008; Mormann et al., 2011; Quian Quiroga et al., 2005; Sigala & Logothetis, 2002; Sugase,
What are the preferred stimuli for visual neurons?
What do those fc units in Fig. 1A actually want? That is, what types of images would trigger high activation in those fc units? We know already from Fig. 1C that images of cells lead to high activation in fc unit 1, images of Labradors lead to high activation in fc unit 2, etc. Therefore, it seems reasonable to argue that fc unit 1 “wants” images of cells, fc unit 2 “wants” images of Labradors and so on. One might even go on to describe fc unit 2 as a “Labrador unit,” as we have been doing. But
Models versus real brains
These deep convolutional bottom-up computational models cast a doubt on claims about semantic encoding based on category-selective responses and provide a null hypothesis to compare against. Yet, these computational models are a far cry from real biological systems in all sorts of ways and therefore it is fair to question to what extent we can extrapolate conclusions from these computational models to the types of representations manifested by real neurons. Advocates of semantics would rightly
In search of abstraction in the brain
What type of experimental data would provide evidence in favor of abstract semantic information? Returning to the examples used in the definition of semantics, it would be nice to show neuronal responses that are similar for a tennis ball and a tennis racket and yet very different between a tennis ball and a lemon. In other words, it would be nice to show (i) images that have a similar visual appearance (e.g., a tennis ball and a lemon) and yet they trigger very different responses, and (ii)
Semantics and the least common sense
Common sense, or general semantic knowledge about the world, is hard to find. The definition of semantics including linguistic-like information, at least taken literally, suggests that we should be looking for a high level of abstraction, beyond what can be described by current visual object recognition models. One practical issue to tackle semantics is that it is difficult to study language in non-human animals. Strangely, there are even investigators that have claimed that language is unique
Data availability
All the code used to generate Figs. 1 and 2 is available for download from: http://klab.tch.harvard.edu/resources/Categorization_Semantics.html.
We cannot provide the images used in the experiments in Figs. 1 and 2. However, we provide the synset identification numbers, which can be used to freely download all the images from the following site: http://image-net.org/.
References (105)
- et al.
A sparse object coding scheme in area V4
Current Biology
(2011) - et al.
Transformation of shape information in the ventral pathway
Current Opinion in Neurobiology
(2007) - et al.
Representation of multiple, independent categories in the primate prefrontal cortex
Neuron
(2010) - et al.
How does the brain solve visual object recognition?
Neuron
(2012) - et al.
Medial axis shape coding in macaque inferotemporal cortex
Neuron
(2012) Neural coding: Computational and biophysical perspectives
Physics of Life Reviews
(2004)Single neuron approaches to human vision and memories
Current Opinion in Neurobiology
(2007)- et al.
Timing, timing, timing: Fast decoding of object information from intracranial field potentials in human visual cortex
Neuron
(2009) - et al.
Modulation of visual responses by behavioral state in mouse visual cortex
Neuron
(2010) - et al.
Sparse coding of sensory inputs
Current Opinion in Neurobiology
(2004)