Chapter Seven - What do neurons really want? The role of semantics in cortical representations

https://doi.org/10.1016/bs.plm.2019.03.005Get rights and content

Abstract

What visual inputs best trigger activity for a given neuron in cortex and what type of semantic information may guide those neuronal responses? We revisit the methodologies used so far to design visual experiments, and what those methodologies have taught us about neural coding in visual cortex. Despite heroic and seminal work in ventral visual cortex, we still do not know what types of visual features are optimal for cortical neurons. We briefly review state-of-the-art standard models of visual recognition and argue that such models should constitute the null hypothesis for any measurement that purports to ascribe semantic meaning to neuronal responses. While it remains unclear when, where, and how abstract semantic information is incorporated in visual neurophysiology, there exists clear evidence of top-down modulation in the form of attention, task-modulation and expectations. Such top-down signals open the doors to some of the most exciting questions today toward elucidating how abstract knowledge can be incorporated into our models of visual processing.

Section snippets

Assumptions and definitions

I focus here on triggering activity in the sense of firing rates defined as spike counts in short windows spanning tens of milliseconds. This is by no means the only or agreed upon relevant property of cortical neurons, there has been extensive discussion about neural codes (see for example, Kreiman, 2004). A neuron might contribute to representing information by firing only a few spikes at a precise time in concert with other spikes in the network. A neuron may also represent information by not

Neuronal responses in visual cortex, the classical view

The introduction of techniques to record the activity of neurons in the beginning of the last century led to decades of experiments interrogating neuronal responses to visual stimulation. The history of studying neuronal tuning properties in visual cortex is the history of visual stimuli. How do we investigate the feature preferences of a neuron in visual cortex? We need to decide which stimuli to use in the experiments. The central challenge in answering this question is that it is essentially

Computational models of ventral visual cortex

Inspired by neuroanatomy and neurophysiology, many investigators have developed computational models that capture the basic principles that progressively transform a pixel-like representation of inputs into complex features that can be linearly decoded to recognize objects (Deco & Rolls, 2004; DiCarlo et al., 2012; Fukushima, 1980; Mel, 1997; Olshausen, Anderson, & Van Essen, 1993; Riesenhuber & Poggio, 1999; Serre et al., 2007; Wallis & Rolls, 1997). More recently, this family of models has

Category-selective responses do not imply semantic encoding

In many typical neuroscience experiments, investigators may present images containing objects belonging to different categories (Desimone et al., 1984; Freedman, Riesenhuber, Poggio, & Miller, 2001; Hung et al., 2005; Kiani, Esteky, Mirpour, & Tanaka, 2007; Kourtzi & Connor, 2011; Kreiman, Koch, & Fried, 2000b; Liu et al., 2009; Logothetis & Sheinberg, 1996; Meyers, Freedman, Kreiman, Miller, & Poggio, 2008; Mormann et al., 2011; Quian Quiroga et al., 2005; Sigala & Logothetis, 2002; Sugase,

What are the preferred stimuli for visual neurons?

What do those fc units in Fig. 1A actually want? That is, what types of images would trigger high activation in those fc units? We know already from Fig. 1C that images of cells lead to high activation in fc unit 1, images of Labradors lead to high activation in fc unit 2, etc. Therefore, it seems reasonable to argue that fc unit 1 “wants” images of cells, fc unit 2 “wants” images of Labradors and so on. One might even go on to describe fc unit 2 as a “Labrador unit,” as we have been doing. But

Models versus real brains

These deep convolutional bottom-up computational models cast a doubt on claims about semantic encoding based on category-selective responses and provide a null hypothesis to compare against. Yet, these computational models are a far cry from real biological systems in all sorts of ways and therefore it is fair to question to what extent we can extrapolate conclusions from these computational models to the types of representations manifested by real neurons. Advocates of semantics would rightly

In search of abstraction in the brain

What type of experimental data would provide evidence in favor of abstract semantic information? Returning to the examples used in the definition of semantics, it would be nice to show neuronal responses that are similar for a tennis ball and a tennis racket and yet very different between a tennis ball and a lemon. In other words, it would be nice to show (i) images that have a similar visual appearance (e.g., a tennis ball and a lemon) and yet they trigger very different responses, and (ii)

Semantics and the least common sense

Common sense, or general semantic knowledge about the world, is hard to find. The definition of semantics including linguistic-like information, at least taken literally, suggests that we should be looking for a high level of abstraction, beyond what can be described by current visual object recognition models. One practical issue to tackle semantics is that it is difficult to study language in non-human animals. Strangely, there are even investigators that have claimed that language is unique

Data availability

All the code used to generate Figs. 1 and 2 is available for download from: http://klab.tch.harvard.edu/resources/Categorization_Semantics.html.

We cannot provide the images used in the experiments in Figs. 1 and 2. However, we provide the synset identification numbers, which can be used to freely download all the images from the following site: http://image-net.org/.

References (105)

  • J.H. Reynolds et al.

    The normalization model of attention

    Neuron

    (2009)
  • T. Serre et al.

    A quantitative theory of immediate visual recognition

    Progress in Brain Research

    (2007)
  • S. Vaziri et al.

    Representation of gravity-aligned scene structure in ventral pathway visual cortex

    Current Biology

    (2016)
  • T. Allison et al.

    Face recognition in human extrastriate cortex

    Journal of Neurophysiology

    (1994)
  • R. Berwick et al.

    Why only us: Language and evolution

    (2015)
  • B. Chapman et al.

    Development of orientation preference maps in ferret primary visual cortex

    Journal of Neuroscience

    (1996)
  • T. Coogan et al.

    Hierarchical organization of areas in rat visual cortex

    The Journal of Neuroscience

    (1993)
  • G. Deco et al.

    Computational neuroscience of vision

    (2004)
  • R. Desimone et al.

    Stimulus-selective properties of inferior temporal neurons in the macaque

    Journal of Neuroscience

    (1984)
  • R. Desimone et al.

    Neural mechanisms of selective visual attention

    Annual Review of Neuroscience

    (1995)
  • A.K. Engel et al.

    Invasive recordings from the human brain: Clinical insights and beyond

    Nature Reviews. Neuroscience

    (2005)
  • E.N. Eskandar et al.

    Role of inferior temporal neurons in visual memory. I. Temporal encoding of information about visual images, recalled images, and behavioral context

    Journal of Neurophysiology

    (1992)
  • M. Fabre-Thorpe et al.

    Rapid categorization of natural images by rhesus monkeys

    Neuroreport

    (1998)
  • D.J. Felleman et al.

    Distributed hierarchical processing in the primate cerebral cortex

    Cerebral Cortex

    (1991)
  • D. Freedman et al.

    Categorical representation of visual stimuli in the primate prefrontal cortex

    Science

    (2001)
  • J. Freeman et al.

    A functional and perceptual signature of the second visual area in primates

    Nature Neuroscience

    (2013)
  • I. Fried et al.

    Single neuron studies of the human brain. Probing cognition

    (2014)
  • K. Fukushima

    Neocognitron: A self organizing neural network model for a mechanism of pattern recognition unaffected by shift in position

    Biological Cybernetics

    (1980)
  • J.L. Gallant et al.

    Selectivity for polar, hyperbolic, and Cartesian gratings in macaque visual cortex

    Science

    (1993)
  • H. Gelbard-Sagiv et al.

    Internally generated reactivation of single neurons in human Hippocampus during free recall

    Science

    (2008)
  • G.M. Ghose et al.

    Spatial summation can explain the attentional modulation of neuronal responses to multiple stimuli in area V4

    Journal of Neuroscience

    (2008)
  • C.D. Gilbert et al.

    Top-down influences on visual processing

    Nature Reviews. Neuroscience

    (2013)
  • C.G. Gross

    How inferior temporal cortex became a visual area

    Cerebral Cortex

    (1994)
  • K. He et al.

    Mask R-CNN

  • K. He et al.

    Deep residual learning for image recognition

    arXiv

    (2015)
  • J. Hegde et al.

    A comparative study of shape representation in macaque visual areas v2 and v4

    Cerebral Cortex

    (2007)
  • S. Higuchi et al.

    Formation of mnemonic neuronal responses to visual paired associates in inferotemporal cortex is impaired by perirhinal and entorhinal lesions

    PNAS

    (1996)
  • D. Hubel

    Evolution of ideas on the primary visual cortex, 1955–1978: A biased historical account

  • D.H. Hubel et al.

    Receptive fields, binocular interaction and functional architecture in the cat's visual cortex

    The Journal of Physiology

    (1962)
  • D.H. Hubel et al.

    Receptive fields and functional architecture of monkey striate cortex

    The Journal of Physiology

    (1968)
  • C.P. Hung et al.

    Fast read-out of object identity from macaque inferior temporal cortex

    Science

    (2005)
  • L. Isik et al.

    What is changing when: Decoding visual information in movies from human intracranial recordings

    NeuroImage

    (2017)
  • J.P. Jones et al.

    The two-dimensional spectral structure of simple receptive fields in cat striate cortex

    Journal of Neurophysiology

    (1987)
  • N. Kanwisher et al.

    The fusiform face area: A module in human extrastriate cortex specialized for face perception

    Journal of Neuroscience

    (1997)
  • C. Keysers et al.

    The speed of sight

    Journal of Cognitive Neuroscience

    (2001)
  • R. Kiani et al.

    Object category structure in response patterns of neuronal population in monkey inferior temporal cortex

    Journal of Neurophysiology

    (2007)
  • E. Kobatake et al.

    Neuronal selectivities to complex object features in the ventral visual pathway of the macaque cerebral cortex

    Journal of Neurophysiology

    (1994)
  • C. Koch

    Biophysics of computation

    (1999)
  • Z. Kourtzi et al.

    Neural representations for object perception: Structure, category, and adaptive coding

    Annual Review of Neuroscience

    (2011)
  • G. Kreiman

    On the neuronal activity in the human brain during visual recognition, imagery and binocular rivalry

  • Cited by (0)

    View full text