Improved object recognition using neural networks trained to mimic the brain’s statistical properties

Abstract

The current state-of-the-art object recognition algorithms, deep convolutional neural networks (DCNNs), are inspired by the architecture of the mammalian visual system, and are capable of human-level performance on many tasks. As they are trained for object recognition tasks, it has been shown that DCNNs develop hidden representations that resemble those observed in the mammalian visual system (Razavi and Kriegeskorte, 2014; Yamins and Dicarlo, 2016; Gu and van Gerven, 2015; Mcclure and Kriegeskorte, 2016). Moreover, DCNNs trained on object recognition tasks are currently among the best models we have of the mammalian visual system. This led us to hypothesize that teaching DCNNs to achieve even more brain-like representations could improve their performance. To test this, we trained DCNNs on a composite task, wherein networks were trained to: (a) classify images of objects; while (b) having intermediate representations that resemble those observed in neural recordings from monkey visual cortex. Compared with DCNNs trained purely for object categorization, DCNNs trained on the composite task had better object recognition performance and are more robust to label corruption. Interestingly, we found that neural data was not required for this process, but randomized data with the same statistical properties as neural data also boosted performance. While the performance gains we observed when training on the composite task vs the “pure” object recognition task were modest, they were remarkably robust. Notably, we observed these performance gains across all network variations we studied, including: smaller (CORNet-Z) vs larger (VGG-16) architectures; variations in optimizers (Adam vs gradient descent); variations in activation function (ReLU vs ELU); and variations in network initialization. Our results demonstrate the potential utility of a new approach to training object recognition networks, using strategies in which the brain – or at least the statistical properties of its activation patterns – serves as a teacher signal for training DCNNs.

Introduction

Deep convolutional neural networks (DCNNs) have recently led to a rapid advance in the state-of-the-art object recognition systems (Lecun, Bengio, & Hinton, 2015). At the same time, there remain critical shortcomings in these systems (Rajalingham et al., 2018). We asked whether training DCNNs to respond to images in a more brain-like manner could lead to better performance. DCNN architectures are directly inspired by that of the mammalian visual system (MVS) (Hubel & Wiesel, 1968), and as DCNNs improve at object recognition tasks, they learn representations that are increasingly similar to those found in the MVS (Gu and van Gerven, 2015, Mcclure and Kriegeskorte, 2016, Razavi and Kriegeskorte, 2014, Yamins and Dicarlo, 2016). Consequently, we expected that forcing the DCNNs to have image representations that were even more similar to those found in the MVS, could lead to better performance.

Previous work showed that the performance of smaller “student” DCNNs could be improved by training them to match the image representations of larger “teacher” DCNNs  (Hinton et al., 2015, Mcclure and Kriegeskorte, 2016, Romero et al., 2015), and that DCNNs could be directly trained to reproduce image representations formed by the V1 area of monkey visual cortex (Kindel, Christensen, & Zylberberg, 2019). These studies provide a foundation for the current work, in which we used monkey V1 as a teacher network for training DCNNs to categorize images. We then tested the hypothesis that DCNNs trained with the monkey V1 as a teacher would outperform those trained without this teacher signal. By several relevant metrics (including accuracy), we found that performance was increased when monkey V1 was used as a teacher. Importantly, the monkey V1 data were collected in response to different images than the ones in the object recognition task. As a result, our approach, of using the brain as a teacher signal, can leverage pre-existing, and publicly-available neural data, without necessarily requiring new neuroscience experiments for each new machine learning task. Moreover, we also trained DCNNs with random teacher signals that matched the statistics of monkey V1 neural activations, and found that those outperformed networks trained without a teacher signal. This emphasizes a potential role for using the statistical properties of neural activations as a form of regularizer, that could be useful for training DCNNs.

Related recent work has demonstrated success in using neural data to train machine learning models. One study found that using fMRI measurements of human brain activations from subjects viewing images could guide Support Vector Machines (SVMs) decision boundaries (Fong, Scheirer, & Cox, 2018). In this study, the authors weighted the training data based on how easy it was for the human brain to recognize the example as a member of a class (Fong et al., 2018). This is different from our work where we train deep convolutional neural networks with a two-part cost function that explicitly trains on matching the neural representations, as opposed to the previous approach which weighted the cost function on specific training examples. In our work, the images shown to the animal during the collection of neural data were not category-labeled images from a machine learning benchmark task. For contrast, in the recent Fong et al. study (Fong et al., 2018), the neural data had to be collected for the same image set used for the categorization task. Previous work from Peterson et al. demonstrated that training deep convolutional neural networks with human perceptual uncertainty makes classification more robust to variations in the test set and to adversarial examples Peterson, Battleday, Griffiths, and Russakovsky (2020). In this study, they use human guidance to change the labels for training, incorporating uncertainties. We instead focus on changing the cost function – incorporating neural data into the network evaluation – without considering behavioral reports of uncertainty. Finally, previous work from Linsley, Shiebler, Eberhardt, and Serre (2019) demonstrated that using human behavioral data to add supervisory attention guidance improved object recognition performance. This is again quite a different approach from ours: while they focused on behavioral data, we instead used signals recorded from visual cortical neurons.

Notably, we did not aim to achieve state-of-the-art classification performance in this work: we instead sought to test whether the use of neural data as a “teacher” signal could robustly improve DCNN performance. For that reason, we studied a wide variety of network properties: different architectures and network sizes; different activation functions; and different optimizers. Our results indicate that, over all of these variations, (1) DCNNs trained to mimic monkey V1 (or surrogate data matching the statistics of monkey V1) have better object recognition performance, (2) DCNNs trained to mimic monkey V1 make fewer errors, and of these errors, more of them are within the correct superclass, and (3) DCNNs trained to mimic monkey V1 are more robust to label corruption. While the performance gains we observed were somewhat modest, based on the robustness of those performance gains, we anticipate that future work could productively apply our new training method to other networks, potentially improving on the current state-of-the-art object recognition systems.