Representation learning as distribution matching

This work adopts the view that the brain learns representations of the world in an unsupervised manner. In the standard formalization of representation learning, this requires learning two components. First, a generative model defines how representations relate to the inputs. This model should be optimized to ensure that representations convey as much information as possible about the world [20]. Secondly, the brain should infer the representations that are consistent with new stimuli. Because sensory information is ambiguous, the brain should infer representations probabilistically. This involves performing Bayesian inference over the learned generative model. Further details of this standard formalization of sensory learning can be found in various textbooks [31, 32].

To take a simple example, one generative model might assume that data is composed of a mixture of discrete categories (Fig 1a), as in Gaussian Mixture Model. In this case one infers which category might explain new stimuli. Simultaneously, the generative model (e.g. the means and variances of each category) are optimized to match the data. Other examples of probabilistic representation learning differ in the form of their generative models, which span from linear transformations (e.g. in Probabilistic Principal Components Analysis) to the very complex hierarchical, nonlinear generative models that may be found in the brain.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Learning representations with a discriminating interneuron.

a) Representations are defined by a generative model that relates them to inputs. Organisms must learn to infer these representations from inputs (for example, to infer a visual category). b) This computational problem is equivalent to matching the joint probability distribution over inputs and representations during inference and generation. c) The brain may solve this problem with neurons that act as adversarial discriminators (red) of whether activity in neighboring neurons (blue) is dreamed or is stimulus-evoked. To achieve this, their input synapses would switch from a Hebbian to anti-Hebbian plasticity with the switch from wake to sleep.

https://doi.org/10.1371/journal.pcbi.1011484.g001

It is crucial to note that because of the generality of this objective, the representations that are learned (i.e. the latent factors in the generative model) may represent nearly anything. Nothing in the objective itself ensures that representations will be ‘useful’ for any particular task. This further consideration of usefulness must instead be supplied by assumptions in the generative model (such as that the data falls into discrete categories). This generality is powerful in that it allows us to reason about representation learning in the brain without overly specifying what is learned. However, it also has the disadvantage that it is only partially determined.

For the purposes of explanation we will examine a bipartite system. Formally, we define a sensory system that consists of neural activity in a more abstract, ‘higher’ brain area, z, and activity in a more concrete, ‘input’ brain area x. The generative model maps z to x, while the inference model maps x to z.

A generative model defines a probability distribution over x for each z. This distribution can be written as p_θ(x|z), and it depends on the parameters of the top-down connections θ. If one samples spontaneous activity from a prior distribution over representations p_θ(z), and propagates this spontaneous activity down to the input population, one obtains samples from a ‘generative distribution’ p_θ(x). This can be thought of as a ‘dreamed’ or ‘fantasized’ set of inputs. The objective of generative modeling is to match the generative distribution p_θ(x) with the empirical, stimulus-evoked distribution, such that generated and real samples cannot be distinguished. It can also be shown that this objective is equivalent to predicting inputs as best as possible, measured by the negative log likelihood (−log p_θ(x)), also known as the surprise [33]. Learning a generative model is thus equivalent to matching the distributions of generated and real activity in the inputs.

The objective of inference is also a distribution-matching problem. Given new stimuli, an organism may wish to infer what factors in its model of the world best explain sensory stimuli. This can be done with a bottom-up network, sometimes called an encoder or recognition model. Formally, the bottom-up network has the role of mapping an observed x to a distribution over z, q_ϕ(z|x). The objective of learning is to perform Bayesian inference over the generative model. Learning will adjust the parameters of this inference model, ϕ, such that q_ϕ(z|x) approximates the posterior distribution of p_θ(z|x), i.e. the z could have generated an observed x under the generative model. If learning is successful, the bottom-up connections ‘invert’ the top-down connections and can infer self-consistent representations for new stimuli.

While inference and generation are two separate objectives of matching distributions, they may be combined into a single objective. We can write the joint distribution over x and z in the inference mode as q_ϕ(x, z) = q_ϕ(z|x)q(x). Here q(x) denotes the probability that x occurs empirically; the use of q reminds that it is a wake-phase distribution. Similarly, we can define the joint generative distribution, p_θ(x, z) = p_θ(z|x)p_θ(x) = p_θ(x|z)p_θ(z). The joint distributions over x and z are aligned if and only if both the generative distribution is aligned (p_θ(x) = q(x)) and the inference distribution is aligned to the true posterior (q_ϕ(z|x) = p_θ(z|x)). (If the joint distribution is aligned, so must any marginal distribution. The implication in the opposite direction can be proved by the definition of the joint distribution). Taken together, performing both inference and generation requires aligning q_ϕ(x, z) with p_θ(x, z). The joint probability distribution of pairs of (real data, inferred representation) must match the joint distribution of (sampled representation, generated data) pairs (Fig 1b).

Aligning distributions with an adversarial algorithm

Any practical algorithm for sensory learning must maximize a measure of similarity between the inferred and generated joint distributions. The insight of the adversarial approach is that such a measure can be learned if one has a large number of samples of the two distributions. If no critic can discriminate the distribution from which a sample pulls, then the two distributions are the same. The goal is to change the distributions until they are indistinguishable in the eyes of the discriminator.

Adversarial algorithms represent an especially flexible strategy as they only require samples from distributions in order to align them. In contrast, many alternative algorithms for approximate inference (including Monte Carlo and variational inference methods) require the evaluation of the likelihood of a sample under either distribution. When the sample space is very high-dimensional and dimensions are correlated (as with neurons), this likelihood depends on all dimensions and may be quite complicated to compute. Being ‘likelihood-free’ thus eliminates this computational burden and makes adversarial algorithms attractive for approximate inference over complex generative processes [22, 23, 27].

Biologically, this makes adversarial algorithms compatible with the ‘sampling hypothesis’, which states that that neural activity represents a sample from the distribution encoding the brain’s uncertainty over neural representations [34, 35]. In this scheme, greater uncertainty in inference corresponds to greater neural variability. When implemented in models of the cortex, sampling procedures reproduce various neural phenomena such as task-dependent neuronal correlations, divisive normalization, stimulus-modulated noise variability, inhibition-dominated transients at stimulus onset, and oscillations [36, 37]. The strategy of representing uncertainty via sampling stands in contrast to neurons explicitly representing the parameters of probability distributions, such as their variance [38]. (It is worth noting that this is not a pure dichotomy, however, as one may also represent densities by sampling the component distributions of a mixture [39]). Committing to a sampling strategy to represent uncertainty, rather than an implicit distributional code, makes it more straightforward to implement an adversarial algorithm. The goal of representation learning becomes the alignment of the distributions of (stochastic) neural activity in the inference and generative modes.

The machine learning literature has extensively studied adversarial algorithms and their multiple potential formulations. The objective of the discriminator may be posed in multiple ways [40]. No one formulation is clearly more biologically relevant than others. For concreteness, and for the purposes of explanation and modeling, we consider the objective associated with the Wasserstein GAN [41]. However, the adversarial hypothesis is general to this implementation choice.

In the Wasserstein GAN formulation, the discriminator aims to maximize the difference of the average of the discriminator’s activity in one phase with the average of its activity in the other. Representing the discriminator as D(x, z), its objective to maximize is: (1)

During the phase in which the network samples from the inference model q_ϕ(x, z), the discriminator maximizes its output. During the phase in which the network samples from the generative model p_θ(x, z), the discriminator minimizes its output. Note that Wasserstein GANs require an additional regularizing penalty so that the discriminator does not sharply vary with small changes in its input: ∇_x,zD(x, z) ≤ 1. Pseudocode for this algorithm, also introduced in [22, 23] with minor differences, is shown in Algorithm 1. We discuss the biological predictions of this objective in the next section.

Interneurons as discriminators of inference and generative phases

The specific hypothesis of this paper is that specific cell types in the cortex act as discriminators of activity to guide sensory learning (Fig 1). These cells receive connections from local populations of cells, and then project back to these local populations to control plasticity. As the local population shifts between internally- and externally-biased modes of information, these cells learn to discriminate, i.e. to have higher firing rates in one phase compared to the other. In the section we describe this circuit and the key identifying attributes of the discriminating cells.

Computational analysis

The adversarial framework has already been shown to be capable of approximate inference over a generative model in neural network settings [22–25, 42, 43]. These publications provide a general proof of principle of the effectiveness of adversarial approximate inference.

In this section we present additional neural network simulations to demonstrate the capabilities of an adversarial approach, including both advantages and disadvantages. It is important to note that artificial neuronal network models differ from biology in many aspects. Our approach is not to include all possible biological details, or to claim that each computational component in these models has a direct corresponding element in the brain. For example, these ‘neurons’ do not spike and have no dendritic trees, ion channels, or cell types. The architectures are not directly modeled on brain connectivity. We assume that biologically plausible credit assignment is solved and use backpropagation to calculate gradients. Finally, we make several arbitrary choices about the batch size, learning rate, and optimizer that are not meant as hypotheses about biology. In these and in other aspects our models should not be interpreted as earnest simulations of the brain. Rather, each experiment proves a general capability (or failure) of the algorithm which together can help adjudicate whether it is powerful enough for sensory learning in the brain.

Approximate inference over recurrent generative models.

One crucial test of biological feasibility that has not yet been verified in the machine learning literature is whether this algorithm can work perform approximate inference over recurrent generative models. This capability is important to demonstrate as it would mark a significant advantage over competing theories of how the brain performs approximate inference.

Many previous theories of how the brain might learn generative models and approximate inference require calculating likelihoods. For example, many variational inference algorithms proposed as brain models that minimize the ‘surprise’ for new data, which is the negative log likelihood of that data under the generative model [13, 19, 29, 30]. However, local recurrence complicates this likelihood-based strategy because it introduces dependencies between neurons. To circumvent this problem, these previous theoretical models prevent such dependencies by eliminating the problematic recurrence, in effect sacrificing biological realism for computational tractability.

In Fig 2, we present a trivial task designed as the minimal case in which local recurrence causes dependencies that complicate variational models. We define a target population (cells B and C), which also driven by external stimulation from a ‘teacher’ cell T. Additionally, we define a neuron (A) whose output is used to model the distribution of B and C induced by T. If learning succeeds, the joint distribution p(B, C) should be the same when driven by T as when driven by A. However, neurons B and C interconnect. This causes dependencies in their distribution given neuron A (Fig 2b).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. A simple case illustrating the problem of predicting activity in a population with internal connections.

a) We created a toy task in which the goal is to adjust the connections away from neuron A to better model the activity of neurons B and C. The “target population” [B, C] is also driven by an external teaching signal from neuron T. All connections evoke Gaussian-distributed postsynaptic responses, and all neurons sum their inputs linearly. b) When learning the A → B and A → C connections by predicting the postsynaptic neuron (e.g. maximizing log p(B|A), ignoring recurrence), the predictions fail to match the joint distribution. The left plot shows the true joint distribution p(B, C), which is correlated and non-Gaussian due to the B → C connection internal to the target population, while the right plot shows . c) The adversarial strategy successfully aligns the distribution. The discriminator sees B and C and gates plasticity at the predictive connections from A. d) The alignment can be quantified with the KL divergence of the binned histogram between the learned and target distribution. We plot the trajectories from 5 random runs; all networks share the same initialization. e) Another measure of success is the distance of the parameters of the outward connections from A from their optimal value, which we call the parameter error.

https://doi.org/10.1371/journal.pcbi.1011484.g002

In a variational solution to this problem, the synapses from A would learn by attempting to maximize the log probability of observing B and C given A: log p(B, C|A). If B and C were independent given A, then the A → B synapses and the A → C synapses could learn with local information (a Hebbian rule). However, recurrence between B and C introduces a dependency, meaning that log p(B, C|A) ≠ log p(B|A) + log p(C|A). Thus B and C cannot be modeled independently. Synapses from A need access to all A, B, and C’s activity in order to learn. Indeed, because of the dependency between B and C given A, learning a model of B and C by predicting them separately causes poor learning (Fig 2b). Ultimately, the problem posed by recurrence for variational methods in the brain is that synapses now also require nonlocal information about many other neurons’ activity.

To solve this problem adversarially, one can introduce a discriminator. This discriminator sees the activity of A, B, and C, and controls plasticity at the synapses from A onto B and C. Learning proceeds by alternating between phases. In the ‘external’ phase, B and C are driven by A, whereas in the ‘internal’ phase they are driven by the hidden cell T. The discriminator learns to classify these phases by attempting to increase its activity in one phase and decrease it in another. The synapses from A attempt to affect the activity of the discriminator in the opposite way. This adversarial strategy allows the synapses from the source neuron A to correctly model their distribution (Fig 2c). Overall, this occurs because the discriminator observes the entire network state and summarizes it in a low-dimensional way that remains useful for learning.

We next aimed to demonstrate this algorithm in a higher-dimensional neural network with recurrence. We created a neural network with both a recognition model and generative model of raw inputs (Fig 3), both of which are stochastic, nonlinear, and recurrent. The generative model and inference model are identical in architecture and are as follows. First, a 2-layer fully-connected neural networks output the mean and variance of a diagonal Gaussian over the latent z or inputs x, which is then sampled. Then, this sample is given to a stochastic network of the same architecture, the output of which is interpolated with the original sample via an arithmetic mean. This step models a single step of recurrence. As a result of the nonlinearity and interpolation, the conditional distributions q_ϕ(z|x) and p_θ(x|z) are not Gaussian. This recurrence induces dependencies among neurons in each layer, given the previous layer.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Approximate inference in a generative model of MNIST digits using a recurrent, stochastic autoencoder.

a) In both inference and generation, samples are processed through a multilayer network that parameterizes a multivariate Gaussian. The network is then sampled and undergoes nonlinear stochastic recurrence. b) Performance metric: Following Berkes et al. [3], we empirically verify if the joint distributions of ‘sleep’ and ‘wake’ phases match by comparing the frequency over a binned histogram. We quantize the vectors of x and z by rounding to the nearest integer and quantify the histograms of the frequency of each pattern across 32 units’ activity when observing/generating MNIST digits. Perfect performance corresponds to all points lying on the y = x diagonal. c) As a baseline, we train the architecture with the standard variational autoencoder objective function (a.k.a. the ELBO), approximating the inferred distribution as a Gaussian and essentially ignoring the stochastic recurrence. This approach unsurprisingly fails, with patterns in ‘sleep’ occurring at different frequencies than in ‘wake’. d) In the adversarial wake/sleep algorithm, a global discriminator observes both z and x during wake and sleep phases. e) This algorithm enables realistic generation but often results in poor reconstruction. f) The frequency of patterns over a subsample of units is much more similar in sleep and wake than in (c). g) The oscillatory algorithm: During the wake phase, inputs are reconstructed through the generative model, and the discriminator attempts to classify real from reconstructed. The discriminator also aims to classify wake from sleep. h) Reconstruction improves due to oscillations without negatively affecting generation. i) Matched patterns for the oscillatory algorithm.

https://doi.org/10.1371/journal.pcbi.1011484.g003

We trained this network with the adversarial wake/sleep algorithm. In the offline ‘sleep’ phase, random representations are sampled and the network generates novel inputs x from its learned model. In the ‘wake’ phase, the network sees a batch of digits and the network produces representations z by sampling from q_ϕ(z|x). The goal of learning is to produce the z that could have generated those inputs, and to be able to generate realistic inputs from the prior. To assist in this goal, a discriminator sees both inputs and representations and aims to classifies pairs of (x, z) as being from wake or sleep.

By training against this discriminator, we found the network learned to generate realistic digits from its prior (Fig 3d). This demonstrates that adversarial approach is sufficient to train a model of the inputs p_θ(x) even in recurrent architectures. However, the empirical evaluation surfaced an issue with the adversarial wake/sleep algorithm. While generation is of good quality, reconstructing digits produces different digits than those input (Fig 3e). This suggests that inference is not optimal. Reconstruction maps to the manifold of MNIST digits, but elsewhere. This is despite the optimal solution being perfect inversion. A wide search over learning rates, weight decay, and the β parameters of the Adam optimizer did not produce better reconstructions.

Unfortunately, the same factors that make this task interesting (its dimensionality and recurrence) also make it difficult to provide a quantitative evaluation of quality. Direct measures like the log-likelihood or KL divergence are not tractable, and kernel density estimation is prohibited by the dimensionality. However, a quantitative measure is possible if the data is downsampled. We thus employed the approach of Berkes et al. [3] and empirically quantified the overlap of the wake and sleep joint distributions. After quantizing the inputs and latents to the nearest integer and subsampling to 32 units (16 x and 16 z), we counted how many times each unique pattern over the 32 units appeared in wake or sleep. The quality can be compared by visual inspection, as the closeness of patterns to the diagonal.

For comparison, we also trained this same recurrent architecture with a strategy that ignores recurrence and attempts to estimate the likelihood of the inferred representation under the generative prior. Specifically, we trained this architecture using the standard variational autoencoder (VAE) loss function, also known as the variational free energy or ELBO [58]. This is possible because the probability distributions in inference (absent recurrence) are Gaussian and can be reparameterized. However, this strategy can only work by ignoring how recurrence changes the inferred probability distribution to something non-Gaussian. Thus, the loss function relies on an incorrect assumption. Unsurprisingly, we found that this significantly hampered training. The histograms of 32 random units appear severely different; patterns that occur during inference rarely occur during sampling (Fig 3c). This underscores the importance of accounting for recurrence and its consequences for the form of conditional probability distributions.

We next trained the network with the oscillatory objective. There are two differences with this approach. First, the discriminator is local to each layer, a feature that is more biologically plausible (Fig 3g). Second, in the processing of information, an input is transformed into a representation z, and then that representation is fed into the internal model to produce a reconstructed input x′. The local discriminator attempts to classify the original input x from x′. In addition, we also employed the approach of the VAE-GAN of using the hidden layers of the discriminator as a metric between inputs and reconstructions [43]. We found that this approach was much more stable (Fig 3h).

This experiment demonstrates the effectiveness of adversarial representation learning even in settings where recurrence complicates stochastic representations. Our intention is not to establish a machine learning baseline; very similar algorithms exist in the literature (e.g. [22, 43]. Rather, it is to convey the computational advantage of this algorithm for approximate inference in a more similar algorithmic setting that might be found in the brain.

Biological evidence and candidate cell types

The adversarial brain hypothesis presents several strong predictions about connectivity, learning, and sensory processing. How plausible is this concept? Is there enough biological evidence to suggest a specific cell type as a discriminator? In the following sections, we review the relevant literature and outline criteria to refine this search. Ultimately, this hypothesis can only be confirmed through experiments testing its specific predictions (refer to Fig 4).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. An experimental program to test the adversarial hypothesis.

a) A key first step is to test whether the brain’s learning algorithms actually do align the distribution of activity in wake and sleep. This would begin with identifying neurons in chronic recordings that have the same distribution of activity patterns in wake and sleep. A subset of these neurons could then be perturbed during sleep to produce a new distribution (pink). Over the course of several nights, it should be the the case that the distribution of stimulus-evoked activity should align with the perturbed distribution. This experiment could identically be run by perturbing the waking activity and observing learning in sleep. b) A next step is to identify if this alignment is mediated by discriminating interneurons. This might begin with a survey of gene transcription in interneurons in wake and sleep, with the prediction that some class should exclusively transcribe genes associated with postsynaptic long-term depression (LTD) in sleep, and genes associated with postsynaptic long-term potentiation (LTP) in waking states. Blocking this synaptic plasticity in only this cell type should prevent the alignment observed in the experiment in (a), even though it concerns different cell types.

https://doi.org/10.1371/journal.pcbi.1011484.g004

Due to their local connectivity, these neurons would be classified as non-projecting neurons or interneurons by definition. Although non-projecting excitatory neurons exist, the majority of interneurons are inhibitory. An additional identifying factor is their ability to control local plasticity, particularly during the critical period. Somatostatin-positive interneurons meet this criterion [61], as do 5-HT_3AR⁺ cells in Layer 1 [62]. To influence plasticity in this manner, the discriminating interneurons must have specific control over signals that regulate plasticity in those cells. For instance, spike bursts elicit significant calcium releases (thereby affecting plasticity) but are strongly regulated by the action of apical-dendrite targeting inhibition [63]. In this way, dendrite-specific inhibition can gate plasticity within cortical networks in a manner consistent with the effect of discriminating cells.

A further prediction of this theory is that the effect of discriminating cells on local plasticity must change from positive to negative gain with the overall phase of sensory processing (e.g., wake to sleep, or phases of an oscillation). For this to happen, there must be a signal that communicates the phase of processing and affects plasticity accordingly. Acetylcholine, a neuromodulator implicated in the control of sleep and initiation of dreaming, is a strong candidate for this role [64–66]. Interestingly, acetylcholine also plays a crucial role in cortical plasticity [67, 68]. For example, if acetylcholine is prevented from being released [69], cortical learning is impaired. Tellingly, this also occurs if acetylcholine’s effect upon somatostatin-positive interneurons is blocked [61], suggesting that it is a key modulator of these interneurons’ affect on plasticity. This suggests a biologically plausible strategy whereby acetylcholine, or another neuromodulator, might mediate the switching of the discriminator’s affect upon local plasticity with the phase of processing.

Connections to local discriminatory interneurons should exhibit a plasticity rule that switches polarity with the phase of processing, from a simple Hebbian to an anti-Hebbian rule. While there is no known cell-type specific evidence for this phenomenon, there is evidence for such switches in plasticity at more general scale. Synaptic strengths across the cortex are known to homeostatically increase during wakefulness and decrease during sleep [70–72]. Phasic changes in plasticity have also been observed at the timescale of oscillations, at least in hippocampus [73–75]. These phenomena are broadly consistent with the adversarial hypothesis’s prediction of plasticity rules that change synchronously with the phase.

It is useful to compare the properties of an adversarial algorithm with others in the theoretical literature that also use phasic mode switches for learning internal models [13, 30, 76]. In particular, there is a a notable difference in prediction for sub-cellular and dendritic responses. Here, in the feedback phase of the oscillation, top-down feedback should drive somatic activity. Other algorithms for Bayesian inference require that feedback only be integrated in dendritic compartments for comparison with bottom-up activity, with the difference between the two driving learning [13, 30, 76]. In reality, feedback into apical dendrites can have a large effect on somatic activity [63]. The adversarial framework can thus explain why oscillations affect somatic activities rather than just apical dendrites.

A fundamental requirement of this hypothesis is that the brain operates in two distinct phases of information processing: one for stimulus-evoked inference and another for self-generated activity. A strong candidate for the second phase is dreaming sleep, and particularly REM sleep. Numerous computational studies have suggested that cortical activity during sleep represents samples from a generative model [17, 28, 66, 77]. However, this perspective remains controversial. Much of the evidence is phenomenological; dreams exhibit many properties of the world, such as meaningful objects and sounds, and are rarely mere repetitions of past experiences [78]. There is also evidence of a similarity between neural activity in dreams and waking states. For instance, it is possible to decode visual imagery in dreams from neural activity using algorithms trained on waking activity [2]. Intriguingly, a recent study found that neural ensembles in V1 are replayed in sleep, although with greatest similarity in slow-wave sleep [79]. However, some argue that the surreal and temporally discontinuous nature of dreams indicates they are purposefully perturbed in some manner, and thus not straightforward samples from the brain’s generative model [80]. This aspect of dreams must be reconciled with the generative model framework. If the brain only aligns local distributions of activity between wake and sleep, for example, this might allow for unusual pairings of activity at the level of distant brain areas. The answer to this broader question about the statistical nature of activity in dreaming sleep is directly relevant to the plausibility of the adversarial hypothesis.

Alternatively, instead of dreaming sleep (or in addition to), we have proposed here that oscillations could represent the necessary mode switches between bottom-up and top-down signals. For a literature review of this possibility, we direct readers to the perspective offered in Honey et al. [17]. One example line of evidence is that the visual cortex is more sensitive to external visual stimuli at the troughs of individual cycles of alpha oscillations [81, 82]. Another consequence is that unexpected stimuli should evoke stronger observable oscillations in activity than expected stimuli. After all, for expected (i.e., well-modeled) stimuli, the top-down predictions are accurate enough to be indistinguishable from bottom-up signals, and no observable change is expected with a phase shift. Therefore, we anticipate a correlation between the power of the oscillation and surprise. In the auditory cortex, MEG and EEG studies show increases in gamma power in response to unexpected auditory stimuli [83, 84], omissions of expected musical beats [85], and unexpected mismatches between auditory and visual cues [86]. In the hippocampus, both theta and gamma ranges increase after unexpected stimuli [87]. A final prediction is that the oscillations should ascend up sensory hierarchies. As information feeds forward it appears as a traveling wave. Indeed, oscillations in the gamma range have been found to ascend up the visual hierarchy [88]. These numerous findings together support the notion that oscillatory waves could indeed represent switches between generative and inference modes that could be leveraged for learning.

Possible experiments

The confirmation of this hypothesis would require a dedicated research program, but it is not beyond the capabilities of current experimental methods. A series of experiments that confirm progressively more specific predictions may make it possible to positively identify a discriminating interneuron and thus firmly establish whether the brain learns with an adversarial strategy. Here, we outline one such research program.

The first experiment in this series is the most general. It must first be demonstrated that the brain aligns the probability distribution between neural activity in a phase of sleep (or oscillation) and stimulus-evoked activity. A further study investigating this could closely resemble that of Berkes et al. [3], which compared the distribution of activity in the ferret visual cortex when eyes were open and closed. Instead of comparing spontaneous closed-eye activity, one could compare activity during phases of LFP oscillations or during sleep. These distributions should match in adult animals. This is perhaps the simplest experiment and is likely achievable with current recordings, as it only requires a set of neurons to be recorded longitudinally over multiple hours, or even days, while simultaneously tracking the phase of processing (oscillation or wake/sleep). A positive outcome in this experiment would provide key evidence for the broad theoretical framework, tracing back to at least the Wake/Sleep algorithm [20], that the brain learns inference over a generative model by switching between phases of processing.

Perturbations would enable a deeper test of this general framework (Fig 4a). During sensory learning, one could selectively silence neural activity during a stage of sleep or half of an oscillatory cycle. This perturbs the generative distribution. By tracking neurons over multiple days, one could then observe if activity changes in the waking or opposing phase to match that perturbed distribution. If this capability is confirmed, it would indicate that the probability distribution of activity in these two phases is actively regulated to match. This key feature of the framework suggests that the brain learns representations (i.e., probabilistic inference over factors in a generative model) by comparing two phases. This general capability would then provide the starting point for experiments into the cell-type specific mechanisms that lead to it.

The identity of the discriminator is the major element that differs between our proposal and that of Gershman and Deperrois et al. [27, 28]. To identify a discriminating interneuron among the many non-projecting cell types, it may be most efficient to search for signatures of the plasticity switch with phase. One technique for this search is genetic methods for observing single-cell gene transcription, as these are now relatively inexpensive and can be scaled to many neurons [89]. Using single-cell RNA sequencing (scRNAseq) taken at various phases of sleep, for example, one could list candidate cell types whose expression of factors relating to postsynaptic LTP and LTD is dominated by the sleep cycle. This unique prediction may be observable in datasets available with today’s technology [90]. This would provide a first-line survey for discriminating cells (Fig 4b). A further step is to establish the causal role of these candidate cell types upon local sensory plasticity. By optogenetically perturbing their activity while tracking plasticity, one should see that their gating effect on plasticity shifts with phase. Similar experiments could be done for the discriminators proposed by Gershman and Deperrois et al., namely, the excitatory projection cells in the sensory cortex (in the case of Deperrois et al.) or in the prefrontal cortex (in the case of Gershman). Through such perturbations, one could effectively determine the identity of the discriminator.

Software and hardware

The experiments presented were coded in Pytorch v1.8. All experiments were run on NVIDIA GeForce GTX 1080Ti GPUs. Running on a single card, the experiments take about 5 minutes to train the MNIST architecture. Searches over training details required around 100 times these totals.

MNIST architecture

For the MNIST task [91], we used the train set for training and the test set to display reconstructions. MNIST is licensed CC BY-SA 3.0. Before training and generation, we rescaled inputs to the range [−1, 1].

The architecture of both the inference and generation networks was a fully-connected, two-layer neural network with 512 hidden units and 128 output units (or 784*2 in the case of generation). The hidden nonlinearities were Exponential Linear Units (ELUs). Half of the output units specify the standard deviation and half the mean of a diagonal Gaussian over the outputs. This iswasthen sampled stochastically, and this sample fed to nonlinear recurrence with the same architecture as the inference/generation networks. Finally, the output was recombined via an arithmetic mean with the inputs.

The prior distribution over the latent variables was a standard normal distribution. Sampling new inputs involves sampling from this prior, followed by processing through the generation network.

Discriminator architecture

We trained two adversarial algorithms in the MNIST architecture: one purely wake/sleep and the other additionally with an oscillation.

In the wake/sleep algorithm, the discriminator was an ensemble of discriminator cells, each of which sees a random subselection of the entire network state. Each of these ‘cells’ is itself a 3-layer neural network. The network had linear outputs, and each internal layer had 512 hidden units and a nonlinearity of a LeakyReLU unit with negative slope 0.2. The final discrimination decision was an average over all cells in the network state.

In the oscillatory algorithm, the discriminator was identical except that the connectivity of the discriminator cells was restricted such that each cell only observes a selection of units from either the inputs x or the representation z.

Algorithmic details

For all adversarial objectives we used the Wasserstein-GAN [41] with a gradient penalty of λ = 1 [46]. As is standard practice, the gradient penalty is applied to a random interpolation of the inputs and the generated samples.

The high-level algorithm in the wake/sleep MNIST task is unchanged from previous applications of adversarial representation learning [22, 23]. In the ‘wake’ phase, inputs were fed through the inference network to obtain samples of inferred representations. These were paired and given as input to the discriminator. The objective function of the discriminator in this phase was the mean of its output across the batch, whereas the objective function of the inference network was the negative of the discriminator’s mean. In the ‘sleep’ phase, the prior over representations was sampled and inputs are generated through the generative network. These were paired and given as input to the discriminator. The objective function of the discriminator in this phase was the negative of the mean of its output across the batch, whereas the objective function of the inference network was the discriminator’s mean. In both phases the discriminator was additionally regularized with a Lipschitz gradient penalty.

The oscillatory algorithm in this MNIST architecture was similar to the VAE-GAN algorithm [43]. In the oscillatory algorithm, the ‘wake’ phase differs in that an input is transformed into a representation z, and then that representation is fed into the internal model to produce a reconstructed input x′. The local discriminator attempted to classify the original input x from x′. In addition, as in the VAE-GAN, the hidden layers of the discriminator were used as a metric between inputs and reconstructions. The ‘wake’-phase objective of the inference and generative networks was to oppose the discriminator and also minimize the distance between the input and the reconstructions under the discriminator’s metric. This special ‘wake’ phase objective was added in sum to the wake/sleep objective described above.

For comparison, we also trained the MNIST architecture with a standard ELBO. This objective function minimizes the reconstruction error, while simultaneously minimizing the KL divergence of the inferred representation from the generative prior. This requires an estimation of the entropy of the inferred distribution, which is not available analytically because of the stochastic recurrence. As stated in the Results, we ignore this recurrence and take the entropy as that of the Gaussian parameterized before recurrence.

Optimization

For all tasks we used the Adam optimizer [92] with β₁ = 0.5, β₂ = 0.99, and weight decay of 2 × 10⁻⁵. The batch size was 512 except for the Moving MNIST task, which for memory reasons was 32. The learning rate of the inference and generation networks was 10⁻⁴ and the discriminator’s learning rate of 4 × 10⁻⁴. These learning rates decreased by a factor of 0.96 between epochs 200 and 250. The total number of training epochs for all tasks was 400. These hyperparameters were chosen from previous literature (specifically [22]). Random hyperparameter searches were also conducted, but these did not overly improve results. In general we found that adversarial training is very fragile to the balance of architecture, learning rate, and optimizer between the generator and discriminator. Architectures were tuned by hand by visually examining the quality of reconstructions on the training set and generations from random noise.

4-neuron toy experiment

In this experiment, all nodes are modeled as linear-summing units with stochastic Gaussian outputs. The initial weights are chosen at random from a standard normal distribution. To enable error propagation through the Gaussian stochasticity, we employed the reparameterization trick.

For the non-adversarial baseline, we aimed to minimize the log probability of the two nodes (when driven by T) under the connections from A, assuming their independence. We trained the weights of A using gradient descent with a learning rate of 0.001.

For the adversarial algorithm, we created a discriminator, consisting of a small neural network with a single hidden layer of 20 units and a Leaky ReLU nonlinearity with a negative slope of 0.2. We employed the Wasserstein GAN objective function with a gradient penalty of 0.1 and optimized all weights using RMSProp with a learning rate of 0.001. When comparing this algorithm to the non-adversarial baseline, we started with the same random initialization of weights. The remaining stochasticity arises from the sample generation process.

To assess the performance, we report the empirical KL divergence between the binned hexagonal histograms of the 2-dimensional distribution over B and C. We used a grid size of 25 bins per side, with an extent from [-5, 5] on the x-axis and [-5, 10] on the y-axis.