WFOM imaging data collection.

WFOM is an in-vivo optical imaging technique capable of recording real-time neuronal activity and hemodynamics across the dorsal cortex of awake, behaving mice. We leverage transgenic mice expressing calcium-sensitive green fluorescent protein (GCaMP) in excitatory neurons, which can provide an ensemble spatiotemporal image of neuronal activity when excited with blue light from a light emitting diode (LED) illumination through a thinned skull cranial window. Reflectance measurements using green and red LEDs can be spectroscopically converted to dynamic maps of oxy-, deoxy- and total hemoglobin, signals which can also be used to remove the effects of absorption cross-talk on neuronal fluorescence signals.

During awake WFOM imaging experiments, mice are head-fixed and positioned on a low-resistance custom-made horizontal wheel that enables them to move and walk, and imaged for up to 90 minutes per session. The mouse’s behavior is recorded using synchronous webcam cameras capturing body and pupil movements. Specific details for each experimental dataset are provided in the Results section.

WFOM imaging system details.

Our custom-built WFOM system consists of three high powered LEDs (Thorlabs, M490L2, M535L2 and M625L3) at wavelengths of 490 nm (Blue), 535 nm (Green), 625 nm (Red), that are strobed to capture hemodynamic (red and green) and neural (blue) fluctuations simultaneously. Synchronous images are captured using an Andor Zyla sCMOS camera running at 31.22 Hz and an exposure time of 23.4 ms (resulting in an imaging frame rate of 10.4 Hz for each variable). A long-pass filter mounted in front of the camera blocks blue excitation light (FF01-496/LP-25, Semrock) in addition to an infrared shortpass filter to block behavioral monitoring light. Band-pass filters are placed in front of each LED (FF01-475/28 for blue, FF01-530/43 for green, and FF01-623/24 for red). For hemodynamic correction, red and green reflectance measurements are used to derive estimated values of hemodynamic absorption contributions to (blue) excitation and (green) fluorescence emission light as described previously [2]. The mouse’s behavior is recorded during WFOM imaging using two independent webcams (PS3 Eye), with a filtered infrared LED light source illuminating the mouse and a matching infrared filter on the webcams to prevent cross-talk with imaging light.

Animal preparation.

All animal procedures were reviewed and approved by the Institutional Animal Care and Use Committee at Columbia University (protocol AC-AAAS3453). For all WFOM imaging experiments, we used adult C57BL/6J-Tg(Thy1-GCaMP6f) mice (purchased from Jackson Labs and bred in-house). Mice were initially anesthetized with isoflurane and underwent a thinned-skull craniotomy over the cortex between coronal and lambdoid sutures, and implanted with a laser-cut acrylic head plate for restraint. The thinned-skull craniotomy was then protected by an optically clear cyanoacrylate layer (applied during surgery) to improve transparency and reduce bone regrowth. After surgery, all mice underwent a two-day post-operative recovery period with analgesia before imaging began.

SCAPE imaging data collection.

SCAPE microscopy is a single-objective high-speed 3D volumetric light sheet imaging technique capable of imaging cellular-level GCaMP activity in awake behaving mice at both high spatial and temporal resolution [1, 20] (Fig 3A and 3B). A mouse was prepared using viral transfection to produce sparse GCaMP6f labeling in apical dendrites of layer 5 neurons, and SCAPE data was acquired through a glass cranial window while the mouse was head-fixed but awake. SCAPE datasets were acquired with 488 nm (blue) excitation was acquired for a 60-second trial at 9.4 volumes per second. The size of the imaging volume was 374 x 1032 x 174 microns with a voxel size of 2.5 x 1.27 x 1.17 microns (x-y-z) [1, 18, 20].

Animal preparation.

Adult wild-type C57BL mice were initially anesthetized with isoflurane and underwent viral injection (AAV9.Syn.GCaMP6f) into the barrel cortex to label layer 5 cortical neurons with the calcium sensitive fluorescent protein GCaMP6f. Mice were then implanted with a glass cranial window (dura removed) and a head-fixation head plate and allowed to recover with analgesia. After several weeks, apical dendrites reaching up to cortical layers 1–3 were sparsely expressing the fluorescent calcium indicator and could be recorded using SCAPE microscopy.

Dimensionality reduction.

To estimate the primary temporal components of the data in an unsupervised manner, bilateral GCaMP data from the awake experiment was k-means clustered into 18 components via correlation distance measure. These clusters were then ordered according to their centroid position from the front to the back of the brain, and each component was assigned colors from the Matlab™ ‘jet’ color map (Fig 2B). We ensured that input data for this clustering corresponded to periods when the mouse was not running. Eighteen basis timecourses H(n,t) were then obtained using eroded k-means spatial clusters as regions of interest (ROIs). Non-Negative Least Squares (NNLS) fitting was then used to generate 18 spatial maps corresponding to the weight of each time-course W(s,n) in each pixel of the raw dataset V(s,t) [24]. Re-multiplying W and H yields model data V’(s,t) that can be compared to the original data to observe information that was lost in the dimensionality-reduced linear approximation (e.g. see S1 and S2 Movies). The same eroded k-means ROIs were used to extract basis time-courses from the ketamine/xylazine-anesthetized dataset, and then NNLS was used to generate that dataset’s corresponding spatial components and linear representation.

Video stream generation.

Using the color assignments shown in Fig 2B, spatial components W and temporal components H were multiplied and remixed to create dynamic color visualizations of neural activity. To clearly show periods of increased neural activity, all values below zero are represented as black in movies, so careful selection of this baseline is needed if significant decreases are present (here, the baseline chosen was the mean of the interval used to calculate GCaMP %ΔF/F the k-means clusters, i.e. at rest).

Analog audio stream generation + pitch encodes position.

H(n,t) was used to create an audio stream by multiplying each temporal component with a unique audio frequency sinusoid and then summing all of the components together (Fig 1B). The note pitches chosen were an ascending Cmin⁷ chord (C, E♭, G, and B♭, spanning 5 octaves across 18 components), ordered according to the centroid position of the W components ascending from back to front of the brain. Fig 2B shows the ordering of the 18 components and the corresponding musical note assigned to each. We note that it is important not to simply use arbitrary integer values for note frequencies, and to instead use natural note frequencies from the chromatic 12-tone scale to improve listenability. Finally, the audio stream was added as a soundtrack to the neural data spatial component video V’(s,t,c).

Effects of anesthesia on brain dynamics.

S1 and S2 Movies show the outputs from the above analysis on a mouse before and after induction of ketamine/xylazine anesthesia respectively. As can be appreciated from these representations, the awake brain exhibits a variety of different activation patterns, typical of resting state datasets previously reported [2], whereas ketamine/xylazine anesthesia resulted in a dramatic, repetitive rostro caudal wave of neural activity. This pattern is consistent with prior reports of slow wave neural dynamics under ketamine [25–27]. Audiovisual representations of this activity reveal a consistent rhythmic pattern from high to low notes, and yet it is also possible to perceive that each wave is not completely unidirectional, as some waves retrace forwards or originate in peripheral components.

Dimensionality reduction.

After background subtraction and detrending of the 4-dimensional SCAPE dataset, each voxel’s time series V(s_x,s_y,s_z,t_1-n) was checked for values that exceeded a z-score of 4. If more than 5% of the values in a voxel exceeded this threshold over the full time-series it was passed to the next clustering step, otherwise it was removed. The remaining voxels were then clustered into 85 components using K-means clustering. The resulting basis time courses were then used to create spatial maps using NNLS. These 85 components were then manually pruned down to 43 components by removing those that did not show well-defined morphologies in their spatial maps. Each 3D spatial component was median filtered (3x3x3 voxel filter) spatially to reduce background.

The resulting 43 spatial (W) and temporal (H) components are shown in Fig 3C. In this case, the unmixed spatial components W(i,j,k,n) are 3-dimensional in space. Components were color-coded according to the component’s centroid in the lateral dimension. S3 Movie (S3 Movie) shows results as 3-view maximum intensity projections (MIPs).

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Audiovisualization of a 4D SCAPE microscopy dataset.

A-B) Diagrams of SCAPE objective and mouse placement. C) Maximum Intensity Projections (MIP), top and side view of one time-point in the raw dataset during a dendritic firing event. D) Basis timecourses (H), arranged spatially on the left, and then arranged in order of first spike, with the original spatial color assignments kept. Audible note frequency follows the spike ordered components on the right, rather than the spatially arranged components on the left. E) MIP of top and side projection of all 43 spatial components after unmixing, color coded spatially from right (red) to left (pink). Inset shows four example components, which are also highlighted below in D. See S2 Fig for individual images of all spatial components. See S3 Movie for dynamic audiovisualization.

https://doi.org/10.1371/journal.pone.0297435.g003

Digital audio stream generation using MIDI encoding + pitch encodes event sequence.

To improve both the ease of listening and clarity of the audio stream compared to experiment 1, instead of simply summing sinusoids and varying their volume, here we encoded neural activity as a piano note. The percussive nature of a piano strike (fast onset followed by slow decay) allows the listener to easily hear the complexity of multiple event onsets while still perceiving the duration of each event.

To convert extracted time-courses H(n,t) into musical notes, distinct neural events need to be identified and represented by their magnitude, time of onset and duration of activity. For microscopy data in which events clearly emerge from a dark background, event identification is possible with a simple threshold, although care should be taken not to set the threshold too low, to avoid grouping multiple events into one sustained note. All events are then represented based on their peak amplitude, onset and duration (Fig 3C), such that small, weaker events are present but audibly scaled to match their magnitude. This digitization can then easily be converted into the MIDI message format in which each note is assigned a; note on (sec), note off (sec), note strength (ranging from 0 to 127), chromatic key choice (0 to 127, with 60 as middle C, or C3), and instrument (here, we used piano). This format can be converted into music using any VST, including Garage Band or REAPER, and is performed automatically in PyAnthem (see S1 Appendix).

In this example, rather than using note pitch to encode spatial position (as in experiment 1), we chose to sort components by the timing of their first event (Fig 3D). This means that that notes increase in pitch over time, but will gradually become more mixed as the same neuron fires a second, third or fourth time during the trial.

Audiovisualization of SCAPE microscopy of mouse dendritic activity.

The resulting audiovisualization is shown in S3 Movie. Ordering notes according to event timing gives a unique perspective on the relative timing and frequency of firing events of each neuron. We note that labeling of neurons in this case was sparse, and therefore does not represent the full activity of the somatosensory region. However, this representation permits assessment of the firing properties of single neurons that could be readily compared to real-time behavioral recordings, presentation of stimuli or the performance of tasks.

We also note that this analysis sought a deliberately low-dimensional representation of this data, which clustered similar time-courses together to yield groups of well-correlated pixels that belong to a dendritic tree, and differ sufficiently from other pixels over time to differentiate them from another dendritic tree (see S2 Fig for individual components). This low dimensionality removed higher order spatiotemporal noise in the dataset, but also did not seek to discover subtler differences in the timing of activity patterns along each dendrite or that may have differed within a dendritic tree from one event to the next.

Dimensionality reduction and video stream generation.

The same approach as in experiment 1 was used to reduce the collected neural data into 12 components (k-means clustering, followed by NNLS to extract spatial maps W). The same eroded ROIs, defined by the neural data were then also used to obtain basis timecourses for the hemodynamic data, followed by NNLS to extract the hemodynamic W. Color remixing was used to create movies of both hemodynamic and neural datasets in the same way as experiment 1. Colors were chosen from a jet color map, and assigned from the front (red) to the back (blue) of the dorsal surface of the brain (see Fig 4C) in an identical way for both neural and hemodynamic components. A composite movie was generated that includes behavioral and pupil data, after temporal synchronization between all of the video streams.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 4. Audiovisualization of both neural and hemodynamic WFOM data with simultaneous behavior in an awake mouse.

A) Example frames of behavioral, pupil, GCaMP and hemodynamic data. B) Plot of temporal basis timecourses, extracted using k-means clustering. C) Color mixed representation of NNLS output W, and reconstructed color mixed dataset at example timepoints from both neural (top) and hemodynamic (bottom) datastreams. The blue bar in B represents the time period of example data shown here. See S4 Movie for full audiovisualization.

https://doi.org/10.1371/journal.pone.0297435.g004

Combined analog and digital audio stream generation + instrument encodes measurable.

The audio stream was generated using the analog method (Fig 1B) for hemodynamic activity and the digital method (Fig 1C) for neural activity. The neural data was encoded as piano notes, while the slower, more continuous nature of hemodynamic fluctuations were encoded as violin. These two instruments provide a clear audible difference in the two dynamic datasets, despite their simultaneity. A Cmin⁷ chord was used for both data streams, with notes ascending from the back (rostral) to the front (caudal) of the brain. It should be noted that all audible sounds represented positive signals, which are dependent on the chosen baseline. Thus, it was important for this audiovisualization to select a baseline that was during a relatively quiet period of activity to avoid losing sub-threshold events.

Audiovisualization of neurovascular dynamics during spontaneous behavior.

S4 Movie (S4 Movie) shows four simultaneous video streams–behavioral, pupil, neural and hemodynamic measures of the awake behaving mouse. First, it is easily observed that sharp increases in neural activity, represented by the piano notes, are typically followed by slower matching hemodynamic chords, consistent with the properties of neurovascular coupling in the brain acting as a delayed spatiotemporal low pass filter of neural activity [28]. The behavior of the mouse is also clearly depicted in the audiovisualization of brain data. Onsets of running are striking events including significant activation of hindpaw regions, with hemodynamics of the region following behind. However, differences in the sequence of neural activation for different running bouts can also be discerned. At 15 seconds, the mouse begins grooming, and these finer movements are well represented by short, quiet notes that represent activation of forepaw somatosensory and motor regions. Activation of the bilateral visual cortex is seen at the start of the run when the WFOM illumination LEDs first turn on (see S1 Fig for an anatomical map of functional regions of the mouse cortex).

Choosing a suitable baseline and thresholds is particularly important for audiovisualization of datasets that are more smoothly varying over time. If a chosen baseline is too high or low, important audio events may not reach threshold, or may be saturated and difficult to distinguish from one another. The baseline of basis time-courses should always be inspected, and should generally be chosen from a period of relative quiescence. Data can also be filtered or detrended (as in experiment 2) or represented on a non-linear or logarithmic scale to emphasize specific data differentials where appropriate. Where signal decreases below baseline are of particular interest, they could be represented as an additional sound or instrument, and could be represented as grayscale or an additional color in spatial representations.