Using Deep Convolutional Neural Networks to Develop the Next Generation of Sensors for Interpreting Real World EEG Signals Part 1: Sensing Visual System Function in Naturalistic Environments

• ¹ DCS Corporation (United States), United States
• ² United States Army Research Laboratory, United States

Brain computer interface (BCI) feature extraction and classification models conventionally optimize accuracy for a specific user and task. Although this training method is valid for laboratory experiments, games, and several assistive technologies, it limits the application of the system in scenarios in which the end-user’s exact task and circumstances cannot be recreated in the lab. As interest in passive BCIs, used to monitor healthy individuals performing everyday tasks, continues to expand it is critical that the field focus on generalization as well as accuracy as the context surrounding everyday tasks can change dynamically and without warning. Outside of within-task subject-subject transfer, there has been relatively little research into the generalization capabilities of BCIs, and their potential use as general-purpose sensors that reliably operate independent of context. In our previous work with electroencephalogram- (EEG-) based BCIs, we investigated experiment-experiment transfer by applying convolutional neural network (CNN) BCI models, trained on one or more experiments, to held-out test experiments. Models trained on a pool of multiple experiments, as opposed to a single experiment, performed better on the unseen test experiment. Consequently, we believe pooled-experiment models to provide best performance if the exact task or stimuli is unknown or could change. Here we describe the first of two efforts using deep CNNs to develop BCI models that generalize across individuals and across application domains. We use the EEGNet algorithm, a CNN previously shown to generalize well to a variety of BCI paradigms in both event-related and oscillatory contexts. The convolutional architecture is inspired by standard temporal and spatial filters often used in EEG feature extraction. This architecture has been shown to work well in across-subject transfer learning, and across-domain learning for visual target detection. We apply this cross-domain approach to a variety of experiments to assess visual system function. Each were designed to elicit a P300 signal, a positive voltage potential occurring over the visual cortex approximately 300ms in response to a rare and relevant stimulus. Unless otherwise noted, we train our model on five out of the six available experiments detailed in Table I, with the sixth experiment held out as test set. We validate our model with Area Under Curve (AUC) where possible, and also from our ability to re-create prior empirical findings, such as: 1) sensitivity to P300 amplitude, latency, and rise time, and 2) workload-related effects due to the regulation of visual system function by user state. Using the 2Hz rapid serial visual presentation (RSVP) dataset as our test set (n=18), with all other experiments in Table I as the training set, we achieve a mean across-experiment AUC of 0.7987 (±0.0756 std) distinguishing background from target and distractor images. Background stimuli are perceptually distinct from target stimuli, whereas distractors are not. Although distractors differ from the target in a measurable way, they can elicit attenuated P300 responses. The amplitude of P300 response is expected to increase as the stimulus more closely resembles a target image. The scores for the deep learning model (DLM) similarly exhibit significant differences (p<0.01) in response to the image categories, with the lowest and highest scores being assigned to background and target images, respectively. Smaller target-target intervals (TTI) measurably attenuate P300 amplitude, shown in Figure 1. In a 5Hz RSVP experiment (n = 18) with variable TTI (Table I), the DLM scores show sensitivity to TTI when time-locked to stimulus onset as well as (up to) 1000ms post-onset. The red and green lines (Figure 2), sampling DLM scores from 600ms and 1000ms after target image onset, indicate a potential post-P300 response that is also sensitive to changes in P300 amplitude. In this same 5Hz dataset, which has a mean across-experiment AUC of 0.6420 (±0.0393), subjects press a button in response to target images. With this, we can investigate P300 rise time, which is associated with faster reaction times, illustrated in Figure 3. Though subsampled in time, DLM outputs show sensitivity to rise time (i.e. reaction time) effects (Figure 4). As demand for attentional resources (via workload) increases, P300 amplitude, latency, and (potentially) recovery time have been observed to change. To investigate this, we compare the P300 waveforms of the guided fixations dataset in Table I, in which subjects performed a binary visual discrimination task while performing varied intensity auditory N-back tasks, to the fixation-locked DLM scores of system monitoring experiment (not detailed in Table I). In this monitoring dataset, subjects freely viewed a screen for system status updates. After all system conditions were shown, subjects indicated from memory which systems had failed. In a low workload condition subjects only performed the monitoring task, and in a high workload condition they also performed an auditory math task. In the guided fixation dataset, the high workload P300 waveforms have, on average, lower peak positive amplitudes (highlighted red on Figure 5), and less pronounced troughs around the sidelobes (highlighted green on Figure 5). In the monitoring dataset, ground truth of the “targetness” of the attended object is unknown. However, when comparing the high and low workload wave forms in Figure 6, generated by DLM outputs time-locked the highest DLM scores in each workload condition, they appear to reflect the differences between high & low workload states in the side lobes (highlighted green on Figure 6).