Post-stimulus encoding of decision confidence in EEG: toward a brain–computer interface for decision making

All experiments were approved by USC's Institutional Review Board, and all participants gave written informed consent to participate in the study. We developed an experimental paradigm in order to evoke different levels of confidence in response to realistic stimuli. The three-dimensional (3D) models and images used in our experiment are similar to those used in [44]. The timeline for a single trial of the experiment is shown in figure 2(A). The stimulus, a character wearing either a cap or a helmet, was shown against a corridor background for 250 ms. After a 1.75 s post-stimulus gap, subjects were asked whether they saw a cap or a helmet. This post-stimulus gap was necessary in order to dissociate stimulus- and response-locked phenomena, and is discussed in detail below. Following the post-stimulus gap, subjects reported their response via mouse click—left click for helmet, right click for cap. There was no time limit for this response. After a 1.5 s post-response gap, subjects were then asked to report their confidence on a scale from 1 to 10, with a 1 being a random guess and a 10 being complete confidence. The prompt for confidence report was shown for a fixed duration of 2 s, after which the value indicated by the subject was recorded as their confidence. Subjects performed a total of 640 trials in 40 trial blocks. We now describe the rationale behind the gap sizes and the method by which task difficulty and thus confidence level were varied.

Zoom In Zoom Out Reset image size

In order to vary the difficulty of the task, we blurred the stimuli by different amounts on each trial. Specifically, the stimuli were blurred to five different levels (figure 2(B)) using Gaussian kernels of varying sizes, with each blur level appearing an equal number of times in the first block. After the first block, the number of maximally- or minimally-blurred stimuli in each subsequent block was adjusted based on the subject's accuracy in the most recent block. If the accuracy was below 65%, five maximally-blurred stimuli were replaced with minimally-blurred stimuli in order to make the task easier. If the accuracy was above 85%, five minimally-blurred stimuli were replaced with maximally-blurred stimuli in order to make the task harder. This was done in order to keep the accuracy in the range of 65%–75% because this range results in roughly equal numbers of confident and unconfident trials as follows. For highly confident trials, we would expect near 100% accuracy, while for minimally confident trials, we would expect near 50% accuracy (random chance). Therefore, with an equal number of confident and unconfident trials, the total accuracy should be around 75%. We refer to this task as the gap task.

Our 1.75 s post-stimulus gap was necessary in order to separate stimulus- and response-locked phenomenon. ERPs such as the motor-related cortical potential have been shown to start up to 1.5 s prior to a response-related movement [53], so it was necessary for the post-stimulus gap to be longer than this. This gap prevents response-locked activity which may be influenced by the response modality (verbal, button press, etc) from interfering with stimulus-locked activity. Therefore, this gap allows us to observe stimulus-locked activity that is independent of the response modality. Further, without this gap, it is possible that stimulus-related activity can leak into the response-locked epoch, thereby causing stimulus-locked activity to be seen as response-locked. In order to assess the impact of the post-stimulus gap, we also collected data during a stimulus-discrimination task that is identical to the gap task but with no post-stimulus gap, referred to here as the no-gap task (figure 2(A)).

We recruited 16 subjects to participate in our study (average age: 26 years; 2 female, 14 male; 1 left-handed, 15 right-handed). All subjects had normal or corrected-to-normal vision. Eleven subjects participated in the gap task, while eight subjects participated in the no-gap task. Three subjects that participated in the no-gap task also participated in the gap task. Two gap-task subjects were removed from the ERP and source localization analyses due to excessive eye movement artifacts. We collected EEG data using a 256 electrode BioSemi ActiveTwo system. In addition to scalp EEG, two electrodes were placed on the ears for re-referencing. Further, four electrodes were placed around the subjects' eyes in order to collect electrooculogram (EOG) data. This EOG data was used to correct for eye-movement artifacts, as explained in section 2.3. Stimuli were presented on an 11'' laptop screen with a 60 Hz refresh rate, using the PsychoPy Python toolbox [54]. Subjects were seated approximately 70 cm from the screen, and used a USB-connected mouse for all task-related responses. All subjects were right-handed and controlled the mouse with their dominant hand.

We first downsampled the raw EEG data from 2048 Hz to 256 Hz. For the ERP analysis, we applied a 1–40 Hz noncausal bandpass filter (zero-phase order 1406 Chebyshev FIR filter) to remove DC offsets and high-frequency noise. For the classification analysis, we instead used a 1–40 Hz causal bandpass filter (order 4 Butterworth IIR filter), which is more appropriate in a BCI setting. In order to remove eye-movement artifacts, we performed independent component analysis (ICA) using the InfoMax algorithm [55, 56]. We regressed independent components (IC's) onto vertical and horizontal EOG signals and removed the IC's with ${r^2}$ values above a threshold of .4. This procedure resulted in 1–3 IC's being identified for removal. Next, EEG channels that were identified based on a visual inspection as too noisy (mean = 13.5, std = 11.8 channels removed) were replaced via spatial spline interpolation. Lastly, data was re-referenced to the average of signals from the two ear electrodes. Finally, trials identified based on a visual inspection as containing high noise levels were removed from the analysis. For the gap task, 101 of 6840 trials were removed, and for the no-gap task, 0 of 5120 trials were removed. All preprocessing was done using the FieldTrip toolbox [57].

2.4.1. ERP analysis

2.4.2. EEG source localization

We compared six classification algorithms in terms of their ability to decode confidence from single trials of the gap task (figure 3). Specifically, we used logistic regression (log), support-vector machine (SVM), a Riemannian geometry (RG) classifier [71, 72], and three NNs architectures. The NNs included a multi-layer perceptron (MLP) network, a spatial convolutional NN (CNN), and EEGNet, an NN that was designed specifically for use with EEG data [73].

Zoom In Zoom Out Reset image size

For SVM classification, we used a nonlinear SVM with a Gaussian kernel. SVM and logistic regression classifiers were implemented using the scikit-learn Python library [74].

RG classifiers have been shown to be effective in various EEG classification tasks [71, 72, 75]. In this work, we use a similar approach to that used in [73]. This approach involves dimensionality reduction via xDAWN spatial filtering [76], a nonlinear transformation based on RG [71, 72], and classification via logistic regression.

We selected hyperparameters for the logistic, SVM and RG classifiers via grid search on training data within each cross-validation fold. This was done for the SVM's Gaussian kernel size and regularization parameters, the logistic classifier's regularization parameter, and the RG classifier's regularization parameter.

NN classifiers have been shown to have better classification performance than traditional methods in several domains, and have gained popularity for their ability to approximate arbitrary nonlinear functions [77–80]. CNNs are a modification to standard NNs that use two-dimensional (2D) convolution to take advantage of spatial structure in input, thereby reducing the number of trainable parameters. CNNs have shown great success in tasks such as image classification, where the input has spatial structure [81–83]. Lastly, EEGNet uses 1D temporal convolutions and has been shown to outperform other classification methods on several EEG datasets [73].

The input to our spatial CNN is a 3D tensor, where the first two dimensions are spatial, i.e., corresponding to EEG electrode location, and the third dimension is time. In order to use this spatial CNN with our EEG data, we needed a mapping from our 256 3D electrode locations onto pixels in a 2D 16 by 16 image, which we took as the first two dimensions of the CNN input. It was important that this mapping preserve the spatial structure of the electrode coordinates, i.e., that electrodes that were near each other on the EEG cap were also near each other in the 16 by 16 image. Our method for doing this is illustrated in figure 3(C). Essentially, we placed a 16 × 16 point grid under the 3D electrode coordinates, and assigned each grid point to the electrode nearest to it. In order to prevent multiple electrodes from mapping to the same grid point, grid points were assigned sequentially, with each grid-point being assigned to the nearest un-assigned electrode. A rectified linear unit activation was used as the activation in all hidden units in both MLP and CNN classifiers, and both classifiers were implemented using the tensorflow library [84]. The architectures of the MLP and CNN classifiers are shown in figures 3(B) and (D), respectively.

For the EEGNet classifier, we used 256 Hz data from 0 to 1 s post-stimulus, or 256 samples per channel, as input (figure 3(A), right). For all other classifiers, we used 8 Hz data from 0.2 to 0.7 s post-stimulus, or four samples per channel, as input (figure 3(A), left). We used a different sampling rate and longer time duration for the EEGNet input in order to match the characteristics of the data used to validate it in prior work. Indeed, more samples are required to get the most out of EEGNet's temporal convolution layers. Note that despite this, EEGNet was not the most performant classifier (see results). To obtain ground-truth binary class labels, we thresholded confidence reports at the 80th percentile across all subjects—trials with a reported confidence at or above this value had a ground-truth label of 'confident', while those below this value had a ground-truth label of 'unconfident'.

We assessed the performance of all the classifiers via the area under the curve (AUC) measure [85]. Briefly, each classifier takes a single trial of neural activity as input, and computes a continuous-valued score as output. This score is then thresholded to determine the predicted discrete class (confident or unconfident) of the associated input. By varying this classification threshold, we also vary the true-positive rate (TPR) and false-positive rate (FPR) of the classifier. The receiver operating characteristic (ROC) curve plots the TPR of a classifier against its FPR for various classification thresholds, and the AUC is the area under this curve. An AUC of 1 indicates perfect classification, while an AUC of .5 indicates that a classifier does no better than chance. All classifiers were evaluated via five-fold stratified cross-validation. This was done using the scikit-learn implementation of cross-validation [74].

In order to ensure that non-causal training, i.e., test trials occurring before training trials, was not providing an unrealistic advantage to the classifiers, we performed a control analysis where for each subject, classifiers were trained on the first 80% of trials and tested on the final 20% of trials. In order to control for the effect of stimulus blur level, we performed an additional analysis where the most and least blurred stimuli were excluded. The results of these control analyses are presented in appendix D.

In addition to studying the neural correlates of confidence and their decoding, our next goal was to determine if confidence decoders with the same accuracy as that observed in our data could be used as part of a BCI to improve a user's task performance. To do this, we devised a simulated BCI framework based on behavioral data collected in our experimental task, including decision accuracies and confidence reports (figure 4). We simulated a general stimulus-discrimination task where a user must determine if a stimulus belongs to one of two categories. The simulated BCI aims to help the user with making the correct decision. To do so, the BCI decodes the user's confidence after each stimulus presentation and repeats the stimulus until the decoded confidence is above a given threshold. Once this confidence threshold is reached, the user is allowed to respond.

Zoom In Zoom Out Reset image size

The details of the developed simulation framework are as follows: when the stimulus is first presented, the simulated user's true confidence, $c$, is drawn from a distribution based on real confidence reports collected during the gap task. The simulated decoder will then determine if the user's confidence is above threshold (confident) or below threshold (unconfident). In order to simulate a realistic, imperfect decoder, we set the FPR and TPR of the simulated decoder to be the same as what the best classifier achieves for confidence classification on the real EEG data. Specifically, we chose the point on the classifier's ROC curve that maximizes decoder accuracy when 20% of trials are confident and 80% are unconfident—this reflects the fact that our threshold for confidence is placed at the 80th percentile of the initial confidence distribution. We found this maximum accuracy point on the ROC curve as follows: for a given number of positive (confident) samples $p$ and negative (unconfident) samples $n$, the point on the ROC curve with highest accuracy is the point with a tangent line of slope $n/p$ closest to the top left corner [86].

Given the FPR and TPR corresponding to the chosen point on the ROC curve, the decoder (figure 4, 'Decoder' box) is simulated as follows. If the user's confidence is above threshold (confident), then the decoder's output is drawn from a Bernoulli distribution with parameter p equal to the TPR—because TPR is the probability that the decoder will correctly output 'confident' when the subject is actually confident. If the user's confidence is below threshold (unconfident), then the decoder's output is drawn from a Bernoulli distribution with parameter p equal to the true-negative rate (TNR), which is equal to 1 minus the FPR—because TNR is the probability that the decoder will correctly output 'unconfident' when the subject is truly unconfident. For both cases, a Bernoulli outcome of '1' corresponds to a decoder output of 'confident' while an outcome of '0' corresponds to a decoder output of 'unconfident'. If the decoder outputs 'unconfident', then the BCI repeats the stimulus. We then update the confidence according to the following rule:

$$\begin{equation}{c_{i + 1}} = {c_i} + {\text{ }}u\left( {1 - {c_i}} \right)\end{equation} \tag{ 1 }$$

where ${c_i} \in \left[ {0,1} \right]$ is the user's confidence after the $i$th stimulus repetition, and $u \in \left[ {0,1} \right]$ is a confidence update parameter that determines how much the confidence increases after each stimulus presentation (figure 4, 'Stimulus Presentation' box). Low values of $u$ indicate a difficult task where several repeated presentations of the stimulus are required to gain above-threshold confidence, while high values indicate an easier task where fewer stimulus repetitions are needed. For example, a version of our gap task where all the stimuli have the maximum blur level would correspond to a low value of $u$. The BCI continues the stimulus repetitions until the decoder outputs 'confident', at which point the user is allowed to respond. The user's decision is simulated by a weighted coin flip, where the probability of correctness linearly increases from .5 at $c = 0$ to .98 at $c = 1$. This linear increase in confidence closely mirrors the confidence–accuracy curve observed in our gap task behavioral data (figure 4, 'Decision' box).

To measure the relative effectiveness of the BCI decoder, we also simulated a control task where the stimulus was repeated randomly instead of based on a BCI decoder. For the control simulation, after each stimulus presentation, there was a 50% chance that the user would be allowed to respond, and a 50% chance that the stimulus would be repeated. This corresponds to a control decoder with an FPR and TPR of .5. Importantly, to account for the fact that a post-stimulus gap is needed to decode confidence but not needed for this random control BCI, we designed each repetition of the control task to take less time than each repetition of the BCI task. Specifically, each BCI repetition takes 700 ms, while each control repetition takes 450 ms. The BCI repetition time was chosen because our logistic classifier uses data from up to 700 ms after stimulus onset. The control repetition time was chosen based on a stimulus duration of 250 ms and a reaction time of 200 ms, for a total of 450 ms. This is roughly the amount of time that a subject would need to complete a trial without any BCI intervention.

Task performance within the simulation was measured via a penalized bitrate, which we define as

$$\begin{equation}b = {\text{ }}\frac{{{n_{{\text{corr}}}} - k*{n_{{\text{incorr}}}}}}{T}\end{equation} \tag{ 2 }$$

where ${n_{{\text{corr}}}}$ is the number of correct responses, ${n_{{\text{incorr}}}}$ is the number of incorrect responses, $T$ is the total time taken to complete all trials, and $k \in \left[ {0,{\text{ }}\infty } \right)$ is the error penalty. Large values of $k$ indicate that errors are very costly, while small values indicate that they are not. This penalized bitrate measures the number of correct answers per unit time, but with an additional term that serves to penalize incorrect responses. By varying the parameters $u$ (1) and $k$ (2), we can simulate tasks with varying levels of difficulty and error cost. We varied $u$ from .01 to 1 in steps of .01 and varied $k$ from 0 to 10 in steps of 1. For each combination of parameters, we simulated 100 000 BCI trials and 100 000 control trials.

Behavioral analysis showed that our subjects successfully performed the task as instructed, and that their subjective confidence reports were good indicators of correctness on individual trials. The Brier score [87], defined as the mean-squared error between reported confidence and correctness, was significantly better than chance level for all blur levels (p < .05, paired t-test, figures 5(A) and (E)). This indicates that confidence was predictive of correctness across blur levels. Subjects also had significantly above-chance accuracy for each blur level (p < .05, paired t-test, figures 5(B) and (F)). Accuracy increased monotonically with reported confidence, further indicating that subjects' confidence reports were well-calibrated (figures 5(C) and (G)). Lastly, reaction times were significantly faster for confident trials than for unconfident trials in both the gap task and no-gap task (p < .05, Wilcoxon rank sum test) (figures 5(D) and (H)). For the no-gap task, the difference in median reaction times between confident and unconfident trials was 205 ms, while for the gap task, the difference was 12 ms.

Zoom In Zoom Out Reset image size

We performed a grand-averaged ERP analysis, comparing confident trials with unconfident trials for the gap and no-gap task, for both stimulus- and response-locked epochs (figure 6(A)). In the no-gap task, we found that confident and unconfident ERPs at channel Pz were significantly different (p < .05, cluster-based permutation test) both around 500 ms after the stimulus onset and around the response time (figure 6(A), top row), thus making it ambiguous whether the difference was stimulus-locked or response-locked. Topographies depicting ERPs across all channels at both these times also had similar characteristics, with positive differences between conditions appearing over the parietal regions (figure 6(B), topography plots 1 and 2).

Zoom In Zoom Out Reset image size

Interestingly, unlike the no-gap task in which there was ambiguity as to whether the activity was stimulus- or response-locked, in the gap task, this pattern of activity was only observed in the stimulus-locked epoch, and not in the response-locked epoch (figure 6(A), bottom row; figure 6(B), topography plots 3 and 4). Importantly, in the gap task, this stimulus-locked difference occurred long before the response, suggesting that there is no possibility of interference between stimulus- and response-locked activity.

Parietal regions had large ERP differences during the stimulus-locked epoch in the no-gap task, the response-locked epoch in the no-gap task, and the stimulus-locked epoch in the gap task, suggesting that shared brain processes may be involved in all three epochs (figure 6(B), topography plots 1–3). This pattern of activity is consistent with the P300 ERP, which is known to be associated with task-related stimulus processing [88, 89]. Further, the latency of the peak difference between conditions relative to the stimulus onset, around 500 ms, suggests that the observed neural activity corresponds to a cognitive, top-down process, and not to low-level stimulus processing, which typically occurs much earlier post-stimulus (around 100 ms) [88, 90].

Given that the response-locked activity follows a different pattern in presence of a post-stimulus gap (figure 6(B), topography plot 4), these results suggest that confidence-related activity in our task is stimulus-locked and not response-locked. They suggest that in the absence of a gap, it is the stimulus-locked activity that appears as response-locked. When a gap is added, stimulus-locked and response-locked activity are separated, thus removing the ambiguity.

Having identified that confidence-related activity is stimulus-locked in the presence of a gap, we then used eLORETA source-localization analysis to explore which brain regions contain sources of confidence-related activity in the gap task. We used a template head model and template electrode coordinates in order to perform this analysis (section 2.4.2). We first confirmed that using a template head model gives sources that are comparable to those obtained using subject-specific head models based on MRI data and co-registered EEG electrode locations. To this end, we used the Localize-MI dataset, a publicly available dataset developed for the express purpose of validating source-localization methods [66]. The dataset features high-density EEG data that was recorded while subjects received intracranial electrical stimulation. The locations of the stimulating electrodes serve as the ground-truth for source-localization methods. Our analysis revealed that eLORETA with both subject-specific and template anatomy performed significantly better than chance level in localizing the stimulating electrodes (p ⩽ 1.2 × 10⁻¹², paired t-test). Further, performance when using subject anatomy was not significantly better than when using template anatomy (p = .1, paired t-test). This result indicates that template anatomy can indeed be used to perform reliable source analysis. The details of this validation analysis are shown in appendix C.

Having validated our source analysis pipeline, we now discuss the results of our source analysis on the gap task EEG data. EEG data was first down-sampled to 20 Hz and separated into confident and unconfident trials. Next, we computed a confident trial average and an unconfident trial average for each subject. We then used eLORETA to compute the neural sources for both the stimulus-locked epoch (500 ms post-stimulus) and the response-locked epoch (100 ms pre-response) for each subject and condition (confident, unconfident). This left us with one confident source map and one unconfident source map for each subject. Lastly, we used a cluster-based permutation test across subjects to test if there was a significant difference between confident and unconfident sources. We found a significant difference (p < .01) in the stimulus-locked epoch of the gap task, but not in the response locked epoch, with below-threshold clusters in the right occipital and temporal lobes (figure 7). This result supports our previous finding that, in the presence of a post-stimulus gap, confidence-related activity is stimulus-locked and not response-locked.

Zoom In Zoom Out Reset image size

Our results thus far suggest that the neural correlates of confidence are stimulus-locked, and that these correlates appear prior to the subject's response when a gap is present. We next sought to decode confidence from single trial stimulus-locked data in the gap task. Note that while various studies have looked at decoding of confidence, here by doing the decoding in the new gap task, we could ensure that the decoding only used the pre-response activity and that activity related to response execution was not helping the decoding results. Thus, our decoding can inform whether a BCI that requires reliable decoding pre-response can be constructed.

We used a battery of classifiers, described in section 2.5, to decode confidence from single-trial, stimulus-locked data from the gap task. All classifiers were evaluated via five-fold stratified cross-validation. Figure 8 shows the average AUC across all subjects (panel B) and across folds for each subject (panel C). P-values for across-subject and per-subject results were corrected for multiple comparisons via the Benjamini–Hochberg procedure [91]. For pooled results across subjects, all classifiers were able to classify confidence with significantly above chance accuracy (p ⩽ 1 × 10⁻⁸, N = 55, one-sample t-test; figure 8(B)), suggesting that confidence is robustly decodable from single trial stimulus-locked pre-response EEG activity. This shows feasibility of a BCI that needs to decode prior to response. Further, for 8 out of 11 subjects, 5 out of 6 classifiers achieved significant decoding, while for the remaining 3 subjects, at least 2 out of 6 classifiers were significant (p ⩽ .05, N = 5, one-sample t-test; figure 8(C)). Notably, of all the classification methods, the logistic classifier performed best, with an average AUC of .75 across all subjects and folds. Also, logistic classification was significant in 10 out of the 11 individual subjects (figure 8(C)). It is likely that the logistic classifier performed better than the more sophisticated deep learning methods because it requires relatively fewer training samples for accurate model fitting.

Zoom In Zoom Out Reset image size

We used the simulation framework described in section 2.6 to investigate the effectiveness of our confidence decoder in enabling a performance improving BCI. We explored the advantage of such a BCI as a function of task difficulty and error cost, which are quantified by $u$ and $k$ in equations (1) and (2) respectively. The operating points for the BCI and control decoders are shown in figure 9(A). In figure 9(B), we plot the penalized bitrate against the parameters $u$ and $k$ for both the BCI and control cases. We show only regions of the plot where the penalized bitrate is positive. The qualitative interpretation of a negative bitrate is that the user is making so many errors that they are better off not doing the task at all and achieving a bitrate of 0. As such, comparisons between BCI and control conditions for negative bitrates are not meaningful.

Zoom In Zoom Out Reset image size

Our results show that the BCI outperforms the control in the high-difficulty (low $u$), high-error cost (high $k$) regime. In the low-difficulty, low-error cost regime, however, the control has a higher penalized bitrate. This is because control trials are designed to be shorter than BCI trials (section 2.6), and thus for low values of the error penalty $k$, this difference in trial duration outweighs the better decision accuracy that the BCI enables—that is, it outweighs the difference in the bitrate numerator ${n_{{\text{corr}}}} - k*{n_{{\text{incorr}}}}$. In other words, for low values of error penalty $k$, it is better to favor speed over accuracy, and the faster control trials have a higher bitrate. For high error penalties and high task difficulty, this is no longer the case; the longer trial duration of the BCI is offset by the fact that the BCI prevents the user from making costly errors. In other words, when the error penalty $k$ is high, it is better to favor accuracy over speed, and thus BCI trials, which are more accurate, achieve a higher bitrate.

Our simulation analysis thus shows the feasibility for our pre-response decoder to be used in a BCI framework in order to improve task performance specifically when the task is difficult and errors are costly.

To address the question of whether confidence-related activity is stimulus-locked or response-locked, we provided two novel comparisons. First, by comparing a gap and a no-gap task, we showed that the presence of a sufficiently long post-stimulus gap impacts whether or not confidence-related activity appears to be stimulus-locked or response-locked. Specifically, our comparison of the gap and no-gap tasks confirmed the hypothesis that stimulus-locked ERP differences can appear to be response-locked in the absence of a sufficient response delay (gap). To the best of our knowledge, such a comparison of confidence-related neural activity between two stimulus discrimination tasks that differ only in terms of the presence of a gap has not been done before. Second, we compared stimulus-locked and response-locked activity for a post-stimulus gap task. This experimental design and the two novel comparisons allowed us to conclude that confidence related activity is stimulus-locked and not response-locked, and that confidence can be decoded purely from stimulus-locked activity without help from activity related to response execution.

Our finding that in our task confidence does not have a response-locked element is also interesting in light of prior works regarding confidence formation. Prior works have suggested different models for confidence formation, some suggesting that it may precede decision making [43], and some suggesting that it may be a secondary process that occurs after making a decision [92, 93]. The former is more aligned with our finding that the confidence representation in our gap task is exclusively stimulus-locked. Exploring evidence for various theories of confidence formation across different tasks is an interesting topic for future investigations. As mentioned above, one reason that a post-stimulus gap is important is that it prevents stimulus- and response-locked activity from overlapping. For example, the response-locked difference in the no-gap task (figure 6(A), top right) could be due to the difference in reaction times between confident and unconfident trials [94]. The median reaction time for confident trials was 205 ms faster than for unconfident trials (figure 5(D)), and as such, it is possible that the confident response-locked ERP captures the return to baseline following the stimulus-locked response, while the non-confident ERP has already returned to baseline. The post-stimulus gap prevents reaction time differences from influencing the response-locked ERP in this way.

This is, however, not the only reason a post-stimulus gap should be considered. For example, it has been shown that an ERP known as the error negativity (ERN) occurs in the response-locked epoch when stimulus processing continues after an incorrect response is made [95]. ERN amplitudes are typically stronger when a subject is aware that they made an error. It is possible that a long post-stimulus gap prevents subjects from making hasty decisions that lead to 'avoidable' errors that they are aware of, thereby attenuating the ERN. This phenomenon could possibly explain the lack of a response-locked difference between confident and unconfident conditions in the gap task. In studies where confidence, but not error monitoring, is the cognitive state of interest, a gap may be necessary to dissociate confidence-related activity from error-related activity.

Another phenomenon to consider is that of temporal scaling. Prior work has shown that rather than occurring at fixed latencies relative to events, certain neural responses can stretch or compress to fill a stimulus–response gap [96–98]. When characterizing particular neural responses, it is important to understand whether the response occurs at a fixed latency, or if it scales with the stimulus–response gap. Among other reasons, this distinction is important because fixed-latency responses may appear in the response-locked epoch with a small gap, while temporally-scaled responses may not. One way of making this distinction is to compare activity between tasks where the only difference is an enforced stimulus–response gap. In our study, we showed that in the absence of a gap, stimulus-locked and response-locked sources of neural activity do indeed overlap, which can suggest that the neural activity we observed occurs at a fixed latency relative to stimulus.

Our source analysis found below-threshold clusters in the right occipital and temporal lobes, suggesting that these brain regions are involved in the neural representation of confidence. The temporal and occipital lobes are a part of the ventral visual pathway, which is known to be involved in object recognition and visual memory [99]. Our results suggest that these regions may also be involved in representing confidence following decisions in a visual object recognition task. Several types of visual processing have also been shown to be lateralized—for example, evidence suggests that language processing occurs primarily in the left hemisphere, while face recognition occurs primarily in the right hemisphere [100–102]. Some aspects of object-based attention and categorization may also be lateralized to the right hemisphere [103]. These phenomena may contribute to our observation that neural sources of confidence in our task appear to be right-side lateralized.

Recent work in [104] involved the simultaneous recording of EEG and fMRI from subjects performing a perceptual decision-making task, during which subjective confidence reports were collected. In this study, analysis of post-stimulus EEG data revealed confidence-related activity in parietal-occipital and frontal-temporal regions, while fMRI analysis showed activity in the left middle frontal gyrus, premotor cortex, superior parietal lobule, right caudate nucleus, and cerebellum. These EEG results generally match what we find in our source localization analysis in figure 7 (i.e., occipital and temporal sources of neural activity), while the fMRI analysis finds different, although functionally related, brain regions. Specifically, the superior parietal lobule has close links with the occipital lobe and is involved in visuospatial perception [105]. We note that the primary goal of [104] was the development of a team-based decision making BCI, and as such did not perform stimulus- and response-locked comparisons of gap and no-gap versions of their task, which is our main focus.

The use of the gap task allowed us to explore whether, using just stimulus-locked pre-response activity, we can decode confidence. Indeed, the gap duration was chosen to be long enough to prevent activity related to response execution from interfering and confounding the decoding results. Our confidence classification analysis on this gap task showed that various classification algorithms could classify confidence from single-trial stimulus-locked pre-response EEG activity alone. Interestingly, among the considered classifiers, the logistic classifier achieved the highest accuracy, even outperforming the more sophisticated deep learning methods. In order to prevent overfitting on our dataset, we had to keep the size and number of layers in our MLP and CNN classifiers relatively small. It is likely that neural network models could use more complex architectures and achieve higher performance if more data is collected from each subject. However, collecting more training data would come at the cost of time and comfort for the subjects and future users of a BCI, and as such may not be desirable. Here, with a manageable number of trials (640) per subject, we were able to achieve a relatively high AUC of .75.

Beyond the classifier performance itself, an important point of consideration is whether this classifier is viable for use within a BCI framework. Viability requires not only that the classifier's AUC is sufficient to allow for the BCI to improve the user's performance, but also that the classifier can be run in real time. In section 3.5, we performed an extensive simulation that verified the first point, that our classifiers were accurate enough to enable a BCI for decision making. Regarding the second point, we note that the time it takes for the trained classifiers to produce an output once given input data is negligible compared to the other delays present in the BCI system, such as reaction times and ERP latencies. For these reasons, these confidence classifiers are indeed viable for use in a real-time BCI.

We have shown that within our simulation framework, a realistic, imperfect confidence decoder can be used to improve performance on a stimulus discrimination task. This simulation assumes that the BCI has the ability to repeat the stimulus, i.e., that the task takes place in a controlled environment where the stimulus presentation can be controlled by a computer. Further, since the decoder uses stimulus-locked data to estimate the user's confidence, it must have access to the stimulus onset times. It is important to consider whether it is reasonable to expect these conditions to hold in realistic use cases, and what can be done in cases where they do not.

In some realistic situations, it is entirely possible that the stimulus can be controlled by the BCI. For example, in assembly line quality control, a human inspector must look at samples pulled from the assembly line and determine whether or not they are up to standard. We envision a use case where the inspector is presented with images of products coming off the assembly line and must determine whether they are defective or not. In such a case, our BCI setup is directly applicable, as it is possible for the BCI to repeat or adjust the presentation of such images. Additionally, since the BCI can control when the images appear, the decoder would have access to stimulus onset times.

There are, however, situations where the aforementioned conditions do not hold. If the user's task involves reacting to and making decisions regarding stimuli that occur naturally, at random, or in some other way that cannot be controlled by a computer, then our BCI framework would require some adjustments to be applicable. First, an event-detection method is needed so that stimulus-locked data can be collected. Prior work has developed algorithms that can detect behavioral or stimulus events in real time from neural activity [106, 107]. Such algorithms can be used alongside the BCI in order to determine when stimuli appear. Once a stimulus is detected, neural data following the stimulus can then be fed into a classifier as usual. Second, if the BCI cannot repeat the stimulus, then some other feedback method is required in order to improve the user's decision performance. Possible alternatives to stimulus repetition include providing additional task-relevant information to the user [42], providing input from AI or human agents performing the same task, or simply informing the user that they are unconfident and should take more time before making a decision. With the use of an event detection algorithm and an appropriate feedback signal, it should be possible for a BCI to improve the user's performance even when stimulus times are unknown and cannot be controlled by a computer.

We have shown with numerical simulations that our decoder can be used to improve task performance in a BCI framework. Specifically, the BCI improves performance when the cost of error is high, and the task is difficult. An important future direction would be to implement and test such a BCI with human subjects. The end goal of this research would be to develop a BCI that can assist human users in practical, real-world tasks. As stepping stones to this end goal, BCIs can be constructed for and tested with increasingly complex and more realistic tasks. One such complexity would be a task in which stimuli cannot be controlled by the BCI, as discussed in section 4.4. This would require the use of event-detection algorithms [106, 107], and the development and testing of feedback methods that do not involve controlling the stimulus itself as discussed in section 4.4. Future work can explore the use of dynamical modeling methods in the development of decoding algorithms to improve cognitive state decoding [108–116]. Further, adaptively tracking the changes in EEG signals over time or detecting switches in this activity can further improve the performance of such BCIs [117–124].