Detecting beats in the photoplethysmogram: benchmarking open-source algorithms

The datasets used in this study are summarised in table 1, and are now described.

For each dataset, the table indicates the duration of recordings and the total number of beats used in the analysis (shown for the MPSTD beat detector).

2.1.1. Hospital monitoring

2.1.2. Wearable data

First, any PPG signals sampled at over 100 Hz were resampled at this frequency to reduce the time for computational analysis. For signals sampled at multiples of 100 Hz, this was performed using downsampling, and for other signals it was performed using resampling with an antialiasing lowpass filter. Second, signals were band-pass filtered between 0.67 and 8.0 Hz to eliminate non-cardiac frequencies. Third, beats were detected using fifteen open-source PPG beat detectors in turn, as demonstrated for two beat detectors in figure 1. The beat detectors are described in table 2. Beat detection was performed on 20 s windows of PPG signal, overlapping by 5 s. Repeated beat detections due to overlapping windows were eliminated. This approach ensured that beat detectors were not penalised for missing beats at the start or end of a window. Fourth, windows were excluded if they contained a flat line lasting more than 0.2 s (typically caused by sensor disconnection or signal 'clipping'). The beat detectors are available in (Charlton 2022a).

Zoom In Zoom Out Reset image size

For consistency, each beat detector's annotations were used to obtain the corresponding middle-amplitude point of the systolic upslope on each detected PPG pulse wave (Peralta et al 2019), which was used for analysis. This point has been found to provide more accurate timings than peaks or onsets (Peralta et al 2019).

The CapnoBase and PPG-DaLiA datasets contain manual beat annotations which were used as reference beats. In the remaining datasets reference beats were obtained from simultaneous ECG signals by: (i) detecting beats using two separate ECG beat detectors; (ii) identifying 'correct' beats as those which both beat detectors detected within 150 ms of each other; and (iii) excluding from the analysis any 20 s windows in which the two beat detectors did not agree. The two beat detectors were: the 'jqrs' ECG beat detector, which is based on the Pan and Tompkins method (Behar et al 2014, Johnson et al 2014) and the 'rpeakdetect' ECG beat detector (Clifford).

PPG and ECG signals were not necessarily precisely aligned, so the timings of PPG-derived beats and reference ECG-derived beats were aligned as follows. The time difference between each ECG-derived beat and its closest PPG-derived beat was calculated. Those ECG-derived beats for which the absolute time difference was <150 ms were determined to be correctly identified. This process was repeated when offsetting the beats by lags of −10 to 10 s, in increments of 20 ms. The lag which resulted in the highest proportion of beats being correctly identified was accepted as the true lag and used to synchronise the timings of beats. Figure 2(a) shows an example of this time-alignment.

Zoom In Zoom Out Reset image size

The ability of beat detectors to detect beats was assessed by comparing PPG-derived beats with reference beats. Reference beats were determined to be correctly identified if the closest PPG-derived beat was within ±150 ms of a reference beat, as shown in figure 2(b). For each recording, the numbers of reference beats (n_ref), PPG-derived beats (n_PPG), and correctly identified beats (n_correct) were used to calculate the following:

$$\begin{eqnarray}&&{\rm{s}}{ensitivity}( \% ),\quad \mathrm{Se}=\displaystyle \frac{{n}_{\mathrm{correct}}}{{n}_{\mathrm{ref}}}\times 100\end{eqnarray} \tag{ 1 }$$

$$\begin{eqnarray}&&{\rm{p}}{ositive}\,{predictive}\,{value}( \% ),\quad \mathrm{PPV}=\displaystyle \frac{{n}_{\mathrm{correct}}}{{n}_{\mathrm{PPG}}}\times 100\end{eqnarray} \tag{ 2 }$$

$$\begin{eqnarray}&&{F}_{1}\,\mathrm{Score}( \% ),\quad {{\rm{F}}}_{1}=\displaystyle \frac{2\times \mathrm{PPV}\times \mathrm{Se}}{\mathrm{PPV}+\mathrm{Se}}\times 100\end{eqnarray} \tag{ 3 }$$

Beat detectors were ranked according to the F₁ score, which is the harmonic mean of sensitivity and PPV.

The accuracy of PPG-derived heart rates (HRs) was assessed by comparing PPG-derived HRs to reference ECG-derived HRs. A HR (in beats per minute, bpm) was calculated at the time of each PPG-derived beat, from the number of PPG-derived beats in the preceding 8 s window (n_beats ), as

$$\begin{eqnarray}&&{HR}=60\times \displaystyle \frac{{n}_{\mathrm{beats}}-1}{t({n}_{\mathrm{beats}})-t(1)}\end{eqnarray} \tag{ 4 }$$

where t denotes the times of PPG-derived beats. Each HR signal was interpolated using sample-and-hold interpolation at 50 Hz. Performance was assessed as the mean absolute percentage error (MAPE) between time series. A median MAPE of <10% was deemed to be acceptable for HR monitoring. This was based on the acceptable limits of ±10% stated in the AAMI standard (ANSI/AAMI 2002) and implemented using the MAPE statistic in (Consumer Technology Association 2018), although we note that the true threshold of acceptability is likely to vary between applications (Mühlen et al 2021).

Performance statistics are reported as median (25th–75th percentiles). The Wilcoxon rank sum test was used to compare performances between groups, at a significance level of α = 0.05. A Holm-Sidak correction was made to correct for multiple comparisons.

The performance of beat detectors is presented in figure 3 using the F₁ score, and in figure 4 using the HR MAPE. Additional results are provided in appendix A for sensitivity and PPV (figures A1 and A2 respectively). The key findings are as follows.

Zoom In Zoom Out Reset image size

First, eight beat detectors performed very well across all datasets with low levels of movement: AMPD, MSPTD, qppg, PWD, ERMA, SPAR, ABD, and HeartPy. These had median F₁ scores of: ≥99% on the hospital monitoring datasets containing high-quality data (CapnoBase and BIDMC); ≥90% on the hospital monitoring datasets containing real-world data (MIMIC PERform Training and Testing Datasets); and ≥90% on the wearable datasets with low levels of movement (WESAD (meditation) and PPG-DaLiA (sitting)). The remainder of the Results will focus on these eight beat detectors. Figure 5(a) shows an example of (mostly) accurate beat detection during low levels of movement. Of note, the Pulses beat detector performed less well on the PPG-DaLiA (sitting) dataset because its assumed duration of the systolic upslope was no longer valid in these wrist signals acquired at rest.

Second, performance decreased during activities associated with more movement. The eight beat detectors which performed well on data with low levels of movement had median F₁ scores of 93%–96% on PPG-DaLiA (sitting). This performance decreased to 70%–91% on PPG-DaLiA (cycling), 60%–77% on PPG-DaLiA (walking), and 55%–72% on PPG-DaLiA (stair climbing). Performance was also poorer during stress, as shown by median F₁ scores of 59%–70% on WESAD (stress) compared to 71%–80% on WESAD (baseline). This was primarily due to beat detectors missing beats, rather than falsely detecting beats, as shown by the generally lower sensitivities than positive predictive values on PPG-DaLiA (walking) and WESAD (stress) datasets (see appendix A, figures A1 and A2). Figures 5(b)–(d) show examples of beat detection during movement.

Third, the variability in performance between subjects was low during activities associated with low levels of movement, as shown by the relatively low inter-quartile ranges of F₁ scores (indicated by the heights of boxes) on WESAD (meditation) and PPG-DaLiA (sitting). However, performance varied much more between subjects in more challenging datasets, e.g., WESAD (stress) and PPG-DaLiA (walking).

To identify the best-performing beat detectors, we focused on results from the MIMIC PERform Testing and PPG-DaLiA (working) datasets, since these are representative of real-world performance in critical care and daily life respectively. On the MIMIC PERform Testing Dataset, the top scoring beat detectors were MSPTD, AMPD, gppq, ABD, and Pulses (all with F₁ scores of 96.6%–97.5%, whereas the remainder scored ≤ 95.6%). On PPG-DaLiA (working), the top scorers were PWD, MPSTD, AMPD, ABD, gppq, and WFD (all with F₁ scores of 80.0%–81.4%, whereas the remainder scored <79.0%). In addition, MSPTD was the best performing beat detector on 5 out of the 12 WESAD and PPG-DaLiA datasets, and qppg was the best performing beat detector on 4 of these datasets. Therefore, we suggest that MSPTD and qppg performed best, although we note that this is subjective, and that some other beat detectors also performed well (notably ABD and AMPD).

The best-performing beat detectors have complementary performance characteristics: MSPTD tended to have a higher positive predictive value, whereas qppg tended to have higher sensitivity (see appendix A, figures A1 and A2). Figure 5 shows examples of this: qppg sometimes detected additional beats during noise (see figure 5(a) at 0.5 s), whereas MSPTD sometimes missed beats (see figure 5(h)).

The performance of beat detectors was deemed to be acceptable for HR monioring in some use cases but not others (see figure 4). All eight beat detectors which had been found to perform well on data with low levels of movement also had acceptable HR MAPEs of <10% on datasets associated with low and moderate levels of movement (the hospital monitoring datasets, and WESAD (meditation, amusement, baseline) and PPG-DaLiA (sitting, working)). At least some of these beat detectors did not perform acceptably on each of the remaining datasets. None of the eight beat detectors produced acceptable HR errors during stress (see WESAD (stress)). Five of the eight beat detectors (MSPTD, qppg, ABD, AMPD, and ERMA) also produced acceptable errors during less intensive activities (PPG-DaLiA (lunch break), and PPG-DaLiA (car driving)). Only qppg performed acceptably on PPG-DaLiA (cycling). None of the beat detectors performed acceptably during more intensive exercise (PPG-DaLiA (walking), PPG-DaLiA (stair climbing), and PPG-DaLiA (table soccer)).

Zoom In Zoom Out Reset image size

The associations between beat detector performance and the assessed factors are shown in figure 6.

The performance of beat detectors was poorer in AF (figure 6(a)). The eight beat detectors which performed well at rest achieved F₁ scores between 99.4%–99.7% in sinus rhythm (non-AF), compared to 91.8%–97.1% in AF. This was primarily because beat detectors missed beats during AF (see appendix B, figures A3(a) and A4(a)), similarly to performance in movement. Performance was worse in AF subjects than non-AF subjects for all eight beat detectors at the 5% significance level, and four of these differences remained significant after accounting for multiple comparisons (0.2% significance level).

All eight beat detectors performed worse on neonates than adults, as shown in (figure 6(b)). Seven of these differences remained significant after accounting for multiple comparisons. The eight beat detectors achieved F₁ scores between 97.8%–98.5% in adults compared to 84.2%–95.9% in neonates. These beat detectors missed beats, as shown by their lower sensitivities (see appendix B, figure A3(b)). The lower performance in neonates may be because the neonatal PPG signals were of lower quality, as shown by them having lower SNRs (−10.9 (−12.2 to −8.8) dBc in neonates compared to −5.9 (−9.6 to −1.6) dBc in adults). In addition, some beat detectors may have been designed for use with adults, who typically have HRs between 60 and 100 bpm, whereas neonates typically have HRs between 110 and 160 bpm (Fleming et al 2011).

Zoom In Zoom Out Reset image size

Five of the eight beat detectors had lower F₁ scores on White subjects than Black subjects, as shown in (figure 6(c)), although none of these differences were significant after accounting for multiple comparisons.

Zoom In Zoom Out Reset image size

Table 4 presents the proposed assessment framework. The MIMIC PERform datasets are recommended for developing and testing algorithms, and for comparing performance between adults and neonates. Out of the wearable datasets, WESAD is recommended for training and PPG-DaLiA for testing, as the latter allows performance to be assessed during several activities of daily living. The MIMIC PERform AF Dataset is recommended for assessing performance in AF, although it would benefit from inclusion of additional subjects in the future. The CapnoBase and BIDMC datasets were designated as 'preliminary design' datasets as all beat detectors achieved F₁ scores of >93% on these datasets, so it is unlikely they could be used to substantially improve beat detector design.