Abstract
Despite the increasing popularity of ambulatory assessment, the reliability and validity of psychophysiological signals from wearable devices is unproven in daily life settings. We evaluated the reliability and validity of physiological signals (electrocardiogram, ECG; photoplethysmography, PPG; electrodermal activity, EDA) collected from two wearable devices (Movisens EcgMove4 and Empatica E4) in the lab (N = 67) and daily life (N = 20) among adults aged 18–64 with Mindware as the laboratory gold standard. Results revealed that both wearable devices’ valid data rates in daily life were lower than in the laboratory (Movisens ECG 82.94 vs. 93.10%, Empatica PPG 8.79 vs. 26.14%, and Empatica EDA 41.16 vs. 42.67%, respectively). The poor valid data rates of Empatica PPG signals in the laboratory could be partially attributed to participants' hand movements (r = – .27, p = .03). In laboratory settings, heart rate (HR) derived from both wearable devices exhibited higher concurrent validity than heart rate variability (HRV) metrics (ICCs 0.98–1.00 vs. 0.75–0.97). The number of skin conductance responses (SCRs) derived from Empatica showed higher concurrent validity than skin conductance level (SCL, ICCs 0.38 vs. 0.09). Movisens EcgMove4 provided more reliable and valid HRV measurements than Empatica E4 in both laboratory (split-half reliability: 0.95–0.99 vs. 0.85–0.98; concurrent validity: 0.95–1.00 vs. 0.75–0.98; valid data rate: 93.10 vs. 26.14%) and ambulatory settings (split-half reliability: 0.99–1.00 vs. 0.89–0.98; valid data rate: 82.94 vs. 8.79%). Although the reliability and validity of wearable devices are improving, findings suggest researchers should select devices that yield consistently robust and valid data for their measures of interest.
Similar content being viewed by others
Notes
We abbreviated 'Empatica E4' as 'Empatica' throughout this paper for brevity, but it should be noted that our findings are based on the E4 model that we tested and may not necessarily apply to other products from Empatica.
References
Acharya, R. U., Joseph, P. K., Kannathal, N., Lim, C. M., & Suri, J. S. (2006). Heart rate variability: A review. Medical & Biological Engineering & Computing, 44(12), 1031–1051. https://doi.org/10.1007/s11517-006-0119-0
Avelar, D., Dore, R. A., Schwichtenberg, A. J., Roben, C. K., Hirsh-Pasek, K., & Golinkoff, R. M. (2022). Children and parents’ physiological arousal and emotions during shared and independent e-book reading: A preliminary study. International Journal of Child-Computer Interaction, 33, 100507.
Battalio, S. L., Conroy, D. E., Dempsey, W., Liao, P., Menictas, M., Murphy, S., Nahum-Shani, I., Qian, T., Kumar, S., & Spring, B. (2021). Sense2Stop: a micro-randomized trial using wearable sensors to optimize a just-in-time-adaptive stress management intervention for smoking relapse prevention. Contemporary Clinical Trials, 109, 106534.
Benedek, M., & Kaernbach, C. (2010). A continuous measure of phasic electrodermal activity. Journal of Neuroscience Methods, 190(1), 80–91. https://doi.org/10.1016/j.jneumeth.2010.04.028
Béres, S., Holczer, L., & Hejjel, L. (2019). On the minimal adequate sampling frequency of the photoplethysmogram for pulse rate monitoring and heart rate variability analysis in mobile and wearable technology. Measurement Science Review, 19(5), 232–240. https://doi.org/10.2478/msr-2019-0030
Bernston, G. G., Quigley, K. S., & Lozano, D. (2007). Cardiovascular psychophysiology. Handbook of psychophysiology (pp. 182–210). Cambridge University Press.
Berntson, G. G., Bigger, J. T., Jr., Eckberg, D. L., Grossman, P., Kaufmann, P. G., Malik, M., Nagaraja, H. N., Porges, S. W., Saul, J. P., Stone, P. H., & van der Molen, M. W. (1997). Heart rate variability: origins, methods, and interpretive caveats. Psychophysiology, 34(6), 623–648. https://doi.org/10.1111/j.1469-8986.1997.tb02140.x
Bertsch, K., Hagemann, D., Naumann, E., Schächinger, H., & Schulz, A. (2012). Stability of heart rate variability indices reflecting parasympathetic activity. Psychophysiology, 49(5), 672–682. https://doi.org/10.1111/j.1469-8986.2011.01341.x
Bland, J. M., & Altman, D. (1986). Statistical methods for assessing agreement between two methods of clinical measurement. The Lancet, 327(8476), 307–310.
Bland, J. M., & Altman, D. G. (1999). Measuring agreement in method comparison studies. Statistical Methods in Medical Research, 8(2), 135–160. https://doi.org/10.1191/096228099673819272
Boucsein, W., Fowles, D. C., Grimnes, S., Ben-Shakhar, G., Roth, W. T., Dawson, M. E., Filion, D. L., Society for Psychophysiological Research Ad Hoc Committee on Electrodermal, M. (2012). Publication recommendations for electrodermal measurements. Psychophysiology, 49(8), 1017–1034. https://doi.org/10.1111/j.1469-8986.2012.01384.x
Carpenter, S. M., Yap, J., Patrick, M. E., Morrell, N., Dziak, J. J., Almirall, D., Yoon, C., & Nahum-Shani, I. (2023). Self-relevant appeals to engage in self-monitoring of alcohol use: A microrandomized trial. Psychology of Addictive Behaviors, 37(3), 434–446. https://doi.org/10.1037/adb0000855
Carrier, B., Barrios, B., Jolley, B. D., & Navalta, J. W. (2020). Validity and reliability of physiological data in applied settings measured by wearable technology: A rapid systematic review. Technologies, 8(4). https://doi.org/10.3390/technologies8040070
Carrier, B. (2021). Assessing the validity and reliability of several heart rate monitors in wearable technology while mountain biking (Order No. 28546216) [Master’s thesis, University of Nevada, Las Vegas]. ProQuest Dissertations & Theses Global. (2581487999)
Castro-Garcia, J. A., Molina-Cantero, A. J., Merino-Monge, M., & Gomez-Gonzalez, I. M. (2019). An open-source hardware acquisition platform for physiological measurements. IEEE Sensors Journal, 19(23), 11526–11534. https://doi.org/10.1109/jsen.2019.2933917
Choi, A., & Shin, H. (2017). Photoplethysmography sampling frequency: pilot assessment of how low can we go to analyze pulse rate variability with reliability? Physiological Measurement, 38(3), 586–600. https://doi.org/10.1088/1361-6579/aa5efa
Chow, H.-W., & Yang, C.-C. (2020). accuracy of optical heart rate sensing technology in wearable fitness trackers for young and older adults: Validation and comparison study. JMIR mHealth and uHealth, 8(4), e14707. https://doi.org/10.2196/14707
Dawson, M. E., Schell, A. M., & Filion, D. L. (2017). The electrodermal system. Handbook of psychophysiology (pp. 217–243). Cambridge University Press.
Ebner-Priemer, U. W., & Trull, T. J. (2009). Ecological momentary assessment of mood disorders and mood dysregulation. Psychol Assess, 21(4), 463–475. https://doi.org/10.1037/a0017075
Escobar-Linero, E., Munoz-Saavedra, L., Luna-Perejon, F., Sevillano, J. L., & Dominguez-Morales, M. (2023). Wearable Health Devices for Diagnosis Support: Evolution and Future Tendencies. Sensors (Basel), 23(3). https://doi.org/10.3390/s23031678
Euser, A. M., Dekker, F. W., & le Cessie, S. (2008). A practical approach to Bland-Altman plots and variation coefficients for log transformed variables. Journal of Clinical Epidemiology, 61(10), 978–982. https://doi.org/10.1016/j.jclinepi.2007.11.003
Fine, J., Branan, K. L., Rodriguez, A. J., Boonya-Ananta, T., Ajmal, Ramella-Roman, J. C., McShane, M. J., & Cote, G. L. (2021). Sources of Inaccuracy in Photoplethysmography for Continuous Cardiovascular Monitoring. Biosensors (Basel), 11(4). https://doi.org/10.3390/bios11040126
Fowles, D. C., Christie, M. J., Edelberg, R., Grings, W. W., Lykken, D. T., & Venables, P. H. (1981). Publication recommendations for electrodermal measurements. Psychophysiology, 18(3), 232–239. https://doi.org/10.1111/j.1469-8986.1981.tb03024.x
Fuller, D., Colwell, E., Low, J., Orychock, K., Tobin, M. A., Simango, B., Buote, R., Van Heerden, D., Luan, H., Cullen, K., Slade, L., & Taylor, N. G. A. (2020). Reliability and validity of commercially available wearable devices for measuring steps, energy expenditure, and heart rate: Systematic review. JMIR mHealth and uHealth, 8(9), e18694. https://doi.org/10.2196/18694
Georgiou, K., Larentzakis, A. V., Khamis, N. N., Alsuhaibani, G. I., Alaska, Y. A., & Giallafos, E. J. (2018). Can wearable devices accurately measure heart rate variability? A systematic review. Folia Med (Plovdiv), 60(1), 7–20. https://doi.org/10.2478/folmed-2018-0012
Gorman, J. M., & Sloan, R. P. (2000). Heart rate variability in depressive and anxiety disorders. American Heart Journal, 140(4), 577–583.
Gruden, T., Stojmenova, K., Sodnik, J., & Jakus, G. (2019). Assessing Drivers’ Physiological Responses Using Consumer Grade Devices. Applied Sciences, 9(24). https://doi.org/10.3390/app9245353
Kasos, K., Zimonyi, S., Kasos, E., Lifshitz, A., Varga, K., & Szekely, A. (2018). Does the Electrodermal System “Take Sides” When It Comes to Emotions? Applied Psychophysiology and Biofeedback, 43(3), 203–210. https://doi.org/10.1007/s10484-018-9398-0
Kemp, A. H., Quintana, D. S., Felmingham, K. L., Matthews, S., & Jelinek, H. F. (2012). Depression, Comorbid Anxiety Disorders, and Heart Rate Variability in Physically Healthy, Unmedicated Patients: Implications for Cardiovascular Risk. PLoS One, 7(2), e30777. https://doi.org/10.1371/journal.pone.0030777
King, A. C., Hekler, E. B., Grieco, L. A., Winter, S. J., & Sheats, J. L. et al. (2013) Harnessing different motivational frames via mobile phones to promote daily physical activity and reduce sedentary behavior in aging adults. PLoS One 8(4), e62613. https://doi.org/10.1371/journal.pone.0062613
Kleckner, I. R., Jones, R. M., Wilder-Smith, O., Wormwood, J. B., Akcakaya, M., Quigley, K. S., Lord, C., & Goodwin, M. S. (2018). Simple, transparent, and flexible automated quality assessment procedures for ambulatory electrodermal activity data. IEEE Transactions on Biomedical Engineering, 65(7), 1460–1467. https://doi.org/10.1109/Tbme.2017.2758643
Kleckner, I. R., Feldman, M. J., Goodwin, M. S., & Quigley, K. S. (2021). Framework for selecting and benchmarking mobile devices in psychophysiological research. Behav Res Methods, 53(2), 518–535. https://doi.org/10.3758/s13428-020-01438-9
Kunkels, Y. K., van Roon, A. M., Wichers, M., & Riese, H. (2021). Cross-instrument feasibility, validity, and reproducibility of wireless heart rate monitors: Novel opportunities for extended daily life monitoring. Psychophysiology, 58(10), e13898. https://doi.org/10.1111/psyp.13898
Kuppens, P., & Verduyn, P. (2017). Emotion dynamics. Current Opinion in Psychology, 17, 22–26. https://doi.org/10.1016/j.copsyc.2017.06.004
Licht, C. M. M., de Geus, E. J. C., Zitman, F. G., Hoogendijk, W. J. G., van Dyck, R., & Penninx, B. W. J. H. (2008). Association between major depressive disorder and heart rate variability in the netherlands study of depression and anxiety (NESDA). Archives of General Psychiatry, 65(12), 1358–1367. https://doi.org/10.1001/archpsyc.65.12.1358
Licht, C. M. M., de Geus, E. J. C., van Dyck, R., & Penninx, B. W. J. H. (2009). Association between Anxiety Disorders and Heart Rate Variability in The Netherlands Study of Depression and Anxiety (NESDA). Psychosomatic Medicine, 71(5), 508–518. https://doi.org/10.1097/PSY.0b013e3181a292a6
Mauss, I. B., & Robinson, M. D. (2009). Measures of emotion: A review. Cognition & Emotion, 23(2), 209–237. https://doi.org/10.1080/02699930802204677
McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlations coefficients (vol 1, pg 30, 1996). Psychological Methods, 1(4), 390–390. https://doi.org/10.1037//1082-989x.1.4.390
McVay, J. C., & Kane, M. J. (2013). Dispatching the wandering mind? Toward a laboratory method for cuing “spontaneous” off-task thought. Frontiers in Psychology, 4, 570. https://doi.org/10.3389/fpsyg.2013.00570
Menghini, L., Gianfranchi, E., Cellini, N., Patron, E., Tagliabue, M., & Sarlo, M. (2019). Stressing the accuracy: Wrist-worn wearable sensor validation over different conditions. Psychophysiology, 56(11), e13441. https://doi.org/10.1111/psyp.13441
Menghini, L., Cellini, N., Goldstone, A., Baker, F. C., & de Zambotti, M. (2021). A standardized framework for testing the performance of sleep-tracking technology: step-by-step guidelines and open-source code. Sleep, 44(2). https://doi.org/10.1093/sleep/zsaa170
Milstein, N., & Gordon, I. (2020). Validating measures of electrodermal activity and heart rate variability derived from the empatica E4 utilized in research settings that involve interactive dyadic states. Frontiers in Behavioral Neuroscience, 14, 148. https://doi.org/10.3389/fnbeh.2020.00148
Montes, J., & Navalta, J. W. (2019). Reliability of the Polar T31 Uncoded heart rate monitor in free motion and treadmill activities. International Journal of Exercise Science, 12(4), 69–76. https://www.ncbi.nlm.nih.gov/pubmed/31156753.
Nahum-Shani, I., Potter, L. N., Lam, C. Y., Yap, J., Moreno, A., Stoffel, R., Wu, Z., Wan, N., Dempsey, W., & Kumar, S. (2021a). The mobile assistance for regulating smoking (MARS) micro-randomized trial design protocol. Contemporary Clinical Trials, 110, 106513.
Nahum-Shani, I., Rabbi, M., Yap, J., Philyaw-Kotov, M. L., Klasnja, P., Bonar, E. E., Cunningham, R. M., Murphy, S. A., & Walton, M. A. (2021b). Translating strategies for promoting engagement in mobile health: A proof-of-concept microrandomized trial. Health Psychology, 40(12), 974–987. https://doi.org/10.1037/hea0001101
Navalta, J. W., Ramirez, G. G., Maxwell, C., Radzak, K. N., & McGinnis, G. R. (2020). Validity and reliability of three commercially available smart sports bras during treadmill walking and running. Scientific Reports, 10(1), 7397. https://doi.org/10.1038/s41598-020-64185-z
Nunan, D., Jakovljevic, D. G., Donovan, G., Hodges, L. D., Sandercock, G. R. H., & Brodie, D. A. (2008). Levels of agreement for RR intervals and short-term heart rate variability obtained from the Polar S810 and an alternative system. European Journal of Applied Physiology, 103(5), 529–537. https://doi.org/10.1007/s00421-008-0742-6
Ollander, S., Godin, C., Campagne, A., & Charbonnier, S. (2016) A comparison of wearable and stationary sensors for stress detection. 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC), Budapest, Hungary, 2016, pp. 004362–004366. https://doi.org/10.1109/SMC.2016.7844917
Ottaviani, C., Shapiro, D., & Couyoumdjian, A. (2013). Flexibility as the key for somatic health: From mind wandering to perseverative cognition. Biological Psychology, 94(1), 38–43. https://doi.org/10.1016/j.biopsycho.2013.05.003
Paniccia, M., Paniccia, D., Thomas, S., Taha, T., & Reed, N. (2017). Clinical and non-clinical depression and anxiety in young people: A scoping review on heart rate variability. Autonomic Neuroscience-Basic & Clinical, 208, 1–14. https://doi.org/10.1016/j.autneu.2017.08.008
Pietilä, J. et al. (2018). Evaluation of the accuracy and reliability for photoplethysmography based heart rate and beat-to-beat detection during daily activities. In: Eskola, H., Väisänen, O., Viik, J., Hyttinen, J. (eds) EMBEC & NBC 2017. IFMBE Proceedings, vol 65. Springer, Singapore. https://doi.org/10.1007/978-981-10-5122-7_37
Raugh, I. M., Chapman, H. C., Bartolomeo, L. A., Gonzalez, C., & Strauss, G. P. (2019). A comprehensive review of psychophysiological applications for ecological momentary assessment in psychiatric populations. Psychological Assessment, 31(3), 304–317. https://doi.org/10.1037/pas0000651
Robertson, I. H., Manly, T., Andrade, J., Baddeley, B. T., & Yiend, J. (1997). “Oops!”: Performance correlates of everyday attentional failures in traumatic brain injured and normal subjects. Neuropsychologia, 35(6), 747–758. https://doi.org/10.1016/S0028-3932(97)00015-8
Sandercock, G. R., Bromley, P. D., & Brodie, D. A. (2005). The reliability of short-term measurements of heart rate variability. International Journal of Cardiology, 103(3), 238–247. https://doi.org/10.1016/j.ijcard.2004.09.013
Schandry, R. (1981). Heart Beat Perception and Emotional Experience. Psychophysiology, 18(4), 483–488. https://doi.org/10.1111/j.1469-8986.1981.tb02486.x
Shaffer, F., & Ginsberg, J. P. (2017). An overview of heart rate variability metrics and norms. Frontiers in public health, 5, 290215. https://doi.org/10.3389/fpubh.2017.00258
Spruijt-Metz, D., & Nilsen, W. (2014). Dynamic models of behavior for just-in-time adaptive interventions. IEEE Pervasive Computing, 13(3), 13–17. https://doi.org/10.1109/MPRV.2014.46
Stange, J. P., Kleiman, E. M., Mermelstein, R. J., & Trull, T. J. (2019). Using ambulatory assessment to measure dynamic risk processes in affective disorders. Journal of Affective Disorders, 259, 325–336. https://doi.org/10.1016/j.jad.2019.08.060
Stasik-O’Brien, S. M., Brock, R. L., Chmielewski, M., Naragon-Gainey, K., Koffel, E., McDade-Montez, E., O’Hara, M. W., & Watson, D. (2019). Clinical utility of the inventory of depression and anxiety symptoms (IDAS). Assessment, 26(5), 944–960. https://doi.org/10.1177/1073191118790036
Tarvainen, M. P., Niskanen, J. P., Lipponen, J. A., Ranta-Aho, P. O., & Karjalainen, P. A. (2014). Kubios HRV–heart rate variability analysis software. Computer Methods and Programs in Biomedicine, 113(1), 210–220. https://doi.org/10.1016/j.cmpb.2013.07.024
Task Force of the European Society of Cardiology and the North American Society of Pacing and Electrophysiology. (1996). Heart rate variability: standards of measurement, physiological interpretation and clinical use. Circulation, 93(5), 1043. https://www.ncbi.nlm.nih.gov/pubmed/8598068.
Tropschuh, B., Windecker, S., & Reinhart, G. (2022). Study-based evaluation of accuracy and usability of wearable devices in manual assembly. Production & Manufacturing Research, 10(1), 569–582. https://doi.org/10.1080/21693277.2022.2100505
Thomas, J. G., & Bond, D. S. (2015). Behavioral response to a just-in-time adaptive intervention (JITAI) to reduce sedentary behavior in obese adults: Implications for JITAI optimization. Health Psychology, 34(S), 1261.
Troy, A. S., Wilhelm, F. H., Shallcross, A. J., & Mauss, I. B. (2010). Seeing the silver lining: Cognitive reappraisal ability moderates the relationship between stress and depressive symptoms. Emotion, 10(6), 783–795. https://doi.org/10.1037/a0020262
Umair, M., Chalabianloo, N., Sas, C., & Ersoy, C. (2021). HRV and Stress: A mixed-methods approach for comparison of wearable heart rate sensors for biofeedback. IEEE Access, 9, 14005–14024. https://doi.org/10.1109/access.2021.3052131
van der Mee, D. J., Gevonden, M. J., Westerink, J., & de Geus, E. J. C. (2021). Validity of electrodermal activity-based measures of sympathetic nervous system activity from a wrist-worn device. International Journal of Psychophysiology, 168, 52–64. https://doi.org/10.1016/j.ijpsycho.2021.08.003
van Lier, H. G., Pieterse, M. E., Garde, A., Postel, M. G., de Haan, H. A., Vollenbroek-Hutten, M. M. R., Schraagen, J. M., & Noordzij, M. L. (2020). A standardized validity assessment protocol for physiological signals from wearable technology: Methodological underpinnings and an application to the E4 biosensor. Behavior Research Methods, 52(2), 607–629. https://doi.org/10.3758/s13428-019-01263-9
Van Voorhees, E. E., Dennis, P. A., Watkins, L. L., Patel, T. A., Calhoun, P. S., Dennis, M. F., & Beckham, J. C. (2022). Ambulatory heart rate variability monitoring: Comparisons between the Empatica E4 Wristband and Holter Electrocardiogram. Psychosomatic Medicine, 84(2), 210–214. https://doi.org/10.1097/PSY.0000000000001010
Watson, D., O’Hara, M. W., Simms, L. J., Kotov, R., & Chmielewski, M. (2007). Development and validation of the inventory of depression and anxiety symptoms (IDAS). Psychological Assessment, 19(3), 253–268. https://doi.org/10.1037/1040-3590.19.3.253
Weippert, M., Kumar, M., Kreuzfeld, S., Arndt, D., Rieger, A., & Stoll, R. (2010). Comparison of three mobile devices for measuring R-R intervals and heart rate variability: Polar S810i, Suunto t6 and an ambulatory ECG system. European Journal of Applied Physiology, 109(4), 779–786. https://doi.org/10.1007/s00421-010-1415-9
Welk, G. J., Bai, Y., Lee, J. M., Godino, J., Saint-Maurice, P. F., & Carr, L. (2019). Standardizing analytic methods and reporting in activity monitor validation studies. Medicine and Science in Sports and Exercise, 51(8), 1767–1780. https://doi.org/10.1249/MSS.0000000000001966
Wilhelm, F. H., & Grossman, P. (2010). Emotions beyond the laboratory: Theoretical fundaments, study design, and analytic strategies for advanced ambulatory assessment. Biological Psychology, 84(3), 552–569. https://doi.org/10.1016/j.biopsycho.2010.01.017
Williams, D. P., Jarczok, M. N., Ellis, R. J., Hillecke, T. K., Thayer, J. F., & Koenig, J. (2017). Two-week test–retest reliability of the PolarRS800CX™ to record heart rate variability. Clinical Physiology and Functional Imaging, 37(6), 776–781. https://doi.org/10.1111/cpf.12321
Zhang, Y., Weaver, R. G., Armstrong, B., Burkart, S., Zhang, S., & Beets, M. W. (2020). Validity of Wrist-Worn photoplethysmography devices to measure heart rate: A systematic review and meta-analysis. Journal of Sports Sciences, 38(17), 2021–2034. https://doi.org/10.1080/02640414.2020.1767348
Zheng, Y., & Poon, C. C. Y. (2016). Wearable devices and their applications in surgical robot control and p-medicine. 2016 IEEE 20th International Conference on Computer Supported Cooperative Work in Design (CSCWD) (pp. 659–663). IEEE.
Author Note
This research was supported by the National Institute of Mental Health (R01MH118218). During preparation of this manuscript, Dr. Northrup was also supported by K23MH127420. The authors have no conflicts of interest to report.
Author information
Authors and Affiliations
Corresponding author
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Open Practice Statement
The index-level data analyzed in the current study are available in the OSF repository, https://osf.io/28usc/?view_only=6508177708fb4ef380bdbf5dd24b0b1b. As this was a validation study, no hypotheses were pre-registered.
Supplementary Information
Below is the link to the electronic supplementary material.
Rights and permissions
Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.
About this article
Cite this article
Hu, X., Sgherza, T.R., Nothrup, J.B. et al. From lab to life: Evaluating the reliability and validity of psychophysiological data from wearable devices in laboratory and ambulatory settings. Behav Res (2024). https://doi.org/10.3758/s13428-024-02387-3
Accepted:
Published:
DOI: https://doi.org/10.3758/s13428-024-02387-3