Skip to main content
SpringerLink
Log in

Self-report measures for the assessment of human–machine interfaces in automated driving

  • Original Article
  • Published:
Cognition, Technology & Work Aims and scope Submit manuscript

Abstract

For a successful market introduction of Level 3 Automated Driving Systems (L3 ADS), a careful evaluation of human–machine interfaces (HMIs) is necessary. User preference has often focused on usability, user experience, acceptance and trust. However, a thorough evaluation of measures when applied to ADS HMIs is missing. We investigated the appropriateness of nine self-reported measures in terms of reliability and validity. A sample of N = 57 participants completed two 15-min simulator drives with a L3 ADS. They experienced two variations of a HMI that differed in the degree of complying with common guidelines. Consistency analysis identified scales that showed insufficient reliability. Validity examination revealed a three-factorial structure of self-reports for construct validity. These factors are design-orientation, usability-orientation and acceptance-orientation. All measures were sensitive to the HMI manipulation and therefore exhibited criterion-related validity. The present study provides researchers and practitioners in the area of ADS with a recommendation for self-report measure application.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3

Similar content being viewed by others

References

  • AdaptIVe Consortium (2017) Final functional human factors recommendations (Deliverable D3.3)

  • Adell E (2010) Acceptance of driver support systems. Proc Eur Conf Hum Centered Design Intell Transp Syst 2:475–486

    Google Scholar 

  • Adell E, Nilsson L, Várhelyi A (2014) How is acceptance measured? Overview of measurement issues, methods and tools. In: Horberry T, Regan MA, Stevens A (eds) Driver acceptance of new technology theory measurement and optimisation. CRC Press, London, UK, pp 73–89

    Google Scholar 

  • Anthoine E, Moret L, Regnault A, Sébille V, Hardouin J-B (2014) Sample size used to validate a scale: a review of publications on newly-developed patient reported outcomes measures. Health Qual Life Outcomes 12:176. https://doi.org/10.1186/s12955-014-0176-2

    Article  Google Scholar 

  • Auricht M, Stark R, Blume C (2014) Integrating user experience validation into a new engineering development process for advanced driver assistance systems. In: Boyle LN (ed) The 6th international conference of automotive user interfaces and interactive vehicular applications. Seattle, WA, USA

  • Bangor A, Kortum P, Miller J (2009) Determining what individual SUS scores mean: adding an adjective rating scale. J Usability Stud 4(3):114–123

    Google Scholar 

  • Bazilinskyy P, Eriksson A, Petermeijer B, de Winter J (2017) Usefulness and satisfaction of take-over requests for highly automated driving. In: Road safety and simulation international conference (RSS 2017), The Hague, Netherlands

  • Beggiato M, Pereira M, Petzoldt T, Krems JF (2015) Learning and development of trust, acceptance and the mental model of ACC. A longitudinal on-road study. Transp Res Part F Traffic Psychol Behav 35:75–84. https://doi.org/10.1016/j.trf.2015.10.005

    Article  Google Scholar 

  • Brooke J (1996) SUS—a quick and dirty usability scale. Usability Eval Ind 194(189):4–7

    Google Scholar 

  • Bubb H, Bengler K, Grünen RE, Vollrath M (2015) Automobilergonomie. Springer, Berlin

    Book  Google Scholar 

  • Bühner M (2011) Einführung in die Test-und Fragebogenkonstruktion. Pearson Deutschland GmbH, München

    Google Scholar 

  • Campbell DT, Fiske DW (1959) Convergent and discriminant validation by the multitrait-multimethod matrix. Psychol Bull 56(2):81

    Article  Google Scholar 

  • Chien S-Y, Semnani-Azad Z, Lewis M, Sycara K (2014) Towards the development of an inter-cultural scale to measure trust in automation. In: International conference on cross-cultural design

  • Crash Avoidance Metrics Partnership (2016) Automated vehicles research for enhanced safety. NHTSA, Department of Transportation, Washington, DC

    Google Scholar 

  • Cronbach LJ (1951) Coefficient alpha and the internal structure of tests. Psychometrika 16(3):297–334. https://doi.org/10.1007/BF02310555

    Article  MATH  Google Scholar 

  • Cronbach LJ, Meehl PE (1955) Construct validity in psychological tests. Psychol Bull 52(4):281–302. https://doi.org/10.1037/h0040957

    Article  Google Scholar 

  • Davis FD (1985) A technology acceptance model for empirically testing new end-user information systems: theory and results. Massachusetts Institute of Technology, Boston

    Google Scholar 

  • Davis FD (1989) Perceived usefulness, perceived ease of use, and user acceptance of information technology. MIS Quart 13(3):319. https://doi.org/10.2307/249008

    Article  Google Scholar 

  • Eichinger A (2011) Untersuchungskonzepte für die Evaluation von Systemen zur Erkennung des Fahrerzustands: BASt-Forschungsbericht: FE 82.369/2009. Berichte der Bundesanstalt für Straßenwesen 80:45–94

    Google Scholar 

  • Fornell C, Larcker DF (1981) Evaluating structural equation models with unobservable variables and measurement error. J Market Res 18:39–50

    Article  Google Scholar 

  • Forster Y, Naujoks F, Neukum A (2016) Your turn or my turn? Design of a human–machine interface for conditional automation. In: Green P (ed) Proceedings of the 8th international conference on automotive user interfaces and interactive vehicular applications. Ann Arbor, MI, USA

  • Forster Y, Naujoks F, Neukum A (2017) Increasing anthropomorphism and trust in automated driving functions by adding speech output. Intelligent Vehicles Symposium (IV), 2017 IEEE, Redondo Beach, California, USA

  • Forster Y, Kraus J, Feinauer S, Baumann M (2018a) Calibration of trust expectancies in conditionally automated driving by brand, reliability information and introductionary videos: an online study. In: Donmez B, Walker BN, Fröhlich K (Chairs) Proceedings of the 10th international conference on automotive user interfaces and interactive vehicular applications, Toronto, CN

  • Forster Y, Hergeth S, Naujoks F, Krems, JF (2018b) How usability can save the day: methodological considerations for making automated driving a success story. In: Donmez B, Walker BN, Fröhlich K (Chairs) Proceedings of the 10th international conference on automotive user interfaces and interactive vehicular applications, Toronto, CN

  • Forster Y, Hergeth S, Naujoks F, Beggiato M, Krems JF, Keinath A (2019a) Learning and development of mental models in interaction with driving automation: a simulator study. Driving Assessment Conference, Santa Fe, NM, USA

  • Forster Y, Hergeth S, Naujoks F, Beggiato M, Krems JF, Keinath A (2019b) Learning to use automation: behavioral changes in interaction with automated driving systems. Transp Res Part F Traffic Psychol Behav 62:599–614

    Article  Google Scholar 

  • Forster Y, Hergeth S, Naujoks F, Krems JF, Keinath A (2019) Empirical validation of a checklist for heuristic evaluation of automated vehicle HMIs. In: 10th international conference on applied human factors and ergonomics, Washington D.C., USA

  • François M, Osiurak F, Fort A, Crave P, Navarro J (2016) Automotive HMI design and participatory user involvement: review and perspectives. Ergonomics 60(4):541–552

    Article  Google Scholar 

  • Frison A-K, Wintersberger P, Riener A, Schartmüller C (2017) Driving hotzenplotz: a hybrid interface for vehicle control aiming to maximize pleasure in highway driving. In: Boll (ed) Proceedings of the 9th international conference on automotive user interfaces and interactive vehicular applications, Oldenburg, Germany

  • Frison A-K, Wintersberger P, Riener A, Schartmüller C, Boyle LN, Miller E, Weigl K (2019) UX We Trust: investigation of aesthetics and usability of driver-vehicle interfaces and their impact on the perception of automated driving. The 2019 CHI Conference, Glasgow, UK

  • Ghazizadeh M, Lee JD, Boyle LN (2012) Extending the technology acceptance model to assess automation. Cognit Technol Work 14(1):39–49

    Article  Google Scholar 

  • Gold C, Körber M, Hohenberger C, Lechner D, Bengler K (2015) Trust in automation—before and after the experience of take-over scenarios in a highly automated vehicle. Procedia Manufact 3:3025–3032

    Article  Google Scholar 

  • Gold C, Naujoks F, Radlmayr J, Bellem H, Jarosch O (2017) Testing scenarios for human factors research in level 3 automated vehicles. In: International conference on applied human factors and ergonomics, Los Angeles, CA, USA

  • Green P, Levison W, Paelke G, Serafin C (1994) Suggested human factors design guidelines for driver information systems. UMTRI, Michigan

    Google Scholar 

  • Guttman L (1945) A basis for analyzing test–restest reliability. Psychometrika 10:255–282

    Article  MathSciNet  MATH  Google Scholar 

  • Hassenzahl M (2001) The effect of perceived hedonic quality on product appealingness. Int J Hum Comput Interact 13(4):481–499. https://doi.org/10.1207/S15327590IJHC1304_07

    Article  Google Scholar 

  • Hassenzahl M, Burmester M, Koller F (2003) AttrakDiff: Ein Fragebogen zur Messung wahrgenommener hedonischer und pragmatischer Qualität [AttracDiff: A questionnaire to measure perceived hedonic and pragmatic quality]. In: Ziegler J, Szwillus G (eds) Mensch & Computer 2003. Interaktion in Bewegung. B. G. Teubner, Stuttgart, Leipzig, pp 187–196

  • Häuslschmid R, von Bülow M, Pfleging B, Butz A (2017) SupportingTrust in autonomous driving. In: Papadopoulos GA, Kuflik T, Chen F, Duarte C, Fu W-T (Chairs) The 22nd international conference on intelligent user interfaces, Limassol, Cyprus

  • Henrich J, Heine SJ, Norenzayan A (2010) The weirdest people in the world? Behav Brain Sci 33(2–3):61–83. https://doi.org/10.1017/S0140525X0999152X(discussion 83–135)

    Article  Google Scholar 

  • Hergeth S (2016) Automation trust in conditional automated driving systems: approaches to operationalization and design. PhD Thesis. Chemnitz University of Technology. Chemnitz, Germany

  • Hergeth S, Lorenz L, Krems JF (2017) Prior familiarization with takeover requests affects drivers’ takeover performance and automation trust. Hum Factors 59(3):457–470

    Article  Google Scholar 

  • Hoff K, Bashir M (2016) Trust in automation: integrating empirical evidence on factors that influence trust. Hum Factors 57(3):407–434

    Article  Google Scholar 

  • Hornbæk K (2006) Current practice in measuring usability: challenges to usability studies and research. Int J Hum Comput Stud 64(2):79–102. https://doi.org/10.1016/j.ijhcs.2005.06.002

    Article  Google Scholar 

  • Hornbæk K, Law EL-C (2007) Meta-analysis of correlations among usability measures. The 2007 CHI conference, San Jose, California, USA. https://doi.org/10.1145/1240624.1240722

  • International Organization for Standardization (2018) Ergonomics of Human–System Interaction—Part 11: Usability: Definitions and Concepts. Geneva, Switzerland: ISO 9241-11

  • ISO (2012) Road vehicles—ergonomic aspects of transport information and control systems—calibration tasks for methods which assess driver demand due to the use of in-vehicle systems. (ISO, 14198). Geneva, Switzerland

  • Jarosch O, Kuhnt M, Paradies S, Bengler K (2017) It’s out of our hands now! Effects of non-driving related tasks during highly automated driving on drivers’ fatigue. In: Driving Assessment Conference, Manchester Village, Vermont, USA

  • Jian J-Y, Bisantz AM, Drury CG (2000) Foundations for an empirically determined scale of trust in automated systems. Int J Cognit Ergon 4(1):53–71

    Article  Google Scholar 

  • Jones PS, Lee JW, Phillips LR, Zhang XE, Jaceldo KB (2001) An adaptation of Brislin’s translation model for cross-cultural research. Nursing Res 50(5):300–304

    Article  Google Scholar 

  • Kline TJB (1999) The team player inventory: reliability and validity of a measure of predisposition toward organizational team-working environments. J Specialists Group Work 24(1):102–112

    Article  Google Scholar 

  • König M, Neumayr L (2017) Users’ resistance towards radical innovations: the case of the self-driving car. Transp Res Part F Traffic Psychol Behav 44:42–52

    Article  Google Scholar 

  • Kyriakidis M, Happee R, de Winter JCF (2015) Public opinion on automated driving: results of an international questionnaire among 5000 respondents. Transp Res Part F Traffic Psychol Behav 32:127–140

    Article  Google Scholar 

  • Laugwitz B, Held T, Schrepp M (2008) Construction and evaluation of a user experience questionnaire. Symp Aust HCI Usability Eng Group. https://doi.org/10.1002/9783527617272.ch1

    Article  Google Scholar 

  • Lee JD, See KA (2004) Trust in automation: designing for appropriate reliance. Hum Factors 46(1):50–80

    Article  Google Scholar 

  • Lewis JR (2002) Psychometric evaluation of the PSSUQ using data from 5 years of usability studies. Int J Hum Comput Interact 14(3–4):463–488. https://doi.org/10.1515/9783110887242.3

    Article  Google Scholar 

  • Lewis JR, Sauro J (2009) The factor structure of the system usability scale. International conference on human centered design

  • Lüdtke O, Robitzsch A, Trautwein U, Köller O (2007) Umgang mit fehlenden Werten in der psychologischen Forschung. Psychologische Rundschau 58(2):103–117. https://doi.org/10.1026/0033-3042.58.2.103

    Article  Google Scholar 

  • Minge M, Thüring M, Wagner I, Kuhr CV (2016) The meCUE questionnaire: a modular tool for measuring user experience. In: Advances in ergonomics modeling 2016, pp 115–128

  • Naujoks F, Purucker C, Neukum A, Wolter S, Steiger R (2015) Controllability of partially automated driving functions—does it matter whether drivers are allowed to take their hands off the steering wheel? Transp Res Part F Traffic Psychol Behav 35:185–198

    Article  Google Scholar 

  • Naujoks F, Forster Y, Wiedemann K, Neukum A (2016) Speech improves human-automation cooperation in automated driving. In: Workshopband Mensch und Computer 2016. Aachen, Germany

  • Naujoks F, Forster Y, Wiedemann K, Neukum A (2017) Improving usefulness of automated driving by lowering primary task interference through HMI design. J Adv Transp. https://doi.org/10.1155/2017/6105087

    Article  Google Scholar 

  • Naujoks F, Hergeth S, Keinath A, Wiedemann K, Schömig N (2018) Use cases for assessing, testing, and validating the human–machine interface of automated driving systems. Human Factors and Ergonomics Society Annual Meeting, Philadelphia

    Book  Google Scholar 

  • Naujoks F, Hergeth S, Wiedemann K, Schömig N, Forster Y, Keinath A (2019a) Test procedure for evaluating the human–machine-interface of vehicles with automated driving. Traffic Injury Prevent 20:146–151

    Article  Google Scholar 

  • Naujoks F, Wiedemann K, Schömig N, Hergeth S, Keinath A (2019b) Towards guidelines and verification methods for automated vehicle HMIs. Transp Res Part F Traffic Psychol Behav 60:121–136

    Article  Google Scholar 

  • Nees M (2016) Acceptance of self-driving cars: an examination of idealized versus realistic portrayals with a self-driving cars acceptance scale. In: Human Factors and Ergonomics Society 60th Annual Meeting, Washington, D.C

  • Nielsen J, Levy J (1994) Measuring usability: preference vs. performance. Commun ACM 37(4):66–75

    Article  Google Scholar 

  • Nordhoff S, van Arem B, Happee R (2016) Conceptual model to explain, predict, and improve user acceptance of driverless podlike vehicles. Transp Res Record J Transp Res Board 2602:60–67. https://doi.org/10.3141/2602-08

    Article  Google Scholar 

  • Norman D, Miller J, Henderson A (1995) What you see, some of what’s in the future, and how we go about doing it: HI at apple computer. In: Conference companion on human factors in computing systems

  • Nunes A, Reimer B, Coughlin JF (2018) People must retain control of autonomous vehicles. Nature 556(7700):169–171. https://doi.org/10.1038/d41586-018-04158-5

    Article  Google Scholar 

  • Nunnally JC (1978) Psychometric theory. McGraw-Hill, New York

    Google Scholar 

  • O’connor BP (2000) SPSS and SAS programs for determining the number of components using parallel analysis and Velicer’s MAP test. Behav Res Methods Instrum Comput 32(3):396–402. https://doi.org/10.3758/BF03200807

    Article  Google Scholar 

  • Osswald S, Wurhofer D, Trösterer S, Beck E, Tscheligi M (2012) Predicting information technology usage in the car: towards a car technology acceptance model. In: Kun AL, Boyle LN, Reimer B, Riener A (Chairs) Proceedings of the 4th international conference on automotive user interfaces and interactive vehicular applications. symposium conducted at the meeting of ACM, Portsmouth, New Hampshire, US

  • Payre W, Cestac J, Delhomme P (2014) Intention to use a fully automated car: attitudes and a priori acceptability. Transp Res Part F Traffic Psychol Behav 27:252–263. https://doi.org/10.1016/j.trf.2014.04.009

    Article  Google Scholar 

  • Rahman MM, Lesch MF, Horrey WJ, Strawderman L (2017) Assessing the utility of TAM, TPB, and UTAUT for advanced driver assistance systems. Accid Anal Prevent 108:361–373. https://doi.org/10.1016/j.aap.2017.09.011

    Article  Google Scholar 

  • Revelle W, Zinbarg RE (2009) Coefficients alpha, beta, omega, and the glb: comments on Sijtsma. Psychometrika 74(1):145

    Article  MathSciNet  MATH  Google Scholar 

  • Society of Automotive Engineers International J3016 (2018) Taxonomy and definitions for terms related to on-road motor vehicle automated driving systems. SAE International, Warrendale, PA

    Google Scholar 

  • Sijtsma K (2009) On the use, the misuse, and the very limited usefulness of Cronbach’s alpha. Psychometrika 74(1):107–120. https://doi.org/10.1007/s11336-008-9101-0

    Article  MathSciNet  MATH  Google Scholar 

  • Spain RD, Bustamante EA, Bliss JP (2008) Towards an empirically Developed Scale for System Trust: take Two. Proc Hum Factors Ergon Soc Ann Meeting 52(19):1335–1339. https://doi.org/10.1177/154193120805201907

    Article  Google Scholar 

  • Stevens A, Quimby A, Board A, Kersloot T, Burns P (2002) Design guidelines for safety in-vehicle information systems. TRL Limited, Crowthorne

    Google Scholar 

  • Tabachnick BG, Fidell LS (2007) Using multivariate statistics. Allyn & Bacon/Pearson Education, Boston

    Google Scholar 

  • Tullis TS, Boynton TL, Hersh H (1995) Readability of fonts in the windows environment. Conference companion on Human factors in computing systems

  • Van der Laan JD, Heino A, de Waard D (1997) A simple procedure for the assessment of acceptance of advanced transport telematics. Transp Res Part C Emerg Technol 5(1):1–10

    Article  Google Scholar 

  • Vanderhaegen F, Carsten O (2017) Can dissonance engineering improve risk analysis of human–machine systems? Cogn Technol Work 19(1):1–12. https://doi.org/10.1007/s10111-017-0405-7

    Article  Google Scholar 

  • Velicer WF (1976) Determining the number of components from the matrix of partial correlations. Psychometrika 41(3):321–327. https://doi.org/10.1007/BF02293557

    Article  MATH  Google Scholar 

  • Venkatesh V, Morris MG, Davis GB, Davis FD (2003) User acceptance of information technology: toward a unified view. MIS Quart (27:3):425–478

  • Verberne FMF, Ham J, Midden CJH (2012) Trust in smart systems: sharing driving goals and giving information to increase trustworthiness and acceptability of smart systems in cars. Hum Factors 54(5):799–810

    Article  Google Scholar 

  • Walch M, Baumann M, Jaschke L, Weber M, Hock P (2017) Touch screen maneuver approval mechanisms for highly automated vehicles: a first evaluation. Adjunct proceedings of the 9th international ACM conference on automotive user interfaces and interactive vehicular applications, Oldenburg, Germany

  • Waytz A, Heafner J, Epley N (2014) The mind in the machine: anthropomorphism increases trust in an autonomous vehicle. J Exp Soc Psychol 52:113–117. https://doi.org/10.1016/j.jesp.2014.01.005

    Article  Google Scholar 

  • Wickens CD, Hollands JG, Banbury S, Parasuraman R (2015) Engineering psychology and human performance. Psychology Press, Boca Raton

    Google Scholar 

  • Zoellick JC, Kuhlmey A, Schenk L, Schindel D, Blüher S (2019) Assessing acceptance of electric automated vehicles after exposure in a realistic traffic environment. PLOS One. https://doi.org/10.1371/journal.pone.0215969

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. BMW Group, Knorrstr. 147, 80937, Munich, Germany

    Yannick Forster, Sebastian Hergeth, Frederik Naujoks & Andreas Keinath

  2. Chemnitz University of Technology, Wilhelm-Raabe Str. 43, 09120, Chemnitz, Germany

    Yannick Forster & Josef F. Krems

Authors
  1. Yannick Forster
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Sebastian Hergeth
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Frederik Naujoks
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Josef F. Krems
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Andreas Keinath
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Yannick Forster.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Forster, Y., Hergeth, S., Naujoks, F. et al. Self-report measures for the assessment of human–machine interfaces in automated driving. Cogn Tech Work 22, 703–720 (2020). https://doi.org/10.1007/s10111-019-00599-8

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10111-019-00599-8

Keywords

Search

Navigation