Abstract
New technologies are making it possible to provide an enriched view of interaction for researchers using multimodal information. This preliminary study explores the use of multimodal information streams in evaluating user cost. In the study, easy, medium and difficult versions of a game task were used to vary the levels of the cost to user. Multimodal data streams during the three versions were analyzed, including eye tracking, pupil size, hand movement, heart rate variability (HRV) and subjectively reported data. Three findings indicate the potential value of multimodal information in evaluating usability: First, subjective and physiological measures showed significant sensitivity to task difficulty. Second, different user cost levels appeared to correlate with eye movement patterns, especially with a combined eye–hand measure. Third, HRV showed correlations with saccade speed. These results warrant further investigations and take an initial step toward establishing usability evaluation methods based on multimodal information.
This is a preview of subscription content, log in via an institution.
Buying options
Tax calculation will be finalised at checkout
Purchases are for personal use only
Learn about institutional subscriptionsPreview
Unable to display preview. Download preview PDF.
References
Hollnagel, E.: From human factors to cognitive systems engineering: Human–machine interactions in the 21st century. In: Kitamura, M., Kimura, I. (eds.) Anzen-no-Tankyu (Researches on Safety), ERC Publishing, Tokyo (2003)
Marshall, C., Rossman, G.B.: Designing qualitative research, 3rd edn. Sage, Thousand Oaks (1999)
Wilson, G.M., Sasse, M.A.: Investigating the impact of audio degradations on users: Subjective vs. objective assessment methods. In: The Proceedings of OZCHI 2000: Interfacing Reality in the New Millennium, Sydney, Australia, pp. 135–142 (2000)
Sweeney, M., Maguire, M., Shackel, B.: Evaluating user-computer interaction: a framework. International Journal of Man-Machine Studies 38, 689–711 (1993)
Wilson, G.M.: Psychophysiological indicators of the impact of media quality on users. In: The Proceedings of CHI 2001 Doctoral Consortium, pp. 95–96. ACM Press, New York (2001)
Andreassi, J.L.: Psychophysiology: Human Behavior and Physiological Response, 4th edn. Lawrence Erlbaum, Mahwah (2000)
Picard, R.W.: Affective Computing. MIT Press, Cambridge (1997)
Vicente, K.J., Thornton, C., Moray, N.: Spectral analysis of sinus arrhythmia: A measure of mental effort. Human Factors 29, 171–182 (1987)
Ekman, P., Levenson, R.W., Friesen, W.V.: Autonomic nervous system activity distinguishes among emotions. Science 221(4616), 1208–1210 (1983)
Lin, T., Hu, W.H., Omata, M., Imamiya, A.: Do physiological data relate to traditional usability indexes? In: The Proceedings of the Australian Conference on Computer Human Interaction OZCHI’05, Canberra, Australia, November 15–21 (2005)
Meehan, M., Insko, B., Whitton, M., Brooks, F.: Physiological measures of presence in stressful virtual environments. In: The Proceedings of the 29th Annual Conference on Computer Graphics and Interactive Techniques (2002)
Rowe, D.W., Sibert, J., Irwin, D.: Heart rate variability: Indicator of user state as an aid to human–computer interaction. In: Proceedings of the CHI, pp. 480–487 (1998)
Hopman, J.C.W., Kollee, L.A.A., Stoelinga, G.B.A., Van Geijn, H.P., van Ravenswaaij-Arts, C.M.A.: Heart rate variability. Annals of Internal Medicine 118, 436–447 (1993)
Malik, M.: Heart rate variability. Circulation 93, 1043–1065 (1996)
Pomeranz, B., Macaulay, R.J.B., Caudill, M.A., et al.: Assessment of autonomic function in humans by heart rate spectral analysis. Am. J. Physiol. Heart Circ. Physiol. 248, 151–H153 (1985)
Pagani, M., Lombardi, F., Guzzetti, S.: Power spectral analysis of heart rate and arterial pressure variabilities as a maker of sympatho-vagal interaction in man and conscious dog. Cir. Res. 59, 178–193 (1986)
Ettema, J., Ziclhuis, R.L.: Physiological parameters of mental load. Ergonomics 14, 137–144 (1971)
Luczak, I.I., Lauring, W.J.: An analysis of heart rate variability. Ergonomics 16, 85–97 (1973)
Hyndman, B.W., Gregory, J.R.: Spectral analysis of sinus arrhythmia during mental loading. Ergonomics 18, 255–270 (1975)
Boutcher, S.H., Naugent, F.W., Mclaren, P.F.: Heart period variability of trained and untrained men at rest and during mental challenge. Psychophysiology 35, 16–22 (1998)
Hyde, C., Izard, C.E.: Cardiac rhythmicities and attention in young children. Psychophysiology 34, 547–552 (1997)
Tattersall, A.J., Hockey, G.R.J.: Level of operator control and changes in heart rate variability during simulated flight maintenance. Human Factors 37, 682–698 (1995)
Hess, E.H., Polt, J.M.: Pupil size in relation to mental activity during simple problem solving. Science 132, 1191–1192 (1964)
Hoecks, B., Levelt, W.: Pupillary dilation as a measure of attention: A quantitative system analysis. Behavior Research Methods, Instruments, & Computers 25, 16–26 (1993)
Juris, M., Velden, M.: The pupillary response to mental overload. Physiological Psychology 5(4), 421–424 (1977)
Kahneman, D.: Pupillary responses in a pitch-discrimination task. Perception & Psychophysics 2, 101–105 (1967)
Nakayama, M., Takahashi, K.: The act of task difficulty and eye-movement frequency for the oculo-motor indices. In: The Proceedings of the Eye Tracking Research and Application, pp. 37–42 (2002)
Beatty, J.: Task-evoked pupillary responses, processing load and the structure of processing resources. Psychological Bulletin 91(2), 276–292 (1982)
Iqbal, S.T., Zheng, X.S., Bailey, B.P.: Task-evoked pupillary response to mental workload in human–computer interaction. In: The proceeding of CHI, pp. 1477–1480 (2004)
Hess, E.H., Petrovich, S.B.: Pupillary behavior in communication. In: Siegman, A.W., Feldstein, S. (eds.) Nonverbal Behavior and Communication, pp. 327–348. Lawrence Erlbaum, Hillsdale (1987)
Rayner, K.: Eye movements and information processing: 20 years of research. Psychological Bulletin 124(3), 372–422 (1998)
Noton, D., Stark, L.W.: Scanpath in saccadic eye movements while viewing and recognizing patterns. Vision Research 11, 929–942 (1971)
Goldberg, H., Kotval, X.P.: Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics 24, 631–645 (1999)
Hébert, S., Beland, R., Dionne-Fournelle, O., Crête, M., Lupien, S.J.: Physiological stress response to video game playing: The contribution of built-in music. Life Sciences 76, 2371–2380 (2005)
Hart, S.G., Staveland, L.E.: Development of NASA-TLX (task load index): Results of experimental and theoretical research. In: Hancock, P.A., Meshakati, N. (eds.) Human Mental Workload, pp. 39–183. North-Holland, Amsterdam (1988)
Hair, J.F., Anderson, R.E., Tatham, R.L., Black, W.C.: Multivariate Data Analysis, 5th edn. Prentice-Hall, Englewood Cliffs (1998)
Lin, Y., Zhang, W.J., Koubek, R.J.: Effective Attention Allocation Behavior and its Measurement: A Preliminary Study. Interacting with Computers 16(Issue 6), 1195–1210 (2004)
Author information
Authors and Affiliations
Editor information
Rights and permissions
Copyright information
© 2007 Springer Berlin Heidelberg
About this paper
Cite this paper
Lin, T., Imamiya, A., Hu, W., Omata, M. (2007). Combined User Physical, Physiological and Subjective Measures for Assessing User Cost. In: Stephanidis, C., Pieper, M. (eds) Universal Access in Ambient Intelligence Environments. Lecture Notes in Computer Science, vol 4397. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-540-71025-7_20
Download citation
DOI: https://doi.org/10.1007/978-3-540-71025-7_20
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-71024-0
Online ISBN: 978-3-540-71025-7
eBook Packages: Computer ScienceComputer Science (R0)