Skip to main content
SpringerLink
Log in

Effective quality factors of multimodal interaction in simple and complex tasks of using a smart television

  • Published:
Multimedia Tools and Applications Aims and scope Submit manuscript

Abstract

Influencing quality factors, related to the user and system, need to be considered when building a well-designed multimodal interaction system. User groups, access to input modes and tasks were defined as the user and system factors to examine effective factors of multimodal interaction with a smart TV, and its input modes consisted of voice, arrow key, and motion-based pointer modes and their combinations. User group 1 had experience of multimodal interaction with another device, while user group 2 had only experience of unimodal interaction. In addition, the sequential/simultaneous input modes and simple/complex tasks were considered as the system factors. Depending on the task complexity, two experiments were conducted. Nine input modes (three unimodes and six multimodes), sequentially and simultaneously given to both user groups, were investigated for the simple task of menu traversal and the complex task of manipulating broadcasting content, menu traversal, and web content navigation. A subjective rating of the level of preference was recorded in the sequential input mode using a modified Likert-type rating scale, while each participant’s preferred mode was observed in the simultaneous scenario. Additionally, the completion time and error rate were measured in both experiments. When performing the simple task, user group 1 used multimodes more so than group 2. However, in the complex task, both user groups preferred multimodes when modes were simultaneously presented. Considering effective quality factors, input modes of a smart TV should be simultaneously provided with a voice and motion-based pointer multimode.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

References

  1. Alepis E, Virvou M (2012) Multimodal object oriented user interfaces in mobile affective interaction. Multimedia Tools Appl 59(1):41–63

    Article  Google Scholar 

  2. Atrey PK, Hossain MA, El Saddik A, Kankanhalli MS (2010) Multimodal fusion for multimedia analysis: a survey. Multimedia Syst 16(6):345–379

    Article  Google Scholar 

  3. Bangalore S, Johnston M (2009) Robust understanding in multimodal interfaces. Comput Linguist 35(3):345–397

    Article  Google Scholar 

  4. Barthelmess P, Oviatt, SL (2008) Multimodal interfaces: combining interfaces to accomplish a single task. Kortum P (ed) HCI Beyond the GUI, 1st edn. Morgan Kauffman, pp 391–444

  5. Bellik Y, Rebaï I, Machrouh E, Barzaj Y, Jacquet C, Pruvost G, Sansonnet JP (2009) Multimodal interaction within ambient environments: an exploratory study. In: Human-Computer Interaction–INTERACT 2009. Springer, Heidelberg, pp 89–92

    Chapter  Google Scholar 

  6. Chbeir R, Coninx K, Ferri F, Grifoni P (2011) Management and interaction with multimodal information content. Multimedia Tools Appl 54(1):1–5

    Article  Google Scholar 

  7. Chen F, Ruiz N, Choi E, Epps J, Khawaja MA, Taib R, Wang Y (2012) Multimodal behavior and interaction as indicators of cognitive load. ACM Trans Interact Intell Syst 2(4):22

    Article  Google Scholar 

  8. Chittaro L (2010) Distinctive aspects of mobile interaction and their implications for the design of multimodal interfaces. J Multimodal User Interfaces 3(3):157–165

    Article  Google Scholar 

  9. Cohen PR, Oviatt SL (2000) Multimodal interfaces that process what comes naturally. Commun ACM 43(3):45–33

    Article  Google Scholar 

  10. Dumas JS, Fox J (2008) Usability testing: current practice and future directions. In: Sears AL, Jacko JA (eds) The Handbook of Human-Computer Interaction, 2nd edn. Taylor and Francis, pp 1129–1149

  11. Dumas B, Lalanne D, Oviatt SL (2009) Multimodal interfaces: a survey of principles, models and frameworks. In: Lalanne D, Kohlas J (eds) Human Machine Interaction, LNCS 5440. Springer, Heidelberg, pp 3–26

    Chapter  Google Scholar 

  12. Elouali N, Rouillard J, Le Pallec X, Tarby JC (2013) Multimodal interaction: a survey from model driven engineering and mobile perspectives. J Multimodal User Interfaces 7(4):351–370

    Article  Google Scholar 

  13. Gürkök H, Nijholt A (2012) Brain–computer interfaces for multimodal interaction: a survey and principles. Int J Hum Comput Int 28(5):292–307

    Article  Google Scholar 

  14. Herrera-Acuña R, Argyriou V, Velastin SA (2015) A Kinect-based 3D hand-gesture interface for 3D databases. J Multimodal User Interfaces 9(2):121–139

    Article  Google Scholar 

  15. Hornbæk K (2006) Current practice in measuring usability: challenges to usability studies and research. Int J Hum Comput Stud 64(2):79–102

    Article  Google Scholar 

  16. Hürst W, Van Wezel C (2013) Gesture-based interaction via finger tracking for mobile augmented reality. Multimedia Tools Appl 62(1):233–258

    Article  Google Scholar 

  17. Jaimes A, Sebe N (2007) Multimodal human–computer interaction: a survey. Comput Vis Image Underst 108(1):116–134

    Article  Google Scholar 

  18. Karray F, Alemzadeh M, Saleh JA, Arab MN (2008) Human-computer interaction: overview on state of the art. Int J Smart Sens Intell Syst 1(1):137–159

    Google Scholar 

  19. König WA, Rädle R, Reiterer H (2010) Interactive design of multimodal user interfaces. J Multimodal User Interfaces 3(3):197–213

    Article  Google Scholar 

  20. Lee WP, Kaoli C, Huang JY (2014) A smart TV system with body-gesture control, tag-based rating and context-aware recommendation. Knowl Based Syst 56:167–178

    Article  Google Scholar 

  21. Lee M, Kim GJ (2014) Empathetic video clip experience through timely multimodal interaction. J Multimodal User Interfaces 8(3):273–288

    Article  Google Scholar 

  22. Lee SH, Sohn MK, Kim DJ, Kim B, Kim H (2013, January) Smart TV interaction system using face and hand gesture recognition. In: Consumer Electronics (ICCE), 2013 I.E. International Conference on, IEEE, pp. 173–174

  23. Lemmelä S, Vetek A, Mäkelä K, Trendafilov D (2008) Designing and evaluating multimodal interaction for mobile contexts. In: Proceedings of the 10th international conference on Multimodal interfaces, ACM, pp 265–272

  24. Li B, Zhang W, Zhou R, Yang C, Li Z (2012) A comparative ergonomics study: performing reading-based tasks on a large-scale tabletop vs. laptop. Int J Ind Ergono 42(1):156–161

    Article  Google Scholar 

  25. Liu SF, Cheng JH, Chen WJ (2015) A study of product experience obtained from multimodal interactive displays. Multimedia Tools Appl 1–30. doi:10.1007/s11042-015-2564-y

  26. Lopez-Cozar R, Araki M (2005) Spoken, multilingual and multimodal dialogue systems: development and assessment. Wiley, New York

    Google Scholar 

  27. Möller A, Diewald S, Roalter L, Kranz M (2014) A framework for mobile multimodal interaction. In: Proceedings of Mensch & Computer: Interaktiv unterwegs –Freiräume gestalten, Oldenbourg Verlag, pp 355–358

  28. Möller S, Engelbrecht KP, Kühnel C, Wechsung I, Weiss B (2009) Evaluation of multimodal interfaces for ambient intelligence. In: Aghajan H, Augusto JC, Delgado RL (eds) Human-Centric Interfaces for Ambient Intelligence, 1st edn. Elsevier, Amsterdam, pp 347–370

    Google Scholar 

  29. Nogueira PA, Teófilo LF, Silva PB (2015) Multi-modal natural interaction in game design: a comparative analysis of player experience in a large scale role-playing game. J Multimodal User Interfaces 9(2):105–119

    Article  Google Scholar 

  30. Osafo-Yeboah B, Jiang S, Delpish R, Jiang Z, Ntuen C (2013) Empirical study to investigate the range of force feedback necessary for best operator performance in a haptic controlled excavator interface. Int J Ind Ergon 43(3):197–202

    Article  Google Scholar 

  31. Oviatt SL (1999) Ten myths of multimodal interaction. Commun ACM 42(11):74–81

    Article  Google Scholar 

  32. Oviatt SL (2003) Advances in robust multimodal interface design. IEEE Comput Graph 5:62–68

    Article  Google Scholar 

  33. Oviatt SL (2003) Multimodal system processing in mobile environments. In: Proceedings of the 13th annual ACM symposium on User interface software and technology, ACM, pp 21–30

  34. Oviatt SL (2006) Human-centered design meets cognitive load theory: designing interfaces that help people think. In: Proceedings of the 14th annual ACM international conference on multimedia, ACM, pp 871–880

  35. Oviatt SL, Coulston R, Tomko S, Xiao B, Lunsford R, Wesson M, Carmichael L (2003) Toward a theory of organized multimodal integration patterns during human-computer interaction. In: Proceedings of the 5th international conference on Multimodal interfaces, ACM, pp 44–51

  36. Oviatt SL, Coulston R, Lunsford R (2004) When do we interact multimodally?: cognitive load and multimodal communication patterns. In: Proceedings of the 6th international conference on multimodal interfaces, ACM, pp 129–136

  37. Plimmer B (2008) Experiences with digital pen, keyboard and mouse usability. J Multimodal User Interfaces 2(1):13–23

    Article  Google Scholar 

  38. Ratzka A (2013) User interface patterns for multimodal interaction. In: Transactions on pattern languages of programming III. Springer, Heidelberg, pp 111–167

    Chapter  Google Scholar 

  39. Reeves LM, Lai J, Larson JA, Oviatt SL, Balaji TS, Buisine S, Wang QY (2004) Guidelines for multimodal user interface design. Commun ACM 47(1):57–59

    Article  Google Scholar 

  40. Sakamoto K, Aoyama S, Asahara S, Yamashita K, Okada A (2009) Evaluation of viewing distance vs. TV size on visual fatigue in a home viewing environment. In: Consumer Electronics, 2009. ICCE’09. Digest of Technical Papers International Conference on, IEEE, pp 1–2

  41. Schüssel F, Honold F, Weber M (2013) Influencing factors on multimodal interaction during selection tasks. J Multimodal User Interfaces 7(4):299–310

    Article  Google Scholar 

  42. Shaer O, Hornecker E (2010) Tangible user interfaces: past, present, and future directions. Found Trends Hum Comput Interact 3(1–2):4–137. doi:10.1007/s10648-014-9255-5

    Google Scholar 

  43. Sheu JS, Huang YL (2015) Implementation of an interactive TV interface via gesture and handwritten numeral recognition. Multimed Tools Appl 1–22. doi:10.1007/s11042-015-2739-6

  44. Shin DH, Hwang Y, Choo H (2013) Smart TV: are they really smart in interacting with people? Understanding the interactivity of Korean Smart TV. Behav Inform Technol 32(2):156–172

    Article  Google Scholar 

  45. Soysal M, Loğoğlu KB, Tekin M, Esen E, Saracoğlu A, Acar B, Çiloğlu T (2014) Multimodal concept detection in broadcast media: KavTan. Multimedia Tools Appl 72(3):2787–2832

    Article  Google Scholar 

  46. Wechsung I, Engelbrecht KP, Kühnel C, Möller S, Weiss B (2012) Measuring the quality of service and quality of experience of multimodal human–machine interaction. J Multimodal User Interfaces 6(1–2):73–85

    Article  Google Scholar 

  47. Wickens CD (2002) Multiple resources and performance prediction. Theor Issues Ergon 3(2):159–177

    Article  Google Scholar 

  48. Wickens CD (2008) Multiple resources and mental workload. Hum Factors 50(3):449–455

    Article  Google Scholar 

  49. Wickens CD, Sandry DL, Vidulich M (1983) Compatibility and resource competition between modalities of input, central processing, and output. Hum Factors 25(2):227–248

    Google Scholar 

  50. Xie L, Deng Z, Cox S (2014) Multimodal joint information processing in human machine interaction: recent advances. Multimedia Tools Appl 73(1):267–271

    Article  Google Scholar 

  51. Yu J, Kim SM, Choe J, Jung ES (2013) Multi-modal controller usability for smart TV control. J Ergon Soc Korea 32(6):517–528

    Article  Google Scholar 

Download references

Acknowledgments

This research was supported by a Korea University grant.

Author information

Authors and Affiliations

  1. Department of Industrial Management Engineering, Korea University, Anam-ro, Seongbuk-gu, Seoul, 02841, South Korea

    Seong M. Kim, Eui S. Jung & Jaekyu Park

Authors
  1. Seong M. Kim
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Eui S. Jung
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jaekyu Park
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Eui S. Jung.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Kim, S.M., Jung, E.S. & Park, J. Effective quality factors of multimodal interaction in simple and complex tasks of using a smart television. Multimed Tools Appl 76, 6447–6471 (2017). https://doi.org/10.1007/s11042-016-3333-2

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11042-016-3333-2

Keywords

Search

Navigation