ABSTRACT
Collaborative Virtual Environments (CVEs) offer unique opportunities for human communication. Humans can interact with each other over a distance in any environment and visual embodiment they want. Although deictic gestures are especially important as they can guide other humans' attention, humans make systematic errors when using and interpreting them. Recent work suggests that the interpretation of vertical deictic gestures can be significantly improved by warping the pointing arm. In this paper, we extend previous work by showing that models enable to also improve the interpretation of deictic gestures at targets all around the user. Through a study with 28 participants in a CVE, we analyzed the errors users make when interpreting deictic gestures. We derived a model that rotates the arm of a pointing user's avatar to improve the observing users' accuracy. A second study with 24 participants shows that we can improve observers' accuracy by 22.9%. As our approach is not noticeable for users, it improves their accuracy without requiring them to learn a new interaction technique or distracting from the experience.
Supplemental Material
- Martha W. Alibali. 2005. Gesture in Spatial Cognition: Expressing, Communicating, and Thinking About Spatial Information. Spatial Cognition and Computation 5, 4 (2005), 307--331. DOI: http://dx.doi.org/10.1207/s15427633scc0504_2Google ScholarCross Ref
- Mahdi Azmandian, Mark Hancock, Hrvoje Benko, Eyal Ofek, and Andrew D. Wilson. 2016. Haptic Retargeting: Dynamic Repurposing of Passive Haptics for Enhanced Virtual Reality Experiences. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, NY, NY, USA, 1968--1979. DOI: http://dx.doi.org/10.1145/2858036.2858226Google ScholarDigital Library
- Adrian Bangerter and Daniel M. Oppenheimer. 2006. Accuracy in detecting referents of pointing gestures unaccompanied by language. Gesture 6, 1 (2006), 85--102. DOI: http://dx.doi.org/10.1075/gest.6.1.05banGoogle ScholarCross Ref
- R.M. Baños, C. Botella, M. Alcañiz, V. Liaño, B. Guerrero, and B. Rey. 2004. Immersion and Emotion: Their Impact on the Sense of Presence. CyberPsychology & Behavior 7, 6 (dec 2004), 734--741. DOI: http://dx.doi.org/10.1089/cpb.2004.7.734Google ScholarCross Ref
- Steve Benford, John Bowers, Lennart E. Fahlén, Chris Greenhalgh, and Dave Snowdon. 1995. User Embodiment in Collaborative Virtual Environments. In Proceedings of the SIGCHI Conference on Human 2https://github.com/interactionlab/Deictic-Pointing-in-VR Factors in Computing Systems (CHI '95). ACM Press/Addison-Wesley Publishing Co., New York, NY, USA, 242--249. DOI: http://dx.doi.org/10.1145/223904.223935Google ScholarDigital Library
- Steve Benford, Chris Greenhalgh, Tom Rodden, and James Pycock. 2001. Collaborative Virtual Environments. Commun. ACM 44, 7 (July 2001), 79--85. DOI: http://dx.doi.org/10.1145/379300.379322Google ScholarDigital Library
- Johanna Bertram, Johannes Moskaliuk, and Ulrike Cress. 2015. Virtual training: Making reality work? Computers in Human Behavior 43 (2015), 284 -- 292. DOI: http://dx.doi.org/https://doi.org/10.1016/j.chb.2014.10.032Google ScholarDigital Library
- Frank Biocca. 1992. Communication Within Virtual Reality: Creating a Space for Research. Journal of Communication 42, 4 (1992), 5--22. DOI: http://dx.doi.org/10.1111/j.1460--2466.1992.tb00810.xGoogle ScholarCross Ref
- Richard A. Bolt. 1980. Put-that-there: Voice and gesture at the graphics interface. In Proceedings of the 7th annual conference on Computer graphics and interactive techniques (SIGGRAPH '80). ACM, NY, NY, USA, 262--270. DOI: http://dx.doi.org/10.1145/800250.807503Google ScholarDigital Library
- Evren Bozgeyikli, Andrew Raij, Srinivas Katkoori, and Rajiv Dubey. 2016. Point & Teleport Locomotion Technique for Virtual Reality. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '16). ACM, NY, NY, USA, 205--216. DOI: http://dx.doi.org/10.1145/2967934.2968105Google ScholarDigital Library
- George Butterworth and Shoji Itakura. 2000. How the eyes, head and hand serve definite reference. British Journal of Developmental Psychology 18, 1 (2000), 25--50. DOI: http://dx.doi.org/10.1348/026151000165553Google ScholarCross Ref
- George Butterworth and Nicholas Jarrett. 1991. What minds have in common is space: Spatial mechanisms serving joint visual attention in infancy. British Journal of Developmental Psychology 9, 1 (1991), 55--72. DOI: http://dx.doi.org/10.1111/j.2044--835X.1991.tb00862.xGoogle ScholarCross Ref
- Luigia Camaioni. 1997. The Emergence of Intentional Communication in Ontogeny, Phylogeny, and Pathology. European Psychologist 2, 3 (1997), 216--225. DOI: http://dx.doi.org/10.1027/1016--9040.2.3.216Google ScholarCross Ref
- Luigia Camaioni, Paola Perucchini, Francesca Bellagamba, and Cristina Colonnesi. 2004. The Role of Declarative Pointing in Developing a Theory of Mind. Infancy 5, 3 (2004), 291--308. DOI: http://dx.doi.org/10.1207/s15327078in0503_3Google ScholarCross Ref
- Christer Carlsson and Lennart E. Fahlén. 1993. Integrated CSCW Tools Within a Shared 3D Virtual Environment (Abstract). In Proceedings of the INTERACT '93 and CHI '93 Conference on Human Factors in Computing Systems (CHI '93). ACM, NY, NY, USA, 513--. DOI: http://dx.doi.org/10.1145/169059.169452Google ScholarDigital Library
- Lung-Pan Cheng, Eyal Ofek, Christian Holz, Hrvoje Benko, and Andrew D. Wilson. 2017. Sparse Haptic Proxy: Touch Feedback in Virtual Environments Using a General Passive Prop. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 3718--3728. DOI: http://dx.doi.org/10.1145/3025453.3025753Google ScholarDigital Library
- Christopher Codella, Reza Jalili, Lawrence Koved, J. Bryan Lewis, Daniel T. Ling, James S. Lipscomb, David A. Rabenhorst, Chu P. Wang, Alan Norton, Paula Sweeney, and Greg Turk. 1992. Interactive Simulation in a Multi-person Virtual World. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '92). ACM, NY, NY, USA, 329--334. DOI: http://dx.doi.org/10.1145/142750.142825Google ScholarDigital Library
- Giambattista Della Porta. 1593. De refractione Optices Parte: Libri Novem... Ex officina Horatii Salviani, apud Jo. Jacobum Carlinum, & Antonium Pacem.Google Scholar
- Daphne Economou, Ioannis Doumanis, Lemonia Argyriou, and Nektarios Georgalas. 2017. User Experience Evaluation of Human Representation in Collaborative Virtual Environments. Personal Ubiquitous Comput. 21, 6 (Dec. 2017), 989--1001. DOI: http://dx.doi.org/10.1007/s00779-017--1075--4Google ScholarDigital Library
- Tiare Feuchtner and Jörg Müller. 2017. Extending the Body for Interaction with Reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 5145--5157. DOI: http://dx.doi.org/10.1145/3025453.3025689Google ScholarDigital Library
- John M. Foley and Richard Held. 1972. Visually directed pointing as a function of target distance, direction, and available cues. Perception & Psychophysics 12, 3 (01 May 1972), 263--268. DOI: http://dx.doi.org/10.3758/BF03207201Google ScholarCross Ref
- Saul Greenberg, Carl Gutwin, and Andy Cockburn. 1996a. Awareness through fisheye views in relaxed-WYSIWIS groupware. In Graphics interface, Vol. 96. Morgan-Kaufmann, Toronto, Canada, 28--38.Google Scholar
- Saul Greenberg, Carl Gutwin, and Mark Roseman. 1996b. Semantic telepointers for groupware. In Proceedings Sixth Australian Conference on Computer-Human Interaction. 54--61. DOI: http://dx.doi.org/10.1109/OZCHI.1996.559988Google ScholarCross Ref
- Scott W Greenwald, Wiley Corning, and Pattie Maes. 2017a. Multi-user framework for collaboration and co-creation in virtual reality (CSCL '17). 12th International Conference on Computer Supported Collaborative Learning. http://hdl.handle.net/1721.1/108440Google Scholar
- Scott W Greenwald, Alexander Kulik, André Kunert, Stephan Beck, Bernd Fröhlich, Sue Cobb, Sarah Parsons, Nigel Newbutt, Christine Gouveia, Claire Cook, and others. 2017b. Technology and applications for collaborative learning in virtual reality. Philadelphia, PA. https://repository.isls.org/handle/1/210Google Scholar
- Victoria Groom, Jeremy N Bailenson, and Clifford Nass. 2009. The influence of racial embodiment on racial bias in immersive virtual environments. Social Influence 4, 3 (2009), 231--248.Google ScholarCross Ref
- Jan Gugenheimer, Evgeny Stemasov, Julian Frommel, and Enrico Rukzio. 2017. ShareVR: Enabling Co-Located Experiences for Virtual Reality Between HMD and Non-HMD Users. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 4021--4033. DOI: http://dx.doi.org/10.1145/3025453.3025683Google ScholarDigital Library
- Carl Gutwin and Saul Greenberg. 2002. A Descriptive Framework of Workspace Awareness for Real-Time Groupware. Computer Supported Cooperative Work 11, 3 (01 Sep 2002), 411--446. DOI: http://dx.doi.org/10.1023/A:1021271517844Google ScholarDigital Library
- Anthony Guye-Vuillème, Tolga K. Capin, Igor Sunday Pandzic, Nadia Magnenat Thalmann, and Daniel Thalmann. 1999. Nonverbal communication interface for collaborative virtual environments. Virtual Reality 4, 1 (01 Mar 1999), 49--59. DOI: http://dx.doi.org/10.1007/BF01434994Google ScholarDigital Library
- Olof Hagsand. 1996. Interactive multiuser VEs in the DIVE system. IEEE MultiMedia 3, 1 (Spring 1996), 30--39. DOI: http://dx.doi.org/10.1109/93.486702Google ScholarDigital Library
- Edward Twitchell Hall. 1966. The hidden dimension. Doubleday & Co, New York, NY, USA.Google Scholar
- Sandra G. Hart. 2006. NASA-Task Load Index (NASA-TLX); 20 years later. In Proceedings of the Human Factors and Ergonomic Society annual meeting, Vol. 50. SAGE Publications, SAGE Publications, Los Angeles, CA, USA, 904--908. DOI: http://dx.doi.org/10.1177/154193120605000909Google ScholarCross Ref
- John B. Haviland. 2003. How to point in Zinacantán. Psychology Press, Mahwah, New Jersey, USA, 139--169.Google Scholar
- Oliver Herbort and Wilfried Kunde. 2016. Spatial (mis-)interpretation of pointing gestures to distal referents. Journal of Experimental Psychology: Human Perception and Performance 42, 1 (2016), 78--89. DOI: http://dx.doi.org/10.1037/xhp0000126Google ScholarCross Ref
- Oliver Herbort and Wilfried Kunde. 2018. How to point and to interpret pointing gestures? Instructions can reduce pointer--observer misunderstandings. Psychological Research 82, 2 (01 Mar 2018), 395--406. DOI: http://dx.doi.org/10.1007/s00426-016-0824--8Google ScholarCross Ref
- Jason Jerald. 2016. The VR Book: Human-Centered Design for Virtual Reality. Association for Computing Machinery and Morgan & Claypool, New York, NY, USA.Google ScholarDigital Library
- Shunichi Kasahara, Keina Konno, Richi Owaki, Tsubasa Nishi, Akiko Takeshita, Takayuki Ito, Shoko Kasuga, and Junichi Ushiba. 2017. Malleable Embodiment: Changing Sense of Embodiment by Spatial-Temporal Deformation of Virtual Human Body. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 6438--6448. DOI: http://dx.doi.org/10.1145/3025453.3025962Google ScholarDigital Library
- Sotaro Kita. 2003. Pointing: Where language, culture, and cognition meet. Psychology Press.Google ScholarCross Ref
- Pascal Knierim, Valentin Schwind, Anna Feit, Florian Nieuwenhuizen, and Niels Henze. 2018. Physical Keyboards in Virtual Reality: Analysis of Typing Performance and Effects of Avatar Hands. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA. DOI: http://dx.doi.org/10.1145/3173574.3173919Google ScholarDigital Library
- Mikko Kytö, Barrett Ens, Thammathip Piumsomboon, Gun A. Lee, and Mark Billinghurst. 2018. Pinpointing: Precise Head- and Eye-Based Target Selection for Augmented Reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 81, 14 pages. DOI: http://dx.doi.org/10.1145/3173574.3173655Google ScholarDigital Library
- Sarah Lopez, Yi Yang, Kevin Beltran, Soo Jung Kim, Jennifer Cruz Hernandez, Chelsy Simran, Bingkun Yang, and Beste F. Yuksel. 2019. Investigating Implicit Gender Bias and Embodiment of White Males in Virtual Reality with Full Body Visuomotor Synchrony. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 557, 12 pages. DOI: http://dx.doi.org/10.1145/3290605.3300787Google ScholarDigital Library
- Kristina Lundholm Fors. 2015. Production and Perception of Pauses in Speech. Ph.D. Dissertation. University of Gothenburg.Google Scholar
- Sven Mayer, Valentin Schwind, Robin Schweigert, and Niels Henze. 2018. The Effect of Offset Correction and Cursor on Mid-Air Pointing in Real and Virtual Environments. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 653, 13 pages. DOI: http://dx.doi.org/10.1145/3173574.3174227Google ScholarDigital Library
- Sven Mayer, Katrin Wolf, Stefan Schneegass, and Niels Henze. 2015. Modeling Distant Pointing for Compensating Systematic Displacements. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, NY, NY, USA, 4165--4168. DOI: http://dx.doi.org/10.1145/2702123.2702332Google ScholarDigital Library
- Thammathip Piumsomboon, Gun A. Lee, Jonathon D. Hart, Barrett Ens, Robert W. Lindeman, Bruce H. Thomas, and Mark Billinghurst. 2018. Mini-Me: An Adaptive Avatar for Mixed Reality Remote Collaboration. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 46, 13 pages. DOI: http://dx.doi.org/10.1145/3173574.3173620Google ScholarDigital Library
- Thammathip Piumsomboon, Youngho Lee, Gun Lee, and Mark Billinghurst. 2017. CoVAR: A Collaborative Virtual and Augmented Reality System for Remote Collaboration. In SIGGRAPH Asia 2017 Emerging Technologies (SA '17). ACM, NY, NY, USA, Article 3, 2 pages. DOI: http://dx.doi.org/10.1145/3132818.3132822Google ScholarDigital Library
- Katrin Plaumann, Matthias Weing, Christian Winkler, Michael Müller, and Enrico Rukzio. 2017. Towards accurate cursorless pointing: the effects of ocular dominance and handedness. Personal and Ubiquitous Computing (07 Dec 2017). DOI: http://dx.doi.org/10.1007/s00779-017--1100--7Google ScholarCross Ref
- Holger T. Regenbrecht, Thomas W. Schubert, and Frank Friedmann. 1998. Measuring the Sense of Presence and its Relations to Fear of Heights in Virtual Environments. International Journal of Human-Computer Interaction 10, 3 (1998), 233--249. DOI: http://dx.doi.org/10.1207/s15327590ijhc1003_2Google ScholarCross Ref
- Rufat Rzayev, Gürkan Karaman, Katrin Wolf, Niels Henze, and Valentin Schwind. 2019. The Effect of Presence and Appearance of Guides in Virtual Reality Exhibitions. In Proceedings of Mensch Und Computer 2019 (MuC'19). ACM, NY, NY, USA, 11--20. DOI: http://dx.doi.org/10.1145/3340764.3340802Google ScholarDigital Library
- Maria V. Sanchez-Vives and Mel Slater. 2005. From presence to consciousness through virtual reality. Nature reviews. Neuroscience 6, 4 (apr 2005), 332--339. DOI: http://dx.doi.org/10.1038/nrn1651Google ScholarCross Ref
- Chris L. Schmidt. 1999. Adult understanding of spontaneous attention-directing events: What does gesture contribute? Ecological Psychology 11, 2 (1999), 139--174. DOI: http://dx.doi.org/10.1207/s15326969eco1102_2Google ScholarCross Ref
- Valentin Schwind, Pascal Knierim, Lews Chuang, and Niels Henze. 2017a. "Where's Pinky?": The Effects of a Reduced Number of Fingers in Virtual Reality. In Proceedings of the 2017 CHI Conference on Computer-Human Interaction in Play (CHI PLAY'17). ACM, NY, NY, USA, 6. DOI: http://dx.doi.org/10.1145/3116595.3116596Google ScholarDigital Library
- Valentin Schwind, Pascal Knierim, Cagri Tasci, Patrick Franczak, Nico Haas, and Niels Henze. 2017b. "These Are Not My Hands!": Effect of Gender on the Perception of Avatar Hands in Virtual Reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 1577--1582. DOI: http://dx.doi.org/10.1145/3025453.3025602Google ScholarDigital Library
- Valentin Schwind, Jan Leusmann, and Niels Henze. 2019. Understanding Visual-Haptic Integration of Avatar Hands Using a Fitts' Law Task in Virtual Reality. In Proceedings of Mensch Und Computer 2019 (MuC'19). ACM, NY, NY, USA, 211--222. DOI: http://dx.doi.org/10.1145/3340764.3340769Google ScholarDigital Library
- Valentin Schwind, Lorraine Lin, Massimiliano Di Luca, Sophie Jörg, and James Hillis. 2018a. Touch with Foreign Hands: The Effect of Virtual Hand Appearance on Visual-haptic Integration. In Proceedings of the 15th ACM Symposium on Applied Perception (SAP '18). ACM, NY, NY, USA, Article 9, 8 pages. DOI: http://dx.doi.org/10.1145/3225153.3225158Google ScholarDigital Library
- Valentin Schwind, Sven Mayer, Alexandre Comeau-Vermeersch, Robin Schweigert, and Niels Henze. 2018b. Up to the Finger Tip: The Effect of Avatars on Mid-Air Pointing Accuracy in Virtual Reality. In Proceedings of the 2018 CHI Conference on Computer-Human Interaction in Play. DOI: http://dx.doi.org/10.1145/3242671.3242675Google ScholarDigital Library
- Maurício Sousa, Rafael Kufner dos Anjos, Daniel Mendes, Mark Billinghurst, and Joaquim Jorge. 2019. Warping Deixis: Distorting Gestures to Enhance Collaboration. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 608, 12 pages. DOI: http://dx.doi.org/10.1145/3290605.3300838Google ScholarDigital Library
- Mark Jeffrey Stefik, Daniel G. Bobrow, Gregg Foster, Stan Lanning, and Deborah Tatar. 1987. WYSIWIS Revised: Early Experiences with Multiuser Interfaces. ACM Transactions on Information Systems 5, 2 (April 1987), 147--167. DOI: http://dx.doi.org/10.1145/27636.28056Google ScholarDigital Library
- Jonathan Steuer. 1992. Defining Virtual Reality: Dimensions Determining Telepresence. Journal of Communication 42, 4 (1992), 73--93. DOI: http://dx.doi.org/10.1111/j.1460--2466.1992.tb00812.xGoogle ScholarCross Ref
- Haruo Takemura and Fumio Kishino. 1992. Cooperative Work Environment Using Virtual Workspace. In Proceedings of the 1992 ACM Conference on Computer-supported Cooperative Work (CSCW '92). ACM, NY, NY, USA, 226--232. DOI: http://dx.doi.org/10.1145/143457.269747Google ScholarDigital Library
- Janet L. Taylor and D. I. McCloskey. 1988. Pointing. Behavioural Brain Research 29, 1 (1988), 1 -- 5. DOI: http://dx.doi.org/https://doi.org/10.1016/0166--4328(88)90046-0Google ScholarCross Ref
- Jolanda G. Tromp, Anthony Steed, and John R. Wilson. 2003. Systematic Usability Evaluation and Design Issues for Collaborative Virtual Environments. Presence: Teleoperators and Virtual Environments 12, 3 (2003), 241--267. DOI: http://dx.doi.org/10.1162/105474603765879512Google ScholarDigital Library
- Martin Usoh, Kevin Arthur, Mary C Whitton, Rui Bastos, Anthony Steed, Mel Slater, and Frederick P Brooks Jr. 1999. Walking> walking-in-place> flying, in virtual environments. In Proceedings of the 26th annual conference on Computer graphics and interactive techniques. ACM Press/Addison-Wesley Publishing Co., 359--364.Google Scholar
- Juan Pablo Wachs, Mathias Kölsch, Helman Stern, and Yael Edan. 2011. Vision-based Hand-gesture Applications. Commun. ACM 54, 2 (Feb. 2011), 60--71. DOI: http://dx.doi.org/10.1145/1897816.1897838Google ScholarDigital Library
- E. J. Williams. 1949. Experimental Designs Balanced for the Estimation of Residual Effects of Treatments. Australian Journal of Chemistry 2, 2 (01 Jun 1949), 149--168. https://doi.org/10.1071/CH9490149Google ScholarCross Ref
- Bob G Witmer and Michael J Singer. 1998. Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence: Teleoperators and Virtual Environments 7, 3 (jun 1998), 225--240. DOI: http://dx.doi.org/10.1162/105474698565686Google ScholarDigital Library
- Nelson Wong and Carl Gutwin. 2014. Support for Deictic Pointing in CVEs: Still Fragmented After All These Years'. In Proceedings of the 17th ACM Conference on Computer Supported Cooperative Work & Social Computing (CSCW '14). ACM, NY, NY, USA, 1377--1387. DOI: http://dx.doi.org/10.1145/2531602.2531691Google ScholarDigital Library
Index Terms
- Improving Humans' Ability to Interpret Deictic Gestures in Virtual Reality
Recommendations
The Gesture Authoring Space: Authoring Customised Hand Gestures for Grasping Virtual Objects in Immersive Virtual Environments
MuC '22: Proceedings of Mensch und Computer 2022Natural user interfaces are on the rise. Manufacturers for Augmented, Virtual, and Mixed Reality head mounted displays are increasingly integrating new sensors into their consumer grade products, allowing gesture recognition without additional hardware. ...
User-Defined Gestures with Physical Props in Virtual Reality
ISSWhen interacting with virtual reality (VR) applications like CAD and open-world games, people may want to use gestures as a means of leveraging their knowledge from the physical world. However, people may prefer physical props over handheld controllers ...
Nonverbal communication in virtual reality: Nodding as a social signal in virtual interactions
Highlights- We use VR technology to create an immersive environment that simulates social interaction between a human participant and a virtual human.
Graphical abstractDisplay Omitted
AbstractNonverbal communication is an important part of human communication, including head nodding, eye gaze, proximity and body orientation. Recent research has identified specific patterns of head nodding linked to conversation, namely ...
Comments