Mohamed Soliman
ABSTRACT
Single-hand thumb-to-finger microgestures have shown great promise for expressive, fast and direct interactions. However, pioneering gesture recognition systems each focused on a particular subset of gestures. We are still in lack of systems that can detect the set of possible gestures to a fuller extent. In this paper, we present a consolidated design space for thumb-to-finger microgestures. Based on this design space, we present a thumb-to-finger gesture recognition system using depth sensing and convolutional neural networks. It is the first system that accurately detects the touch points between fingers as well as the finger flexion. As a result, it can detect a broader set of gestures than the existing alternatives, while also providing high-resolution information about the contact points. The system shows an average accuracy of 91% for the real-time detection of 8 demanding thumb-to-finger gesture classes. We demonstrate the potential of this technology via a set of example applications.
Supplemental Material
- 2015. NailO: Fingernails as an Input Surface. Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems - CHI '15 (2015), 3015--3018. Google Scholar
Digital Library
- Edwin Chan, Teddy Seyed, Wolfgang Stuerzlinger, Xing-Dong Yang, and Frank Maurer. 2016. User elicitation on single-hand microgestures. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 3403--3414. Google ScholarDigital Library
- Liwei Chan, Yi-Ling Chen, Chi-Hao Hsieh, Rong-Hao Liang, and Bing-Yu Chen. 2015a. Cyclopsring: Enabling whole-hand and context-aware interactions through a fisheye ring. In Proceedings of the 28th Annual ACM Symposium on User Interface Software & Technology. ACM, 549--556. Google ScholarDigital Library
- Liwei Chan, Chi-Hao Hsieh, Yi-Ling Chen, Shuo Yang, Da-Yuan Huang, Rong-Hao Liang, and Bing-Yu Chen. 2015b. Cyclops: Wearable and single-piece full-body gesture input devices. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3001--3009. Google ScholarDigital Library
- Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y. Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: Private and Subtle Interaction Using Fingertips. Proceedings of the 26th annual ACM symposium on User interface software and technology - UIST '13 (2013), 255--260. Google ScholarDigital Library
- Ke-Yu Chen, Shwetak N. Patel, and Sean Keller. 2016. Finexus: Tracking Precise Motions of Multiple Fingertips Using Magnetic Sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 1504--1514. Google ScholarDigital Library
- Andrea Colaço, Ahmed Kirmani, Hye Soo Yang, Nan-Wei Gong, Chris Schmandt, and Vivek K Goyal. 2013. Mime: compact, low power 3D gesture sensing for interaction with head mounted displays. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 227--236. Google ScholarDigital Library
- Artem Dementyev and Joseph A Paradiso. 2014. WristFlex: low-power gesture input with wrist-worn pressure sensors. In Proceedings of the 27th annual ACM symposium on User interface software and technology. ACM, 161--166. Google ScholarDigital Library
- Markus Funk, Sven Mayer, Michael Nistor, and Albrecht Schmidt. 2016. Mobile In-Situ Pick-by-Vision: Order Picking Support Using a Projector Helmet. In Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments (PETRA '16). ACM, New York, NY, USA, Article 45, 4 pages. Google ScholarDigital Library
- Sean G. Gustafson, Bernhard Rabe, and Patrick M. Baudisch. 2013. Understanding Palm-based Imaginary Interfaces: The Role of Visual and Tactile Cues when Browsing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '13). ACM, 889--898. Google ScholarDigital Library
- Chris Harrison, Hrvoje Benko, and Andrew D Wilson. 2011. OmniTouch: wearable multitouch interaction everywhere. In Proceedings of the 24th annual ACM symposium on User interface software and technology. ACM, 441--450. Google ScholarDigital Library
- Chris Harrison, Desney Tan, and Dan Morris. 2010. Skinput: appropriating the body as an input surface. In Proceedings of the SIGCHI conference on human factors in computing systems. ACM, 453--462. Google ScholarDigital Library
- Christopher-Eyk Hrabia, Katrin Wolf, and Mathias Wilhelm. 2013. Whole hand modeling using 8 wearable sensors: biomechanics for hand pose prediction. In Proceedings of the 4th Augmented Human International Conference. ACM, 21--28. Google ScholarDigital Library
- Da-Yuan Huang, Liwei Chan, Shuo Yang, Fan Wang, Rong-Hao Liang, De-Nian Yang, Yi-Ping Hung, and Bing-Yu Chen. 2016. DigitSpace: Designing Thumb-to-Fingers Touch Interfaces for One-Handed and Eyes-Free Interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. ACM, 1526--1537. Google ScholarDigital Library
- Da-Yuan Huang, Ming-Chang Tsai, Ying-Chao Tung, Min-Lun Tsai, Yen-Ting Yeh, Liwei Chan, Yi-Ping Hung, and Mike Y Chen. 2014. TouchSense: expanding touchscreen input vocabulary using different areas of users' finger pads. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 189--192. Google ScholarDigital Library
- Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. 2014. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia. ACM, 675--678. Google ScholarDigital Library
- David Kim, Otmar Hilliges, Shahram Izadi, Alex D Butler, Jiawen Chen, Iason Oikonomidis, and Patrick Olivier. 2012. Digits: freehand 3D interactions anywhere using a wrist-worn gloveless sensor. In Proceedings of the 25th annual ACM symposium on User interface software and technology. ACM, 167--176. Google ScholarDigital Library
- Eyal Krupka, Kfir Karmon, Noam Bloom, Daniel Freedman, Ilya Gurvich, Aviv Hurvitz, Ido Leichter, Yoni Smolin, Yuval Tzairi, Alon Vinnikov, and Aharon Bar Hillel. 2017. Toward Realistic Hands Gesture Interface: Keeping it Simple for Developers and Machines. In Proceedings of ACM CHI. 14. https://sites.google.com/ site/aharonbarhillel/CHI_2017.pdf?attredirects=0 Google ScholarDigital Library
- Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4 (2016), 142. Google ScholarDigital Library
- Christian Loclair, Sean Gustafson, and Patrick Baudisch. 2010. PinchWatch: a wearable device for one-handed microinteractions. In Proc. MobileHCI, Vol. 10.Google Scholar
- Pranav Mistry, Pattie Maes, and Liyan Chang. 2009. WUW-wear Ur world: a wearable gestural interface. In CHI'09 extended abstracts on Human factors in computing systems. ACM, 4111--4116. Google ScholarDigital Library
- Franziska Mueller, Dushyant Mehta, Oleksandr Sotnychenko, Srinath Sridhar, Dan Casas, and Christian Theobalt. 2017. Real-time Hand Tracking under Occlusion from an Egocentric RGB-D Sensor. In Proceedings of International Conference on Computer Vision (ICCV). http: //handtracker.mpi-inf.mpg.de/projects/OccludedHands/Google Scholar
- Masa Ogata, Yuta Sugiura, Yasutoshi Makino, Masahiko Inami, and Michita Imai. 2013. SenSkin: adapting skin as a soft interface. In Proceedings of the 26th annual ACM symposium on User interface software and technology. ACM, 539--544. Google ScholarDigital Library
- Masa Ogata, Yuta Sugiura, Hirotaka Osawa, and Michita Imai. 2012. iRing: Intelligent Ring Using Infrared Reflection. Proceedings of the 25th annual ACM symposium on User interface software and technology - UIST '12 (2012), 131--136. Google ScholarDigital Library
- Alan Poston. 2000. Human engineering design data digest. Washington, DC: Department of Defense Human Factors Engineering Technical Advisory Group (2000).Google Scholar
- Manuel Prätorius, Dimitar Valkov, Ulrich Burgbacher, and Klaus Hinrichs. 2014. DigiTap: an eyes-free VR/AR symbolic input device. In Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology. ACM, 9--18. Google ScholarDigital Library
- Chen Qian, Xiao Sun, Yichen Wei, Xiaoou Tang, and Jian Sun. 2014. Realtime and robust hand tracking from depth. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1106--1113. Google ScholarDigital Library
- Grégory Rogez, Maryam Khademi, JS Supancic III, Jose Maria Martinez Montiel, and Deva Ramanan. 2014. 3D hand pose detection in egocentric RGB-D images. In Workshop at the European Conference on Computer Vision. Springer, 356--371.Google Scholar
- Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-Net: Convolutional Networks for Biomedical Image Segmentation. CoRR abs/1505.04597 (2015). http://arxiv.org/abs/1505.04597Google Scholar
- Kyeongeun Seo and Hyeonjoong Cho. 2014. AirPincher: a handheld device for recognizing delicate mid-air hand gestures. Proceedings of the adjunct publication of the 27th annual ACM symposium on User interface software and technology - UIST'14 Adjunct (2014), 83--84. Google ScholarDigital Library
- Toby Sharp, Cem Keskin, Duncan Robertson, Jonathan Taylor, Jamie Shotton, David Kim, Christoph Rhemann, Ido Leichter, Alon Vinnikov, Yichen Wei, and others. 2015. Accurate, robust, and flexible real-time hand tracking. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 3633--3642. Google ScholarDigital Library
- Ayan Sinha, Chiho Choi, and Karthik Ramani. 2016. Deephand: Robust hand pose estimation by completing a matrix imputed with deep features. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4150--4158.Google ScholarCross Ref
- Srinath Sridhar, Anders Markussen, Antti Oulasvirta, Christian Theobalt, and Sebastian Boring. 2017. WatchSense: On- and Above-Skin Input Sensing through a Wearable Depth Sensor. In Proceedings of ACM CHI. 12. http: //handtracker.mpi-inf.mpg.de/projects/WatchSense/ Google ScholarDigital Library
- Srinath Sridhar, Franziska Mueller, Antti Oulasvirta, and Christian Theobalt. 2015. Fast and robust hand tracking using detection-guided optimization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3213--3221.Google ScholarCross Ref
- Carsten Stoll, Nils Hasler, Juergen Gall, Hans-Peter Seidel, and Christian Theobalt. 2011. Fast articulated motion tracking using a sums of gaussians body model. In Computer Vision (ICCV), 2011 IEEE International Conference on. IEEE, 951--958. Google ScholarDigital Library
- Emi Tamaki, Takashi Miyaki, and Jun Rekimoto. 2009. Brainy hand: an ear-worn hand gesture interaction device. In CHI'09 Extended Abstracts on Human Factors in Computing Systems. ACM, 4255--4260. Google ScholarDigital Library
- Jonathan Taylor, Lucas Bordeaux, Thomas Cashman, Bob Corish, Cem Keskin, Toby Sharp, Eduardo Soto, David Sweeney, Julien Valentin, Benjamin Luff, and others. 2016. Efficient and precise interactive hand tracking through joint, continuous optimization of pose and correspondences. ACM Transactions on Graphics (TOG) 35, 4 (2016), 143. Google ScholarDigital Library
- Jonathan Taylor, Vladimir Tankovich, Danhang Tang, Cem Keskin, David Kim, Philip Davidson, Adarsh Kowdle, and Shahram Izadi. 2017. Articulated distance fields for ultra-fast tracking of hands interacting. ACM Transactions on Graphics (TOG) 36, 6 (2017), 244. Google ScholarDigital Library
- Anastasia Tkach, Mark Pauly, and Andrea Tagliasacchi. 2016. Sphere-Meshes for Real-Time Hand Modeling and Tracking. ACM Transaction on Graphics (Proc. SIGGRAPH Asia) (2016). Google ScholarDigital Library
- Jonathan Tompson, Murphy Stein, Yann Lecun, and Ken Perlin. 2014. Real-time continuous pose recovery of human hands using convolutional networks. ACM Transactions on Graphics (ToG) 33, 5 (2014), 169. Google ScholarDigital Library
- Hsin-Ruey Tsai, Min-Chieh Hsiu, Jui-Chun Hsiao, Lee-Ting Huang, Mike Chen, and Yi-Ping Hung. 2016a. TouchRing. Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct - MobileHCI '16 (2016), 891--898. Google ScholarDigital Library
- Hsin-Ruey Tsai, Cheng-Yuan Wu, Lee-Ting Huang, and Yi-Ping Hung. 2016b. ThumbRing: private interactions using one-handed thumb motion input on finger segments. Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct (2016), 791--798. Google ScholarDigital Library
- Hsin-ruey Tsai, Te-Yen Wu, Da-Yuan Huang, Min-Chieh Hsiu, Jui-Chun Hsiao, Yi-Ping Hung, Mike Y. Chen, and Bing-yu Chen. 2017. SegTouch: Enhancing Touch Input While Providing Touch Gestures on Screens Using Thumb-To-Index-Finger Gestures. Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems - CHI EA '17 (2017), 2164--2171. Google ScholarDigital Library
- Koji Tsukadaa and Michiaki Yasumurab. 2001. Ubi-finger: Gesture input device for mobile use. In Ubicomp 2001 Informal Companion Proceedings. 11.Google Scholar
- Dimitrios Tzionas, Luca Ballan, Abhilash Srikantha, Pablo Aponte, Marc Pollefeys, and Juergen Gall. 2016. Capturing Hands in Action Using Discriminative Salient Points and Physics Simulation. International Journal of Computer Vision 118, 2 (01 Jun 2016), 172--193. Google ScholarDigital Library
- Dimitar Valkov, Ulrich Burgbacher, Klaus Hinrichs, and Computer Graphics. 2014. DigiTap: An Eyes-Free VR/AR Symbolic Input Device. (2014), 9--18. Google ScholarDigital Library
- Chengde Wan, Thomas Probst, Luc Van Gool, and Angela Yao. 2017. Crossing Nets: Combining GANs and VAEs With a Shared Latent Space for Hand Pose Estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Google ScholarCross Ref
- Saiwen Wang, Jie Song, Jamie Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum. Proceedings of the 29th Annual Symposium on User Interface Software and Technology - UIST '16 (2016), 851--860. Google ScholarDigital Library
- Martin Weigel, Tong Lu, Gilles Bailly, Antti Oulasvirta, Carmel Majidi, and Jürgen Steimle. 2015. Iskin: flexible, stretchable and visually customizable on-body touch sensors for mobile computing. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems. ACM, 2991--3000. Google ScholarDigital Library
- Martin Weigel, Aditya Shekhar Nittala, Alex Olwal, and Jürgen Steimle. 2017. SkinMarks: Enabling Interactions on Body Landmarks Using Conformal Skin Electronics. In Proceedings of the 35rd Annual ACM Conference on Human Factors in Computing Systems. ACM. Google ScholarDigital Library
- Eric Whitmire, Mohit Jain, Divye Jain, Greg Nelson, and Ravi Karkar. 2017. DigiTouch : Reconfigurable Thumb-to-Finger Input and Text Entry on Head-mounted Displays. 1, 3 (2017), 1--21. Google ScholarDigital Library
- Christian Winkler, Julian Seifert, David Dobbelstein, and Enrico Rukzio. 2014. Pervasive Information Through Constant Personal Projection: The Ambient Mobile Pervasive Display (AMP-D). In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, New York, NY, USA, 4117--4126. Google ScholarDigital Library
- Katrin Wolf, Anja Naumann, Michael Rohs, and Jörg Müller. 2011. A taxonomy of microinteractions: Defining microgestures based on ergonomic and scenario-Dependent requirements. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) 6946 LNCS, PART 1 (2011), 559--575.Google Scholar
- Sang Ho Yoon, Ke Huo, Vinh P. Nguyen, and Karthik Ramani. 2015. TIMMi: Finger-worn Textile Input Device with Multimodal Sensing in Mobile Interaction. Proceedings of the Ninth International Conference on Tangible, Embedded, and Embodied Interaction - TEI '14 (2015), 269--272. Google ScholarDigital Library
- Sang Ho Yoon, Ke Huo, and Karthik Ramani. 2014. Plex. Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing Adjunct Publication - UbiComp '14 Adjunct (2014), 191--194.Google Scholar
- Matthew D Zeiler. 2012. ADADELTA: an adaptive learning rate method. arXiv preprint arXiv:1212.5701 (2012).Google Scholar
- Cheng Zhang, Anandghan Waghmare, Pranav Kundra, Yiming Pu, Scott Gilliland, Thomas Ploetz, Thad E Starner, Omer T Inan, and Gregory D Abowd. 2017. FingerSound: Recognizing unistroke thumb gestures using a ring. 1, 3 (2017), 1--19. Google ScholarDigital Library
Index Terms
- FingerInput: Capturing Expressive Single-Hand Thumb-to-Finger Microgestures
Recommendations
SoloFinger: Robust Microgestures while Grasping Everyday Objects
CHI '21: Proceedings of the 2021 CHI Conference on Human Factors in Computing SystemsUsing microgestures, prior work has successfully enabled gestural interactions while holding objects. Yet, these existing methods are prone to false activations caused by natural finger movements while holding or manipulating the object. We address this ...
CyclopsRing: Enabling Whole-Hand and Context-Aware Interactions Through a Fisheye Ring
UIST '15: Proceedings of the 28th Annual ACM Symposium on User Interface Software & TechnologyThis paper presents CyclopsRing, a ring-style fisheye imaging wearable device that can be worn on hand webbings to en- able whole-hand and context-aware interactions. Observing from a central position of the hand through a fisheye perspective, ...
TouchCam: Realtime Recognition of Location-Specific On-Body Gestures to Support Users with Visual Impairments
On-body interaction, which employs the user's own body as an interactive surface, offers several advantages over existing touchscreen devices: always-available control, an expanded input space, and additional proprioceptive and tactile cues that support ...
Comments