Sven Mayer
Huy Viet Le
Niels Henze
ABSTRACT
In the last years, touchscreens became the most common input device for a wide range of computers. While touchscreens are truly pervasive, commercial devices reduce the richness of touch input to two-dimensional positions on the screen. Recent work proposed interaction techniques to extend the richness of the input vocabulary using the finger orientation. Approaches for determining a finger's orientation using off-the-shelf capacitive touchscreens proposed in previous work already enable compelling use cases. However, the low estimation accuracy limits the usability and restricts the usage of finger orientation to non-precise input. With this paper, we provide a ground truth data set for capacitive touch screens recorded with a high-precision motion capture system. Using this data set, we show that a Convolutional Neural Network can outperform approaches proposed in previous work. Instead of relying on hand-crafted features, we trained the model based on the raw capacitive images. Thereby we reduce the pitch error by 9.8% and the yaw error by 45.7%.
- Bengio, Y., Courville, A., and Vincent, P. Representation learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence 35, 8 (Aug 2013), 1798--1828. Google Scholar
Digital Library
- Bilgic, B., Chatnuntawech, I., Fan, A. P., Setsompop, K., Cauley, S. F., Wald, L. L., and Adalsteinsson, E. Fast image reconstruction with l2-regularization. Journal of Magnetic Resonance Imaging 40, 1 (2014), 181--191. Google ScholarCross Ref
- Boring, S., Ledo, D., Chen, X. A., Marquardt, N., Tang, A., and Greenberg, S. The fat thumb: Using the thumb's contact size for single-handed mobile interaction. In Proceedings of the 14th International Conference on Human-computer Interaction with Mobile Devices and Services, MobileHCI '12, ACM (New York, NY, USA, 2012), 39--18.Google ScholarDigital Library
- Caruana, R., Lawrence, S., and Giles, C. L. Overfitting in neural nets: Backpropagation, conjugate gradient, and early stopping. In Advances in neural information processing systems (2001), 402--408.Google Scholar
- Chang, W., Kim, K. E., Lee, H., Cho, J. K., Soh, B. S., Shim, J. H., Yang, G., Cho, S.-J., and Park, J. Recognition of grip-patterns by using capacitive touch sensors. In IEEE International Symposium on Industrial Electronics, vol. 4, IEEE (2006), 2936--2941. Google ScholarCross Ref
- Colley, A., and Häkkilä, J. Exploring finger specific touch screen interaction for mobile phone user interfaces. In Proceedings of the 26th Australian Computer-Human Interaction Conference on Designing Futures: The Future of Design, OzCHI '14, ACM (New York, NY, USA, 2014), 539--548. Google ScholarDigital Library
- Coulibaly, P., Anctil, F., and Bobe, B. Daily reservoir inflow forecasting using artificial neural networks with stopped training approach. Journal of Hydrology 230, 3 (2000), 244--257. Google ScholarCross Ref
- Duchi, J., Hazan, E., and Singer, Y. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research 12 (July 2011), 2121--2159.Google Scholar
- Glorot, X., and Bengio, Y. Understanding the difficulty of training deep feedforward neural networks. In In Proceedings of the International Conference on Artificial Intelligence and Statistics. Society for Artificial Intelligence and Statistics, vol. 9 of AISTATS'10, JMLR.org (2010), 249--256.Google Scholar
- Guo, A., Xiao, R., and Harrison, C. Capauth: Identifying and differentiating user handprints on commodity capacitive touchscreens. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces, ITS '15, ACM (New York, NY, USA, 2015), 59--62. Google ScholarDigital Library
- Harrison, C., and Hudson, S. Using shear as a supplemental two-dimensional input channel for rich touchscreen interaction. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '12, ACM (New York, NY, USA, 2012), 3149--3152. Google ScholarDigital Library
- Harrison, C., Schwarz, J., and Hudson, S. E. Tapsense: Enhancing finger interaction on touch surfaces. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST '11, ACM (New York, NY, USA, 2011), 627--636. Google ScholarDigital Library
- Henze, N., Rukzio, E., and Boll, S. 100,000,000 taps: Analysis and improvement of touch performance in the large. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, MobileHCI '11, ACM (New York, NY, USA, 2011), 133--142.Google ScholarDigital Library
- Hinckley, K., Heo, S., Pahud, M., Holz, C., Benko, H., Sellen, A., Banks, R., O'Hara, K., Smyth, G., and Buxton, W. Pre-touch sensing for mobile interaction. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI '16, ACM (New York, NY, USA, 2016), 2869--2881. Google ScholarDigital Library
- Holz, C., and Baudisch, P. The generalized perceived input point model and how to double touch accuracy by extracting fingerprints. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '10, ACM (New York, NY, USA, 2010), 581--590. Google ScholarDigital Library
- Holz, C., and Baudisch, P. Understanding touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, ACM (New York, NY, USA, 2011), 2501--2510. Google ScholarDigital Library
- Holz, C., Buthpitiya, S., and Knaust, M. Bodyprint: Biometric user identification on mobile devices using the capacitive touchscreen to scan body parts. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI '15, ACM (New York, NY, USA, 2015), 3011--3014. Google ScholarDigital Library
- Ioffe, S., and Szegedy, C. Batch normalization: Accelerating deep network training by reducing internal covariate shift. CoRR abs/1502.03167 (2015).Google Scholar
- Kratz, S., Chiu, P., and Back, M. Pointpose: Finger pose estimation for touch input on mobile devices using a depth sensor. In Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces, ITS '13, ACM (New York, NY, USA, 2013), 223--230. Google ScholarDigital Library
- Krizhevsky, A., Sutskever, I., and Hinton, G. E. Imagenet classification with deep convolutional neural networks. In Proceedings of the 25th International Conference on Neural Information Processing Systems, Curran Associates, Inc. (Lake Tahoe, NV, USA, Dec. 2012), 1097--1105.Google Scholar
- Lawrence, N., Seeger, M., and Herbrich, R. Fast sparse gaussian process methods: The informative vector machine. In Proceedings of the 15th International Conference on Neural Information Processing Systems, NIPS '02, MIT Press (Cambridge, MA, USA, 2002), 625--632.Google Scholar
- Le, H. V., Bader, P., Kosch, T., and Henze, N. Investigating screen shifting techniques to improve one-handed smartphone usage. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction, NordiCHI '16, ACM (New York, NY, USA, 2016), 27--37. Google ScholarDigital Library
- Le, H. V., Mayer, S., Bader, P., Bastian, F., and Henze, N. Interaction methods and use cases for a full-touch sensing smartphone. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA '17, ACM (New York, NY, USA, 2017), 2730--2737. Google ScholarDigital Library
- Le, H. V., Mayer, S., Bader, P., and Henze, N. A smartphone prototype for touch interaction on the whole device surface. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, MobileHCI '17, ACM (New York, NY, USA, 2017). Google ScholarDigital Library
- Le, H. V., Mayer, S., Wolf, K., and Henze, N. Finger placement and hand grasp during smartphone interaction. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA '16, ACM (New York, NY, USA, 2016), 2576--2584. Google ScholarDigital Library
- Lorensen, W. E., and Cline, H. E. Marching cubes: A high resolution 3d surface construction algorithm. In Proceedings of the 14th Annual Conference on Computer Graphics and Interactive Techniques, SIGGRAPH '87, ACM (New York, NY, USA, 1987), 163--169. Google ScholarDigital Library
- Mayer, S., Gad, P., Wolf, K., Wozniak, P. W., and Henze, N. Understanding the ergonomic constraints in designing for touch surfaces. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (Vienna, 2017). Google ScholarDigital Library
- Mayer, S., Mayer, M., and Henze, N. Feasibility analysis of detecting the finger orientation with depth camera. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services Adjunct, MobileHCI'17, ACM (New York, NY, USA, 2017), 8. Google ScholarDigital Library
- Murray-Smith, R. Stratified, computational interaction via machine learning. In Eighteenth Yale Workshop on Adaptive and Learning Systems (New Haven, CT, USA, June 2017), 95--101.Google Scholar
- Nair, V., and Hinton, G. E. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning, ICML'10, Omnipress (2010), 807--814.Google ScholarDigital Library
- Oakley, I., Lindahl, C., Le, K., Lee, D., and Islam, M. R. The flat finger: Exploring area touches on smartwatches. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI '16, ACM (New York, NY, USA, 2016), 4238--1249. Google ScholarDigital Library
- Qian, N. On the momentum term in gradient descent learning algorithms. Neural Networks 12, 1 (1999), 145--151. Google ScholarDigital Library
- Ramos, G., Boulos, M., and Balakrishnan, R. Pressure widgets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '04, ACM (New York, NY, USA, 2004), 487--194. Google ScholarDigital Library
- Rogers, S., Williamson, J., Stewart, C., and Murray-Smith, R. Anglepose: Robust, precise capacitive touch tracking via 3d orientation estimation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, ACM (New York, NY, USA, 2011), 2575--2584. Google ScholarDigital Library
- Roudaut, A., Lecolinet, E., and Guiard, Y. Microrolls: Expanding touch-screen input vocabulary by distinguishing rolls vs. slides of the thumb. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '09, ACM (New York, NY, USA, 2009), 927--936. Google ScholarDigital Library
- Simard, P. Y., Steinkraus, D., and Platt, J. C. Best practices for convolutional neural networks applied to visual document analysis. In Proceedings of the Seventh International Conference on Document Analysis and Recognition, vol. 3 of ICDAR '03, IEEE Computer Society (Washington, DC, USA, August 2003), 958--962. Google ScholarCross Ref
- Weir, D., Rogers, S., Murray-Smith, R., and Löchtefeld, M. A user-specific machine learning approach for improving touch accuracy on mobile devices. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology, UIST '12, ACM (New York, NY, USA, 2012), 465--176. Google ScholarDigital Library
- Wilkinson, G., Kharrufa, A., Hook, J., Pursglove, B., Wood, G., Haeuser, H., Hammerla, N. Y., Hodges, S., and Olivier, P. Expressy: Using a wrist-worn inertial measurement unit to add expressiveness to touch-based interactions. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI '16, ACM (New York, NY, USA, 2016), 2832--2844. Google ScholarDigital Library
- Williamson, J. Fingers of a hand oscillate together: Phase syncronisation of tremor in hover touch sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, CHI '16, ACM (New York, NY, USA, 2016), 3433--3437. Google ScholarDigital Library
- Wong, P. C., Fu, H., and Zhu, K. Back-mirror: Back-of-device one-handed interaction on smartphones. In SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications, SA '16, ACM (New York, NY, USA, 2016), 10:1--10:5.Google ScholarDigital Library
- Xiao, R., Schwarz, J., and Harrison, C. Estimating 3d finger angle on commodity touchscreens. In Proceedings of the 2015 International Conference on Interactive Tabletops & Surfaces, ITS '15, ACM (New York, NY, USA, 2015), 47--50. Google ScholarDigital Library
- Zaliva, V. 3d finger posture detection and gesture recognition on touch surfaces. In Control Automation Robotics & Vision (ICARCV), 2012 12th International Conference on, IEEE (2012), 359--364.Google ScholarCross Ref
Index Terms
- Estimating the Finger Orientation on Capacitive Touchscreens Using Convolutional Neural Networks
Recommendations
Estimating 3D Finger Angle on Commodity Touchscreens
ITS '15: Proceedings of the 2015 International Conference on Interactive Tabletops & SurfacesWe describe a novel approach for estimating the pitch and yaw of fingers relative to a touchscreen's surface, offering two additional, analog degrees of freedom for interactive functions. Further, we show that our approach can be achieved on off-the-...
Investigating the feasibility of finger identification on capacitive touchscreens using deep learning
IUI '19: Proceedings of the 24th International Conference on Intelligent User InterfacesTouchscreens enable intuitive mobile interaction. However, touch input is limited to 2D touch locations which makes it challenging to provide shortcuts and secondary actions similar to hardware keyboards and mice. Previous work presented a wide range of ...
Designing finger orientation input for mobile touchscreens
MobileHCI '18: Proceedings of the 20th International Conference on Human-Computer Interaction with Mobile Devices and ServicesA large number of today's systems use interactive touch surfaces as the main input channel. Current devices reduce the richness of touch input to two-dimensional positions on the screen. A growing body of work develops methods that enrich touch input to ...
Comments