ABSTRACT
We show that when users make errors on mobile devices they make immediate and distinct physical responses that can be observed with standard sensors. We used three standard cognitive tasks (Flanker, Stroop and SART) to induce errors from 20 participants. Using simple low-resolution capacitive touch sensors placed around a standard mobile device and the built-in accelerometer, we demonstrate that errors can be predicted at low error rates from micro-adjustments to hand grip and movement in the period shortly after swiping the touchscreen. Specifically, when combining features derived from hand grip and movement we obtain a mean AUC of 0.96 (with false accept and reject rates both below 10%). Our results demonstrate that hand grip and movement provide strong and low latency evidence for mistakes. The ability to detect user errors in this way could be a valuable component in future interaction systems, allowing interfaces to make it easier for users to correct erroneous inputs.
Supplemental Material
- Andrew Campbell, Tanzeem Choudhury, Shaohan Hu, Hong Lu, Matthew K Mukerjee, Mashfiqui Rabbi, and Rajeev DS Raizada. 2010. NeuroPhone: brain-mobile phone interface using a wireless EEG headset. In Proceedings of the second ACM SIGCOMM workshop on Networking, systems, and applications on mobile handhelds. ACM, 3--8. Google ScholarDigital Library
- Julin Candia, Marta C Gonzlez, Pu Wang, Timothy Schoenharl, Greg Madey, and Albert-Lszl Barabsi. 2008. Uncovering individual and collective human dynamics from mobile phone records. Journal of Physics A: Mathematical and Theoretical 41, 22 (2008), 224015.Google ScholarCross Ref
- Chih-Chung Chang and Chih-Jen Lin. 2011. LIBSVM: A library for support vector machines. ACM Transactions on Intelligent Systems and Technology 2 (2011), 27:1--27:27. Issue 3. Google ScholarDigital Library
- Chih-Jen Chang, Chih-Chung; Lin. 2011. LIBSVM: A Library for Support Vector Machines. ACM Trans. Intell. Syst. Technol. 2, 3, Article 27 (May 2011), 27:1--27:27 pages. Google ScholarDigital Library
- Ricardo Chavarriaga, Pierre W Ferrez, and Jose del R Millan. 2008. To err is human: Learning from error potentials in brain-computer interfaces. In Advances in Cognitive Neurodynamics ICCN 2007. Springer, 777--782.Google Scholar
- James Clawson, Kent Lyons, Alex Rudnick, Robert A Iannucci Jr, and Thad Starner. 2008. Automatic whiteout++: correcting mini-QWERTY typing errors using keypress timing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 573--582. Google ScholarDigital Library
- Stephen A Coombes, Kelly M Gamble, James H Cauraugh, and Christopher M Janelle. 2008. Emotional states alter force control during a feedback occluded motor task. Emotion 8, 1 (2008), 104.Google ScholarCross Ref
- Bernardo Dal Seno, Matteo Matteucci, and Luca Mainardi. 2010. Online Detection of P300 and Error Potentials in a BCI Speller. Intell. Neuroscience 2010, Article 11 (Jan. 2010), 1 pages. Google ScholarDigital Library
- Barbara A Eriksen and Charles W Eriksen. 1974. Effects of noise letters upon the identification of a target letter in a nonsearch task. Perception & psychophysics 16, 1 (1974), 143--149.Google Scholar
- Pierre W Ferrez and Jose del R Millan. 2005. You are wrong!-Automatic detection of interaction errors from brain waves. In International Joint Conference on Artificial Intelligence, Vol. 19. LAWRENCE ERLBAUM ASSOCIATES LTD, 1413. Google ScholarDigital Library
- WJ Gehring, MGH Coles, DE Meyer, and E Donchin. 1990. The error-related negativity: an event-related brain potential accompanying errors. Psychophysiology 27, 4 (1990), S34.Google Scholar
- Mehmet Gonen and Ethem Alpaydın. 2011. Multiple kernel learning algorithms. The Journal of Machine Learning Research 12 (2011), 2211--2268. Google ScholarDigital Library
- Joshua Goodman, Gina Venolia, Keith Steury, and Chauncey Parker. 2002. Language modeling for soft keyboards. In Proceedings of the 7th international conference on Intelligent user interfaces. ACM, 194--195. Google ScholarDigital Library
- Yuta Higuchi and Takashi Okada. 2014. User Interface Using Natural Gripping FeaturesGrip UI. (2014).Google Scholar
- Clay B Holroyd and Michael GH Coles. 2002. The neural basis of human error processing: reinforcement learning, dopamine, and the error-related negativity. Psychological review 109, 4 (2002), 679.Google Scholar
- Per-Ola Kristensson and Shumin Zhai. 2005. Relaxing stylus typing precision by geometric pattern matching. In Proceedings of the 10th international conference on Intelligent user interfaces. ACM, 151--158. Google ScholarDigital Library
- M. Lehne, K. Ihme, A.-M. Brouwer, J. van Erp, and T.O. Zander. 2009. Error-related EEG patterns during tactile human-machine interaction. In Affective Computing and Intelligent Interaction and Workshops, 2009. ACII 2009. 3rd International Conference on. 1--9.Google Scholar
- Qiang Li, John Stankovic, Mark Hanson, Adam T Barth, John Lach, Gang Zhou, and others. 2009. Accurate, fast fall detection using gyroscopes and accelerometer-derived posture information. In Wearable and Implantable Body Sensor Networks, 2009. BSN 2009. Sixth International Workshop on. IEEE, 138--143. Google ScholarDigital Library
- I Scott MacKenzie. 2012. Human-computer interaction: An empirical research perspective. Newnes. Google ScholarDigital Library
- Jillian Madison. 2012. Damn you, autocorrect! Random House.Google Scholar
- Perrin Margaux, Maby Emmanuel, Daligault Sébastien, Bertrand Olivier, and Mattout Jérémie. 2012. Objective and subjective evaluation of online error correction during P300-based spelling. Advances in Human-Computer Interaction 2012 (2012), 4. Google ScholarDigital Library
- J Timothy Noteboom, Kerry R Barnholt, and Roger M Enoka. 2001. Activation of the arousal response and impairment of performance increase with anxiety and stressor intensity. Journal of applied physiology 91, 5 (2001), 2093--2101.Google ScholarCross Ref
- Nishkam Ravi, Nikhil Dandekar, Preetham Mysore, and Michael L Littman. 2005. Activity recognition from accelerometer data. In AAAI, Vol. 5. 1541--1546. Google ScholarDigital Library
- Ian H Robertson, Tom Manly, Jackie Andrade, Bart T Baddeley, and Jenny Yiend. 1997. Oops!: performance correlates of everyday attentional failures in traumatic brain injured and normal subjects. Neuropsychologia 35, 6 (1997), 747--758.Google ScholarCross Ref
- Marten K Scheffers and Michael GH Coles. 2000. Performance monitoring in a confusing world: error-related brain activity, judgments of response accuracy, and types of errors. Journal of Experimental Psychology: Human Perception and Performance 26, 1 (2000), 141.Google ScholarCross Ref
- Tom Sharma, Nandita; Gedeon. 2013. Optimal Time Segments for Stress Detection. In Machine Learning and Data Mining in Pattern Recognition, Petra Perner (Ed.). Lecture Notes in Computer Science, Vol. 7988. Springer Berlin Heidelberg, 421--433. Google ScholarDigital Library
- J Ridley Stroop. 1935. Studies of interference in serial verbal reactions. Journal of experimental psychology 18, 6 (1935), 643.Google ScholarCross Ref
- Chi Vi and Sriram Subramanian. 2012. Detecting Error-related Negativity for Interaction Design. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '12). ACM, NY, NY, USA, 493--502. Google ScholarDigital Library
- Chi Thanh Vi, Izdihar Jamil, David Coyle, and Sriram Subramanian. 2014. Error Related Negativity in Observing Interactive Tasks. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '14). ACM, NY, NY, USA, 3787--3796. Google ScholarDigital Library
- Daryl Weir, Henning Pohl, Simon Rogers, Keith Vertanen, and Per Ola Kristensson. 2014. Uncertain text entry on mobile devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. ACM, 2307--2316. Google ScholarDigital Library
Index Terms
- Detecting Swipe Errors on Touchscreens using Grip Modulation
Recommendations
B2B-Swipe: Swipe Gesture for Rectangular Smartwatches from a Bezel to a Bezel
CHI '16: Proceedings of the 2016 CHI Conference on Human Factors in Computing SystemsWe present B2B-Swipe, a single-finger swipe gesture for a rectangular smartwatch that starts at a bezel and ends at a bezel to enrich input vocabulary. There are 16 possible B2B-Swipes because a rectangular smartwatch has four bezels. Moreover, B2B-...
28 frames later: predicting screen touches from back-of-device grip changes
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsWe demonstrate that front-of-screen targeting on mobile phones can be predicted from back-of-device grip manipulations. Using simple, low-resolution capacitive touch sensors placed around a standard phone, we outline a machine learning approach to ...
Exploring back-of-device interaction
UIST '13 Adjunct: Adjunct Proceedings of the 26th Annual ACM Symposium on User Interface Software and TechnologyBack of device interaction is gaining popularity as an alternative input modality in mobile devices, however it is still unclear how the back of device is related to other interactions. My research explores the relationship between hand grip from the ...
Comments