Jun Wang
Grace Ngai
ABSTRACT
The task of summarizing a document is a complex task that requires a person to multitask between reading and writing processes. Since a person's cognitive load during reading or writing is known to be dependent upon the level of comprehension or difficulty of the article, this suggests that it should be possible to analyze the cognitive process of the user when carrying out the task, as evidenced through their eye gaze and typing features, to obtain an insight into the different difficulty levels. In this paper, we categorize the summary writing process into different phases and extract different gaze and typing features from each phase according to characteristics of eye-gaze behaviors and typing dynamics. Combining these multimodal features, we build a classifier that achieves an accuracy of 91.0% for difficulty level detection, which is around 55% performance improvement above the baseline and at least 15% improvement above models built on a single modality. We also investigate the possible reasons for the superior performance of our multimodal features.
Supplemental Material
- Nahla J Abid, Bonita Sharif, Natalia Dragan, Hend Alrasheed, and Jonathan I Maletic. 2019. Developer reading behavior while summarizing java methods: Size and context matters. In 2019 IEEE/ACM 41st International Conference on Software Engineering (ICSE). IEEE, 384--395.Google Scholar
Digital Library
- Suleyman Al-Showarah, AL-Jawad Naseer, and Harin Sellahewa. 2014. Effects of user age on smartphone and tablet use, measured with an eye-tracker via fixation duration, scan-path duration, and saccades proportion. In International Conference on Universal Access in Human-Computer Interaction. Springer, 3--14.Google ScholarCross Ref
- Ralf Biedert, Jörn Hees, Andreas Dengel, and Georg Buscher. 2012. A robust realtime reading-skimming classifier. In Proceedings of the Symposium on Eye Tracking Research and Applications. 123--130.Google ScholarDigital Library
- Ann L Brown and Jeanne D Day. 1983. Macrorules for summarizing texts: The development of expertise. Journal of Verbal Learning and Verbal Behavior 22, 1 (1983), 1--14.Google ScholarCross Ref
- Russell LC Butsch. 1932. Eye movements and the eye-hand span in typewriting. Journal of Educational Psychology 23, 2 (1932), 104.Google ScholarCross Ref
- Shiwei Cheng, Zhiqiang Sun, Lingyun Sun, Kirsten Yee, and Anind K Dey. 2015. Gaze-based annotations for reading comprehension. In Proceedings of the 33rd annual ACM Conference on Human Factors in Computing Systems. 1569--1572.Google ScholarDigital Library
- Evgeny Chukharev-Khudilaynen. 2014. Pauses in spontaneous written communication: A keystroke logging study. Journal of Writing Research 6, 1 (2014), 61.Google ScholarCross Ref
- Leandro L Di Stasi, Rebekka Renner, Peggy Staehr, Jens R Helmert, Boris M Velichkovsky, José J Cañas, Andrés Catena, and Sebastian Pannasch. 2010. Saccadic peak velocity sensitivity to variations in mental workload. Aviation, Space, and Environmental Medicine 81, 4 (2010), 413--417.Google ScholarCross Ref
- Linda Flower and John R Hayes. 1981. A cognitive process theory of writing. College Composition and Communication 32, 4 (1981), 365--387.Google ScholarCross Ref
- Wei Guo and Shiwei Cheng. 2019. An Approach to Reading Assistance with Eye Tracking Data and Text Features. In Adjunct of the 2019 International Conference on Multimodal Interaction. 1--7.Google ScholarDigital Library
- Julie A Jacko. 2012. Human computer interaction handbook: Fundamentals, evolving technologies, and emerging applications. CRC press.Google Scholar
- Casey Keck. 2006. The use of paraphrase in summary writing: A comparison of L1 and L2 writers. Journal of Second Language Writing 15, 4 (2006), 261--278.Google ScholarCross Ref
- Kai Kunze, Yuzuko Utsumi, Yuki Shiga, Koichi Kise, and Andreas Bulling. 2013. I know what you are reading: recognition of document types using mobile eye tracking. In Proceedings of the 2013 International Symposium on Wearable Computers. 113--116.Google ScholarDigital Library
- Jiajia Li, Grace Ngai, Hong Va Leong, and Stephen CF Chan. 2016. Multimodal human attention detection for reading from facial expression, eye gaze, and mouse dynamics. ACM SIGAPP Applied Computing Review 16, 3 (2016), 37--49.Google ScholarDigital Library
- Yanping Liu, Siyuan Guo, Lei Yu, and Erik D Reichle. 2018. Word predictability affects saccade length in Chinese reading: An evaluation of the dynamic-adjustment model. Psychonomic Bulletin & Review 25, 5 (2018), 1891--1899.Google ScholarCross Ref
- Stuart Lloyd. 1982. Least squares quantization in PCM. IEEE Transactions on Information Theory 28, 2 (1982), 129--137.Google ScholarDigital Library
- Michael E Masson. 1982. Cognitive processes in skimming stories. Journal of Experimental Psychology: Learning, Memory, and Cognition 8, 5 (1982), 400.Google ScholarCross Ref
- Kevin Ong, Kalervo Järvelin, Mark Sanderson, and Falk Scholer. 2018. Qwerty: The effects of typing on web search behavior. In Proceedings of the 2018 Conference on Human Information Interaction & Retrieval. 281--284.Google ScholarDigital Library
- Alexandra Papoutsaki, Aaron Gokaslan, James Tompkin, Yuze He, and Jeff Huang. 2018. The eye of the typer: a benchmark and analysis of gaze behavior during typing. In Proceedings of the 2018 ACM Symposium on Eye Tracking Research & Applications. 1--9.Google ScholarDigital Library
- Keith Rayner, Kathryn H Chace, Timothy J Slattery, and Jane Ashby. 2006. Eye movements as reflections of comprehension processes in reading. Scientific Studies of Reading 10, 3 (2006), 241--255.Google ScholarCross Ref
- Peter J Rousseeuw. 1987. Silhouettes: a graphical aid to the interpretation and validation of cluster analysis. J. Comput. Appl. Math. 20 (1987), 53--65.Google ScholarDigital Library
- Dario D Salvucci and Joseph H Goldberg. 2000. Identifying fixations and saccades in eye-tracking protocols. In Proceedings of the 2000 Symposium on Eye Tracking Research & Applications. 71--78.Google ScholarDigital Library
- Charles Lima Sanches, Olivier Augereau, and Koichi Kise. 2017. Using the eye gaze to predict document reading subjective understanding. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Vol. 8. IEEE, 28--31.Google ScholarCross Ref
- Tayyar Sen and Ted Megaw. 1984. The effects of task variables and prolonged performance on saccadic eye movement parameters. In Advances in Psychology. Vol. 22. Elsevier, 103--111.Google Scholar
- Dave M Stampe. 1993. Heuristic filtering and reliable calibration methods for video-based pupil-tracking systems. Behavior Research Methods, Instruments, & Computers 25, 2 (1993), 137--142.Google ScholarCross Ref
- Lisa M Vizer. 2009. Detecting cognitive and physical stress through typing behavior. In CHI'09 Extended Abstracts on Human Factors in Computing Systems. 3113--3116.Google ScholarDigital Library
- Jun Wang, Eugene Yujun Fu, Grace Ngai, and Hong Va Leong. 2019. Investigating Differences in Gaze and Typing Behavior Across Age Groups and Writing Genres. In 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), Vol. 1. IEEE, 622--629.Google Scholar
- Suowei Wu, Zhengyin Du, Weixin Li, Di Huang, and Yunhong Wang. 2019. Continuous Emotion Recognition in Videos by Fusing Facial Expression, Head Pose and Eye Gaze. In 2019 International Conference on Multimodal Interaction. 40--48.Google ScholarDigital Library
- Hui-Chun Yang. 2014. Toward a model of strategies and summary writing performance. Language Assessment Quarterly 11, 4 (2014), 403--431.Google ScholarCross Ref
- Luxin Yang and Ling Shi. 2003. Exploring six MBA students? summary writing by introspection. Journal of English for Academic Purposes 2, 3 (2003), 165--192.Google ScholarCross Ref
- Kun Yi, Yu Guo, Weifeng Jiang, Zhi Wang, and Lifeng Sun. 2020. A dataset for exploring gaze behaviors in text summarization. In Proceedings of the 11th ACM Multimedia Systems Conference. 243--248.Google ScholarDigital Library
Index Terms
- Hand-eye Coordination for Textual Difficulty Detection in Text Summarization
Recommendations
Eye, Head and Torso Coordination During Gaze Shifts in Virtual Reality
Humans perform gaze shifts naturally through a combination of eye, head and body movements. Although gaze has been long studied as input modality for interaction, this has previously ignored the coordination of the eyes, head and body. This article ...
The effect of clicking by smiling on the accuracy of head-mounted gaze tracking
ETRA '12: Proceedings of the Symposium on Eye Tracking Research and ApplicationsThe effect of facial behaviour on gaze tracking accuracy was studied while using a prototype system that integrated head-mounted, video-based gaze tracking and a capacitive facial movement detection for respective pointing and selecting objects in a ...
4th international workshop on pervasive eye tracking and mobile eye-based interaction
UbiComp '14 Adjunct: Proceedings of the 2014 ACM International Joint Conference on Pervasive and Ubiquitous Computing: Adjunct PublicationPrevious work on eye tracking and eye-based human-computer interfaces mainly concentrated on making use of the eyes in traditional desktop settings. With the recent growth of interest in smart glass devices and low-cost eye trackers, however, gaze-based ...
Comments