Dhruva Bansal
ABSTRACT
Deaf children born to hearing parents lack continuous access to language, leading to weaker working memory compared to hearing children and deaf children born to Deaf parents. CopyCat is a game where children communicate with the computer via American Sign Language (ASL), and it has been shown to improve language skills and working memory. Previously, CopyCat depended on unscalable hardware such as custom gloves for sign verification, but modern 4K cameras and pose estimators present new opportunities. Before re-creating the CopyCat game for deaf children using off-the-shelf hardware, we evaluate whether current ASL recognition is sufficient. Using Hidden Markov Models (HMMs), user independent word accuracies were 90.6%, 90.5%, and 90.4% for AlphaPose, Kinect, and MediaPipe, respectively. Transformers, a state-of-the-art model in natural language processing, performed 17.0% worse on average. Given these results, we believe our current HMM-based recognizer can be successfully adapted to verify children’s signing while playing CopyCat.
- Helene Brashear, Valerie Henderson, Kwang-Hyun Park, Harley Hamilton, Seungyon Lee, and Thad Starner. 2006. American sign language recognition in game development for deaf children. In Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility. 79–86. https://doi.org/10.1145/1168987.1169002Google Scholar
Digital Library
- Helene Brashear, Thad Starner, Paul Lukowicz, and Holger Junker. 2006. Using multiple sensors for mobile sign language recognition. In Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibility. 79–86. https://doi.org/10.1109/iswc.2003.1241392Google ScholarCross Ref
- P Margaret Brown and Andrew Cornes. 2015. Mental health of deaf and hard-of-hearing adolescents: what the students say. Journal of deaf studies and deaf education 20, 1 (2015), 75–81. https://doi.org/10.1093/deafed/enu031Google ScholarCross Ref
- Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden. 2020. Sign Language Transformers: Joint End-to-End Sign Language Recognition and Translation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr42600.2020.01004Google ScholarCross Ref
- Helen Cooper and Richard Bowden. 2010. Sign language recognition using linguistically derived sub-units. In Proceedings of 4th workshop on the representation and processing of sign languages: corpora and sign language technologies. 57–61.Google Scholar
- Helen Cooper, Eng-Jon Ong, Nicolas Pugeault, and Richard Bowden. 2012. Sign language recognition using sub-units. The Journal of Machine Learning Research 13, 1 (2012), 2205–2231. https://doi.org/10.1007/978-3-319-57021-1_3Google ScholarCross Ref
- David Coulter, Yijie Wang, and Phil Meadows. [n.d.]. Azure Kinect body tracking joints. https://docs.microsoft.com/en-us/azure/kinect-dk/body-joints. Accessed: 2020-01-07.Google Scholar
- M. Ebrahim Al-Ahdal and M. T. Nooritawati. 2012. Review in Sign Language Recognition Systems. In 2012 IEEE Symposium on Computers Informatics (ISCI). 52–57. https://doi.org/10.1109/ISCI.2012.6222666Google ScholarCross Ref
- Hao-Shu Fang, Shuqin Xie, Yu-Wing Tai, and Cewu Lu. 2017. RMPE: Regional Multi-person Pose Estimation. In ICCV. https://doi.org/10.1109/iccv.2017.256Google ScholarCross Ref
- Barak Freedman, Alexander Shpunt, Meir Machline, and Yoel Arieli. 2013. Depth mapping using projected patterns. US Patent 8,493,496.Google Scholar
- Enrique Garcia-Ceja, Michael Riegler, Anders K. Kvernberg, and Jim Torresen. 2020. User-adaptive models for activity and emotion recognition using deep transfer learning and data augmentation. User Modeling and User-Adapted Interaction 30, 3 (July 2020), 365–393. https://doi.org/10.1007/s11257-019-09248-1Google ScholarDigital Library
- K. Grobel and M. Assan. 1997. Isolated sign language recognition using hidden Markov models. In 1997 IEEE International Conference on Systems, Man, and Cybernetics. Computational Cybernetics and Simulation, Vol. 1. 162–167 vol.1. https://doi.org/10.1109/ICSMC.1997.625742Google ScholarCross Ref
- Jie Huang, Wengang Zhou, Qilin Zhang, Houqiang Li, and Weiping Li. 2018. Video-based sign language recognition without temporal segmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 32.Google ScholarCross Ref
- Hamid Reza Vaezi Joze and Oscar Koller. 2018. Ms-asl: A large-scale data set and benchmark for understanding american sign language. arXiv preprint arXiv:1812.01053(2018).Google Scholar
- Kourosh Khoshelham and Sander Oude Elberink. 2012. Accuracy and resolution of kinect depth data for indoor mapping applications. Sensors 12, 2 (2012), 1437–1454. https://doi.org/10.3390/s120201437Google ScholarCross Ref
- Diederik P. Kingma and Jimmy Ba. 2014. Adam: A Method for Stochastic Optimization. http://arxiv.org/abs/1412.6980 cite arxiv:1412.6980Comment: Published as a conference paper at the 3rd International Conference for Learning Representations, San Diego, 2015.Google Scholar
- Sang-Ki Ko, Chang Jo Kim, Hyedong Jung, and Choongsang Cho. 2019. Neural sign language translation based on human keypoint estimation. Applied Sciences 9, 13 (2019), 2683. https://doi.org/10.3390/app9132683Google ScholarCross Ref
- Oscar Koller, Hermann Ney, and Richard Bowden. 2016. Deep Hand: How to Train a CNN on 1 Million Hand Images When Your Data Is Continuous and Weakly Labelled. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). https://doi.org/10.1109/cvpr.2016.412Google ScholarCross Ref
- Yong-Lu Li, Liang Xu, Xinpeng Liu, Xijie Huang, Yue Xu, Shiyi Wang, Hao-Shu Fang, Ze Ma, Mingyang Chen, and Cewu Lu. 2020. PaStaNet: Toward Human Activity Knowledge Engine. In CVPR.Google Scholar
- Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg, and Matthias Grundmann. 2019. MediaPipe: A Framework for Building Perception Pipelines. CoRR abs/1906.08172(2019). arxiv:1906.08172http://arxiv.org/abs/1906.08172Google Scholar
- Rachel I Mayberry and Ellen B Eichen. 1991. The long-lasting advantage of learning sign language in childhood: Another look at the critical period for language acquisition. Journal of memory and language 30, 4 (1991), 486–512. https://doi.org/10.1016/0749-596x(91)90018-fGoogle ScholarCross Ref
- S. Mazilu, M. Hardegger, Z. Zhu, D. Roggen, G. Tröster, M. Plotnik, and J. M. Hausdorff. 2012. Online detection of freezing of gait with smartphones and machine learning techniques. In 2012 6th International Conference on Pervasive Computing Technologies for Healthcare (PervasiveHealth) and Workshops. 123–130. https://doi.org/10.4108/icst.pervasivehealth.2012.248680Google ScholarCross Ref
- Ross E Mitchell and MICHAELA KARCHMER. 2004. Chasing the mythical ten percent: Parental hearing status of deaf and hard of hearing students in the United States. Sign language studies 4, 2 (2004), 138–163. https://doi.org/10.1353/sls.2004.0005Google ScholarCross Ref
- Elissa L Newport. 1990. Maturational constraints on language learning. Cognitive science 14, 1 (1990), 11–28. https://doi.org/10.1207/s15516709cog1401_2Google ScholarCross Ref
- Robin E Perkins-Dock Ph D, Terrilyn R Battle MS, Jaleassia M Edgerton MS, and Jaqueline N McNeill MS. 2015. A survey of barriers to employment for individuals who are deaf. JADARA 49, 2 (2015), 3.Google Scholar
- Patricia M Sullivan and John F Knutson. 2000. Maltreatment and disabilities: A population-based epidemiological study. Child abuse & neglect 24, 10 (2000), 1257–1273. https://doi.org/10.1016/s0145-2134(00)00190-3Google ScholarCross Ref
- Oliver Turner, Kirsten Windfuhr, and Navneet Kapur. 2007. Suicide in deaf populations: a literature review. Annals of General Psychiatry 6, 1 (2007), 26. https://doi.org/10.1186/1744-859X-6-26Google ScholarCross Ref
- Kimberly A Weaver, Harley Hamilton, Zahoor Zafrulla, Helene Brashear, Thad Starner, Peter Presti, and Amy Bruckman. 2010. Improving the language ability of deaf signing children through an interactive American Sign Language-based video game. (2010).Google Scholar
- Yuliang Xiu, Jiefeng Li, Haoyu Wang, Yinghong Fang, and Cewu Lu. 2018. Pose Flow: Efficient Online Pose Tracking. In BMVC.Google Scholar
- Kayo Yin. 2020. Sign Language Translation with Transformers. ArXiv abs/2004.00588(2020).Google Scholar
- Zahoor Zafrulla, Helene Brashear, Pei Yin, Peter Presti, Thad Starner, and Harley Hamilton. 2010. American sign language phrase verification in an educational game for deaf children. In 2010 20th International Conference on Pattern Recognition. IEEE, 3846–3849. https://doi.org/10.1109/icpr.2010.937Google ScholarDigital Library
Index Terms
- CopyCat: Using Sign Language Recognition to Help Deaf Children Acquire Language Skills
Index terms have been assigned to the content through auto-classification.
Recommendations
American sign language recognition in game development for deaf children
Assets '06: Proceedings of the 8th international ACM SIGACCESS conference on Computers and accessibilityCopyCat is an American Sign Language (ASL) game, which uses gesture recognition technology to help young deaf children practice ASL skills. We describe a brief history of the game, an overview of recent user studies, and the results of recent work on ...
A gesture-based american sign language game for deaf children
CHI EA '05: CHI '05 Extended Abstracts on Human Factors in Computing SystemsWe present a system designed to facilitate language development in deaf children. The children interact with a computer game using American Sign Language (ASL). The system consists of three parts: an ASL (gesture) recognition engine; an interactive, ...
The impacts of deaf culture on designing with deaf children
OzCHI '17: Proceedings of the 29th Australian Conference on Computer-Human InteractionThis paper will discuss the impacts Deaf culture and the traits of individual Deafness have on the conduct of co-design with Deaf children. A series of design sessions were undertaken with a design team composed of young Deaf children, the first author ...
Comments