Abstract
Difficulties in social interaction, verbal and non-verbal communications as well as repetitive and atypical patterns of behavior, are typical characteristics of Autism spectrum disorders (ASD). Advances in computer and robotic technology are enabling assistive technologies for intervention in psychiatric disorders such as autism spectrum disorders (ASD) and schizophrenia (SZ). A number of research studies indicate that many children with ASD prefer technology and this preference can be explored to develop systems that may alleviate several challenges of traditional treatment and intervention. The current work presents development of an adaptive virtual reality-based social interaction platform for children with ASD. It is hypothesized that endowing a technological system that can detect the feeling and mental state of the child and adapt its interaction accordingly is of great importance in assisting and individualizing traditional intervention approaches. The proposed system employs sensors such as eye trackers and physiological signal monitors and models the context relevant psychological state of the child from combination of these sensors. Preliminary affect recognition results indicate that psychological states could be determined from peripheral physiological signals and together with other modalities including gaze and performance of the participant, it is viable to adapt and individualize VR-based intervention paradigms.
Chapter PDF
References
Goodwin, M.S.: Enhancing and Accelerating the Pace of Autism Research and Treatment. Focus on Autism and Other Developmental Disabilities 23(2), 125–128 (2008)
Feil-Seifer, D., Matarić, M.: Toward socially assistive robotics for augmenting interventions for children with autism spectrum disorders. Paper presented at the Experimental Robotics 54 (2009)
Bekele, E., Lahiri, U., Davidson, J., Warren, Z., Sarkar, N.: Robot-Mediated Joint Attention Tasks for Children at Risk with ASD: A Step towards Robot-Assisted Intervention. In: International Meeting for Autism Research (IMFAR), San Deigo, CA (2011)
Andreasen, N.C.: Scale for the assessment of positive symptoms. University of Iowa, Iowa City (1984)
Welch, K., Lahiri, U., Liu, C., Weller, R., Sarkar, N., Warren, Z.: An Affect-Sensitive Social Interaction Paradigm Utilizing Virtual Reality Environments for Autism Intervention. Paper presented at the Human-Computer Interaction, Ambient, Ubiquitous and Intelligent Interaction (2009)
Lord, C., Volkmar, F., Lombroso, P.J.: Genetics of childhood disorders: XLII. Autism, part 1: Diagnosis and assessment in autistic spectrum disorders. Journal of the American Academy of Child and Adolescent Psychiatry 41(9), 1134 (2002)
Diagnostic and Statistical Manual of Mental Disorders: Quick reference to the diagnostic criteria from DSM-IV-TR. American Psychiatric Association, Amer Psychiatric Pub Incorporated, Washington, DC (2000)
Demopoulos, C., Hopkins, J., Davis, A.: A Comparison of Social Cognitive Profiles in children with Autism Spectrum Disorders and Attention-Deficit/Hyperactivity Disorder: A Matter of Quantitative but not Qualitative Difference? Journal of Autism and Developmental Disorders, 1–14 (2012)
Gal, E., Bauminger, N., Goren-Bar, D., Pianesi, F., Stock, O., Zancanaro, M., Weiss, P.L.: Enhancing social communication of children with high-functioning autism through a co-located interface. AI & Society 24(1), 75–84 (2009)
Ploog, B.O., Scharf, A., Nelson, D., Brooks, P.J.: Use of Computer-Assisted Technologies (CAT) to Enhance Social, Communicative, and Language Development in Children with Autism Spectrum Disorders. Journal of Autism and Developmental Disorders, 1–22 (2012)
Ganz, M.L.: The lifetime distribution of the incremental societal costs of autism. Archives of Pediatrics and Adolescent Medicine 161(4), 343–349 (2007)
Bernard-Opitz, V., Sriram, N., Nakhoda-Sapuan, S.: Enhancing social problem solving in children with autism and normal children through computer-assisted instruction. Journal of Autism and Developmental Disorders 31(4), 377–384 (2001)
Parsons, S., Mitchell, P., Leonard, A.: The use and understanding of virtual environments by adolescents with autistic spectrum disorders. Journal of Autism and Developmental Disorders 34(4), 449–466 (2004)
Kandalaft, M.R., Didehbani, N., Krawczyk, D.C., Allen, T.T., Chapman, S.B.: Virtual Reality Social Cognition Training for Young Adults with High-Functioning Autism. Journal of Autism and Developmental Disorders, 1-11 (2012)
Lang, P.J., Bradley, M.M., Cuthbert, B.N.: International affective picture system (IAPS): Technical manual and affective ratings. The Center for Research in Psychophysiology, University of Florida, Gainesville, FL (1999)
Herbener, E.S., Song, W., Khine, T.T., Sweeney, J.A.: What aspects of emotional functioning are impaired in schizophrenia? Schizophrenia Research 98(1), 239–246 (2008)
Lahiri, U., Bekele, E., Dohrmann, E., Warren, Z., Sarkar, N.: Design of a Virtual Reality based Adaptive Response Technology for Children with Autism. IEEE Transactions on Neural Systems and Rehabilitation Engineering: A Publication of the IEEE Engineering in Medicine and Biology Society PP (early access), (99), p. 1 (2012)
Zeng, Z., Pantic, M., Roisman, G.I., Huang, T.S.: A survey of affect recognition methods: Audio, visual, and spontaneous expressions. IEEE Transactions on Pattern Analysis and Machine Intelligence 31(1), 39–58 (2009)
Bekele, E., Zheng, Z., Swanson, A., Crittendon, J., Warren, Z., Sarkar, N.: Understanding How Adolescents with Autism Respond to Facial Expressions in Virtual Reality Environments. IEEE Transactions on Visualization and Computer Graphics 19(4), 711–720 (2013)
Parsons, T.D., Rizzo, A.A., Rogers, S., York, P.: Virtual reality in paediatric rehabilitation: A review. Developmental Neurorehabilitation 12(4), 224–238 (2009)
Lahiri, U., Warren, Z., Sarkar, N.: Design of a Gaze-Sensitive Virtual Social Interactive System for Children With Autism. IEEE Transactions on Neural Systems and Rehabilitation Engineering (99), 1–1 (2012)
Kenny, P., Parsons, T., Gratch, J., Leuski, A., Rizzo, A.: Virtual patients for clinical therapist skills training. Paper presented at the Intelligent Virtual Agents (2007)
Leuski, A., Patel, R., Traum, D., Kennedy, B.: Building effective question answering characters. In: Proceedings of the 7th SIGdial Workshop on Discourse and Dialogue, pp. 18–27. Association for Computational Linguistics (2009)
Bekele, E., et al.: A step towards adaptive multimodal virtual social interaction platform for children with autism. In: Stephanidis, C., Antona, M. (eds.) UAHCI/HCII 2013, Part II. LNCS, vol. 8010, pp. 464–473. Springer, Heidelberg (2013)
Picard, R.W.: Affective computing. MIT Press, Cambridge (1997)
Picard, R.W., Vyzas, E., Healey, J.: Toward machine emotional intelligence: Analysis of affective physiological state. IEEE Transactions on Pattern Analysis and Machine Intelligence 23(10), 1175–1191 (2001)
Liu, C., Conn, K., Sarkar, N., Stone, W.: Online affect detection and robot behavior adaptation for intervention of children with autism. IEEE Transactions on Robotics 24(4), 883–896 (2008)
Rani, P., Liu, C., Sarkar, N., Vanman, E.: An empirical study of machine learning techniques for affect recognition in human–robot interaction. Pattern Analysis & Applications 9(1), 58–69 (2006)
Kim, J., Ande, E.: Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and Machine Intelligence 30(12), 2067–2083 (2008)
Hagan, M.T., Menhaj, M.B.: Training feedforward networks with the Marquardt algorithm. IEEE Transactions on Neural Networks 5(6), 989–993 (1994)
Riedmiller, M., Braun, H.: A direct adaptive method for faster backpropagation learning: The RPROP algorithm. In: IEEE, vol. 581, pp. 586–591 (1993)
Battiti, R.: First-and second-order methods for learning: between steepest descent and Newton’s method. Neural Computation 4(2), 141–166 (1992)
Suykens, J.A.K., Vandewalle, J.: Least squares support vector machine classifiers. Neural Processing Letters 9(3), 293–300 (1999)
Platt, J.C.: 12 Fast Training of Support Vector Machines using Sequential Minimal Optimization (1998)
Salvucci, D.D., Goldberg, J.H.: Identifying fixations and saccades in eye-tracking protocols. Paper presented at the Proceedings of the 2000 Symposium on Eye Tracking Research & Applications (2000)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2014 Springer International Publishing Switzerland
About this paper
Cite this paper
Bekele, E. et al. (2014). Multimodal Interfaces and Sensory Fusion in VR for Social Interactions. In: Shumaker, R., Lackey, S. (eds) Virtual, Augmented and Mixed Reality. Designing and Developing Virtual and Augmented Environments. VAMR 2014. Lecture Notes in Computer Science, vol 8525. Springer, Cham. https://doi.org/10.1007/978-3-319-07458-0_2
Download citation
DOI: https://doi.org/10.1007/978-3-319-07458-0_2
Publisher Name: Springer, Cham
Print ISBN: 978-3-319-07457-3
Online ISBN: 978-3-319-07458-0
eBook Packages: Computer ScienceComputer Science (R0)