An automated assessment framework for atypical prosody and stereotyped idiosyncratic phrases related to autism spectrum disorder

Abstract

Autism Spectrum Disorder (ASD), a neurodevelopmental disability, has become one of the high incidence diseases among children. Studies indicate that early diagnosis and intervention treatments help to achieve positive longitudinal outcomes. In this paper, we focus on the speech and language abnormalities of young children with ASD and present an automated assessment framework in quantifying atypical prosody and stereotyped idiosyncratic phrases related to ASD. For detecting atypical prosody from speech, we propose both the hand-crafted feature based method as well as the end-to-end deep learning framework. First, we use the OpenSMILE toolkit to extract utterance level high dimensional acoustic features followed by a support vector machine (SVM) backend as the conventional baseline. Second, we propose several end-to-end deep neural network setups and configurations to model the atypical prosody label directly from the constant Q transform spectrogram of speech. Third, we apply cross-validation on the training data to perform segments selection and enhance the subject level classification performance. Fourth, we fuse the deep learning based methods with the conventional baseline at the score level to further enhance the overall system performance. For detecting the stereotyped idiosyncratic usage of words or phrases from speech transcripts, we adopt language model, dependency treebank and Term Frequency–Inverse Document Frequency (TF–IDF) in addition to Linguistic Inquiry and Word Count software (LIWC) methods to extract a set of text features followed by a standard SVM backend. We collect a database of spontaneous Mandarin speech recorded during the Autism Diagnostic Observation Schedule (ADOS) Module 2 and Module 3 sessions. The Module 2 part consists of 118 children while the Module 3 part includes 71 children. Experimental results on this database show that our proposed methods can effectively predict the atypical prosody and stereotyped idiosyncratic phrases codes for young children with the risk of ASD. On the two categories classification task, the unweighted accuracy of the aforementioned two tasks are 88.1% and 77.8%, respectively.

Introduction

Autism Spectrum Disorder (ASD) refers to a group of symptoms related to social impairments and communication difficulties. It has become one of the high incidence diseases among children. A recent analysis from the Centers for Disease Control and Prevention estimates that 1 in 68 children has ASD in the United States (Christensen et al., 2016). Early behavioral and educational interventions have been proved to be very successful in many clinical studies. This attaches great significance to the recognition of common ASD behavior patterns and diagnoses at the early stage.

In paralinguistics, prosody relates to several communicative functions such as intonation, tone, pitch, stress, rhythm, etc. Prosody can reflect many important elements of language including the emphasis, contrast and affective state of the speaker (McCann and Peppé, 2003). These are critical information in human communication. Therefore, atypical prosody is one of the common symptoms related to ASD. Specifically, children with ASD may speak in flat, robot-like or a sing-song voice (Fusaroli et al., 2016).

In this work, we not only focus on the speech signal but also study the language patterns for ASD detection. For ASD children who are verbally fluent, they may have various kinds of language communication abnormalities, such as stereotyped, repetitive and idiosyncratic usage of words or phrases. Children with stereotyped idiosyncratic usage of words or phrases often use some inflexible and rigid words and expressions during the conversation. The words or phrases they uttered sometimes may be inappropriate for the context. Moreover, ASD children may create some new and weird words during the conversation.

Both the aforementioned speech and language cues are important for clinicians to perform diagnosis. The Autism Diagnostic Observation Schedule (ADOS) is a standard screening test to help clinicians observe children’s language and behavior patterns relevant to the diagnosis of autism. It consists of a series of structured and semi-structured tasks assessing social interaction, communication, playing, and imaginative usage of materials (Lord et al., 2000). There are four different modules designed mainly according to the subject’s age and linguistic capability. Moreover, speech and language abnormalities are included in all these four modules. The ADOS screening provides codes to quantify the items on an integer scale from “0” to “2” based on the severity of each abnormality category (Gotham et al., 2009). Taking atypical prosody as an example, “0” denotes no abnormal prosody; “1” stands for some changes on pitch/tone, a bit flat/exaggerate intonation, slightly abnormal volume, a little slow/fast/jerky rhythm and “2” implies markedly and consistently abnormalities on the aforementioned aspects (Lord et al., 2000).

In the ADOS screening, therapists need to identify multiple behavior codes related to speech and language, including atypical prosody, stereotyped idiosyncratic phrases, etc. As many research and treatment methods in the psychology field, this kind of evaluation or diagnosis requires experienced experts or clinicians with intensive specialized training. Another issue is the subjective inconsistency between clinicians, which could sometimes make the results ambiguous at some certain levels. Researchers have proposed strategies to utilize speech and language processing techniques to support clinicians with quantitative analysis of ASD children’s prosody (Bone, Black, Lee, Williams, Levitt, Lee, Narayanan, 2012, Chaspari, Provost, Katsamanis, Narayanan, 2012, Bone, Black, Ramakrishna, Grossman, Narayanan, 2015, Bone, Chaspari, Narayanan, 2017) and language patterns (Kumar et al., 2016). Furthermore, since pattern recognition and machine learning methods have demonstrated promising results in modeling behavior symptoms and relationships with expert’s experience (Narayanan, Georgiou, 2013, Xiao, Imel, Georgiou, Atkins, Narayanan, 2015), some automated screening and evaluating tools based on objective measurements directly extracted from recordings are proposed (Gong, Gong, Levy-Lambert, Green, Hogan, Guttag, 2016, Xiao, Can, Gibson, Imel, Atkins, Georgiou, Narayanan, 2016). These automated coding tools showed great potential to be scalable and assist clinicians to analyze the variation trend of a specific symptom in long term monitoring or assessments.

In this paper, we focus on the speech and language abnormalities and present an automated assessment framework to determine the existence and severity level of atypical prosody and stereotyped idiosyncratic phrases for young children under the ADOS Module 2 and 3 setup.

On the speech side, we model the atypical prosody abnormality using both the traditional strategy and the deep learning framework. We demonstrate that the end-to-end techniques can achieve comparable performance against the baseline system even on a small-scale dataset. Since we directly model the ASD related atypical prosody code from the spectrograms in an end-to-end manner, there is no prior domain knowledge required for feature engineering. The fusion of the two systems can further improve the overall system performance at the segment level in terms of the unweighted average recall (UAR). This result shows that the end-to-end framework has great potential in the field of behavior signal processing (BSP) (Black et al., 2013). Moreover, among all the speech segments in an ADOS conversation session, not every segment reflects the atypical prosody information and therefore we adopt the cross validation strategy on the training set to perform segment selection and improve the accuracy on both segment and person levels.

On the language side, besides n-gram language model, categorical word counts from the Linguistic Inquiry and Word Count software (LIWC) these baseline features and maximum entropy classifier (Kumar et al., 2016), we also propose dependency treebank and Term Frequency-Inverse Document Frequency (TF–IDF) these two methods to extract features that are more related to the stereotyped idiosyncratic usage of words or phrases. We concatenate all these four features together and adopt a standard SVM classifier as the backend.

Furthermore, we also investigate the cutoff boundary of the code 0/1/2 by merging 1/2 as a new code to form a binary classification task. Experimental results show that our trained models are more confident at distinguishing between normal and abnormal cases rather than estimating the detailed severity level of abnormal behaviors. In this study, our goal is not just to recognize the three-category code and use it the same way as described in the ADOS manual (adding together all the codes and compare with the cutoff threshold). This two-category code itself can serve as a quantitative measure of atypical prosody or stereotyped idiosyncratic phrases. We can use the proposed method to perform coarse screening. Besides that, we can also fuse the recognized two-category code with other automatic calculated codes from related tests, e.g. respond to name (Liu et al., 2017), response to non-social sound stimuli, joint attention, etc.

The remainder of the paper is organized as follows. Section 2 describes our database. The proposed methods are explained in Section 3 and Section 4, respectively. Experimental results and discussions are presented in Section 5 while conclusions and future works are provided in Section 6.