https://orcid.org/0000-0002-4662-2468
Figures
Abstract
The use of virtual reality in social skills training for high functioning autism spectrum disorder (HFASD) youth has been found to be engaging and enjoyable. Despite the promising results, previous literature indicates that there has been no consensus on the social skills target in the training content. There is also limited research on how evidence-based strategies like cognitive and behaviour techniques are instantiated into the VR environment to teach social skills. The aim of this study is to determine the key components to design a social skills training content using virtual reality for youths with HFASD. The Fuzzy Delphi method (FDM) was used to obtain expert consensus on social skills difficulties and cognitive behavioral techniques included in the content in three phases. In phase 1, a questionnaire was developed from in-depth interviews and scientific literature review. The in-depth interviews were conducted with 13 HFASD youth, 7 parents and 6 experts. In phase 2, 3 experts rated the relevance of the items in the questionnaire using an item-level content validity index (I-CVI) assessment. In phase 3, the questionnaire was distributed to 10 experts to rate their level of agreement on each component using a 7-point Likert scale. Components that received a value above 75%, threshold value (d) ≤ 0.2, fuzzy score (A) ≥ α - cut value = 0.5 and higher rank based on defuzzification score were prioritized to be included in the content. Items that received higher expert consensus on social skills difficulties included assessing non-verbal responses, initiating, maintaining, and leaving conversations, emotional difficulties and difficulties in perspective taking. Cognitive and behavioral techniques that received higher expert consensus were psychoeducation, modelling, relaxation techniques, reinforcements, and perspective-taking questions. These key components can be used as a framework for the development of virtual learning content for social skills training in future studies.
Citation: Ahmad Basri MAF, Wan Ismail WS, Kamal Nor N, Mohd Tohit N, Ahmad MN, Mohamad Aun NS, et al. (2024) Validation of key components in designing a social skills training content using virtual reality for high functioning autism youth—A Fuzzy Delphi method. PLoS ONE 19(4): e0301517. https://doi.org/10.1371/journal.pone.0301517
Editor: Yu-Wei Ryan Chen, The University of Sydney, AUSTRALIA
Received: May 18, 2023; Accepted: March 14, 2024; Published: April 4, 2024
Copyright: © 2024 Ahmad Basri et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the manuscript.
Funding: The authors acknowledge the Transdisciplinary Research Grant Scheme(TRGS), TRGS/1/2020/UKM/02/6/1 funded by the Ministry of Higher Education(MOHE) Malaysia. We would also like to acknowledge Universiti Kebangsaan Malaysia for supporting this research project. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Autism spectrum disorder (ASD) ranges in a spectrum from low functioning (LFASD) to high functioning (HFASD) individuals. Even though the terms “low” and “high” are not officially used in the DSM-5, published past literature and clinicians in the field usually refer to these terms to indicate differences in behaviors, cognition, language, and social communication [1]. It has been suggested that around 44% of ASD individuals have IQ above 85, which is used as a parameter indicative of HFASD [2]. Even though those with HFASD have normal cognitive ability, social skills difficulties remain a central deficit [3].
Virtual reality (VR) has been recognized as a promising new intervention application that can improve social skills in ASD individuals, mainly due to the enjoyability and engaging aspects of VR that motivates ASD users to learn [4]. Previous studies using VR interventions found improvements in social communication [5], social attention [6], play skills [7], emotional regulation skills [8], and social cognition [9] among individuals with ASD in different age groups.
Despite the promising results in earlier studies, there has been a lack of consensus and inconsistency on the priority of which social skills difficulty to be targeted in the virtual learning content [4, 10–12]. Thai and Nathan-Roberts [11] concluded that “there is a large variation in which specific social skills should be targeted” (p. 1471) in their systematic review of 12 intervention studies on social skills using VR. Similarly, Mesa-Gresa et al. [10] found a large variety of social skills targets in their systematic review of 31 articles. The social skills targets across studies varied from improving emotion recognition [13–16], communication skills [17], social understanding [18], collaboration and reciprocity in behavior and communication [19], performance in facial affect and recognition [20], social responsiveness and executive functioning skills [21], as well as nonverbal and gesturing communication [22]. Hence, determining which social skills target to be prioritized in designing the learning content for social skills training using VR is important.
There is also a dearth of research on the “what” and “how” evidence-based techniques are used to teach social skills in VR form [4, 10–12, 23, 24]. Parsons and Mitchell [24] suggested that contents in VR can be designed to instantiate both behavioral and cognitive approaches to learning to bridge a user-centered design with evidence-based strategies. Even though it is widely known that cognitive and behavioral techniques are effective in physical social skills training [23], there has been a lack of studies exploring how these techniques are instantiated in the form of VR [4, 10–12, 24]. Hence, determining which cognitive and behavioral techniques can be utilized in training social skills using VR is important in designing the learning content.
This study was conducted to determine expert consensus on the selection of social skills difficulties and cognitive and behavioral techniques to be included in the social skills training content using VR for HFASD youths.
Methodology
The Fuzzy Delphi method (FDM) was used to obtain expert consensus on the key components to be included in the content. The FDM is an adaptation from the classic Delphi method, in which it adds a set of fuzzy numbering sets while maintaining the Delphi method itself [25]. The FDM reduces the cycle process to avoid loss of data, leading to a higher economic efficacy in time and cost [26]. The FDM has been widely used to validate the components for training contents due to its ability to obtain the value of fuzzy scores in the form of ranking that can be used as a determinant and priority of an element according to expert consensus [27–29]. This study consisted of three phases:
Phase 1
The objective in Phase 1 was to explore the needs analysis of the social skills training content. This was conducted using in-depth interviews and scientific literature review.
The in-depth interviews were conducted with 13 HFASD youths, 7 parents and 6 experts, chosen through a purposive maximal variation sampling technique. We purposely and intentionally selected these participants with specific criteria to strengthen data triangulation from different perspectives. The HFASD youth shared their perspectives from lived experience, parents shared their experience raising and living with HFASD individuals, while the experts shared their experience from a professional perspective. These three groups were chosen as they were identified as key stakeholders in the process of designing a user-friendly learning content [30]. The objective of the in-depth interviews was to explore the social skills difficulties that are faced by HFASD and the cognitive behavioural techniques that can help them to learn social skills in everyday life.
The scientific literature review was explored using four primary databases, which are PubMed, APA PsycINFO, Scopus and Google Scholar using keywords such as (autism OR ASD OR Asperger) AND (social skills OR social communication skills) AND (virtual reality OR VR). Recent articles from January 2010 to December 2022 were exclusively reviewed in the analysis due to the fast-paced advancement in technology in this century, as technology plays a crucial role in VR-related studies. The objective of this is to explore the social skills difficulties and cognitive behavioral techniques that have been used in social skills training using virtual reality.
The inclusion criteria for the groups were:
HFASD Youths
Diagnosed with HFASD by a mental health professional or developmental pediatrician, aged between 15 to 24 years, and able to converse in English or Malay.
Parents
Mother, father, or caregiver of a diagnosed youth with HFASD, and able to converse in English or Malay.
Experts
A clinical psychologist, psychiatrist, speech therapist, or occupational therapist with a minimum of five years of professional experience conducting social skills training with HFASD youth, familiar with cognitive and behavior techniques, and able to converse in English or Malay.
Procedure
Ethics approval was granted from the National University of Malaysia Research Ethics Committee on 11th March 2021 (Ref. No: UKM PPI/111/8/JEP-2021-092). Potential participants were contacted via telephone. They were informed about the study and invited for the interview once they fulfilled the inclusion criteria. Written informed consent for participants above 18 years was obtained via email. For participants under 18 years, written consent was obtained from the parents or guardians. Participants were recruited between April to August 2021. The time and date for the interview was arranged according to the participants’ convenience. All interviews were conducted online due to movement restriction orders during the peak of COVID-19 period in Malaysia. The main language of interview was English, Malay or combination of both, based on the preference of the participants. A semi-structured interview protocol was used. The flow of the questions was mainly influenced by the participants’ cues and context. The questions were mainly to explore the social skills difficulties faced by HFASD youths as well as the cognitive and behavioral techniques that helped in improving their social skills. All interviews were recorded and saved in Health Insurance Portability and Accountability Act 1996 (HIPAA) compliant applications. All authors had access to information that identified the participants during and after data collection. The privacy and confidentiality of the participants’ background information were protected as all participants were identified as initials in recordings, transcripts, analysis, and data reporting.
Data analysis
The recorded interview videos were transcribed verbatim and then transferred to Nvivo 12. The qualitative analysis begins with open coding in which the transcripts are read line by line and participants’ exact phrases are used as initial codes. Each of the codes are compared with other data in the process of developing a theme. Codes that match each other were categorized under the same themes, while codes that did not match were categorized into a new theme. Similar themes were then analyzed whether it fits into a broader categorization. This approach follows an inductive method.
Then, we consistently reviewed the data to identify any quotes that may have been overlooked and should have been incorporated into the established themes and sub-themes. This approach follows a deductive method. This process continues until all relevant quotes from the transcripts have been categorized under themes or subthemes and there are no longer new themes that emerge from the transcripts.
The qualitative data was triangulated between each of the groups (HFASD, parents and experts). The themes and sub-themes were then verified with the scientific literature review findings. The themes and sub-themes that had support from literature review were converted into items in a questionnaire, which was then reviewed in Phase 2.
Phase 2
The objective in Phase 2 was to review the relevance of items in the questionnaire developed in Phase 1 using an item-level content validity index (I-CVI) assessment. The I-CVI assessment is a critical step in enhancing the validity of the questionnaire [31]. Three experts were chosen by purposive sampling technique, in accordance with Lynn [32] who suggested 3 experts as the minimum requirement. These experts were selected based on the following criteria:
- A clinical psychologist, psychiatrist, speech therapist, or occupational therapist with a minimum of five years of professional experience conducting social skills training with HFASD youths.
- Familiar with cognitive and behavior techniques
- Able to converse in English or Malay.’
- Did not participate in the previous phase of this study.
Procedure
Potential participants were contacted via telephone. They were informed about the study and invited to become an expert assessor once they fulfilled the inclusion criteria. Written informed consent was obtained via email. Participants were recruited between January to March 2022.The questionnaire developed in Phase 1 was distributed to the three experts. They rated the relevance of each item according to a 4-point ordinal scale: 1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, 4 = highly relevant Items rated 3 or 4 are given a value of 1.00, while items rated as 1 or 2 are given a value of 0 [31].
Data analysis
The value of I-CVI was calculated using the formula; I-CVI = n/N, in which ‘n’ is the number of experts agreed and ‘N’ is the sum of evaluator. Only items with the value of 1.00, which means that all three experts agree on the relevance of the item (rated 3 or 4), were accepted [32]. Items that received scores below 1.00 (rated 1 or 2 by at least 1 expert) were omitted. Past literature states that when the experts are fewer than 5 people, the I-CVI value must be 1.00 –that means, all experts must agree that the item is relevant [32]. The questionnaire containing the items that were accepted in this phase was used in the Fuzzy Delphi method in Phase 3.
Phase 3
The objective in Phase 3 was to determine an expert consensus on the selection of social skills difficulties and cognitive behavioral techniques to be included in the social skills training content using VR. Ten experts chosen by purposive sampling were involved in the process of Fuzzy Delphi method, consistent with suggestions from previous literature [33]. The inclusion criteria of experts in this phase were:
- Clinical psychologist, psychiatrist, speech therapist or occupational therapist with a minimum of five years of professional experience conducting social skills training with HFASD youth.
- Familiar with cognitive behavior techniques.
- Able to converse in English or Malay.
- Did not participate in the previous phases of this study.
Procedure
Potential participants were contacted via telephone. They were informed about the study and invited to become an expert assessor once they fulfilled the inclusion criteria. Written informed consent was obtained via email. Participants were recruited between June to July 2022. The questionnaire developed and reviewed in the previous phases were distributed to the experts via email, who rated their level of agreement on the items using a 7-point Likert scale, ranging from “Strongly Disagree = 1” to “Strongly Agree = 7” as shown in Table 1.
Data analysis
- Step 1: Insert Likert scales into Microsoft Excel
All the expert’s responses were inserted into a Microsoft Excel template [34–37] to analyze the fuzzy delphi scores. - Step 2: Convert the likert scales into triangular fuzzy numbers as suggested in Table 1. This conversion was conducted automatically by Microsoft Excel [34–37].
- Step 3: Determine fuzzy average values from fuzzy scale.
The fuzzy average values, m = (m1, m2, m3) was calculated automatically using the Microsoft Excel template [34–37]. - Step 4: Specify the threshold value, d-construct.
This was calculated automatically using the Microsoft Excel template [34–37]. The vertex method was used to calculate the distance between two fuzzy numbers, as computed by Microsoft Excel using the following formula [34–37]:
An expert agreement is reached if the distance between the average value and the experts’ evaluation of data is smaller or equal to the threshold value (d) 0.2. If it was higher, the item was rejected [34–37]. - Step 5: Gain a 75% consensus among the experts. This was calculated automatically using the Microsoft Excel template [34–37].
- Step 6: Defuzzification process.
The defuzzification process, which is a process used to obtain the ranking of each item in the survey, was conducted. The formula for calculating the defuzzification value was A = (1/3) * (m1 + m2 + m3). The value that was used in defuzzification is the α-cut, which was the median for 0 and 1, α-cut = (0+1)/2 = 0.5. Items with a value of α-cut below 0.5 indicated a rejection, and items with a value above 0.5 indicated acceptance of expert consensus. The value of defuzzification also indicates the ranking for each item. A higher value provided a higher ranking, indicating a higher importance of the item. This was calculated automatically using the Microsoft Excel template [34–37].
Overall, Items that received a threshold value (d) ≤ 0.2, value above 75%, fuzzy score (A) ≥ α - cut value = 0.5 and higher rank based on defuzzification score were prioritized to be included in the content.
Results
Phase 1
Profile of participants.
As shown in Table 2, 13 HFASD youths, 7 parents and 6 experts were involved in the in-depth interview. The gender proportion was equal. In terms of race, 65.4% (n = 17) were Malays, 24% (n = 6) Chinese and 12% (n = 3) Indians. The mean age of the HFASD youth was 19.77 years (SD = 2.74), ranging between 15 to 24 years old. In the parents’ group, the mean age was 48.29 (SD = 5.38), ranging between 42 to 56 years of age. The six experts included four clinical psychologists, one child and adolescent psychiatrist, and one speech language therapist.
Results analysis
The extensive literature review and in-depth interviews results on social skills difficulties identified 5 major themes and 19 sub-themes as shown in Table 3. The major themes were social cognition, social communication skills, social emotions, social language, and stereotyped social behaviors. The results on cognitive and behaviour techniques helpful in training social skills in ASD identified 6 cognitive strategies and 9 behaviour strategies. The sub-themes were converted into items in a questionnaire and reworded in a phrase that would assist the experts to understand better, as shown in Table 5.
Phase 2
As shown in Table 4, the three experts were a child and adolescent psychiatrist, a pediatrician, and a family medicine specialist. All of them had more than 10 years of working experience. Two of them were PhD holders while the other one had master’s degree as the highest level of education.
Results analysis
Sub-themes derived from Phase 1 of this study were converted into items in a questionnaire and reworded into phrases that would be easier to understand as shown in Table 5. Out of the 19 items in the social skills difficulties domain, 15 were accepted, and 4 were rejected. Out of 15 items in the cognitive and behavior techniques domain, 11 were accepted and 4 were rejected. In summary, items of 15 social skills difficulties and 11 cognitive behavior techniques that were rated as most relevant are retained in the questionnaire, which is used in the fuzzy delphi method in Phase 3.
Phase 3
As shown in Table 6, the 10 experts in this phase included 5 clinical psychologists, 1 psychiatrist, 2 developmental psychologists, and 2 psychologists with expertise in digital content development. Most of them (70%, n = 7) had more than 10 years of working experience, while the others (30%, n = 3) had experience between 5 to 10 years. Half of the experts (50%, n = 5) had master’s degree as the highest level of education while the other half (50%, n = 5) were PhD holders.
Results analysis
All 15 items on social skills difficulties met the criteria for item acceptance. As shown in Table 7, the threshold value was ≤ 0.2, expert consensus exceeded 75% and fuzzy score (A) was more than 0.5. These items were then sorted by ranking according to the value of fuzzy score.
As shown in Table 8, items that were rated as high-ranking were difficulties assessing non-verbal response of other people, starting, maintaining, and leaving conversation, social anxiety, understanding another person’s perspective, maintaining appropriate eye contact, inappropriate voice tone and being aware of physical distance between people when talking.
The results of the Fuzzy Delphi analysis in cognitive behavior techniques are shown in Table 9. In general, all 11 items met all three criteria for item acceptance. The threshold value was ≤ 0.2, expert consensus exceeded 75% and fuzzy score (A) was more than 0.5.
These items were then sorted by ranking according to the value of fuzzy score, as shown in Table 10. Items that were rated at high rankings were psychoeducation, modelling, relaxation techniques, reinforcements, role plays, immediate feedback, skill rehearsal, perspective taking questions, Socratic questioning, as well as awareness on cognitive bias.
Discussion
The result of this study provides a clear framework and guideline for researchers, clinicians, and trainers to develop and implement social skills training content using VR which has been validated by expert consensus. The components in this framework could be used as a standardized design for future research on the development of social skills training contents using VR for HFASD youths. This design could ensure that the content meets the needs of the HFASD youths and improve levels of efficacy in training.
Social skills difficulties
One interesting point from this finding was the importance of considering the management of difficult emotions in social skills training. The experts rated ‘social anxiety’ highly as a social skills difficulty and ‘relaxation techniques’ as a strategy to help manage emotions and enhance social skills practice. This is highly important to acknowledge as HFASD individuals commonly report high levels of anxiety and fear of rejection whenever they try to practice their social skills in the real-world environment [38]. Furthermore, the benefit of VR itself, which can simulate real-world social communication scenarios, can provide a safe space for ASD individuals to train themselves without fear of failure or rejection [38].
The experts also rated highly on assessing people’s non-verbal response, as well as difficulties in starting, maintaining, and leaving a conversation—all of which are fundamental challenges in individuals with ASD due to mind-blindness, which means they struggle to attribute mental states to others [39]. This was consistent with the focus of Program for the Education and Enrichment of Relational Skills (PEERS®), developed by Laugeson [40], a face-to-face manualized social skills program with the highest number of efficacy studies [41]. The reason this is important is that high-functioning autism youth often lack the knowledge of the typical steps followed by their neurotypical peers [26]. They are hesitant to engage in conversations because, when they do, they frequently face rejection, leading to emotional trauma and social isolation [42]. Furthermore, these social steps of assessing people’s non-verbal response, starting, maintaining, and leaving conversations are relevant to teach because they are versatile and can be applied in various contexts, such as school, college, the workplace, and family gatherings [40].
Cognitive behavior techniques
Results are consistent with systematic review findings that one of the most used techniques of behavior skills is modelling, including video modelling [41]. Recent studies also show promising results on how video modelling can be instantiated in spherical video-based virtual reality (SVVR) for autistic users [43]. SVVR, also known as 360-degree videos or immersive videos, involves positioning users at the center of a spherical environment [44]. In this setup, users engage with the content and context by moving their head, as detected by a connected Head-Mounted Display (HMD). Recent studies indicate that there is some supporting evidence indicating that skills learned in SVVR through video modelling can be generalized to real-world situations [44–47].
Research has also shown preliminary data on how relaxation techniques can be incorporated into VR and the results are promising. For example, in a study by Meindl et al. [46], they explored the use of relaxation techniques within exposure therapy in SVVR to an adult with autism gradually confront his needle phobia. They developed a special SVVR program where the participant watched a 360-degree video in a VR headset, making it feel like he was in a doctor’s office. The researchers found that incorporating relaxation techniques in this SVVR method was better than regular exposure therapy because it could replicate real-life situations like doctors’ offices using affordable VR technology. Results indicated that the participant became more comfortable with medical procedures and different healthcare providers. Similarly, Maskey et al. [48] worked with eight autistic adults to assess a VR-based intervention for anxiety management. The participants underwent a session with a therapist to learn anxiety management techniques, followed by four 20-minute sessions in a virtual reality (VR) environment called the Blue Room, tailored to their specific anxiety triggers. The study found five out of eight participants showed lasting improvements in their real-life anxiety-related situations six months after the VR sessions. Both studies demonstrate the potential effectiveness of incorporating relaxation techniques in VR-based treatment for anxiety in autistic adults, which was previously unexplored. Nevertheless, it is crucial to highlight that both investigations featured a notably limited number of participants and did not conduct a comparative analysis of results between the cohort receiving the intervention and the control group.
Considerations for future research in VR social skills training
Future research, educational programs, and therapeutic measures should prioritize the integration of anxiety management as a pivotal component within the framework of social skills training. The omission of emotional regulation from social skills training could impede the program’s efficacy, given that it constitutes a substantial impediment for individuals on the autism spectrum in their socialization endeavors [38].
Another important aspect is to explore the social steps for initiating, joining, and maintaining a conversation which are relevant and in accordance with the youth. This is called the ecologically valid social skills steps—which means the steps taught for social skills training should be pragmatic, practical, and appropriate such as those used by socially successful youths [40].
Considering cultural context could also be beneficial since social skills are influenced by cultural variations. For instance, a study modified the PEERS® model for Dutch culture and observed different reactions to gossip in Dutch and American adolescents. Americans appeared surprised, questioning the significance of gossip, while the Dutch remained indifferent, displaying no reaction [49]. These cultural distinctions emphasize the necessity for developing culturally sensitive VR-based social skills training for diverse populations. This underscores the importance of future research focusing on creating culture and norm-specific VR social skills training programs.
Future research should also put more effort in prioritizing the authenticity of the real-world image that it tries to simulate, especially in relation to non-verbal gestures such as facial expressions. This comes at the light of Parsons’ [30] argument of the extent to which the 3-D images and the immersive characteristics of VR could really represent the real-world situation, considering that social skills require very specific verbal and non-verbal gestures (e.g.: smile, eye contact, smirking). The recent emergence of SVVR interventions for ASD, which relies on 360-degree videos to generate the virtual environment with relatively less complex and cost-effective development [43] seems to provide a promising solution to increase authenticity of the virtual environment and characters.
Another direction of future research is the feasibility and effectiveness of the training when cognitive and behavioral techniques are instantiated in VR environment. These techniques have been well documented as effective in physical social skills training [23, 41, 50], but whether it is feasible and effective when instantiated in VR remains a central question, though previous studies indicate promising results [8, 46, 48].
Study limitations
There are several limitations to this research. The Phase 1 scientific literature review was constrained to examining social skills challenges and cognitive behavioral techniques that were already employed in VR, potentially neglecting the potential of the specific challenges and techniques that has yet to be implemented within the VR context. Other limitations include the methodology of fuzzy delphi method itself. When expert opinions are represented by fuzzy numbers, there is a lack of reasoning behind the scores, and it could lead to inaccuracy in decision-making. Loss of information could also occur when only distance measures are used to assess the weight and value of opinions between experts [51].
Conclusion
This study has established a validated framework on the design of social skills training using virtual reality for HFASD. This framework plays a significant role in the development of the training program as it provides standardization by reducing ambiguity, diversity, and discrepancy of opinions, and enhances the quality of selected elements to be included in the training module. It also provides guidance in developing a module that meets the specific needs of the targeted users.
References
- 1. Bal VH, Farmer C, Thurm A. Describing Function in ASD: Using the DSM-5 and Other Methods to Improve Precision. J Autism Dev Disord. 2017; 47 (9): 2938–2941. pmid:28681254
- 2. Maenner MJ, Shaw KA, Bakian AV, Bilder DA, Durkin MS, Esler A, et al. Prevalence and Characteristics of Autism Spectrum Disorder Among Children Aged 8 Years—Autism and Developmental Disabilities Monitoring Network, 11 Sites, United States, 2018. MMWR Surveill Summ. 2021 Dec 3;70(11):1–16. pmid:34855725; PMCID: PMC8639024.
- 3. Frye RE. Social Skills Deficits in Autism Spectrum Disorder: Potential Biological Origins and Progress in Developing Therapeutic Agents. CNS Drugs. 2018 Aug;32(8):713–734. pmid:30105528; PMCID: PMC6105175.
- 4. Mosher MA, Carreon AC. Teaching social skills to students with autism spectrum disorder through augmented, virtual and mixed reality. Research in Learning Technology 2021; 29. https://doi.org/10.25304/rlt.v29.2626.
- 5. Zhang L, Warren Z, Swanson A, Weitlauf A, Sarkar N. Understanding Performance and Verbal-Communication of Children with ASD in a Collaborative Virtual Environment. J Autism Dev Disord. 2018 Aug;48(8):2779–2789. pmid:29560561; PMCID: PMC6041154.
- 6. Kumazaki H, Yoshikawa Y, Yoshimura Y, Ikeda T, Hasegawa C, Saito DN, et al. The impact of robotic intervention on joint attention in children with autism spectrum disorders. Mol Autism. 2018 Sep 4;9:46. pmid:30202508; PMCID: PMC6122716.
- 7. Zhao H, Swanson AR, Weitlauf AS, Warren ZE, Sarkar N. Hand-in-Hand: A Communication-Enhancement Collaborative Virtual Reality System for Promoting Social Interaction in Children with Autism Spectrum Disorders. IEEE Trans Hum Mach Syst. 2018 Apr;48(2):136–148. Epub 2018 Jan 23. pmid:30345182; PMCID: PMC6193496.
- 8. Yuan SNV, Ip HHS. Using virtual reality to train emotional and social skills in children with autism spectrum disorder. London J Prim Care (Abingdon). 2018 Jun 7;10(4):110–112. pmid:30083244; PMCID: PMC6074644.
- 9. Didehbani N, Allen T, Kandalaft M, Krawczyk D, Chapman S. Virtual Reality Social Cognition Training for children with high functioning autism. Comput. Hum. Behav. 2016; 62: 703–711. https://doi.org/10.1016/j.chb.2016.04.033
- 10. Mesa-Gresa P, Gil-Gómez H, Lozano-Quilis JA, Gil-Gómez JA. Effectiveness of Virtual Reality for Children and Adolescents with Autism Spectrum Disorder: An Evidence-Based Systematic Review. Sensors (Basel). 2018 Aug 1;18(8):2486. pmid:30071588; PMCID: PMC6111797.
- 11. Thai E, Nathan-Roberts D. Social Skill Focuses of Virtual Reality Systems for Individuals Diagnosed with Autism Spectrum Disorder: A Systematic Review. Proc. Hum. Factors Ergon. Soc. Annu. Meet. 2018; 62(1): 1469–1473. https://doi.org/10.1177/1541931218621333.
- 12. Glaser N, Schmidt M. Systematic Literature Review of Virtual Reality Intervention Design Patterns for Individuals with Autism Spectrum Disorders. Int. J. Hum. Comput. Interact. 2022; 38(8): 753–788.
- 13. Ip HHS, Wong SWL, Chan DFY, Byrne J, Li C, Yuan VSN, et al. Enhance Emotional and Social Adaptation Skills for Children with Autism Spectrum Disorder: A Virtual Reality Enabled Approach. Comp Educ. 2018; 117: 1–15.
- 14. Bekele E, Wade J, Bian D, Fan J, Swanson, A, Warren Z, et al. Multimodal Adaptive Social Interaction in Virtual Environment (MASI-VR) for Children with Autism Spectrum Disorders (ASD). In Virtual Reality (VR). IEEE: Piscataway, NJ, USA. 2016; 121–130.
- 15. Lorenzo G, Lledó A, Pomares J, Roig R. Design and Application of an Immersive Virtual Reality System to Enhance Emotional Skills for Children with Autism Spectrum Disorders. Comp Educ. 2016; 98: 192–205.
- 16.
Ip HH, Wong SW, Chan DF, Byrne J, Li C, Yuan VS, et al. Virtual Reality Enabled Training for Social Adaptation in Inclusive Education Settings for School-aged Children with Autism Spectrum Disorder (ASD). In Proceedings of the International Conference on Blending Learning, Beijing, China, 19–21 July 2016; Springer: Cham, Switzerland, 2016.
- 17.
Taryadi BR, Kurniawan I. The Improvement of Autism Spectrum Disorders on Children Communication Ability with PECS Method Multimedia Augmented Reality-Based. In Journal of Physics: Conference Series, 1st ed. Rahayu YS, Ekawati R, Nandiyanto ABD, Lukito A, Wibawa SC, Eds; IOP Publishing: Bristol, UK, 2018; 947: 1–7, ISSN 1742-6588.
- 18. Cheng Y, Huang CL, Yang CS. Using a 3D Immersive Virtual Environment System to Enhance Social Understanding and Social Skills for Children with Autism Spectrum Disorders. Focus Autism Other Dev. Disabl. 2015; 30: 222–236.
- 19. Parsons S. Learning to Work Together: Designing a Multi-user Virtual Reality Game for Social Collaboration and Perspective-Taking for Children with Autism. Int. J. Child-Comp Interact. 2015; 6: 28–38.
- 20. Bekele E, Crittendon J, Zheng Z, Swanson A, Weitlauf A, Warren Z, et al. Assessing the Utility of a Virtual Environment for Enhancing Facial Affect Recognition in Adolescents with Autism. J. Autism Dev. Disord. 2014; 44: 1641–1650. pmid:24419871
- 21. Stichter JP, Laffey J, Galyen K, Herzog M. iSocial: Delivering the Social Competence Intervention for Adolescents (SCI-A) in a 3D Virtual Learning Environment for Youth with High Functioning Autism. J. Autism Dev. Disord. 2014; 44: 417–430. pmid:23812663
- 22. Cai Y, Chia NK, Thalmann D, Kee NK, Zheng J, Thalmann NM. Design and Development of a Virtual Dolphinarium for Children with Autism. IEEE Trans. Neural Syst. Rehab. Eng. 2013; 21: 208–217. pmid:23362251
- 23. Laugeson EA, Park MN. Using a CBT Approach to Teach Social Skills to Adolescents with Autism Spectrum Disorder and Other Social Challenges: The PEERS® Method. J Rat-Emo Cognitive-Behav Ther. 2014. 32: 84–97. https://doi.org/10.1007/s10942-014-0181-8
- 24. Parsons S, Mitchell P. The potential of virtual reality in social skills training for people with autistic spectrum disorders. J Intellect Disabil Res. 2002 Jun;46(Pt 5):430–43. pmid:12031025.
- 25. Murray TJ, Pipino LL, van Gigch JP. A Pilot Study of Fuzzy Set Modification of Delphi. Hum Sys Mngmt. 1985; 5(1): 76–80.
- 26. Tahriri F, Mousavi M, Hozhabri Haghighi S, Siti Zawiah MD. The application of fuzzy Delphi and fuzzy inference system in supplier ranking and selection. J Ind Eng Int. 2014. 10(66). https://doi.org/10.1007/s40092-014-0066-6
- 27. Mahmud MI, Sidek MN, Wan Marzuki WJ, Salleh A, Abu Yazid AB. Need analysis of development on career readiness content using fuzzy delphi method. Cent. Eur. J. Manag. 2022. 30(3).
- 28. Ismail N, Mohamed S, Hamzah M. The Application of the Fuzzy Delphi Technique to the Required Aspect of Parental Involvement in the Effort to Inculcate Positive Attitude among Preschool Children. Creative Education. 2019. 10: 2907–2921.
- 29. Mohamad SNA, Embi M, Nordin N. Determining e-portfolio elements in learning process using fuzzy delphi analysis. Int. Educ. Stud. 2015; 8(9).
- 30. Parsons S. Authenticity in Virtual Reality for assessment and intervention in autism: A conceptual review. Edu. Res. Rev. 2016; 19: 138–157. https://doi.org/10.1016/j.edurev.2016.08.001.
- 31. Polit DF, Beck CT, Owen SV. Is the CVI an acceptable indicator of content validity? Appraisal and recommendations. Res Nurs Health. 2007 Aug;30(4):459–67. pmid:17654487.
- 32. Lynn MR. Determination and quantification of content validity. Nurs Res. 1986; 35: 382–385. pmid:3640358
- 33. Skulmoski GJ, Hartman FT, Krahn J. The Delphi method for graduate research. J. Inf. Technol. 2007; 6(1):1–21.
- 34. Jamil MRM, Siraj S, Hussin Z, Noh NM, Sapar AA. Introduction of Fuzzy Delphi Method in Design and Developmental Research. 2014. Bangi: Minda Intelek.
- 35. Noh NM, Hajar S, Abd Razak SH, Alias N, Siraj S, & Jamil MRM, et al. Usage of Facebook: The Future Impact of Curriculum Implementation on Students in Malaysia. Procedia—Soc & Beh Science. 2013; 103: 1261–1270.
- 36. Noh NM, Siraj S, Jamil MRM, Hussin Z, Sapar AA. Design of Guidelines on Learning Psychology in the Use of Facebook as a Medium for Teaching & Learning in Secondary School. Turkish Online Jour Edu Tech. 2015; 14: 39–44.
- 37. Noh NM, Siraj S, Hajar SH, Jamil MRM, Hussin Z. Application of Fuzzy Delphi Method as a Vital Element in Technology as a Tool in Design Thinking Based Learning. Asia Pac Jour of Educators & Edu. 2019; 34: 129–151. https://doi.org/10.21315/apjee2019.34.7
- 38. Maskey M, Lowry J, Rodgers J, McConachie H, Parr JR. Reducing specific phobia/fear in young people with autism spectrum disorders (ASDs) through a virtual reality environment intervention. PLoS One. 2014 Jul 2;9(7):e100374. pmid:24987957; PMCID: PMC4079659.
- 39. Berenguer C, Miranda A, Colomer C, Baixauli I, Roselló B. 2018. Contribution of Theory of Mind, Executive Functioning, and Pragmatics to Socialization Behaviours of Children with High-Functioning Autism. J. Autism Dev. Disord. 2018; 48(2): 430–441.
- 40.
Laugeson EA. The science of making friends: Helping socially challenged teens and young adults. San Francisco, CA: Jossey-Bass; 2013.
- 41. Hall LJ, Leinert S, Jacquez J. A Review of Social Skills Manuals for Adolescents with Autism Spectrum Disorder. Curr Dev Disord Rep. 2018. 5: 77–88. https://doi.org/10.1007/s40474-018-0134-5.
- 42. Sasson N, Faso D, Nugent J. et al. Neurotypical Peers are Less Willing to Interact with Those with Autism based on Thin Slice Judgments. Sci Rep. 2017; 7: 40700. pmid:28145411
- 43. Glaser N, Newbutt N, Palmer H. Schmidt M. Video-Based Virtual Reality Technology for Autistic Users: An Emerging Technology Report. Tech Know Learn 28. 2023; 925–935. https://doi.org/10.1007/s10758-022-09594-x.
- 44. Schmidt M, Schmidt C, Glaser NBeck D, Lim M, Palmer H. Evaluation of a Spherical Video-based Virtual Reality Intervention Designed to Teach Adaptive Skills for Adults with Autism: A Preliminary Report. Int. Learn. Env. 2019. https://doi.org/10.1080/10494 820.2019.15792 36
- 45.
Gelsomini M, Garzotto F, Matarazzo V, Messina N, Occhiuto D. Creating Social Stories as Wearable Hyper-immersive Virtual Reality Experiences for Children with Neurodevelopmental Disorders. In: Proceedings of the 2017 conference on interaction design and children. 2017; 431–437.
- 46. Meindl JN, Saba S, Gray M, Stuebing L, Jarvis, A. Reducing Blood Draw Phobia in an Adult with Autism Spectrum Disorder using Low-cost Virtual Reality Exposure Therapy. J. App. Res. Int. Dis. 2019; 32(6): 1446–1452.
- 47. Dixon DR, Miyake CJ, Nohelty K, Novack MN, Granpeesheh D. Evaluation of an Immersive Virtual Reality Safety Training used to Teach Pedestrian Skills to Children with Autism Spectrum Disorder. Beh. Anal. Pract. J. Art. 2019; 13: 631–640. pmid:32953391
- 48. Maskey M, Rodgers J, Ingham B, Freeston M, Evans G, Labus M, et al. Using Virtual Reality Environments to Augment Cognitive Behavioral Therapy for Fears and Phobias in Autistic Adults. Autism Adulthood. 2019 Jun 1;1(2):134–145. Epub 2019 Apr 13. pmid:31032480; PMCID: PMC6485262.
- 49. Idris S, Jagersma G, van Pelt B, Jacobs S, Laugeson E, Hillegers M, et al. Development and preliminary testing of the Dutch version of the Program for the Education and Enrichment of Relational Skills (PEERS®). Research in Autism Spectrum Disorders. 2020; 78. 101629.
- 50. Weston L, Hodgekins J, Langdon PE. Effectiveness of cognitive behavioural therapy with people who have autistic spectrum disorders: A systematic review and meta-analysis. Clin Psychol Rev. 2016 Nov;49:41–54. Epub 2016 Aug 4. pmid:27592496.
- 51. Foth T, Efstathiou N, Vanderspank-Wright B, Ufholz L-A, Du¨tthorn N, Zimansky M, et al. The use of Delphi and Nominal Group Technique in nursing education: a review. International Journal of Nursing Studies. 2016; 60:112–20. pmid:27297373.