Abstract
Caused by a deviance of their reward system, autistic people show attention deficits for learning content outside their special fields of interest. This can lead to significant problems, especially in formal learning situations. A promising approach to increase attention is the use of game-based learning concepts. The effect of individual playful aspects could be shown in existing learning systems. However, these do not provide consistent game experiences, which may result in a decreasing motivation for training. Therefore, this paper presents requirements as well as a related game concept to integrate the learning content with a playful narrative in order to promote motivation and attention for socio-emotional training.
1 Introduction
Promoting learning means taking on specific and sometimes even unexpected challenges from the perspective of diversity and inclusion. These may be so unusual, at least with regard to the inclusion of autistic individuals, that they may require specific consideration of particular forms of teaching. In this article, the specific challenges in the learning of autistic people are considered and a dedicated solution is presented to address these challenges.
The term autism refers to people with certain similarities, some of these are determined by diagnostic criteria [3]. However, autistic people also have numerous individual differences, which makes the phenomenon of autism appearing comparatively blurred. Therefore, the broader term autism spectrum disorder (ASD) was introduced to replace the previous names of different types of autism such as Kanner Autism, Asperger Syndrome and High Functional Autism.
Autistic people are mainly impaired in terms of learning because of their narrow interests and their difficulty to take in externally controlled learning content with appropriate attention [33]. These deficits in the variety of interests and in the ability to conscious attention control are elementary components of the autistic syndrome [1]. In this context, the better inclusion of autistic people and, thus, appropriate consideration of their needs in the educational system has a high potential for social benefit. This is especially true since autistic people – despite good educational qualifications – are far above average affected by unemployment [44].
This problem has a particularly unfavorable effect if autists are to be trained in the perception of emotions by looking at human facial expressions, which is a necessary basis for social interaction. The trainability of emotion perception by IT-supported systems is currently investigated in psychological research (cf. Section 2). In principle, positive effects on motivation and attention could be determined when autistic learners use IT-supported training [5].
Autists usually do not have a great self-interest in looking at faces [51]. Consequently, there is a problem with attention and motivation for such training. A system for training the recognition of emotions from looking at facial expressions can, therefore, benefit in particular from appropriate forms of promoting attention and motivation. Computer games intrinsically motivate and can, therefore, support self-motivated learning. Since computer games are often among the special interests of autistic people [39] the integration of such training sessions is promising [35].
In the following, related work and requirements for such a game-based training are presented. Building on this, a training concept addressing these special requirements as well as a technical design for the implementation of the concept will be shown. Finally, evaluation approaches and first findings are discussed. This is the extended and updated version of [57], which was awarded as best paper at the DeLFI conference in 2018.
2 Related Work
In addition to other impairments, autistic individuals have particular difficulties in recognizing other people’s emotions from their facial expressions [7]. It plays an essential role that autistic people tend to avoid looking at the eye area and instead prefer to observe the mouth region. However, since some emotions are primarily readable by important signals in the area of the eyes, emotion recognition is impaired [13]. Existing training systems for socio-emotional competencies use, among other things, video recordings with which the learner can practice the interpretation of different emotion expressions [4], [23], [38], [19], [55]. The effectiveness of these systems and the generalizability of the trained content on everyday life is a current research topic. However, many of these training systems have problems when it comes to persistent user motivation. Also, there is a focus on training systems for children with an autism spectrum disorder in the current research [9]. Particularly, mobile technology can help autistic persons in their everyday life and advance outside a clinical scope [37].
A common approach to promote engagement, motivation, and fun in learning contexts is to employ gamification or game-based approaches. There are examples that kept players interested and in a game for hours, weeks, and even years [28]. The feasibility of computer games was investigated in an interview study with 58 adults with autism spectrum which showed that there is a strong interest in computer games [39]: Playing is motivated by stress relief, immersion, time use, and social connection. On the one hand, the participants reported that they enjoy game aspects such as achievements, creativity as well as the story. On the other hand, most dislike game violence, sexual content, and game design problems. Also, negative game aspects such as addiction and negative social interactions were mentioned by the participants.
Gamification is defined as “the use of game design elements in non-game contexts” [15, p. 9]. Typically, elements such as rewards, high scores, or badges are used to enhance the learning content (see [27] for an overview). An overview of gamified interventions can be found in [9]. However, this approach was described as “chocolate-covered broccoli” [22] because the game design elements are just added to the learning content and not intertwined with it. An analysis in [46] shows that such reward systems initially encourage learners to play longer sessions, but this effect usually disappears after a few sessions.
That’s why there are more game-centered approaches such as serious games, educational games, and game-based learning: Serious games are designed with a primary goal different from pure entertainment [14], [16]. Therefore, they have a “game” dimension and a “serious” dimension. In general, serious games try to “contextualize the player’s experience in challenging, realistic environments, supporting situated cognition” [16, p. 1]. However, serious games often focus on the serious part whereas game-based learning approaches often focus on the learning dimension [14], [28]. A good indicator of this is the challenge of “finding an appropriate balance between gaming and learning activities” [34, p. 96]. There are a lot of (often task-oriented) approaches for serious games, e. g. for complex learning contexts such as in physics [28], business, military, or medicine [16] and also for individuals with autism [52]. However, the immersive potential of educational games is fragile [34]. It is believed that better promotion of motivation and attention is hampered by the inadequate implementation of game-based approaches [56]. Therefore, a key point for game-based approaches is that the learning tasks need to be “appropriately designed and tightly coupled with the narrative” [50, p. 98]. It is not sufficient to have a plot where exercises are embedded and appear from time to time quite isolated from the narrative. In particular, the existing approaches already lack an intrinsically motivating narrative. Closely linked to this, assessment is another key point for educational games. It is very important that assessing the current performance of the user does “not destroy or impair motivation, immersion, flow experience or the game’s storyline” [34, p. 97]. In particular, “flow seems to have a positive influence on performance enhancement, learning and engagement” also “gathered evidence suggests that the flow has a positive influence on game-based learning outcome” [43].
Being a game is not just about having a good narrative but also requires a certain level of freedom so that the player can be an active part of the game [18]. Without the ability for decisions and choices the player might feel powerless and the activity might not be a game at all [18]. In general, there is a “highly interactive, deliberately generating tension between the degree of control the story imposes and the player’s freedom of interaction” [48]. Particularly, enhancing the player’s perception of autonomy has been shown to increase intrinsic motivation [47]. According to [46], the opportunities for making choices are often limited to instructionally irrelevant aspects.
Particularly notable in this context is the research and serious game development of [25]. They extended and modified an existing game (The Elder Scrolls V: Skyrim) for social skills training. They adapted a role-playing game and invented a narrative where the player is held captive on a ship and has to escape. The player needs to talk to all crew members and gain trust to subtly receive items which aid in his escape. The basis is a detailed dialogue system where relationship values are calculated. This game, however, is not integrated into everyday life and, thus, may suffer from decreased interest.
To sum up, there is a research gap for training systems that consistently implement game-based training concepts as well as deeply integrate the training into a narrative (as suggested by [20], [24], [34]) and into the everyday life (e. g., by using mobile technology). However, motivational interventions might still be required in case learners do not act for a longer period of time [34].
3 Requirements for a Game-Based Emotion Training
An essential feature of computer games and, thus, of game-based learning is the integration of the learning content into a narrative [32], [34]. Therefore, a game concept for socio-emotional training must meaningfully connect the training contents with the narrative of the computer game [50], [17]. This requirement is of the utmost importance. Without any close intertwining of the learning content in the gameplay and the story, the learning content cannot be considered part of the game, and thus a substantial part of the positive effects of the game-based approach cannot be fully effective.
Typically, clinical trials have a duration of 5 to 12 weeks to study the effects of an intervention [4], [30], [55]. Accordingly, a game that encourages training must remain interesting for players over a longer period of time and offer new challenges as well. In addition, it is important for effective training that regularity and minimum duration of individual training sessions are maintained [19]. To meet these requirements, a game mechanic may be important to regulate game time and duration. Furthermore, a regulatory mechanism should also counteract excessive game use.
Likewise, the game concept should reach as many potential users as possible:
Although the prevalence of autism shows an asymmetric diagnosis ratio of about four men to one woman, female autistics are no exception [10]. The concept should appeal equally to male and female players alike.
Already in the preschool age, the training of socio-emotional competencies for autistic people is possible and meaningful [26]. But even healthy people of higher age show deficits in the socio-emotional functions [45], albeit less so than in autistic individuals. The usefulness of the socio-emotional training thus affects a very broad age range. Consequently, a game concept should be sought, which is consistently appealing and suitable for persons of this broad age spectrum.
Also, the game concept should appeal to both experienced players and inexperienced players and it should provide an easy entry point [21].
4 Training Concept
The training and game concept described here bears the working title Lodur. The main part of Lodur is based on the popular game Minecraft with certain additions and modifications to the familiar items of the game. A mobile app integrating the socio-emotional exercises completes the concept with elements that enable time- and location-independent play. With regard to the system of game-based learning according to [29], Lodur embeds the learning tasks directly into the game and intertwines them with the narrative. The following sections explain the individual game concepts and their meaning for the actual training.
4.1 Narrative Embedding of the Learning Contents
The meaningful combination of the training sessions with the narrative context of the game is one of the most important, but also the most difficult challenges in developing a suitable game concept [34]. This does not only apply to the training of socio-emotional competencies. The narrative context primarily serves motivation, but the learning content must fit in harmoniously with the plot. Also, the game flow must not be disturbed by the training elements, but should even be promoted if possible. [20], [24], [34] therefore propose to place learning content as essential units of the game. This allows players to grasp the training of skills as part of the game. Furthermore, the training elements should be offered frequently and regularly.
The player talks to a non-playable character in the game and at the same time uses the mobile app to communicate with the alien game character called “Kralmik”.
Many autistic people feel like aliens on a foreign planet whose rules and inhabitants they do not understand [8]. In order to take up this world view, the development of the narrative context makes use of this thought: The player gets stranded on an alien planet where humanoid beings exist but whose language and facial expressions he cannot interpret. This provides the starting point for the narrative involvement of the facial expressions exercises: The illusion is created that the player possesses a translation tool that can translate the language and facial expressions into the familiar modes of expression – this tool is operated by the player in reality through a mobile app as depicted in Figure 1. The game can offer the facial expressions exercises so that the recognition of human facial expressions facilitates progress in the game. This is mapped by calculating a relationship value between player and non-player characters, that improves upon successful recognition of facial expressions. A higher relationship value affects the quality of game rewards positively [25]. As a result, the player can implicitly achieve better rewards. For example, the player could be contacted by a trader via the mobile app. The trader wants to sell the player a resource that would advance him in the current game section. However, the trader is annoyed because half of his goods are spoiled by the carelessness of his assistant. If the player recognizes this from the text and facial expressions of the trader and solves the included training task, he receives the resources at a discounted price or other helpful tools which are not available otherwise. Due to the nature of mobile apps being available everywhere, this can happen even if the player is not directly playing Lodur on the PC, but is currently traveling.
The implicit connection between the results in the facial expressions recognition and the game rewards supports the deep connection between the learning units and the game. With an explicit reward, the player may be more likely to recognize what he is rewarded for, and may thus identify the learning units as such. It is very important that there is no overly negative feedback for a wrong answer – instead, the game remains playable, but the progress might be slower. The decisive factor here is the voluntary nature of learning. First and foremost, the game is played for the sake of playing. The teaching content is learned voluntarily because it enables or facilitates the progress in the already exciting game. Like a child who voluntarily learns a foreign language because the game it wants to play is only available in that specific language. Not because of language learning, but because it recognizes this ability as important for playing. Although here, as with almost all forms of teaching, it can not be ensured that the learning content is actually learned, nevertheless, this voluntariness can be regarded as a strong driving force. However, implicit rewards also have another effect that may be appropriate in the context of learning support from autistic individuals: Autists have a disorder of the inner reward system, which does not make material rewards work as usual [33]. On the one hand, autistic people benefit from intrinsic motivation, while obvious external rewards can even be annoying. Implicit rewards, on the other hand, can make it logically reversible which actions favorably influence the course of the game and, thus, indirectly support intrinsic motivation. However, they must be used carefully so that they do not appear to be extrinsically motivating.
In Minecraft, one can develop new tools by combining resources. For example, the combination of iron and sticks yields an iron pickaxe. In Lodur, this option is intentionally limited. Instead, new tools can be obtained from non-player characters after a dialogue (i. e., a training task). However, there is still a variety of activities available, such as mining, farming, raising animals, build buildings, etc. So one can play freely and decide for oneself if one wants to unlock tools because the game is still playable without tools like in common freemium games (here, however, without paying money, but by solving training tasks). The motivation is, however, even if stones can be broken down with the bare hands, that it goes much faster with a suitable tool, such as a pickaxe. Tools are just one example – rewards can also be resources and access to new buildings or areas in the game world. Lodur’s detailed storyline, derived from these basic considerations, is described in more detail in the following section.
4.2 Promoting a Long-Lasting Motivation: Lodur’s Storyline
Long-lasting game motivation is particularly promoted by adventure games and simulations [2]. Adventure games can stay interesting for a longer time based on a very extensive and varied story or narrative. In simulation games, many development opportunities are usually available in order to allow the players to continuously experience new facets of the game and to develop new strategies.
Lodur combines both game genres and consists of three phases: An introductory phase, a mid-game phase, and an end-game phase (see Figure 2). At the beginning (the first two phases) Lodur has the character of an adventure game. The game is driven by a strong narrative and the player has to solve predetermined tasks or puzzles. Later on, the game smoothly changes its character into an economic simulation with very extensive and versatile development opportunities.
Phases of the game Lodur.
The introductory phase sets the narrative context and also serves as a tutorial that teaches the basics of control and play (compare Section 4.5). As already mentioned in Section 4.1, autistics often feel as if they are on a foreign planet whose inhabitants do not understand them. Lodur‘s plot picks up on this feeling by putting the player in the role of a captain of a spaceship. At the end of the introductory phase, the spacecraft crashes on an alien planet and the player is forced to contact the locals in order to survive and live on the planet. Since the locals and their kind are alien to him, the player must first learn to read the emotions of the persons he meets from their “alien” faces. A translation tool, the mobile app, helps the player to translate the alien Minecraft faces into human faces.
At this point, the game is gradually changing over to the mid-game phase. Here, still supported by a strong narrative, the elements of an economic simulation are introduced by a non-player character named “Kralmik” (e. g., he gives the hint “You need to farm crops.”, see Figure 3). Through training tasks embedded in the narrative, new game elements can be unlocked.
A player uses his learned skills to farm.
By successfully completing a training task, the player has unlocked a merchant who is now selling raw materials.
Finally, the game goes into the end-game phase which finally changes the character of the game to an economic simulation. The narrative is not hard-scripted anymore and is now driven by predefined events that are triggered randomly or by certain player activities. More and more inhabitants of the planet come into contact with the player, for example, traders (see Figure 4). Through training exercises embedded in spontaneous short narrative events, the player can further accelerate his progress. In the economic simulation, the player is free in his choices on what activities to perform. Different tools enable various activities, each of which simulates a specific profession. The game, comprising different habitats, allows for each profession again to carry out very diverse activities. As a result, the combination of the two genres allows players to get smoothly into the narrative and then constantly brings new challenges, new game concepts to discover as well as more and more degrees of freedom to the player.
4.3 Regulation of Playing Time and Duration
The regulation of the playing time and duration has several objectives. Incentives for regular use should be created, in this case once a day. Furthermore, the player should be motivated to a minimum usage of one hour per session. In addition, these incentives should decrease if they are used excessively to prevent abuse, up to possible gambling addiction. Explicitly, handling excessive gameplay is important as addiction was explicitly mentioned in an interview study with adults with autism as a possible negative game aspect [39]. Thus, these features are not intended to patronize the player, as the right amount of choice enhances the gameplay experience [48]. In addition, the promotion of self-control has a meaningful secondary therapeutic goal. Self-control can be trained through self-responsible decisions and represents an important part of media literacy regarding the use of computer games by children and adolescents.
The main mechanism for the regulation of the playing time of Lodur is a special play element, so-called Boosters. They are distributed just once a day through the game. They reinforce the player’s income in terms of game currency for one hour. This way, the player can earn more resources once a day and for one hour. If a player does not play for a full day, the booster will expire. This creates an incentive to play daily. After playing an hour with the booster, the game becomes less motivating in terms of reduced award creation. However, the game does not prohibit playing without a booster. So it remains a decision of the player to continue beyond the recommended playing time.
4.4 Re-Activation of Passive Players
Some players interrupt or even finish gaming sessions, although the game would still be interesting for them [34]. This can happen through changed living conditions, such as vacation, short-term lack of leisure or even by short-term intensive use of another game. Under certain circumstances, these players can be reclaimed for a game (and thus for training) by remembering mechanisms. Lodur issues special game objects to all players at intervals of approx. two weeks and informs them via notifications using the mobile app. Thus, players can be reached directly in their everyday life and not only at their desktop PCs, as recommended by [37]. Even for active players, it is interesting to see what items will become available with the next “payout wave”. Examples of such game objects are particularly effective tools and special items that can be used to get into otherwise hidden parts of the game world.
4.5 Challenges for Different Levels of Gaming Experience
As already stated, Lodur is based on the engine of the game Minecraft. Since this environment is relatively popular, there will be many players who have already gained experience in other contexts.
However, not all players are already experienced in Minecraft. Therefore, Lodur comes with several support mechanisms for introducing the Minecraft-based game mechanics to newcomers. The first support mechanism is a relatively elaborate introductory phase, which teaches newcomers the important concepts of the game and how to control the game. This is embedded in the narrative as a short “fitness test” which must be completed. In fact, this fitness test is an obstacle course where the player learns how to move inside Minecraft. Even after the introduction, complex processes, such as the construction of a manifold building, are supported by automatisms. Some of the automatisms are only available as rewards after finishing a learning task.
Experienced players do not have to take advantage of all these support mechanisms and, therefore, have room to prove their skills: They can make use of the extensive and versatile development possibilities of the economic simulation and develop sophisticated strategies. Of course, they can also adjust existing or build manifold buildings on their own, explore the world, and freely use all available activities (compare Sections 4.1 and 4.2).
4.6 Support for a Wide Range of Ages and Diversity of Players
Supporting a wide range of ages poses a greater problem of considering the limited literacy of younger children. [54] showed that especially younger players with less practice in reading longer text passages are disturbed in their presence by written text within the game world. For that reason, all the essential dialogues within the story are provided in both written and spoken language. Even the mobile app provides spoken dialogues. Preliminary tests on a Lodur prototype confirmed the effectiveness of this approach [57]. This also provides support for diversity, e. g. for players with cognitive impairments or illiteracy.
5 Technical Design
This section describes and explains the technological choices, media design, system architecture, and the mobile app in more detail.
5.1 Representation of the 3D World
The 3D world is represented by the game world of Minecraft. The selection of the technology for the representation of the 3D world was partly based on criteria that promote efficient project handling: For example, Minecraft offers a very simple way to design scenes. Almost everyone involved in the project was very familiar with Minecraft’s world editor in advance. This allowed the extensive scenes for the narrative background to be created very efficiently. The importance of this aspect is not to be underestimated for the development of a complex game.
Furthermore, Minecraft can be extensively modified via a Java-based API called Bukkit. This API is comparatively well supported by an active developer community. This is also very helpful in terms of rapid development progress. By using this API a server plugin can be created. Such a server plugin can be used to interact with the Minecraft server (e. g., by reacting to game events) and to adapt the game mechanics. For the game scenario, the Minecraft game world has been extended to include voice output and modified textures, allowing for a highly customized gaming experience. Not least, the decision for Minecraft is also justified by a strategic factor: Minecraft is one of the most popular and widely used computer games of our time.[1] This has a beneficial effect on the acceptance of the game-based therapy concept. The disadvantage, however, is the license costs for Minecraft.
5.2 Media Design
The media design of Lodur takes up the special needs of autistic people, who can, for example, be easily disturbed by unnecessary design elements [36]. Therefore, the general principle of reducing the user interface is consistently implemented by Lodur (see Figure 5). However, a minimum of detail is needed to convey the narrative comprehensively.
Reduced design of the game.
Moreover, the minimalist appearance of Minecraft characters and their facial expressions do precisely support the narrative of an alien world with unknown socio-emotional interactions.
Beyond the reduced design, we also strictly followed the design guidelines for socio-emotional training systems as identified in previous studies [56]:
The complexity of tasks shall be primary reflected in content and as less as possible in design.
The difficulty of tasks shall consider individual progress instead of a rigid level system.
The training tasks shall be tightly interwoven with the game in order to avoid tiring.
The target group (here: people with ASD) shall not be further reduced (e. g. to certains age groups).
5.3 System Architecture
Technically, Lodur consists of three components and is played on two different devices. The central entry point to the game is in a simplified 3D world, which is offered via the Minecraft client on a desktop PC. An additional part of the game is accessed via a mobile app. Both are connected in the background by a central server infrastructure. The architecture (see Figure 6) allows synchronization of gameplay across both devices.
Architecture of the game Lodur.
As a Minecraft server, the widely used and community developed open-source SpigotMC[2] server is used which provides the Bukkit API for creating server plugins. Lodur is developed as such a server plugin in order to manipulate and control the Minecraft game in-depth – for example the available options are limited, so that not all tools can be “crafted” as usually possible in Minecraft (compare Section 4.1). The Lodur Spigot plugin connects to a Java server with a REST-interface in the logic layer which is used for maintaining the current game state. Also, the plugin controls the gameplay and particularly drives the narrative (i. e., creates a new world based on a template for every player, drives non-playable characters, plays audio, unlocks doors, plants objects used for quests/tasks etc.). The phases of the narrative are modeled as sequences of different quests/tasks with pre-conditions which control the progress of the phases [24]. The modeled tasks (including dependencies to other tasks) as well as the game state, in particular, the number of achievements, solved tasks and trainings accomplished, are persisted in an SQL database.
As Lodur is played in the 3D Minecraft world using the Minecraft client on a desktop computer as well as on a mobile phone, the player switches between both clients. The app is used to display more complex media than the Minecraft world allows (for example, the video footage of the emotions played by actors). The pairing of the mobile phone (see Section 5.4) with the Minecraft game is also handled by the plugin using a random code which is displayed in Minecraft and needs to be entered in the mobile app. Both, the Spigot plugin and the mobile app, regularly interact with the Java server in the logic layer in order to synchronize the current game state (i. e., achievements and quests/tasks).
In terms of software, one important point to keep in mind is that the Spigot API is not thread-safe so that only one central thread is allowed to intervene in the game. However, network communication is necessary for the synchronization of game states. This should not take place in the central thread so as not to affect the Minecraft server in its other tasks due to latencies. For this reason, the network communication takes place in an independent thread. Synchronization information is first placed in a thread-safe queue. The Spigot API provides a mechanism to periodically execute program sections on the central thread. In such a section, the synchronization data is transferred to the game world. A similar approach is also necessary in the case of the mobile app for the Android API.
This embedded training tasks consists of a dialogue with the alien “Kralmik”, which translates his mimics into video and offers possible answers.
5.4 Mobile App
The mobile app provides the main/substantive learning content and also lists the current as well as the solved quests. The user interface is tailored to the narrative context but still is based on a minimal and reduced design adapted to the target group [56]. The start of the exercises in the mobile app is initiated and controlled by the progress of the desktop game. This supports the deep integration of the learning content into the game. The exercises can also be carried out at a later point in time. Therefore, the app offers the benefits of mobile technology, such as time and location independence, and also follows the approach of micro-learning.
Figure 7 shows the integration of specific learning content in the mobile app. An actor who expresses an emotion with his face is shown as a video stimulus. The player can make a selection on the right side of the screen, based on the emotion he believes he has recognized in the actor’s face. The speech output has a special benefit at this point: The player can completely focus on the current stimulus. This promotes the therapeutic goal of focusing visual attention on the facial expressions. The correct recognition of the emotion enables the selection of best matching answers, which in turn improves the relationship to this non-player character and results in implicit rewards (compare Section 4.1).
The use of different devices within a single game does not only carry benefits. The change of the game device could disturb the immersion and, therefore, result in a hindrance of the flow experience. Hence, preserving the flow through the game device change has been part of our research. Our preliminary investigations showed an unexpectedly low interference with immersion. The reason for this may be that both devices are perceived as a toy and, thus, the device switch becomes a customary within a game activity. Also, there are commercial computer games such as Fallout 4 which employ a smartphone as a second input device for the game. Similarly to Lodur, the app has a counterpart within the game and displays statistics, a map, a log and inventory of the player.[3] The Fallout 4 app, however, can only be used within the same network when the game is running.
6 Evaluation and Discussion
Assessing the suitability of apps like Lodur comprises two major fields. First, from a computer science or media perspective, there are issues of usability, user experience, cognitive load, acceptance etc. to be examined. Standardized questionnaires like System Usability Scale (SUS), User Experience Questionnaire (UEQ), NASA Task Load Index (NASA-TLX) or Technology Acceptance Model (TAM) are available for this purpose. This established repertoire of evaluation instruments provides cursory information on aspects of human-machine-interaction, but seldomly reveals deeper insights on single design aspects leading to the resulting scores. However, such questionnaires are commonly applied to our developed apps, and for our family of ASD training apps we could demonstrate good or very scores in these fields [40], [42]. This confirms the validity of the design guidelines applied [56]. Moreover, we could prove the effectiveness of a game-based approach for socio-emotional training with regard to the intensity of flow experience [49].
The second perspective – clinical effectiveness in people with ASD – is harder to evaluate. Basically, it has been shown that computer-based systems can help autists raise more awareness and motivation [6]. Unfortunately, samples are often rather small, consistent testing procedures and control groups are missing, or results are rather superficial and hard to transfer to other scenarios [9]. While there is only little evidence for many ASD training apps [31], the effectiveness of selected game-based training systems for socio-emotional skills for autistic children has already been confirmed [30]. Likewise, the effectiveness of the video stimuli used in Lodur and in related apps has already been verified [19].
On this basis, a related adaptive system for computer-assisted training of socio-emotional competencies [40] with the same stimuli, but without a similarly sophisticated game-based approach, is currently being investigated in a clinical study. During the training, data relating to the flow experience as well as indicators for attention, reaction rates and error rates in the completion of the various training tasks are collected. There are two control groups; one with conventional, non-IT training and another one with app-based training without socio-emotional aspects.
A preliminary study [57] with a total of ten subjects (male, ages 9–16 years), eight neurologically typical and two autistic adolescents, demonstrated the comprehensibility of the introductory phase of Lodur and the narrative involvement of the exercises in game sessions of about 30–60 minutes. For this, the players were observed while playing in order to monitor if there are difficulties in completing the tasks. Afterwards, semi-structured interviews on the content of the story and on the game experience in comparison to other games were carried out. The high level of demonstrated motivation to continue playing was very promising. The first appearance of the app was perceived as very stimulating by many subjects. The app could also be used by all players without a noticeable training period. However, there were problems with the recognition of new tasks. The app was redesigned accordingly, so this issue is now fixed.
An in-depth study is currently in preparation with 25 adult volunteers providing comparative findings on flow and attention. These results shall be contrasted with the results of the non-gaming-based clinical trial mentioned above. However, Lodur does not collect data through explicit questionnaires, but rather through instrumentation directly embedded within the game and in playful tests. The measurement of flow and attention takes the form of mini-games, which are based on modified psychology attention tests [41] and in-situ flow tests [43]. This variation has been chosen to allow trouble-free gaming and not distract from the game with a questionnaire, although comparability of measurements still has to be proven. However, this approach bears huge potential not only for game-based settings, but also for virtual or augmented reality applications [53].
7 Summary and Future Work
The development of various training systems for socio-emotional competencies, despite the progress made in terms of the quality of the stimuli and the interaction concepts used, still shows some weaknesses in terms of promoting motivation and attention, especially for autistic users. The training concept presented in this article picks up on these weaknesses by using an innovative game concept. As autism is highly individual, our design decisions address key features of the autistic spectrum, including:
Reduced user interface design against sensory overload
Self-perception: the feeling of living in another world
Enthusiasm for technology, IT and computer games
Intertwining of the narrative and the learning contents
Subject of learning: Recognizing emotions from facial expressions
Adaptive game progress related to the individual learning process
A larger study will examine the effect on motivation and attention in more detail. Also, effects will be compared with closely related, but not game-based training applications for socio-emotional competences. Furthermore, the capability for short-term transferability of attention to foreign content should be able to be appropriately reviewed. Since autistic attention deficit disorder [33] is a key issue for inclusion within our educational system, this subject is highly relevant. Therefore, a study of the transferability and effectiveness of the presented game-based learning concept on institutional learning content would be welcome. Further research may shift towards such longitudinal effects.
As a final remark, there are several ethical questions associated with the use (and potential misuse) of such training technology. The consequences for individuals and for society as a whole of having emotion recognition and emotion-sensitive systems fully operational are still to be understood [42]. For instance, using a training system for social cognition as an invisible assessment tool is not intended, but possible, e. g., by employers. Moreover, the mere availability of such a training may shift societal norms towards a mainstreamed understanding of how people should behave in order to be accepted (in terms of tolerance) and legally allowed (e. g., for granting public aids). Ethical guidelines have to be further shaped for this.
Funding source: Bundesministerium für Bildung und Forschung
Award Identifier / Grant number: 16SV7241
Funding statement: Parts of the work described here were funded by the German Federal Ministry of Education and Research under grant number 16SV7241 in the joint project Emotisk.
About the authors
Dr. Sven Strickroth graduated in computer science at Clausthal University of Technology. Afterwards, as a research assistant, he investigated e-assessment systems for programming, e-learning community platforms, and lesson planning tools. Since receiving his PhD in 2016 he works as a coordinator of a university-wide e-learning project and PostDoc at the University of Potsdam.
Dipl.-Inf. Dietmar Zoerner graduated in computer science at the Technical University of Berlin. Since 1988 he worked in software development for several industrial projects. Since 2011 Dietmar took a position as a research assistant at the University of Potsdam working on e-learning projects. Since April 2015 he has been working in the research project EMOTISK, which ended in 2018.
Dipl.-Inf. Tobias Moebert graduated in computer science at the University of Potsdam. With the successful funding of the ZIM project MOTIVATE (Mobile Training Via Adaptive Technologies), he was able to take up a position as a research assistant at the University of Potsdam in October 2013. He coordinated the project from a scientific and technical point of view until December 2015. Since January 2016 he has been working in the research project EMOTISK (Emotion-sensitive systems for training social behavior), which ended in 2018. He is currently working on two ethics-focused projects.
Anna Morgiel, B. Sc. obtained her Bachelor in Computer Science at the University of Potsdam in 2018. She was a student worker in a research project on emotion-sensitive training tools. Before leaving the scientific sector, she evaluated the use of face reading apps for such training.
Prof. Dr.-Ing. habil. Ulrike Lucke graduated in Computer Science at the University of Rostock. She received her PhD in 2006 and her habilitation in 2011 with works on educational technology and academic infrastructures. Since 2010 she is a professor of computer science at the University of Potsdam. Research of her group is focused on educational technology from empirical-analytical as well as from design-oriented perspective. This combines different disciplinary views against the background of socio-technical complexity. Ulrike Lucke is PI of several funded projects and (co-)author of numerous publications in these fields.
Acknowledgment
We thank the team of Prof. Dziobek at Humboldt University in Berlin for this valuable cooperation. In addition, we deeply thank the computer science students at the University of Potsdam involved in the implementation of the game.
References
[1] Allen, G.; Courchesne, E.: Attention function and dysfunction in autism. In: Frontiers in Bioscience, 6, D105–D119, 2001.Search in Google Scholar
[2] Amory, A.; Naicker, K.; Vincent, J.; Adams, C.: The use of computer games as an educational tool: Identification of appropriate game types and game elements. In: British Journal of Educational Technology, 30(4), 311–321, 2005.Search in Google Scholar
[3] American Psychiatric Association: Diagnostic and statistical manual of mental disorders. 5th ed. Arlington, VA: American Psychiatric Association, 2013.Search in Google Scholar
[4] Bölte, S.; Feineis-Matthews, S.; Leber, S.; Dierks, T.; Hubl, D.; Poustka, F.: The Development and Evaluation of a Computer-Based Program to Test and to Teach the Recognition of Facial Affect. In: International Journal of Circumpolar Health, 61/2, 61, 2002.Search in Google Scholar
[5] Bölte, S.: Computer- und Informationstechnik. In: Autismus. Spektrum, Ursachen, Diagnostik, Intervention, Perspektiven, 1st ed. Bern: Huber, 2009.Search in Google Scholar
[6] Bölte, S.; Golan, O.; Goodwin, M. S.; Zwaigenbaum, L.: What can innovative technologies do for Autism Spectrum Disorders? In: Autism, 14(3), 155–159, 2010.Search in Google Scholar
[7] Bons, D.; van den Broek, E.; Scheepers, F.; Herpers, P.; Rommelse, N.; Buitelaaar, J. K.: Motor, emotional, and cognitive empathy in children and adolescents with autism spectrum disorder and conduct disorder. In: Journal of abnormal child psychology, 41, 425, 2013.Search in Google Scholar
[8] Brauns, A.: Buntschatten und Fledermäuse. Mein Leben in einer anderen Welt. 8. Aufl. München: Goldmann, 2004.Search in Google Scholar
[9] Camargo, M. C.; Barros, R. M.; Brancher, J. D.; Barros, V. T.; Santana, M.: Designing Gamified Interventions for Autism Spectrum Disorder: A Systematic Review. In Joint International Conference on Entertainment Computing and Serious Games, pp. 341–352, Springer, 2019.Search in Google Scholar
[10] Christensen, D. L. et al.: Prevalence and characteristics of autism Spectrum disorder among children aged 8 years, autism and developmental disabilities monitoring network, 11 sites, United States, 2012. In: MMWR Surveill Summ, 65/3, 1, 2016.Search in Google Scholar
[11] Castro, T.; Lucke, U.: The Process of Inclusive Design. In: Proc. 16th IEEE Int. Conf. on Advanced Learning Technologies (ICALT), 2016.Search in Google Scholar
[12] Csikszentmihalyi, M.: Flow: The psychology of optimal experience, New York: Harper Perennial, 1990.Search in Google Scholar
[13] Dalton, K. M.; Nacewicz, B. M.; Johnstone, T.; Schaefer, H. S.; Gernsbacher, M. A.; Goldsmith, H. H.; Alexander, A. L.; Davidson, R. J.: Gaze fixation and the neural circuitry of face processing in autism. In: Nature Neuroscience, 8, 519–526, 2005.Search in Google Scholar
[14] Djaouti, D.; Alvarez, J.; Jessel, J.-P.: Classifying serious games: the G/P/S model. In: Handbook of research on improving learning and motivation through educational games: Multidisciplinary approaches, pp. 118–136, IGI Global, 2011.Search in Google Scholar
[15] Deterding, S.; Dixon, D.; Khaled, R.; Nacke, L.: From game design elements to gamefulness: defining gamification. In: Proceedings of the 15th International Academic MindTrek Conference, pp. 9–15, ACM, 2011.Search in Google Scholar
[16] De Gloria, A.; Bellotti, F.; Berta, R.; Lavagnino, E.: Serious Games for education and training. In: International Journal of Serious Games, 1(1), 2014. DOI:10.17083/ijsg.v1i1.11.Search in Google Scholar
[17] Dickey, M. D.: “Ninja Looting” for instructional design: The design challenges of creating a game based learning environment, 2006.Search in Google Scholar
[18] Dignan, A.: Game frame: Using games as a strategy for success, Simon and Schuster, 2011.Search in Google Scholar
[19] Dziobek, I.; Rosenblau, G.; Kliemann, D.; Kappelhoff, H.; Heekeren, H.: Ein neues Softwaretraining für komplexe Emotionserkennung: Social Cognition Training Tool (SCOTT). In: 7. Wissenschaftliche Tagung Autismus-Spektrum (WTAS), Frankfurt, 2014.Search in Google Scholar
[20] Fisch, S. M.: Making educational computer games “educational”, 2005.Search in Google Scholar
[21] Gee, J. P.: What video games have to teach us about learning and literacy, Palgrave MacMillan, 2003.Search in Google Scholar
[22] Glasemann, M.; Kanstrup, A. M.; Ryberg, T.: Making chocolate-covered broccoli: designing a mobile learning game about food for young people with diabetes. In: Proceedings of the 8th ACM conference on Designing Interactive Systems, pp. 262–271, 2010.Search in Google Scholar
[23] Golan, O.; Ashwin, E.; Granader, Y.; McClintock, S.; Day, K.; Leggett, V.; Baron-Cohen, S.: Enhancing emotion recognition in children with autism spectrum conditions: an intervention using animated vehicles with real emotional faces. In: Journal of autism and developmental disorders, 40, 269, 2010.Search in Google Scholar
[24] Göbel, S. et al.: Narrative Game-based Learning Objects for Story-based Digital Educational Games. In: Narrative, 14, 16, 2009.Search in Google Scholar
[25] Gay, V.; Leijdekkers, P.; Pooley, A.: Building social awareness for teens and young adults with autism via gamification. In: Joint International Conference on Serious Games, pp. 116–127, 2016.Search in Google Scholar
[26] Herbrecht, E.; Bölte, S.: Training sozialer Fertigkeiten. In: Autismus. Spektrum, Ursachen, Diagnostik, Intervention, Perspektiven, 1st ed. Bern: Huber, 2009.Search in Google Scholar
[27] Hamari, J.; Koivisto, J.; Sarsa, H.: Does Gamification Work? – A Literature Review of Empirical Studies on Gamification. In: Proceedings of the 47th Hawaii International Conference on System Sciences, Hawaii, USA, January 6–9, 2014.Search in Google Scholar
[28] Hamari, J.; Shernoff, D. J.; Rowe, E.; Coller, B.; Asbell-Clarke, J.; Edwards, T.: Challenging games help students learn: An empirical study on engagement, flow and immersion in game-based learning. In: Computers in human behavior, 54, 170–179, 2016.Search in Google Scholar
[29] Kerres, M.; Bormann, M.; Vervenne, M.: Didaktische Konzeption von Serious Games: Zur Verknüpfung von Spiel- und Lernangeboten. In: MedienPädagogik, 2009.Search in Google Scholar
[30] Kirst, S.; Diehm, R.; Wilde-Etzold, S.; Ziegler, M.: Fostering socio-emotional competencies in children with autism spectrum condition: Results of a randomized controlled trial using the interactive training app „Zirkus Empathico“. Poster at IMFAR 2017, San Francisco.Search in Google Scholar
[31] Kim, J. W.; Nguyen, T.; Gipson, S. Y. T. et al.: Smartphone Apps for Autism Spectrum Disorder: Understanding the Evidence. In: Journal of Technology in Behavioral Science, 3, 1–4, 2018.Search in Google Scholar
[32] Klimmt, C.: Ego-Shooter, Prügelspiel, Sportsimulation? Zur Typologisierung von Computer- und Videospielen. In: Medien- und Kommunikationswissenschaft, 49/4, 480, 2001.Search in Google Scholar
[33] Kohls, G.; Antezana, L.; Mosner, M. G.; Schultz, R. T.; Yerys, B. E.: Altered reward system reactivity for personalized circumscribed interests in autism. In: Molecular autism, 9, 2018.Search in Google Scholar
[34] Kickmeier-Rust, M. D.; Albert, D.: Micro-adaptivity: Protecting immersion in didactically adaptive digital educational games. In: Journal of Computer Assisted Learning, 26(2), 95–105, 2010.Search in Google Scholar
[35] Kohls, G.; Schultz, R.: Computerized health games to promote social perceptual learning in autism. In: Autism Spectrum News, Summer, 18, 2011.Search in Google Scholar
[36] Lawson, W.: The passionate mind: How people with autism learn. pp. 104–107, Jessica Kingsley Publishers, 2010.Search in Google Scholar
[37] Lofland, K. B.: The Use of Technology in the Treatment of Autism. In: Cardon, T. (ed.), Technology and the Treatment of Children with Autism Spectrum Disorder, Autism and Child Psychopathology Series, Cham: Springer, 2016.Search in Google Scholar
[38] McHugh, L.; Bobarnac, A.; Reed, P.: Brief Report: Teaching Situation-Based Emotions to Children with Autistic Spectrum Disorder. In: Journal of autism and developmental disorders, 41, 1423, 2011.Search in Google Scholar
[39] Mazurek, M. O.; Engelhardt, C. R.; Clark, K. E.: Video games from the perspective of adults with autism spectrum disorder. In: Computers in Human Behavior, 51, 122–130, 2015.Search in Google Scholar
[40] Moebert, T.; Schneider, J.; Zoerner, D.; Tscherejkina, A.; Lucke, U.: How to use Socio-Emotional Signals for Adaptive Training. In: Augstein, M.; Herder, E.; Wörndl, W. (eds.), Personalized Human-Computer Interaction, 103–132, DeGruyter Oldenbourg, 2018.Search in Google Scholar
[41] Moosbrugger, H.; Goldhammer, F.: FAIR, Frankfurter Aufmerksamkeits-Inventar, Bern: Verlag Hans Huber, 1996.Search in Google Scholar
[42] Müller, I.; Moebert, T.; Lucke, U.: Lessons Learned from Designing Adaptive Training Systems: An Ethical Perspective. In: Artificial Intelligence Supported Educational Technologies, Springer, in print.Search in Google Scholar
[43] Perttula, A.; Kiili, K.; Lindstedt, A.; Tuomi, P.: Flow experience in game based learning – a systematic literature review. In: International Journal of Serious Games, 4/1, 57–72, 2017.Search in Google Scholar
[44] Proft, J.; Gawronski, A.; Krämer, K.; Schoofs, T; Kockler, H.; Vogeley, K.: Autismus im Beruf. In: Zeitschrift für Psychiatrie, Psychologie und Psychotherapie, 64/4, 277–285, 2016.Search in Google Scholar
[45] Reiter, A. M.; Kanske, P.; Eppinger, B.; Li, S. C.: The Aging of the Social Mind – Differential Effects on Components of Social Understanding. In: Scientific Reports, 7, 2017.Search in Google Scholar
[46] Ronimus, M.; Kujala, J.; Tolvanen, A.; Lyytinen, H.: Children’s engagement during digital game-based learning of reading: The effects of time, rewards, and challenge. In: Computers & Education, 71, 237–246, 2014.Search in Google Scholar
[47] Ryan, R. M.; Rigby, C. S.; Przybylski, A.: The Motivational Pull of Video Games: A Self-Determination Theory Approach. In: Motiv Emot, 30, 344, 2006. DOI:10.1007/s11031-006-9051-8.Search in Google Scholar
[48] Swartout, W.; van Lent, M.: Making a game of system design. In: Communications of the ACM, 46/7, 32, 2003.Search in Google Scholar
[49] Tscherejkina, A.; Morgiel, A.; Moebert, T.: Computergestütztes Training von sozio-emotionalen Kompetenzen durch Minispiele. In: Evaluation von User Experience. E-Learning Symposium 2018, 41–52, 2018.Search in Google Scholar
[50] Waraich, A.: Using narrative as a motivating device to teach binary arithmetic and logic gates, 2004.Search in Google Scholar
[51] Wang, C.; Shimojo, E.; Shimojo, S.: Don’t look at the eyes: Live interaction reveals strong eye avoidance behavior in autism. In: Journal of Vision, 15/12, 648, September 2015.Search in Google Scholar
[52] Whyte, E. M.; Smyth, J. M.; Scherf, K. S.: Designing serious game interventions for individuals with autism. In: Journal of autism and developmental disorders, 45(12), 3820–3831, 2015.Search in Google Scholar
[53] Zender, R.; Knoth, A. H.; Fischer, M. F.; Lucke, U.: Potentials of Virtual Reality as an Instrument for Research and Education. In: i-com, 18/1, 3–15, 2019.Search in Google Scholar
[54] Zahorik, P.; Jenison, R. L.: Presence as being-in-the-world. In: Presence: Teleoperators and Virtual Environments, 7/1, 78, 1998.Search in Google Scholar
[55] Zoerner, D.; Schütze, J.; Kirst, S.; Dziobek, I.; Lucke, U.: Zirkus Empathico: Mobile Training of Socio-Emotional Competences for Children with Autism. In: Proceedings of the IEEE 16th International Conference on Advanced Learning Technologies (ICALT) 2016, 448–452, 2016.Search in Google Scholar
[56] Zoerner, D.; Moebert, T.; Lucke, U.: IT-gestütztes Training sozio-emotionaler Kognition für Menschen mit Autismus. In: Informatik Spektrum, 40, 546–555, 2017.Search in Google Scholar
[57] Zoerner, D.; Moebert, T.; Morgiel, A.; Strickroth, S.; Lucke, U.: Spielbasierte Förderung von Motivation und Aufmerksamkeit für sozioemotionales Training bei Autismus. In: Proceedings of DeLFI 2018 – Die 16. E-Learning Fachtagung Informatik, 2018.Search in Google Scholar
© 2020 Walter de Gruyter GmbH, Berlin/Boston
