Keywords

1 Introduction and Background

Popular movies such as Rain Man, A Beautiful Mind and The Imitation Game often depicting the leading characters with high-functioning autism spectrum disorder (ASD) with exceptional math skills (referred to as autistic savants) which is largely inconsistent with research [1]; in fact, only 10% of individuals with ASD have such savant skills and is rare in the population [22]; instead, prior studies showed that during their middle school years, students with ASD usually under-perform an average of 5 years below their neuro-typically developing (NT) peers in mathematics [23]. Plausible explanations of such a gap have been attributed to the population’s impairments of working memory, cognitive development, visual-spatial which in turn might undermine their learning abilities in one or more mathematical domains [5, 6, 9, 13, 16, 20]. Whitby [24] also attributed such a gap appeared in their middle school years to the increasing difficulty of this population’s understanding of the abstract nature of mathematics and their impairments in generalizing acquired skills to the real-world settings.

In addition, quite a number of individuals with ASD have posited the fact that they tend to rely more on the visual mental representations instead of verbal ones in understanding abstract concepts in their daily life [7, 11]; a term referred to as ‘thinking in pictures’ [7, 15]. However, due to the wide variability of impairments each individual with ASD demonstrates, the bias towards the adoption of visual mental representations might be suitable to be interpreted at the group, not individual level [15], which thereby demanding avenues for further research. One of our present study goals aims to address this issue by recruiting a population of users that have not been included in previous studies: Chinese children with ASD [21].

Meanwhile, the overwhelming number of prior studies attempted to tap into the intertwined relationships among these factors, but little previous empirical evidence directly addresses the issue of how some fundamental math presentation strategies might affect the learning outcome which motivates our study. In particular, in present study, we aim to gain access to the additional information with regards to the complex interrelationship between presentation features, contexts and learning outcome afforded through the referential understanding of children’s gaze.

2 Related Work

Four indirect lines of prior research are relevant to our present study which will be briefly reviewed here.

2.1 Mathematics Learning of Individuals with ASD

Popular media has portrayed individuals with high-functioning ASD with exceptional math skills (referred to as autistic savants), which is largely inconsistent with research [1]. However, earlier studies showed that during their middle school years, students with ASD usually under-perform an average of 5 years below their neuro-typically developing (NT) peers in mathematics [23]. Plausible explanations of such a gap have been attributed to the population’s impairments of some cognitive impairment which in turn might undermine their learning abilities in one or more mathematical domains [5, 6, 9, 13, 16, 20]. Whitby [24] also attributed such a gap appeared in their middle school years to the increasing difficulty of this population’s understanding of the abstract nature of mathematics and their impairments in generalizing acquired skills to the real-world settings.

In addition, quite a number of individuals with ASD have posited the fact that they tend to rely more on the visual mental representations instead of verbal ones in understanding abstract concepts in their daily life [7, 11]; a term referred to as ‘thinking in pictures’ [7]. However, due to the wide variability of impairments each individual with ASD demonstrates, the bias towards the adoption of visual mental representations might be suitable to be interpreted at the group, not individual level [15], which thereby demanding avenues for further research.

2.2 Eye-Tracking for Behavioral Analysis During the Decision-Making Process

Eye-tracking technology has allowed the shift from traditional outcome-based measurement of decision making (as inferred by choices and preferences, etc.) to process-tracing based methodology [4]. The achievement of the latter has relied on the traditional mouse–tracking methods [17], and eye-tracking which could examine attention, information acquisition, arousal, etc. at the cognitive and neurological level [2,3,4, 19, 25]. For example, Fiedler et al. [3] analyzed the dynamics of risk preferences in two gambles and verified that the visual attention to the outcome of a gamble tends to increase when the probability of winning increases. In addition, the cognitive efforts and arousal indexed by pupil dilation had also been observed to increase. Zhou et al. [25] quantitatively measured the quality of the decision-making process via physiological sensors such as GSR and an eye tracker in an attempt to alter the user interface where human and computer can augment each other’s strengthens. Franco-Watkins and Johnson [4] observed increased pupil dilation during the decision making process and is influenced by presentation format.

2.3 Eye-Tracking for Behavioral Analysis in ASD Research

Eye-tracking technology has been commonly adopted in the autism research area both clinically and empirically due to a number of advantages it has provided [8, 18]: (1) it may be ideally suited to tap into many abnormalities this population are plagued with; (2) it is objective and easily implemented; (3) it is safe to be used from infancy to adulthood [10, 14, 18]. In our study, we will adopt the affordable version of eye-tracking device, Tobii EyeXTM Footnote 1which is marketed to develop eye-gazed based games; such an affordable eye-tracker will not comprise the quality of the collected eye-gaze data; instead, it offers us the same advantages as those of its more expensive peers (such as Tobii Eye GlassTM) by providing additional information with regards to the complex interrelationship between presentation features, contexts and learning outcome. As of present, the overwhelming number of prior works on eye-tracking attempt to probe into the unusual patterns of visual preferences among the population, among many, [12, 14, 18].

2.4 Preferences of Presentation Format Among Children with ASD

One of the notable studies is the one by Kunda and Goel [15] which examined the long-standing assumption of whether individuals with ASD exhibit a deposition towards visual mental representation over verbal mental representation, referred to as the ‘thinking in pictures’ hypothesis. In the context of a number of cognitive tasks, experiment results strongly support the hypothesis that individuals with ASD tend to tap the visual mental presentation for completing these tasks. However, Kunda and Goel urged researchers to take precautions in interpreting the data within the context of the tasks; and called for more in-depth studies at the behavioral and neurobiological levels.

3 Our Math Learning Application: Prototype Version

3.1 Application Design and Math Training Problem Presentation Styles

The application is running on a Windows Tablet (Win10), and consists of a “Training Material (TM)” module, a “Learning Materials (LM)” module and some audio files to (1) read out each question; (2) feedbacks on students’ performance (praise or encouragement). Instead of providing a ‘right’ or ‘wrong’ feedback on accurate and inaccurate answers, we use more encouraging feedback for the child to engage. Hence, three presentation strategies were included: mathematical digit, pictures, and audio file (see Figs. 1, 2 and 3).

Fig. 1.
figure 1

The two presentation styles in the app: Digit (DIG) and Image (IMG)

Full size image
Fig. 2.
figure 2

The Application screen after the child answers the question where navigation buttons are shown

Full size image
Fig. 3.
figure 3

The child’s performance record

Full size image

Stage one—the training session.

At the beginning, the TM module consists of short questions on addition, subtraction and multiplication. Figures 1, 2 and 3 shows the presentation styles and performance record for future analysis.

Meanwhile, the audio instruction of this math problem: “yi jia yi” (one plus one in Chinese) will be played. The app will then animate the arithmetic practice, as shown in Fig. 2 as an example.

Again, the audio instruction of the answer to this math question: “er” (two in Chinese) will be played, followed by a rewarding praise (‘well done’) will be played if the answer is accurate; a redirection sentence will be played to encourage the child to try it again if the answer is inaccurate. The user may choose to go back, repeat, or to the next item (Fig. 2). When the training session is over, a log data showing both the child’s performance as well as the corresponding styles he/she had chosen will be recorded (Fig. 3).

The recorded data (as shown in Fig. 3) can then be saved and analyzed to offer valuable and objective information on the child’s preferences, which then can be fed into the system for personalization.

Stage two—the training session.

A learner can choose one of the presentation styles before each question is presented (as shown in Figs. 4, 5 and 6).

Fig. 4.
figure 4

The mathematical digit style during the training session

Full size image
Fig. 5.
figure 5

The Picture Style during the training session

Full size image
Fig. 6.
figure 6

The audio style where the question will be spoken out in Chinese

Full size image

Again, all learners’ in-application interaction as well as the performance will be recorded (see Fig. 3).

Feed-back from Participatory Design.

The prototype of the application had been tested with three children with ASD in order to obtain feedbacks from both the children and their teachers who accompany them during the testing; interviews with teachers were also conducted. The results of their feedback had been incorporated into the design of next two versions of the application (v2.0 and v3.0).

4 Our Math Learning Application: Eye-Tracking Version (v2.0 and v3.0)

A second version of the application was designed in order to accommodate these feedbacks (participatory design). The application was developed using C# in Microsoft Visual Studio Enterprise 2015; an eye-tracking application (written in C# for Tobii EyeXTM) is running at the background of the Math app which can be utilized to collect the eye-gaze data. Figure 7 captures the screenshots of the system.

Fig. 7.
figure 7

The screenshot of the eye-tracker enabled learning application (v2.0) (Color figure online)

Full size image

During the integration testing, we found out that since a working Tobii EyeX shows an array of three red lights right below the application (as highlighted in green in Fig. 7), it might pose a potential problem to the children who are not aware of an eye-tracking device before. Therefore, in the pilot testing, we adopted the third version of our application which uses a non-intrusive eye tracker Tobii Eye X2-60 (v3.0). Figure 8 shows a screenshot of the application while Fig. 9 shows the images used in the application (v3.0).

Fig. 8.
figure 8

A screenshot of the learning application used in the pilot study (v3.0)

Full size image
Fig. 9.
figure 9

Imagines used in the application (v3.0)

Full size image

In order to assess our assumption of the possible link between presentation types and learning outcomes, we conducted a pilot study which will be described in details in the next section.

5 Study Results and Discussions

5.1 Participants, Apparatus and Study Procedure

Participants.

One nine-year old child with low functioning ASD (LFASD) and one six-year old child with high functioning ASD (AFASD) participated in the pilot study.

Apparatus.

A Dell workstation with 27 inches 4 K monitor with standard mouse and keyboard was used. A light and portable Tobii X2-60 eye-tracking device was mounted on the monitor.

Study procedure.

Due to the high usability of the learning application and conversation with the teachers, no training was provided. Each child was asked to sit in front of the computer and use mouse to complete a total of twelve math questions (simple addition). The researcher demonstrated how to move the mouse and click a correct answer and asked the child to answer the rests. Figure 10 showed the study environment where the dash-line highlighted device is the Tobii X2-60 which was mounted below the monitor.

Fig. 10.
figure 10

The study environment where the dash-line highlighted Tobii X2-60 eye-tracker was mounted below the monitor

Full size image

Study measures.

Following those in previous eye-tracking studies [2,3,4, 18, 25], we adopted heat-maps and eye-gaze plots to measure the subjects’ preferences of the screen objects: image or mathematical symbol. Heat maps show where the subject concentrated their eye-gaze and the duration of the gaze at a given point. A red spot over an area suggest that the subject (or group of subjects) focused on the given point for a longer period of time which could be inferred to demonstrate their interest on the visual spot over other spots (e.g. Fig. 11). Eye-gaze plots, on the other hand, do not combine more than one user in the screen; it visually shows the temporal movement of the user’s fixation on the screen. The size of the dot denotes the duration of the eye-gaze time (Fig. 12).

Fig. 11.
figure 11

The eye-gaze heat-map of the child with LFASD (Color figure online)

Full size image
Fig. 12.
figure 12

The gaze plots of the child with LFASD

Full size image

5.2 Results and Discussions

Since each question has three optional answers, the chance of correctly answering it by random is 1/3, or 4 out of the total 12 questions. The child with LFASD failed to answer any question. In fact, he looked at the monitor for the first four seconds and then looking around (outside the monitor) for the next 70 s until he left. Figure 11 showed the heat-map of his eye-gaze data. The heat-map data showed no preferences towards either presentation styles.

Figure 12 present the gaze plots of the child where no preferences over either symbols were observed (size of the eye-gaze plot does not vary much). It was revealed that the child has not started to learn math, which provides clues to his eye-gaze patterns on the screen.

The child with HFASD solved 8 problems correctly (three wrongly). Out of 11 questions, he clicked pictures 5 times (once wrongly) and digits 6 times (twice wrongly). The selected pictures are clipart car (selected twice), and real doll, animated wheels, and animated book (once for each). His eye-gazing results are shown in Figs. 13 and 14.

Fig. 13.
figure 13

The eye-gaze heat-map of the child with HFASD

Full size image
Fig. 14.
figure 14

The gaze plots of the child with HFASD during one problem-solving task

Full size image

The heat-map data suggested that the child focused more attention on the images (Fig. 13). The gaze plots data further supported it. Figures 14 and 15 showed two of them when he was solving two math question.

Fig. 15.
figure 15

The gaze plots of the child with HFASD during a successful problem-solving task

Full size image

6 Concluding Remarks

In our present study, we aim to compare the effects of three different presentation styles during the training for children with ASD’s mathematical skills (image-based, mathematical digit-based and audio-based ones) through the behavioral analysis and eye-tracking technology. Except for one LFASD child who has not officially started the math education, eye-tracking data indicated that the HFASD child show more interest in images.

Although it is still too early to draw conclusions on the preferences of images over mathematical symbols and the influences of appropriate and personalized design elements in math learning application on children’s performance, our study nevertheless highlight the importance of tuning in such finer grained design elements so as to facilitate personalized learning.

In the future, more in-depth studies on viewing patterns could add to our understanding of the old question of whether individuals with ASD tend to ‘think’ in pictures. We are currently working on a prototype of adaptive math learning application that could react to the child’s preferences over the screen objects (via observation of their eye-gaze patterns).