Predicting expert–novice performance as serious games analytics with objective-oriented and navigational action sequences
Introduction
Generally speaking, the purpose of analytics is to discover value-added properties based on user-generated data. For instance, to stakeholders in the digital and mobile entertainment gaming industry, the purpose of game analytics is to create new revenue out of player-generated gameplay data (i.e., monetization). Since serious games was originally envisioned (Krulak, 1997) to be advanced training tools for the improvement of decision-making skills and job performance in trainees/learners, the purpose of serious games analytics would logically be to gain insights (methods, metrics, and policies) for the stakeholders. While the insights may include the improvement of serious game design, stakeholders are much more interested in the ability of the ‘tool’ to develop skills, raise performance, and improve bottom-line – through the reduction of training cost and its impact on return of investment (see Kozlov and Reinhold, 2007, Loh, 2012a).
Even though, to date, few serious games have an assessment component, the concern for the lack of appropriate methods and metrics for performance measurement is not new (Michael & Chen, 2005). For example, Crookall (2010) asserted that serious games and simulations could add more values by incorporating appropriate debriefing tools for performance assessment and improvement. Seif El-Nasr, Drachen, and Canossa (2013) advocated turning user-generated data into game analytics for monetization. Loh, Sheng, and Ifenthaler (in press) explored various methodologies to measure, assess, and improve performance of training and learning in serious games in a new edited volume, entitled Serious Games Analytics. To better meet the needs of various stakeholders, one has to discover useful metrics (when none is available) for measuring human performance with serious games, identify strong predictors (from among many other weaker ones) for incorporation into methods of ‘best-practice’, and maximize the value of the analytics for return of investment and performance improvement.
Serious games analytics requires a two-step process. The first step is the collection of user-generated data to ascertain what has been done in the training environment, in order to extrapolate what has been learned. Bellotti, Kapralos, Lee, Moreno-Ger, and Berta (2013) reported that the most prevalent methods found in the serious games literature are user surveys and pretest–posttest methods. After analyzing more than 510 data collection techniques used before, after and during digital games, Smith, Blackmore, and Nesbitt (in press) reported that the majority of techniques (before and after) were questionnaires, followed by some kind of test (38% and 31%, respectively). In contrast, these two methods were inefficient for ‘during game’ data collection (0% and 1.5%, respectively). This is because ex situ methods (such as questionnaires, surveys, and pre/post-tests) are less objective compared to in situ user-generated data for serious games analysis: user-surveys are self-reported data (Fan et al., 2006, Hoskin, 2012) and pre/posttest treats games as an impenetrable Black Box (Loh, 2012b). In comparison, in situ data collection methods, using telemetry, trace players’ actions and behaviors from within the gaming habitat itself. User interactions with (predetermined) game events – e.g., objectives met, number of enemies killed, navigational route(s) taken, items discovered, etc., are directly recorded via ‘event listener’ functions in situ and then stored as game logs (on the local machine) or in an online database (on a remote machine). In situ data are relatively free of ‘noise’ caused by human data-input errors, in addition to being more objective by nature.
While a game log (being a plain text file) is the easiest to produce and is probably the most common option in game analytics research (Drachen, Thurau, Togelius, Yannakakis, & Bauckhage, 2013), its usefulness is limited to post hoc (after action) report because a log needs to be further processed before analysis. A better alternative is to capture and store the user-generated data using an online database (Loh et al., 2007, Zoeller, 2013) – especially if an online gaming architecture is already in play. The advantage of online database over game log is that the data are already optimized, and are therefore, immediately available for real-time analytics and ad hoc reporting (Ellis, 2014). The concept of real-time analytics of serious games by combining telemetry and online database to facilitate ad hoc reporting is not new, viz. Information Trails (Loh, 2006). The Information Trails communicates its analytics in both ad hoc (real-time) and post hoc analysis by way of a visualization component, viz. Performance Tracing Report Assistant (or PeTRA; Loh, 2012a, Loh, 2012b).
Once the user-generated gameplay data become available, the second step in the serious games analytics process is to mine or analyze the data for any hidden patterns therein – e.g., via statistical or machine learning methods and pattern recognition techniques. As there can be literally thousands of predictors available (depending on the size of the data), the key is to weed out the weak predictors by identifying the strong ones. The goal is to (a) identify features in serious games (design) for the measurement of players’ skills, abilities, and knowledge, (b) assess players’ performance as evidence of usefulness of serious games, and (c) facilitate the formulation of new policies/insights for performance improvement in the human trainee/learners and the serious games for training/learning.
Section snippets
Expert–novice differences
A clear understanding of the differences between experts and novices can help us understand how knowledge is acquired, target the differences for (re)training and improvement, and teach new (systemic) skills in a variety of situations: e.g., robotic surgery, aviation, sports, music performance, strategic thinking, disaster preparation, behavior recognition, and many others. Research about behavioral and cognitive differences between skilled individuals and novices – or expertise in general,
Materials and methods
In order to maintain the validity of Jaccard coefficients for comparison between those calculated in this study and those from the previous studies by Loh and Sheng, 2013, Loh and Sheng, 2014, Loh and Sheng, 2015, we made the necessary arrangement for the same serious game to be used in our study. (Readers are referred elsewhere for the detailed descriptions of the serious games, Guardian 2.0.) Permission to conduct the experiment was obtained from Human Subject Review Board and all players had
Results and discussion
We traced a total of 534,837 raw (gameplay) data points in situ using the Information Trails empowered serious game, Guardian 2.0. A total of 62 players (55 novices, 7 experts) from a mid-western public university participated in this study. All players attempted the Challenge Round, but only 56 of them (49 novices, 7 experts) completed Round 2. Six novices had to be dropped from Challenge Round due to technical issues and/or network problems.
We obtained the simple (bivariate) correlations
Conclusions
As Box and Draper (1987) wrote, “Essentially, all models are wrong, but some are useful” (p. 424). We presented PLS-DA as a classification (DA) method to predict expert–novice performance for serious games analytics and the insights gained from the predictive modeling. By using PLS-DA, we reduced he dimensions from the original six predictor variables to two components and presented the model in parsimony – the simplest model that also accurately predicts as many experts/novices as possible in
Acknowledgments
This research was made possible in part through funding from the Defense University Research Instrumentation Program (DURIP) from the U.S. Army Research Office. The authors wished to thank Mr. Ting Zhou and Dr. JaeHwan Byun for their assistance in data collection, and Ms. Ariel Yining Loh for editing the manuscript.
References (72)
- et al.
The effect of time pressure on risky choice behavior
Acta Psychologica
(1981) - et al.
The effects of video game playing on attention, memory, and executive control
Acta Psychologica
(2008) - et al.
Training “Greeble” experts: A framework for studying expert object recognition processes
Vision Research
(1998) Decision support for real-time, dynamic decision-making tasks
Organizational Behavior and Human Decision Processes
(2005)- et al.
Web Search behavior of Internet experts and newbies
Computer Networks
(2000) - et al.
A study on thinking strategy between experts and novices of computer games
Computers in Human Behavior
(2003) Current practice in measuring usability: Challenges to usability studies and research
International Journal of Human-Computer Studies
(2006)- et al.
Maximum Similarity Index (MSI): A metric to differentiate the performance of novices vs. multiple-experts in serious games
Computer in Human Behavior
(2014) - et al.
Visual attention during brand choice: The impact of time pressure and task motivation
International Journal of Research in Marketing
(1999) From novice to expert decision behaviour: A qualitative modelling approach with Petri Nets
Advances in Human Factors/Ergonomics
(1995)
Serious games for knee replacement surgery procedure education and training
Procedia – Social and Behavioral Sciences
Analysis of revisitations in online games
Entertainment Computing
Characteristics of the mental representations of novice and expert programmers: An empirical study
International Journal of Man-Machine Studies
Perceptual-cognitive expertise in sport: Some considerations when applying the expert performance approach
Human Movement Science
Prediction of user’s web-browsing behavior: Application of markov model
IEEE Transactions on Systems, Man, Cybernetics, Part B: Cybernetics
Classification tools in chemistry. Part 1: Linear models. PLS-DA
Analytical Methods
From novice to expert performance: Memory, attention and the control of complex sensori-motor skills
Do action video games improve perception and cognition?
Frontiers in Psychology
Empirical model building and response surfaces
Studies on the telegraphic language: The acquisition of a hierarchy of habits
Psychological Review
DashMash: A mashup environment for end user development
Confounding in educational computing research
Journal of Educational Computing Research
Dangers in the evaluation of instructional media
Academic Medicine
Media will never influence learning
Educational Technology Research and Development
Serious games, debriefing, and simulation/gaming as a discipline
Simulation and Gaming
Thought and Choice in Chess
Evaluating motion: Spatial user behaviour in virtual environment
International Journal of Arts and Technology
The five-stage model of adult skill acquisition
Bulletin of Science, Technology and Society
Peripheral vision: Expertise in real world contexts
Organization Studies
Real-time analytics: Techniques to analyze and visualize streaming data
Cited by (17)
Expert-novice classification of mobile game player using smartphone inertial sensors
2021, Expert Systems with ApplicationsCitation Excerpt :In Anwar et al. (2016) and Anwar et al. (2018), EEG signals of the participants were analyzed to identify expert-novice level of the mobile game player during game play. In Loh, Sheng, and Li (2015), objective-oriented and navigational action sequences were used to analyze and predict the expert-novice performance of the game players. It was evaluated that how some specific objectives are attained during game play for classification.
The effect of sustained vs. faded scaffolding on students’ argumentation in ill-structured problem solving
2018, Computers in Human BehaviorCitation Excerpt :Spector and Koszalk (2004) maintained that learning progress can be viewed as cognitive changes into the direction of expert-like knowledge structure. According to the literature on expertise and novice differences (Anderson, 2015; Ifenthaler, 2010, 2014; Loh, Sheng, & Li, 2015), the expert's knowledge structure is organized in meaningful ways that reflect a holistic understanding of knowledge structure, which is an indicator of deep learning. The similarity to the expert argumentation was not only true for the week that learners had access to the scaffold (Week 2), but also in the transfer week (Week 3).
Hybrid Multi-Agent Strategy Discovering Algorithm for human behavior
2017, Expert Systems with ApplicationsCitation Excerpt :They obtained the data for defining the decision support system (DSS) manually, whereas our contribution presents a mechanism to extract the patterns to be considered during the decision-making process automatically. Loh, Sheng, and Li (2015) have analyzed navigational action sequences in order to differentiate between expert and novice performance, and to track how the behavior of novices improves throughout training rounds. They divided the domain map into several smaller grids, and compared the similarity of navigational patterns by using the Jaccard similarity.
Comparison of similarity measures to differentiate players' actions and decision-making profiles in serious games analytics
2016, Computers in Human BehaviorCitation Excerpt :As players interact with the gaming/training elements in the serious games, they generate action data that can be traced using software telemetry (Chung, 2015; Zoeller, 2013) and other specifically created assessment frameworks, such as Information Trails (Loh, 2012, pp. 123–144; Loh, Anantachai, Byun, & Lenox, 2007). A database of captured action data can be (data)mined to yield analytics and insights for training and improvement: e.g., visualize players’ navigational paths (Chittaro, Ranon, & Ieronutti, 2006; Loh, Sheng, & Li, 2015; Zacharias, 2006), measure changes in their proficiency levels (Loh & Sheng, 2015a), differentiate the novices from the experts (Loh & Sheng, 2014, 2015b), and diagnose potential problems (Liu, Shen, Mei, Ji, & Miao, 2013; Mihail et al., 2015), to name a few. Hopefully, in the near future, serious games can mature into the “tools for improving decision-making skills and performance” (Krulak, 1997; Michael & Chen, 2005; Sawyer & Rejeski, 2002) – as originally intended.
DeepSkill: A methodology for measuring teams’ skills in massively multiplayer online games
2024, Multimedia Tools and Applications
- 1
Address: Department of Counseling, Quantitative Methods, and Special Education, Southern Illinois University, 625 Wham Drive, Carbondale, IL 62901-4618, USA. Tel.: +1 618 453 6913.
- 2
Address: Virtual Environment Lab (V-LAB), Department of Curriculum & Instruction, Southern Illinois University, 625 Wham Drive, Carbondale, IL 62901-4610, USA. Tel.: +1 618 503 0188.