Elsevier

Computers in Human Behavior

Volume 49, August 2015, Pages 147-155
Computers in Human Behavior

Predicting expert–novice performance as serious games analytics with objective-oriented and navigational action sequences

https://doi.org/10.1016/j.chb.2015.02.053Get rights and content

Highlights

  • New metrics are needed to produce insights in serious games analytics.

  • Action sequences can be coded using task- or tile-based approach.

  • Size of map grids can affects the accuracy in performance prediction.

  • Grid size affects tile-based coding of action sequences and performance prediction.

  • The optimal map grid size must be determined independently for each game.

Abstract

Previous research differentiated expert vs. novice performances based on how (dis)similar novices’ action sequences were from that of the expert’s by way of similarity measures. Action sequences were coded using an ‘objective-oriented’ (or task-based) approach based on the sequence of objectives/tasks completed in-game. Findings from these studies suggest that the task-based similarity measures is a better predictor than (a) distance traversed, and (b) time (of completion).

In this study, we suggested an alternative method to code action sequences of experts and novices by way of a ‘navigational’ (or tile-based) approach. We divided a game-map into grids/tiles of different sizes to facilitate tracing of the path traversed by players in game and proceeded to test the effect of grid sizes on differentiating between experts and novices. We further compared the two different action sequence coding approaches and their abilities to measure players’ competency improvement in serious games. The results of the study showed that the size of game grids does matter, and that both task-based and tile-based action sequence coding approaches are useful for serious games analytics.

Introduction

Generally speaking, the purpose of analytics is to discover value-added properties based on user-generated data. For instance, to stakeholders in the digital and mobile entertainment gaming industry, the purpose of game analytics is to create new revenue out of player-generated gameplay data (i.e., monetization). Since serious games was originally envisioned (Krulak, 1997) to be advanced training tools for the improvement of decision-making skills and job performance in trainees/learners, the purpose of serious games analytics would logically be to gain insights (methods, metrics, and policies) for the stakeholders. While the insights may include the improvement of serious game design, stakeholders are much more interested in the ability of the ‘tool’ to develop skills, raise performance, and improve bottom-line – through the reduction of training cost and its impact on return of investment (see Kozlov and Reinhold, 2007, Loh, 2012a).

Even though, to date, few serious games have an assessment component, the concern for the lack of appropriate methods and metrics for performance measurement is not new (Michael & Chen, 2005). For example, Crookall (2010) asserted that serious games and simulations could add more values by incorporating appropriate debriefing tools for performance assessment and improvement. Seif El-Nasr, Drachen, and Canossa (2013) advocated turning user-generated data into game analytics for monetization. Loh, Sheng, and Ifenthaler (in press) explored various methodologies to measure, assess, and improve performance of training and learning in serious games in a new edited volume, entitled Serious Games Analytics. To better meet the needs of various stakeholders, one has to discover useful metrics (when none is available) for measuring human performance with serious games, identify strong predictors (from among many other weaker ones) for incorporation into methods of ‘best-practice’, and maximize the value of the analytics for return of investment and performance improvement.

Serious games analytics requires a two-step process. The first step is the collection of user-generated data to ascertain what has been done in the training environment, in order to extrapolate what has been learned. Bellotti, Kapralos, Lee, Moreno-Ger, and Berta (2013) reported that the most prevalent methods found in the serious games literature are user surveys and pretest–posttest methods. After analyzing more than 510 data collection techniques used before, after and during digital games, Smith, Blackmore, and Nesbitt (in press) reported that the majority of techniques (before and after) were questionnaires, followed by some kind of test (38% and 31%, respectively). In contrast, these two methods were inefficient for ‘during game’ data collection (0% and 1.5%, respectively). This is because ex situ methods (such as questionnaires, surveys, and pre/post-tests) are less objective compared to in situ user-generated data for serious games analysis: user-surveys are self-reported data (Fan et al., 2006, Hoskin, 2012) and pre/posttest treats games as an impenetrable Black Box (Loh, 2012b). In comparison, in situ data collection methods, using telemetry, trace players’ actions and behaviors from within the gaming habitat itself. User interactions with (predetermined) game events – e.g., objectives met, number of enemies killed, navigational route(s) taken, items discovered, etc., are directly recorded via ‘event listener’ functions in situ and then stored as game logs (on the local machine) or in an online database (on a remote machine). In situ data are relatively free of ‘noise’ caused by human data-input errors, in addition to being more objective by nature.

While a game log (being a plain text file) is the easiest to produce and is probably the most common option in game analytics research (Drachen, Thurau, Togelius, Yannakakis, & Bauckhage, 2013), its usefulness is limited to post hoc (after action) report because a log needs to be further processed before analysis. A better alternative is to capture and store the user-generated data using an online database (Loh et al., 2007, Zoeller, 2013) – especially if an online gaming architecture is already in play. The advantage of online database over game log is that the data are already optimized, and are therefore, immediately available for real-time analytics and ad hoc reporting (Ellis, 2014). The concept of real-time analytics of serious games by combining telemetry and online database to facilitate ad hoc reporting is not new, viz. Information Trails (Loh, 2006). The Information Trails communicates its analytics in both ad hoc (real-time) and post hoc analysis by way of a visualization component, viz. Performance Tracing Report Assistant (or PeTRA; Loh, 2012a, Loh, 2012b).

Once the user-generated gameplay data become available, the second step in the serious games analytics process is to mine or analyze the data for any hidden patterns therein – e.g., via statistical or machine learning methods and pattern recognition techniques. As there can be literally thousands of predictors available (depending on the size of the data), the key is to weed out the weak predictors by identifying the strong ones. The goal is to (a) identify features in serious games (design) for the measurement of players’ skills, abilities, and knowledge, (b) assess players’ performance as evidence of usefulness of serious games, and (c) facilitate the formulation of new policies/insights for performance improvement in the human trainee/learners and the serious games for training/learning.

Section snippets

Expert–novice differences

A clear understanding of the differences between experts and novices can help us understand how knowledge is acquired, target the differences for (re)training and improvement, and teach new (systemic) skills in a variety of situations: e.g., robotic surgery, aviation, sports, music performance, strategic thinking, disaster preparation, behavior recognition, and many others. Research about behavioral and cognitive differences between skilled individuals and novices – or expertise in general,

Materials and methods

In order to maintain the validity of Jaccard coefficients for comparison between those calculated in this study and those from the previous studies by Loh and Sheng, 2013, Loh and Sheng, 2014, Loh and Sheng, 2015, we made the necessary arrangement for the same serious game to be used in our study. (Readers are referred elsewhere for the detailed descriptions of the serious games, Guardian 2.0.) Permission to conduct the experiment was obtained from Human Subject Review Board and all players had

Results and discussion

We traced a total of 534,837 raw (gameplay) data points in situ using the Information Trails empowered serious game, Guardian 2.0. A total of 62 players (55 novices, 7 experts) from a mid-western public university participated in this study. All players attempted the Challenge Round, but only 56 of them (49 novices, 7 experts) completed Round 2. Six novices had to be dropped from Challenge Round due to technical issues and/or network problems.

We obtained the simple (bivariate) correlations

Conclusions

As Box and Draper (1987) wrote, “Essentially, all models are wrong, but some are useful” (p. 424). We presented PLS-DA as a classification (DA) method to predict expert–novice performance for serious games analytics and the insights gained from the predictive modeling. By using PLS-DA, we reduced he dimensions from the original six predictor variables to two components and presented the model in parsimony – the simplest model that also accurately predicts as many experts/novices as possible in

Acknowledgments

This research was made possible in part through funding from the Defense University Research Instrumentation Program (DURIP) from the U.S. Army Research Office. The authors wished to thank Mr. Ting Zhou and Dr. JaeHwan Byun for their assistance in data collection, and Ms. Ariel Yining Loh for editing the manuscript.

References (72)

  • H. Sabri et al.

    Serious games for knee replacement surgery procedure education and training

    Procedia – Social and Behavioral Sciences

    (2010)
  • R. Thawonmas et al.

    Analysis of revisitations in online games

    Entertainment Computing

    (2011)
  • S. Wiedenbeck et al.

    Characteristics of the mental representations of novice and expert programmers: An empirical study

    International Journal of Man-Machine Studies

    (1993)
  • A.M. Williams et al.

    Perceptual-cognitive expertise in sport: Some considerations when applying the expert performance approach

    Human Movement Science

    (2005)
  • Adar, E., Teevan, J., Dumais, S. T. (2008). Large scale analysis of web revisitation patterns. In Proceedings of the...
  • M.A. Awad et al.

    Prediction of user’s web-browsing behavior: Application of markov model

    IEEE Transactions on Systems, Man, Cybernetics, Part B: Cybernetics

    (2012)
  • D. Ballabio et al.

    Classification tools in chemistry. Part 1: Linear models. PLS-DA

    Analytical Methods

    (2013)
  • S.L. Beilock et al.

    From novice to expert performance: Memory, attention and the control of complex sensori-motor skills

  • Bellotti, F., Kapralos, B., Lee, K., Moreno-Ger, P., & Berta, R. (2013). Assessment in and of Serious Games: An...
  • W.R. Boot et al.

    Do action video games improve perception and cognition?

    Frontiers in Psychology

    (2011)
  • G.E.P. Box et al.

    Empirical model building and response surfaces

    (1987)
  • W.L. Bryan et al.

    Studies on the telegraphic language: The acquisition of a hierarchy of habits

    Psychological Review

    (1899)
  • C. Cappiello et al.

    DashMash: A mashup environment for end user development

  • Chi, E. H., Pirolli, P., Chen, K., Pitkow, J. (2001). Using information scent to model user information needs and...
  • R.E. Clark

    Confounding in educational computing research

    Journal of Educational Computing Research

    (1985)
  • R.E. Clark

    Dangers in the evaluation of instructional media

    Academic Medicine

    (1992)
  • R.E. Clark

    Media will never influence learning

    Educational Technology Research and Development

    (1994)
  • D. Crookall

    Serious games, debriefing, and simulation/gaming as a discipline

    Simulation and Gaming

    (2010)
  • A. de Groot

    Thought and Choice in Chess

    (1978)
  • A. Drachen et al.

    Evaluating motion: Spatial user behaviour in virtual environment

    International Journal of Arts and Technology

    (2011)
  • Drachen, A., Thurau, C., Togelius, J., Yannakakis, G. N., & Bauckhage, C. (2013). Game data mining. In: M. Seif...
  • S.E. Dreyfus

    The five-stage model of adult skill acquisition

    Bulletin of Science, Technology and Society

    (2004)
  • Dreyfus, S. E., & Dreyfus, H. L. (1980). A five-stage model of the mental activities involved in directed skill...
  • H.L. Dreyfus et al.

    Peripheral vision: Expertise in real world contexts

    Organization Studies

    (2005)
  • B. Ellis

    Real-time analytics: Techniques to analyze and visualize streaming data

    (2014)
  • Cited by (17)

    • Expert-novice classification of mobile game player using smartphone inertial sensors

      2021, Expert Systems with Applications
      Citation Excerpt :

      In Anwar et al. (2016) and Anwar et al. (2018), EEG signals of the participants were analyzed to identify expert-novice level of the mobile game player during game play. In Loh, Sheng, and Li (2015), objective-oriented and navigational action sequences were used to analyze and predict the expert-novice performance of the game players. It was evaluated that how some specific objectives are attained during game play for classification.

    • The effect of sustained vs. faded scaffolding on students’ argumentation in ill-structured problem solving

      2018, Computers in Human Behavior
      Citation Excerpt :

      Spector and Koszalk (2004) maintained that learning progress can be viewed as cognitive changes into the direction of expert-like knowledge structure. According to the literature on expertise and novice differences (Anderson, 2015; Ifenthaler, 2010, 2014; Loh, Sheng, & Li, 2015), the expert's knowledge structure is organized in meaningful ways that reflect a holistic understanding of knowledge structure, which is an indicator of deep learning. The similarity to the expert argumentation was not only true for the week that learners had access to the scaffold (Week 2), but also in the transfer week (Week 3).

    • Hybrid Multi-Agent Strategy Discovering Algorithm for human behavior

      2017, Expert Systems with Applications
      Citation Excerpt :

      They obtained the data for defining the decision support system (DSS) manually, whereas our contribution presents a mechanism to extract the patterns to be considered during the decision-making process automatically. Loh, Sheng, and Li (2015) have analyzed navigational action sequences in order to differentiate between expert and novice performance, and to track how the behavior of novices improves throughout training rounds. They divided the domain map into several smaller grids, and compared the similarity of navigational patterns by using the Jaccard similarity.

    • Comparison of similarity measures to differentiate players' actions and decision-making profiles in serious games analytics

      2016, Computers in Human Behavior
      Citation Excerpt :

      As players interact with the gaming/training elements in the serious games, they generate action data that can be traced using software telemetry (Chung, 2015; Zoeller, 2013) and other specifically created assessment frameworks, such as Information Trails (Loh, 2012, pp. 123–144; Loh, Anantachai, Byun, & Lenox, 2007). A database of captured action data can be (data)mined to yield analytics and insights for training and improvement: e.g., visualize players’ navigational paths (Chittaro, Ranon, & Ieronutti, 2006; Loh, Sheng, & Li, 2015; Zacharias, 2006), measure changes in their proficiency levels (Loh & Sheng, 2015a), differentiate the novices from the experts (Loh & Sheng, 2014, 2015b), and diagnose potential problems (Liu, Shen, Mei, Ji, & Miao, 2013; Mihail et al., 2015), to name a few. Hopefully, in the near future, serious games can mature into the “tools for improving decision-making skills and performance” (Krulak, 1997; Michael & Chen, 2005; Sawyer & Rejeski, 2002) – as originally intended.

    View all citing articles on Scopus
    1

    Address: Department of Counseling, Quantitative Methods, and Special Education, Southern Illinois University, 625 Wham Drive, Carbondale, IL 62901-4618, USA. Tel.: +1 618 453 6913.

    2

    Address: Virtual Environment Lab (V-LAB), Department of Curriculum & Instruction, Southern Illinois University, 625 Wham Drive, Carbondale, IL 62901-4610, USA. Tel.: +1 618 503 0188.

    View full text