Empowering higher education students to monitor their learning progress: opportunities of computerised classification testing

2.1 Online self-assessment in higher education

Online self-assessment in higher education describes the assessment of students learning with digital tools, including information and communication technologies (Conrad and Openo, 2018). This does not restrict online self-assessment to fully online courses and can also be implemented in a blended learning format (Gikandi et al., 2011). Online self-assessments may take on different pedagogical functions as part of online learning environments (Webb and Ifenthaler, 2018), for example, scaffolding students to complete a task and measuring how much support they need (Ahmed and Pollitt, 2010) or providing students with semantically rich and personalised feedback, as well as adaptive prompts for reflection (Ifenthaler, 2012; Schumacher and Ifenthaler, 2021). Other examples of online self-assessments include a pedagogical agent acting like a virtual coach tutoring learners and providing feedback when needed (Johnson and Lester, 2016) as well as an analysis of a learner’s decisions during a digital game or simulation (Bellotti et al., 2013). Other online self-assessments use multimedia-constructed response items for authentic learning experiences (Lenhart, 2015) or provide students with an emotionally engaging virtual world experience that unobtrusively documents the progression of a person’s leadership and ethical development over time (Turkay and Tirthali, 2010). Thus, online self-assessments offer a broad range of pedagogical functions, including a medium for communication, a learning assistant, a judge, a test administrator, a performance prompt, a practice arena or a performance workspace (Webb et al., 2013).

To facilitate learning, online self-assessment tasks should be learning tasks that are related to the defined learning outcomes and distributed across the learning and course period (Carless, 2007). Furthermore, online self-assessments are a useful method for encouraging learners' ownership (Bennett, 2011; Wanner and Palmer, 2018) of their learning and self-regulation (Panadero et al., 2017). Hence, online self-assessments enable learners to develop independence from relying on external feedback (Andrade, 2010). However, online self-assessments demand and foster evaluative judgment of learners (Tai et al., 2018). Thus, online self-assessments might be particularly challenging for learners with lower levels of domain or procedural knowledge (Sitzmann et al., 2010). Hence, the feedback generated internally by the learners could be complemented and further enhanced with external feedback (Butler and Winne, 1995). Such external feedback may help learners adjust their self-monitoring (Sitzmann et al., 2010). Among others, the feedback provided should clearly define expectations (i.e. criteria, standards, goals), be timely, sufficiently frequent and detailed, be on aspects that are malleable through the students, be on how to close the gap, in a way learners can react upon it (Nicol and Macfarlane‐Dick, 2006).

It is expected that digitally supported and data-driven systems may enhance external feedback while meeting several specific requirements, such as follows:

• adaptability to different subject domains;

• flexibility for experimental as well as learning and teaching settings;

• management of huge amounts of data;

• rapid analysis of complex and unstructured data;

• immediate feedback for learners and educators; and

• generation of automated reports of results for educational decision-making (Ifenthaler et al., 2010).

2.2 Computerized classification testing

CCT has a long history in psychological and educational research as well as pedagogical practice (van der Linden and Glas, 2000). CCT systems aim at classifying learners into two or more categories (van Groen, 2012). Classifications with two categories (Huebner, 2012; van Groen et al., 2019), as well as three and more categories (Eggen and Straetmans, 2000), are frequently implemented. CCT uses various methodological approaches and algorithms for classifying learners with the least number of items (Thompson, 2007). The Sequential Probability Ratio Test (SPRT) is a frequently used algorithm in CCT systems. SPRT follows a decision matrix to decide which one out of two hypotheses is more correct (Wald, 1947). Such algorithms enable CCT systems to select and present the most appropriate assessment items to individual learners (Spray and Reckase, 1996) in comparison to an expected standard or predefined benchmark (Parshall et al., 2002). According to Frick (1990), SPRT algorithms are less complex and more practical for implementation as well as require less time for rendering decisions. For instance, Frick (1992) found that the SPRT algorithm could classify a learner as a master (advanced learner in a specific knowledge domain) vs a nonmaster (novice learner in a specific knowledge domain) using an average of ten assessment items.

2.3 Data-driven dashboards

Dashboards are customizable control panels displaying features that may adapt to a specified process in real-time. Dashboards in the context of learning analytics are being developed and implemented to visualize the analytics results of learner-generated data and other relevant information. Such visualizations are expected to create awareness and reflection among learners (Roberts et al., 2017). The functions of visualizations include exploration, discovery, summarising, presenting, comparing and enjoying (Verbert et al., 2014).

Current research on dashboards aims to identify which data are meaningful to different stakeholders in education and how data can be presented to support learning processes and outcomes (Bodily and Verbert, 2017; Sahin and Ifenthaler, 2021). The objectives of recent dashboard research include the following:

• increasing awareness about the learning process;

• supporting cognitive processes;

• identifying students at risk;

• providing immediate feedback;

• displaying achievement level;

• providing procedural information;

• supporting decision-making;

• informing;

• showing participant relationships;;

• comparing with peers; and

• reflecting learning activities.

Most visualization techniques stem from statistics, including bar charts, line graphs, tables, pie charts and network graphs.

A recent systematic literature review identified 76 studies focusing on various dashboard features and stakeholder applications (Sahin and Ifenthaler, 2021). Contrary to previous findings (Leitner et al., 2019), this current state of research identified an increase in empirical research studies. Nevertheless, the available research does not include sufficient experimental and controlled evidence with a specific focus on the design of dashboards for online self-assessment.

2.4 Current study

This project focused on providing a CCT system that can easily be embedded as an online self-assessment feature into the existing legacy environment of higher education institutions, empowering students with online self-assessments to monitor their learning progress and following strict data protection regulations.

This study has three aims:

1. to investigate the interaction with two different versions of the CCT system (without dashboard vs with dashboard) during the course of a semester;

2. to examine changes in the intended use and perceived usefulness of two different versions (without dashboard vs with dashboard) of the CCT system; and

3. to compare the self-reported confidence levels of two different versions (without dashboard vs with dashboard) of the CCT system.

3.1 Design and participants

Through evidence-based design, the implementation of the CCT system for supporting online self-assessments was part of a larger initiative for establishing data-driven features into the existing legacy system of the higher education institution. For instance, learning analytics features and promoting functions were implemented into the learning management system (Klasen and Ifenthaler, 2019).

This design-based research study was conducted over the course of two semesters for two similar study cohorts as part of a lecture in a bachelor’s program on research methodology in the field of education. In the first iteration of the CCT system, students had the opportunity to interact with the CCT system without the support of a dashboard. The second iteration of the CCT system included a dashboard focussing on individual learning progression (Figure 1) and group comparisons (Figure 2).

The dashboard included information on the individual self-assessment results, i.e. the number of mastery subject areas, number of correct answers, number of incorrect answers, number of total responses and number of total attempts. In addition to the information displayed on the individual dashboard (Figure 1), the group dashboard included averaged information about other anonymized learners, which enabled the learner to compare individual and group performance (Figure 2).

A total ofN = 194 students (138 female; 56 male) from a European university in the area of economic and business education participated in the study. The participants’ average age was 23.15 years (SD = 2.34), while they studied for an average of 5.71 semesters (SD = 1.49). The first study cohort (without a dashboard; SC1) included N = 107 students (71 female; 36 male), and their average age was 23.32 (SD = 2.51). The second cohort (with dashboard; SC2) consisted of N = 87 students (67 female; 20 male) with an average age of 22.94 (SD = 2.11). Both study cohorts are similar with regard to their course enrolment, prior knowledge and study experience. Ethics consent was obtained for this research project.

3.2 Computerized classification testing system

At the start of the semester, the CCT system was embedded as an online self-assessment feature in the productive learning management system. Various subject areas for self-assessment were defined:

• research approaches;

• research process;

• research designs;

• statistical correlations;

• advanced statistical correlations;

• significant differences;

• advanced significant difference; and

• research quality criteria.

A total of N = 256 assessment items (true/false and multiple-choice) were available in the item bank. The CCT system was available throughout the semester.

The CCT SPRT algorithm was implemented for classifying learners into two categories (van Groen, 2012): master and nonmaster. In addition, a random item selection feature was implemented. Figure 3 shows an example of an individualized performance chart learners received after a completed round of online self-assessments. The charts provide an overview of the individual task performance (blue line in Figure 3) as well as the thresholds for being master (red line in Figure 3) and nonmaster (yellow line in Figure 3).

A total of nine indicators were implemented for classifying the performance of learners (see Table 1). Principal component analysis (PCA) as a feature selection algorithm was conducted to determine which of the indicators collected by the CCT system are more important in students' behavioral engagement (Table 1). PCA is used to reduce the number of components in the available data set (Fabrigar et al., 1999). Accordingly, it is expected that the communality values of the indicators are high. Factor scores were examined to determine which indicators provided more information, i.e. the importance of participants’ engagement with the CCT system. The most important indicators for learners' behavioral engagement in the CCT system are the number of responses, the number of attempts, the number of correct answers and the number of incorrect answers. These indicators explain 91% of learners' behavioral engagement structure. These indicators were implemented in the CCT system dashboards.

3.3 Materials and instruments

Instruments for collecting co-variates included the FSI (Schiefele et al., 1993), a study interest questionnaire (Cronbach’s α = 0.90), the short version of the LMI-K (Schuler and Prochaska, 2001), which is an achievement motivation inventory (Cronbach’s α = 0.94), measures focusing on perceived usefulness (Cronbach’s α = 0.90) (Davis, 1989) and confidence (Cronbach’s α = 0.88) as well as demographic data (e.g. gender, age, study experience).

FSI and LMI-K data collection tools were used for internal validity. Learners’ confidence and perceived usefulness level were tested according to the study cohorts (SC1: CCT without dashboard; SC2: CCT with dashboard).

3.4 Procedure and analysis

Participants received a brief introduction to the CCT system and could access it anytime during the semester. At the end of the semester, participants were asked to complete the following surveys:

• FSI;

• LMI-K;

• perceived usefulness;

• confidence in the CCT system; and

• demographic data.

The course performance was assessed through an open-ended exam with a duration of 90 min. The examined questions related to the five areas of the CCT system. The exam scores were classified into high and low performance, where low performance indicated an increased need for learning support.

Log data from the CCT environment were collected using a time-stamped sequence format. Log data of the CCT system represented two categories:

1. interactions with the assessment items; and

2. interactions with the dashboard.

Interactions with the assessment items included the login frequency, number of correct responses, number of master subjects, number of incorrect answers and number of total attempts. The dashboard interactions considered the time spent on individual or group dashboards.

As a standard data-protection practice, all data were stored and analyzed using an anonymized procedure. Time-stamped log-data from the CCT system included the following:

• login frequency;

• login duration;

• correct answers;

• incorrect answers;

• number of responses;

• master subject areas;

• nonmaster subject areas;

• test attempts; and

• subject area attempts.

Data were cleaned and combined for descriptive and inferential statistics using SPSS version 27.

4.1 Computerized classification testing system interaction

For answering the first research question, the participants’ CCT system interactions were determined based on the CCT system’s log-data. Log-data consisted of the number of responses, test attempts, number of correct answers and number of incorrect answers. We computed an independent t-test to check for differences in participants’ CCT system interaction between the study cohorts (SC1 vs SC2). The independent t-test analysis revealed a highly significant difference in CCT system interaction between SC1 (without dashboard; M = −0.31; SD = 0.59) and SC2 (with dashboard; M = 0.37; SD = 1.25), t(192) = 4.936, p < 0.001, d = 0.69.

Accordingly, participants of the second study cohort (SC2; dashboard available) interacted significantly higher with the CCT system than participants of the first study cohort (SC1; no dashboard).

4.2 Intended use and perceived usefulness

Regarding the second research question, the participants self-reported intended use and perceived usefulness were analyzed. We computed two independent t-tests to check for differences in participants’ intended use and perceived usefulness of the CCT system between the study cohorts (SC1 vs SC2). The first independent t-test analysis revealed a significant difference in intended use between SC1 (without dashboard; M = 17.72; SD = 2.85) and SC2 (with dashboard; M = 16.59; SD = 3.11), t(192) = 2.641, p < 0.01, d = 0.38. The second independent t-test analysis revealed a significant difference of perceived usefulness between SC1 (without dashboard; M = 20.87; SD = 3.55) and SC2 (with dashboard; M = 19.47; SD = 3.72), t(192) = 2.672, p < 0.01, d = 0.39.

Accordingly, participants of the first study cohort (SC1; without dashboard) reported significantly higher intended use and perceived usefulness of the CCT system than participants of the second study cohort (SC2; with dashboard).

4.3 Confidence

For answering the third research question, the participant’s self-reported confidence level was examined. We computed an independent t-test to check for differences in participants’ confidence level regarding the CCT system between the study cohorts (SC1 vs SC2). The independent t-test analysis revealed a significant difference in the confidence level between SC2 (with dashboard; M = 7.71; SD = 1.81) and SC1 (without dashboard; M = 7.08; SD = 1.80), t(192) = 2.415, p < 0.01, d = 0.35.

Accordingly, participants of the second study cohort (SC2; dashboard available) reported significantly higher confidence level in the CCT system than participants of the first study cohort (SC1; no dashboard).

5.1 Summary of findings

The findings indicate that the students used the CCT system intensively throughout the semester. In-depth PCA revealed that the number of total responses, the number of attempts, correct answers and incorrect answers seem to be valid metrics for being included in the design of future self-assessment or learning analytics systems (Park and Jo, 2015; Tempelaar et al., 2015).

The first research question investigated the use of the two different versions (without dashboard vs with dashboard) of the CCT system during the course of a semester. The findings revealed a significant difference between the two study cohorts, with a medium effect of more interactions by students having the dashboard available. The more options a digital system provides, the more interactions are expected. In-depth post-hoc log-data analysis confirmed that students spent similar time with the self-assessment items; however, additionally used the dashboard for reflecting their learning progress (Roberts et al., 2017).

The second research question examined the intended use and perceived usefulness of two different versions (without dashboard vs with dashboard) of the CCT system. Contrary to the expectations based on previous research, students of the first study cohort (SC1; without dashboard) reported significantly higher intended use and perceived usefulness of the CCT system than participants of the second study cohort (SC2; with dashboard) (both differences included a small effect). While the dashboard design followed current state-of-the-art visualization features (e.g. line graphs) (Schwendimann et al., 2016), students were unable to make use of the visualization to foster their learning process. Accordingly, dashboards may require features students want for supporting their learning, e.g. personalised scaffolds or adaptive content recommendations (Schumacher and Ifenthaler, 2018).

The third research question tested for differences concerning the confidence level toward the CCT system. Interestingly, students of the second study cohort (SC2; dashboard available) reported significantly higher confidence levels of the CCT system than participants of the first study cohort (SC1; no dashboard) (small effect). In contrast to the intended use and perceived usefulness of the CCT system, the available dashboard seems to increase confidence in using the CCT system. Hence, data visualisations related to self-assessments seem to foster students’ trust in CCT systems (Pardo and Siemens, 2014).

Additional analysis focusing on differences in the psycho-educational structures of the students were tested. FSI and LMI did not reveal a statistically significant difference between the two study cohorts.

5.2 Research implications and contribution

The findings support the assumption that self-assessments and related feedback support learning processes and impact the learning performance (Adler and Benbunan-Fich, 2015; Azevedo and Bernard, 1995). Thus, students who were classified with more master subject areas outperformed students who were classified with fewer master subject areas. However, in contrast to previous findings (Jo et al., 2015), students’ characteristics did not contribute to the prediction of the student’s performance.

The reported small and medium effect sizes highlight the importance of further understanding what learners expect from learning analytics dashboards and related visualisations (Bennett and Folley, 2021; Schumacher and Ifenthaler, 2018). Visualisation techniques such as line chart, bar chart, progress bar, timeline and pie chart seem to be limited in supporting student learning (Sahin and Ifenthaler, 2021). Accordingly, inappropriate designs can negatively affect the learning processes of the learners (Bodily et al., 2018; Schwendimann et al., 2016). To design an effective dashboard design, it is necessary to establish a theoretical connection with human cognition and perception, situation awareness and visualisation technologies (Yoo et al., 2015). In addition, dashboard visualisations may include contextually appropriate presentations, visual language and social framing (Sarikaya et al., 2018). Further, affective dispositions of learners which may be triggered through dashboards and visualisations could be considered when designing future implementations, for instance, understanding the need for autonomy, relatedness or interest when designing learning analytics dashboards (Eseryel et al., 2014; Howell et al., 2018; Roberts et al., 2017). Hence, the acceptance and effectiveness of dashboards highly depend on the benefits learners may expect, including a clear, simple and fit-for-purpose design (Pokhrel and Awasthi, 2021). However, dashboard designs are expected to be unique regarding the design of the visualisation and are also expected to fit the current needs of learners when they are used (Teasley et al., 2021). While the currently implemented dashboard is being further developed, the learners expectations are being evaluated in a participatory design approach (Könings et al., 2014).

5.3 Limitations and future research

This study has several limitations that need to be addressed. The sample included a select group of participants from one university all enrolled in a specific course, thus prohibiting generalisations of results. This fact limits the external validity of our findings (Campbell and Stanley, 1963). Accordingly, future studies shall include participants within and across different subject domains and from different higher education institutions. The administered self-reporting inventories for assessing participants' dispositions are limited as they can only gather perceptions after using the CCT system and of which the learners are aware (Veenman, 2013). Still, using other methodological approaches to investigate dispositions toward the CCT system, such as think-aloud protocols influence the behavior as students might be more aware of their actions or might feel interrupted (Schraw, 2010) (Schraw, 2010). Therefore, opportunities for a combination of a multi-method approach to assess learners' dispositions seem to be a reasonable future approach (Azevedo et al., 2010). Last, while our sample was large enough to achieve statistically significant results, the explained variance and respected effect sizes were rather moderate. This indicates that besides the tested variables others may have influenced the outcomes that were not tested in this study.

Accordingly, the CCT system is being further developed for seamless integration of self-assessment and related feedback to further reveal unforeseen opportunities for future student cohorts (Saqr et al., 2017). As the next step of this research project, dynamic student-facing dashboards will be implemented that are based on assessment indicators (Bodily et al., 2018). Different dashboard designs will be tested in quasi-experimental studies that will be embedded in a productive digital learning environment. It is thought that these dashboards provide students with meaningful insights into monitoring their performance and their deficiencies or strengths. Thus, it will be possible for learners to control their learning processes and support learners’ autonomy (Carless and Boud, 2018; Matcha et al., 2020). In conclusion, the design of digitally supported learning environments requires valid formative (self-assessment) data to better support the current needs of the learner.