3.1.1 Target data: Teacher and student performance.

We use student grades and SET scores as two different target features in our analysis, each one of which represents a study in itself with various models using features described in Section 3.1.5.

In the first analysis, which focuses on student grades, we infer associations with student performance during COVID-19 lockdown in the spring of 2020. We obtain grades for all courses with exams from 1 March to 31 August 2020. Grades are given based on the Danish grading scale, from lowest to highest: -3, 00, 02 (lowest passing grade), 4, 7, 10, and 12. The passing grades correspond to grades in the European Credit Transfer System from E to A. In addition, some courses only assign the non-numeric grades of pass and fail. We remove all pass/fail data points and only include data points for students who received a pass since more students appear to have taken exams and submitted them blank in 2020 than in 2019, while more student failed in 2019 because of not showing up. As this may be related to the online format of exams, we exclude data points with failing grades. As the grading values are meant to be distributed normally (as confirmed by the histogram and quantile-quantile plot—see S1 and S2 Figs), we do not process the target further.

For the second analysis, we use the SET score on a five-point Likert scale as the target, which we treat as a numeric variable, even though it cannot be assumed to be normally distributed (as the histogram and quantile-quantile plot in S3 and S4 Figs show). The validity of our approach is supported by Derrick and White [28] and Norman [29] and appears to be the best method, since data is not uniformly distributed and therefore not suitable for a multinominal logistic regression.

Both grade and SET scores were in 2020 on comparable levels with 2019.

3.1.2 Teacher survey data and teacher population.

The first author conducted the survey of CBS teachers in June 2020. Distributed using the survey software Qualtrics, the survey was designed to evaluate the first semester affected by COVID-19 to examine the challenges teachers faced and to identify any potential for future blended learning strategies. Most of the questions were based on a five-point Likert scale, though some provided answer categories for questions like “What tools do you plan to use next semester?”. A few of the questions permitted free-text comments, but this study does not address those.

The survey questions were divided into two parts: personal questions for the teacher and then question about the specific courses they taught. The former contained questions about:

• The teacher’s home workstation and surrounding environment, e.g., working position, stillness, and appearance or absence of children.
• The teacher’s technical background, e.g., online teaching experience, IT skills, and attitude toward online teaching.
• Any didactic changes or adaptions, and tech use in general during the semester.

The latter had questions about:

• The teacher’s assessment of their own workload in each of their courses and whether their courses were successful.
• The teacher’s assessment of, e.g., students and exams for each of their courses.

The survey was given to 837 teachers at CBS, including core and adjunct faculty who taught the spring semester of 2020. Of these, 328 completed the first part of the survey on teacher background information, while 244 completed the second part. Table 1 shows the teacher demographics of those who answered the second part compared to the total population of teachers who taught in the spring semester of 2020.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Demographics of the teachers.

https://doi.org/10.1371/journal.pone.0291689.t001

The population of teachers who completed the part of the survey on courses is significantly different (according to a Student’s t-test) from the background population regarding age and number of courses taught.

The teachers (n = 244) who completed the part of the survey on courses represent all of the 11 departments at CBS and their response rate ranged from 21–42%, which indicates that each department is fairly represented. This is also the case for types of faculty, where the response rate was 17–50% for all faculty, with the lowest percentage among part-time faculty. The population comprised 33 nationalities, primarily Danish (n = 153), German (n = 29), and Swedish (n = 6). There were 90 teachers (37% of respondents) of a different nationality than Danish who responded (unreported (n = 1)). For comparison, 30% of all teachers who taught the spring semester of 2020 were not Danish, which we designate as international.

Figs 2 and 3 illustrate the distribution of answers to personal and the course questions, though with “Do not know” responses removed. S1 Table contains all questions.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 2. Teachers’ survey, personal questions.

Answers to background-related questions of the teacher survey. The questions with answers on the Likert scale are designed such that the higher value, the better. See also the full teacher survey description in S1 Table.

https://doi.org/10.1371/journal.pone.0291689.g002

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 3. Teachers’ survey, course questions.

Answers to course-related questions of teacher survey. The questions with answers on the Likert scale are designed such that the higher value, the better. See also the full teacher survey description in S1 Table.

https://doi.org/10.1371/journal.pone.0291689.g003

Due to data pre-processing, as described in Section 3.1.5, we narrow the teacher population down further for each of our two model targets. The dataset for the grade target includes data from 162 unique teachers, while the dataset for the SET score target includes 114 unique teachers. The target populations resemble the general population with respect to the teachers’ age and the number of courses they taught. For gender and nationality (whether teachers are Danish or not), there are no overall significant differences between the background population and the two target datasets. S2 and S3 Tables present a comparison of the teacher demographics between the two datasets and the population of all teachers who taught in the spring semester of 2020.

3.1.3 Student data population.

As of May 2020, there were 14,137 students enrolled at CBS, 49% females, 25% internationals, and their average age was 24.6 years.

For student grade performance, we only included students who received a passing numerical grade on an ordinary exam. Once other data cleaning was performed (Section 3.1.5), 6,702 unique students remained in the grade dataset and had relatively similar demographics compared to the entire population of students enrolled in the spring semester of 2020 (S4 Table). However, the population of the grade dataset is significantly different (according to a Student’s t-test) from the background population regarding nationality, age, and program level. Most master’s students who enrolled in 2018 were in the midst of writing their thesis in the spring semester of 2020 and therefore did not have any course grades that semester. This was also the case for bachelor’s students who enrolled in 2017, although most of them were taking courses while writing their bachelor thesis.

For SET scores, we have fewer data as only a fraction of students evaluated their teachers. Once other data cleaning was performed (Section 3.1.5), only 1,581 unique students remained in the SET score dataset (compared to 6,702 in the grade dataset). S5 Table presents their distribution relative to the background dataset.

Overall, the population for the SET score is significantly different from the background population regarding age, gender, and program level.

3.1.4 Course data.

From the course catalogue we obtain information about whether exams contained a group assignment, i.e., whether the students were required to perform group work (written or oral). Moreover, we obtain information about changes to exam formats made during the lockdown, such as from oral to written, or from a written exam on campus to a written exam at home.

3.1.5 Data pre-processing and final feature set.

Substantial data pre-processing is needed for the analysis and modeling performed in this paper. First, the data is transformed so that each data point concerns a student-teacher-course relationship. This is done to investigate the effect of teacher and course features on student grade performance and on SET scores.

While data pre-processing is almost always needed, the fact that we used data from several systems require us to undertake numerous transformations, filtering, and joins. Moreover, most of the data is not collected purposefully for modeling in this paper, which is why transformation and feature engineering are necessary to create data of a format suited for our analyses. Finally, due to the aim of our analysis, part of the data was filtered out since we deemed it irrelevant to our analysis. S1 Text provides a complete list of the data transformation, feature engineering, and filtering of data.

The features of the analysis are based on both the teacher survey data and administrative data about teachers, students, and courses (see Fig 1). Table 2 lists the features in full.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 2. List of features used for the models.

https://doi.org/10.1371/journal.pone.0291689.t002

First, there are administrative features concerning courses: required group work and changes in exam formats. Second, the administrative data also includes demographic measures such as gender, age, position (for teachers), department (for teachers), nationality, and program level (for students), some of which are mainly control variables. For instance, departments may have a variety of circumstances that are beyond the scope of our analysis. And, third, there is teacher assessments, which are explained thematically in Section 3.1.2.

We want to explore whether demographic measures or program level influenced student performance or whether it was a combination of both. Likewise, we want to examine teacher backgrounds more broadly, such as their technical skills, experience with online teaching, attitudes toward online teaching, and how these influenced student and teacher performance. We also use these features to potentially reveal various reasons for how a particular course was assessed since teachers may have used differing online mitigation strategies, depending on the course.

3.2.1 Linear regression models.

For both targets, we use the lm function in the statistical programming language R to fit the multiple linear regression models using the ordinary least square method [30]. Table 2 lists the features we used. We do not include any interactions between these features in the linear regression models. The resulting regression estimates are scaled by dividing by two standard deviations [31].

The scaled regression estimates, together with their estimation uncertainty, are visualized using dot-and-whisker plots [32], where the dots represent the regression estimate and the whiskers span the 99.95% confidence interval. The significant regression estimates provides an idea of which features have a direct (linear) effect on our targets.

3.2.2 Random forest regression models.

Due to the complexity of the domain, it is unlikely that all effects are linear. Thus, to investigate the potential nonlinearity and interaction effects, we also train random forest models [33] on the same data. There are several advantages to using random forest models compared to other advanced machine learning models. First, they allow fairly interpretable output in the form of feature importance. Moreover, contrary to linear regression, random forest models can detect non-linear relationships and are less sensitive to confounders. Finally, they can detect interaction effects between several features.

We train the random forest models in R [30] using the ranger package [34], which is a rapid implementation of random forest models in accordance with Breiman [33]. We run the random forest model with 1000 trees and perform hyperparameter tuning of how many features to sample at each split (called mtry in ranger) and the minimum node size of the final trees. We use the tidymodels [35] framework to perform this hyperparameter tuning with a grid search and 10-fold cross validation.

We use the random forest models to extract feature importance values (representing direct effects) to compare with the multiple linear regression estimates. For feature importance, we use impurity importance from the ranger package since it measures importance as a decrease in impurity, where impurity is measured as the variance of the target [34]. That is, the high importance of a feature means that it has been used in numerous tree splits to create a high decrease in impurity, which intuitively signals the high importance of the feature for predicting the target. The significance of the feature importance is also calculated in the form of p-values using the ranger package (based on a permutation method originally suggested by [36]) and 6000 permutations.

To detect which features interact with other features to create effects on the targets in our random forest models, we use the R package iml [37], which provide multiple model agnostic tools for explainable artificial intelligence. More specifically, for our two random forest models, we use Friedman and Popescu’s H-statistics [38] and partial dependence plots [39].

For each of the random forest models, we first calculate the total H-statistics for all features, which indicate whether and to what extent a feature interacts in the model with all the other features. Next, we select the features with the highest total H-statistics and calculate all two-way H-statistics for the selected features, which indicate any other features they interact with the most. Again, we select the seemingly most dominant two-way interactions and visualize them as partial dependence plots [40].

Since calculations of the total and two-way H-statistics are computationally heavy, sampling is used, which in turn introduce uncertainty. As a result, we run 10 to 25 different samples using cloud computing resources for each calculation of H-statistics to get a more robust result.

3.2.3 Modeling strategy and evaluation.

In contrast to classical statistical modeling, we adopt a hypothesis-free approach, as is common in machine learning and also adopted by others, such as [41]. In this approach we train predictive machine learning models without any prior hypothesis of what might have an effect, thus providing the possibility of discovering a larger variety of associations.

For both targets (student grades and SET scores), we first fit a multiple linear regression model with the entire set of features (Table 2) on the largest dataset without any missing values. Moreover, we also fit a random forest model for both targets, using the hyper-parameter tuning strategy explained above.

We introduce GPA prior to 2020 (Historical GPAs, student) in our feature set to control for it in the models with grade as target. Thus, any associations with grade, we find, are particular to the spring semester of 2020. The same approach is used for the models with SET score as target, i.e. we introduce SET score of teachers prior to 2020 (Historical SET score, teacher). We also control for student assessments of the course in general (SET score, course) in the SET score models.

Apart from these four models (linear regression and random forest models for both grade and SET score targets), we built a collection of alternative models to validate the original choice of models. For student grade performance, we built alternative models using the following alternative input data:

• Excluding all features from the teacher survey
• Weighting of data points based on estimated teacher’s share in the course

For the SET score models we also built alternative models using the following alternative feature sets:

• Excluding the overall course evaluation feature
• Including a historical, course mean evaluation
• Excluding all features from the teacher survey

In addition, we creat a random forest classification model instead of a regression model.

We train these alternative models to see if they show any substantial improvements in performance and to test the stability of the associations in the models. Table 3 shows the predictive performance of these models compared to the original models, while S2 Text describes the associations found in the alternative models in more detail.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 3. Performance metrics for basic and alternative models.

https://doi.org/10.1371/journal.pone.0291689.t003

We evaluate the predictive performance of both the linear regression and the random forest models using r-squared (R²). We try to avoid overfitting in the random forest models by using out-of-bag samples upon which the reported R² is calculated [33]. The out-of-bag R² measure (OOB R²) provides a realistic measure, allowing us to not reduce our training set by splitting it into testing and training sets, which would be undesirable due to our sample size. However, our sample size is large enough such that overfitting is not an issue in our linear regression models.

Additionally, we use both model types, along with the most prominent interactions, when evaluating the importance of each feature. Where the linear regression model outputs polarity of the estimate and significance of the linear dependencies, the random forest model exhibits the significance of the strength of the feature’s influence through feature importance techniques on the responses/output grade and SET score, regardless of whether they are linear or not. Finally, interaction measures can point to features that might be affected heavily by interaction with other features, which also indicates the importance of a certain feature.

5.1.1 Teachers’ time use.

The first theme we discuss in our findings concerns the preferences, skills, and working style of the teachers. Our findings show that ambiguity exists in the relationship between time use and the student grades, in the sense that the time teachers spend on course is not significant in either the linear regression or random forest models. The time teachers generally spent on work has a significant association with the grade in the random forest model, but the direction is neither clearly positive or negative, and the correlation in the linear regression model is not significant. However, there is a clear negative association between sharing responsibility for caring for children at home and grades in the linear regression model, and for those who had more than 40% of the responsibility, our findings show a clear negative association in the random forest model. While the teachers who have to take a larger share of caring for their children have a tendency to give lower grades, teachers who have children actually tend to give higher grades in general. This finding can be interpreted in multiple ways, though. Maybe teachers with children were not the ones who had the hardest time delivering quality teaching or maybe they had a harder time and were more forgiving toward their students. The fact that teachers without children also face challenges might also be supported by the fact that quiet working conditions are negatively associated with student grades.

For the SET score, we see no association with the amount of time spend on work in general, which is also the case for share of responsibility for taking care of children at home. However, we find a significantly positive relationship between whether teachers spent more time on a course than usual and the SET score they receive.

This suggests that the many teachers who spend additional time on their courses actually make a difference in terms of the students perception of their courses. The fact that this does not have a significant correlation with the student grades in our model can be due to other reasons, such as the teachers who spend more time on courses also expecting more of their students. In general, we expect the relationship between the time teachers spend on preparing courses and students performance in terms of grades to be complex, especially during a pandemic lockdown, and worth further examination in general. It is reassuring that the teachers who put in an extra effort regarding teaching are also rewarded in their SET score; however, it seems partially irrelevant if student learning is not also positively affected. Finally, it worth noting that approximately 80% of the teachers answered that they spent more time on courses than usual, which potentially could introduce a spurious effect.

This time balance should be further studied in relation to the research output of the same teachers since academic staff are required to both teach and do research. Do the extra time spent on adapting teaching to online teaching have a negative impact on their research? We expect so, but this is beyond the scope of this paper and a topic for future research.

5.1.2 Teacher peer consulting.

Peer consultation among teachers is positively associated with grades in both the linear regression and random forest models, which may indicate that a collective effort supports performance and is important to promote in times of isolation, both among peer teachers and students, to improve learning. It is widely accepted in the literature that peer collaboration is important for students [41–44]; however, we do not know much about the effect of peer consulting among teachers. We suggest that peer collaboration during crises appears to be equally, if not more, important for teachers, both in terms of the performance of their students and their own well-being. While peer consulting among teachers has a positive association with the student grades, there is no significant association with the SET score, which is a bit surprising, as consulting among teachers is expected to lead to better teaching, also from the students’ point of view.

5.1.3 Technical skills, dedication, and quiet working conditions.

One somewhat counterintuitive finding is that possessing high-tech skills, a positive attitude towards online teaching, and quiet working conditions for teachers seem to have a negative association with grades, but it must be remembered that grades are not a perfect measure of performance since they are also highly teacher dependent. For example, if teachers are struggling, they may feel sorry for their students and award higher grades.

When looking at the use of tools, there is a clear correlation between experience with a certain technology/didactics and the willingness to use it in the future. Our findings show that using one’s own videos and the future intention to use one’s own videos in teaching are positively associated with student grades, but that the shortcut of using others’ videos has a negative association with grades. In addition, using other people’s videos, for example from YouTube, even has a negative association with the SET score. Thus, this shortcut is questionable as a teaching tool, even though video recordings seem to have great learning potential. A first glance at the student comments in the teaching survey data shows that students are generally satisfied with their instructors’ videos, which supports this analysis.

5.1.4 Gender and grade.

One interesting finding relates to the debate about gender and grades. Female teachers tended to give slightly higher grades than usual—an association which is significant in the random forest model (and the alternative linear regression model using weights, as is described S2 Text), though not quite significant in the linear regression model (p = 0.0035), with a significance level of 0.0005.

Female students seem to receive lower grades than male students during the COVID-19 lockdown, especially among Danish students, which is confirmed by the random forest model but not the linear model. In addition, the gender association with grades seems to be larger when exams are changed from being on campus to taking place at home. The change seems to favor the younger men (Table 4 and Fig 7). A go-to explanation at first sight would be to couple this observation with the fact that it is easier for students to cheat during a home exam than at the university, which other studies have argued [45, 46]. These studies also find that the association is gender biased, in the sense that men cheat more often. This could potentially have an effect on student grades in our model. However, the fact that our analyses showed a negative linear association with student grades when written exams were moved from on campus to at home may indicate that cheating did not occur (successfully) very often.

Previous studies have shown that, under normal conditions, the skills of females are underrated [47, 48]. The present study showed that the gender bias was amplified during the COVID-19 lockdown in terms of grades. On the other hand, we found no significant change in the SET score of female teachers compared to male teachers during the lockdown.

5.1.5 Do students at bachelor level perform better?

Bachelor students received a very slightly higher grade, as did students enrolled after 2018. Although newer students are more vulnerable to, for example, sudden changes in teaching because they are less used to study at an institution of higher education, they either managed to overcome the challenges satisfactorily or the teachers may have felt sorry for them, as suggested above.

In terms of students’ age, we find that their grades peak around the age of 22, with lower grades shown for both younger and older students on average. However, it is likely not age, as such, that causes the differences in performance, but the fact that age often is a proxy for other aspects. At age 22, students may have reached a certain level of maturity but have not taken a long sabbatical or changed degree programs. On the other hand, more complex dynamics may also explain this. For instance, we see that student age is one of the features that interacts most with other features (Table 4). Among younger students, men seem to perform better than women (Fig 7).

Student age also have a non-linear association with SET scores, as students who were 25 years old give the best SET scores, while younger and older students rate their teachers with a lower SET score. Most 25-year-old students are in a master’s program and potentially receive more dedicated and specialized teaching that may result in them being more forgiving, resulting in higher SET scores. Section 5.2 discusses the importance of non-linear relationships and interaction in more detail.

5.1.6 Teacher assessment of students compared to normal.

When teachers were asked to look at student performance compared to normal, we see a clear signal in the SET score model, in the sense that teachers who assess their students as performing much better than normal also receive higher SET scores. It seems natural that students who impress their teachers are also treated in a positive way that also causes them to evaluate their teacher more positively. Interestingly, however, if the teacher assessed the students as performing better than usual, the students did not get higher grades. We have no good explanation for this.

5.1.7 Administrative data versus survey data.

In our basic models, we include both administrative data and survey data derived from many different questions. We also introduce alternative models, some which only use administrative data. Interestingly, the alternative model for which only administrative data is used does not perform much worse than the full model (Table 3).

However, we see that several interesting and meaningful features of the survey do indeed have a significant impact, such as teacher assessment of the course and students, as well as their own skills and their attitudes toward online teaching. Thus, we find it justified and valuable to add additional survey data of this kind in this type of educational data mining and learning analytics studies.

5.2.1 Importance of non-linearity and interaction.

In general, we find it intriguing that features often used as independent variables in learning analytics, such as age of students and age of teachers, have non-linear association with both SET scores and student grades. Although it is not new that non-linearity exists in educational data mining and learning analytics [25, 26], the features that actually interact or have a non-linear association have not been discussed previously, to our knowledge. In this paper, we have tried to bridge this gap by using techniques from interpretable machine learning to investigate interactions in more detail.

Our models show that that being of average age, for both students and teachers, have an apparently positive association with both student grades and SET scores (Sections 4.2 and 5.1.5); that is, age has a non-linear association with the two targets in our models. This aspect is clearly something that should be taken into account when analyzing student learning and teachers performance. Thus, research should incorporate models that can deal with both non-linearity and interaction to a greater extent.

5.2.2 Measuring performance and quality in education.

The success of the transformation to online teaching during the COVID-19 lockdown can be looked at from at least two perspectives: 1) How well did the students perform compared to normal? And 2) How well did the teachers perform compared to normal? This study attempted to address both questions.

Grades are used widely in the learning analytics and educational data mining literature ([20]) as a measure of student performance, and we agree overall that it often can be used as such. We also included a question in the teacher survey (Section 3.1.2) on the general assessment of students compared to normal which was aggregated at course level. The SET scores in our study can also be seen as an appropriate way to evaluate teachers’ performance in the classroom. However, a measure of performance in relation to the two questions above will, of course, always be proxies. High SET scores and grades might not always reflect the good performance of quality teaching, the quality of a course, or the learning of the students [49].

As the output shows, student features can be associated with teachers’ SET scores (though perhaps not their actual teaching quality), and teacher features can be associated with grading. In this regard our findings are interesting since the SET score and grade analyses do not seem to overlap in terms of explaining features, as seen in Section 4. Apparently, what constitutes good teacher performance and good student performance measured using these standard targets are not the same. An understanding of teaching quality needs to take both into account—at least during the COVID-19 lockdown. This approach may be generalizable and should be investigated further. Interestingly, teachers’ general assessment of students’ performance in class (compared to normal) seems to be the only feature that comes close to being significant for both targets. This could therefore be considered as a potential future measure of teaching quality for teacher surveys.

Other measures that are not included in this study but that can partly serve as proxies for the quality of education are, for example, employment among graduates and their salaries. However, these measures are difficult to compare across educational institutions, and even within the same country or over time, as jobs may be easier to find in some regions than others, just as some sectors may pay more than others. Additionally, we cannot measure them until several years after a study is over.