Applications of deep language models for reflective writings

Abstract

Social sciences expose many cognitively complex, highly qualified, or fuzzy problems, whose resolution relies primarily on expert judgement rather than automated systems. One of such instances that we study in this work is a reflection analysis in the writings of student teachers. We share a hands-on experience on how these challenges can be successfully tackled in data collection for machine learning. Based on the novel deep learning architectures pre-trained for a general language understanding, we can reach an accuracy ranging from 76.56–79.37% on low-confidence samples to 97.56–100% on high confidence cases. We open-source all our resources and models, and use the models to analyse previously ungrounded hypotheses on reflection of university students. Our work provides a toolset for objective measurements of reflection in higher education writings, applicable in more than 100 other languages worldwide with a loss in accuracy measured between 0–4.2% Thanks to the outstanding accuracy of the deep models, the presented toolset allows for previously unavailable applications, such as providing semi-automated student feedback or measuring an effect of systematic changes in the educational process via the students’ response.

Appendices

A: Classifiers: technical details

In this section we describe technical specifics of training of the selected shallow and deep classifiers.

1.1 A.1: Data splits

To make sure that no sentences are present multiple times in our dataset, we start by removing the duplicate sentences regardless of the context. To minimize the chance of multiple occurrences of sentences that are close-to the same among the splits, we order the sentences alphabetically. We split the ordered list of sentences with their associated contexts in a ratio of 90:5:5 to a train, validation and test split, resulting in a total of 6,097 of training, 340 of validation and 340 of test sentences. We pick a validation set based on a confidence threshold matching the training confidence threshold, i.e. the classifier trained on sentence of minimal confidence of 4 is tuned on a validation set with a minimal threshold of 4 as well. Confidence filtering on all splits is applied only to the selected split so that it is reproducible and no samples can be exchanged between splits when testing on different confidence threshold.

1.2 A.2: Shallow classifiers

We report the tuned hyperparameters of the shallow classifiers, and report their comparative results of accuracy with their respective optimal parameters.

1.2.1 A.2.1: Preprocessing

To minimize a dimensionality of bag-of-words representation, we lowercase and remove the stemming all the words of sentence and its context. We limit the vocabulary of bag-of-words to top-n most-common words, except the stop-words occurring in more than $50\%$ of sentences. We consider n as a hyperparameter of every shallow classifier.

The final bag-of-words representation with context, used for all the shallow classifiers, is a concatenation of a bag-of-words vector of a sentence and its context.

1.2.2 Hyperparameters

We have performed an exhaustive hyperparameter search on a validation data set of the training confidence, on the following parameters of shallow classifiers:

• (Shared) use of context $\in \{\text {True}, \text {False}\}$

• (Shared) preceding and succeeding context window size $\in \{1, 2,\ldots , 5\}$,

• (Shared) BoW size: limitations to top-n tokens: number of tokens $\in \{100, 150,\ldots , 1000\}$

• (Shared) Tokenization for a specific language $\in \{True, False\}$

• Random Forrest: depth $\in \{2, 3, 4, 5\}$,

• Random Forrest: number of estimators $\in \{100, 200, \ldots , 500\}$

• SVM: kernel function $\in \{linear, polynomial, radial\}$

1.2.3 Results

Fig. 6

Per-sentence accuracy of reflection classification of the evaluated shallow classifiers, with their respective hyperparameters tuned on a validation set. Results for both Czech instance (left) and English instance (right) of the introduced CEReD dataset

Figure 6 shows results for all the evaluated classifiers with the hyperparameters listed in Section 1 tuned on a validation split.

B: Deep classifiers

We extend the description of the training process from Section 3.4.2 with technical specifics relevant for reproduction or easier further customization.

We experiment with two multilingual representatives of transformer family: Multilingual BERT-base-cased (Devlin et al., 2019), and XLM-RoBERTa-large (Conneau et al., 2020). We split the samples using the same methodology as described in 1.

We segment the units of text based on Wordpiece (Turc et al., 2019) or Sentencepiece (Kudo & Richardson, 2018) model, respectively, built for the supported languages of the particular model. We utilize a context of the classified sentences in a concatenation, separated by a special symbol “<s>”, similarly to other sequence-pair problems, such as answer extraction, or entailment classification.

By default, we train the neural classifiers using an effective batch size of 32, i.e. the weights of the models are adjusted using the gradients aggregated over the 32 training samples. We set the warmup to 10% of the total training steps and by default, we schedule the training for 20 epochs. We use early-stopping on evaluation accuracy with a patience over 500 training steps. A training of a single classifier takes approximately 8 hours on two GPUs of Nvidia Tesla T4.

For final evaluation, we pick the model with the highest validation accuracy, as measured in every 50 training steps. Figure 7 shows a comparison of validation loss and accuracy of Multilingual-BERT-base-cased and XLM-RoBERTa-large trained on samples with minimal confidence threshold of 4.

Due to the computational complexity and the fact that we find our results quite consistent to adjustments of aforementioned parameters, we do not perform a systematic hyperparameter search of parameters of deep classifiers and set these only by our best knowledge and intuition.

Fig. 7

Values of validation accuracy (left) and loss (right) during the training process, for XLM-RoBERTa-large and BERT-base-cased trained and validated on sentences with confidence over 4 propose a clear dominance of accuracy by XLM-RoBERTa model

C: Literature review of applying machine learning to reflective writing

Table 4 Overview of a literature applying machine learning to reflective writing

D: Parameters of Generalized linear mixed models

Table 5 Estimated regression parameters, standard errors, z-values and P-values for GLMMs presented