The promise and pitfall of automated text-scaling techniques for the analysis of jurisprudential change

Abstract

I consider the potential of eight text-scaling methods for the analysis of jurisprudential change. I use a small corpus of well-documented German Federal Constitutional Court opinions on European integration to compare the machine-generated scores to scholarly accounts of the case law and legal expert ratings. Naive Bayes, Word2Vec, Correspondence Analysis and Latent Semantic Analysis appear to perform well. Less convincing are the performance of Wordscores, ML Affinity and lexicon-based sentiment analysis. While both the high-dimensionality of judicial texts and the validation of computer-based jurisprudential estimates pose major methodological challenges, I conclude that automated text-scaling methods hold out great promise for legal research.

Appendix

1.1 Interpreting unsupervised models

Surely, one can use the word parameter estimates of a Wordfish model to interpret and validate the dimension being scaled. Figure 9 depicts the $\psi$ and $\beta$ values of all words appearing in the corpus. Words with high $\psi$ value, like “Europa”, are words that appear in similar proportion across documents. Words with non-zero $\beta$ value are those that effectively discriminate among the documents (here negative values can be interpreted as associated with Euroscepticism and positive values with integration-friendliness). I highlighted some terms, which can be related to the frames emerging from Figs. 3 and 8. However, even if the results feel right–as they would seem here–it could that the second largest or third largest dimension is, in fact, the relevant one. But the researcher will not know unless she scales these dimensions too. Something Wordfish does not allow.

As explained in Sect. 3, the dimensions generated by an unsupervised model can be explored and interpreted by inspecting the words associated with these dimensions. Figure 12 shows how words likeee “Volk”, “Souveränität”, “Nationalstaat” (nation state), “Demokratie”, “Ultraviresakt” (ultra vires action), “Vorlagepflicht” (duty to request a preliminary reference) and even “Griechenland” (Greece) to see how they relate to dimension 2 from CA. Here two distinct frames emerge from the analysis, namely an integration-friendly frame associated with positive values on the y-axis and a Eurosceptic, state-centred frame associated with negative values on the y-axis. This, again, is consistent with what scholars say about the German Constitutional Court’s rhetoric (Thym 2009; Stein 2011; Weiler 1995; Calliess 2012; Tomuschat 2010; Möllers 2011).

Fig. 12

Correspondence analysis: argumentation frames in German constitutional rulings on European integration on dimension 2

Figure 13 depicts the $\psi$ and $\beta$ values of the Wordfish model for all words appearing in the corpus. Words with high $\psi$ value, like “Europa”, are words that appear in similar proportion across documents. Words with non-zero $\beta$ value are those that effectively discriminate among the documents. Here negative values can be interpreted as associated with Euroscepticism and positive values with integration-friendliness. The plot—notably the words “terrorismus” and datei”—suggests that Wordfish collapses the first and second dimension of CA into a single dimension.

Fig. 13

Wordfish: $\psi$ and $\beta$ values of terms in German constitutional rulings on European integration

1.2 Word embeddings

See Table 2

Table 2 Top 25 words by cosine similarity of the word vector “verfassungsidentität” for pre-trained Word2Vec embeddings

1.3 Validation

See Fig. 14.

Fig. 14

Correlation of computer-generated estimates in pairwise comparisons with mean value of expert ratings ($N=153$). Plot shows mean estimate of Spearman correlation with 95% confidence interval. Results for supervised methods exclude the two training opinions, Solange I and Solange II. Except for Wordscores, ML Affinity and Dictionary, all models are significant at $p<0.01$