Variables are valuable: making a case for deductive modeling

1.1 Statistical models as images

“All models are wrong, but some are useful” is a common aphorism in statistics, famously formulated by George Box (1979). In his view, models can only provide useful or illuminating “approximations”. We might say they are like the pipe in Magritte’s work The Treachery of Images (1929). It cannot be packed, lighted, or smoked; it is not a real pipe, only its visual representation (see Figure 1). Similarly, a statistical model offers us a concise abstraction, a singular representation or picture of the linguistic phenomenon under study.^[2]

Figure 1:

Magritte’s La trahison des images ‘The treachery of images’ (1929), Los Angeles County Museum of Art.²

Accordingly, we should bear in mind that a statistical model is not the reality; it is just an (abstract) image of the reality that we study. However, this image should highlight relevant aspects to help us understand the phenomenon of interest. We might then conceive of a statistical method as the camera that will allow us to take a picture of the reality we are interested in. The model will take the picture for us, but what the picture shows will depend on the camera’s settings, as well as on other circumstances that surround the “reality” and might affect the image indirectly (e.g., natural lighting; noise in the data). The photographer’s/researcher’s task is to find an appropriate configuration of the camera/model, given the purpose and conditions at hand.

1.2 Variable selection: adjusting the exposure level in our cameras

A researcher will have an overall goal regarding their data, such as pursuing a “confirmatory” or “exploratory” line of investigation, and in many contexts it is essential to state the approach taken (cf. Agresti 2002: 212; Vasishth and Nicenboim 2016). We think that it also helps to consider, for each individual variable, why it is part of the investigation. In multifactorial research settings, different objectives may lead us to include a variable in our study:

1. Exploration: We want to explore the influence of a variable, but have no particular expectation about whether or in what way that influence will show.

2. Confirmation: Previous findings or theory suggest an effect for a variable; we want to test whether the effect is present in our dataset.

3. Estimation: We expect a certain effect (from previous findings or theory) and want to evaluate its magnitude in our dataset.

4. Arbitration: Previous findings or theories suggest different expectations regarding the effect of a variable; we want to test which of them our data supports.

In settings with multiple input (i.e., independent) variables, our line-up will include predictors of different types. As we proceed to statistical modeling, the way in which a procedure treats a variable is blind to the researcher’s original objective – the model does not distinguish between cases (i) to (iv). For example, if we follow through with an automatized variable selection procedure, we end up treating every variable on a par and as dispensable if so judged by statistical criteria (which should only apply to objectives (i) and (ii), if at all). This means we put aside whatever expectations we may have had about the variable on theoretical grounds. Yet, when we discuss our results, the expectations and theoretical objectives may be crucial as the basis of our interpretation of the results. We will argue that these expectations and objectives should also be relevant to the procedure of data analysis. This means that research aims can and should guide every step of data analysis and, if need be, have priority over automatic processes.

When using a regression model to account for linguistic variation, researchers have to adjust a number of settings when they run an analysis (or “take a picture” of the reality/phenomenon in focus). Deciding on the variables that must be included in the model is the first and possibly the most important adjustment required. To continue with the camera metaphor, we can use automatic settings or make manual adjustments. We might then think of variable selection as the “exposure level” of regression models, i.e., how much light do we need in order to get the shot that comes closest to a reasonable and informative image of the reality? Likewise, when fitting a regression model, a sensible recruitment of variables is required, and the question arises which out of a set of candidate variables are to be part of the model in the end. This is “a question with no definite answer” (Baayen 2013: 347). Variable selection thus has been described as a balancing act “between high within-dataset accuracy on the one hand, and high predictive accuracy for new data on the other” (Johnson 2008: 90). Speelman (2014: 529) adds the warning that an overly detailed model runs the risk of “overfitting to the ‘noise’ in the data”, while a reduced one may lead to “misreading or oversimplifying the patterns in the data”.

Textbooks on statistics for Linguistics typically suggest – or at least describe – an Occam’s razor approach to model selection, namely the idea that among competing hypotheses, the one with the fewest assumptions should be preferred (cf. Upton 2017: 90). This translates into preferring a model with fewer coefficients – to “get as good a fit as possible with a minimum of predictive variables” (Johnson 2008: 90) – and often implies following an automatic elimination procedure in order to reach a “minimal adequate model” that ultimately drops the variables that do not make a significant contribution to explaining the variance in the dependent variable. There are two popular strategies towards model minimalism: a backward elimination procedure starts with a maximal specification of the statistical model, one that includes all predictors. It then works its way backwards to a parsimonious constellation. Forward selection, on the other hand, builds up towards this minimal specification step-by-step, by adding predictors to the focal model until the minimal adequate set is reached. Forward and backward procedures can also be combined (bidirectional model selection). These techniques feature prominently in the quantitative linguist’s bookshelf. For instance, Johnson (2008: 90) suggests stepwise forward variable selection; Baayen (2008) mentions sequential ANOVA tests as a forward stepwise procedure (2008: 199), as well as backwards elimination (2008: 205), but does not explain variable selection techniques in detail. Hosmer et al. (2013: 90–93) promote variable selection and minimal adequate models, although not by purely statistical means but by a process of “purposeful variable selection” that involves seven different steps. Gries (2013) also suggests (stepwise) variable selection on the basis of Occam’s razor (2013: 285), although he does warn that “automatic model selection processes can be dangerous: different algorithms can result in very different results” (2013: 292). Levshina (2015) mentions the option of “retain[ing] all theoretically relevant factors in the model” (2015: 149) and is cautious about stepwise selection procedures (2015: 152); similar words of caution come from Agresti (2002: 214), who states that “algorithmic selection procedures are no substitute for careful thought in guiding the formulation of models”. However, most current research papers on linguistic variation that use regression modeling seem to implicitly or explicitly follow a “minimal adequate” modeling approach.

An “alternative” to significance-based variable selection is deductive modeling or “effect estimation”, as suggested by Harrell (2015: 98), for instance: “[b]y effect estimation is meant point and interval estimation of differences in properties of the responses between two or more settings of some predictors”. This method implies pre-selecting variables of interest on the basis of theory and previous research and then reading them off the model as it is generated. Thus, a deductive statistical model includes all theoretically relevant factors and provides information on the effect of each factor, especially the direction and size of the effect. Further reduction of the model is not desirable in this approach, since eliminating a factor from the model would be to eliminate the information on its effect. This means that studies using this approach do not aim for a parsimonious model, or as such for predictive accuracy. Instead, they explicitly assess relevant factors of a variation. They follow Agresti’s (2002) advice:

It is sensible to include a variable that is central to the purposes of the study and report its estimated effect even if it is not statistically significant. Keeping it in the model may help reduce bias in estimated effects of other predictors and may make it possible to compare results with other studies. (Agresti 2002: 214)

We believe that deductive modeling is in keeping with the aims of many contemporary linguistic studies: it is a theoretically informed strategy that takes background knowledge and research objectives into account, and it requires the researcher to be clear about their aims and expectations.

Drawing on a random selection of some of our favorite linguistic research papers, we observe that most apply some form of significance-based variable selection and they are more or less explicit about it. Studies such as Lohmann (2011), Fonteyn and Van de Pol (2016), Kaatari (2016) or Hilpert and Saavedra (2020), for instance, report minimal adequate models but do not offer much detail on the procedure followed for variable selection. Some of them do provide model comparison between the minimal and a “saturated” (Kaatari 2016: 548) or full model, including all variables, by means of the Akaike Information Criterion (AIC). Other papers, such as Wolk et al. (2013), Rosemeyer (2016), Pijpops and Speelman (2017) or Levshina (2016) are more precise about their procedure. Using mixed-effects logistic regression, Wolk et al. (2013) explain their backward selection method:

[f]irst, we constructed models containing all predictors and all putatively relevant interactions. These models were then reduced by removing predictors and interactions that did not have reliable effects, and the new models were compared to the fuller ones by means of the Akaike Information Criterion. (Wolk et al. 2013: 16)

Rosemeyer (2016: 19) states the use of a backward fitting process based on ANOVA and reports C and AIC scores to show that no explanatory power (in the statistical sense) is lost by excluding variables. Pijpops and Speelman (2017: 227) differentiate between “hypothesis-driven” and “nuisance” variables, thus stating the objectives for the variables (where “nuisance” roughly corresponds to “exploration” above). After running a bidirectional stepwise selection, all their hypothesis-driven variables make it into the final model – however, this selection procedure does not pay heed to the initial distinction between variables, and if the outcome had been different (i.e., one or more hypothesis-driven variables dropped), the researchers could not have reported all the relevant results. Last, taking a Bayesian approach, Levshina (2016: 253) reports a model that “contains all 17 variables of interest as fixed effects” – so essentially a deductive model – and notes that a “more parsimonious model with only those predictors whose 95% credible intervals do not include zero […] reveals highly similar results”.

Overall, we see that most cowboys – especially those with training and experience in the use of statistical methods – are aware of the fact that their cameras need adjustment, and that the automatic filter is to be used with caution. However, linguists who are new to statistical methods might be misled into thinking that generating minimal models is the one (and only) standard procedure that grants validity and reliability. Some misconceptions might also arise that minimal modeling provides some kind of advantage when dealing with unruly data (e.g., multicollinearity, scarcity of data points, empty cells in the analysis). As pointed out by Harrell (2015),

[f]or reasons of developing a concise model or because of a fear of collinearity or of a false belief that it is not legitimate to include “insignificant” regression coefficients when presenting results to the intended audience, stepwise variable selection is very commonly employed. Variable selection is used when the analyst is faced with a series of potential predictors but does not have (or use) the necessary subject matter knowledge to enable her to prespecify the “important” variables to include in the model. (Harrell 2015: 67)

Importantly, “important” refers to theoretical, not statistical, importance here. This importance motivates the inclusion of a variable (except perhaps for purely exploratory variables about which no “subject matter knowledge” is present). With minimal modeling, important information, which was originally in our data, might be left out of a model due to the categorical character of variable selection: a predictor either stays in the model or is dropped. Modeled effects, however, are gradual – coefficients, standard errors, t-/p-values are all continuous measures – so a categorical in/out decision may cause us to lose relevant detail. Moreover, it can be a form of cherry-picking by statistical significance, as Harrell (2015: 63) explains: “Variable selection is an example where the analysis is systematically tilted in one’s favor by directly selecting variables on the basis of p-values of interest, and all elements of the final result (including regression coefficients and p-values) are biased”. This bias can lead to overrating the statistical significance of the effects that remain in the model (cf. Heinze et al. 2018: 435).

Minimal models also hinder comparability: when a model is applied to new data, the two results may be different after the variable selection process. Because models are inherently dependent on the observed data, two minimal models including the same candidate variables but run on separate datasets may contain different variables on the basis of their contribution to explaining the variance in the dependent variable. In the camera metaphor, variable selection might make it less straightforward to look back at the two pictures to compare them and find differences as regards perspectives or shades of color. Assessing such differences can be part of a research design or of a research cycle to create cumulative knowledge (cf. Schmidt 1996). In this paper, we argue that a pre-defined set of variables might be useful when these variables are tested on different dependent variables and then compared (e.g., different loci of phonetic reduction in the same morphophonological unit).

1.3 Deductive versus predictive modeling: a matter of research design

There seems to be little awareness in the (linguistic) research community of the distinction between “explanatory” and “predictive” modeling (cf. Shmueli 2010). On the one hand, explanatory modeling applies “statistical models to data for testing causal hypotheses about theoretical constructs”. On the other hand, predictive modeling applies “a statistical model or data mining algorithm to data for the purpose of predicting new or future observations” (Shmueli 2010: 291). These definitions of explanatory and predictive modeling refer to the purpose of modeling (not the method). The kind of research setting we address here, empirical research based on cognitive or social theories of language, can be considered “theory-heavy” (Shmueli 2010: 290). The questions asked in this setting typically call for an explanatory modeling approach, as they aim to test or refine theoretical constructs.^[3] This is why it is central to distinguish between these two different dimensions (explanatory vs. predictive) in order to build a model that best “fits” the data as well as the purpose: we should consider that “[e]xplanatory power and predictive accuracy are different qualities; a model will possess some level of each” (Shmueli 2010: 305). One purpose of the present paper is to highlight the distinction between predictive and explanatory modeling and how they lead to a minimal or deductive use of variables, respectively.

As an initial step, both approaches require some kind of pre-selection of candidate variables. In the research setting described above, this will ideally be based on subject matter knowledge (theoretical and empirical literature, domain expertise). Then, a predictive approach is interested only in information that will reliably “predict” an outcome, typically producing a “minimal adequate” model.^[4] Thus, predictive modeling entails that some of the variables we or others have hypothesized to have an effect on a given outcome might simply disappear because we are interested in forecasting future results rather than in comprehensively accounting for the existent effects. The explanatory approach, however, requires a careful formulation of expectations as regards the effects of given variables (in relation to their status as in Section 1.2.), and then to estimate them as precisely as possible. This is achieved by a deductive modeling strategy that does not exclude “inefficient” variables. The fact that prediction implies economy of predictors should not be used to argue that deductive models are “uneconomic”, undiscerning, or unreliable. It is the researcher’s responsibility to craft deductive models that allow for a useful interpretation of effects in a dataset. There are ways to attain this, and there are advantages to this approach.

In what follows, we show the ramifications of both the minimal and deductive approaches to regression modeling by considering their applications to data from a previous corpus study. This case study illustrates some advantages of deductive models: they rely explicitly on prior knowledge, they are responsive to our linguistic objectives, and they allow for estimation and comparison across datasets (in this case, different loci of variation).

2.1 Deductive models versus backward variable selection

We begin with the model for fricative devoicing ([v] versus [f]) in have to. With the deductive approach, we include the five independent variables above. The resulting full model is presented in Table 1.^[8]

Trained in “star-gazing” (McElreath 2016: 167), our attention will turn to the rightmost column to scan for significant effects. But richer information is gained from the coefficients and their standard errors. These provide the effect estimation that the model is designed for. Coefficients are odds ratios on the logarithmic scale, which is not the most intuitive of concepts, but their interpretation is straightforward (see, e.g., Jaccard 2001: 7–8 for a concise explanation of log odds ratios). Positive values indicate positive effects (in this case, on the probability of devoicing), negative values indicate negative effects; large values indicate large effects, small values indicate small effects.^[9] The standard error (S.E.) shows the precision with which the effect is measured in the data (the smaller S.E., the more precise). p-values and significance thresholds, on the other hand, do not inform us of the direction, size, and precision of the effect. An effect may be “significant” because it is large, or because it is precise, or both.

In estimating effects based on coefficients and standard errors, we acknowledge what the data at hand reveal (the coefficient) and apprehend the degree of uncertainty (the standard error) that stems from the fact that the data are only a sample. Standard errors often reflect the amount of data that an effect is measured on. In Table 1, the largest S.E. is for situation=public, because there are only 22 tokens of this factor level; the observed effect is in the expected direction (less devoicing, i.e., more canonical forms, in public speeches), but given the low number of data points, the estimate comes with a high degree of uncertainty.

As we now take the “parsimonious modeling” approach, we will reduce the model in Table 1 to a “minimal adequate model” by statistical variable selection. There are several methods and tools available for this (see Levshina 2015: 149–152 for an overview). We will use stepwise backward variable selection with the R functions fastbw() (package rms [Harrell 2017]) and step() (based on drop1(), from the package stats, which comes with the basic installation of R). Stepwise backward variable selection methods start from the full model and take out the least predictive variable; this is repeated until the resulting model would be significantly “worse” (in terms of predictive power) than the previous one. Both fastbw() and step() apply the Akaike Information Criterion (AIC) for model comparison, though step() tends to be more conservative than fastbw(). The resulting minimal models are shown in Table 2 (step()) and Table 3 (fastbw()).

A first essential insight is that there is not the one “minimal adequate model”. The different selection functions produce different results. Model 2 retains speech rate along with stress accent, Model 3 drops it. Having to choose between models, the researchers are forced to make a decision that they may have hoped to leave to automatized statistical procedures. We might resolve this by a principled decision for the “most minimal” (Model 3) or for the smallest AIC (Model 2), or – ideally – by smallest AIC and ANOVA comparison (which in this case suggests that Model 2 is a significantly better fit). Yet, the case serves to show that trying to obtain the most “objective” result by running a variable selection function is deceptive – human decisions are still involved.

Comparing the full model (Model 1) and Model 2, we see that Model 2 has a lower AIC and almost equal C, so, if we are aiming for parsimony, Model 2 wins. We would probably even report the same basic findings from either model, such as a higher probability of devoicing in slow speech and with heavy stress accent. This is reassuring, as it shows that results – at least when they are clear enough – will not be turned on their head by changing one methodological decision. However, the deductive full model and a focus on coefficients and standard errors allow us to quantify and assess “significant” and “non-significant” effects alike, rather than merely reporting them as present or absent.

Moreover, there is the issue of comparability: the effect of a factor can be compared across cases when models with the same variables are applied. We will do this by looking at the same factors for the realization of /t/ (full or lenited) and of the final vowel ([ʊ] or [ə]) in have to. As above, we present for each variation the full model and “minimal adequate models” as produced by drop1()/step() and fastbw().

Overall, the results in Table 4 are in line with what we would expect in a case of articulatory reduction, such that reduction is disfavored in formal situations, and before pauses as well as (slightly) in slow speech – while the effect of stress accent cannot be ascertained. If we glance over the models, we find, again, that the estimates of the reported effects do not differ much between the full and minimal models. The difference is in what the minimal models leave out; most strikingly, fastbw() drops the variable situation, which produces a significant effect in the other models (/t/-lenition is less likely in professional situations compared to private conversations). So the choice of a variable selection method would lead to either stating that the situational context is a determinant of /t/-lenition or that it has “no effect” – two opposite conclusions, drawn from the same dataset. What we rather want is to quantify the effects and assess their reliability. We see from the full model that the effect of situation=professional matches our expectation (as casual situations favor reduction) with a fair degree of certainty; the effect of situation=public actually shows the expected direction but cannot be reliably ascertained as it is only a very slight trend on a small number of observations. This is the information we ask for in an “effect estimation” scenario.

In fact, when situation is taken out, the estimates for other effects also change – speech rate has a larger coefficient, and to the strict “star-gazer” it may look as though following sound=other has “lost” one star. These changes are not dramatic when considered with care, but the danger is real: “it may happen, that after eliminating a potential confounder another adjustment variable’s coefficients moves closer to zero, changing from ‘significant’ to ‘nonsignificant’ and hence leading to the elimination of that variable in a later step” (Heinze and Dunkler 2017: 8).

Regarding the variation in the final vowel (Table 5), the variable selection functions both retain only the factor Following sound, and the only significant effect is for the labial/dental level (which disfavors reduction to schwa). This is also quite clear from the full model. We might be tempted to say that this time variable selection really only clears the model from uninformative clutter. But let’s recall that the aim of variable selection – at least theoretically – is predictive modeling, that is, constructing a model that can optimally predict new data (and does not overestimate spurious effects in the present data). A measure for predictive power is the C-index, and with C = 0.611, the minimal model in Table 5 is clearly not good at predicting.^[10] (Neither is the full model, of course.) This corresponds to the different inherent purposes of the approaches: The full model allows us to estimate the effects of a set of pre-selected variables, and in this case we find that most of them show no reliable association with the dependent variable; with the “minimal adequate model” we try to make predictions about the probability of variants, and in this case we have to admit defeat – the minimal model is not adequate.

2.2 Comparability across models and visualization

We have stated that one motivation for keeping all the (theoretically relevant) variables in our models is their direct comparability across variations. In the present case, we can compare the effect of each factor on fricative devoicing, /t/-lenition and final vowel reduction in have to. We illustrate this by visualizing the coefficients and standard errors of the three full models (Figure 2).^[11]

Figure 2:

Comparison of effects: Coefficients and standard errors of the full models.

Figure 2 contains all the information needed to interpret the variations at hand. Effects are plotted on the x-axis: Negative effects show to the left of the zero line, positive ones to the right; the distance from zero indicates the size of the effect, the error bars its degree of uncertainty. While each color/shape represents the model for one variation, the effects are grouped by factor (level). With this, we can compare how the effects differ between variations in direction and size. For example, we can identify factors that show a clear effect on variation but not the others (e.g., situation=professional), as well as those which barely have any effect at all (e.g., following sound=velar); and we can observe that effects on fricative devoicing (red dots) and /t/-lenition (green triangles) tend to pull in opposite directions (e.g., for following sound=other, speech rate, stress accent). This is of interest as it shows that there is a variant “haf to/hafta” as opposed to an articulatory reduced form “havda”.

Reading off the effects from a coefficient plot like Figure 2 is convenient for a general comparison. For more detail, they can be visualized as in Figure 3, where the estimated effects of speech rate are plotted against the dependent variables.^[12]

Figure 3:

Effects of speech rate on fricative devoicing, /t/-lenition and final vowel reduction in have to.

While higher speech rates increase the odds for /t/-lenition (middle panel), fricative devoicing shows the opposite effect (less devoicing in faster speech, left panel); and there is a slight tendency towards final vowel reduction in rapid speech (right panel), an effect we might have expected but that is weak (and non-significant) in the model. Also, the rate of /t/-lenition overall is lower than that of fricative devoicing or final vowel reduction, which is apparent in the positions of the regression lines. Again, this direct comparison is only possible if the variable in question (here, speech rate) is included in each model, and it is most reliable when each model includes exactly the same set of independent variables. These conditions are met with pre-specified, “deductive” models but not with “minimal adequate” models.

What if we tried comparisons like those in Figures 2 and 3 with models that have undergone statistical variable selection? To illustrate the difference, we will show this with the more radically trimmed-down models, those suggested by fastbw(). The effects plot, showing the coefficients and standard errors from the three models, is presented in Figure 4; the effects can be read in the same way as in Figure 2. As for variable selection, we can only note the presence and absence of effects in Figure 4 (e.g., no effects on fricative devoicing for following sound, or on the final vowel for speech rate and stress accent); the variables situation and year of birth are not even shown, as they have been eliminated from all three models. Thus, instead of showing an accurate comparison, we have turned some effects into a yes/no question – we have given up information for the sake of parsimony. Even if we wanted to show only “significant” effects, this is not achieved, as the coefficient of an individual factor level can still be close to zero (e.g., following sound=velar).

Figure 4:

Coefficients and standard errors of the minimal models.

The logic of eliminating variables is based on null hypothesis testing. The observed data do not allow us to reject a null hypothesis, for example that there is no effect of situation on /t/-lenition, with sufficient certainty. We end up excluding the variable, and make no statement about its effect – not even a statement on its effect size being rather too small or its variance too large due to small token numbers. As Cumming (2012: 8) notes, “[s]uch dichotomous decision making seems likely to prompt dichotomous thinking, which is a tendency to see the world in an either-or way” (emphasis in original). A variable either contributes to explaining the distribution in the dependent variable, or not. What we should promote instead (according to Cumming 2012, and we agree) is estimation thinking, i.e., a focus on effect sizes. The question then is not “yes/no” but “how much”. Again, it is the full models that have this very focus.

On the other hand, with the minimal models of Figure 4, we are probably inclined to make some “how much” statements too, e.g., that the effect of speech rate on /t/-lenition is quite large. Apart from the potential issue of accuracy (cf. Harrell 2015: 69–70), this means that we have taken two steps: First, we test variables on a “yes/no”-question (does it make a significant contribution to the model?); then, if the answer is “yes”, we ask “how much” (how large and in what direction is its effect?). It seems that the first step is unnecessary. Compare this to the full models (as seen in Figure 2), where “how much” is all we ask. The answer for some effects may be “practically zero”, which is tantamount to “no effect”; but it still is an answer that can be specified (by coefficient and standard error) and compared to other effect sizes.