A meta-analysis of syntactic priming in language production

Highlights

• 73 papers on syntactic priming in production are meta-analyzed.

• The priming effect is robust, with a large effect of lexical overlap.

• Studies that investigate moderators of priming are underpowered.

• We do not find evidence of p-hacking in the syntactic priming literature.

Abstract

We performed an exhaustive meta-analysis of 73 peer-reviewed journal articles on syntactic priming from the seminal Bock (1986) paper through 2013. Extracting the effect size for each experiment and condition, where the effect size is the log odds ratio of the frequency of the primed structure X to the frequency of the unprimed structure Y, we found a robust effect of syntactic priming with an average weighted odds ratio of 1.67 when there is no lexical overlap and 3.26 when there is. That is, a construction X which occurs 50% of the time in the absence of priming would occur 63% if primed without lexical repetition and 77% of the time if primed with lexical repetition. The syntactic priming effect is robust across several different construction types and languages, and we found strong effects of lexical overlap on the size of the priming effect as well as interactions between lexical repetition and temporal lag and between lexical repetition and whether the priming occurred within or across languages. We also analyzed the distribution of p-values across experiments in order to estimate the average statistical power of experiments in our sample and to assess publication bias. Analyzing a subset of experiments in which the primary result of interest is whether a particular structure showed a priming effect, we did not find evidence of major p-hacking and the studies appear to have acceptable statistical power: 82%. However, analyzing a subset of experiments that focus not just on whether syntactic priming exists but on how syntactic priming is moderated by other variables (such as repetition of words in prime and target, the location of the testing room, and the memory of the speaker), we found that such studies are, on average, underpowered with estimated average power of 53%. Using a subset of 45 papers from our sample for which we received raw data, we estimated subject and item variation and give recommendations for appropriate sample size for future syntactic priming studies.

Introduction

When someone is primed with a syntactic structure X and is then asked to produce a new sentence, it is claimed that they are more likely to use that same structure X than if they had instead heard some other structure Y. This phenomenon, syntactic priming (also sometimes called structural priming or syntactic persistence), has been an important topic of study in psycholinguistics since Bock (1986). Syntactic priming has been used to test theories of event structure (Bunger, Papafragou, & Trueswell, 2013), social interaction (Branigan, Pickering, McLean, & Cleland, 2007), bilingualism (Bernolet et al., 2007, Bernolet et al., 2013, Schoonbaert et al., 2007), syntactic surprisal (Jaeger & Snider, 2013), childhood linguistic representations (Messenger, 2010), amnesia (Ferreira, Bock, Wilson, & Cohen, 2008), autism (Slocombe et al., 2013), aphasia (Verreyt et al., 2013), implicit learning (Kaschak, Kutta, & Jones, 2011), and human mating behavior (Coyle & Kaschak, 2012). Perhaps most critically, syntactic priming has been used as evidence for the abstractness of syntactic operations (Bock, 1986, Bock, 1989). See Ferreira and Bock, 2006, Heydel and Murray, 2000, Pickering and Ferreira, 2008 for critical reviews of this literature.

As a phenomenon that has become central to the field of psycholinguistics, syntactic priming is ripe for a cumulative quantitative analysis. One of the goals of this meta-analysis is to assess the current state of knowledge in the field by aggregating data and evaluating it quantitatively. All else being equal, how big is the syntactic priming effect? What is the range of variation one could expect? How much bigger should it be when there is lexical overlap between the prime and target? Can the existing literature be trusted, or does it suffer from publication bias?

While there have been several large-scale critical reviews of syntactic priming, there has not been a systematic, large-scale quantitative meta-analysis (but see Jaeger & Snider (2013) for a meta-analysis of three earlier experiments). Meta-analyses, whereby a group of studies are gathered and quantitatively analyzed together, can be useful for assessing what we have learned through a large body of distinct studies and for exploring whether these studies are exploring the same underlying phenomena (Lipsey & Wilson, 2001). Meta-analyses dramatically increase statistical power – that is, the probability of detecting a true effect – by pooling data together. In our meta-analysis, for instance, we included data from over 5000 unique participants, whereas no single experiment in our sample used more than 144. For these reasons, increased use of meta-analysis in the social sciences has been widely recommended as a way to investigate the reliability of published results (Button et al., 2013b, Cumming, 2013, Simonsohn et al., 2014b).

In this paper, we report three results: a standard meta-analysis, an analysis of publication bias, and recommendations for sample size in future priming studies. We define effect size as the log odds ratio of the proportion of target structures produced in the prime condition to the proportion of target structure produced in the no-prime condition. For 45 of the 73 papers in our sample, we obtained raw data from the authors and used it to derive estimates of effect size and standard error. From the remaining papers, we estimated the effect size and standard error using the published estimates. Along with effect sizes and their associated standard errors, we also collected information on several key manipulations that can potentially modulate the priming effect, including the construction used, lexical repetition, lag, and whether the priming is within or across languages. Using these variables, we estimated the average effect size of syntactic priming given various experimental conditions.

As a secondary analysis, we assessed the extent to which the set of papers in our study suffer from publication bias and low power. Indeed, there have been meta-analyses in other branches of psychology alleging widespread publication bias (Ioannidis et al., 2014, Landy and Goodwin, 2015), low reproducibility (Open Science Collaboration, 2015), and low statistical power (Button et al., 2013a, Button et al., 2013b). Low statistical power can lead to inflated false-positive rates in the literature and unreliable results (Gelman & Carlin, 2014). To assess publication bias and statistical power, we used p-curve, a tool developed for that purpose which works by analyzing the distribution of significant p-values in the literature (Simonsohn et al., 2014a, Simonsohn et al., 2014a). Using the raw data gathered from the study authors, we did a power analysis and give guidelines on how to run syntactic priming studies with sufficient statistical power.

In addition to quantifying the state of the field, there are a number of open questions in syntactic priming that we can investigate using this method. For instance, Pickering and Ferreira (2008) describe conflicting evidence as to just how long-lived syntactic priming is. Here, we provide evidence that, as Hartsuiker, Bernolet, Schoonbaert, Speybroeck, and Vanderelst (2008) suggest, syntactic priming decays relatively slowly but the effect of lexical overlap decays quickly. We also show that, when there is lexical overlap between the prime and target, syntactic priming is very strong in a speaker’s second language – much stronger than any observed priming within a first language. This collection of priming results, analyzed together for the first time, yields the strongest support yet to claims that priming is an abstract process largely independent of modality or task.