Original Article
Statistically significant papers in psychiatry were cited more often than others

https://doi.org/10.1016/j.jclinepi.2006.11.014Get rights and content

Abstract

Objective

Citations by other researchers are important in the dissemination of research findings. We aimed to investigate whether preferential citation of statistically significant articles exists in the psychiatric literature.

Study Design and Settings

We analyzed all original research papers published in 1996 in four psychiatric journals. Using a standardized questionnaire, from each paper, we extracted the primary outcome and its statistical significance. The number of citations, excluding authors' “self-citations,” received by April 2005 was obtained. Regression analysis was used to relate citation frequency to statistical significance, adjusting for confounders.

Results

Of 448 extracted papers, 368 used statistical significance testing and 287 (77.8%) reported P < 0.05. The median number of citations for papers reporting “significant” and “nonsignificant” results was 33 vs. 16. After adjustment for journal, study design, reporting quality, whether outcome confirmed previous findings and study size, the ratio of the number of citations per article for articles reporting “P < 0.05” on the primary outcome to those reporting “P > 0.05” was 1.63 (95% confidence interval [CI]: 1.32, 2.02, P < 0.001).

Conclusion

Authors cite studies based on their P-value rather than intrinsic scientific merit. This practice skews the research evidence. Systematic study registration and inclusion in meta-analysis should be encouraged.

Introduction

An article can be regarded as having been cited if it is mentioned in the list of references of another publication. Citation bias occurs when the chances of a paper being cited by others are associated in some way with its results. For example, authors of published articles may tend to cite studies that support their position. This criterion may operate as a filter that favors some research findings and thereby introduces a preferential citation bias into their interpretation by readers [1].

Preferential citations can be based on statistical significance of primary outcome [1] or, for example, on whether an article supports the effectiveness of a new intervention, or some other fashionable hypothesis in the field [2], [3]. Some authors define citation bias as preferential citations based on nationality or discipline of the publishing journal [4]. Citation bias is a specific type of dissemination bias. The wider term “dissemination bias” includes bias due to selective publication and other publication-related biases caused by the time, type and language of publication, selective citation of references, database index bias, and biased media attention [1].

Medical research is beset by dissemination bias. Selective publication, based on statistical significance, is well documented [1], [5]. More recently, evidence of selective reporting of statistically significant outcomes in randomized trials has emerged [6]. Here, we investigate selective citation bias based on statistical significance of the primary outcome in the psychiatric literature.

It has been suggested that authors are less likely to cite studies that were nonsignificant, negative, or neutral in outcome [1], [7], [8]. Arguably, more frequent citation of positive and statistically significant findings is likely to be an intrinsic consequence of the mechanical (mis)use of “hypothesis testing” methodology [9] carrying over to favor the inclusion of studies that support the proposed hypothesis more often than those that do not. If authors covertly select papers for citation in this way, they present a distorted view of the evidence to their readers, often leaving them with an exaggerated view of the medical evidence.

There is already some evidence of citation bias in clinical trials. Kjaergard and Gluud [10] found preferential citation of hepatobiliary randomized clinical trials reporting significant differences on the primary outcome. Comparison of frequency of citation of all controlled cholesterol lowering trials using coronary heart disease or death as the end point showed that supportive trials were cited almost six times more often than others, even though meta-analysis showed lowering cholesterol concentration was not beneficial [7]. A survey identified that even a database (ACP Journal Club) summarizing findings from original studies has a preference for summarizing therapeutic trials that have a positive outcome [11]. Koren and Klein [12] demonstrate that there is bias against negative studies in newspaper reports of medical research. However, Gluud [13] notes in her review that empirical evidence concerning selective citation bias of statistically significant articles is still scarce and not so convincing.

Analysis of the relationship between the outcomes of statistical tests and the citations received are likely to face several potential confounding factors. The prestige and visibility of journals is a potential confounding factor, which should be adjusted for. Further, different subfields of psychiatry emphasize different research methods differently, and this is reflected in the fact that some psychiatric journals publish more epidemiological reports than clinical trials or biological studies. Some authors have found the number of citations received to be associated with the study design [14] and the direction of the outcome [7], or that the poor quality of methodological reporting has led to exaggerated positive findings [10], [15], [16]. A larger number of coauthors has also been shown to increase the number of received citations [17], [18], [19]. Exclusion of authors' self-citations in the citation count can, and should in our view, be used to control this effect.

Previous work in this area has been limited to intervention trials. Further it has either been based on a relatively small number of studies [7] or has adjusted for a limited set of confounders [10]. Further, self-citation has not been excluded, nor sample size or journal taken into account. Here, we investigate preferential citation in the psychiatric literature, adjusting for these and other factors, in both noninterventional and intervention studies. We focus on psychiatric research, an area in which the first author has worked for many years.

Section snippets

Set of articles

We selected four general English-language psychiatric journals: The American Journal of Psychiatry (AJP), Archives of General Psychiatry (AGP), the British Journal of Psychiatry (BJP), and Nordic Journal of Psychiatry (NJP). AJP and AGP have consistently been the top two psychiatric research journals (Garfield's impact factor [IF] 7.61 and 11.21, respectively), whereas BJP is the most visible and cited psychiatric journal outside the United States (IF 4.18) [20]. NJP (IF 0.89) represents the

Results

The statistical significance of the primary outcome of articles in the four psychiatric journals is summarized in Table 1. All the eligible articles were classified according to whether or not the authors had reported or decided that the primary outcome was statistically significant or nonsignificant. If the results of formal statistical significance testing or confidence interval estimation were not included in the reporting of the primary finding, the article was classified as not evaluating

Discussion

The purpose of this work was to investigate whether papers with statistically significant findings on the primary outcome had an increased probability of being cited by other authors. We found articles containing significant primary findings were cited more often, an association that could not solely be explained by the journal of publication or the study design. The influence of the outcome of statistical testing on the decision to refer to a particular published finding is consistent with a

Acknowledgments

This study was supported by the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft, FOR534, AN 365/2-1).

References (37)

  • A.W. Chan et al.

    Empirical evidence for selective reporting of outcomes in randomized trials: comparison of protocols to published articles

    JAMA

    (2004)
  • U. Ravnskov

    Cholesterol lowering trials in coronary heart disease: frequency of citation and outcome

    BMJ

    (1992)
  • P.C. Gotzsche

    Reference bias in reports of drug trials

    BMJ

    (1987)
  • D.G. Altman

    Statistics in medical journals: developments in the 1980s

    Stat Med

    (1991)
  • G. Koren et al.

    Bias against negative studies in newspaper reports of medical research

    JAMA

    (1991)
  • L.L. Gluud

    Bias in clinical intervention research

    Am J Epidemiol

    (2006)
  • J. Miettunen et al.

    The effect of statistical methods and study reporting characteristics on the number of citations: a study of four general psychiatric journals

    Scientometrics

    (2003)
  • K.F. Schulz et al.

    Empirical evidence of bias. Dimensions of methodological quality associated with estimates of treatment effects in controlled trials

    JAMA

    (1995)
  • Cited by (0)

    View full text