Original ArticleStatistically significant papers in psychiatry were cited more often than others
Introduction
An article can be regarded as having been cited if it is mentioned in the list of references of another publication. Citation bias occurs when the chances of a paper being cited by others are associated in some way with its results. For example, authors of published articles may tend to cite studies that support their position. This criterion may operate as a filter that favors some research findings and thereby introduces a preferential citation bias into their interpretation by readers [1].
Preferential citations can be based on statistical significance of primary outcome [1] or, for example, on whether an article supports the effectiveness of a new intervention, or some other fashionable hypothesis in the field [2], [3]. Some authors define citation bias as preferential citations based on nationality or discipline of the publishing journal [4]. Citation bias is a specific type of dissemination bias. The wider term “dissemination bias” includes bias due to selective publication and other publication-related biases caused by the time, type and language of publication, selective citation of references, database index bias, and biased media attention [1].
Medical research is beset by dissemination bias. Selective publication, based on statistical significance, is well documented [1], [5]. More recently, evidence of selective reporting of statistically significant outcomes in randomized trials has emerged [6]. Here, we investigate selective citation bias based on statistical significance of the primary outcome in the psychiatric literature.
It has been suggested that authors are less likely to cite studies that were nonsignificant, negative, or neutral in outcome [1], [7], [8]. Arguably, more frequent citation of positive and statistically significant findings is likely to be an intrinsic consequence of the mechanical (mis)use of “hypothesis testing” methodology [9] carrying over to favor the inclusion of studies that support the proposed hypothesis more often than those that do not. If authors covertly select papers for citation in this way, they present a distorted view of the evidence to their readers, often leaving them with an exaggerated view of the medical evidence.
There is already some evidence of citation bias in clinical trials. Kjaergard and Gluud [10] found preferential citation of hepatobiliary randomized clinical trials reporting significant differences on the primary outcome. Comparison of frequency of citation of all controlled cholesterol lowering trials using coronary heart disease or death as the end point showed that supportive trials were cited almost six times more often than others, even though meta-analysis showed lowering cholesterol concentration was not beneficial [7]. A survey identified that even a database (ACP Journal Club) summarizing findings from original studies has a preference for summarizing therapeutic trials that have a positive outcome [11]. Koren and Klein [12] demonstrate that there is bias against negative studies in newspaper reports of medical research. However, Gluud [13] notes in her review that empirical evidence concerning selective citation bias of statistically significant articles is still scarce and not so convincing.
Analysis of the relationship between the outcomes of statistical tests and the citations received are likely to face several potential confounding factors. The prestige and visibility of journals is a potential confounding factor, which should be adjusted for. Further, different subfields of psychiatry emphasize different research methods differently, and this is reflected in the fact that some psychiatric journals publish more epidemiological reports than clinical trials or biological studies. Some authors have found the number of citations received to be associated with the study design [14] and the direction of the outcome [7], or that the poor quality of methodological reporting has led to exaggerated positive findings [10], [15], [16]. A larger number of coauthors has also been shown to increase the number of received citations [17], [18], [19]. Exclusion of authors' self-citations in the citation count can, and should in our view, be used to control this effect.
Previous work in this area has been limited to intervention trials. Further it has either been based on a relatively small number of studies [7] or has adjusted for a limited set of confounders [10]. Further, self-citation has not been excluded, nor sample size or journal taken into account. Here, we investigate preferential citation in the psychiatric literature, adjusting for these and other factors, in both noninterventional and intervention studies. We focus on psychiatric research, an area in which the first author has worked for many years.
Section snippets
Set of articles
We selected four general English-language psychiatric journals: The American Journal of Psychiatry (AJP), Archives of General Psychiatry (AGP), the British Journal of Psychiatry (BJP), and Nordic Journal of Psychiatry (NJP). AJP and AGP have consistently been the top two psychiatric research journals (Garfield's impact factor [IF] 7.61 and 11.21, respectively), whereas BJP is the most visible and cited psychiatric journal outside the United States (IF 4.18) [20]. NJP (IF 0.89) represents the
Results
The statistical significance of the primary outcome of articles in the four psychiatric journals is summarized in Table 1. All the eligible articles were classified according to whether or not the authors had reported or decided that the primary outcome was statistically significant or nonsignificant. If the results of formal statistical significance testing or confidence interval estimation were not included in the reporting of the primary finding, the article was classified as not evaluating
Discussion
The purpose of this work was to investigate whether papers with statistically significant findings on the primary outcome had an increased probability of being cited by other authors. We found articles containing significant primary findings were cited more often, an association that could not solely be explained by the journal of publication or the study design. The influence of the outcome of statistical testing on the decision to refer to a particular published finding is consistent with a
Acknowledgments
This study was supported by the German Research Foundation (DFG, Deutsche Forschungsgemeinschaft, FOR534, AN 365/2-1).
References (37)
- et al.
Citation bias of hepato-biliary randomized clinical trials
J Clin Epidemiol
(2002) - et al.
A survey identified publication bias in the secondary literature
J Clin Epidemiol
(2006) - et al.
Early extreme contradictory estimates may appear in published research: the Proteus phenomenon in molecular genetics research and randomized trials
J Clin Epidemiol
(2005) Genetic associations: false or true?
Trends Mol Med
(2003)- et al.
Publication bias in clinical research
Lancet
(1991) - et al.
Publication and related biases
Health Technol Assess
(2000) - et al.
The citation bias: fad and fashion in the judgment and decision literature
Am Psychol
(1984) - et al.
Comprehensiveness and bias in reporting clinical trials. Study of reviews of pneumococcal vaccine effectiveness
Can Fam Physician
(1995) - et al.
Citation bias in medical journals
Scientometrics
(1999) - et al.
Publication decision revisited: the effect of the outcome of statistical tests on the decision to publish and vice versa
Am Stat
(1995)