Mehau Kulyk/Science Photo Library

At the Joint Colloquium of the Cochrane & Campbell Collaborations in Keystone in October 2010, we ran a workshop about the problems of detecting research misconduct,[1] and had a wonderful discussion with participants. The US Office of Research Integrity defines research misconduct as: “fabrication, falsification, or plagiarism in proposing, performing, or reviewing research, or in reporting research results; fabrication is making up data or results and recording or reporting them; falsification is manipulating research materials, equipment, or processes, or changing or omitting data or results such that the research is not accurately represented in the research record; plagiarism is the appropriation of another person's ideas, processes, results, or words without giving appropriate credit; research misconduct does not include honest error or differences of opinion”.[2] The Committee on Publication Ethics (COPE) also outlines publication and research misconduct in its flowcharts for editors, and highlights redundant (duplicate) publication, changes in authorship, undisclosed conflicts of interest, and ethical problems as additional types of misconduct.[3] Cochrane Review authors, as they analyse the entirety of primary research evidence in a specific area, are well placed to identify many of these types of research and publication misconduct. Indeed, Professor Sir Iain Chalmers urged systematic reviewers, not so long ago, to harness their unique opportunity to detect plagiarism.[4]

Misconduct can seriously distort the scientific research record, and several articles have indicated that misconduct is widespread. Fanelli et al. found that 2% of scientists surveyed admitted to having fabricated, falsified or modified data or results at least once, and up to 34% admitted other questionable research practices; in surveys asking about the behaviour of colleagues, rates were 14% for falsification, and up to 72% for other questionable research practices.[5] Other articles have also highlighted misconduct, including erroneous presentation of significant results in abstracts,[6] selective outcome reporting,[7] and other issues.[8]

Our workshop considered whether Cochrane Review authors do enough to detect and report research and publication misconduct – important tasks they may do better than anyone else. Searching The Cochrane Library in July 2010 (using the search term “data NEAR/2 falsifi* in All Text”) we found that only five Cochrane Reviews mentioned data fabrication,[9, 10, 11, 12, 13] and two reported data falsification.[14, 15] In all instances the misconduct had already been discovered by other authors and editors, mostly in journals. Only one Cochrane Review clearly stated (in a table) that a primary study was excluded because of plagiarism.[16] By contrast, redundant publication was reported in 440 of 4372 Cochrane Reviews included in Issue 9, 2010 of the Cochrane Database of Systematic Reviews. This considerable number may reflect that guidance to authors on how to handle duplicate data is described in the Cochrane Handbook; however, the Cochrane Handbook does not guide authors on how to report and highlight instances of redundant publication. Additionally, while the Cochrane Handbook highlights that authors should be aware of fraudulent studies, retracted publications, errata and comments, authors are not guided on whether and where they should report these issues when they are identified.

Risk of bias is now addressed extensively in current Cochrane Reviews, and GRADE provides a methodology for grading the quality of evidence. Both the risk of bias and GRADE encourage a considered assessment of the limitations of primary studies: however, these assessments focus primarily on identifying poor methodology rather than poor ethical integrity. Only assessment of incomplete outcome reporting and selective reporting can identify actual misconduct. Discussants at the workshop noted that experienced systematic reviewers become sceptical about studies from certain authors, sites, and even countries known for having poor research integrity records, and can be particularly cautious when appraising trials where too many participants have been recruited from a single site. When Cochrane Review authors think they have discovered misconduct, it is not clear what they should do; they have no guidance on how to report misconduct within their Cochrane Reviews, and how and when to contact editors of primary journals. In some cases authors may exclude a suspect study because it does not meet the prespecified inclusion criteria, and their concerns about the study may remain unheard.

The Cochrane Database of Systematic Reviews now has authors, editors, Editor‐in‐Chief, publisher, impact factor, and content such as Editorials, and therefore is similar in many respects to medical journals (although the author role is often partly fulfilled by editors and Cochrane Review Group staff, as staff work more closely with authors than editors of ‘traditional’ journals). Yet despite its journal status, the Cochrane Database of Systematic Reviews provides no guidance for authors via the Cochrane Handbook and Cochrane Policy Manual on reporting misconduct. By contrast, all Cochrane Review Groups are members of COPE, which provides editors with specific advice on both unresolved and proven cases of misconduct and, when necessary, on referring cases to authors' institutions and licensing bodies. Several Cochrane editors have now used the COPE forum meetings to discuss specific anonymised cases, but is COPE's advice shared with other Cochrane Review authors so that they can learn from it, and are subsequent actions reported anywhere in The Cochrane Library? And could The Cochrane Library publish notices of concern when appropriate, which might lead to the retraction of one or more primary studies if necessary?

Workshop participants agreed that it is time for the Cochrane Handbook and Cochrane Policy Manual to include guidance on detecting and reporting misconduct in primary studies within Cochrane Reviews. First of all, the Cochrane Handbook must clearly acknowledge the possibility of finding misconduct in primary studies. Secondly, it should provide examples of misconduct so that authors can clearly recognise it. Thirdly, it should advise on how and when to describe and report misconduct (for example, selective reporting is highly prevalent in primary studies but will not always amount to serious misconduct or warrant exclusion of a study from a review, and duplicate publication may sometimes be warranted). Options for reporting proven misconduct might include prominent flagging of relevant reviews; using special tables to list studies where there was misconduct; and providing a short glossary of terms for research misconduct that differentiates appropriate practices such as data transformation procedures from inappropriate data manipulation. Using the right terms would ensure consistent reporting within Cochrane Reviews and would enable reliable searches of completed Cochrane Reviews.

COPE provides algorithms which recommend graded responses to different types and degrees of misconduct. The Cochrane Handbook could provide similar algorithms about reporting, handling, and referring cases of misconduct, recommending actions including:

• Flagging up cases to Cochrane Review Group editors.

• Writing to the authors of suspect primary studies to seek explanations for apparent misconduct: asking “have I misunderstood?” This would fit well with the already‐common practices of asking authors of primary studies for further information and explanations, and for access to raw data.

• Contacting the journal(s) that published the study if the authors of a primary study do not reply or do not respond adequately.

• Informing the Editor in Chief of The Cochrane Library.

• Informing the publication arbiter.

Other options debated in the workshop included appointing a Collaboration committee of fraudbusters, creating some other Cochrane independent body to report to (anonymously?) and to judge and handle cases. Internal referrals to such a body would relieve Cochrane Reviewer authors of the associated ethical, professional, legal, and practical burdens and would guard against making malicious accusations. Referral of anonymised cases to an external body (e.g. COPE) was also discussed, as was a more radical proposal to build a ‘name and shame’ website for collating proven cases of misconduct, and contacting authors' peers in confidence to discuss potential misconduct.

We believe that Cochrane Review authors should report newly discovered and previously known cases of misconduct in primary studies, should avoid using euphemisms, and should have proper support and guidance from The Cochrane Collaboration on how to handle misconduct. Care must be taken to avoid tarnishing reputations without cause or due process, advice in the Cochrane Handbook and Cochrane Policy Manual should be ethically and legally sound, and such delicate activities require oversight from the Cochrane Editorial Unit. The Collaboration might also encourage editors or other journals to send alerts about retractions and notices of concern, so that together they can work towards clean, unbiased systematic reviews and a reliable research record.