Open science and the future of peer-review
Science builds on itself. New knowledge is gained in the context of the enlightenment of earlier discoveries and, for the foundations to remain solid each discovery must be accurate and reliable. But science in the real world is messy, and advances haltingly and piecewise. Often, prior information is incomplete, so conclusions drawn need revising in the light of new, later evidence. This means that self-reference and self-checking are cornerstones of the scientific method; and, having reported our experiments, it is vital that readers are able to repeat them and understand how we reached our conclusions. Various studies have shown that, for a study to be successfully reproduced, information presented in the literature is often inadequate and the underlying data not readily available1 - a significant drawback. Several commentators have concluded that weaknesses in the way that research investigations are currently conducted, and how their results are disseminated via article publication have become detrimental to the scientific process2–7. As the technological sophistication of science increases, and the equipment used becomes more specialised, the data generated are harder to represent in traditional media, and reporting how experiments were performed so that independent researchers can repeat them becomes progressively harder.
Open science is a movement that seeks to ensure that the results and the data of scientific research are, and continue to be, available to all. One way in which reproducibility issues can be tackled is through the use of open-science and open-data practices8,9. As attendees of the AllBio: Open Science & Reproducibility Best Practice Workshop, we discussed how the problem of keeping science transparent and reproducible in an increasingly technology-driven, and specialised, domain could be addressed.
One route, at the heart of scientific endeavour, is through the peer-review process. Peer review is an important gatekeeper and key component of scientific discourse. Before research findings can be formally accepted, they must be evaluated and commented upon by other domain experts, who provide advice about the quality or validity of the work to journal editors and/or readers (whether via pre-publication peer review, open peer review or post-publication peer-review systems). Importantly, peer review happens at a personal, rather than institutional, level and is carried out by individuals; it is therefore an ideal mechanism for engaging most researchers, given that all scientists peer review or are peer reviewed.
Peer review oaths
The peer-review process is not infallible10–12; its weaknesses are many and varied, including the time available to perform thorough reviews, reviewers’ expertise, journals’ perception of relevance/interest/impact, and so on. Arguably, one of the most significant problems – certainly the one that generates most friction – is that reviewers can safely dispense self-serving and biased critiques, fully protected by the mask of anonymity. In some instances, anonymity may be the only pragmatic option, but in the interest of objectivity and constructive discourse it shouldn’t be the cornerstone of the peer-review process. Scientists have become sufficiently frustrated by these issues to devise ad hoc solutions to help safeguard the quality of reviews, and allow reviewers to affirm that they will review in an ethical and professional way, and encourage clearer review processes. This has led to the articulation of various reviewer’s oaths (e.g.,13–15); it is these that inspired our discussions and prompted this article.
An outcome of our debate was the formulation of an addition to these oaths that helps codify the role of reviewers, encouraging and promoting best practices to ensure that the science they review is as open and reproducible as possible. The amendment includes guidelines not only on how to review professionally, but also on how to support transparent, reproducible and responsible research, while optimising its societal impact and maximizing its visibility.
The open-science peer-review oath (which can be found on figshare for reuse16) we therefore propose is as follows (Box 1).
Box 1. Open peer review oath.
Principle 1: I will sign my name to my review
Principle 2: I will review with integrity
Principle 3: I will treat the review as a discourse with you; in particular, I will provide constructive criticism
Principle 4: I will be an ambassador for the practice of open science
Towards ‘open science friendly’ reviews
The declaration in your review
Peer-review oaths tend to be short declarations that reviewers make at the start of their written comments, typically dictating the terms by which they will conduct their reviews. The novelty in our amendment is the addition of a new declaration, which we recommend including in review preambles:
I will be an ambassador for open science
For it to be successful requires reviewers to follow the spirit of the statement throughout their reviews, which may involve a little additional work. The statement itself is derived from the rationale outlined below. It is for reviewers to decide how to implement this on a case-by-case basis – it is case specific, but nevertheless provides a framework for nurturing useful and hopefully fruitful discussions.
Guidelines for open science reviewers
Here is how we envisage the review process from an open-science reviewer's point of view:
I will work with you to help improve your research, as I believe that peer review should be an open, supportive and collaborative process. I will therefore sign my review and state my identity.
I will, before I review, ensure that I have everything (raw data, detailed methods/protocols, etc.) necessary to make a complete, independent assessment.
I will encourage the application, and, where necessary, provide guidance for any open-science best practices relevant to my field that would help support transparency, reproducibility, re-use and integrity of your research.
I will check that any data and software code are consistent with the text, that any digital object identifiers and accession numbers are correct and correctly cited, and that any models presented are archived, referenced and accessible.
I will check that the data and any software code and support documentation that underpin the published concept are made available in a manner that provides long-term unrestricted access.
I will advise, where necessary, on how to achieve better transparency and availability (in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards), with the understanding that adherence to best open-science practices may require further effort from you.
I will support others in writing open reviews, where it is appropriate for me to do so.
I will decline to review if I am not an appropriate reviewer (whether because of my expertise or because of my relationship with the author(s)). In doing so, I will provide journal editors with an honest appraisal of these issues, and will openly explain how I reached my decision so that alternative reviewers may be found.
Conclusion
We believe that combining the open-science principle with some of the other key peer-review principles will help the scientific community to repeat published experiments, and to reach the same, or similar, conclusions.
Author contributions
Dan Maclean, Ivo Grigorov, Michael Markie, Teresa Attwood, Konrad Förstner, Jean-Karim Heriche and Neil Chue Hong conceived and designed the oath and prepared the first draft of the manuscript. All the other authors in the working group were involved in the revision of the draft manuscript and have agreed to the final content.
Competing interests
MM is currently employed by F1000Research. His role at the journal does not include any involvement in the pre-publication editorial checks, or with the refereeing process.
Grant information
ALLBIO - Broadening the Bioinformatics Infrastructure to unicellular, animal, and plant science, Project reference: 289452, Funded under: FP7-KBBE. We would also like to thank The Genome Analysis Centre (TGAC, Norwich, UK) and the Biotechnology and Biological Sciences Research Council (BBSRC, UK).
Acknowledgements
We would like to thank The Genome Analysis Centre (TGAC, Norwich, UK) for organising and hosting the workshop.
We would also like to thank Peter Murray Rust for looking over the preprint of the manuscript (https://zenodo.org/record/12273) and contributing an additional principle to the oath.
References
- 1.
Collins FS, Tabak LA:
Policy: NIH plans to enhance reproducibility.
Nature.
2014; 505(7485): 612–3. PubMed Abstract
| Publisher Full Text
| Free Full Text
- 2.
Ioannidis JP:
Why most published research findings are false.
PLoS Med.
2005; 2(8): e124. PubMed Abstract
| Publisher Full Text
| Free Full Text
- 3.
Ioannidis JP, Allison DB, Ball CA, et al.:
Repeatability of published microarray gene expression analyses.
Nat Genet.
2009; 41(2): 149–55. PubMed Abstract
| Publisher Full Text
- 4.
Begley GC, Ioannidis JP:
Reproducibility in Science: Improving the Standard for Basic and Preclinical Research.
Circ Res.
2015; 116(1): 116–126. PubMed Abstract
| Publisher Full Text
- 5.
Prinz F, Schlange T, Asadullah K:
Believe it or not: how much can we rely on published data on potential drug targets?
Nat Rev Drug Discov.
2011; 10(9): 712. PubMed Abstract
| Publisher Full Text
- 6.
Hines WC, Su Y, Kuhn I, et al.:
Sorting out the FACS: a devil in the details.
Cell Rep.
2014; 6(5): 779–81. PubMed Abstract
| Publisher Full Text
- 7.
Vasilevsky NA, Brush MH, Paddock H, et al.:
On the reproducibility of science: unique identification of research resources in the biomedical literature.
PeerJ.
2013; 1: e148. PubMed Abstract
| Publisher Full Text
- 8.
Molloy JC:
The Open Knowledge Foundation: open data means better science.
PLoS Biol.
2011; 9(12): e1001195. PubMed Abstract
| Publisher Full Text
| Free Full Text
- 9.
Pereira S, Gibbs RA, McGuire AL:
Open access data sharing in genomic research.
Genes (Basel).
2014; 5(3): 739–747. PubMed Abstract
| Publisher Full Text
| Free Full Text
- 10.
Patel J:
Why training and specialization is needed for peer review: a case study of peer review for randomized controlled trials.
BMC Med.
2014; 12(1): 128. PubMed Abstract
- 11.
Glen AS:
A New “Golden Rule” for Peer Review?
Bull Ecol Soc Am.
2014; 95(4): 431–434. Publisher Full Text
- 12.
Siler K, Lee K, Bero L:
Measuring the effectiveness of scientific gatekeeping.
Proc Natl Acad Sci U S A.
2014; pii: 201418218. PubMed Abstract
| Publisher Full Text
- 13.
Watson M:
The reviewer’s oath. 2013. Reference Source
- 14.
Alexander S:
The Peer Reviewer’s Oath. 2014. Reference Source
- 15.
Verger A:
My Reviewer Oath. 2014. Reference Source
- 16.
MacLean D, Aleksic J, Attwood TK, et al.:
Open Science Peer Review Oath.
figshare.
2014. Publisher Full Text
Comments on this article Comments (5)
Passive review:
Follows all of Principles 1 through 3, and elements x - xii of Principle 4. Here, the idea of an open review basically is a review that is signed to make sure the process has appropriate integrity.
Active review:
Includes all elements of Passive review, as well as items xiii and xiv of Principle 4:
- xiii) I will check that the data, software code and digital object identifiers are correct, and the models presented are archived, referenced, and accessible
- xiv) I will comment on how well you have achieved transparency, in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards, so that your experiments can be repeated independently
The idea of an open review here seems to be that, in addition to ensuring integrity in the review process, there is also extra work being done beyond a standard review, and the person who does such work should be credited for doing so.I am uncertain about where elements xv and xvi, as written:
- xv) I will encourage deposition with long-term unrestricted access to the data that underpin the published concept, towards transparency and re-use;
- xvi) I will encourage central long-term unrestricted access to any software code and support documentation that underpin the published concept, both for reproducibility of results and software availability
would fit. These are neither active nor passive, and as written, they don't match the review function, but are even more active, more a collaboration than a review. I suggest that they be rephrased as:- xv) I will check that the data that underpin the published concept are made available in a manner that provides long-term unrestricted access, towards transparency and re-use;
- xvi) I will check that any software code and support documentation that underpin the published concept are made available in a manner that provides long-term unrestricted access, both for reproducibility of results and software availability
so that they could be part of an Active review.Of course, this specific remedy is just a suggestion, but I the overall point I want to make is that the added work to be done by the reviewer beyond what is now standard needs to be explicitly considered in both the oath itself as well as the description of the oath.
Passive review:
Follows all of Principles 1 through 3, and elements x - xii of Principle 4. Here, the idea of an open review basically is a review that is signed to make sure the process has appropriate integrity.
Active review:
Includes all elements of Passive review, as well as items xiii and xiv of Principle 4:
- xiii) I will check that the data, software code and digital object identifiers are correct, and the models presented are archived, referenced, and accessible
- xiv) I will comment on how well you have achieved transparency, in terms of materials and methodology, data and code access, versioning, algorithms, software parameters and standards, so that your experiments can be repeated independently
The idea of an open review here seems to be that, in addition to ensuring integrity in the review process, there is also extra work being done beyond a standard review, and the person who does such work should be credited for doing so.I am uncertain about where elements xv and xvi, as written:
- xv) I will encourage deposition with long-term unrestricted access to the data that underpin the published concept, towards transparency and re-use;
- xvi) I will encourage central long-term unrestricted access to any software code and support documentation that underpin the published concept, both for reproducibility of results and software availability
would fit. These are neither active nor passive, and as written, they don't match the review function, but are even more active, more a collaboration than a review. I suggest that they be rephrased as:- xv) I will check that the data that underpin the published concept are made available in a manner that provides long-term unrestricted access, towards transparency and re-use;
- xvi) I will check that any software code and support documentation that underpin the published concept are made available in a manner that provides long-term unrestricted access, both for reproducibility of results and software availability
so that they could be part of an Active review.Of course, this specific remedy is just a suggestion, but I the overall point I want to make is that the added work to be done by the reviewer beyond what is now standard needs to be explicitly considered in both the oath itself as well as the description of the oath.