Introduction

The peer review system can be tracked back to 1731 in the Royal Society, with the first peer review model being used as early as in 1752 where a knowledgeable group of peers reviewed the manuscripts, and their recommendations would later influence the decision of the editor-in-chief [1]. It is considered to be the best available system and currently serves as the gold standard and gatekeeper of Science, despite several aspects of it being controversial [2].

Over the years, its process of blinding has evolved in different types which include single-blind peer review (SBPR), double-blind peer review (DBPR), triple-blind peer review (TBPR), quadruple-blind peer review (QBPR), and open peer review (OPR) (Table 1) [3, 4]. In general, it is perceived that DBPR and OPR offer higher objectivity and fairness and may increase review quality [5, 6]. However, each system has strengths and weaknesses [7], and which one is best still remains a debate.

Table 1 The different types of peer review. SBPR, single-blinded peer review; DBPR, double-blinded peer review; TBPR, triple-blinded peer review; QBPR, quadruple-blinded peer review; OPR, open peer review; ID, identity
Full size table

Another controversial issue pertains to reviewer suggestion (author-suggested reviewers (ASR)) or exclusion (non-preferred reviewers (NPR)) option by the authors during the submission process. ASR may offer reduced cost and increased speed of the review but may increase bias or introduce fraud as there been several cases of abuse with authors impersonating peer reviewers and reviewing their own work, a phenomenon called “fake peer-review”(FPR) [8, 9]. Conversely, NPR may exclude some knowledgeable, fair but strict reviewers [10].

To the authors’ knowledge, there has been no study of the aforementioned controversial issues in orthopaedic journals, and there is very limited literature on the subject looking at specific disciplines [8]. This study has the following aims: (1) establish the current peer-reviewed practices in the discipline of orthopaedic surgery and correlate them to the journal’s impact factor; (2) given those findings, provide recommendations to improve the system in orthopaedic surgery; those findings and lessons learned may also be extrapolated to other academic medicine disciplines to address this critical gap in the literature.

It is hoped that this manuscript will help contribute to addressing some of the potentially correctable flaws of our currently imperfect peer review system which definitely have an impact in its integrity, fairness, and equality that are relevant to all parties involved (editors, authors, reviewers, and readers) in all fields of medicine.

Materials and methods

The authoritative Journal Citation Reports™ (JCR) database of Clarivate Analytics was queried in March of 2022. The journal category of “orthopaedics” belonging to the science-citation index expanded was selected, and the name of the journal, its ranking, and its 2020 impact factor (IF) were extracted [11]. Furthermore, journals with submission by invitation only were excluded. For each journal, the official webpage or their partner’s webpage where the journal was hosted was thoroughly examined in order to extract information about the type of the peer review used for evaluating the submitted papers. For journals which did not have clear information or no information at all regarding their established peer review practices, the Journal was directly contacted via email prompting for a quick survey with questions regarding its peer review practices (Appendix 1—Online survey).

If no answer was received after seven days, a reminder email was sent and then at regular intervals. If no answer was received after four total attempts, the type of peer review for the journal was marked as “unclear.” All journals were categorized in one of the four peer review types presented in Table 1.

Furthermore, a sample email account was created and registered in the electronic submission system of each journal in order to access the submission process and investigate whether the journal offers/mandates do not offer the options of ASR or NPR. The maximum number of allowed ASR or NPR was also recorded, and if it was ten or more, it was recorded as “no limit.” The IF of the journals, obtained as described, was compared with the acquired data to investigate for potential correlations with the ASR or NPR options [11].

Statistical analysis

A point-biserial correlation was made for the types of peer review of the journals in correlation with the journal’s IFs. Moreover, further analysis investigated potential correlations between the ASR and NPR practices and IF. p values below 0.05 were considered statistically significant. The software used for the data analysis was IBM Statistical Package for the Social Sciences (SPSS®) version 26.

Results

Orthopaedic journals

Out of the 82 orthopaedic journals from the JCR query, four were excluded since they allowed submission only with invitation (Clinics in Podiatric Medicine and Surgery, Foot and Ankle Clinics, Hand Clinics, and Orthopaedic Clinics of North America). The remaining 78 journals as categorized in the area of “orthopaedics” by the JCR were included for the data analysis and had a mean IF of 2.672 (range 0.500–7.000) (Table 2).

Table 2 The included journals based on the Journal Citation Reports™ (JCR) database of Clarivate Analytics®. ISSN, the International Standard Serial Number; eISSN, electronic ISSN; ASR, author-suggested reviewers; NPR, non-preferred reviewers; JIF, journal impact factor
Full size table

Peer review type

Only 47 of the 78 journals (60.3%) had a clear statement of their peer review type on the website. From the remaining 31 journals queried by email, 20 responses were received (64.5% response rate), with 19 accepted responses and one invalid response, with a total of 11 journals not responding at all after four attempts. Of the 78 journals, nine journals used SBPR (11.5%) (Table 2), 52 journals used DBPR (66.7%) (Table 2), two journals used TBPR (2.6%) (Table 2), three journals used OPR (3.8%) (Table 2), and in 12 journals, this was unclear (15.4%) (Table 2, Fig. 1). None of the journals used QBPR. A point-biserial correlation was made for the types of peer review vs IFs, but there was no statistically significant correlation between any type of peer review.

Fig. 1
figure 1

Peer review blinding practices of orthopedic journals (in decreasing order)

Full size image

Reviewer options (ASR and NPR)

In 77 of 78 journals, information regarding the optional choice or mandatory requirement of ASR or NPR during the submission process was acquired.

ASR

Forty-three journals (55.1%) did not offer the ASR option, whereas 34 journals (43.6%) did, and for one journal, this was unclear. From the 34 journals that included the ASR, in 26, it was optional, whereas in eight, it was mandatory in order to proceed with the submission process (Table 2, Fig. 2). In 29 journals (85.3%), there was no limit in the number of ASR, and in four, it was limited to up to three reviewers (11.8%), whereas one journal allowed the suggestion of up to five reviewers (2.9%).

Fig. 2
figure 2

ASR practices by orthopaedic journals (in decreasing order)

Full size image

A point-biserial correlation was performed with coefficient of 0.205 (p = 0.071) for the option of ASR vs IF, but no statistically significant correlation was shown. Of the eight journals with mandatory-ASR, five required suggestions of at least two reviewers (62.5%), two required at least three reviewers (25%), and one required at least one reviewer (12.5%). There was no correlation between IF and the maximum number of ASR as well.

NPR

Of the 77 journals only 27 (34.6%) allowed the author to be able to suggest NPR during the submission process (Table 2, Fig. 3). In 24 (88.9%), there was no limit in the number of NPR, in two, the limit was up to three reviewers (7.4%), and in one, the limit was up to five reviewers (3.7%). A point-biserial correlation coefficient of 0.331 (p = 0.003) indicated a statistically significant positive correlation about offering the option to suggest non-preferred reviewers and the journals’ IF. However, no significant association was observed on point-biserial correlation between number of allowed non-preferred reviewers vs journal IF.

Fig. 3
figure 3

NPR practices by orthopaedic journals (in decreasing order)

Full size image

Discussion

Despite challenges, imperfections, and yet little proof that it works [12], the peer review system still remains the gold standard that safeguards the integrity of science [2]. Furthermore, although Dr. Lachmann quoted that “peer-review is in science what democracy is in politics, i.e. not the most efficient system, but the least corruptible” [13], there can still be unfairness, corruption, fraud, and lack of integrity. It is therefore important to have a clear picture of the peer-reviewed practices within a certain discipline for all parties involved and continuously strive to improve the ethical standards by stimulating objectivity and fairness. This study indicates that 2/3 (66%) of orthopaedic journals use the DBPR system, with the SBPR system following with 11.5%. By removing the 15.4% “unclear” component, DBPR rises up to 79%, SBPR to 13.5%, and the remaining OPR and TBPR with small percentages to 4.5% and 3% respectively. This “unclear” component is important, as potential authors and reviewers may not know if their information is shared during the peer review process. The authors of this paper recommend that all journals should have a clear indication of their peer review practices on their website and also in their electronic submission system, so that the rules of the game are disclosed to all parties, therefore ensuring better transparency.

In SBPR, the authors’ identities are revealed to the reviewers, and therefore, it has been criticized by many that it lacks fairness and impartiality. When authors that are previously well known and/or originate from prestigious institutions may receive favourable critiques, the inverse may be true for junior, less known investigators [3, 4, 14]. Additionally, further discrimination may be present secondary to ethnic origin, native language, institution, etc. [15]. We found that at least one in ten orthopaedic journals still employs SBPR, but no correlation with the IF was found. This is encouraging, as a recent study in the discipline of medical imaging showed that this was employed by 52% of the journals, and interestingly enough, journals with higher IF used this model more frequently [8]. Another study looking at dental journals showed that journals with a higher IF were most likely to have SBPR in place compared to the lower IF which was DBPR [16]. Given the obvious concerns raised above, the reasons for the persistence of SBPR are not clear. Interestingly, a 2016 study using a single orthopedic journal showed that when prestigious authors submit a manuscript, acceptance rate was 87% vs 68% in the case of SBPR vs DBPR, with higher reviewer ratings in all categories [14]. Based on the current study, the authors suggest that SBPR should be upgraded to DBPR as a first step to increase fairness of the submission process.

DBPR aims at eliminating the aforementioned biases, and this study found that it is the predominant system used by orthopaedic surgery journals (66.7%), which contrasts with some other medical disciplines [8]. Taking it to other fields, a survey of 590 chemistry journals showed that 97% did not embrace DBPR in 2007 [17], but this seems to be shifting just recently [18]. However, there are still limitations and remaining concerns with the DBPR: (1) Some studies (e.g., systematic reviews and randomized controlled trials) may have been registered in some public databases [19] or deposited in pre-print servers (e.g., medRxiv, www. https://www.medrxiv.org/) prior to peer review; (2) After removing the authors’ names and institutions, it is logistically sometimes difficult and time consuming to remove all blinding information; for example, it may be difficult to truly blind all manuscripts in a way that does not hint the reviewer, including removing discriminative self-revealing phrases such as “we have shown that” [4, 20] and also self-citations [21]. To theoretically mitigate those concerns, scrutiny by the editorial team of every manuscript demands resources, increases costs, may result in increased publication times, and is technically difficult [4]. Despite this fact, in a frequently cited survey by Ware and Monkman, 56% voted in favor of the DBPR versus 15% for the SBPR [22], and 80% of reviewers in nursing journals preferred DBPR with high levels of satisfaction [5]. Based on the above, and to ensure proper DBPR, as well as to lower administrative burden, journals should clearly state in the instructions to authors that sentences and/or hints that may potentially betray the authors’ identity should be excluded from the manuscript.

In DBPR, the editor still knows the authors’ identities, and therefore, this still carries a risk of bias. Therefore, theoretically, by automatically deidentifying this information during the submission process, additional blinding of the editor may be achieved through the so-called TBPR. The current study showed that this practice has only been implemented by two of the orthopaedic journals (Table 2) (2.6%). Possible reasons for this might be that, on one hand, this would require significant increase in the responsibilities and workload of administrative staff, and not all journals can handle that [3], and on the other hand, this might not be feasible when close communication between authors and editors is required [23]. Similar to the case of the DBPR, the authors speculate that the TBPR would require even more administrative resources. Furthermore, QBPR may be achieved by deidentifying the handling editor, to minimize the so-called “desk-rejections,” which take place as soon as the paper reaches one of the editors [24]. In this study, no orthopaedic journal has been found using this system.

In OPR, the identities of all the parties are known, and in some cases, all documentation of the entire peer review process, along with the names of reviewers, is fully available in the final published version of the manuscript [25]. Advocates of OPR often state that there is more accountability, and the reviews are increasingly constructive, have better quality, and promote a culture of “partnership” in publication [3]. Furthermore, OPR is a valuable resource for researchers who study the peer review system and also provides a better recognition of reviewer’s contribution to the manuscript [26]. However, there are concerns with OPR as far as a lower reviewer invitation acceptance rate [27] increased fear of retaliation especially for junior researchers and less critical feedback as reviewers may be reluctant to be strict [28]. Only three orthopaedic journals (3.8%) were found to practice OPR which shows a reluctancy to transition to this system (Table 2). Wolfram et al. in their 2018 study identified that there is a trend towards increasing the number of OPR journals since 2001 (174 vs 38 respectively), and it was mostly medicine and natural sciences that were adopting this model [29]. Of note, the “transparency” of OPR slightly differs between journals due to different OPR implementations. It is paramount thus that OPR journals clearly outline their open peer review process providing adequate information regarding the OPR implementation, the decision making process, and the editorial transparency [30].

There is currently a peer review crisis [2], in which 75% of journal editors reported that the hardest issue is finding reviewers to do the work [31], and that 4.70 invitations would be required for one to be accepted [32]. Therefore, the option of ASR to resolve those logistics would be compelling. However, this may come at a potentially enormous cost. Firstly, the mandatory option of ASR may raise some ethical questions as the authors are coerced to interfere with the peer review process and that would be a violation of their rights [9]. Second, the acceptance rate is higher, as authors may suggest friends and colleagues who would provide more favourable reviews, therefore compromising fairness [33, 34]. Last, but not least, FPR may account for at least 15% of retractions since 2012 [35], and only some few examples include Springer nature retraction of 107 articles from a single journal, 64 articles throughout ten journals [36], and SAGE retraction of 60 papers from a single journal [37]. Furthermore, Wang et al. reported that FPR was the most frequent cause for retraction in open access journals with the highest number of retractions and especially in journals with impact factor (IF) less than two [38]. The true magnitude of this problem cannot be calculated as it is impossible to detect all the cases, which poses a serious credibility issue to science [35]. Therefore, the editorial practice of asking (or even mandating) ASR during the submission process may compromise the journal, affecting its quality and, theoretically, its impact factor in the long term. This study found that 43.5% of the journals allowed ASR, and from these journals, 23.5% had this option as a mandatory requirement. We speculate that some of these journals may have a smaller pool of reviewers, and therefore, it is hard for them to find reviewers; however, this is an alarming finding, as the validity of the peer review system in orthopedic surgery may be thus compromised. To help eliminate bias, the authors recommend to eliminate the ASR option and, at a minimum, have the reviewers explicitly state their relationship(s) with the authors and vice versa (e.g., past mentors or past co-authors).

Noteworthy, this study also found that 34.6% of the journals allowed NPR options for the submitting authors. The option of having NPR correlated positively with the IF of the journals. Although it seems to have noble motives in eliminating reviewers with potential conflicts of interest, it can also be used unethically to increase the chances of publication, by deliberately rejecting specific knowledgeable reviewers that are known for their strict judgment and would have reviewed the paper in a fair but negative manner [10]. In turn, reviewers have an ethical obligation to alert the editor of their potential conflicts of interest and turn down the invitation to review [39]. Given the above findings, we recommend that the authors submitting a manuscript should justify in detail their NPR choice(s), and editors should only accept them based on solid grounds (even by querying the NPR in cases of doubt).

This study has some limitations: (1) The criterion for including an orthopaedic journal was its participation in the ISI Web of Science Journal Citation Reports which provides the journal information and official IF. The non-indexed journals were not evaluated. Nevertheless, this study gives a good idea of the baseline practices of those journals indexed in this prestigious database which could also be used to monitor any future changes. (2) Although correlations with the IF were made, and the journals were listed in decreasing IF ranking in Table 2, it is well known that the IF should not be looked at as an absolute measure of a journal’s quality [40]. (3) This study used information publicly available in the journals’ websites and sent survey-emails to the editors of journals with no online available information; no other means were considered to contact the non-responders (eg. telephone or social media). (4) This study answered quantitatively how many orthopaedic journals are using DBPR; however, it did not answer the question as to how effectively DBPR is achieved as alluded to previously. (5) This study did not compare orthopaedic surgery with other disciplines, because of scarcity in the literature. All three of these last limitations may be the objective of future studies.

Conclusion

This study addresses serious issues in peer-reviewed practices in academic medicine that are not yet known for individual disciplines, let alone the fact that they have not yet been resolved. By investigating the paradigm of orthopaedic surgery, several alarming challenges are identified. Although currently two thirds of orthopaedic journals use DBPR, still one in ten uses less objective SBPR, and about four in ten allow ASR all of which may introduce bias and increase the potential for fraud. Unexpectedly, in 15% of cases, the model used is unclear, and the authors and reviewers may thus be unaware of the “rules of the game.” These findings may vary among disciplines, e.g., medical imaging [8]; however, there is significant gap in the literature. It is urgent to conduct similar research in the future in other disciplines to discover, compare and contrast their similarities and discrepancies, and apply the lessons learnt to help ensure maximal fairness and objectivity across the board in academic medicine.