Figures
Abstract
Background
Comprehensive monitoring of the quality of systematic reviews (SRs) and meta-analyses (MAs) of endoscopic ultrasound (EUS) requires complete and accurate reporting and methodology.
Objective
To assess the reporting and methodological quality of SRs/MAs on EUS diagnosis and to explore the potential factors influencing articles’ quality.
Methods
The quality of the reporting and methodology was evaluated in relation to the adherence of papers to the PRISMA checklist and the AMSTAR quality scale. The total scores for every criterion and for every article on the two standards were calculated. Data were evaluated and analyzed using SPSS17.0 and RevMan 5.1 in terms of publication time, category of reviews, category of journals, and funding resource.
Results
A total of 72 SRs/MAs was included, but no Cochrane Systematic Reviews (CSRs) were obtained. The number of SRs/MAs ranged from 1 in 1998 to 15 in 2013; 88.1% used the QUADAS tool; the average overall scores by PRISMA statement and AMSTAR tool were 19.9 and 5.4, respectively. Scores on some items showed substantial improvement after publication of PRISMA and AMSTAR. However, no reviews followed the criterion of protocol and registration, and only 11.1% of articles fulfilled the criterion of literature search. SRs/MAs from the Science Citation Index (SCI) were of better quality than non-SCI studies. Funding resource made no difference to quality. Regression analysis showed that time of publication and inclusion in the SCI were significantly correlated with total scores on the two standards.
Conclusion
The reporting and methodological quality of SRs/MAs on EUS diagnosis has improved measurably since PRISMA and AMSTAR checklists released. It is hoped that CSR in this field will be produced. Literature searching and protocol criteria, as well as QUADAS-2 tool need to be addressed more in the future. Time of publication and SCI relate more to the overall quality of SRs/MAs than does funding resource.
Citation: Liu D, Jin J, Tian J, Yang K (2015) Quality Assessment and Factor Analysis of Systematic Reviews and Meta-Analyses of Endoscopic Ultrasound Diagnosis. PLoS ONE 10(4): e0120911. https://doi.org/10.1371/journal.pone.0120911
Academic Editor: Jakob Pietschnig, Universitat Wien, AUSTRIA
Received: June 22, 2014; Accepted: February 9, 2015; Published: April 23, 2015
Copyright: © 2015 Liu et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited
Data Availability: All relevant data are within the paper and its Supporting Information files.
Funding: The authors have no support or funding to report.
Competing interests: The authors have declared that no competing interests exist.
Introduction
Endoscopic ultrasound (EUS) is a rapidly developing technique, which fully takes advantage of both endoscopy and ultrasonography. Over the last two decades, it has been widely applied in initial diagnosis and effective therapy in a wide spectrum of patients [1]. Compared with other diagnostic tests such as CT, MRI and ERCP, EUS provides an excellent complement in preoperative diagnosis and staging of tumors especially in pancreatic malignancy [2,3]. Moreover, on its foundation, fine-needle aspiration (FNA) has been widely used in the diagnosis of tumor tissues [3].
In the age of evidence-based medicine, systematic reviews(SRs)/meta-analyses(MAs) were developing rapidly and recognized as a higher level of evidence than other research types for issues of evidence-based medicine [4], particularly in the Cochrane systematic review (CSR), which was famous for its rigorous methodology [5,6]. An increasing number of SRs/MAs had emerged to assist clinicians in making evidence-based decisions, but clinicians and investigators lacked confidence to use the evidence especially systematic review of diagnostic test accuracy [7].
With the rapid development of advanced techniques, a number of standards were produced one after another, attempting to solve problems of research quality in recent years [8–10]. It is known that inadequate reporting can lead to false credence and biased results, thus misleading clinicians. [11]. Consequently, PRISMA (Preferred Reporting Items for Systematic Review and Meta-Analyses) statement was developed in 2009 after elaborative revision [8,11], providing a standardized framework to help authors report their findings in a complete and transparent manner. Ever since its release, more and more researchers have utilized it to assess SRs/MAs’ reporting quality, yet some special subtle differences were not shown in diagnostic reviews [12]. Meanwhile, the assessment of methodological quality was also a key step in relation to its reliability and validity [13]. Among all currently existing methodological quality assessment tools, AMSTAR (Assessment of Multiple Systematic Reviews) was considered a reliable and valid measurement tool [14,15]. The instrument had been developed over a decade based on empirical evidence, expert consensus and previous tools such as Overview Quality Assessment Questionnaire (OQAQ, 1991)[15]. Regarded as the best quality assessment tool of SRs/MAs by CADTH (The Canadian Agency for Drugs and Technologies in Health), it involves novel ideas in flow diagram, publication bias and language restriction [16].
Recently, the two standards have been widely accepted and utilized due to their reliability and reproducibility [17,18], but the reporting and methodological quality of SRs/MAs on EUS diagnosis was still unclear around the world. Therefore, this study aims to evaluate their qualities and explore the potential factors influencing the articles’ quality such as publication time, category of reviews, category of journals and funding resources.
Methods
Literature search
A comprehensive literature search was conducted in multiple databases including PubMed (1966~2013.7), Web of Science (1980~2013.7), The Cochrane Library (~2013.7), EMBASE.com (1974~2013.7), Chinese Biomedical Literature Database (CBM, 1978~2013.7), China National Knowledge Infrastructure (CNKI, 1994~2013.7) and Wanfang Database (1997~2013.7). Our search terms included “endoscopic ultrasound”, “endosonography”, “ultrasongraphy”,“systematic review” and “meta-analysis”. The syntaxes were adjusted corresponding to different database. Details of the search strategies were in S1 Text of Supporting information. The reference lists of all articles selected for the review were screened for potentially relevant articles not identified by the initial search. The electronic search was complemented by a hand search of related journals to ensure that all eligible studies were captured.
Inclusion criteria and study selection
To be eligible, SRs/MAs of diagnostic studies on EUS, which is defined as a technology to distinguish between patients with and without disease, were included, while conference abstracts, case reports and dissertations about the accuracy of EUS were excluded. No language was restricted.
Two investigators (Liu DL and Jin JX) independently reviewed titles and abstracts. Diagnostic reviews often reported a number of diagnostic effect index to describe review’s accuracy such as sensitivity, specificity, ±LR,±PV, DOR, AUC or accuracy and the highest frequency is “sensitivity”, “specificity” or “accuracy”. Thus, the full texts were remained to exclude the reviews without mentioning the term “accuracy” or “sensitivity” or “specificity”. Disagreements were resolved by discussion.
Quality assessment
The PRISMA statement with a 27-point checklist has been a reliable tool commonly used to evaluate the overall reporting quality of SRs/MAs [9,16]. To assess the degree of compliance, every item was rated as “yes” for total compliance, “unclear” for partial compliance or”no” for non-compliance, corresponding to the score values of ‘1’, ‘0.5’ or ‘0’, respectively. In addition to reporting criteria, AMSTAR tool, as a generic methodological standard, has been used to evaluate the quality of research. By contrast with global assessment, AMSTAR has been well received with a good practicability [13]. It consisted of 11 points and the terms of “yes”, “no”, “can’t answer” and “not applicable” were used to assess these criteria. Then the total scores for every criterion and for every article on the two standards were calculated.
Data extraction and data analysis
Data was extracted and listed in predesigned table, including publication time, study type, category of target disorder, first author country, category of journals and funding resource. Considering that the endorsement of PRISMA resulted in increases of reporting and methodological quality [19], we had an interest in investigating instruction for authors of included journals to evaluate the endorsement of PRISMA.
Two investigators (Liu DL and Jin JX) then independently extracted data in terms of the above items. A third investigator (Tian JH) was responsible for adjudicating any disagreements until a consensus was reached. As PRISMA statement was released in 2009 and AMSTAR tool in 2007, articles were divided into groups by the publication time. CSR has gone through burgeoning development due to rigorous management and robust registration platform [5,17]. The distinctiveness of Science Citation Index (SCI) in category of journals and funding resources may result in a significant difference. Thus, the factors of subgroup analysis were presented as follows: publication time (≥2009 vs. ≤2008 in PRISMA; ≥2007 vs. ≤2006 in AMSTAR), category of reviews (CSR vs. Non-CSR), category of journals (SCI vs. Non-SCI) and funding resources (Fund vs. Non-Fund).
According to these subgroups, each article was classified and the data was analyzed using SPSS 17.0 and RevMan 5.1, respectively. A p value ≤0.05 indicated a statistically significant difference, which was further assessed by visual inspection of a forest plot and 95% CI for the pooled quality assessment.
Results
Literature search
The original search yielded 481 results (Fig 1) and 291 studies were retained through duplicating. A further 172 studies were excluded by abstract review and the remaining 119 articles were reviewed in depth. By intensive screening of full texts, reference lists of included paper, and hand search, 13 non-diagnosis articles and 34 reviews were excluded. Finally, 72 articles were discussed in this study. See in S2 Text of Supporting information.
Characteristics of Included Studies
The 72 reviews were extracted from 41 journals and the majority of journals (36.6%, 15/41) were from the United States. The two journals of the highest frequency were Gastrointestinal Endoscopy (24.4%, 10/41), and Digestive Diseases and Sciences (14.6%, 6/41). There were 33 journals from Science Citation Index (SCI) with IF varying from 1 to 30 (the highest was JAMA, IF: 30.387 in 2013), and the IF of the majority (75.8%, 25/33) ranged from 2 to 6. After reviewing all these journals, only one-fifth (9/41) of them endorsed PRISMA in instructions for authors including PRISMA flow diagram and checklist.
Table 1 summarizes the characteristics of all SRs/MAs in the field of EUS diagnosis. Of all the 72 articles, 90.3% (65/72) were written in English and 9.7% (7/72) in Chinese. Articles from SCI journals accounted for 87.5% (63/72) and 11 (15.3%) articles had funds support mainly from the government. The diseases primarily involved pancreas and gastro-esophageal. Moreover, quality assessment tool in primary studies of diagnostic accuracy was mentioned in 58.3% (42/72) of the reviews, among which 88.1% (37/42) used the QUADAS tool, which met the demand of Cochrane Handbook 5.1, whereas STARD tool was rarely used.
As show in Fig 2, the number of publications on SRs/MAs of EUS diagnosis has been increasing over the last two decades. The first SR/MA was published in 1998, and 88.9% (64/72) articles went into publication after 2007. Fig 3 displays that publications with their authors coming from the United States and China play a prominent role in our study, but only 5 reviews are composed by cross-national authors.
Reporting Quality
After assessing the compliance of papers with the 27-item PRISMA checklist, we got an overall score of 19.9±3.5, but not a single review met all the listing criteria in PRISMA statement, with its full details given in Table 2. For more than one-third items (10/27), 80.0% articles are in compliance with the criteria, but for the item of Rationale, all articles have met the criterion. On the contrary, there were 7 items whose compliance rates are below 50.0%, which were mainly considered in risk of bias and searching. Then, the quality of the remaining items was medium between 50% and 80% in compliance with PRISMA statement.
As no CSRs were obtained in this study, category of reviews (CSR vs. non-CSR) could not be analyzed. Thus, analysis for the other three potential factors was performed carefully and the result is shown in Table 2. It can be seen that 68.0% (49/72) reviews were published after 2009. Between the two periods (≥2009 vs. ≤2008), difference is statistically significant in a number of items, including title, structured summary, risk of bias in individual in the method part, study selection and risk of bias across studies in the result part. However, no difference was found in rationale, protocol, and summary of evidence after the release of PRISMA. Apart from Table 2, the forest plot (Fig 4) also clearly displays the difference with a mean of 3.22 (95% CI: 1.59 to 4.89). In addition, Table 2 and Fig 4 jointly indicate that SRs/MAs from SCI are of higher quality than non-SCI studies. To be specific, there exists a significant difference in items such as risk of bias in individual, study selection and conclusions. Yet there was no evidence suggesting a difference in the last item “funding of studies”.
Methodological Quality
Compliance with the AMSTAR checklist, the score of all SRs/MAs was mere 5.4±2.2. Most articles (88.9%, 64/72) were published after 2007, the release time of AMSTAR. Table 3 displays that the compliance rate with a certain item in the checklist ranges from 1.4% to 90.3%, and nearly half (5/11) of them are beyond 50%. The optimum item was “characteristic of the included studies” the same as PRISMA. However, there were still 6 items with their compliance rates less than 50%. The worst condition occurs in the items of “preliminary design” (1.4%) and “comprehensive literature search” (11.1%).
After 2007, significant difference was found in four items including study selection and data extraction, eligibility criterion, characteristics of the included studies, and statistical methods. Generally speaking, the overall quality of articles from SCI was better than that of non-SCI papers, especially in the item of “a list of studies provided”. Fig 4 reveals a great improvement with a mean of 4.11 (95% CI: 3.12 to 5.10) after AMSTAR’s release. Similarly, funding resource had no influence on methodological quality.
Multiple linear regression analysis
To assess the independent contributions of variables in affecting the quality of reporting or methodology, two multiple linear regression models were established with backward selection procedure (αin = 0.05,αout = 0.1, model 1-PRISMA, model 2-AMSTAR) in Table 4. After assessing all potential factors by SPSS 17.0 software, funding resource is excluded from models apparently, while the other two factors are retained (p<0.05) in both models.
After controlling the factor of SCI in model 1, the final score got improved to 3.915 by publication time (95%CI: 2.444, 5.387). All other results are presented clearly in Table 4. In both models, the total score of quality assessment was greatly involved with time of publication and inclusion in the SCI. Moreover, it indicates that the standardized regression coefficient of publication time is higher than that of SCI (0.524>0.372 in model 1, 0.613>0.391 in model 2).
Discussion
Summary of evidence
In general, the number of SRs/MAs of EUS diagnosis is increasing annually. EUS is widely applied in the diagnosis of digestive system disease. The majority of SRs/MAs have been conducted in the United States and China. Although cross-national research is becoming common, only 5 of the assessed reviews of EUS diagnosis were cross-national. Of the 72 SRs/MAs assessed, 57 were from the SCI and 11 had funding resources. This study showed that the reporting and methodology of EUS diagnosis was of average quality, with overall average scores on the PRISMA and AMSTAR criteria of 19.9 and 5.4, respectively. Subgroup analysis for time of publication showed a substantial improvement in certain aspects of adherence after the publication of generic criteria, but there is still considerable room for improvement in some criteria, especially protocol and searching, which require urgent measures in order to improve.
One conclusion we can draw is that less attention has been paid to protocol and registration, because none of the reviews reported on this criterion, and this was true of reviews across the time period sampled. There were various reasons why researchers refused to be registered. Silagy’s (2002) report compared CSRs with their previously published protocols. Over 90% of studies showed major changes, revealing the limitations of protocol and registration[20,21]. However, measures have since been taken to improve this process, such as reporting the reason for changing especially the specified outcomes[22], and it is clear that publication of protocols adds scientific credibility and improves research standards[23]. Simultaneously, for non-CSR research, the PROSPERO registration platform was created to help researchers control publication and reporting bias, and to reduce unplanned duplication of SRs/MAs[24,25]. These measures may help to enhance the development of high-quality SRs.
Both PRISMA and AMSTAR statements mention the importance of a literature search, but our findings showed that compliance with these items was very poor (below 20%). Incomplete search strategies may produce reporting bias and lead researchers or clinicians to misinterpret the reliability of the evidence [8,9]. Since the lack of a literature search can restrict the overall quality of a study, future authors should focus on ways to refine the searching process and pay more attention to the following requirements: 1) search by two authors, 2) search at least two electronic databases as well as hand searching, 3) seek grey literature and recount references, 4) not restrict language, and 5) report the literature search process [26,27]. Although it is difficult to establish a robust searching strategy, this is an area that needs to show improvement in future studies. In addition, only half of SRs/MAs described eligibility criteria in detail. This information needs to be made more apparent and presented much more clearly by use of appropriate tables and figures.
Through subgroup analysis and regression analysis for factors that may affect the quality of the research, we concluded that time of publication and SCI relate more to the overall quality of SRs/MAs than does funding resource. The introduction of PRISMA and AMSTAR statements has largely improved the quality of SRs/MAs. The role of the SCI as a potential factor in the quality of the research suggests that this should be a requirement for future reviews. Only one-fifth of journals require reporting strictly according to the PRISMA statement and this seems to be irrelevant to IF. Since this proportion is so low, the PRISMA statement should feature more in journal instructions to authors, especially for non-SCI journals. Although each set of reporting standards has distinctive advantages, understanding and dissemination of a generic assessment tool was still inadequate. The task of collating information in order to produce guidance on criteria that present researchers with problems will continue [28]. Although the Cochrane Database of Systematic Reviews (CDSR) is rigorously managed and has a robust registration platform [20], we could not distinguish CSR from non-CSR articles because the full text of CSRs was not currently available. It is hoped that research in this field will have a chance to appear on the CDSR.
In addition to including the PRISMA statement in journal instructions to authors, reviewers need to pay more attention to those items that our research identified as having a low reporting rate, such as literature searching. For authors and researchers, using PRISMA and AMSTAR is an effective way to design research in the primary stage, and to test its effectiveness in the final stage. Understanding and mastering relevant procedures is essential for criteria to be successfully met. Future methodological researchers need to focus more on literature searching and seek more guidance concerning criteria that they find problematic.
According to the Cochrane Handbook 5.1, the QUADAS tool is regarded as a common tool to assess the quality of primary diagnostic research [29]. The superiority of the revised tool (QUADAS-2) was apparent in the quality assessment of diagnostic accuracy studies [30,31]. However, nearly half of SRs/MAs did not report using any quality assessment tool. This low proportion may be because QUADAS is not as popular as we thought. Future researchers should be encouraged to apply QUADAS-2.
Strengths and Limitations
Although we assessed the SRs/MAs strictly according to the standardized statements, the results of this evaluation were not without limitations. First, four English databases and three Chinese databases were searched, instead of a specific professional journal or a non-English and Chinese database. Second, a further potential limitation was publication bias. There were several SRs/MAs from the same author, such as Puli, Srinivas R, who published ten articles of the articles we studied. Third, although two reviewers had received rigorous training before undertaking quality assessment, their differing understanding of the two standards is a potential limitation. Last, the weight of the measured item on these two statements may be different, but we regarded each item as the same weight.
Despite these limitations, our research clearly evaluated the need for improving quality for reviews/analyses of a diagnostic approach. First, this was the first article that used two kinds of independent standards to evaluate reporting and methodological quality in global diagnosis SRs/MAs. Second, the findings could be important in the field of EUS diagnosis when clinicians need to refer to SRs/MAs to judge the effectiveness of EUS diagnosis. Third, this study was carefully designed and searched seven main databases. Finally, it is the first, to our knowledge, to analyze potential factors (publication time, SCI, and funding) affecting the quality of SRs/MAs. Our findings could guide future research in this diagnostic field.
Conclusion
With the introduction of PRISMA and AMSTAR statements, the reporting and methodological quality of SRs/MAs of EUS diagnosis has improved measurably over the last two decades. It is hoped that research in this field will have a chance to appear on CDSR. Similarly, literature searching and protocol and registration criteria need to be addressed more in the future. Time of publication and SCI relate more to the overall quality of SRs/MAs than does funding resource. Future researchers should be encouraged to apply QUADAS-2.
Supporting Information
S2 Text. References to systematic reviews in this review.
https://doi.org/10.1371/journal.pone.0120911.s002
(DOCX)
Author Contributions
Conceived and designed the experiments: JHT KHY. Performed the experiments: DLL JXJ. Analyzed the data: DLL JXJ. Contributed reagents/materials/analysis tools: DLL JHT. Wrote the paper: DLL JHT KHY. Provided language and structural improvements to manuscript: JXJ.
References
- 1. Johanson JF, Cooper G, Eisen GM, Freeman M, Goldstein JL, Jensen DM, et al. (2002) Quality assessment of endoscopic ultrasound. Gastrointest Endosc Jun;55(7):798–801. pmid:12024130
- 2. Kinney T (2010) Evidence-based imaging of pancreatic malignancies. Surg Clin North Am Apr; 90(2):235–49. pmid:20362784
- 3. Law JK 1, Stoita A, Weaver W, Gleeson FC, Dries AM, Blackford A, et al. (2014) Endoscopic ultrasound-guided fine needle aspiration improves the pre-operative diagnostic yield of solid-pseudopapillary neoplasm of the pancreas: an international multicenter case series (with video). Surg Endosc Sep; 28(9):2592–8. pmid:24718662
- 4. Bero LA, Jadad AR (1997) How Consumers and Policymakers Can Use Systematic Reviews for Decision Making. Annals of Internal Medicine Jul 1;127(1):37–42. pmid:9214251
- 5. Jadad AR, Cook DJ, Jones A, Klassen TP, Tugwell P, Moher M, et al. (1998) Methodology and reports of systematic reviews and meta-analyses: a comparison of Cochrane reviews with articles published in paper-based journals. JAMA Jul 15;280(3):278–80. pmid:9676681
- 6. Jadad AR, Moher M, Browman GP, Booker L, Sigouin C, Fuentes M, et al. (2000) Systematic reviews and meta-analyses on treatment of asthma: critical evaluation. BMJ Feb 26;320(7234):537–40. pmid:10688558
- 7. Manchikanti L, Derby R, Wolfer L, Singh V, Datta S, Hirsch JA. (2009) Evidence-based medicine, systematic reviews, and guidelines in interventional pain management: part 5 Diagnostic accuracy studies. Pain Physician May-Jun;12(3):517–40. pmid:19461821
- 8. Moher D, Liberati A, Tetzlaff J, Altman DG; PRISMA Group (2009) Preferred reporting items for systematic reviews and meta-analyses: the PRISMA statement. Ann Intern Med Aug 18;151(4):264–9, W64. pmid:19622511
- 9. Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. (2007) Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol Feb 15;7:10. pmid:17302989
- 10. Biondi-Zoccai GG, Lotrionte M, Abbate A, Testa L, Remigi E, Burzotta F, et al. (2006) Compliance with QUOROM and quality of reporting of overlapping meta-analyses on the role of acetylcysteine in the prevention of contrast associated nephropathy: case study. BMJ Jan 28;332(7535):202–9. pmid:16415336
- 11. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JP, et al. (2009) The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate health careinterventions: explanation and elaboration. PLoS Med Jul 21;6(7):e1000100. pmid:19621070
- 12. Willis BH, Quigley M (2011) The assessment of the quality of reporting of meta-analyses in diagnostic research: a systematic review. BMC Med Res Methodol Dec 9;11:163. pmid:22151233
- 13. Al Faleh K, Al-Omran M (2009) Reporting and methodologic quality of Cochrane Neonatal review group systematic reviews. BMC Pediatr. Jun 17;9:38. pmid:19534780
- 14. Shea BJ, Hamel C, Wells GA, Bouter LM, Kristjansson E, Grimshaw J, et al. (2009) AMSTAR is a reliable and valid measurement tool to assess the methodological quality of systematic reviews. J Clin Epidemiol Oct;62(10):1013–20. pmid:19230606
- 15. Shea BJ, Bouter LM, Peterson J, Boers M, Andersson N, Ortiz Z, et al. (2007) External validation of a measurement tool to assess systematic reviews (AMSTAR). PLoS One Dec 26;2(12):e1350. pmid:18159233
- 16. Sequeira-Byron P, Fedorowicz Z, Jagannath VA, Sharif MO (2011) An AMSTAR assessment of the methodological quality of systematic reviews of oral healthcare interventions published in the Journal of Applied Oral Science (JAOS). J Appl Oral Sci Oct;19(5):440–7. pmid:21986647
- 17. Kung J, Chiappelli F, Cajulis OO, Avezova R, Kossan G, Chew L, et al. (2010) From Systematic Reviews to Clinical Recommendations for Evidence-Based Health Care: Validation of Revised Assessment of Multiple Systematic Reviews (R-AMSTAR) for Grading of Clinical Relevance. Open Dent J Jul 16;4:84–91. pmid:21088686
- 18. Ma B, Guo J, Qi G, Li H, Peng J, Zhang Y, et al. (2011) Epidemiology, quality and reporting characteristics of systematic reviews of traditional Chinese medicine interventions published in Chinese journals. PLoS One. 6(5):e20185. pmid:21633698
- 19. Panic N, Leoncini E, de Belvis G, Ricciardi W, Boccia S (2013) Evaluation of the endorsement of the preferred reporting items for systematic reviews and meta-analysis (PRISMA) statement on the quality of published systematic review and meta-analyses. PLoS One Dec 26;8(12):e83138. pmid:24386151
- 20. Shea B, Moher D, Graham I, Pham B, Tugwell P (2002) A comparison of the quality of Cochrane reviews and systematic reviews published in paper-based journals. Eval Health Prof Mar;25(1):116–29. pmid:11868441
- 21. Silagy CA, Middleton P, Hopewell S (2002) Publishing protocols of systematic reviews: comparing what was done to what was planned. JAMA Jun 5;287(21):2831–4. pmid:12038926
- 22. Kirkham JJ 1, Altman DG, Williamson PR (2010) Bias due to changes in specified outcomes during the systematic review process. PLoS One Mar 22;5(3):e9810. pmid:20339557
- 23.
Macaskill P, Gatsonis C, Deeks JJ, Harbord RM, Takwoingi Y (2010) Analysing and Presenting Results. In Cochrane Handbook for Systematic Reviews of Diagnostic Test Accuracy. Edited by Deeks JJ, Bossuyt PM, Gatsonis C. Oxford, United Kingdom: The Cochrane Collaboration 46–47. https://doi.org/10.1002/14651858.MR000022.pub3 pmid:24022476
- 24. Booth A, Stewart L (2013) Trusting researchers to use open trial registers such as PROSPERO responsibly. BMJ Oct 2;347:f5870. pmid:24088555
- 25. Booth A, Clarke M, Dooley G, Ghersi D, Moher D, Petticrew M, et al. (2012) The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev Feb 9;1:2. pmid:22587842
- 26. Faggion CM Jr, Atieh MA, Park S (2013) Search strategies in systematic reviews in periodontology and implant dentistry. J Clin Periodontol Sep;40(9):883–8. pmid:23834263
- 27. Devillé WL, Buntinx F, Bouter LM, Montori VM, de Vet HC, van der Windt DA, et al. (2002) Conducting systematic reviews of diagnostic studies: didactic guidelines. BMC Med Res Methodol Jul 3;2:9. pmid:12097142
- 28. Popovich I, Windsor B, Jordan V, Showell M, Shea B, Farquhar CM. (2012) Methodological quality of systematic reviews in subfertility: a comparison of two different approaches. PLoS One 7(12):e50403. pmid:23300526
- 29.
Higgins J.P.T., Green S. (Eds.) (2011) Cochrane Handbook for Systematic Reviews of Interventions, Version 5.1.0 (updated March 2011), The Cochrane Collaboration, 2008. Available: www.cochrane-handbook.org.
- 30. Whiting PF, Rutjes AW, Westwood ME, Mallett S, Deeks JJ, Reitsma JB, et al. (2011) QUADAS-2: a revised tool for the quality assessment of diagnostic accuracy studies. Ann Intern Med Oct 18;155(8):529–36. pmid:22007046
- 31. Whiting P, Rutjes AW, Reitsma JB, Bossuyt PM, Kleijnen J (2003) The development of QUADAS: a tool for the quality assessment of studies of diagnostic accuracy included in systematic reviews. BMC Med Res Methodol Nov 10;3:25. pmid:14606960