Identification of Randomized Controlled Trials From the Emergency Medicine Literature: Comparison of Hand Searching Versus MEDLINE Searching,☆☆,

https://doi.org/10.1016/S0196-0644(99)70268-4Get rights and content

Abstract

Study objective: As part of an ongoing project to identify all the randomized controlled trials (RCTs) in the emergency medicine literature, in association with the Cochrane Collaboration, 2 discrete studies were undertaken; the first, to compare motives for active participation in hand searching of the literature by emergency medicine professionals, and the second, to compare hand searching with MEDLINE searching of a number of emergency medicine journals. Methods: All listed members of the British Association for Emergency Medicine (BAEM) and the Society for Academic Emergency Medicine (SAEM) received a standard letter outlining the objectives of the project, with 1 of 3 headings assigned on an alternate basis. Recruited volunteers hand searched journals prioritized from the emergency medicine literature. Each issue of each journal was hand searched for RCTs. In addition, a comprehensive MEDLINE search was conducted for each journal. The yields of RCTs from the 2 searching methods were compared for all journals and for each journal individually. Results: No clear motivation for participation in this work could be ascertained because of the low response rates from BAEM and SAEM (10.1% and 1.8%, respectively). Only 18 (29.0%) of the 62 journals identified were indexed by MEDLINE. In the 14 journals indexed by MEDLINE for which hand searching was completed, a total of 710 RCTs were identified by a combination of the 2 approaches; of these, 592 (83.4%) were identified by hand searching alone and 483 (68.0%) by MEDLINE searching alone. Both methods identified 365 (51.4%) RCTs; hand searching revealed an additional 227 (32.0%) that were not identified by MEDLINE searching, and MEDLINE searching found 118 (16.6%) that were not identified by hand searching. The difference between the proportions identified by hand searching and by MEDLINE searching (15.4%; 95% confidence interval [CI], 12.7% to 17.9%) was statistically significant (McNemar’s χ2 test, 1 df , 33.8; P <.0001). This difference was not significant for 8 of the journals. Conclusion: The response rates from mailing to members of the relevant professional organizations letters requesting participation in this work were very low and suggested that such an approach was not cost-effective. However, no formal costing exercise was undertaken. Searching results showed that, in the 14 emergency medicine journals indexed by MEDLINE for which hand searching was completed, hand searching led to identification of additional RCTs (primary articles) not found through MEDLINE searching. However, hand searching, although statistically significantly better than MEDLINE searching, failed to identify some of the RCTs found by MEDLINE searching, suggesting that hand searching is not a “gold standard” method and that the dual approach, promoted by the Cochrane Collaboration, may be the optimal approach for journals indexed by MEDLINE. [Langham J, Thompson E, Rowan K: Identification of randomized controlled trials from the emergency medicine literature: Comparison of hand searching versus MEDLINE searching. Ann Emerg Med July 1999;34:25-34.]

Section snippets

INTRODUCTION

With the advent of new journals, the amount of scientific literature is ever increasing. As a result, health care professionals are more likely to rely on review articles to keep abreast of developments. Review articles are therefore very influential, but they often are subjective appraisals and syntheses of selected published works and consequently carry a potential for the introduction of bias by the reviewers.1 In addition, reviews carried out nonsystematically may lead to seriously

MATERIALS AND METHODS

Volunteers to assist with hand searching of the emergency medicine literature were identified by 4 different approaches. First, all listed members of the British Association for Emergency Medicine (BAEM) and the Society for Academic Emergency Medicine (SAEM) received a standard letter outlining the objectives of the work and the background to the Cochrane Collaboration and requesting help with the hand searching of the emergency medicine literature. Every member from the alphabetical listing

RESULTS

The results of the comparison of motives for active participation in hand searching of the literature by emergency medicine professionals are presented as response rates from members of the BAEM and SAEM by type of letter heading in Table 1.

. Comparison of motives for active participation in hand searching of the emergency medicine literature by emergency medicine professionals.

ApproachTotal No. of ContactsNo. of Responders (% of Contacts)Total No. of Actual Hand Searchers (% of Contacts)

DISCUSSION

The response rates from mailing to members of the relevant professional organizations letters requesting participation in this work were very low and suggested that such an approach was not cost-effective. However, no formal costing exercise was undertaken, nor do any formal cost comparisons for other recruitment methods exist. The higher response rate from the British organization may have occurred because the Cochrane Collaboration was established in the United Kingdom and was only emerging

Acknowledgements

We would like to acknowledge the complimentary subscriptions received from 11 journals and the efforts of the 46 volunteers who retrospectively and prospectively hand searched 18 journals. In addition, we thank Iain Chalmers, Kay Dickersin, Chris Silagy, Caroline Goldfrad, Phil Alderson and, more recently, the anonymous journal referees for comments on earlier drafts of this manuscript.

Journals providing complimentary subscriptions: Injury , Burns, Accident and Emergency Nursing , Resuscitation

References (19)

  • D Schwartz et al.

    Explanatory and pragmatic attitudes in therapeutical trials

    J Chronic Dis

    (1967)
  • I Chalmers et al.

    The Oxford Database of Perinatal Trials: Developing a register of published reports of controlled trials

    Control Clin Trials

    (1986)
  • DL Sackett et al.

    Clinical Epidemiology : A Basic Science for Clinical Medicine

    (1991)
  • EM Antman et al.

    A comparison of results of meta-analyses of randomized controlled trials and recommendations of clinical experts: Treatment for myocardial infarction

    JAMA

    (1992)
  • R Peto

    Why do we need systematic overviews of randomized controlled trials?

    Stat Med

    (1987)
  • Anonymous

    Systematic overview of controlled trials (meta-analyses) helps clarify treatment effects

    Drugs Ther Bull

    (1992)
  • I Chalmers et al.

    Getting to grips with Archie Cochrane’s agenda

    BMJ

    (1992)
  • K Dickersin et al.

    Identifying relevant studies for systematic reviews

    BMJ

    (1994)
  • RW Scherer et al.

    Full publication of results initially presented in abstracts: A meta-analysis

    JAMA

    (1994)
There are more references available in the full text version of this article.

Cited by (40)

  • Handsearching had best recall but poor efficiency when exporting to a bibliographic tool: case study

    2020, Journal of Clinical Epidemiology
    Citation Excerpt :

    Handsearching involves a manual, page-by-page examination of the entire contents of relevant journals, conference proceedings, and abstracts [2,4,7,9,16–18]. There is evidence that handsearching is effective when compared with bibliographic database searching and that handsearching can identify studies (or study data) which may be missed by other search methods [4,5,7,13,16,17,19–26]. While handsearching is known to be an effective method of study identification, it is resource intensive [5].

  • Medication adherence interventions that target subjects with adherence problems: Systematic review and meta-analysis

    2016, Research in Social and Administrative Pharmacy
    Citation Excerpt :

    Searches were conducted in 19 research registers (e.g., Research Portfolio Online Reporting Tools), and investigators were contacted to obtain research reports of those studies.94,107,108 Hand searches were conducted in 57 journals where multiple eligible papers in the parent project were published.109,110 Potentially eligible studies were imported into bibliographic software and subsequently tracked with study-specific custom fields and terms.

  • MEDLINE, EMBASE, and Cochrane index most primary studies but not abstracts included in orthopedic meta-analyses

    2009, Journal of Clinical Epidemiology
    Citation Excerpt :

    It also provides some important data regarding the percentage of relevant articles one may expect to find in MEDLINE and EMBASE when searching for this topic. The results we found were similar to those of other works completed in nonsurgical disciplines that investigated the performance of databases, such as MEDLINE and EMBASE [15,16,20,22–27]. We believe that this is the first study attempting to quantify the recall rates of the major biomedical databases as they relate to primary research used for orthopedic-related meta-analyses.

  • The Cochrane Library as a Resource for Evidence on Out-of-Hospital Health Care Interventions

    2007, Annals of Emergency Medicine
    Citation Excerpt :

    To our knowledge, this is the first time that the evidence in the Cochrane Library has been assessed for relevance to out-of-hospital health care interventions. Although the number of systematic reviews and reports of out-of-hospital trials identified is low in comparison to that of other medical specialities, the number of trials identified by this study is much larger than those reported in previous out-of-hospital literature searching efforts.3-5,7,8,17,18 In conclusion, although the Cochrane Library is a valuable resource for out-of-hospital interventions, the topic coverage is sparse.

View all citing articles on Scopus

Supported by the Researchand Development Directorateof the North West RegionalHealth Authority, UK (grant No. 0925/704267), and the Faculty of Accident and Emergency Medicine, England, and the Cochrane Injuries Group.

☆☆

Address for reprints: Kathy Rowan, MSc, PhD, Scientific Director, ICNARC, Tavistock House,Tavistock Square, London WC1H 9HR, UK; 0171-388-2856,fax 0171-388-3759;E-mail [email protected].

47/1/97704

View full text