Multisite assessment of emergency medicine resident knowledge of evidence-based medicine as measured by the Fresno Test of Evidence-Based Medicine

Methods

Results

Of the 124 submitted FTEBMs for grading, only 51.6% (64/124) had all 12 questions answered. A total of 48.4% (60/124) of FTEBMs was submitted with one or more question left unanswered. The most common question left unanswered was quantitative question 9 (risk reduction) (36.3% (45/124)) (Table 1).

Most residents (50.8%, 124/244) (35.4% PGY-1, 25.8% PGY-2, 26.6% PGY-3, 12.1% PGY-4) completed the demographics survey and FTEBM assessment during the study period (Table 2). Male residents (81/124, 65.3%) outnumbered female residents (34.7%, 43/124), and most residents self-identified as Caucasian (76/124, 61.3%) and held a doctorate in osteopathic medicine (109/124, 87.9%) (Table 2). A minority of the residents held an additional degree at the Master’s level and above (29/124, 23.4%) (Table 2). Residents most frequently reported an “average” ability (72/124, 58.1%) to assess the quality of medical evidence compared with their peers (Table 2). The overall mean score in those residents who reported an “average” or “above average” baseline knowledge in EBM was 36.0% (46.1/128) (32.8–39.1%). No correlation was noted with the self-perceived knowledge in EBM and overall test scores on the FTEBM (T _b= 0.021; p=0.768). The residency program’s focus on EBM was most often rated as “average” (41/124, 33.1%) or “above average” (41/124, 33.1%) by the participating residents (Table 2). The overall mean score on the FTEBM of residents rating their institutions as having an “average” or “above average” focus on EBM was 34.9% (44.7/128) (32.2–37.6%). No correlation was noted with the perceived institutional focus on EBM and the overall test scores on the FTEBM (T _b =−0.018; p=0.792).

The overall mean performance for the FTEBM was 35.2% (45.1/128) (32.7–37.8%). Performance on the FTEBM did not vary by postgraduate year in residency training with PGY-1 through PGY-4 performing on average at 32.3% (41.3/128), 36.9% (50.7/128), 37.7% (48.3/128), and 34.6% (44.3/128), respectively, on the examination (p=0.356) (Table 3). Residents holding a doctorate in osteopathic medicine performed the same as residents holding a doctorate in allopathic medicine on the overall examination (35.6% [45.6/128] vs. 32.9% [42.1/128]; p=0.511) (Table 3). No effect on overall performance was detected among those holding an additional advanced degree at the Master’s level or above as compared to those without (38.9% [49.8/128] vs. 34.1% [43.6/128]; p=0.117). Male residents significantly outperformed female residents on the FTEBM (37.1% (47.5/128) vs. 31.7% (40.6/128); p=0.049) (Table 3). When assessed by question type (theoretical or quantitative), no significant differences based upon resident demographics were detected (Table 3).

Among the core components of EBM assessed by the FTEBM, residents showed the most room for growth on understanding the components of statistical significance and diagnostic study design, with only 5.6 and 6.5% of residents scoring above 50% on the questions addressing these topics (Table 4). Conversely, 73.4% of residents performed above the 50% criterion on questions addressing clinical study design (Table 4). The training level of residents was significantly associated with increased performance on questions addressing clinical study design (PGY-1=46.6%, PGY-2=53.9%, PGY-3=59.3%, PGY-4=70.0%, p=0.014) and evidence search strategies (PGY-1=39.2%, PGY-2=54.3%, PGY-3=51.5%, PGY-4=30.0%; p=0.029). A linear relationship between years of postgraduate education and clinical study design was detected (p=0.002).

Discussion

This study represents the largest independent assessment of emergency medicine residents’ knowledge of evidence-based practice in the literature to date. A recent systematic review on the development of EBM curriculums for emergency medicine noted several studies assessing a resident’s knowledge on topics commonly covered in EBM but that few utilized a validated assessment tool [11]. Conducted in 2010, Friedman et al. [12] sought to describe the impact of EBM on the change in patient-directed care by emergency medicine residents. Although the authors did not assess baseline knowledge, they did describe how following a search and review of the literature, the residents altered the course of care for 16.3% of patients. While the data presented in this manuscript suggest an improvement of literature search skills over the course of residency training, it appears that additional work could be completed by faculty to further train residents on how to interpret the papers that are sourced.

In a group of residents at a single training institution, Mohr et al. [13] reported a baseline score on the FTEBM of 82.3%. In comparison, this graduate medical education program represents an academic health center program that is usually assumed to be more replete in EBM didactics compared to a community hospital-based residency program such as ours. The highest score among residencies with more than 10 participants in our study was 48.0%. Bentley et al. [14] estimate that scores start low in emergency medicine residents and then increase linearly from PGY-1 through PGY-4 with a starting point of 25% and endpoint of 77%.

Based upon the current study and prior literature, it appears that emergency medicine residents have a limited understanding of the basic theoretical and statistical concepts that are needed to effectively understand and practice EBM. Although no overall differences in FTEBM scores were observed based on postgraduate years, more experienced residents did show an increase in knowledge on searching for clinical literature (PGY-3 outperforming PGY-1 residents by 12.3% points) and on study design (PGY-3 residents outscoring PGY-1 residents by 12.7% points). Unlike many of the items on the FTEBM, these items represent the initial phases of clinical evidence. It is likely that residents garner a more ad hoc search strategy as they learn to utilize available online resources over the course of their training. Although residents learn to better identify literature, the remainder of the items on the FTEBM do not show training-year performance-related increases. Thus, while the residents may be able to locate literature, it is unlikely that they are able to evaluate the scope of the reported findings based upon statistical analysis.

Residency programs may be well served to increase the amount of time spent teaching both research-focused concepts and biostatistics in order to better prepare trainees to incorporate the principles of EBM into future practice. Although not addressed in the current study, when asked about attitudes toward research, residents indicate little perceived impact of EBM on their medical training [15]. These attitudes represent a concerning barrier toward future training.

Based upon these results and the current body of literature, the open question that remains is what educational activities can be implemented during residency to improve understanding of the core concepts of EBM. Although sparse in the literature, several interventions have attempted to improve knowledge of research appraisal. Validated instructional techniques for postgraduates include dedicated journal clubs [16], individual research methodology workshops [17], and full curricula associated with monthly instruction [18]. Unfortunately, only a minority of graduate medical education programs have an established EBM curriculum [19].

Conclusions

The surveyed emergency medicine residents show an incomplete understanding of the basic concepts in EBM as tested by the FTEBM. No difference in knowledge was seen based on the postgraduate level of training, on the different medical school degrees, or if a resident held an advanced degree in addition to that of medicine. A heightened focus on teaching the basic theoretical and statistical concepts of research is needed if residents are truly expected to not only meet the specific milestones prior to graduation but also effectively treat patients with an eye toward EBM.