The effect of prior experience on diagnostic reasoning: exploration of availability bias

Introduction

Classic work on diagnostic reasoning has established that clinicians propose an initial diagnostic hypothesis quite rapidly and very early in a patient encounter [1], [2], [3], [4]. These early diagnostic hypotheses have been shown to determine final diagnostic accuracy; if the correct diagnostic hypothesis was advanced in the first few minutes of a clinical encounter, clinicians were likely to be correct (95%), whereas if the hypothesis was not advanced in the first few minutes, the likelihood of an eventual accurate diagnosis was much lower (20%) [4]. In another study in a primary care setting, the accuracy of clinicians’ “diagnostic hypotheses” was 78% when based only on a review of the chief complaint alone [3]. Subsequent review of the patient history contributed a smaller amount to diagnostic performance (16%), while the physical exam and diagnostic tests did not influence clinicians’ diagnostic accuracy.

This early hypothesis generation process is presumed to be based on prior examples [1]. In the course of becoming an expert, a clinician will encounter hundreds of clinical cases. These encounters may be seen as representative of core distinguishing features; relying on memory for these prior cases may be both an accurate and efficient way to sort between the features of one potential diagnosis relative to another [5], [6]. Even for novices, prior examples can provide scaffolding to understand how current case features relate to prior experiences; given the relatively small subset of cases that novice clinicians have to draw upon; this approach may be less effective in terms of accuracy [7]. Indeed, several studies have shown that prior examples can have both positive and negative effects on diagnostic accuracy [7], [8], [9], [10], [11], [12]. While none of these specifically evaluated the influence of expertise, the common sense conclusion from this classic research is that more experienced physicians generate more accurate hypotheses [1]. Yet recent literature [12], [13], [14], [15], [16], [17], [18], [19] has unilaterally emphasized the potential for error due to reliance on prior examples. Kahneman described this as the process by which:

“…people assess the … probability of an event by the ease with which instances or occurrences can be brought to mind. If the availability heuristic is applied, then such factors will affect the … subjective probability of events. Consequently, the use of the availability heuristic leads to systematic biases.” [18]

Typical descriptions of this process, often called “availability bias,” [12], [13], [15], [16], [17], [18], [19] do not specify the contextual factors (e.g., clinical signs or symptoms, patient demographics) that may influence ease of recall. However, results from previous studies of availability bias suggest two mechanisms that may act independently of expertise: the current case may facilitate recall of the clinical reasoning pathway and relevant clinical knowledge of differential diagnoses [19], [20], [21], or resemblance between a prior case and the current case—even related to non-clinical features such as hair color—may be sufficient to facilitate recall [8]. Of particular concern, research on the availability bias in domains outside of medicine rarely include expertise as a factor. With few exceptions, studies of cognitive bias outside medicine use materials that are “knowledge or experience sparse” and dependent more on statistical or logical analysis than specialist knowledge [22]. Consequently, they provide no insight into the effect of experience or expertise. Within medicine, a few studies have evaluated how expertise is associated with the influence of cognitive bias [21], [23]. In one study, exposure to prior similar cases was measured through self-report and the nature of similarity was not controlled [21]. Additionally, participants reviewed very few cases. In the second study, clinical outcomes for 12 cases were presented in a positive versus a negative frame, but the effects of this biasing technique were minimal [23]. Indeed, medical students did not demonstrate bias. More experienced physicians demonstrated a small bias for only 2 cases and responded in idiosyncratic ways, suggesting a stronger influence of unique personal experience [23]. The empirical question remaining is whether the influence of a prior example varies when cases share similar features or dissimilar features, and also if these variations are influenced further by the clinician’s level of experience.

This study aimed to measure the influence of a single similar prior case on the diagnostic hypothesis generation process. As it is difficult to control what prior cases physicians have seen or remember, study participants were exposed to identical sets of recent prior cases. A preliminary exposure phase created the experience of a prior case. In this exposure phase, participants confirmed a working diagnosis. A second test phase measured the influence of the prior case. The similarity of case features in the exposure and test phases was manipulated to evaluate the effect of case similarity on the diagnostic accuracy of clinicians with varying amounts of experience. Competing clinical reasoning theories predict two potential outcomes: 1) if experts rely more on analytical reasoning and discount prior examples, they demonstrate no influence of a prior similar case; 2) if experts rely more on similarity to prior exemplars, the influence of a prior similar case may be stronger for experts compared to novices [20], [24].

Materials and methods

Design summary

The study involved clinicians at three levels of expertise—pre-clerkship medical students, emergency medicine residents, and staff emergency medicine physicians. In essence, we engineered experiences with specific prior cases. We then measured if there was an influence of these prior cases on a set of test cases. Each participant first went through an exposure phase involving a total of eight case vignettes. They then diagnosed 10 test cases. The exposure and test phases created four experimental conditions and one control condition. This study design is summarized in Figure 1.

Figure 1:

Study flow diagram – all participants completed an exposure phase and a test phase.

*Control case diagnoses: adrenal insufficiency, anaphylaxis, glyburide and mastoiditis.

Results

For the eight 50/50 cases overall, Diagnosis A represented 41%, Diagnosis B 53% and other diagnoses 6% of all responses. The percent by condition for Diagnosis A: a) 36%, b) 36%, c) 44% and d) 47%. Accuracy for filler cases varied: 68% for medical students and 98% for residents and staff.

Specific similarity

As shown in Figure 2, the results examining accuracy assessed by Diagnosis A only did not reveal any influence of specific similarity of patient characteristics (F(3,712)=7.28, p=0.28). (Similar results are found for Diagnosis B (F(3,712)=4.87, p=0.19)). Contrary to the proposed hypothesis there was no evidence of a gradient of accuracy. Although condition d did produce roughly 50/50 accuracy, this was not unique.

Figure 2:

Specific Similarity – Diagnostic accuracy results for conditions a to d, determined by examining the influence of condition on Diagnosis A only. Similar results are seen when analyzed for Diagnosis B. Error bars are standard error for condition.

Post-hoc results

Broad category priming

Figure 3 shows the analysis by level of expertise and condition a–c, when either Diagnosis A or B is given, as well as condition d (unbiased 50/50 cases). There was an effect of experience on diagnostic accuracy; medical students achieved an accuracy of 85%, medical residents achieved an accuracy of 98%, and faculty achieved an accuracy of 98%, and (F(2,712)=35.6, p<0.0001). There was a main effect of condition, F (3,712)=3.91, p<0.01, with condition d (no bias) having a lower accuracy. There was also an interaction between condition and experience level, F(6, 712)=3.55, p<0.01.

Figure 3:

Category priming – diagnostic accuracy results determined by examining the influence of conditions a to d on Diagnosis A and B. The near ceiling effect for staff and residents indicates that Diagnosis A and B were entered with equal frequency and were not influenced by the exposure phase. An influence of the exposure phase is only seen with medical students. Error bars are standard error for condition.

Discussion

This study was designed to critically examine the role of prior examples; specifically, to test whether prior examples are beneficial or harmful (availability bias), and whether a prior example affects all levels of expertise equally. In a post-hoc analysis we also examined whether the effect of similarity operates at a specific case feature level or at a broader category priming level.

The study found a small, but significant, main effect of prior examples. On more detailed examination, the effect of a single example on downstream performance was restricted to novice medical students. In this population, the effect was fairly large, ranging from 11 to 20% higher accuracy. Conversely, there was no detectable impact on more experienced clinicians, perhaps in part because there was a ceiling effect with mean accuracy scores approaching 100%. Indeed, for the more experienced physicians, accuracy for filler cases was 98%. In the literature, availability is typically framed as a cognitive bias leading to increased errors. However, this may be an artifact of the experimental conditions. Most of the studies that demonstrate negative effects of availability use priming cases that resemble the test case but are from a different, incorrect, diagnostic category [12]. Under these circumstances, availability leads to error. However in the present study and some others [7] availability contributes to an increase in accuracy.

In prior work, the effect of expertise on cognitive bias is not well studied. Kahneman [18] has claimed that cognitive biases, such as availability, are not amenable to instruction or related to experience:

“What can be done about biases? … How can we improve judgments and decisions … ? The short answer is that little can be achieved without a considerable investment of effort. …System 1 is not readily educable….” [18], p. 416.

However, he does not present any evidence to substantiate this claim. By contrast, some studies of cognitive biases in clinical reasoning have shown that biases diminish with expertise [21], [23]. Indeed, the present study showed no effect of availability (i.e., recent exposure) in the two experienced cohorts. Our findings suggest that the effect of a single, recent case diminishes with expertise. As acquisition of expertise in part reflects a larger mental inventory of exemplars, it is reasonable that the effect of any one exemplar, particularly a very recent one, diminishes, which is compatible with an exemplar-based model of reasoning.

While availability was present in novices, it emerged with two critical characteristics: First, there was a significant and substantial benefit arising from exposure to a prior similar case. Second, the benefit appeared to be a consequence of category priming [26], [27]. This suggests that the prior experience facilitated retrieval of a broader differential diagnosis.

It is challenging to integrate the current findings with those of prior work. Mamede [12] used a very broad definition of availability, assuming that any diagnosis that may be on the differential diagnosis for the test case may be more available from prior exposure. In that respect, their conclusion is similar to the present study. However Hatala [8] showed specific similarity effects in ECG interpretation derived from matching of contextual features only. Similarly, a number of studies in dermatology [31], [32] have shown specific effects based on similarity in appearance.

One explanation for the variability may be that studies that show effects of specific similarity are based on cognitively-focused tasks; interpretation of ECGs or skin lesions, or maintaining a clinical interview. In a cognitively-focused task, participants may have limited attention for multiple contextual features or limited ability to consider multiple sources of information. Hence, any effect of prior examples may result primarily from matches to a limited set of information. It may be that the manipulations in the present study simplified the task, encouraging broader connections.

Conclusions

In this study, novices showed an effect of a previous example on diagnostic accuracy. Moreover, the effect had two specific characteristics. First, it improved accuracy, in contrast to the prevailing view that availability leads to greater errors. Second, the phenomenon arose at the level of category priming, increasing the potential breadth of the differential diagnosis. The diagnoses provided by intermediate and expert clinicians did not appear to be influenced by the experimental manipulation of the availability bias.