Satisfaction of Search in Chest Radiography 2015

doi:10.1016/j.acra.2015.07.011

Academic Radiology

Volume 22, Issue 11, November 2015, Pages 1457-1465

https://doi.org/10.1016/j.acra.2015.07.011 Get rights and content

Rationale and Objectives

Two decades have passed since the publication of laboratory studies of satisfaction of search (SOS) in chest radiography. Those studies were performed using film. The current investigation tests for SOS effects in computed radiography of the chest.

Methods

Sixty-four chest computed radiographs half demonstrating various “test” abnormalities were read twice by 20 radiologists, once with and once without the addition of a simulated pulmonary nodule. Receiver-operating characteristic detection accuracy and decision thresholds were analyzed to study the effects of adding the nodule on detecting the test abnormalities. Results of previous studies were reanalyzed using similar modern techniques.

Results

In the present study, adding nodules did not influence detection accuracy for the other abnormalities (P = .93), but did induce a reluctance to report them (P < .001). Adding nodules did not affect inspection time (P = .58) so the reluctance to report was not associated with reduced search. Reanalysis revealed a similar decision threshold shift that had not been recognized in the early studies of SOS in chest radiography (P < .01) in addition to reduced detection accuracy (P < .01).

Conclusions

The nature of SOS in chest radiography has changed, but it is not clear why.

Advances in Knowledge

SOS may be changing as a function of changes in radiology education and practice.

Section snippets

Experimental Conditions

We used the same two conditions used in previous SOS demonstrations: presentation of each chest radiograph with and without a simulated pulmonary nodule. The detection accuracy for native, subtle lesions was assessed with and without the addition of a digitally added simulated pulmonary nodule. This created two cases with the same background anatomy and actual lesions perfectly matched for the two conditions (Fig 1). Simulated and native lesions were not spatially superimposed, and the native

Results

Figure 2 presents the observed ROC points averaged over readers for detecting the native test abnormalities. Each ROC point is the average of 20 readers. Comparing crosses to open circles provides a visual comparison of the non-SOS and SOS conditions.

Discussion

The SOS effect observed in this experiment differs from that previously reported for chest radiography 1, 2. The previous experiments reported a decrement in detection accuracy of detecting test abnormalities with the addition of nodules. In contrast, there was no accuracy difference in the present study with the addition of the nodules. Instead, there were reduced FP fractions of the ROC points and reduced TP fractions of the ROC points, indicating a threshold shift toward more conservative

Acknowledgment

Supported by USPHS Grant R01 EB 00145 from the National Institute of Biomedical Imaging and Bioengineering, Bethesda, Maryland.

References (17)

K.S. Berbaum et al.
Proper ROC analysis and joint ROC analysis of the satisfaction of search effect in chest radiography
Acad Radiol
(2000)
K.S. Berbaum et al.
Satisfaction of search from detection of pulmonary nodules in computed tomography of the chest
Acad Radiol
(2013)
D.D. Dorfman et al.
A contaminated binormal model for ROC data - Part II. A formal model
Academic Radiology
(2000)
S.L. Hillis et al.
Recent developments in the Dorfman-Berbaum-Metz procedure for multireader ROC study analysis
Acad Radiol
(2008)
K.S. Berbaum et al.
The cause of satisfaction of search effects in contrast studies of the abdomen
Acad Radiol
(1996)
K.S. Berbaum et al.
Can order of report prevent satisfaction of search in abdominal contrast studies?
Acad Radiol
(2005)
K.S. Berbaum et al.
Satisfaction of search in diagnostic radiology
Invest Radiol
(1990)
E. Samei et al.
Simulation of subtle lung nodules in projection chest radiography
Radiology
(1997)

There are more references available in the full text version of this article.

Cited by (34)

Investigating the impact of cognitive biases in radiologists’ image interpretation: A scoping review
2023, European Journal of Radiology
Image interpretation is a fundamental aspect of radiology. The treatment and management of patients relies on accurate and timely imaging diagnosis. However, errors in radiological reports can negatively impact on patient health outcomes. These misdiagnoses can be caused by several different errors, but cognitive biases account for 74 % of all image interpretation errors. There are many biases that can impact on a radiologist’s perception and cognitive processes. Several recent narrative reviews have discussed these cognitive biases and have offered possible strategies to mitigate their effects. However, these strategies remain untested. Therefore, the purpose of this scoping review is to evaluate the current knowledge on the extent that cognitive biases impact on medical image interpretation.
Scopus and Medline Databases were searched using relevant keywords to identify papers published between 2012 and 2022. A subsequent hand search of the narrative reviews was also performed. All studies collected were screened and assessed against the inclusion and exclusion criteria.
Twenty-four publications were included and categorised into five main themes: satisfaction of search, availability bias, hindsight bias, framing bias and other biases. From these studies, there were mixed results regarding the impact of cognitive biases, highlighting the need for further investigation in this area. Moreover, the limited and untested debiasing methods offered by a minority of the publications and narrative reviews also suggests the need for further research. The potential of role of artificial intelligence is also highlighted to further assist radiologists in identifying and mitigating these cognitive biases.
Cognitive biases can impact radiologists’ image interpretation, however the effectiveness of debiasing strategies remain largely untested.
Effect of a comprehensive deep-learning model on the accuracy of chest x-ray interpretation by radiologists: a retrospective, multireader multicase study
2021, The Lancet Digital Health
Chest x-rays are widely used in clinical practice; however, interpretation can be hindered by human error and a lack of experienced thoracic radiologists. Deep learning has the potential to improve the accuracy of chest x-ray interpretation. We therefore aimed to assess the accuracy of radiologists with and without the assistance of a deep-learning model.
In this retrospective study, a deep-learning model was trained on 821 681 images (284 649 patients) from five data sets from Australia, Europe, and the USA. 2568 enriched chest x-ray cases from adult patients (≥16 years) who had at least one frontal chest x-ray were included in the test dataset; cases were representative of inpatient, outpatient, and emergency settings. 20 radiologists reviewed cases with and without the assistance of the deep-learning model with a 3-month washout period. We assessed the change in accuracy of chest x-ray interpretation across 127 clinical findings when the deep-learning model was used as a decision support by calculating area under the receiver operating characteristic curve (AUC) for each radiologist with and without the deep-learning model. We also compared AUCs for the model alone with those of unassisted radiologists. If the lower bound of the adjusted 95% CI of the difference in AUC between the model and the unassisted radiologists was more than −0·05, the model was considered to be non-inferior for that finding. If the lower bound exceeded 0, the model was considered to be superior.
Unassisted radiologists had a macroaveraged AUC of 0·713 (95% CI 0·645–0·785) across the 127 clinical findings, compared with 0·808 (0·763–0·839) when assisted by the model. The deep-learning model statistically significantly improved the classification accuracy of radiologists for 102 (80%) of 127 clinical findings, was statistically non-inferior for 19 (15%) findings, and no findings showed a decrease in accuracy when radiologists used the deep-learning model. Unassisted radiologists had a macroaveraged mean AUC of 0·713 (0·645–0·785) across all findings, compared with 0·957 (0·954–0·959) for the model alone. Model classification alone was significantly more accurate than unassisted radiologists for 117 (94%) of 124 clinical findings predicted by the model and was non-inferior to unassisted radiologists for all other clinical findings.
This study shows the potential of a comprehensive deep-learning model to improve chest x-ray interpretation across a large breadth of clinical practice.
Annalise.ai.
The role of clinical history in the interpretation of chest radiographs
2021, Radiography
Citation Excerpt :
The Royal College of Radiologists (RCR) state that causes of error can be perceptual. This umbrella term includes observational, where the finding was identifiable but missed, and satisfaction of search which occurs when lesions remain undetected after detection of an initial lesion.39,40 The RCR also describes cognitive error, where the finding was appreciated but attributed to the wrong cause, and system error due to inadequate, misleading or incorrect clinical information.41
This review will appraise the literature pertaining to the influences that clinical history has on the action of assessing the chest radiograph.
There remains conflicting evidence on the impact of clinical history on chest radiography. Some research suggests that clinical history has the potential to influence the reporter in a negative way by limiting their search strategy to a more focussed search. Image interpretation is more accurate when reporters are allowed to conduct a free search of the chest image, untainted by preconceived concepts.
Clinical history needs to be accessed appropriately to aid and not stifle accurate image interpretation. Reporters need to be aware of the potential bias clinical history can introduce to their reporting and develop strategies to alleviate this as much as possible.
A greater understanding of the potential bias of clinical history on the process of image interpretation is required by all reporters. Reporters need to develop an approach and strategy when accessing clinical history. Novice reporters need to be educated regarding the impact of clinical history on their reporting.
A comparison of computer-assisted detection (CAD) programs for the identification of colorectal polyps: performance and sensitivity analysis, current limitations and practical tips for radiologists
2018, Clinical Radiology
To directly compare the accuracy and speed of analysis of two commercially available computer-assisted detection (CAD) programs in detecting colorectal polyps.
In this retrospective single-centre study, patients who had colorectal polyps identified on computed tomography colonography (CTC) and subsequent lower gastrointestinal endoscopy, were analysed using two commercially available CAD programs (CAD1 and CAD2). Results were compared against endoscopy to ascertain sensitivity and positive predictive value (PPV) for colorectal polyps. Time taken for CAD analysis was also calculated.
CAD1 demonstrated a sensitivity of 89.8%, PPV of 17.6% and mean analysis time of 125.8 seconds. CAD2 demonstrated a sensitivity of 75.5%, PPV of 44.0% and mean analysis time of 84.6 seconds.
The sensitivity and PPV for colorectal polyps and CAD analysis times can vary widely between current commercially available CAD programs. There is still room for improvement. Generally, there is a trade-off between sensitivity and PPV, and so further developments should aim to optimise both. Information on these factors should be made routinely available, so that an informed choice on their use can be made. This information could also potentially influence the radiologist's use of CAD results.
Now You See It, But Would You Later? Examining the Mechanisms of Satisfaction of Search in the Fatigued Radiologist
2017, Academic Radiology
The Impact of Fatigue on Satisfaction of Search in Chest Radiography
2017, Academic Radiology
Citation Excerpt :
Specifically, we wanted to find out whether SOS under conditions of fatigue would lead to an SOS effect on detection accuracy as documented in the original SOS studies (15,16) or an SOS effect on decision thresholds as found in the more recent ones (17,18). The same set of 64 CR chest cases from the most recent chest SOS study (17) was used. All cases were acquired from clinical studies and anonymized in accordance with local institutional review board approval.
To assess the nature of the satisfaction of search (SOS) effect in chest radiography when observers are fatigued; determine if we could replicate recent findings that have documented the nature of the SOS effect to be due to a threshold shift rather than a change in diagnostic accuracy as in earlier film-based studies.
Nearing or at the end of a clinical workday, 20 radiologists read 64 chest images twice, once with and once without the addition of a simulated pulmonary nodule. Half of the images had different types of “test” abnormalities. Decision thresholds were analyzed using the center of the range of false-positive (FP) and true-positive (TP) fractions associated with each receiver operating characteristic (ROC) point for reporting test abnormalities. Detection accuracy was assessed with ROC technique and inspection time was recorded.
The SOS effect was confirmed to be a reduction in willingness to respond (threshold shift). The center of the FP range was significantly reduced (FP = 0.10 without added nodules, FP = 0.05 with added nodules, F(1,18) = 19.85, P = 0.0003). The center of the TP range was significantly reduced (TP = 0.39 without added nodules, TP = 0.33 with added nodules, F(1,18) = 10.81, P = 0.004).
This study suggests that fatigue does not change the nature of the SOS effect, but rather may be additive with the SOS effect. SOS reduces both TP and FP responses, whereas fatigue reduces TPs more than FPs.

View all citing articles on Scopus

View full text

Original InvestigationSatisfaction of Search in Chest Radiography 2015

Rationale and Objectives

Methods

Results

Conclusions

Advances in Knowledge

Section snippets

Experimental Conditions

Results

Discussion

Acknowledgment

Acad Radiol

Acad Radiol

Academic Radiology

Acad Radiol

Acad Radiol

Acad Radiol

Satisfaction of search in diagnostic radiology

Invest Radiol

Simulation of subtle lung nodules in projection chest radiography

Radiology

Original Investigation
Satisfaction of Search in Chest Radiography 2015