Elsevier

Academic Radiology

Volume 22, Issue 5, May 2015, Pages 580-590
Academic Radiology

Original Investigation
Estimating Screening-Mammography Receiver Operating Characteristic (ROC) Curves from Stratified Random Samples of Screening Mammograms: A Simulation Study

https://doi.org/10.1016/j.acra.2014.12.011Get rights and content

Rationale and Objectives

To evaluate stratified random sampling (SRS) of screening mammograms by (1) Breast Imaging Reporting and Data System (BI-RADS) assessment categories, and (2) the presence of breast cancer in mammograms, for estimation of screening-mammography receiver operating characteristic (ROC) curves in retrospective observer studies.

Materials and Methods

We compared observer study case sets constructed by (1) random sampling (RS); (2) SRS with proportional allocation (SRS-P) with BI-RADS 1 and 2 noncancer cases accounting for 90.6% of all noncancer cases; (3) SRS with disproportional allocation (SRS-D) with BI-RADS 1 and 2 noncancer cases accounting for 10%–80%; and (4) SRS-D and multiple imputation (SRS-D + MI) with missing BI-RADS 1 and 2 noncancer cases imputed to recover the 90.6% proportion. Monte Carlo simulated case sets were drawn from a large case population modeled after published Digital Mammography Imaging Screening Trial data. We compared the bias, root-mean-square error, and coverage of 95% confidence intervals of area under the ROC curve (AUC) estimates from the sampling methods (200–2000 cases, of which 25% were cancer cases) versus from the large case population.

Results

AUC estimates were unbiased from RS, SRS-P, and SRS-D + MI, but biased from SRS-D. AUC estimates from SRS-P and SRS-D + MI had 10% smaller root-mean-square error than RS.

Conclusions

Both SRS-P and SRS-D + MI can be used to obtain unbiased and 10% more efficient estimate of screening-mammography ROC curves.

Section snippets

Materials and methods

We begin with a large population of clinical mammogram cases for which we wish to estimate the screening-mammography ROC curve. For example, in this study, we used approximately the 49,500 cases of the Digital Mammography Imaging Screening Trial (DMIST) (9). From this large population of clinical cases we build a smaller observer study case set. The goal is to construct the observer study case set to be as small as possible for efficient (low uncertainty) and unbiased estimation of

Results

Table 5, Table 6, Table 7 show the bias of AUC estimates from simulated RS, SRS-P, SRS-D, and SRS-D + MI case sets. There was essentially no bias in all AUC estimates from RS, SRS-P, and SRS-D + MI case sets (Table 5, Table 7), but there was clearly bias from SRS-D case sets, with AUC estimates biased lower from 0.03 to 0.28 compared to the expected AUC value of 0.75 (Table 6). Thus, RS and SRS-P case sets produced essentially unbiased AUC estimates, whereas SRS-D case sets produced clearly

Discussion

This Monte Carlo simulation study of three SRS methods for construction of observer study case sets from a large clinical case population shows that the methods of SRS-P and SRS-D + MI can produce unbiased and 10% more efficient (ie, smaller estimation variance) AUC estimates compared to the standard method of RS. The method of SRS-P has been known for a long time (8), but is not widely used in laboratory observer studies; the results of our study suggest that it might deserve more attention

Acknowledgments

The content of this article is solely the responsibility of the authors and does not necessarily represent the official views of any of the supporting organizations.

References (22)

  • C.J. D'Orsi et al.

    Breast imaging reporting and data system: ACR BI-RADS-mammography

    (2003)

    • Cited by (1)

    • Comparison of Contrast-Enhanced Mammography With Conventional Digital Mammography in Breast Cancer Screening: A Pilot Study

      2019, Journal of the American College of Radiology
      Citation Excerpt :

      The rationale for building the case set with cancer incidence greater than in the general population was to allow for statistical analysis because low incidence of de novo cancer limits detection of change before and after addition of recombined images. We simulated the screening pool using a cancer incidence rate that is between the stratified random sampling with proportional and disproportional allocation as described by Zur et al [24]. Information regarding tissue density and background parenchymal enhancement (BPE) was collected for each case from the initial CEM imaging report.

    This work was supported in part by the National Cancer Institute (NCI) of the National Institutes of Health (NIH) through grant CA092361.

    1

    Present address: The Hospital for Sick Children, 555 University Avenue, Toronto, ON, M5G 1X8, Canada.

    2

    Present address: Computation Institute, Searle Chemistry Laboratory, 5735 South Ellis Avenue, Chicago, IL 60637.

    View full text
    Copyright © 2015 AUR. Published by Elsevier Inc. All rights reserved.