Skip to main content
SpringerLink
Log in

Methods for Evaluating Prediction Performance of Biomarkers and Tests

  • Conference paper
  • First Online:
Book cover Risk Assessment and Evaluation of Predictions

Part of the book series: Lecture Notes in Statistics ((LNSP,volume 215))

Abstract

This chapter covers material presented in a short course at the 2011 International Conference on Risk Assessment and Evaluation of Predictions. Methods for evaluating the performance of markers to predict risk of a current or future clinical outcome are reviewed. Specifically, we discuss criteria for evaluating a risk model including: calibration, accurate classification and benefit for decision making using the model. Measures for making comparisons between models are described. The role of risk reclassification techniques is discussed. We present a detailed example.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution
Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 139.00
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 179.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

References

  1. Baker, S.: Putting risk prediction in perspective: relative utility curves. J. Natl. Cancer Inst. 101(22), 1538–1542 (2009)

    Article  Google Scholar 

  2. Baker, S., Kramer, B.: Evaluating a new marker for risk prediction: decision analysis to the rescue. Discov. Med. 14(76), 181–188 (2012)

    Google Scholar 

  3. Bura, E., Gastwirth, J.: The binary regression quantile plot: assessing the importance of predictors in binary regression visually. Biom. J. 43(1), 5–21 (2001)

    Article  MathSciNet  MATH  Google Scholar 

  4. Cook, N.: Use and misuse of the receiver operating characteristic curve in risk prediction. Circulation 115(7), 928–935 (2007)

    Article  Google Scholar 

  5. Cook, N., Ridker, P.: Advances in measuring the effect of individual predictors of cardiovascular risk: the role of reclassification measures. Ann. Intern. Med. 150(11), 795–802 (2009)

    Article  Google Scholar 

  6. Gail, M., Costantino, J.: Validating and improving models for projecting the absolute risk of breast cancer. J. Natl. Cancer Inst. 93(5), 334–335 (2001)

    Article  Google Scholar 

  7. Gail, M., Pfeiffer, R.: On criteria for evaluating models of absolute risk. Biostatistics 6(2), 227–239 (2005)

    Article  MATH  Google Scholar 

  8. Gu, W., Pepe, M.: Measures to summarize and compare the predictive capacity of markers. Int. J. Biostat. 5(1), Article 27 (2009). doi:10.2202/1557-4679.1188

    Google Scholar 

  9. Harrell, F.: Regression Modeling Strategies: With Applications to Linear Models, Logistic Regression, and Survival Analysis. Springer, New York (2001)

    Book  Google Scholar 

  10. Hastie, T., Tibshirani, R., Friedman, J.: The Elements of Statistical Learning. Springer, New York (2001)

    Book  MATH  Google Scholar 

  11. Huang, Y., Pepe, M.: A parametric ROC model-based approach for evaluating the predictiveness of continuous markers in case-control studies. Biometrics 65(4), 1133–1144 (2009)

    Article  MathSciNet  MATH  Google Scholar 

  12. Huang, Y., Pepe, M.S.: Semiparametric methods for evaluating the covariate-specific predictiveness of continuous markers in matched case-control studies. J. R. Stat. Soc., Ser. C (Appl. Stat.) 59(3), 437–456 (2010)

    Google Scholar 

  13. Janes, H., Pepe, M.S., Bossuyt, P.M., Barlow, W.E.: Measuring the performance of markers for guiding treatment decisions. Ann. Intern. Med. 154, 253–259 (2011)

    Article  Google Scholar 

  14. Kerr, K.F., McClelland, R.L., Brown, E.R., Lumley, T.: Evaluating the incremental value of new biomarkers with integrated discrimination improvement. Am. J. Epidemiol. 174(3), 364–374 (2011)

    Article  Google Scholar 

  15. Lemeshow, S., Hosmer Jr, D.: A review of goodness of fit statistics for use in the development of logistic regression models. Am. J. Epidemiol. 115(1), 92–106 (1982)

    Google Scholar 

  16. Parikh, C.R., Devarajan, P., Zappitelli, M., Sint, K., Thiessen-Philbrook, H., Li, S., Kim, R.W., Koyner, J.L., Coca, S.G., Edelstein, C.L., Shlipak, M.G., Garg, A.X., Krawczeski, C.D., TRIBE-AKI Consortium: Postoperative biomarkers predict acute kidney injury and poor outcomes after pediatric cardiac surgery. J. Am. Soc. Nephrol. 22(9), 1737–1747 (2011)

    Article  Google Scholar 

  17. Pencina, M., D’Agostino, R., D’Agostino, R., Vasan, R.: Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat. Med. 27(2), 157–172 (2008)

    Article  MathSciNet  Google Scholar 

  18. Pencina, M., D’Agostino Sr, R., Steyerberg, E.: Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat. Med. 30(1), 11–21 (2011)

    Article  MathSciNet  Google Scholar 

  19. Pepe, M.: Problems with risk reclassification methods for evaluating prediction models. Am. J. Epidemiol. 173(11), 1327 (2011)

    Article  Google Scholar 

  20. Pepe, M., Janes, H.: Gauging the performance of SNPs, biomarkers, and clinical factors for predicting risk of breast cancer. J. Natl. Cancer Inst. 100(14), 978–979 (2008)

    Article  Google Scholar 

  21. Pepe, M., Feng, Z., Gu, J.: Comments on ‘Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond’ by MJ Pencina et al. Stat. Med. 27(2), 173–181 (2008). doi:10.1002/sim.2929

    Article  MathSciNet  Google Scholar 

  22. Pepe, M., Feng, Z., Huang, Y., Longton, G., Prentice, R., Thompson, I., Zheng, Y.: Integrating the predictiveness of a marker with its performance as a classifier. Am. J. Epidemiol. 167(3), 362 (2008)

    Article  Google Scholar 

  23. Pepe, M., Kerr, K., Longton, G., Wang, Z.: Testing for improvement in prediction model performance. Stat. Med. 32(9), 1467–1482 (2013)

    Article  MathSciNet  Google Scholar 

  24. Pfeiffer, R., Gail, M.: Two criteria for evaluating risk prediction models. Biometrics 67(3), 1057–1065 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  25. Sargent, D.J., Conley, B.A., Allegra, C., Collette, L.: Clinical trial designs for predictive marker validation in cancer treatment trials. J. Clin. Oncol. 23(9), 2020–2027 (2005)

    Article  Google Scholar 

  26. Seymour, C.W., Kahn, J.M., Cooke, C.R., Watkins, T.R., Heckbert, S.R., Rea, T.D.: Prediction of critical illness during out-of-hospital emergency care. JAMA 304(7), 747–754 (2010)

    Article  Google Scholar 

  27. Steyerberg, E.: Clinical Prediction Models: A Practical Approach to Development, Validation, and Updating. Springer, New York (2009)

    Book  Google Scholar 

  28. Steyerberg, E., Borsboom, G., van Houwelingen, H., Eijkemans, M., Habbema, J.: Validation and updating of predictive logistic regression models: a study on sample size and shrinkage. Stat. Med. 23(16), 2567–2586 (2004)

    Article  Google Scholar 

  29. Vickers, A., Elkin, E.: Decision curve analysis: a novel method for evaluating prediction models. Med. Decis. Mak. 26(6), 565 (2006)

    Article  Google Scholar 

  30. Vickers, A.J., Kattan, M.W., Daniel, S.: Method for evaluating prediction models that apply the results of randomized trials to individual patients. Trials 5, 8–14 (2007)

    Google Scholar 

  31. Vickers, A.J., Cronin, A.M., Begg, C.B.: One statistical test is sufficient for assessing new predictive markers. BMC Med. Res. Methodol. 11, 13 (2011)

    Article  Google Scholar 

  32. Wilson, P., D’Agostino, R., Levy, D., Belanger, A., Silbershatz, H., Kannel, W.: Prediction of coronary heart disease using risk factor categories. Circulation 97(18), 1837–1847 (1998)

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Fred Hutchinson Cancer Research Center, Seattle, Washington, USA

    Margaret Pepe & Holly Janes

  2. University of Washington, Seattle, Washington, USA

    Margaret Pepe

Authors
  1. Margaret Pepe
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Holly Janes
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Margaret Pepe .

Editor information

Editors and Affiliations

  1. University of Maryland, College Park, Maryland, USA

    Mei-Ling Ting Lee

  2. National Cancer Institute Div. Cancer Epidemiology & Genetics, Bethesda, Maryland, USA

    Mitchell Gail

  3. National Cancer Institute, Bethesda, Maryland, USA

    Ruth Pfeiffer

  4. Centers for Disease Control and Prevention, Atlanta, Georgia, USA

    Glen Satten

  5. Department of Biostatistics, Harvard School of Public Health, Boston, Massachusetts, USA

    Tianxi Cai

  6. Department of Mathematics, Imperial College London, London, United Kingdom

    Axel Gandy

Rights and permissions

Reprints and permissions

Copyright information

© 2013 Springer Science+Business Media New York

About this paper

Cite this paper

Pepe, M., Janes, H. (2013). Methods for Evaluating Prediction Performance of Biomarkers and Tests. In: Lee, ML., Gail, M., Pfeiffer, R., Satten, G., Cai, T., Gandy, A. (eds) Risk Assessment and Evaluation of Predictions. Lecture Notes in Statistics, vol 215. Springer, New York, NY. https://doi.org/10.1007/978-1-4614-8981-8_7

Download citation

Publish with us

Policies and ethics

Search

Navigation