Skip to main content
Springer Nature Link
Log in

External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study

  • Original
  • Published:
Intensive Care Medicine Aims and scope Submit manuscript

Abstract

Objective

External validation of three prognostic models in adult intensive care patients in South England.

Design

Prospective cohort study.

Setting

Seventeen intensive care units (ICU) in the South West Thames Region in South England.

Patients and participants

Data of 16,646 patients were analysed.

Interventions

None.

Measurements and results

We compared directly the predictive accuracy of three prognostic models (SAPS II, APACHE II and III), using formal tests of calibration and discrimination. The external validation showed a similar pattern for all three models tested: good discrimination, but imperfect calibration. The areas under the receiver operating characteristics (ROC) curves, used to test discrimination, were 0.835 and 0.867 for APACHE II and III, and 0.852 for the SAPS II model. Model calibration was assessed by Lemeshow-Hosmer C-statistics and was Χ2 =232.1 for APACHE II, Χ2 =443.3 for APACHE III and Χ2 =287.5 for SAPS II.

Conclusions

Disparity in case mix, a higher prevalence of outcome events and important unmeasured patient mix factors are possible sources for the decay of the models' predictive accuracy in our population. The lack of generalisability of standard prognostic models requires their validation and re-calibration before they can be applied with confidence to new populations. Customisation of existing models may become an important strategy to obtain authentic information on disease severity, which is a prerequisite for reliably measuring and comparing the quality and cost of intensive care.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1.

Similar content being viewed by others

Explore related subjects

Discover the latest articles and news from researchers in related subjects, suggested using machine learning.

References

  1. Le Gall JR, Lemeshow S, Saulnier F (1993) Development of a new scoring system, the SAPS II, from a European/North American multicenter study. JAMA 270:2957–2963

    PubMed  Google Scholar 

  2. Knaus WA, Draper EA, Wagner DP et al. (1985) APACHE II: a severity of disease classification system. Crit Care Med 10:818–829

    Google Scholar 

  3. Knaus WA, Wagner DP, Draper EA et al. (1991) The APACHE III prognostic system: risk prediction of hospital mortality for critically ill hospitalized adults. Chest 100:1619–1639

    CAS  PubMed  Google Scholar 

  4. Wong DT, Crofts SL, Gomez M, McGuire GP, Byrick RJ (1995) Evaluation of predictive ability of APACHE II system and hospital outcome in Canadian intensive care patients. Crit Care Med 23:1177–1183

    Google Scholar 

  5. Sirio CA, Tajimi K, Tase C, Knaus WA, Wagner DP, Hirasawa H et al. (1992) An initial comparison of intensive care in Japan and the United States. Crit Care Med 20:1207–1215

    Google Scholar 

  6. Markgraf R, Deutschinoff G, Pientka L, Scholten T (2000) Comparison of Acute Physiology and Chronic Health Evaluation score II and III and Simplified Acute Physiology Score II: a prospective cohort study evaluating these models to predict outcome in a German multidisciplinary intensive care unit. Crit Care Med 28:26–33

    CAS  PubMed  Google Scholar 

  7. Castella X, Artigas A, Bion J, Kari A (1995) A comparison of severity of illness scoring systems for intensive care unit patients: results of a multicenter, multinational study. Crit Care Med 23:1327–1335

    Google Scholar 

  8. Apolone G, Bertolini G, D'Amico R, Iapichino G, Cattanco A, De Salvo G, Melotti RM (1996) The performance of SAPS II in a cohort of Italian ICUs: results from GiViTI. Intensive Care Med 22:1368–1378

    CAS  PubMed  Google Scholar 

  9. Moreno R, Reis Miranda D, Fidler V, Van Schilfgaarde R (1998) Evaluation of two outcome prediction models on an independent database. Crit Care Med 26:50–61

    Google Scholar 

  10. Rowan KM, Kerr JH, Major E, Mc Pherson K, Short A, Vesssey MP (1993) Intensive Care Society's APACHE II study in Britain and Ireland II: outcome comparisons of intensive care units after adjustment for case mix by the American APACHE II method. BMJ 307:977–981

    CAS  PubMed  Google Scholar 

  11. Zimmerman JE, Wagner DP, Draper EA et al. (1998) Evaluation of Acute Physiology and Chronic Health III predictions of hospital mortality in an independent database. Crit Care Med 20:1317–1326

    Google Scholar 

  12. Pappachan JV, Millar B, Bennett D, Smith GB (1999) Comparison of outcome from intensive care admission after adjustment for case mix by the APACHE III prognostic system. Chest 115:802–805

    CAS  PubMed  Google Scholar 

  13. Bastios PG, Sun X, Wagner DP, Knaus WA, Zimmerman JE (1996) Application of the APACHE III prognostic system in Brazilian intensive care units: a prospective multicenter study. Intensive Care Med 22:564–570

    Article  PubMed  Google Scholar 

  14. Rowan KM, Kerr JH, Major E et al. (1994) Intensive Care Society's Acute Physiology and Chronic Health Evaluation (APACHE II) study in Britain and Ireland: a prospective, multicenter, cohort study comparing two methods for predicting outcome for adult intensive care patients. Crit Care Med 22:1392–1401

    Google Scholar 

  15. Livingston BM, MacKirdy FN, Howie JC, Jones R, Norrie JD (2000) Assessment of the performance of five intensive care scoring models within a large Scottish database. Crit Care Med 28:1820–1827

    CAS  PubMed  Google Scholar 

  16. Beck DH, Taylor BL, Millar B, Smith GB (1997) Prediction of outcome from intensive care: a prospective cohort study comparing APACHE II and III in a UK intensive care unit. Crit Care Med 25:9–15

    CAS  PubMed  Google Scholar 

  17. Hanley JA, McNeil BJ (1982) The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology 143:29–36

    CAS  PubMed  Google Scholar 

  18. Hosmer DW, Lemeshow S (1989) Applied logistic regression. John Wiley, New York, pp 135–149

  19. Lemeshow S, Hosmer DW (1982) A review of the goodness of fit statistics for use in the development of logistic regression models. Am J Epidemiol 115: 92–106

    CAS  PubMed  Google Scholar 

  20. Altman DC, Royston P (2000) What do we mean by validating prognostic models? Stat Med 19:453–473

    Article  CAS  PubMed  Google Scholar 

  21. Harrell FE, Lee K (1984) Regression modelling strategies for improved prognostic prediction. Stat Med 3:143–152

    PubMed  Google Scholar 

  22. Goldhill DR, Withington PS (1996) The effect of case mix adjustment on mortality as predicted by APACHE II. Intensive Care Med 22:415–419

    CAS  PubMed  Google Scholar 

  23. Rowan KM, Kerr JH, Major E et al. (1993) Intensive Care Society's APACHE II study in Britain and Ireland I: variations in case mix of adult admissions to general intensive care units and impact on outcome. BMJ 307:972–977

    CAS  PubMed  Google Scholar 

  24. Moreno R, Apolone G, Reis Miranda D (1998) Evaluation of the uniformity of fit of general outcome prediction models. Intensive Care Med 24:40–47

    CAS  PubMed  Google Scholar 

  25. Metnitz PGH, Vesely H, Valentin A, Popow C, Hiesmayr M, Lenz K, Krenn CG Steltzer H (1999) Evaluation of an interdisciplinary data set for national intensive care unit assessment. Crit Care Med 27:1486–1491

    CAS  PubMed  Google Scholar 

  26. Murphy-Filkins RL, Teres D, Lemeshow S, Hosmer DW (1996) Effect of changing patient mix on the performance of intensive care unit severity-of-illness models: how to distinguish a general from a speciality intensive care unit. Crit Care Med 24:1968–1973

    Google Scholar 

  27. Lemeshow S, Teres D, Klar J et al. (1993) Mortality Probability Models (MPM II) based on an international cohort of intensive care unit patients. JAMA 270:2478–2486

    CAS  PubMed  Google Scholar 

  28. Beck DH, McQuillan PJ, Smith GB (2002) Waiting for the break of dawn? The effects of discharge TISS scores, discharge time and discharge facilities on mortality after intensive care. Intensive Care Med 28:1287–1293

    PubMed  Google Scholar 

  29. Miller ME, Hui SL, Tierney WM (1991) Validation techniques for logistic regression models. Stat Med 10:1213–1226

    CAS  PubMed  Google Scholar 

  30. Zhu PG, Lemeshow S, Hosmer DW, Klar J, Avrunin J, Teres D (1996) Factors affecting the performance of the models in the Mortality Prediction Model II system and strategies of customization: a simulation study. Crit Care Med 24:57–63

    Google Scholar 

  31. Glance LG, Osler TM, Papadakos P (2000) Effect of mortality rate on the performance of the Acute Physiology and Chronic Health Evaluation II: a simulation study. Crit Care Med 28:3424–3428

    CAS  PubMed  Google Scholar 

  32. Rivera-Fernandez R, Vazquez-Mata G, Bravo M, Aguayo-Hoyos E, Zimmerman JE, Wagner DP, Knaus WA (1998) The APACHE III prognostic system: customized mortality predictions for Spanish ICU patients. Intensive Care Med 24:574–581

    CAS  PubMed  Google Scholar 

  33. Moreno R, Apolone G (1997) Impact of different customization strategies in the performance of a general severity score. Crit Care Med 25:2001–2008

    Google Scholar 

  34. Beck DH, Smith GB, Pappachan JV (2002) The effects of two methods for customising the original Simplified Acute Physiology Score (SAPS) II for intensive care patients from South England. Anaesthesia; 57:785–793

  35. Sirio CA, Shepardson LB, Rotondi AJ, Cooper GS, Angus DC, Harper DL, Rosenthal GE (1999) Community-wide assessment of intensive care outcomes using a physiologically based prognostic measure. Chest 115:793–801

    CAS  PubMed  Google Scholar 

  36. Teres D, Lemeshow S (1998) As American as apple pie and APACHE (editorial). Crit Care Med 26:1297–1298

    Google Scholar 

  37. Ridley S (2001) Critical care outcomes. Anaesthesia 56:1–3

    CAS  PubMed  Google Scholar 

  38. Smith GB, Taylor BL, McQuillan PJ, Nials E (1995) Rationing intensive care: intensive care provision varies widely in Britain. BMJ 10:1412–1413

    Google Scholar 

  39. Metcalfe A, McPherson K. Sloggett A (1997) Mortality among appropriately referred patients refused admission to intensive care units. Lancet 350:7–11

    CAS  PubMed  Google Scholar 

  40. Goldfrad C, Rowan K (2000) Consequences of discharges from intensive care at night. Lancet 355:1138–1142

    CAS  PubMed  Google Scholar 

Download references

Acknowledgements

We acknowledge the directors and staff of the following ICUs for their co-operation and enthusiasm in the collection of data for this study: Ashford Hospital, Middlesex; Crawley Hospital; East Surrey Hospital, Redhill; Epsom General Hospital; Frimley Park Hospital; Haywards Heath Hospital; Kingston Hospital; Mayday Hospital, Croydon; Queen Mary's Hospital, Roehampton; Royal Surrey County Hospital, Guildford; St. Helier Hospital, Carshalton; St. Peter's Hospital, Chertsey; St. Richard's Hospital, Chichester; Worthing Hospital; General ICU, St. George's Hospital, Tooting; Queen Alexandra Hospital, Portsmouth and St. Mary's Hospital, Portsmouth.

Author information

Author notes

  1. Dieter H. Beck

    Present address: Lindauerstr.10, 10781, Berlin, Germany

Authors and Affiliations

  1. Department of Anaesthesiology and Intensive Care, Charité Hospital, Humboldt University, Schumannstrasse 20-21, 10098, Berlin, Germany

    Dieter H. Beck

  2. Department of Intensive Care Medicine, Queen Alexandra Hospital, Portsmouth, Hampshire, PO6 3LY, UK

    Gary B. Smith

  3. Department of Anaesthesia, Southhampton General Hospital, Southampton, UK

    John V. Pappachan

  4. Critical Audit Ltd., St.George's Hospital Medical School, London, UK

    Brian Millar

Authors
  1. Dieter H. Beck
    View author publications

    You can also search for this author inPubMed Google Scholar

  2. Gary B. Smith
    View author publications

    You can also search for this author inPubMed Google Scholar

  3. John V. Pappachan
    View author publications

    You can also search for this author inPubMed Google Scholar

  4. Brian Millar
    View author publications

    You can also search for this author inPubMed Google Scholar

Corresponding author

Correspondence to Dieter H. Beck.

Additional information

Work performed in the Department of Intensive Care Medicine, Portsmouth Hospitals NHS Trust, Queen Alexandra Hospital, Portsmouth PO6 3LY, Hampshire, UK.

Funding support: This study was supported by departmental funds. There was no conflict of interest.

Rights and permissions

Reprints and permissions

About this article

Cite this article

Beck, D.H., Smith, G.B., Pappachan, J.V. et al. External validation of the SAPS II, APACHE II and APACHE III prognostic models in South England: a multicentre study. Intensive Care Med 29, 249–256 (2003). https://doi.org/10.1007/s00134-002-1607-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00134-002-1607-9

Keywords