Chest

Chest

Volume 115, Issue 3, March 1999, Pages 793-801
Journal home page for Chest

Clinical Investigations in Critical Care
Community-Wide Assessment of Intensive Care Outcomes Using a Physiologically Based Prognostic Measure: Implications for Critical Care Delivery From Cleveland Health Quality Choice

https://doi.org/10.1378/chest.115.3.793Get rights and content

Study objectives

To examine the applicability of a previously developed intensive care prognostic measure to a community-based sample of hospitals, and assess variations in severity-adjusted mortality across a major metropolitan region.

Setting

Twenty-eight hospitals with 38 ICUs participating in a community-wide initiative to measure performance supported by the business community, hospitals, and physicians.

Patients

Included in the study were 116,340 consecutive eligible patients admitted to medical, surgical, neurologic, and mixed medical/surgical ICUs between March 1, 1991, and March 31, 1995.

Main outcome measures

The risk of hospital mortality was assessed using a previous risk prediction equation that was developed in a national sample, and a reestimated logistic regression model fit to the current sample. The standardized mortality ratio (SMR) (actual/predicted mortality) was used to describe hospital performance.

Results

Although discrimination of the previous national risk equation in the current sample was high (receiver operating characteristic [ROC] curve area = 0.90), the equation systematically overestimated the risk of death and was not as well calibrated (Hosmer-Lemeshow statistic, 2407.6, 8 df, p < 0.001). The locally derived equation had similar discrimination (ROC curve area = 0.91), but had improved calibration across all ranges of severity (Hosmer-Lemeshow statistic = 13.5, 8 df, p = 0.10). Hospital SMRs ranged from 0.85 to 1.21, and four hospitals had SMRs that were higher or lower (p < 0.01) than 1.0. Variation in SMRs tended to be greatest during the first year of data collection. SMRs also tended to decline over the 4 years (1.06, 1.02, 0.98, and 0.94 in years 1 to 4, respectively), as did mean hospital length of stay (13.0, 12.4, 11.6, and 11.1 days in years 1 to 4; p < 0.001). However, excluding the increasing (p < 0.001) number of patients discharged to skilled nursing facilities attenuated much of the decline in standardized mortality over time.

Conclusions

A previously validated physiologically based prognostic measure successfully stratified patients in a large community-based sample by their risk of death. However, such methods may require recalibration when applied to new samples and to reflect changes in practice over time. Moreover, although significant variations in hospital standardized mortality were observed, changing hospital discharge practices suggest that in-hospital mortality may no longer be an adequate measure of ICU performance. Community-wide efforts with broad-based support from business, hospitals, and physicians can be sustained over time to assess outcomes associated with ICU care. Such efforts may provide important information about variations in patient outcomes and changes in practice patterns over time. Future efforts should assess the impact of such community-wide initiatives on health-care purchasing and institutional quality improvement programs.

Section snippets

Hospitals

The study was conducted in 38 ICUs in 28 hospitals participating in CHQC.10 Nineteen of the study ICUs were mixed medical and surgical units, eight ICUs were medical, eight ICUs were surgical, and three were neurologic and/or neurosurgical. Thirteen additional ICUs in study hospitals that specialized in coronary care (n = 11) or cardiovascular surgery (n = 2) were excluded from the study, as per CHQC protocols. Five hospitals were members of the Council of Teaching Hospitals of the Association

Results

The mean age of study patients was 63 years and 52% were men (Table 1). Forty-one percent of patients were admitted through the emergency department; 37% of patients (n = 42,416) were postoperative (ie, admitted to the ICU after undergoing a surgical procedure), and 63% of patients (n = 73,924) were nonoperative. The 10 most common ICU admission diagnoses accounted for nearly 50% of admissions and included the following: angina (n = 10,046); congestive heart failure (n = 8,007); trauma to the

Discussion

The current study represents one of the largest evaluations of variations in ICU mortality, and the first study (to our knowledge) to include all hospitals providing critical care services in a single metropolitan region. In analyses of > 116,000 patients admitted to ICUs in 28 hospitals over a 4-year period, several important findings emerge. First, an existing ICU risk stratification tool can be successfully implemented in a diverse spectrum of hospitals, as part of an ongoing collaboration

References (53)

  • S Tilley et al.

    Data initiatives: building a managed care network utilizing clinical performance data

    Am J Med Qual

    (1996)
  • J Wennberg et al.

    Small area variations in health care delivery

    Science

    (1973)
  • MR Chassin et al.

    Variations in the use of medical and surgical services by the Medicare population

    N Engl J Med

    (1986)
  • AL Siu et al.

    Inappropriate use of hospitals in a randomized trial of health insurance plans

    N Engl J Med

    (1986)
  • JE Wennberg et al.

    The Dartmouth atlas of health care

    (1998)
  • W McClure

    Competition and the pursuit of quality: a conversation with Walter McClure [interview by John K. Iglehart]

    Health Affairs

    (1988)
  • DV Shaller et al.

    Reforming the market for value in health care: the Cleveland experience

    J Occup Med

    (1991)
  • R Winslow

    Cincinnati firms cutting costs with hospital-ranking system

    Wall Street Journal

    (1993)
  • WA Knaus et al.

    Short-term mortality predictions for critically ill hospitalized adults: science and ethics

    Science

    (1991)
  • S Lemeshow et al.

    Modeling the severity of illness of ICU patients: a systems update

    JAMA

    (1994)
  • SM Shortell et al.

    Organizational assessment in intensive care units (ICUs): construct development, reliability, and validity of the ICU nurse-physician questionnaire

    Med Care

    (1991)
  • MS Blois

    Medicine and the nature of vertical reasoning

    N Engl J Med

    (1988)
  • GE Rosenthal et al.

    Use of intensive care units for patients with low severity of illness

    Arch Intern Med

    (1998)
  • WA Knaus et al.

    Variations in mortality and length of stay in intensive care units

    Ann Intern Med

    (1993)
  • JA Hanley et al.

    The meaning and use of the area under a receiver operating characteristic (ROC) curve

    Radiology

    (1982)
  • LI Iezzoni et al.

    Illness severity and costs of admissions at teaching and nonteaching hospitals

    JAMA

    (1990)

    • Cited by (111)

    • Predictive Modeling Using Artificial Intelligence and Machine Learning Algorithms on Electronic Health Record Data: Advantages and Challenges

      2023, Critical Care Clinics

    • A novel mortality prediction model for the current population in an adult intensive care unit

      2018, Heart and Lung: Journal of Acute and Critical Care
      Citation Excerpt :

      Conventional predictive scoring models may not be well-calibrated in various geographical regions and often display lack of fit when evaluated in different critical care populations. This contributes to failure to predict actual mortality with an accepted degree of accuracy.5–13 For instance, in a surgical ICU, the Acute Physiology and Chronic Health Evaluation (APACHE) II14 and the Simplified Acute Physiology Score (SAPS) III15 showed poor calibration.

    • Predictors of early postdischarge mortality in critically ill patients: A retrospective cohort study from the California Intensive Care Outcomes project

      2011, Journal of Critical Care

    • A model for identifying patients who may not need intensive care unit admission

      2010, Journal of Critical Care
      Citation Excerpt :

      Data used to develop and internally validate the APACHE IV risk for active treatment model were collected from January 1, 2002, to December 31, 2003, for a nonrandomized observational cohort of 131 615 admissions to 104 ICUs at 45 US hospitals. Details about the characteristics of these hospitals, ICUs, and patients have been described elsewhere [19]. Of these 104 ICUs, 56 (53.8%) recorded active treatment data only on ICU day 1 and were excluded from the analysis.

    • Severity of disease scoring systems and mortality after non‐cardiac surgery

      2018, Brazilian Journal of Anesthesiology

    • Benchmarking, public reporting, and pay-for-performance: A mixed-methods survey of California pediatric intensive care unit medical directors

      2011, Pediatric Critical Care Medicine
      Citation Excerpt :

      However, quality improvement efforts and quality comparisons between doctors and hospitals have elicited concerns about physician autonomy (20), the quality of physician-patient relationships (16), and accuracy in conveying quality assessments to the public. Benchmarking has been widely applied across ICUs in recent years as part of more comprehensive efforts to improve quality of critical care, reduce medical errors, and minimize ICU costs (21–24). Severity-adjusted mortality rates and length of stay between different ICUs is a standard benchmark for both adult and pediatric populations (25, 26).

    View all citing articles on Scopus

    Dr. Rosenthal was supported by a Career Development Award from the Health Services Research and Development Service, US Department of Veterans Affairs.

    Financial disclosure: Drs. Sirio and Harper have provided consultingservices to APACHE Medical Systems, Inc (AMS). Dr. Harper is Executive Director of the Cleveland Health Quality Choice Program. AMS holds thecommercial copyright on the hospital mortality equations. APACHE and APACHE III are trademarks of AMS. While both the equations and the APACHE database are protected by commercial copyright, they areavailable to researchers for independent verification and furtheranalysis by contacting the authors or AMS.

    View full text
    Copyright © 1999 The American College of Chest Physicians. Published by Elsevier Inc. All rights reserved.