Introduction

Growing concerns about the quality of care and patient safety have increased the importance of monitoring intensive care units (ICUs) in health care organizations. In response to increasing demands to improve the quality of care, performance measures for intensive care have been developed [13]. As patient mortality is a major health care outcome, many studies have included this measure as a quality outcome indicator. However, mortality rates vary among ICUs due to differences in patient case mix and disease severity [46]. Several risk-adjustment models [1] have been developed to compare ICU mortality rates among institutions; these models include the Acute Physiology and Chronic Health Evaluation (APACHE) system, the Mortality Prediction Model (MPM), and the Simplified Acute Physiology Score (SAPS). Additionally, Render et al. [7] have proposed an automated ICU risk-adjustment tool, and the Critical Care Outcome Prediction Equation (COPE) model was developed as a hospital mortality prediction model using only administrative data [8].

There is an increasing demand for performance measurement in ICU benchmarking in health care organizations. Several indicators for ICU performance have been developed [911], which include mechanical ventilation-associated indicators such as the prevention of ventilator-associated pneumonia, protocol-driven ventilator weaning, daily sedation interruption policy, low tidal volume ventilation in acute lung injury/adult respiratory distress syndrome (ALI/ARDS), ventilator-associated pneumonia rate, average days on mechanical ventilation, and mortality (crude and severity-adjusted). Therefore, mechanically ventilated patients have been shown to be an important target for ICU performance evaluation.

The Diagnosis Procedure Combination (DPC) system in Japan was introduced in 2004, and has since become the standard method of payment in the health care financial system. Administrative data in this system include records of patient information and daily medical care. From these data, the types of all tests, medications, and procedures, as well as the use of intensive or special care and nursing services, are itemized on a daily basis. Procedures such as mechanical ventilation, renal replacement therapy, and the use of vasoactive agents are available from DPC data, and have been reported to be closely associated with mortality [1116]. However, the utilization rates and patterns of these procedures vary among intensivists, and the inclusion of data concerning these procedures may therefore support the accurate prediction of mortality.

The aims of this study were to use administrative data to develop an in-hospital mortality equation that includes patient demographics and clinical factors, and procedures administered during the ICU admission as independent variables; and to examine the viability of the equation in conducting retrospective evaluations of ICUs.

Methods

Data sources and case selection criteria

All data for the study were extracted from the Japanese DPC database, which was collected by the DPC Research Group. Data were obtained from July 1st to December 31st for each of the 3 years from 2008 to 2010. Of the participant hospitals that comprise the database, we included 437 acute care teaching hospitals with ICUs (including surgical ICUs, medical ICUs, and surgical-medical ICUs); we obtained data from 282 ICUs in 2008, 310 ICUs in 2009, and 364 ICUs in 2010. The initial study population included all patients aged ≥20 years treated in an ICU at any of the sample hospitals. We identified the time of ICU entry and the dates of ICU stay based on specific codes in the administrative data. The data did not indicate whether a patient had been previously hospitalized in another ICU, but critical care patients are rarely transferred from one center to another in Japan. We therefore assumed that patients entering the ICU had not been transferred from another ICU.

Because the focus of this analysis was to develop accurate mortality prediction equations on mechanically ventilated patients undergoing critical care interventions, we analyzed patients who required mechanical ventilation ≥2 days after ICU entry, which was identified from the corresponding codes. Non-invasive positive pressure ventilation was not included in the analysis.

Development of the prediction model and potential risk factors

We utilized a split-half approach to prediction model development, using data from 2008 as the test dataset and data from 2009 and 2010 as the validation datasets. The primary measure used was in-hospital mortality. The in-hospital mortality prediction equation was constructed using the test dataset and evaluated using the validation datasets. Coefficients obtained from the test dataset were applied to cases in the validation datasets to calculate the predicted mortality.

Model development was based on up to 9 variables (Table 1). Age was used as a continuous variable. To determine the reason for ICU entry, patients who underwent surgery on the first ICU day were considered to be surgical cases. In these cases, patients who had both emergency hospital admission and underwent surgery on the day of hospital admission or the following day were defined as emergency surgical cases, whereas those who did not undergo emergency surgery were defined as elective surgical cases. All other patients were considered to be medical cases. To define admission categories, items in the administrative database pertaining to the course of admission were used. The emergency admission category indicates hospital admission after transport by ambulance or an unexpected admission. Organ failure was identified according to the study conducted by Angus et al. [17]. The DPC system in Japan utilizes the International Classification of Diseases, 10th Revision (ICD-10) coding, rather than ICD-9 CM or ICD-10 AM, and ICD-10 AM codes were therefore translated to ICD-10 codes for identification of primary diagnosis and organ failure (Table 2). The Charlson score is a clinical comorbidity index that predicts the 10-year survival of a patient based on a range of comorbid conditions (e.g., heart disease, cancer, and liver disease). The score can be calculated from ICD codes available from administrative data, and increasing scores have been shown to have strong associations with mortality [18, 19].

Table 1 Candidate variables used to develop the in-hospital mortality prediction equation
Table 2 Recoding of ICD-10 AM and ICD-9 CM codes to ICD-10 codes

Renal replacement therapy and pressors/vasoconstrictors were included in the candidate variables due to their reported association with 28-day hospital mortality in a previous study [20]. Renal replacement therapy included continuous renal replacement therapy, intermittent renal replacement therapy, plasma absorption, and plasma exchange, but excluded peritoneal dialysis due to its rare utilization for ICU patients. Pressors/vasoconstrictors included dopamine, dobutamine, norepinephrine, epinephrine, and vasopressin, but excluded the use of epinephrine in cardiopulmonary resuscitation. We were unable to distinguish whether dopamine was given as a low (renal) dose or for cardiovascular support, but found no evidence to support the possibility that low-dose dopamine was used [21]. We therefore assumed that dopamine was used for cardiovascular support.

Relationships between the individual candidate variables and in-hospital mortality were analyzed with χ 2 tests or Student t tests using the test dataset, depending on the type of data. After exclusion of variables with P > 0.25, the remaining variables were subjected to multiple logistic regression analyses (stepwise backward selection method). The model was constructed using variables with P < 0.05, and the C-statistics were calculated. Two models were developed: Model 1 included independent variables of demographics and clinical factors, and Model 2 included independent variables of procedures and therapies administered at any time during ICU admission (in addition to the variables in Model 1).

Prediction model performance

Calibration of the model was evaluated using the Hosmer–Lemeshow χ 2 test. A well-calibrated model has a low χ 2 value (<15.5; df = 8) and a high P value (>0.05). The discrimination of the model was assessed by the C-statistics, for which a value 0.9–1.0 was determined to represent high accuracy, 0.7–0.9 moderate accuracy, and 0.5–0.7 low accuracy.

Prediction model validation

The mortality prediction equation was cross-validated using the validation datasets to demonstrate the predictive validity of the prediction equation obtained from multiple logistic regression analysis of the test dataset. Coefficients derived from analysis of the test dataset were applied to the validation dataset in order to calculate predicted mortality. The performance of the equation was tested using the C-statistics [95 % confidence interval (CI)] for the validation datasets from 2009 and 2010. All statistical analyses were performed using SPSS 18.0J (SPSS Inc., Chicago, IL, USA).

Results

The final samples for analysis from 2008, 2009 and 2010 comprised the following: the numbers of hospitals per year were 282, 310 and 364, respectively; numbers of ICU patients were 38,625, 71,243 and 49,230, respectively; numbers of ventilated patients were 5,807, 10,610, and 7,576, respectively. The incidence of mechanically ventilated patients in ICU entry was approximately 15 % for all 3 years. The details of patient characteristics for the sample are shown in Table 3. Preliminary univariate analyses showed significant differences between survivors and non-survivors in all patient characteristics except for gender in the three study years. In 2008 (test dataset), intracranial haemorrhage was the most frequent condition in the diagnostic category for non-survivors (10.4 %), while myocardial ischemia was the most frequent condition for survivors (13.2 %). All patient characteristics (excluding gender) were thus included as independent variables in the consequent multiple logistic analysis.

Table 3 Patient characteristics in 2008, 2009, and 2010

Coefficients of the variables, odds ratios (OR), and 95 % CIs are shown in Table 4. Factors associated with a high risk of death in Models 1 and 2 were hemopoietic malignancy (OR = 4.92, 95 % CI = 2.15–11.24) and lung malignancy (OR = 6.00, 95 % CI = 3.41–10.55), respectively. The Hosmer–Lemeshow χ 2 values (P values) in Models 1 and 2 were 11.9 (0.15) and 15.6 (0.05), respectively; the models’ C-statistics (95 % CI) in the test dataset were 0.70 (0.69–0.72) and 0.78 (0.77–0.79), respectively (Table 5). Applying the final equation to the validation dataset showed similar discrimination when compared with that of the test dataset.

Table 4 Variable coefficients used in the hospital mortality prediction models
Table 5 Model discrimination for the prediction equation in test and validation datasets

Discussion

In this study, we developed and validated in-hospital mortality prediction equations in mechanically ventilated patients using data from Japan; we showed moderate discrimination in a model using patient demographics, clinical factors, and procedures administered during the ICU admission (Model 2). Mechanically ventilated patients in the ICU are frequently the subjects of epidemiological studies [2224]. The risk-adjustment methods previously employed in these studies frequently include APACHE, MPM and SAPS [2536], but these methods are primarily dependent on organ scores that require physiological data. Ohno-Machado et al. [25] obtained C-statistics for mortality prediction models using APACHE-II, APACHE-III, MPM0, MPM24, MPM-II0, MPM-II24, SAPS, and SAPS-II; and found that these all had C-statistics ≥0.8, except for SAPS. In contrast to these models, Duke et al. [8] derived the COPE model using solely administrative data. This model has advantages in that it can predict mortality with relatively few variables from routinely-available data. The COPE model also includes mechanical ventilation as an intensive care therapy which has been shown to be strongly associated with hospital mortality. Therefore, for an analysis focusing on mechanically ventilated patients using DPC data, there was a need to develop a new robust mortality prediction tool that did not include the independent variable of mechanical ventilation. In addition, because these models were developed specifically using Japanese data, and because of the ability of the DPC database to identify the details of medical care for mechanically ventilated patients in a uniform format from numerous hospitals, this approach has the capacity to support comparative evaluations of ICU performance using multicenter analysis in Japan.

Model 2 from our analysis may serve as a possible alternative model due to displaying moderate accuracy in the C-statistics. If the prognoses of mechanically ventilated patients are to be required, the use of existing scoring systems using physiological data may be more useful. But the intended applications of this equation (Model 2) lie in the retrospective evaluations of ICU performance in a multicenter analysis based on the identical format of DPC data introduced in 2004 in Japan.

The prediction equation in this study has the following advantages over existing models: the variables used in our equation utilize information that can be routinely obtained from administrative data. These variables are submitted by doctors and nurses in a timely manner on a daily basis rather than at or after discharge, which ostensibly improves the reliability of the data. Also, the model uses only 8 variables, which facilitates its generalizability and application. However, there is also the risk of coding errors, especially in ICU patients [8].

There are several limitations in the present study. First, we did not compare our model with scoring systems using physiological data, since our data did not include severity scores. Therefore, we cannot determine the relative accuracy of the model compared with other systems. Second, the administrative data include information on a ‘calendar day’ basis, rather than an hourly basis, and therefore the first ICU day was defined by a calendar day. This provides no distinction regarding the use of renal replacement therapy and pressors/vasoconstrictors before or after ICU entry on the first ICU day. However, these resources are mostly used under monitoring in the ICU. Third, the indications for renal replacement therapy and pressors/vasoconstrictors varied among the hospitals, which may have resulted in therapeutic bias. Fourth, the administrative data do not indicate if renal replacement therapy was given for chronic or acute renal failure, or for a non-renal indication; or if pressors/vasoconstrictors were used to treat hypovolemic or septic shock. Fifth, different admission criteria among the ICUs could have produced a selection bias that affected mortality. Our model has a therapeutic bias similar to that of the COPE model, including the use of mechanical ventilation, renal replacement therapy, and pressors/vasoconstrictors. However, it is likely that there is little if any inappropriate application of these therapies due to ethical considerations. Finally, since our study sample included approximately 30 % of all ICUs in Japan, and did not include university hospitals and non-teaching hospitals, further verification and modification of the model may be required in a larger sample of patients and ICUs from a greater variety of hospital types.

The absence of physiological data is disadvantageous since diagnosis is not possible, but our model has the additional advantage in that it uses administrative data routinely collected for all patients with a high level of accuracy. Comparison of ICU performance using administrative data has applications for benchmarking and quality improvement, and our model establishes a method for the comparative evaluation of ICU performance.

Conclusions

The hospital mortality prediction equation for mechanically ventilated patients in intensive care proposed in this study is based solely on administrative data, and uses a relatively small number of variables that can be easily collected. In addition to the COPE model, Model 2 can be used to evaluate illness severity of mechanically ventilated patients based on administrative data and may be applicable to future critical care studies.