Abstract
Objective
To develop an equation model of in-hospital mortality for mechanically ventilated patients in adult intensive care using administrative data for the purpose of retrospective performance comparison among intensive care units (ICUs).
Design
Two models were developed using the split-half method, in which one test dataset and two validation datasets were used to develop and validate the prediction model, respectively. Nine candidate variables (demographics: age; gender; clinical factors hospital admission course; primary diagnosis; reason for ICU entry; Charlson score; number of organ failures; procedures and therapies administered at any time during ICU admission: renal replacement therapy; pressors/vasoconstrictors) were used for developing the equation model.
Setting
In acute-care teaching hospitals in Japan: 282 ICUs in 2008, 310 ICUs in 2009, and 364 ICUs in 2010.
Participants
Mechanically ventilated adult patients discharged from an ICU from July 1 to December 31 in 2008, 2009, and 2010. Main Outcome Measures: The test dataset consisted of 5,807 patients in 2008, and the validation datasets consisted of 10,610 patients in 2009 and 7,576 patients in 2010. Two models were developed: Model 1 (using independent variables of demographics and clinical factors), Model 2 (using procedures and therapies administered at any time during ICU admission in addition to the variables in Model 1). Using the test dataset, 8 variables (except for gender) were included in multiple logistic regression analysis with in-hospital mortality as the dependent variable, and the mortality prediction equation was constructed. Coefficients from the equation were then tested in the validation model.
Results
Hosmer–Lemeshow χ 2 are values for the test dataset in Model 1 and Model 2, and were 11.9 (P = 0.15) and 15.6 (P = 0.05), respectively; C-statistics for the test dataset in Model 1and Model 2 were 0.70 and 0.78, respectively. In-hospital mortality prediction for the validation datasets showed low and moderate accuracy in Model 1 and Model 2, respectively.
Conclusions
Model 2 may potentially serve as an alternative model for predicting mortality in mechanically ventilated patients, who have so far required physiological data for the accurate prediction of outcomes. Model 2 may facilitate the comparative evaluation of in-hospital mortality in multicenter analyses based on administrative data for mechanically ventilated patients.
Similar content being viewed by others
Introduction
Growing concerns about the quality of care and patient safety have increased the importance of monitoring intensive care units (ICUs) in health care organizations. In response to increasing demands to improve the quality of care, performance measures for intensive care have been developed [1–3]. As patient mortality is a major health care outcome, many studies have included this measure as a quality outcome indicator. However, mortality rates vary among ICUs due to differences in patient case mix and disease severity [4–6]. Several risk-adjustment models [1] have been developed to compare ICU mortality rates among institutions; these models include the Acute Physiology and Chronic Health Evaluation (APACHE) system, the Mortality Prediction Model (MPM), and the Simplified Acute Physiology Score (SAPS). Additionally, Render et al. [7] have proposed an automated ICU risk-adjustment tool, and the Critical Care Outcome Prediction Equation (COPE) model was developed as a hospital mortality prediction model using only administrative data [8].
There is an increasing demand for performance measurement in ICU benchmarking in health care organizations. Several indicators for ICU performance have been developed [9–11], which include mechanical ventilation-associated indicators such as the prevention of ventilator-associated pneumonia, protocol-driven ventilator weaning, daily sedation interruption policy, low tidal volume ventilation in acute lung injury/adult respiratory distress syndrome (ALI/ARDS), ventilator-associated pneumonia rate, average days on mechanical ventilation, and mortality (crude and severity-adjusted). Therefore, mechanically ventilated patients have been shown to be an important target for ICU performance evaluation.
The Diagnosis Procedure Combination (DPC) system in Japan was introduced in 2004, and has since become the standard method of payment in the health care financial system. Administrative data in this system include records of patient information and daily medical care. From these data, the types of all tests, medications, and procedures, as well as the use of intensive or special care and nursing services, are itemized on a daily basis. Procedures such as mechanical ventilation, renal replacement therapy, and the use of vasoactive agents are available from DPC data, and have been reported to be closely associated with mortality [11–16]. However, the utilization rates and patterns of these procedures vary among intensivists, and the inclusion of data concerning these procedures may therefore support the accurate prediction of mortality.
The aims of this study were to use administrative data to develop an in-hospital mortality equation that includes patient demographics and clinical factors, and procedures administered during the ICU admission as independent variables; and to examine the viability of the equation in conducting retrospective evaluations of ICUs.
Methods
Data sources and case selection criteria
All data for the study were extracted from the Japanese DPC database, which was collected by the DPC Research Group. Data were obtained from July 1st to December 31st for each of the 3 years from 2008 to 2010. Of the participant hospitals that comprise the database, we included 437 acute care teaching hospitals with ICUs (including surgical ICUs, medical ICUs, and surgical-medical ICUs); we obtained data from 282 ICUs in 2008, 310 ICUs in 2009, and 364 ICUs in 2010. The initial study population included all patients aged ≥20 years treated in an ICU at any of the sample hospitals. We identified the time of ICU entry and the dates of ICU stay based on specific codes in the administrative data. The data did not indicate whether a patient had been previously hospitalized in another ICU, but critical care patients are rarely transferred from one center to another in Japan. We therefore assumed that patients entering the ICU had not been transferred from another ICU.
Because the focus of this analysis was to develop accurate mortality prediction equations on mechanically ventilated patients undergoing critical care interventions, we analyzed patients who required mechanical ventilation ≥2 days after ICU entry, which was identified from the corresponding codes. Non-invasive positive pressure ventilation was not included in the analysis.
Development of the prediction model and potential risk factors
We utilized a split-half approach to prediction model development, using data from 2008 as the test dataset and data from 2009 and 2010 as the validation datasets. The primary measure used was in-hospital mortality. The in-hospital mortality prediction equation was constructed using the test dataset and evaluated using the validation datasets. Coefficients obtained from the test dataset were applied to cases in the validation datasets to calculate the predicted mortality.
Model development was based on up to 9 variables (Table 1). Age was used as a continuous variable. To determine the reason for ICU entry, patients who underwent surgery on the first ICU day were considered to be surgical cases. In these cases, patients who had both emergency hospital admission and underwent surgery on the day of hospital admission or the following day were defined as emergency surgical cases, whereas those who did not undergo emergency surgery were defined as elective surgical cases. All other patients were considered to be medical cases. To define admission categories, items in the administrative database pertaining to the course of admission were used. The emergency admission category indicates hospital admission after transport by ambulance or an unexpected admission. Organ failure was identified according to the study conducted by Angus et al. [17]. The DPC system in Japan utilizes the International Classification of Diseases, 10th Revision (ICD-10) coding, rather than ICD-9 CM or ICD-10 AM, and ICD-10 AM codes were therefore translated to ICD-10 codes for identification of primary diagnosis and organ failure (Table 2). The Charlson score is a clinical comorbidity index that predicts the 10-year survival of a patient based on a range of comorbid conditions (e.g., heart disease, cancer, and liver disease). The score can be calculated from ICD codes available from administrative data, and increasing scores have been shown to have strong associations with mortality [18, 19].
Renal replacement therapy and pressors/vasoconstrictors were included in the candidate variables due to their reported association with 28-day hospital mortality in a previous study [20]. Renal replacement therapy included continuous renal replacement therapy, intermittent renal replacement therapy, plasma absorption, and plasma exchange, but excluded peritoneal dialysis due to its rare utilization for ICU patients. Pressors/vasoconstrictors included dopamine, dobutamine, norepinephrine, epinephrine, and vasopressin, but excluded the use of epinephrine in cardiopulmonary resuscitation. We were unable to distinguish whether dopamine was given as a low (renal) dose or for cardiovascular support, but found no evidence to support the possibility that low-dose dopamine was used [21]. We therefore assumed that dopamine was used for cardiovascular support.
Relationships between the individual candidate variables and in-hospital mortality were analyzed with χ 2 tests or Student t tests using the test dataset, depending on the type of data. After exclusion of variables with P > 0.25, the remaining variables were subjected to multiple logistic regression analyses (stepwise backward selection method). The model was constructed using variables with P < 0.05, and the C-statistics were calculated. Two models were developed: Model 1 included independent variables of demographics and clinical factors, and Model 2 included independent variables of procedures and therapies administered at any time during ICU admission (in addition to the variables in Model 1).
Prediction model performance
Calibration of the model was evaluated using the Hosmer–Lemeshow χ 2 test. A well-calibrated model has a low χ 2 value (<15.5; df = 8) and a high P value (>0.05). The discrimination of the model was assessed by the C-statistics, for which a value 0.9–1.0 was determined to represent high accuracy, 0.7–0.9 moderate accuracy, and 0.5–0.7 low accuracy.
Prediction model validation
The mortality prediction equation was cross-validated using the validation datasets to demonstrate the predictive validity of the prediction equation obtained from multiple logistic regression analysis of the test dataset. Coefficients derived from analysis of the test dataset were applied to the validation dataset in order to calculate predicted mortality. The performance of the equation was tested using the C-statistics [95 % confidence interval (CI)] for the validation datasets from 2009 and 2010. All statistical analyses were performed using SPSS 18.0J (SPSS Inc., Chicago, IL, USA).
Results
The final samples for analysis from 2008, 2009 and 2010 comprised the following: the numbers of hospitals per year were 282, 310 and 364, respectively; numbers of ICU patients were 38,625, 71,243 and 49,230, respectively; numbers of ventilated patients were 5,807, 10,610, and 7,576, respectively. The incidence of mechanically ventilated patients in ICU entry was approximately 15 % for all 3 years. The details of patient characteristics for the sample are shown in Table 3. Preliminary univariate analyses showed significant differences between survivors and non-survivors in all patient characteristics except for gender in the three study years. In 2008 (test dataset), intracranial haemorrhage was the most frequent condition in the diagnostic category for non-survivors (10.4 %), while myocardial ischemia was the most frequent condition for survivors (13.2 %). All patient characteristics (excluding gender) were thus included as independent variables in the consequent multiple logistic analysis.
Coefficients of the variables, odds ratios (OR), and 95 % CIs are shown in Table 4. Factors associated with a high risk of death in Models 1 and 2 were hemopoietic malignancy (OR = 4.92, 95 % CI = 2.15–11.24) and lung malignancy (OR = 6.00, 95 % CI = 3.41–10.55), respectively. The Hosmer–Lemeshow χ 2 values (P values) in Models 1 and 2 were 11.9 (0.15) and 15.6 (0.05), respectively; the models’ C-statistics (95 % CI) in the test dataset were 0.70 (0.69–0.72) and 0.78 (0.77–0.79), respectively (Table 5). Applying the final equation to the validation dataset showed similar discrimination when compared with that of the test dataset.
Discussion
In this study, we developed and validated in-hospital mortality prediction equations in mechanically ventilated patients using data from Japan; we showed moderate discrimination in a model using patient demographics, clinical factors, and procedures administered during the ICU admission (Model 2). Mechanically ventilated patients in the ICU are frequently the subjects of epidemiological studies [22–24]. The risk-adjustment methods previously employed in these studies frequently include APACHE, MPM and SAPS [25–36], but these methods are primarily dependent on organ scores that require physiological data. Ohno-Machado et al. [25] obtained C-statistics for mortality prediction models using APACHE-II, APACHE-III, MPM0, MPM24, MPM-II0, MPM-II24, SAPS, and SAPS-II; and found that these all had C-statistics ≥0.8, except for SAPS. In contrast to these models, Duke et al. [8] derived the COPE model using solely administrative data. This model has advantages in that it can predict mortality with relatively few variables from routinely-available data. The COPE model also includes mechanical ventilation as an intensive care therapy which has been shown to be strongly associated with hospital mortality. Therefore, for an analysis focusing on mechanically ventilated patients using DPC data, there was a need to develop a new robust mortality prediction tool that did not include the independent variable of mechanical ventilation. In addition, because these models were developed specifically using Japanese data, and because of the ability of the DPC database to identify the details of medical care for mechanically ventilated patients in a uniform format from numerous hospitals, this approach has the capacity to support comparative evaluations of ICU performance using multicenter analysis in Japan.
Model 2 from our analysis may serve as a possible alternative model due to displaying moderate accuracy in the C-statistics. If the prognoses of mechanically ventilated patients are to be required, the use of existing scoring systems using physiological data may be more useful. But the intended applications of this equation (Model 2) lie in the retrospective evaluations of ICU performance in a multicenter analysis based on the identical format of DPC data introduced in 2004 in Japan.
The prediction equation in this study has the following advantages over existing models: the variables used in our equation utilize information that can be routinely obtained from administrative data. These variables are submitted by doctors and nurses in a timely manner on a daily basis rather than at or after discharge, which ostensibly improves the reliability of the data. Also, the model uses only 8 variables, which facilitates its generalizability and application. However, there is also the risk of coding errors, especially in ICU patients [8].
There are several limitations in the present study. First, we did not compare our model with scoring systems using physiological data, since our data did not include severity scores. Therefore, we cannot determine the relative accuracy of the model compared with other systems. Second, the administrative data include information on a ‘calendar day’ basis, rather than an hourly basis, and therefore the first ICU day was defined by a calendar day. This provides no distinction regarding the use of renal replacement therapy and pressors/vasoconstrictors before or after ICU entry on the first ICU day. However, these resources are mostly used under monitoring in the ICU. Third, the indications for renal replacement therapy and pressors/vasoconstrictors varied among the hospitals, which may have resulted in therapeutic bias. Fourth, the administrative data do not indicate if renal replacement therapy was given for chronic or acute renal failure, or for a non-renal indication; or if pressors/vasoconstrictors were used to treat hypovolemic or septic shock. Fifth, different admission criteria among the ICUs could have produced a selection bias that affected mortality. Our model has a therapeutic bias similar to that of the COPE model, including the use of mechanical ventilation, renal replacement therapy, and pressors/vasoconstrictors. However, it is likely that there is little if any inappropriate application of these therapies due to ethical considerations. Finally, since our study sample included approximately 30 % of all ICUs in Japan, and did not include university hospitals and non-teaching hospitals, further verification and modification of the model may be required in a larger sample of patients and ICUs from a greater variety of hospital types.
The absence of physiological data is disadvantageous since diagnosis is not possible, but our model has the additional advantage in that it uses administrative data routinely collected for all patients with a high level of accuracy. Comparison of ICU performance using administrative data has applications for benchmarking and quality improvement, and our model establishes a method for the comparative evaluation of ICU performance.
Conclusions
The hospital mortality prediction equation for mechanically ventilated patients in intensive care proposed in this study is based solely on administrative data, and uses a relatively small number of variables that can be easily collected. In addition to the COPE model, Model 2 can be used to evaluate illness severity of mechanically ventilated patients based on administrative data and may be applicable to future critical care studies.
References
Kuzniewicz MW, Vasilevskis EE, Lane R, Dean ML, Trivedi NG, Rennie DJ, Clay T, Kotler PL, Dudley RA. Variation in ICU risk-adjusted mortality: impact of methods of assessment and potential confounders. Chest. 2008;133:1319–27.
McMillan TR, Hyzy RC. Bringing quality improvement into the intensive care unit. Crit Care Med. 2007;35(Suppl):S59–65.
Leary T, Ridley S, Burchett K, Kong A, Chrispin P, Wright M. Assessing critical care unit performance: a global measure using graphical analysis. Anaesthesia. 2002;57:751–5.
Knaus WA, Draper EA, Wagner DP, Zimmerman JE. An evaluation of outcome from intensive care in major medical centers. Ann Intern Med. 1986;104:410–8.
Knaus WA, Wagner DP, Zimmerman JE, Draper EA. Variations in hospital mortality and length-of-stay from intensive care. Ann Intern Med. 1993;118:753–61.
Zimmerman JE, Shortell SM, Rousseau DM, Duffy J, Gillies RR, Knaus WA, Devers K, Wagner DP, Draper EA. Improving intensive care: observations based on organizational case studies in nine intensive care units: a prospective, multicenter study. Crit Care Med. 1993;21:1443–51.
Render ML, Welsh DE, Kollef M, Lott JH 3rd, Hui S, Weinberger M, Tsevat J, Hayward RA, Hofer TP. Automated computerized intensive care unit severity of illness measure in the Department of Veterans Affairs: preliminary results. Crit Care Med. 2000;28:3540–6.
Duke GJ, Santamaria J, Shann F, Stow P, Pilcher D, Ernest D, George C. Critical care outcome prediction equation (COPE) for adult intensive care. Crit Care Resusc. 2008;10:35–41.
Curtis JR, Cook DJ, Wall RJ, Wall RJ, Angus DC, Bion J, Kacmarek R, Kane-Gill SL, Kirchhoff KT, Levy M, Mitchell PH, Moreno R, Pronovost P, Puntillo K. Intensive care unit quality improvement: a ‘how-to’ guide for the interdisciplinary team. Crit Care Med. 2006;34:211–8.
de Vos M, Graafmans W, Keesman E, Westert G, van der Voort PH. Quality measurement at intensive care units: which indicators should we use? J Crit Care. 2007;22:267–74.
Terblanche M, Adhikari NK. The evolution of intensive care unit performance assessment. J Crit Care. 2006;21:19–22.
Lin YF, Ko WJ, Chu TS, Chen YS, Wu VC, Chen YM, Wu MS, Chen YW, Tsai CW, Shiao CC, Li WY, Hu FC, Tsai PR, Tsai TJ, NSARF Study Group. The 90-day mortality and the subsequent renal recovery in critically ill surgical patients requiring acute renal replacement therapy. Am J Surg. 2009;198:325–32.
Brar H, Olivier J, Lebrun C, Gabbard W, Fulop T, Schmidt D. Predictors of mortality in a cohort of intensive care unit patients with acute renal failure receiving continuous renal replacement therapy. Am J Med Sci. 2008;335:342–7.
Soubrier S, Leroy O, Devos P, Nseir S, Georges H, d’Escrivan T, Guery B. Epidemiology and prognostic factors of critically ill patients treated with hemodiafiltration. J Crit Care. 2006;21:66–72.
Pinsky MR. Hemodynamic evaluation and monitoring in the ICU. Chest. 2007;132:2020–9.
Lundberg GD. Practice parameter for the use of fresh-frozen plasma, cryoprecipitate, and platelets. JAMA. 1994;271:777–81.
Angus DC, Linde-Zwirble WT, Lidicker J, Clermont G, Carcillo J, Pinsky MR. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. 2001;29:1303–10.
Deyo R, Cherkin D, Ciol M. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45:613–9.
Sundararajan V, Henderson T, Perry C, Muggivan A, Quan H, Ghali WA. New ICD-10 version of the Charlson Comorbidity Index predicted in-hospital mortality. J Clin Epidemiol. 2004;57:1288–94.
Umegaki T, Sekimoto M, Hayashida K, Imanaka Y. An outcome prediction model for adult intensive care. Crit Care Resusc. 2010;12:96–103.
Dunning J, Khasati N, Barnard J. Low dose (renal dose) dopamine in the critically ill patient. Interact Cardiovasc Thorac Surg. 2004;3:114–7.
Kim IB, Fealy N, Baldwin I, Bellomo R. A pilot study of the epidemiology and associations of pulse pressure variation among non-cardiac surgery critically ill patients. Crit Care Resusc. 2011;13:156–61.
Lombardi R, Nin N, Lorente JA, Frutos-Vivar F, Ferquson ND, Hurtado J, Apeztequia C, Desmery P, Raymondos K, Tomicic V, Cakar N, González M, Elizalde J, Nightingale P, Abrouq F, Jibaja M, Arabi Y, Moreno R, Matamis D, Anzueto A, VENTILA Group. An assessment of the Acute Kidney Injury Network creatinine-based criteria in patients submitted to mechanical ventilation. Clin J Am Soc Nephrol. 2011;6:1547–55.
Peñuelas O, Frutos-Vivar F, Fernández C, Anzueto A, Epstein SK, Apeztequía C, González M, Nin N, Raymondos K, Tomicic V, Desmery P, Arabi Y, Pelosi P, Kuiper M, Jibaja M, Matamis D, Ferquson ND, Ventila Group. Characteristics and outcomes of ventilated patients according to time to liberation from mechanical ventilation. Am J Resp Crit Care Med. 2011;184:430–7.
Ohno-Machado L, Resnic FS, Matheny ME. Prognosis in critical care. Annu Rev Biomed Eng. 2006;8:567–99.
Rosenberg AL. Recent innovations in intensive care unit risk-prediction models. Curr Opin Crit Care. 2002;8:321–30.
Afessa B, Gajic O, Keegan MT. Severity of illness and organ failure assessment in adult intensive care units. Crit Care Clin. 2007;23:639–58.
Cook DA. Methods to assess performance of models estimating risk of death in intensive care patients: a review. Anaesth Intensive Care. 2006;34:164–75.
Junger A, Engel J, Benson M, Hartmann B, Röhriq R, Hempelmann G. Risk predictors, scoring systems and prognostic models in anesthesia and intensive care. Part II. Intensive care. Anasthesiol Intensivmed Notfallmed Schmerzther. 2002;37:591–9.
Schusterschitz N, Joannidis M. Predictive capacity of severity scoring systems in the ICU. Contrib Nephrol. 2007;156:92–100.
Kramer AA. Predictive mortality models are not like fine wine. Crit Care. 2005;9:636–7.
Chen FG, Khoo ST. Critical care medicine: a review of the outcome prediction in critical care. Ann Acad Med Singapore. 1993;22:360–4.
Harrison DA, D’Amico G, Singer M. Case mix, outcome, and activity for admissions to UK critical care units with severe acute pancreatitis: a secondary analysis of the ICNARC Case Mix Programme Database. Crit Care. 2007;11:S1.
Strand K, Flaatten H. Severity scoring in the ICU: a review. Acta Anaesthesiol Scand. 2008;52:467–78.
Capuzzo M, Moreno RP, Le Gall JR. Outcome prediction in critical care: the Simplified Acute Physiology Score models. Curr Opin Crit Care. 2008;14:485–90.
Duke GJ, Piercy M, DiGiantomasso D, Green JV. Comparison of intensive care outcome prediction models based on admission scores with those based on 24-hour data. Anaesth Intensive Care. 2008;36:845–9.
Author information
Authors and Affiliations
Corresponding author
About this article
Cite this article
Umegaki, T., Nishimura, M., Tajimi, K. et al. An in-hospital mortality equation for mechanically ventilated patients in intensive care units. J Anesth 27, 541–549 (2013). https://doi.org/10.1007/s00540-013-1557-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00540-013-1557-0