Intensive care faces economic challenges. Therefore, evidence proving both effectiveness and efficiency, i.e., cost-effectiveness, of delivered care is needed. Today, the quality of care is an important issue in the health care debate. How do we measure quality of care and how accurate and representative is this measurement? In the following report, several topics which are used for the evaluation of intensive care unit (ICU) performance are discussed: (1) The use of general outcome prediction models to determine the risk of patients who are admitted to ICUs in an increasing variety of case mix for the different intensive care units, together with three major limitations; (2) As critical care outcomes research becomes a more established entity, mortality is now only one of many endpoints that are relevant. Mortality is a limited outcome when assessing critical care performance, while patient interest in quality of life outcomes is relevant; and (3) The Quality Indicators Committee of the Society of Critical Care Medicine recommended that short-term readmission is a major performance indicator of the quality of intensive care medicine.

The intensive care unit (ICU) is a hospital unit delivering continuous surveillance and highly specialized care to critically ill patients, either medical or surgical. Patients’ conditions are life-threatening and require comprehensive care[1]. Established approximately five decades ago, the ICU is now a fundamental part of hospital care. It presents itself as the knowledge that aims to help patients with extended needs of care and organ support[2].

Intensive care faces economic challenges. Therefore, evidence proving both effectiveness and efficiency, i.e., cost-effectiveness, of delivered care is needed. ICUs consume a significant proportion of health care resources, accounting for up to 20% of a hospital’s cost[3-8]. By 2005, critical care medicine costs in the United States were estimated to be $81.7 billion, accounting for 4.1% of the national health expenditures and 0.66% of the gross domestic product[9]. The United States spends 15% of the gross domestic product on health care (9%-11% in Germany, France and Canada; 7%-8% in Spain and the United Kingdom). Intensive care costs are estimated to be increasing throughout the developed world[4,7,10-17].

Today, quality of care is an important issue in the health care debate[18]. All countries struggle to optimize quality of care while minimizing costs. Assessment of clinical performance is obligatory for the evaluation of both the effectiveness and efficiency of care[19] and therefore several questions arise: How do we measure quality of care and how accurate and representative is this measurement?

The goal of intensive care medicine is to achieve the best outcome for critically ill patients and this is usually accompanied by the use of very complex care[2,20]. All patients carry both an intrinsic (disease-related) and an extrinsic (care-related) risk at the same time[2,21]. There is an ever-increasing acknowledgement of the wide variation in the quality of care across ICUs and its effect on outcome. Indicators to evaluate the quality of care are progressively being used and focus on patient outcome[18,22]. Finding a solid technique to determine the performance of single ICUs has been a difficult pursuit for the last 30 years[19].

Each new development in critical care treatment over the past 30 years has been implemented to improve the quality of care. Therefore, the extrinsic risks that patients carry should be as low as possible. Ideally, quality of care performance research should give more information about the extrinsic rather than the intrinsic risks. Presently, ICU performance evaluation is becoming increasingly difficult because of the presence of an increasing variety in patient case mix for the different intensive care units. Since the development of prediction mortality models in the early 1980s, physicians have tried to normalize certain ICU populations through the use of severity of illness measurements. At the time that a general outcome prediction model (GOPM) was developed, the intrinsic risk had been adjusted in such a way that performance mainly illuminated the extrinsic risk factors. Most published approaches concerning the evaluation of ICU performance adopt more or less identical methods: the development of a GOPM and its calibration in a suitable database. Such models are then applied to different cohorts of ICU patients and the comparison of the predicted number of deaths with the actual number is used as a reference for the clinical behavior of the unit[15]. For over 30 years, outcome research in critical care relied heavily on these risk adjustment methods (GOPM) to assess and quantify the risk of patients admitted to ICUs[2]. Using several GOPMs, this methodology has become the “gold standard” to compare ICUs across different geographical areas or within a specific individual nation or other specific subgroups[19]. Various risk adjustment systems have been created or updated and are used in daily practice.

In the use of general outcome prediction models, several limitations should be considered: (1) Most systems produce a single estimate, known as the standardized mortality ratio (SMR). A single estimate reflects that the performance of an ICU is steady over the whole spectrum of the severity of illness[23]. In other words, an ICU with a “good” performance (low SMR) is believed to be homogeneously good for both low-risk and high-risk patients; in the same way, an ICU with a “bad” performance (high SMR) is assumed to be uniformly bad. However, since performance can vary not only between ICUs but also within the same unit across patients and doctors, this assumption is likely not true[19]. Several studies have provided conclusive documentation that the clinical performance of ICUs may vary over the array of severity of illness[2,19,24-27]; (2) It is unknown whether variations in SMR reflect quality of care or case mix differences. Debate continues whether higher than predicted mortality (high SMR) is a warning about the quality of care or rather reflects a difference of case mix between hospitals[20,28]. In the past, GOPMs have been revised or even updated to newer versions to predict expected death more accurately. However, many years elapsed before a new GOPM version was used. Although the newer third and fourth versions of the APACHE prognostic model were developed many years ago[29-31], the APACHE II score is still one of the most widely used[9,20,32]; and (3) There is no consensus as to which GOPM must be used for which type of ICU (general mixed unit, specialized unit, or even in different sub-populations). For critical care physicians, there are three overall GOPMs for predicting overall mortality used for performance evaluations: the APACHE model[29-31,33], the MPM system[34-36] and the SAPS model[26,27,37-39]. These scoring systems differ in the choice and relative weight given to patient characteristics and physiological parameters[18,22]. Quality of care performance evaluation should be done with the same and ideally most reliable outcome prediction model for each intensive care unit. Because there is no consensus as to which GOPM should be used, they seem to be used randomly. Within the Netherlands, since 2008, all 61 participating ICUs in the NICE registry started using the APACHE IV prognostic model[18].

Until today, one of the most used ICU performance measurements is the SMR[20]. The SMR was developed in a period when the evaluation of quality of care was done exclusively through primary patient outcome (short-term mortality). Some authors evaluated the use of SMR as an indicator of ICU quality of care and debated its specific relevance[21,40,41]. The SMR value gives insight into the observed mortality compared with the associated predicted mortality but it does not give insight into the health status of these patients.

As critical care outcomes research becomes more established, entity mortality is now only one of many endpoints that is relevant and mortality is a limited outcome evaluation method when assessing critical care performance. The health-related quality of life (HrQoL), described as the level to which a patient’s health status affects the subjective appraisal of his or her contentment with life, seems to be a better indicator, especially from the patient-centered view[42]. ICU and hospital survival will always have an important role in the evaluation of performance at the moment different units or hospitals are being benchmarked. Consequently, in the last decade, the QoL has gained great interest when both physicians and patients’ relatives mention patient outcome. Therefore, QoL clearly challenges survival whenever we address secondary (long-term) patient outcome.

Difficulties are being foreseen when using health status as a performance benchmark[43] because of the great diversity in intrinsic risk that patients carry in different ICUs (i.e., specialized units, general mixed units)[20]. How should we use health status as performance benchmark? Should we cross-section the mean health status of a given cohort against the general population norm or must we compare individual outcome with individual pre-admission values? The latter will invariably provide more patient oriented and thus clinically relevant outcome values but also result in an administrative burden. A third possibility is to compare such an individual QoL value with a predicted individual health status.

The capability of calculating a patient’s QoL after ICU admission could be useful in many ways. Firstly, it could help patients and their relatives to make decisions. Secondly, it could help families to prepare themselves to care for the patient after hospital discharge. Thirdly, it could help critical care physicians to give useful information, avoid unrealistic expectations and possibly help in making treatment decisions. Fourthly, it could help society to realize in which ICUs patients have a good prospect of recovery and give health policy makers and insurance companies insight into the needs of ICUs[42-44].

A completely different benefit using HrQoL as a performance benchmark could be the possibility of follow-up evaluation of patients’ health status after ICU or hospital discharge. Post-ICU patients are known to express a reduced HrQoL compared to the general population. It is still not clear to what extent and how long this reduced HrQoL persists, although this effect may be long-lasting[45]. Therefore, a continuous survey as part of regular after care for each individual patient would be the ideal way to investigate this, providing the possibility of better managing patients in which HrQoL does not increase as expected.

The Quality Indicators Committee of the Society of Critical Care Medicine recommended that readmission within 48 h is a major performance indicator of the quality of intensive care medicine[46,47]. Readmitted patients are most often the sickest in the ICU; therefore, it is an unexpected and unfavorable event for the patient and is associated with a more severe outcome[48-57]. Moreover, a strategy to reduce premature discharges in patients at high risk of in-hospital death could result in a reduction of post-ICU mortality (Daly et al[58]: 39% reduction in mortality)[48,57,58]. In times of great pressure on ICU capacity, should we not be more careful in deciding which patient may be discharged and who has a greater risk of readmission? Ideally, such decisions are made on sound criteria rather than subjective parameters. In the last 10 years, several authors have proven that it is difficult to analyze and predict readmission risk for ICU patients in general[49-52]. Various authors concluded that patients readmitted to the ICU had a higher severity of illness score at the time of initial ICU discharge compared to single ICU admission patients[47,50,51,59]. Ideally perhaps, severity of illness is scored on a daily basis and discharge is initiated from these values. Unfortunately, these severity of illness scores have not been validated after the first 24 h of ICU admission. The Sequential (Sepsis-related) Organ Failure Assessment score (SOFA score) is used to track a patient’s status during the admission to the ICU (also validated to be used after 24 h). The SOFA score is a scoring system to determine the extent of a person’s organ function or rate of failure[60-63]. This particular score has been validated to predict ICU mortality[64]. Nevertheless, the possible association with readmission has not been evaluated as yet. Currently, there are hardly any systematic studies of how daily severity of illness score changes from admission to initial discharge predict ICU readmission[32,52]. Besides the severity of illness score, there is also an association between nursing workload and post-ICU mortality[65,66]. The Therapeutic Intervention Scoring System (TISS) has been widely applied to assess workload and resource allocation in intensive care, measuring treatment intensity[67-69]. Consequently, attempts have been made to use TISS scores to categorize the level of care that patients require and even to evaluate the care required after ICU discharge[1,68]. Several authors have shown an association of the TISS value of the last ICU day with post-ICU mortality[65,66,69,70] and therefore indirectly the association with ICU readmission. Smith et al[66] concluded in their research that the mean TISS scores in patients readmitted to the ICU were significantly higher than in patients who did not require readmission[65,66].

For a couple of years, Spanish physicians have shown great interest in this topic and developed the Sabadell score system, a modification of the McCabe score[71-73]. They have validated the relevance of the Sabadell score as a method for classifying patient’s ward survival at discharge from the ICU[74] and even found an association of the Sabadell score with ICU readmission. Unfortunately, the lack of reliable predictors of ICU readmission prevents the clinical efficacy of this variable. However, this information may improve the ability to predict the readiness for discharge for individual patients and improve the efficiency of intensive care units[47]. Would critical care physicians have more information about patients’ disease status when they use a combination of several systems (TISS, severity of illness score and Sabadell score) as a prediction measurement for ICU discharge readiness? This value could also give an indication of whether the patient could be discharged to the normal ward or if he should first be admitted to a step-down unit (high dependency unit).

Hospital death rates would be particularly useful if patients and physicians could use the statistics for a given diagnosis to select a hospital that offers the best prospect of survival. If the data are only partially corrected for differences in the health status of patients they must be used with caution[23]. Variations in case mix, ICU demographics, clinical and non-clinical factors not addressed by the present severity of illness scores must be quantified to improve the accuracy of future prediction models. If the variation between ICUs is important, it will impair the stability of the equations used to calculate predicted mortality and preclude the use of indirect standardization in the evaluation of differences between ICUs. These GOPMs consider the relationship between performance and severity of illness as constant although performance can vary within ICUs according to the level of severity of illness in patients. Hypothetically, performance should be evaluated through the combination of survival (SMR) and the health status (QoL) at the time of discharge. As yet, this combination of both outcome measurements has not been used in a single benchmark value. Therefore, future research should focus on predicting quality of life together with an accuracy study of the TISS, severity of illness and the Sabadell scores to identify and weigh the specific variables for readmission. The moment when outcome research can predict the short-term (ICU discharge) QoL of a critically ill patient during the first 24 h of ICU admission will give physicians and health care policy makers an up-to-date and reliable evaluation of quality of care in the ICU for the future.

The authors declared that they have no competing interests. Each author certifies that he has participated sufficiently in the intellectual content, the analysis of data if applicable and the writing of this work to take public responsibility for it. Each has reviewed the final version of this work, believes it represents valid work and approves it for publication. We herewith state that all authors provided an equal contribution to this study and act in concordance to the “uniform requirements”.