Scoring systems have not been evaluated in oncology patients. We aimed to assess the performance of Acute Physiology and Chronic Health Evaluation (APACHE) II, APACHE III, APACHE IV, Simplified Acute Physiology Score (SAPS) II, SAPS III, Mortality Probability Model (MPM) II₀ and Sequential Organ Failure Assessment (SOFA) score in critically ill oncology patients.

To compare the efficacy of seven commonly employed scoring systems to predict outcomes of critically ill cancer patients.

We conducted a retrospective analysis of 400 consecutive cancer patients admitted in the medical intensive care unit over a two-year period. Primary outcome was hospital mortality and the secondary outcome measure was comparison of various scoring systems in predicting hospital mortality.

In our study, the overall intensive care unit and hospital mortality was 43.5% and 57.8%, respectively. All of the seven tested scores underestimated mortality. The mortality as predicted by MPM II₀ predicted death rate (PDR) was nearest to the actual mortality followed by that predicted by APACHE II, with a standardized mortality rate (SMR) of 1.305 and 1.547, respectively. The best calibration was shown by the APACHE III score (χ² = 4.704, P = 0.788). On the other hand, SOFA score (χ² = 15.966, P = 0.025) had the worst calibration, although the difference was not statistically significant. All of the seven scores had acceptable discrimination with good efficacy however, SAPS III PDR and MPM II₀ PDR (AUROC = 0.762), had a better performance as compared to others. The correlation between the different scoring systems was significant (P < 0.001).

All the severity scores were tested under-predicted mortality in the present study. As the difference in efficacy and performance was not statistically significant, the choice of scoring system used may depend on the ease of use and local preferences.

The application of prognosticating scoring systems is considered as an important phase in intensive care units (ICUs) since these severity scoring systems estimate the probability of mortality for patients. These scores help the physicians to facilitate resource utilization or continuous quality improvement and to stratify the patients for clinical research[1,2]. ICU scoring systems can help both patients as well as their attendants to select from further treatment options. Further, the scores calculated by these scoring systems help in evaluating the impact of newer treatment modalities and organizational changes which in turn contributes towards the development of treatment standards. In addition to the above, the scoring systems’ outcomes also help in benchmarking ICU performance and comparing the scores secured by different ICU patient populations so as to find out the differences in mortality. However, these systems are unreliable in predicting the clinical outcomes of an individual though it has proven efficacy in predicting mortality for a particular patient cohort[3].

Acute Physiology and Chronic Health Evaluation (APACHE) II and Simplified Acute Physiology Score (SAPS) II are arguably the two most-commonly used and validated tools used in the prediction of ICU patient outcomes[4,5]. These scoring systems were developed in the 1980s and have become outdated due to technological and clinical advancements in critical care management of patients in recent years. Hence, there is a need to develop new scoring systems that include APACHE IV, SAPS III and Mortality Probability Model (MPM) II0[6-9]. Such newly-created systems encompass a large number of variables and are highly complicated to compute.

In addition, both validity and utility of the existing scoring systems may be questionable in terms of current patient population compared to the patient population during which they were developed. These scores are widely used and the scoring systems have been validated for a notable time to predict the outcome in general medical or surgical procedures conducted upon critically ill patients. However, whether these systems can predict the mortality accurately among cancer patients remains unknown[10]. There is a dearth of studies that compare different generations of scoring systems and especially the ones used upon cancer patients admitted in medical oncology ICUs. Only a few studies have assessed their usefulness in cancer patients with conflicting results. Moreover, geographic variations in patient populations and the types of cancer necessitate that these scores should be evaluated for different populations[11]. Therefore, the current study is aimed at analyzing the efficacy of seven commonly-used scoring systems to predict the mortality amongst patients admitted in oncology ICUs.

A retrospective observational cohort study was carried out at a multi-disciplinary onco-medical ICU of a tertiary care center in India. We have an advanced ICU setup and 24-h intensivist coverage with state-of-the-art facilities. Approval for the study and a consent waiver from the institutional ethics committee was obtained.

The data from the records of adult patients who were admitted between January 2018 and February 2020, i.e., 2 years, was collected and analyzed. If the patient was readmitted to the ICU more than once during his/her hospital stay, only the first admission was included in the study. Patients who had ICU stays of less than 12 h, post-operative patients and those admitted from or discharged to another ICU were excluded from the study. Patients fulfilling inclusion criteria were serially recruited. The researchers collected the following data; baseline patient characteristics, indication for ICU admission, type of malignancy, presence of metastasis, need for vasopressor, renal and mechanical ventilation (MV), length of ICU and hospital stay, and ICU and hospital mortality. The data, required to compute various scores, was collected and calculated specified by the procedures.

During the study period, the data from 400 patients who fulfilled the inclusion criteria were included in the final analysis. Thirty-eight patients were excluded because 31 were admitted from or discharged to another ICU, five were post-operative patients and two had ICU stays less than 12 h. Their baseline characteristics are given in Table 1 and the comparison between various scores is given in Table 2.

Discrimination

The efficacy of various scores is given in Figure 1. All the scores tested in the current study exhibited good efficacy, even though there was no statistically significant difference between AUROCs and SAPS III PDR. On the other hand, MPM II₀ PDR (AUROC = 0.762) yielded the best performance (Table 5).

Open in New Tab   Full Size Figure   Download Figure

Figure 1 Comparison between the area under the receiver operating characteristic curves of APACHE II, APACHE III, APACHE IV SAPS-II, SAPS-III, SOFA score and MPM II₀ -PDR in discriminating survivors from non-survivors. APACHE: Acute Physiology and Chronic Health Evaluation; SAPS: Simplified Acute Physiology Score; SOFA: Sequential Organ Failure Assessment; MPM: Mortality Probability Model.

Scoring system	AUC	P value	95%CI	Cut off	Sensitivity	Specificity
APACHE II	0.688	< 0.001	0.637-0.739	> 18.5	67.5%	62.7%
APACHE III	0.720	< 0.001	0.672-0.769	> 78.5	46.8%	87.6%
APACHE IV	0.708	< 0.001	0.659-0.758	> 72.5	53.7%	79.3%
SAPS II	0.734	< 0.001	0.685-0.782	> 34.5	76.2%	60.4%
SAPS III PDR	0.762	< 0.001	0.715-0.808	39.0	44.3%	92.0%
SOFA Score	0.715	< 0.001	0.665-0.764	> 7.5	58.0%	79.3%
MPM II0 PDR	0.762	< 0.001	0.714-0.810	36.45	71.3%	69.9%

AUC: Area under the curve; CI: Confidence interval; APACHE: Acute Physiology and Chronic Health Evaluation; SAPS: Simplified Acute Physiology Score; SOFA: Sequential Organ Failure Assessment; MPM: Mortality Probability Model; PDR: Predicted death rate.

The current study compared various mortality prediction scoring systems and found that all the scores under-predicted the mortality in critically-ill cancer patients. Amongst the scoring systems considered, mortality predicted by MPM PDR was the closest to that of the actual mortality with an SMR of 1.305. AUROC values showed that all of the seven scoring systems had good efficacy and acceptable discrimination. MPM PDR and SAPS III PDR achieved the best discrimination. We found the best sensitivity in SAPS II score (76.2%) and best specificity in SAPS III PDR score (92%). The Lemeshow-Hosmer goodness-of fit tests showed that the APACHE III score had the best calibration although there was no statistically significant difference.

In the current study, all of the scores were significantly higher among non-survivors (P value < 0.001) as reported in the literature[13-18]. However, all the scores tested in this study underestimated the mortality (SMR > 1), like previous studies[14,15,19,20].

Discrimination is the ability to determine the patients who may die and who will survive. Measures of discrimination include sensitivity, specificity and AUROC curve. But no single scoring system excelled in all of the three areas. SAPS III PDR and MPM II₀ PDR (AUROC = 0.762) had the best AUROC values whereas sensitivity was at its best for SAPS II and specificity was at its best for SAPS III PDR. However, these differences were not statistically significant. In the current study, AUROC outcomes showed that discrimination is acceptable in all the scoring systems tested as reported in the literature[14-16,20-22]. All the severity illness scores showed good efficacy with no statistically significant difference in AUROCs.

Calibration evaluates the accuracy of the degree of correspondence between the estimated probability of mortality and the observed actual mortality. Calibration is good if the predicted mortality is close to the observed mortality. APACHE III (4.704) had the best calibration with P = 0.788. This infers that it had the least statistically significant discrepancy between the predicted and observed mortality. Good calibration of these scores have also been reported by other authors[14-16,20].

A significant correlation was found among various scoring systems (P < 0.001) as per linear regression analysis. This correlation may be attributed to the overlap of multiple variables, considered for calculating the scores. Sculier et al[21] also reported an excellent correlation between APACHE II and SAPS II in their study on oncology patients. ICU mortality rate among cancer patients was reportedly high and in the range of 30% to 77%[23-26]. The overall ICU mortality rate in the current study was 43.5%. Even though it is higher, the ICU mortality of the current cohort does not differ from the mortality reported in similar studies conducted earlier[23,24]. The hospital mortality rate in the current study was 57.8% which is again similar as reported earlier[27,28].

Use of MV and vasopressor support have a direct association with hospital mortality. Similar studies conducted earlier have also reported the need for organ support in the form of MV. At times, vasopressor use is directly associated with increased mortality among cancer patients[29]. An ideal scoring system is the need of the hour. This system should be well calibrated, easy to compute, able to have high levels of discrimination and predict mortality rates with high accuracy based on the easily-available patient parameters. Additionally, an ideal score also needs to be dynamic, reflecting the change in management and case mix over time. In this search for an ideal scoring system, newer scoring systems have been developed. However, these systems are highly complex in nature, demand huge sets of patient data and need computer assistance to calculate the scores. Hence, the development of an ideal scoring system has a long way to go.

The accuracy of scoring systems may differ over a period of time and may produce varied results in different countries due to differences in ethnicity, patient population, healthcare systems, ICU structure and organization. So, its accuracy cannot be generalized and all such models need external validation in independent patient populations to prove its reproducibility. Therefore, it becomes imperative to compare and test the validity of scoring systems under different geographical areas and upon different patient populations. The current study is one of the few studies conducted on the Indian subcontinent and the researchers have compared a huge number of scoring systems developed for cancer patients in a significantly large cohort of patients.

The current study has a limitation to address, i.e. being a single center retrospective study where concerns may arise in terms of generalizing the conclusions arrived in this study. The missing data may have also led to information bias. Nonetheless, the study has several salient features such as the comparison of seven scoring systems, fairly large sample size, well-defined study protocol and the inclusion of only medical oncology patients.

The current study concludes that all of the scoring systems considered for this study cohort under-predicted the mortality. However, the APACHE III score had the least discrepancy between the predicted and observed mortality. There was no statistically significant difference in efficacy and all the scores tested had good calibration and acceptable discrimination. Hence, the choice of scoring system in critically-ill oncology patients should not only be based on the performance of the score, but also on other factors such as ease of use and local preferences.