https://orcid.org/0000-0001-6833-3811
Figures
Abstract
Objective
This study aimed to assess the performance of a commonly used ICU severity score (SAPS3) and determine whether an alternative scoring system may be more accurate across all age strata.
Methods
Retrospective cohort study in a general ICU in Brazil. A secondary analysis was performed with clinical and epidemiological data, present in the first 24 hours of unit admission. Then, a binary logistic regression, followed by cross-validation, was made to develop a novel prognostic tool. ICU mortality was the primary outcome evaluated.
Results
A total of 3042 patients were included over the study period between August 2015 and July 2018 with a median age of 67 ± 18.4 years. SAPS3 performed fairly in prediction of ICU mortality, particularly in the 80 years or older subset. Multivariable regression identified variables independently associated with mortality that were used to develop the Age Calibrated ICU Score (ACIS) tool that performed similarly to SAPS3 across age categories, being slightly superior in the very elderly population (AUC 0.80 vs 0.72).
Citation: Menezes RC, Ferreira IBB, Carmo TA, Telles GP, Pugas PLD, Otero ML, et al. (2020) Are prognostic tools losing accuracy? Development and performance of a novel age-calibrated severity scoring system for critically ill patients. PLoS ONE 15(11): e0240793. https://doi.org/10.1371/journal.pone.0240793
Editor: Andrea Cortegiani, University of Palermo, ITALY
Received: May 30, 2020; Accepted: October 3, 2020; Published: November 4, 2020
Copyright: © 2020 Menezes et al. This is an open access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Data Availability: All relevant data are within the paper and its Supporting Information files. The raw data used in our analyses are publicly available at the following link: http://bioinfo03.bahia.fiocruz.br:8787/PONE-D-20-16383R2/Dataset_ACIS.xlsx
Funding: 1. Funder: Fundação Oswaldo Cruz, Brazil a. Award number: Intramural Research Program b. Recipient: Bruno B. Andrade 2. Funder: Conselho Nacional de Desenvolvimento Científico e Tecnológico, Brazil a. Award number: senior fellowship b. Recipient: Bruno B. Andrade 3. Funder: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior, Brazil a. Award number: fellowship (finance code 001) b. Recipient: Kiyoshi F. Fukutani 4. Funder: Coordenação de Aperfeiçoamento de Pessoal de Nível Superior a. Award number: PhD fellowship b. Recipient: Rodrigo C. Menezes 5. Funder: Fundação de Amparo à Pesquisa do Estado da Bahia a. Award number: PhD fellowship b. Recipient: María B. Arriaga Funding statement: The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have declared that no competing interests exist
Introduction
The world population is aging with the rate elderly individuals comprising the fastest growing group. By 2050, 16% of the world population is expected to be 65 years or older; moreover, in Latin America, this stratum is expected to double from 56 million to 144 million individuals. As a result of this higher life expectancy and concurrent increased prevalence of comorbidities, patients over 80 years will invariably constitute a greater proportion of intensive care units (ICU) [1]. These patients form a particularly heterogeneous population, with more comorbidities and higher mortality compared to younger patients. Increasingly, modern ICU care teams aims to identify elderly patients who will benefit from treatment in the ICU. In this context, accurate prognosis is paramount to triage resources to those most likely to survive their illness [2–5]. This challenge has become more apparent in the current coronavirus (COVID-19) pandemic, where ICU capacity has become scarce with a surge of critically ill elderly patients. This has left physicians with few tools to accurately determine treatment decisions, often using age as one of the primary factors for ICU admission [6].
The benefit of high cost invasive treatment in elderly patients is unclear. Moreover, prior severity scoring systems have been derived and validated in populations disparate from patients currently admitted to the ICU. Scoring systems, such as the Simplified Acute Physiology Score, have been validated and long used in ICUs around the world to assess the severity of illness. However, these scores did not include an aging population at time of validation, suggesting that their discriminatory capacity may diminish in the coming years with a complex very elderly population [7–9]. This study aims to determine the accuracy of the Simplified Acute Physiology Score (SAPS3) in patients admitted to a general ICU and identify variables associated with mortality to develop an alternative age calibrated scoring system.
Methods
Ethics statement
All clinical investigations were conducted according to the principles expressed in the Declaration of Helsinki. The Ethics approval and waiver of consent to participate was approved by the Research Ethics Committee of Hospital Ana Nery under the number 2.571.265 and CAAE 52892315.1.0000.0045.
Study design and procedures
Observational analytical retrospective cohort study carried conducted from August 2015 to October 2019 in a general ICU of 22 beds in Salvador, Bahia, Brazil. All patients admitted to the ICU with complete data and over 18 years were included. Data was obtained for patient registries and recorded in the Epimed Monitor system.
Covariables included: age, weight, height, sex, comorbidities (arterial hypertension, diabetes, previous myocardial infarction, malignancy, asthma, peripheral vascular disease, structural cardiovascular disease, chronic atrial fibrillation, liver disease, stroke, dementia, alcohol consumption, tobacco consumption, psychiatric disease peptic disease, hypothyroidism, hyperthyroidism, dyslipidemias, reduced level of consciousness, neurological seizures, dependence (independent or minor dependence/bedridden), admission diagnosis, length of ICU and hospital stay, physiologic and laboratory data (lowest mean arterial pressure, highest heart rate, highest respiratory rate, highest temperature, highest leukocyte count, lowest platelets count, highest creatinine, highest arterial lactate, urea and BUN), within the first day of admission, complications, use of vasopressors and mechanical ventilation, ICU mortality and, SAPS3, Charlson Comorbidity Index and MFI scores. The MFI is an instrument to assess the severity of frailty syndrome and is comprised by 11 items [10]. The primary outcome was mortality at ICU discharge.
Statistical analysis
Categorical variables were expressed as frequencies and percentages and analyzed by Fisher’s exact test or Chi-Squared. Continuous variables with normal distribution were expressed as means (standard deviation, SD) and means between groups were compared with independent T-test. Non-normal continuous variables were expressed as median (interquartile range, IQR) and compared with Mann-Whitney U test. Normality was assessed by the D'agostino test. The area under the receiver operating characteristic (AUC) curve was used to determine the discriminate capacity. Probability of ICU death by age was calculated using Kaplan-Meyer curves.
A binary logistic regression, backward stepwise method, was used to identify characteristics independently associated with ICU mortality. Through an analysis of variances, we evaluated the interactions between the study variables and only the interactions with a p <0.05 were entered into regression. The K (10) Fold Cross Validation was performed using Classification And Regression Training package (CARET) available in R [11]. The resampling was performed to evaluate the models on the data sample, using a parameter called “k” that refers to the number of groups the data sample was split into. One proportion of the data was used to discovery the classification and the rest to validate and measure the prediction power of a limited data. Continuous variables were then dichotomized, setting a cut-off value based on the Youden Index J on AUROC analysis. A new regression was made with the dichotomized variables, to identify the adjusted odds ratios and to develop the prognostic tool.
Results
Patient characteristics
During the study period, 3,042 patients were admitted to the ICU, with 867 patients 80 years or older, 646 between 70–79, 575 between 60–69 and 954 under 60 (Fig 1). The mean age was 67 ± 18.4 years with a female predominance (53.3%). Eighty-one percent of admissions were non-surgical primarily with cardiovascular, infection/sepsis and neurological diagnoses (21.1%, 17.5% and 16.4% respectively). A total of 463 (15.2%) deaths occurred. The mean value of SAPS3 was 46.2 ± 12.3, corresponding to a predicted mortality of 16.9% with SAPS3. The average ICU length of stay was 7. ± 12.15 days and the average length of hospital stay prior to ICU admission was 2.2 ± 10.2 days (Table 1).
The 80 years or older subset had a mean age of 87 ± 5.1 years with a predominance of female patients (64.9%). Majority were non-surgical (93.7%) with similar reasons for ICU admission to the general cohort of infection/sepsis, cardiovascular and neurological (27.5%, 20.8% and 16.3% respectively). A total of 247 (27.9%) deaths occurred. The average ICU length of stay was 8.8 ± 12.2 days and the average length hospital stay prior to ICU admission was 2 ± 7.1 days. Additional information can be found in Table 2.
Comparison between non-survivors and survivors
In the whole population, those who died had a mean age of 76.2 ± 15.6 years with a female predominance (52.4%) compared with survivors whose mean age of 65.3 ± 18.3 years and a predominance of females (53.5%). The non-survivors presented with lower BMI, longer ICU length of stay, higher admission SAPS3, Charlson Comorbidity Index (CCI) and Modified Frailty Index (MFI) scores. These and other physiological data can be found in the S1 Table.
The non-survivors in the 80 years or older subgroup did not vary in their mean age compared to survivors (87.7 ± 5.1 vs 87 ± 5.1 year, respectively). Non-survivors had lower Body Mass Index (BMI), ICU length of stay, and higher SAPS3, CCI and MFI scores values, as seen in Table 2. Data regarding other age strata are summarized in the S2 Table.
Derivation of a novel severity score
Multivariable regression analysis yielded 14 variables that were used to develop an Age Calibrated ICU Score (ACIS) and a punctuation was assigned to each variable based on their adjusted OR values (Fig 2). Then, an approximation to an integer was made to facilitate usability, without significant compromise of its AUC. The ACIS is a score where the following variables are present or absent during admission and they add up to a maximum of 27 points: patient with minor dependence or bedridden (2 points); sepsis (2 points); need of vasopressor (3 points), need of mechanical ventilation (3 points) Glasgow lower than 15 (2 points), immunosuppression (2 points), malignancy (2 points), arterial lactate above 2.64 mmol/L (2 points), serum creatinine above 1.2 (2 points), being 80 years or older (2 points), heart rate over 100 beats per minute (2 points), BMI less than 23 (1 points), being readmitted 24 hours after ICU discharge (2 points), being admitted after elective surgery (-3 points). There was a small, difference in discriminate function between SAPS3 and ACIS in the general ICU population (AUC 0.83 CI 95% 0.82 to 0.84) vs AUC 0.85 (95% CI 0.84 to 0.87) (Fig 3A). When stratified by age, SAPS3 performed fairly in prediction of ICU mortality in the 80 years or older population with an AUC of 0.72 (CI 95% 0.69 to 0.75; P < 0.0001). In contrast, the ACIS discriminate function was significantly, but slightly, superior to SAPS3 in the 80 years or older population (AUC 0.80 vs 0.72, P < 0.001) as demonstrated in Fig 3B. The accuracy of SAPS3 and ACIS in the other age strata is shown S1 Fig. ACIS presented a performance slightly superior to SAPS3 in the age group 80 years or older, this difference being statistically significant, and a similar performance in the other age groups (S3 Table). When comparing the predictive capacity of both scores for hospital mortality, SAPS3 and ACIS had similar performance in the general population (AUC 0.84 vs. 0.84 p = 0.75) and a difference in the 80 years or older group (AUC 0.73 vs. 0.78; p = 0.004) (S4 Table) and 59 years or younger group (0.90 vs. 0.86; p = 0.05), the last not being statistically significant.
(A) Adjusted and unadjusted binary regression model for ICU mortality. Univariable analysis yielded unadjusted odds of death. Multivariable regression adjusted for differences in baseline characteristics (variables of p<0.1 identified in univariable analysis). (B) ROC curve after K-10 fold-validation, showing model accuracy.
Comparison between SAPS3 and ACIS receiver operating characteristic (ROC) curves for prediction of ICU mortality on ICU general population (A) and 80 years or older population (B). The ACIS novel score outperformed the SAPS3 in the 80 years or older subset with similar discriminate function in the other age strata.
Discussion
This study evaluated the accuracy of the SAPS3 severity score in a large heterogeneous ICU cohort comprised of very elderly patients. In the general population, SAPS3 demonstrated good accuracy in prediction of ICU mortality and it performed fairly in the very elderly subset of our critically ill cohort. Prior studies have demonstrated similar reductions in the accuracy of SAPS3 and other severity scores in elderly patients admitted to the ICU [9]. Notably, the novel ACIS tool outperformed the SAPS3 in our 80 years or older population and most other age strata.
The ACIS novel scoring system is a practical bedside tool that determines disease severity and subsequent ICU mortality in the general and 80 years or older ICU population. Furthermore, the ACIS offers an alternative severity scoring system in clinical trials to determine study inclusion and to compare the severity of enrolled individuals with future or prior randomized studies [12–14]. In the absence of an accurate tool to determine the severity of those admitted to the ICU, critical care trials will continue to be burdened by misclassified heterogeneous populations. Inaccuracy of commonly used ICU scores is an underappreciated reality in both clinical management of those admitted to the ICU and interpretation of results from randomized clinical trials, particularly in the 80 years or older subset.
Factors associated with survival, rather than mortality in our cohort also revealed that patients from 60–69 years of age who were independent prior to admission with a BMI classification as overweight (25–29) had a higher probability of survival.
While our large cohort study confirmed the reasonable performance of SAPS3 in elderly ICU patients and derived the ACIS tool, there are several limitations that must be acknowledged. First, as a single center study there may be unknown factors related to the ICU population impacting the poor performance of SAPS3 or improved discriminate function of the ACIS novel system in our elderly subset. Given the retrospective collection of all relevant clinical data, there was a limitation due to unavailability or inaccessibility to variables that were not available in our electronic record, such as some laboratory tests and palliative care. Due to our relatively large cohort of 80 years or older, patient characteristics are unlikely to affect the performance of SAPS3 or ACIS. Second, as a study conducted in a tertiary ICU in an urban setting in Brazil, there may be local epidemiologic factors that could interfere with generalizability in the wider ICU population. While severity scores such as SAPS3 have been extensively studied in Europe and in the United States, this study represents one of the first performed in a resource limited setting to specifically evaluate the performance of SAPS3 and derive a novel severity tool. Our findings call into question the routine use of SAPS3 in the ICU, particularly among elderly patients whose prognosis may be inaccurately determined. Lastly, emerging evidence suggests that it may be the change in severity scores over time from admission that more accurately predicts ICU mortality.
In conclusion, the ACIS described here offers a robust and simple alternative to existing severity scores to predict ICU mortality and aid in triage, particularly in an increasingly elderly critical care population.
Supporting information
S1 Table. General survivors and non-survivors’ characteristics.
https://doi.org/10.1371/journal.pone.0240793.s001
(DOCX)
S3 Table. Comparison of ACIS and SAPS3 stratified by age.
https://doi.org/10.1371/journal.pone.0240793.s003
(DOCX)
S4 Table. Comparison of ACIS and SAPS3 stratified by age for hospital mortality.
https://doi.org/10.1371/journal.pone.0240793.s004
(DOCX)
S1 Fig.
Comparison between SAPS3 (A) and ACIS (B) receiver operating characteristic (ROC) curves stratified by age was performed to test accuracy of the indicated scores in prediction of ICU mortality. Discriminate function of the ACIS was superior to SAPS3 in the 80 years or older subset with comparable performance in the other age interval
https://doi.org/10.1371/journal.pone.0240793.s005
(TIF)
S2 Fig. Modified Kaplan-Meier analysis for an ACIS cutoff of 5 points.
Survival probability is significantly decreased in those over 75 years old with an ACIS score ≥5.
https://doi.org/10.1371/journal.pone.0240793.s006
(TIF)
S3 Fig. Determination of cut-off values using the associated Youden criterion on the ROC curve.
https://doi.org/10.1371/journal.pone.0240793.s007
(TIF)
References
- 1.
United Nations (2019) World Population Prospects 2019, Volume II. 1238
- 2. Boumendil A, Somme D, Garrouste-Orgeas M, Guidet B (2007) Should elderly patients be admitted to the intensive care unit? Intensive Care Med 33:1252 pmid:17404703
- 3. Muessig JM, Masyuk M, Nia AM, Franz M, Kabisch B, Kelm M, et al. (2017) Are we ever too old? Medicine (Baltimore) 96:e7776 pmid:28906362
- 4. Flaatten H, de Lange DW, Artigas A, et al (2017) The status of intensive care medicine research and a future agenda for very old patients in the ICU. Intensive Care Med 43:1319–1328 pmid:28238055
- 5. McDermid RC, Bagshaw SM (2011) Octogenarians in the ICU: Are you ever too old? Crit Care. https://doi.org/10.1186/cc10018 pmid:21371344
- 6. Remuzzi A, Remuzzi G (2020) COVID-19 and Italy: what next? Lancet. https://doi.org/10.1016/S0140-6736(20)30627-9 pmid:32178769
- 7. Metnitz PGH, Moreno RP, Almeida E, Jordan B, Bauer P, Campos RA, et al. (2005) SAPS 3—From evaluation of the patient to evaluation of the intensive care unit. Part 1: Objectives, methods and cohort description. Intensive Care Med 31:1336–1344 pmid:16132893
- 8. Minne L, Ludikhuize J, de Rooij SEJA, Abu-Hanna A (2011) Characterizing Predictive Models of Mortality for Older Adults and Their Validation for Use in Clinical Practice. J Am Geriatr Soc 59:1110–1115
- 9. Sanchez-Hurtado L, Ángeles-Veléz A, Tejeda-Huezo B, García-Cruz J, Juárez-Cedillo T (2016) Validation of a prognostic score for mortality in elderly patients admitted to Intensive Care Unit. Indian J Crit Care Med 20:695–700 pmid:28149026
- 10. Farhat JS, Velanovich V, Falvo AJ, Horst HM, Swartz A, Patton JH, et al. (2012) Are the frail destined to fail? Frailty index as predictor of surgical morbidity and mortality in the elderly. J Trauma Acute Care Surg 72:1526–1531 pmid:22695416
- 11. Kuhn M (2008) Building Predictive Models in R Using the caret Package. J Stat Softw. https://doi.org/10.18637/jss.v028.i05
- 12. (2017) Early, Goal-Directed Therapy for Septic Shock—A Patient-Level Meta-Analysis. N Engl J Med 376:2223–2234. pmid:28320242
- 13. Rochwerg B, Oczkowski SJ, Siemieniuk RAC, et al (2018) Corticosteroids in Sepsis. Crit Care Med 46:1411–1420. pmid:29979221
- 14. Molnar L, Berhes M, Papp L, Nemeth N, Fulesdi B (2015) Cerebral autoregulation testing in a porcine model of intravenously administrated E. coli induced fulminant sepsis. Crit Care 19:P1.