Development and validation of a nomogram to predict the mortality risk in elderly patients with ARF

Database and subjects

Subject data were retrieved from Medical Information Mart for Intensive Care III database version 1.4 (MIMIC-III v1.4) and the Second Affiliated Hospital and Yuying Children’s Hospital of Wenzhou Medical University in China. The MIMIC-III is a clinical database comprising the information of 46,520 patients who were admitted to the ICU of Beth Israel Deaconess Medical Center (BIDMC) in Boston, MA, from 2001 to 2012 (Johnson et al., 2016). MIMIC-III is a large, freely accessible database for international researchers upon a use agreement (certification number: 31355221). The database was approved by the institutional review boards (IRB) of the Massachusetts Institute of Technology (MIT) and BIDMC, and consent was obtained for the original data collection (unidentified health information of patients was used); therefore, the informed consent was waived in our study. The MIT and BIDMC are responsible for waiving the informed consent.

Participants

ARF was identified from ICD-9 code in the MIMIC-III database. For patients with multiple ICU admissions, we included only the first ICU admission. The primary outcome in this study was patients’ 30-day mortality (died within 30 days after hospitalization).

Data extraction

Demographical and laboratory variables were extracted from MIMIC-III database by Structure Query Language (SQL) at the first 24 h of ICU admission. We collected the following data: age, gender, vital signs, mechanical ventilation (including invasive and non-invasive), percutaneous oxygen saturation (SpO2), white blood cell (WBC), hemoglobin, platelet, albumin, bilirubin, blood urea nitrogen (BUN), lactate, activated partial thromboplastin time (APTT), prothrombine time (PT), 24 h urine output, sequential organ failure assessment (SOFA) score, and Acute Physiology Score III (APS-III). If the laboratory variables were examined more than once, the greatest severity value associated with the illness was used in our study. SOFA and APS-III scores were calculated within the first 24 h after the ICU admission. Patients with missing data, length of stay in ICU <24 h, and who were younger than 60 years old (Lloyd-Sherlock et al., 2012) were excluded from this study. Finally, 759 ARF patients were included in our analysis. All the scripts used to calculate the SOFA and APS-III scores were available from GitHub website (https://github.com/MIT-LCP/mimic-code/tree/master/concepts). Meanwhile, the same clinical data of ARF patients admitted to ICU in the Second Affiliated Hospital of Wenzhou Medical University from January 2010 to January 2020 were collected retrospectively. Inclusion criteria were length of stay in ICU ≥ 24 h, age ≥ 60 years old, and meeting the diagnostic criteria for acute respiratory failure (Tierney et al., 2020). Cases with missing data, advanced cancer, pregnancy, automatic discharge, and abandonment of treatment were excluded from the study. This study was approved by the Ethics Committee of the Second Affiliated Hospital of Wenzhou Medical University. IRB approval number: (2020) Ethical Approval No. 94.

Statistics analysis

Continuous data are presented as mean ± standard deviation (SD) or median (IQR) according to the normal or non-normality distribution. Kolmogorov–Smirnov test was performed to determine normal distribution. Categorical variables were presented as frequency(proportion). Continuous data were compared using the Student t-test or Mann–Whitney U test and proportion variables were compared using chi-squared test or Fisher exact tests, as appropriate. For the development of the nomograms, the univariate and multivariable logistic regression were used to identify prognosis factors from the training data set. A nomogram was formulated based on the results of multivariable analysis, a final model selection was performed by a backward stepdown selection process with the Akaike information criterion (Harrell Jr, KL & Mark, 1996). The ‘rms’ package was used for nomogram and calibration curve (Liu et al., 2020). The accuracy of the nomogram to predict the 30-day mortality of ARF was quantified using the concordance index (C-index), which is equal to the area under the Receiver Operating Characteristic (ROC) curve and ranges from 0.5 to 1 (Uno et al., 2011). The difference of C-index was compared by DeLong’s non-parametric approach (DeLong, DeLong & Clarke-Pearson, 1988). The calibration of the model is assessed by the calibration curves and determined using the Hosmer–Lemeshow goodness-of-fit test in the training set and validation set (Grant, Collins & Nashef, 2018). The ‘rmda’ package was performed for decision curve analysis (DCA) by quantifying the net benefits to assess the clinical value of the model (Vickers & Elkin, 2006). We did the statistics analyses and figures production using R software (version 3.6.1). All statistics tests were two-sided, and P values <0.05 were considered statistically significant.

Patient characteristics

A total of 1,432 patients were included in this study. Patients from MIMIC-III database (759 cases) were categorized into the training set, the 30-day mortality in the training set were 38.6%. The 673 patients from our hospital were categorized into the validation set, the 30-day mortality was 40.5%. The patient characteristics and laboratory findings of the training and validation sets are shown in Table 1. There were no statistically significant differences between the training and validation set.

Prognostic factors in the nomogram

Baseline demographic, laboratory variables, including SOFA and APS-III score for the prediction of 30-day mortality were determined using univariate logistic regression firstly. The heart rate, respiratory rate, systolic pressure, spo2, bilirubin, albumin, lactate, APTT, PT, BUN and 24 h urine output were prognostic factors of 30-day mortality in univariate logistic regression analysis. All above statistically significant prognostic factors were entered into the multivariable logistic regression for adjusting the confounding factors for 30-day mortality. The heart rate, respiratory rate, systolic pressure, SPO₂, albumin and 24 h urine output were independent prognostic factors for 30-day mortality (Table 2).

Prognostic nomogram for 30-day mortality

We established this nomogram, elderly ARF mortality prediction nomogram (e-ARF-MPN), which incorporated the significant prognostic factors from the multivariable analysis (Fig. S1, Fig. 1). This e-ARF-MPN had a C-index of 0.741 (95% confidence interval [CI], 0.706–0.777) for predicting the 30-day mortality in ARF patients. Meanwhile, we developed two more nomograms according to SOFA and APS-III scores (Fig. S2). The SOFA nomogram had a C-index of 0.653 (95% CI [0.613–0.693]) and the APS-III nomogram had a C-index of 0.707 (95% CI [0.670–0.744]) (Supplemental Information). Our e-ARF-MPN had a better predictive power than both the SOFA (P < 0.001) and APS-III (P = 0.039) scores. For example, the calibration curve for e-ARF-MPN predicted 30-day mortality observed for in ARF patients in the training set was better than the SOFA and APS-III calibration curves (Fig. 2). Hosmer-Lemeshow test showed nonsignificant statistic (chi-square = 9.132, P = 0.2197) in the training set.

External validation of the nomogram

The C-index of established e-ARF-MPN was 0.687 (95% CI [0.646–0.727]) for predicting 30-day mortality in the validation set. The C-index of APS-III and SOFA were 0.677 (95% CI [0.635–0.719]) and 0.613 (95% CI [0.569–0.657]), respectively. It has similar predictive power to the APS-III (P > 0.05), but significantly higher than the SOFA (P < 0.001). The calibration curves revealed adequate fit of the e-ARF-MPN and APS-III predicting the 30-day mortality in the validation set, which is significantly better than the calibration curve for SOFA (Fig. 3). Hosmer-Lemeshow test showed nonsignificant statistic (chi-square = 10.086, P = 0.2025) in validation set.

Decision Curve Analysis of the nomogram

The decision curve analysis (DCA) showed that this e-ARF-MPN had a large threshold probability range than the SOFA and APS-III scores. For example, whether mechanical ventilation should be used in a patient with respiratory failure. The decision curve analysis of our nomogram showed that the nomogram assisted- mechanical ventilation decision adds more net benefit to respiratory failure patients than either the treat-all-patients (all patients treated with mechanical ventilation) or treat-none-patients (no patients treated with mechanical ventilation) when the threshold probability is more than 8%. Compared to the net benefit of SOFA- and APS-III-assisted decisions, at the same threshold probability, e-ARF-MPN showed higher net benefit. It unveiled the clinical utility of proposed e-ARF-MPN (Fig. 4).

Currently, urinary output and serum creatinine are used to evaluate kidney function

However, a study has shown that serum creatinine was an unreliable indicator of acute changes in renal function (Li et al., 2016). Our study also showed that urinary output was superior to serum creatinine in predicting short-term mortality of elderly patients with ARF. Although the assessment of AKI stage is not necessarily based on urine volume, the initial postoperative urine volume was considered an accurate predictor of delayed graft function (Kim et al., 2019). The reduction of urinary output can be attributed to insufficient blood flow to the kidneys, due to reduced blood volume and systolic pressure. Albumin, synthesized by the liver, is considered an important factor associated with malnutrition among patients. It tends to improve the microcirculatory performance which supports the maintenance of major organ functions (Liu, Yang & Wei, 2017). Thus, albumin was regarded as an important biomarker to evaluate the poor prognosis of hospitalized patients (Akirov et al., 2017). Our research also showed that the risk of death increased gradually with the decrease of plasma albumin. Therefore, plasma albumin may play an important role in predicting the mortality of elderly patients with ARF.

Finally, the e-ARF-MPN incorporates 6 items of heart rate, respiratory rate, systolic blood pressure, SpO2, urinary output, and plasma albumin. In order to prove the calibration of the nomogram, clinical data was collected from different institutions. As is well known, the internal validity associated with the explanation of the results, and the external validity related to the generalizability of the results (Hong et al., 2019; Huebschmann, Leavitt & Glasgow, 2019). Through the internal and external validation data set analysis, the calibration of our e-ARF-MPN has been proved to be highly consistent, which was more accurate than APS-III (B) and SOFA scores. At present, SOFA score has been widely used in assessment of critical diseases (Gole, Srivastava & Neeraj, 2020; Mebazaa et al., 2018), especially in the prognosis of multiple organ failure. When compared with APS-III and SOFA scores, nomogram developed in this study has fewer indicators but has better discrimination and calibration. This means that our nomogram may be popularized to predict the outcome of elderly patients with ARF.

However, evaluating the clinical usefulness of our nomogram depends on how much it benefits the patient, not just its popularization (Huang et al., 2016). DCA is a novel method that has been widely used in the evaluation of clinical research effectiveness (Hijazi et al., 2016; Prabhu et al., 2019; Talluri & Shete, 2016). It offers insight into clinical consequences on the basis of threshold probability, from which the net benefit could be derived (Balachandran et al., 2015). According to the DCA results, the application value of our nomogram is better than that of APS-III (B) and SOFA scores.

Our study has several limitations. First, our study was single center study and this e-ARF-MPN was only been validated in our hospital. We need to validate our e-ARF-MPN in broad external population. Second, our e-ARF-MPN is only applicable to the elderly ARF patients. Third, we reported 30-day all-cause mortality instead of ARF specific cause of death. Fourth, the time frame of our database collection i.e., from 2001 to 2012 is broad. Recent advance in the ARF treatment, i.e., noninvasive ventilation, high frequency oscillatory ventilation, and early intubation, might have a role in mortality.