External validation of the REMEMBER score
Armin Darius Peivandi1* 
Henryk Welp1 Sebastian Kintrup2 Nana Maria Wagner2,† Angelo Maria Dell’Aquila1,†
- 1Department of Cardiothoracic Surgery, University Hospital Muenster, Muenster, Germany
- 2Department of Anesthesiology, Intensive Care and Pain Therapy, University Hospital Muenster, Muenster, Germany
Background: The use of venoarterial extracorporeal membrane oxygenation (VA-ECMO) after coronary artery bypass grafting (CABG) is associated with high in-hospital mortality rates. The pRedicting mortality in patients undergoing venoarterial Extracorporeal MEMBrane oxygenation after coronary artEry bypass gRafting (REMEMBER) score has been created to predict in-hospital mortality in this subgroup of patients. The aim of this study is to externally validate the REMEMBER score.
Methods: All CABG patients who received VA-ECMO during or after the operation at our center between 01/2012 and 12/2021 were included in the analysis. Discrimination was assessed using concordance statistics, visualized by ROC curve analysis. Calibration-in-the-large and Calibration slope were tested separately.
Results: A total of 107 patients (male: n = 78, 72.9%) were included in this study. The in-hospital mortality rate in our cohort was 45.8% compared with 55% in the original study. The REMEMBER score median predicted mortality rate was 52% (76.9–36%). However, the REMEMBER score showed low discriminative ability [AUC: 0.623 (p = 0.0244; 95% CI = 0.524–0.715)] and inaccurate calibration (intercept = 0.25074; p = 0.0195; slope = 0.39504; p = 0.0303), indicating poor performance.
Conclusions: The REMEMBER score did not predict in-hospital mortality and was therefore not applicable in our cohort of patients. Additional external validation studies in a multicenter setting are therefore advisable.
Introduction
Background and objectives
Postcardiotomy extracorporeal life support is associated with high in-hospital mortality rates. In a retrospective analysis, Raffa et al. found that the survival-to-discharge rate in patients was only 37% (1). Their retrospective study included not only patients who had undergone coronary artery bypass grafting (CABG) but also those who had undergone valve and aortic surgeries. The in-hospital mortality rate for the subgroup of patients receiving venoarterial extracorporeal membrane oxygenation (VA-ECMO) after CABG was equally high. In a meta-analysis, including an analysis of the data from 12 centers, the in-hospital mortality rate was 64.2% (2).
Furthermore, VA-ECMO therapy is associated with huge costs for the healthcare system. Recently, it was shown that the median cost of a hospital stay with ECMO therapy exceeded 100,000 $ per patient (3).
In light of these issues, a scoring system to identify patients who are likely to profit from VA-ECMO after CABG could be a beneficial tool for clinicians.
In order to predict in-hospital mortality in patients undergoing VA-ECMO after CABG, the pRedicting mortality in patients undergoing venoarterial Extracorporeal MEMBrane oxygenation after coronary artEry bypass gRafting score (REMEMBER score) was introduced recently. This score has six pre-ECMO parameters: older age, left main disease, inotropic score > 75, CK-MB > 130 IU/L, serum creatinine > 150 µmol/L, and platelet count < 100 × 109/L. In the REMEMBER score mentioned above, score calculation was based on 166 patients from a single center, and the results showed good discrimination (AUC 0.85, 95% CI 0.79–0.91) and calibration (Hosmer–Lemeshow χ2 p = 0.644) in this cohort (4).
Hence, the REMEMBER score is designed as an assessment instrument to select patients who can potentially benefit from VA-ECMO after CABG.
However, construction data of the REMEMBER score were obtained from only one institution. Therefore, a broader and potential worldwide application remains to be assessed.
In this regard, we aim to externally validate the REMEMBER score in our cohort of patients who received VA-ECMO after CABG.
Material and methods
Source of data/study design
This study is a retrospective, external validation study of a previously published scoring system (4). TRIPOD criteria provided the basis for this study and article structuring (5).
Participants
The reported validation is a single-center analysis. All CABG patients who received VA-ECMO during or after the operation at our center between 01/2012 and 12/2021 were included in the analysis.
Outcome
The outcome, predicted by the REMEMBER score, was in-hospital mortality.
Predictors
Score variables were age, left main disease, inotropic score > 75, CK-MB > 130U/L, serum creatinine > 150 µmol/L, and platelet count < 100 × 109/L. Points were classified as weighted and assigned in the original score publication (4). The lowest values recorded within 6 h before ECMO implantation were considered for analysis. If the laboratory values were not measured within the last 6 h before ECMO (elective patients), the last values before implantation were taken for score calculation.
Missing data
Patients with missing data (missing variables for score calculation) (n = 11) were excluded from the analysis.
Sample size
Exclusion criteria were adapted in accordance with the original score publication (age <18 years, VV-ECMO, ECMO before CABG operation or more than 7 days after operation, and concomitant major cardiac surgeries) (4). After the exclusion process, our study included a total of 107 patients.
Statistical analysis methods
Patient characteristics were reported as either n (%), median (Q 3–1), or mean (std). The normality of continuous variables was assessed using the Shapiro–Wilk test. The patients were divided into four risk groups (I: lowest risk, IV: highest risk) as described in the original study (4). In-hospital mortality rates of the different risk groups were assessed. The ABCD model for validation was used for score analysis (6). For calibration analysis, the model proposed by Cox et al. (7) was used. Calibration-in-the-large and Calibration slope were assessed. In this regard, the predicted mortality rate was obtained using the prediction formula described in the original score analysis [1/(1 + exp. (3.856–0.220 * score)] (4). Discriminative score ability was quantified using concordance (C)—statistic, visualized by a ROC curve.
Ethical approval
The local ethics committee (“Ethikkommission der Ärztekammer Westfalen Lippe und der Westfälischen Wilhelms Universität”) provided ethical approval for a retrospective analysis of ECMO patients (2015-547-f-S). Because of the retrospective nature of the study design, informed consent of each individual was not needed and was therefore waived.
Results
Participants
A total of 107 patients (male: n = 78, 72.9%) with a median age of 68 (Q 3–1: 74–62) who received VA-ECMO during or after CABG surgery between 2012 and 2021 were included in our study. Unfortunately, 49 patients (45.8%) died during their hospital stay. Those who underwent left coronary artery main stenosis accounted for 54.2% (n = 58). In 55 patients (51.4%), VA-ECMO was implanted intra-operatively, and in 52 patients (48.6%), implantation was done after surgery. Detailed patient characteristics and score variables are summarized in Table 1.
Table 1. Patient characteristics pre-VA-ECMO (score constructing variables: lowest values 6 h pre-ECMO) and in-hospital mortality.
Model performance
The REMEMBER score median predicted mortality rate in our cohort was 52.6%. The observed mortality rates in the risk groups were I: 28%, II: 44.7%, III: 64.3%, and IV: 36.8%. This means that in-hospital mortality in the highest risk group (IV) was lower compared with risk groups II and III (Figure 1). This indicates overestimation of the scores in high values.
Figure 1. Observed in-hospital mortality rates in different risk groups, built according to the original study from I (lowest risk) to IV (highest risk): risk group IV showed a lower in-hospital mortality rate than risk groups II and III, indicating a mortality overestimation by the score in higher values.
Figure 2 shows the REMEMBER score discrimination in our cohort of patients. Although the score median predicted mortality was close to observed mortality, the score showed a low discriminative ability with an AUC of 0.623 (p = 0.0244; 95% CI = 0.524–0.715.
Figure 2. ROC curve showing low discrimination of the REMEMBER score in our cohort [AUC of 0.623 (p = 0.0244; 95% CI = 0.524–0.715)].
As shown in Figure 3, the REMEMBER score was also not accurately calibrated in our cohort. Calibration-in-the-large (intercept) was 0.25074 (std. error 0.10566, t-value 2.3731, p = 0.0195) and the Calibration slope was 0.39504 (std. error 0.17991, t-value 2.1958, p = 0.0303). This indicates that low prediction values of the score underestimate mortality, while higher values overestimate mortality.
Figure 3. Calibration curve, demonstrating that low prediction values of the REMEMBER score underestimate mortality, while higher values overestimate the same [calibration-in large (intercept) 0.25074 (std. error 0.10566, t-value 2.3731, p = 0.0195), Slope 0.39504 (std. error 0.17991, t-value 2.1958, p = 0.0303)].
Comment
The REMEMBER score was developed to predict in-hospital mortality after VA-ECMO implantation following CABG surgery. Within our cohort of 107 CABG patients who suffered from peri- or postoperative cardiogenic shock requiring VA-ECMO implantation, the score was not able to predict in-hospital mortality.
When discussing the reasons for the poor performance of this score in our cohort of patients, three main issues should be considered:
First, the establishment of a scoring system consists of three essential steps: derivation, validation, and impact analysis (8). While derivation depends on the original population of the score, validation is an ongoing task, comprising external analyses in different study populations. In our study, we intended to provide the basis for such a validation process. As no population of patients is exactly equal, we also noted some differences in the baseline characteristics of our patients.
In comparison with the development data of the REMEMBER score, the patients in our cohort were older (68 vs. 61 years). They had a higher incidence of left main disease (54.2% vs. 31%), a lower median CK-MB (65 vs. 143 IU/L), a lower median serum creatinine level (97.2 vs. 121 µmol/L), almost equal median platelet counts (200 vs. 203 × 109/L), and a lower median inotropic score (50 vs. 75). In our cohort, emergency surgery was more common (62.6% vs. 15%). In addition, the collective of Wang and colleagues’ consisted of 83 (50%) off-pump CABG patients, whereas almost every patient in our collective had undergone cardiopulmonary bypass [nOPCAB = 3, (2.8%)]. In addition, our cohort showed a higher median EuroSCORE (8 vs. 6) and a higher percentage of cardiopulmonary resuscitation (CPR) (20.6 vs. 17%) (4).
The differences, especially in EuroSCORE, on-pump vs. off-pump ratio, and CPR, indicate that our cohort of patients reflects a more typical present-day postcardiotomy VA-ECMO cohort than the original study population. An ideal scoring system should be applicable in such a cohort so that it may become clinically relevant in everyday patient care.
In the REMEMBER score–deriving cohort, VA-ECMO was established through peripheral cannulation. As most of the VA-ECMOs in our cohort were inserted through central arterial cannulation [67 (62.6%)], we performed a subanalysis of peripheral cannulated patients to investigate whether the REMEMBER score was too specific. In this subgroup, however, score discrimination was even worse (AUC: 0.465, 95% CI: 0.306–0.629, p: 0.7034).
Although differences in study populations have an impact on score performance, an ideal scoring system should be applicable in all cohorts of plausibly related patients. Thus, external validation is a sine qua non for clinical application and, hence, impact analysis. In this regard, the REMEMBER score did not achieve promising results in our cohort of patients.
Second, a question arises as to whether the right variables were chosen for score construction in the first place. Another score that aims to assess survival prognosis in VA-ECMO patients is the PREDICT VA-ECMO score, developed by Wengenmayer and colleagues. This score mainly focuses on pH, lactate levels, and serum bicarbonate concentrations. These three parameters were missing in the REMEMBER score (9). The importance of serum lactate levels as a predictive variable in mechanical circulatory support is also highlighted in a multicenter prospective cohort study by Scolari et al. They found serum lactate and lactate clearance at 24 h to be independent predictors of short-term survival (10). In comparison with the REMEMBER score, both studies focus on predictions made shortly after VA-ECMO implantation. The impact of pre-ECMO lactate levels on ECMO survival remains to be assessed in CABG patients.
The SAVE score, yet another published score for mortality prediction in VA-ECMO patients, comprises 12 different parameters. In contrast to the REMEMBER score, this score includes more perioperative parameters, such as, but not limited to, acute organ failure (liver, central nervous system, kidneys) and respiratory and hemodynamic parameters (11).
In addition, when examining the score constructing variables of the REMEMBER score, it is found that the pre-ECMO platelet count was used as a predicting parameter. A recently published study showed that a relative decrease in platelets on day 1 from ECMO initiation was an independent risk factor for mortality (12). In contrast, major complications resulting from ECMO, such as bleeding and heparin-induced thrombocytopenia, affect the platelet count. In this regard, platelet count serves as a surrogate parameter. Against this background, it becomes questionable whether the pre-ECMO platelet count alone is able to predict mortality. More studies are needed in the future to examine this issue.
Third, perioperative treatment protocols can also have an impact on survival on VA-ECMO after coronary surgery. In this regard, weaning strategies and conditions and a switch to a left-ventricular-assist device (LVAD) are only two examples of confounders affecting survival and therefore misleading score results (13).
In the original REMEMBER score study, 80% of the patients had a running intra-aortic balloon pump treatment prior to VA-ECMO implantation. As this setting leads to improved coronary perfusion and a beneficial reduction in afterload, ECMO implantation might have taken place under more “balanced” circumstances (14). This could have ultimately weakened the mortality prediction of the score once again.
Conclusion
In our external validation study, we found that the REMEMBER score did not adequately forecast in-hospital mortality. This was true for both discrimination and calibration. The potential reasons for the poor score performance in our patient cohort have been discussed in detail.
Limitations
This validation study is a single-center study with 107 patients. Additional external validation in different cohorts is needed to conclude whether the REMEMBER score should be verified or dismissed.
Data availability statement
The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.
Ethics statement
The studies involving human participants were reviewed and approved by Ethikkommission der Ärztekammer Westfalen Lippe und der Westfälischen Wilhelms Universität. Written informed consent for participation was not required for this study in accordance with the national legislation and institutional requirements.
Author contributions
AP wrote the original draft and conducted the statistical analysis; HW and SK were involved in data acquisition and writing–review and editing; NW was engaged in supervision of the study and writing—review and editing; AD was involved in the conception and supervision of the study and writing—review and editing. All authors contributed to the article and approved the submitted version.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1. Raffa GM, Gelsomino S, Sluijpers N, Meani P, Alenizy K, Natour E, et al. In-hospital outcome of post-cardiotomy extracorporeal life support in adult patients: the 2007–2017 Maastricht experience. Crit Care Resusc. (2017) 19(Suppl 1):53–61. PMID: 29084502.29084502
2. Biancari F, Dalén M, Perrotti A, Fiore A, Reichart D, Khodabandeh S, et al. Venoarterial extracorporeal membrane oxygenation after coronary artery bypass grafting: results of a multicenter study. Int J Cardiol. (2017) 241:109–14. doi: 10.1016/j.ijcard.2017.03.120
3. Chung M, Zhao Y, Strom JB, Shen C, Yeh RW. Extracorporeal membrane oxygenation use in cardiogenic shock: impact of age on in-hospital mortality, length of stay, and costs. Crit Care Med. (2019) 47(3):e214–21. doi: 10.1097/CCM.0000000000003631
4. Wang L, Yang F, Wang X, Xie H, Fan E, Ogino M, et al. Predicting mortality in patients undergoing VA-ECMO after coronary artery bypass grafting: the REMEMBER score. Crit Care. (2019) 23(1):11. doi: 10.1186/s13054-019-2307-y
5. Collins GS, Reitsma JB, Altman DG, Moons KG. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ. (2015) 350:g7594. doi: 10.1136/bmj.g7594
6. Steyerberg EW, Vergouwe Y. Towards better clinical prediction models: seven steps for development and an ABCD for validation. Eur Heart J. (2014) 35(29):1925–31. doi: 10.1093/eurheartj/ehu207
7. Cox DR. Two further applications of a model for binary regression. Biometrika. (1958) 45:562–5. doi: 10.1093/biomet/45.3-4.562
8. Labarère J, Renaud B, Fine MJ. How to derive and validate clinical prediction models for use in intensive care medicine. Intensive Care Med. (2014) 40(4):513–27. Erratum in: Intensive Care Med. (2014) 40(6):925. doi: 10.1007/s00134-014-3227-6
9. Wengenmayer T, Duerschmied D, Graf E, Chiabudini M, Benk C, Mühlschlegel S, et al. Development and validation of a prognostic model for survival in patients treated with venoarterial extracorporeal membrane oxygenation: the PREDICT VA-ECMO score. Eur Heart J Acute Cardiovasc Care. (2019) 8(4):350–9. doi: 10.1177/2048872618789052
10. Scolari FL, Schneider D, Fogazzi DV, Gus M, Rover MM, Bonatto MG, et al. Association between serum lactate levels and mortality in patients with cardiogenic shock receiving mechanical circulatory support: a multicenter retrospective cohort study. BMC Cardiovasc Disord. (2020) 20(1):496. doi: 10.1186/s12872-020-01785-7
11. Schmidt M, Burrell A, Roberts L, Bailey M, Sheldrake J, Rycus PT, et al. Predicting survival after ECMO for refractory cardiogenic shock: the survival after veno-arterial-ECMO (SAVE)-score. Eur Heart J. (2015) 36(33):2246–56. doi: 10.1093/eurheartj/ehv194
12. Wang L, Shao J, Shao C, Wang H, Jia M, Hou X. The relative early decrease in platelet count is associated with mortality in post-cardiotomy patients undergoing venoarterial extracorporeal membrane oxygenation. Front Med. (2021) 8:733946. doi: 10.3389/fmed.2021.733946
13. Bertoldi LF, Pappalardo F, Lubos E, Grahn H, Rybczinski M, Barten MJ, et al. Bridging INTERMACS 1 patients from VA-ECMO to LVAD via impella 5.0: de-escalate and ambulate. J Crit Care. (2020) 57:259–63. doi: 10.1016/j.jcrc.2019.12.028
Keywords: extracorporeal life support, coronary artery bypass grafting, score system, cardiac failure, risk score
Citation: Peivandi AD, Welp H, Kintrup S, Wagner NM and Dell’Aquila AM (2023) External validation of the REMEMBER score. Front. Cardiovasc. Med. 10:1192300. doi: 10.3389/fcvm.2023.1192300
Received: 23 March 2023; Accepted: 17 July 2023;
Published: 28 July 2023.
Edited by:
Guo-wei Tu, Fudan University, ChinaReviewed by:
Ilija Djordjevic, University Hospital of Cologne, GermanyMiha Antonic, Maribor University Medical Centre, Slovenia
© 2023 Peivandi, Welp, Kintrup, Wagner and Dell'aquila. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Armin Darius Peivandi armindarius.peivandi@ukmuenster.de
†These authors have contributed equally to this work