Article Text
PDF
Abstract
Objective We aimed to develop and validate a score to assess inpatient complexity and compare its performance with two currently used but not validated tools to estimate complexity (ie, Charlson Comorbidity Index (CCI), patient clinical complexity level (PCCL)).
Methods Consecutive patients discharged from the department of medicine of a tertiary care hospital were prospectively included into a derivation cohort from 1 October 2016 to 16 February 2017 (n=1407), and a temporal validation cohort from 17 February 2017 to 31 March 2017 (n=482). The physician in charge assessed complexity. Potential predictors comprised 52 parameters from the electronic health record such as health factors and hospital care usage. We fit a logistic regression model with backward selection to develop a prediction model and derive a score. We assessed and compared performance of model and score in internal and external validation using measures of discrimination and calibration.
Results Overall, 447 of 1407 patients (32%) in the derivation cohort, and 116 of 482 patients (24%) in the validation cohort were identified as complex. Eleven variables independently associated with complexity were included in the score. Using a cut-off of ≥24 score points to define high-risk patients, specificity was 81% and sensitivity 57% in the validation cohort. The score’s area under the receiver operating characteristic (AUROC) curve was 0.78 in both the derivation and validation cohort. In comparison, the CCI had an AUROC between 0.58 and 0.61, and the PCCL between 0.64 and 0.69, respectively.
Conclusions We derived and internally and externally validated a score that reflects patient complexity in the hospital setting, performed better than other tools and could help monitoring complex patients.
- primary care
- general medicine (see internal medicine)
- quality in health care
- social medicine
- internal medicine
Data availability statement
Data are available on reasonable request from fabian.liechti@insel.ch (ORCID ID 0000-0003-1006-6903).
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
- primary care
- general medicine (see internal medicine)
- quality in health care
- social medicine
- internal medicine
Strengths and limitations of this study
This is a prospective cohort study of consecutive, unselected, adult inpatients discharged from the department of medicine of a large university hospital.
We derived and validated an easily usable score that accurately assesses patient complexity in medical inpatients that may help monitoring the proportion of complex patients (Patient Complexity Assessement (PCA) score).
The reference standard used to define complexity was the physician’s judgement, which per definition is partly subjective.
The PCA score has been developed at a single tertiary hospital and may not consider a comprehensive list of important indicators.
The PCA score includes values available only at discharge and indicators are not modifiable.
Introduction
One-fourth of patients are estimated to be complex in the primary care setting, while this proportion is not well known in the hospital setting.1–4 Generally, those patients using more resources, time and/or effort are regarded as complex patients, although no universal definition of patient complexity is available. Complexity is not limited to multimorbidity and chronicity of disease but depends also on multiple other aspects, including psychological, social, economic and environmental factors.1 2 5–7 Complex patients challenge the current structures, for example, they have a higher probability of future emergency department utilisation (without higher mortality rates) and show suboptimal use of the healthcare system.2 8–10 Identifying complex patients is of economic, epidemiological and social importance because it may help to better allocate resources and improve healthcare utilisation.5 11
The only available assessment method to identify complex inpatients is currently the physician’s assessment, which limits the monitoring of patient complexity over time.10 12 13 The Charlson Comorbidity Index (CCI), originally developed and validated to predict mortality,14 has been assessed as a proxy for patient complexity in the primary care setting. However, agreement between the primary care physician’s assessment and the CCI to identify complex patients was only modest.1 2 5 No such assessment has been yet performed in the hospital setting. The patient clinical complexity level (PCCL) is calculated for each treatment episode to indicate the effect of complications and comorbidities in a patient. The PCCL ranges from 0 (no complication or comorbidity) to 4 (very severe complication or comorbidity), according to a complex algorithm.14 15 Identification of complex patients at discharge could help to identify those, who would profit from more intense follow-up, for example, by general practitioners or social workers, although effectiveness of such interventions would have to be proven first.
In order to simplify and standardise the identification of complex patients, we aimed to develop and validate a new score to help identifying the most complex inpatients (Patient Complexity Assessment, PCA score) using readily available administrative and clinical data. Our hypothesis was that some data routinely collected during a hospitalisation can be used as a valuable surrogate to physician’s assessment. We then compared the performance of the newly developed PCA score to the CCI, and the PCCL used in the Swiss DRG system to allocate reimbursement according to multimorbidity.14 15
Methods
Study design and participants
This study was a prospective cohort of consecutive, unselected, adult inpatients discharged from the department of medicine of a large University hospital (Inselspital, Bern University Hospital, Bern, Switzerland) between 1 October 2016 and 31 March 2017. The only exclusion criterion was a previous study inclusion. We originally planned to consider around 35 variables in the prediction model. With an estimated proportion of complex patients of one fourth, we preset the sample size of the derivation cohort to be 1400 (rule of thumb of 10 outcomes per variable tested).16 17 We predefined, that if more than 1400 patients will be included during the study period of 6 months, we would use these patients to externally validate the prediction model. Patients enrolled before 16 February 2017 were allocated to the derivation sample (derivation and internal validation cohort), and patients enrolled after this date were allocated to an external validation sample (temporal validation cohort). During their first admission, all patients included in the study gave their written general consent to the use of their routine data for research purposes. We reported the study in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis statement.18
Patient and public involvement
No patient involved.
Study outcome and predictor variables
The primary outcome was the predictive accuracy of the PCA against the treating physician’s judgement as the gold standard to identify complex general internal medicine inpatients. Complex patients were defined as those using more resources, time and/or effort while hospitalised. The resident (or supervising consultant) was asked by a trained study nurse to assess at time of discharge the level of complexity of the entire hospital stay of her/his patient without providing any specific scoring system (complex or not-complex).
The CCI was originally developed to predict 10-year survival by using an algorithm based on addition of score points for specific diagnoses.14 19 The PCCL was derived from the electronic health record (no complication or comorbidity: 0; light complication or comorbidity: 1; moderate complication or comorbidity: 2; severe complication or comorbidity: 3; very severe complication or comorbidity: 4 and is defined by SwissDRG.20 21
For all patients, information regarding International Classification of Disease (ICD) codes and other potential indicators for patient complexity were collected retrospectively through the electronic health record of the hospital. Candidate predictor variables have been selected based on a previous survey among general internists in the hospital setting which asked them to identify factors that contribute to patient complexity,4 and on a selection of readily available potential predictors to have a broad spectrum of candidate predictors. Variables that were not routinely collected were removed (ie, variables with more than 25% missing data, such as aspartate amino transferase, C reactive protein and albumin at discharge). Collinearity between variables was assessed using Pearson correlation coefficients. In case of strong correlation (r>0.7), only the strongest univariate predictor was kept. A final list of 52 indicators was considered in denoting complexity: baseline demographic information (age, gender, living area (rural vs urban, defined according to the Swiss Federal Statistical Office based on the patient’s place of residence), marital status, institutional care before admission, nationality (Swiss vs non-Swiss), hospital variables (urgent vs elective admission, number of previous hospitalisation in the last 12 months, patient destination (death, home, other hospital, nursing home, rehabilitation, other), stay on the intensive care unit, internal transfer), drugs (for each group of the Anatomical, Therapeutic and Chemical classification categories) at admission and at discharge and polypharmacy (≥10 drugs,22 at admission and discharge), main diagnosis (cancer, chronic obstructive pulmonary disease, dementia, depression, heart failure, pneumonia, sepsis, stroke, substance abuse, syncope, malnutrition, based on the 10th Revision of the ICD and Related Health Problems code), number of diagnoses at discharge, CCI, laboratory values (haemoglobin, leucocyte count and thrombocyte count, serum sodium and creatinine) at admission (first lab values at admission) and discharge (last lab values before discharge), number of interventions and costs (normal vs high costs, ie, ≥ the 75th empirical percentile value) during hospitalisation of blood products, drugs, imaging procedures, physiotherapy and nursing workload.
Missing data
When missing, the value of haemoglobin and creatinine at discharge was assumed to be identical to the value at admission. When missing, the value of sodium and platelet count at discharge was considered normal. For other potential indicators of complexity, we assumed data to be missing at random and imputed missing data using single imputation by chained equations. To compare performance measures of the PCA with the CCI and PCCL, patients with missing values for the PCCL variable (n=3 for the derivation, n=11 for the validation dataset) were removed prior to analysis.
Statistical analysis
Multivariable logistic regression analysis with backward selection was used in the derivation set to predict complexity based on 52 potential indicators of complexity variables registered during hospitalisation, removing variables with a p>0.1. Calibration of the final model was evaluated by constructing a calibration curve, estimating the calibration slope, calculating the difference between the mean observed proportion and mean predicted proportion of patients with high complexity (calibration-in-the-large) and the Brier score (overall measure of accuracy) in the derivation and validation set. The predictors from the final model were used to create a comprehensible score using the regression coefficient-based scoring technique.23 Beta-coefficients were divided by the lowest coefficient and rounded up to the closest integer to generate score points, indicating increasing risk by higher scores. The discriminatory power of the score was assessed by calculating the area under the receiver operating characteristic (AUROC) curve.
The validity of performance measures was investigated by performing internal and external validation. For internal validation we used 1000 bootstrap samples, drawing samples with replacement from the derivation sample.24 The bootstrap-corrected performance estimates were calculated by subtracting the optimism from the performance of the original model. The 95% CIs for the bootstrapped performance measures were derived using the percentile method. External validation was made by estimating the same performance measures in the external validation cohort (temporal validation).
The clinical usefulness of the developed score was assessed with a decision-curve analysis investigating whether the use of the complexity score instead of the CCI alone was associated with benefit gains relative to the prediction complexity.25
Applying PCA, CCI and PCCL, we calculated the score of each patient and split the patient sample into a high-risk and a low-risk group. The reference point (cut-off) of each scoring system was chosen in order to make the frequency of patients in the high-risk category as close as possible to 30% (ie, approximating the frequency of observed complex patients). To determine the accuracy of this method to predict complexity, we estimated sensitivity, specificity and positive and negative predictive value in both the derivation and validation set for PCA and in the derivation set for CCI and PCCL.
R V.3.3.1 was used for statistical analysis.
Results
A total of 1889 patients were included in the study (figure 1). Patients enrolled before 16 February 2017 were allocated to the derivation sample (n=1407), patients enrolled after this date (n=482) were allocated to the temporal validation sample. In the derivation cohort, 447 patients (31.8%) were clinically judged as complex and 116 (24.1%) patients in the validation cohort. The patients in the two cohorts presented with similar baseline characteristics (table 1 and online supplemental material table S1 and S2). The overall median age was 80 years (IQR 75–86 years).
Supplemental material
Baseline characteristics for all patients (derivation and validation cohort) stratified by complexity, as number and percentage or median and IQR for categorical and continuous variables, respectively
Flow chart. Derivation sample (derivation and internal validation cohort) and external validation sample (temporal validation cohort).
After backward selection, 11 of the 52 potential predictors were used to derive the PCA score (table 2). Besides diagnosis-related factors, they represented demographic characteristic, hospital variables, medication and laboratory values. Highest score points were assigned to leucocytosis (at discharge only, 16 points and at admission and discharge, 10 points) followed by age under 60 years, high nurse workload (costs above 75th percentile for nursing expenses), and abnormal serum creatinine at discharge (≥100 µmol/L).
PCA score weighted according to coefficients
The prediction model showed a good accuracy, with a Brier score of 0.17 and 0.15 in internal and external validation, respectively. The calibration curve showed fair agreement between predicted and observed proportions of complexity in the derivation cohort and slightly lower observed proportions than predicted probabilities in the validation cohort (graphs not shown). The calibration-in-the-large coefficient of −0.51 (95% CI −0.74 to −0.27) in the validation cohort implies that the mean observed proportion was lower than the mean predicted probability. However, the calibration curve slope was satisfactory in internal and external validation (0.93 (95% CI 0.80 to 1.05) and 0.96 (95% CI 0.74 to 1.18)), respectively.
The median score was 17 points in the derivation and validation cohort (mean 18.77 and 19.03, respectively). The minimal score was 0 points in both cohorts, the maximal score reached was 54 points in the derivation cohort and 53 points in the validation cohort (theoretically maximal 81 score points). The score’s AUROC curve was 0.77 (95% CI 0.74 to 0.79) and 0.78 (95% CI 0.74 to 0.82) in internal and external validation.
We classified patients as low and high complexity risk (table 3) according to the selected cut-off of 24 points (approximating the frequency of observed complex patients of 30%). The proportion of patients categorised as complex (ie, score ≥24 points) was 30% and 28% in the derivation and validation dataset, respectively. Sensitivity was 57% in both the derivation and validation dataset. The specificity was 83% and 81%, respectively. Positive predictive values were 61% and 49% in the derivation and validation cohort, respectively, while negative predictive values were 81% and 86%, respectively. The discriminatory power of the PCA score was robust with an AUROC of 0.77 (95% CI 0.74 to 0.79) in internal validation (bootstrap-corrected value) and 0.78 (95% CI 0.74 to 0.82) in external validation (table 4 and online supplemental figure S2).
Supplemental material
Stratification of observed versus predicted complex patients applying the PCA score
In comparison, predictive accuracy of the CCI was lower compared with the PCA score. The AUROCs were low with 0.58 (95% CI 0.55 to 0.62) and 0.62 (95% CI 0.56 to 0.68) in the derivation and validation cohort, respectively (table 4). Sensitivity of the CCI reached 36% (95% CI 31% to 40%) and 41% (95% CI 31% to 50%) in derivation and validation cohort, respectively, while specificity was 76% (95% CI 73% to 78%) and 75% (95% CI 71% to 80%), respectively. The decision curve analysis (online supplemental figure S1) indicates a superiority of the PCA score compared with the CCI to predict complexity.
Measures of performance to predict complexity
AUROCs of PCCL were between those of CCI and PCA score with 0.64 (95% CI 0.61 to 0.67) and 0.69 (95% CI 0.64 to 0.75) in the derivation and validation cohort, respectively (table 4). Sensitivity was 52% (95% CI 47% to 56%) and 61% (95% CI 51% to 70%), respectively, while specificity was 73% (95% CI 71% to 76%) and 75% (95% CI 70% to 79%).
Discussion
We derived and validated the PCA score that accurately assessed patient complexity in medical inpatients. The final score of eleven independent and readily available factors, included age, hospital variables, diagnosis-related aspects and laboratory variables. The PCA score showed overall good performance with a discriminatory power of 0.78 that surpasses other comorbidity-based tools such as the CCI and the PCCL.
In this cohort of medical inpatients, 32% and 24% were considered ‘complex’ by the treating physician, in the derivation and the validation cohort, respectively. This first estimate of patient complexity in the hospital setting is consistent with a previous assessment in an outpatient population where 26% of total 4302 patients were categorised as being complex by a primary care physician.1 Based on these data, the authors later derived a model to identify around 20% of 143 372 primary care patients as complex. Using the model and outpatient CCI or PCCL, only modest agreement between the methods was observed (37% and 40%, respectively).2 Therefore, a tool not solely based on multimorbidity, such as the newly developed PCA score, seems to better identify complex patients.
In the present study, age was an inverse indicator of complexity. In a previous study of outpatients, mean age of complex patients was 60 years vs 48 years in non-complex patients.1 Nonetheless, the same study reported noteworthy age-related variability: in younger patients the association of certain diagnoses (eg, alcohol-related diseases) with complexity was stronger, and deprivation as contributor to complexity is independent of age.1 5 In our setting, discharge planning processes for older patients may be better established (eg, including hospital social services, decision making based on patient’s provision and possibility for indiscriminate discharge to geriatric rehabilitation facilities or nursing homes) compared with younger patients.1 9 26 27 Treating physicians may therefore perceive the discharge planning process of some younger patients as difficult and categorise these patients as complex. Furthermore, young non-complex patients may more often be treated as outpatients or by specialist’s clinics instead of our tertiary care general internal medicine ward. Elective admissions to a tertiary hospital may represent a cohort of rather complex patients preselected by primary care physicians and smaller hospitals (21% elective admissions in complex patients vs 14% in non-complex patients). The inverse relationship between age and complexity, and the relationship between elective admissions and complexity may therefore represent structural incentives to hospitalise complex younger patients which overburden outpatient care. It is also possible that these patients are only perceived as more complex by the treating physician because patients admitted directly to the medical ward are pending initial workup otherwise provided in the emergency department.
Patients with high costs of imaging procedures may reflect the patients with more severe diseases or more diagnosis uncertainty. Similarly, high costs for laboratory analysis may be explained by a higher need of costly or repeated measurements in more complex patients. High costs for care/nursing were indicators of complexity highlighting some concordance between the nurse workload and the medical complexity.
In our study multimorbidity (defined as a number of more than six diagnoses) was an indicator of complexity. Comorbidity-based scores, that is, the CCI, are commonly used to identify complex patients. Indeed, in the study of Grant et al the proportion of multimorbid patients identified by a CCI of 2 or more was higher in complex patients, that is, 26% of complex patients were multimorbid vs 9% of non-complex patients.1 However, many multimorbid patients are not complex and not all complex patients are multimorbid. In our cohort (derivation and validation datasets together) 34% of polymorbid patients (CCI≥2) were complex vs 24% in the group of CCI <2. Comparably, nearly one half of patients with a CCI of 2 or greater were classified as non-complex in the study of Grant et al.1 Therefore, a system to identify complexity should not depend on diagnosis alone.
In the PCA score, malnutrition was a risk factor of complexity. Malnutrition in hospitalised patients is associated with more complications, increased mortality, longer hospital stays and higher costs.28 29 Therefore, malnutrition and complexity may both reflect a cluster of severe and chronic disease as well as socioeconomic circumstances.1
Antineoplastic and immunomodulating medication at admission was an indicator of complexity. These drugs are used for oncologic patients, but also in patients with rheumatological disease or after receiving organ transplants. These patients may be complex because of challenging infectious diseases, end-of-life issues and interdisciplinary care. Abnormal values of serum creatinine and leucocyte counts at discharge were denoting complexity whether the values were normal or abnormal at admission. These patients may also requiring more interaction between specialists and may complicate the discharge process.
Personal characteristics or mental health issues and use of psychoactive medication, that is, narcotics, selective serotonin reuptake inhibitors, benzodiazepines, smoking cessation agents and antipsychotics, have been described as characterising complex patients, especially in younger patients.1 This is in line with the observation that in the PCA score, use of medication affecting the nervous system at admission (including antipsychotics, mood-stabilisers, sedatives, analgesics including opioids, anticonvulsive medication and anti-dementia drugs) was an indicator of complexity. These patients may challenge the known pathways of the healthcare system, for example, by parallel use of general internal medicine and psychiatric resources.
There are several limitations of the study. First, we used physician’s assessment to define complexity, which per definition is subjective. Nonetheless, there is no better standard reference (gold standard) and the proportion of patients identified as complex was similar in previous studies.1 2 Second, the PCA score has been developed at a single tertiary hospital in Switzerland and therefore may not be generalisable to other settings, for example, other health care systems. However, costs and nursing workload are not measured as absolute values but as those above the 75th percentile, making it transferable to other settings. Also, some patients may appear as complex in one setting, while they will be judged as non-complex in other settings (eg, primary care vs university hospital), nevertheless the proportion of complex patients in out setting was similar to the one in primary care.1 Therefore, in other health systems the final indicators may vary, which might be considered when validating the PCA score. Third, it is likely that our model does not consider every important indicator, but it allows deriving an easily usable tool which kept its fair sensitivity and good specificity in our external validation. Fourth, the PCA score includes values available only at discharge, which makes patient-aimed interventions during the hospitalisation difficult. This is however also true for alternative assessment tools, such as the CCI and the PCCL, which had a lower performance in identifying complex patients in our cohort. Fifth, imputation of missing data may have changed the outcome of the study. However, potential predictors with more than 25% missing data were excluded. Sixth, most of the included indicators are not modifiable. For example, a patient will still be complex if receiving less imaging procedures to reduce costs.
To our knowledge, the PCA score is the first tool to identify complex medical patients in the hospital setting. It can easily be calculated and is therefore predestined to be used for population-based studies as it does not involve individual judgement of a physician. With its prospective design and inclusion of a large number of medical inpatients, this study has a strong design.
Identification of complex patients by this simple tool using electronically available data may help monitoring the proportion of complex patients in the hospital setting and comparing patient complexity level between hospitals. Thereby, the PCA score might improve the monitoring of resources distribution and coordination of care, for example, by flagging complex patients to general practitioners or social workers for closer follow-up or low-threshold service.
Data availability statement
Data are available on reasonable request from fabian.liechti@insel.ch (ORCID ID 0000-0003-1006-6903).
Ethics statements
Ethics approval
The study was approved by the local ethics committee (Kantonale Ethikkommission Bern, ID 2016–01319).
Acknowledgments
We acknowledge the support by Roland Angerer (coding data) and Barbara Ammann (workload nurse).
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors All authors had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. FDL: Analysis and interpretation of data, drafting of the manuscript, critical revision of the manuscript for important intellectual content. TB: Acquisition of data, analysis and interpretation of data, critical revision of the manuscript for important intellectual content. AR: Acquisition of data, analysis and interpretation of data, critical revision of the manuscript for important intellectual content. MCR: Statistical analysis, analysis and interpretation of data, critical revision of the manuscript for important intellectual content. AL: Analysis and interpretation of data, critical revision of the manuscript for important intellectual content. TT: Analysis and interpretation of data, critical revision of the manuscript for important intellectual content. JDD: Study concept and design, acquisition of data, analysis and interpretation of data, administrative, technical and material support, drafting of the manuscript, critical revision of the manuscript for important intellectual content.
Funding This work was supported by SGIM foundation. JDD is supported by Swiss National Science Foundation (grant number 170656). TT is supported by an Early Postdoc. Mobility Award from the Swiss National Science Foundation (SNSF P2ZHP3_177999) and a Fellowship Award from the CanVECTOR Network.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.