Prediction of 30-Day Hospital Readmissions for All-Cause Dental Conditions using Machine Learning

Introduction

Hospital readmission is a common occurrence that can impose a multitude of burdens such as increased costs and an increased risk of mortality.¹ Studies indicate that the percentage of patients readmitted to hospitals in 30 days following discharge may be as high as 20%.² Due to its cost and reoccurrence, in 2012 the Center for Medicare & Medicaid Services (CMS) instituted the Hospital Readmissions Reduction Program to reduce the frequency of rehospitalization by establishing financial penalties for hospitals with readmission rates deemed excessive.³ The goal to reduce readmission spurred interest in studying and predicting hospital readmission.

Efforts to predict hospital readmission attempt to identify those at greatest risk and to uncover factors or trends leading to readmission. Due to the increasing availability of large databases, artificial intelligence (AI) and machine learning (ML) offer a potentially effective mean to predict risk factors influencing a variety of healthcare issues.⁴ Unlike traditional modeling, ML utilizes higher-order and nonlinear interactions between predictors which could demonstrate better model performance,⁵ and by identifying patterns in learning processes could help predict future events. Standard methods to predict hospital readmission typically rely on a limited set of clinical data using simple calculations such as a modified LACE score that exams at the length of hospital stay, acuity on admission, comorbidity and emergency department visits.⁶ When comparing standard methods to ML, some researchers found that ML scores were better at predicting hospital readmissions.⁶ However, others showed mixed results,⁷ highlighting the need for further research to clarify discrepancies in predictive models.

Much of the published literature focused primarily on hospital readmission for surgical and other medical conditions, overlooking the importance of dental health. Globally, oral health and oral diseases such as dental caries, periodontal disease, lesions and oropharyngeal cancers are major public health concerns with poor oral health negatively affecting one’s general health and quality of life.^8,9 In addition to its importance to overall well-being, in 2016 the United States spent more than $100 billion on dental expenditures, representing over a 3% increase from the previous year.^10,11 Despite potential clinical and financial impact, factors associated with hospital readmissions related to oral or dental issues remain unexplored.

This study aimed to explore the risk factors related to dental hospital readmission for patients with all-cause dental admissions (ACDA), and to build ML algorithms to predict whether an individual patient will be at risk of readmission within 30 days following hospital discharge for all-cause dental conditions (ACDC) . This predictive model created from artificial intelligence should be useful in decreasing risk and occurrences of hospital readmission following dental procedures.

Methods

Data

Data for this study came from the 2013 Nationwide Readmissions Database (NRD). The NRD is a part of the Healthcare Cost and Utilization Project and aims to provide nationally representative data on hospital readmissions for all types of payers and the uninsured. It contains data from approximately 85% of the State Inpatient Databases from 21 participating states in the United States. The 2013 NRD contains data for 14,325,172 hospital discharges. It has four distinct files (core file, severity file, diagnostic and procedure code file, hospital characteristics file) containing various data to enable more in-depth analyses.

In order to identify ACDA for inclusion into this study, we included patients with ICD-9-CM dental diagnostic codes (beginning with 52) and procedure codes (beginning with 23, 24 and 27, and some selected codes beginning with 96, 97 and 99). Appendix 1 shows the top dental diagnosis and procedure codes among all of them. This study excluded patients who died during a hospital stay, hospital stays less than one day, and those discharged after November 2013. Index events were identified by using inclusion and exclusion criteria stated above.

After applying the inclusion and exclusion criteria, 11,851 cases were identified for 30-day readmissions for ACDA patients in 2013. The initial dataset contained 236 variables, but some variables were eliminated resulting in a final total of 55 potential predictor variables and one outcome variable used for the analyses (Appendix 2). The eliminated variables included ID variables considered unlikely to be related to hospital readmission, variables showing extremely high correlation with existing ones, or variables used to weight for hospitals and discharges, as well as those that were used to filter specific patients. To prepare for further variable selection and preliminary analyses, the application of listwise deletion eliminated a small portion of cases with missing data (4.3%). A final total of 11,341 cases remained in the 30-day hospital readmission dataset for this study.

Outcome Variable

The primary outcome variable was dichotomous (yes/no on whether patients had readmissions within 30 days following discharge of an index hospitalization with dental-related procedures). Altogether, 1254 cases (11.1%) had 30-day hospital readmissions for ACDA patients over the one-year study period.

Potential Predictor Variables

Based on literature review and consultation with an experienced dentist, the study defined three categories of information from the 55 potential predictor variables: (1) a demographic category such as age and gender, (2) a socioeconomic category such as median household income, primary payer, patient location, total charges, etc., and (3) a clinical category such as length of hospital stay, hospital urban-rural designation, emergency department service indicator, number of chronic conditions, number of diagnosis, number of procedures, AHRQ comorbidity measures, etc. Appendix 3 contains a detailed summary of the potential predictors across the Readmission groups.

Statistical Analyses

Traditional statistical analyses were used to further process the data in order to select a parsimonious set of variables for subsequent ML analyses. Continuous variables and categorical variables were analyzed to examine whether there were statistically significant differences at α=0.05 between the readmission group and non-readmission group. For categorical variables, Fisher’s exact test was used when the total sample size was less than 30 or the expected cell count was less than 5, or alternatively a simulated Chi-squared test applied when the dataset was too large for Fisher’s exact test; otherwise the variables were tested using Chi-squared test. For continuous variables, a t-test was used if the variable was normally distributed; otherwise a Wilcoxon rank sum test was used. In sum, this selection process identified 49 variables (out of the 55) with statistically significant relationships to the outcome variable “Readmission” for use in subsequent feature selection by ML to build the 30-day readmission risk prediction models. All statistical analyses were performed using R 4.0.0 software.¹²

Machine Learning Methods

In preparing for the creation of preliminary risk prediction models using ML methods, the dataset was randomly split into two mutually exclusive sets, a training set with 70% of the data, and a testing set with 30% of the data. The training set was used for the machine (ie, computer) to learn the information contained in the data and then generate the risk prediction algorithms. The testing set was used to validate the performance of the prediction algorithms generated from the training set against the new data (eg, data unseen by the computer during the learning stage) contained in the testing set.

The data revealed an imbalanced binary outcome in which only 11.1% of patients had readmission and 88.9% did not. An imbalanced training set provides less information on the minority class (eg, 11.1%) and can bias prediction, thus potentially contributing to inaccuracy of risk prediction. In order to adjust for the imbalanced data and to increase accuracy, data resampling techniques were applied to balance the data. Specifically, oversampling of cases was applied to the minority class so that the dataset for ML has a balance of readmitted and did not readmit cases.

Data standardization and normalization are required for some ML methods such as k-Nearest Neighbor (k-NN), but not for the tree-based decision algorithms like decision trees (DT) or random forests (RF).¹³ This study applied Z-score standardization for continuous predictors and dummy coding for the categorical variables. The standardized training dataset was utilized in the Support Vector Machine (SVM), k-Nearest Neighbor (k-NN) and Artificial Neural Network (ANN) ML methods as well as Logistic Regression (LR) in this study. We intentionally used LR (a traditional statistical technique) in this study so that we could compare its results with the ML methods.

RF, a tree-based ML algorithm, also known as an ensemble learning method, can be applied to both classification and regression tasks. A bagging (or bootstrap aggregating) technique was used in RF to build independent identically distributed trees, to grow deep trees, and to reduce the variance of an estimated prediction function. During the construction of RF, the out-of-bag error rate was calculated by averaging the prediction error of each unselected training sample through the bootstrap sampling for training. Aggregation of predictions can help reduce both bias and variance.

As a supervised learning technique (eg, the presence of predefined outcome variable to guide the learning process), one popular application of RF is important feature selection. Feature selection helps to reduce the dimensions without much loss of the information, leads to a decrease in training time and model complexity and prevents the data overfitting. This study used the randomForest package in R to select the important features for building risk prediction models using the ML models. As an ensemble of individual DT, RF also uses the Gini Impurity to evaluate the importance of a variable on predicting the outcome across all of the individual trees in the forest. Relative importance of the features (eg, displaying the risk factors) was generated as a result.

The study applied five methods (DT, SVM, k-NN, ANN and LR) to build risk prediction algorithms. The 10-fold validation was then employed to estimate the model on test data and model performances on the testing set evaluated by using the confusion matrix (accuracy, sensitivity, specificity, precision, etc.) and the area under the receiver operating characteristic curve (AUC). According to United States federal regulations (45 CFR 46, category 4), this is a secondary analysis of existing data and the NRD data were de-identified and publicly available (see https://www.hcup-us.ahrq.gov/db/nation/nrd/NRD_Introduction_2013.jsp); thus, this research was exempt from Institutional Review Board approval. This study conforms to STROBE guidelines.

Results

Tables 1 and 2 summarize demographic data and compare key factors between the readmission and no readmission groups, respectively. Among the 11,341 patients who were discharged, 11.1% (n=1254) had hospital readmission within 30 days and 89.9% (n=10,087) did not. The sample consisted of 55.9% male (n=6338) and 44.1% female (n=5003). The mean age at admission was 42.4 years. The mean length of hospital stay was 7.9 days and the average total medical charges were $76,522. The average number of diagnoses for each patient was 9.7, and the average number of procedures was 3.8. The average number of chronic conditions was 3.7. There were 22.8% of patients with Medicare insurance (n=2590), 28.7% with Medicaid (n=3250) and 24.5% with private insurance (n=2781). Other groups included self-pay 15.8% (n=1792) and 8.1% other payers or with no hospital charges (n=928). Among the sample, 38.8% experienced minor loss of function (n=4399) and 33.9% (n=3845) moderate loss of function. Patients readmitted to the hospitals tended to be older, incur higher medical bills, have longer hospital lengths of stay, have more diagnoses, procedures and experience more chronic and severe illnesses. They also had a greater proportion of Medicare and/or Medicaid insurance (Table 2). The average hospital charge for a patient with 30-day readmission was $131,004, compared to $69,749 for those without readmission. If one-third of the readmitted cases (ie, 418 cases) could be prevented, it represents a cost saving of more than $25 million (ie, ($131,004 - $69,749) x 418 = $25,604,590) for the twenty-one states in the NRD.

Table 1 Demographic Characteristics of Patients

Table 2 Descriptive Summary of the Top Eight Predictors of 30-Day Readmission by Readmission and No Readmission Groups

Based on the largest contribution to decreasing Gini index, the top eight most important features selected by the random forest algorithm included total charges, number of diagnosis on this record, age in years at admission, number of chronic conditions, length of hospital stay, number of procedures on this record, primary expected payer, and severity of illness subclass (Figure 1). These eight selected features were then used to construct the risk prediction models using DT, SVM, k-NN, ANN and LR.

Figure 1 Important feature selection using random forest (The top eight features selected from random forest were used to build hospital readmissions risk prediction models. A higher “Mean Decrease in Gini” in x-axis indicates a higher purity (less noise, less bias) contributed by variable, and higher variable importance).

Figure 2 displays the 30-day hospital readmission risk prediction algorithms generated by DT. By using each individual patient’s information about the top eight features, healthcare providers can follow the DT diagram in Figure 2 to get an estimated readmission classification, and thus arrive at a conclusion of whether a patient will likely be readmitted 30 days within hospital discharge. Unlike DT, many ML methods produce highly complicated algorithms which act like a black-box and do not provide clarity as to how to arrive at the likelihood that a patient will be readmitted or not. Figure 2 also illustrates an example of such a black-box prediction algorithm developed from ANN.

Figure 2 Risk prediction models for 30-day hospital readmission generated by decision tree (top diagram) and artificial neural network (bottom diagram).

Figure 3 depicts the AUC of the risk prediction models for the 30-day readmission from the five methods. High AUC value represents high prediction accuracy of the risk prediction models. The AUCs were similar across five methods using the eight predictors, with ANN having the highest AUC of 0.739, and LR having the second-highest AUC of 0.735. The precision was low across all methods at around 0.200. Figure 3 also graphically summarizes the performance indices of each method, including sensitivity, specificity, accuracy, precision and AUC.

Figure 3 Performance metrics of the risk prediction models for 30-day hospital readmission when conducting model validation using various methods.

Discussion

In 2009, Jencks et al² highlighted the issue of hospital readmissions in a milestone article. Using Medicare claims data, they found that about 1 in 5 patients discharged from the hospital were readmitted within 30 days. Both from a cost and quality perspective, the CMS made readmission as focus, initially concentrating on heart disease and pneumonia, but more recently expanding to other conditions such as joint replacement surgery. Ideally, tools to forecast those at greatest risk for readmission could identify individuals most likely to benefit from interventions and to develop metrics to evaluate providers and organizations.

Little research exists about predictive models and ML tools for readmission for those hospitalized with ACDC. Likewise, research related to preventable dental related hospitalizations is also limited.¹⁴ To our knowledge, this study represents the first in the United States to explore ACDC hospital readmissions and to apply AI to develop risk prediction tools. Our results indicate that about 11% of ACDC patients were readmitted within 30 days, a finding that helps establish a benchmark to assess interventions and for institutions and hospitals to evaluate their performance. In comparison, all-cause 30-day readmission rates vary between 7.1% and 14.4%.¹⁵ Similar to other types of hospital readmission, those patients insured by Medicare and Medicaid experienced higher dental hospital readmission rates. The 11% 30-day dental hospital readmission rate found in this study certainly exceeds the 30-day rate for total hip arthroplasty (5.6%) and total knee arthroplasty (3.3%). Although the exact percent of preventable readmissions remains unclear, based on a median estimate of 30% being preventable,¹⁶ our study estimates that the annual cost-saving opportunity for 30-day dental hospital readmissions may exceed $25 million dollars for the twenty-one states in the NRD. With an average annual dental expenditure of $290.86 per person,¹¹ this $25 million represents dental care for over 88,000 individuals. Comparable to readmission research for heart disease and pneumonia, age, number of conditions, disease acuity and level of function correlated with an increased readmission risk for 30-day dental hospital readmissions. While intuitively it makes sense that “sicker” patients are at greater risk, this is the first study to document which co-morbidities correlate with ACDC hospital readmissions. Understanding risk factors can help guide strategies and research aimed at reducing readmission rates.

Despite numerous attempts to develop accurate prediction models for readmission, most models perform poorly.¹⁷ In contrast, all of the five models in this study provided quite decent AUC, accuracy, sensitively and specificity. Precision was poor though, meaning that there could be large variation in future model performance. The poor precision is likely a result of the small number of predictors (ie, eight). Future effort can be focused on adding more predictors to increase the precision. After all, readmission likely represents a broad array of factors that interact in complex ways that challenge scientific discovery. Nonetheless, most predictive models rely on traditional analytic methods but this study is novel in its application of ML methods from AI using only eight features to predict all-cause 30-day readmission for dental patients, and it can inform individualized patient treatment decision. The prediction models built by ANN provided better model performance. The DT, LR and other methods also had good AUCs. However, ML works best with large data sets and the small number of readmission cases and predictors might not provide robust enough data for ML techniques to outperform LR. We intentionally limited the final number of predictors to be very small in this study in order to produce a parsimonious model for practical use in clinical settings for all hospitals. Further studies using either a larger number of readmission cases or larger number of predictors in the final model will help confirm ML performance against traditional analyses such as LR and also help assess each model’s applicability to other diagnoses. A surprising finding was that the length of stay for index admissions was unexpectantly long at almost eight days. However, a study examining hospital admissions in Florida observed admission rates and diagnostic codes for non-traumatic dental conditions (NTDC) (Morón et al, 2019) consistent with our findings suggesting that our study correctly identified dental admissions. Another study identified hospital charges of over $53,000 per NTDC admission (Tomar et al, 2019) suggesting a relatively high acuity level and complexity for ACDC admissions. Still the exact reasons for an average 8-day length of hospital stay remains uncertain and suggests an area of future research and perhaps new opportunities for identifying cost savings.

Limitations

Finding the best method to build a model depends on the setting, population and condition. While this study employed several ML methods, further fine-tuning using grid search to optimize hyperparameter combination (eg, using different or more predictors, adjusting the number of branches in a tree), or utilizing additional ML methods might improve in future models. Moreover, trying different feature selection sets and multiple feature combinations might improve prediction performance, but such a strategy would require extensive repeat testing and refinement. Future research can continue this effort to search and refine the models and algorithms. Another limitation was that because some numeric feature distributions were highly right skewed, log transformed data was used to produce better results. However, the trade-off in using log transformed data is the difficulty in how to directly interpret the results. Another limitation is that while the study examined risk factors, it did not explore which dental conditions associated with readmission. While this study provides insights that can guide strategies to prevent readmissions, we have yet to carry out the next step in the future to fully realize the impact on clinical decisions. The next step will deploy the ML algorithms generated from this study by programming an app for mobile phone and/or computer desktop, and use the app for real-time clinical decision support in hospitals.

The NRD database represents a single year of data and so it is unknown if the data are consistent with other time periods, and any time trends cannot be identified. Another limitation is that the performance of the ML analyses was limited by the relatively small number of readmission cases. Also, the NRD is constructed from State Inpatient Databases and contains hospital readmission records from only twenty-one states, so readmissions occurring in other states are not captured. Another drawback is that race and ethnicity are not identified and thus the impact of these factors cannot be assessed.

Conclusion

The 30-day readmission rate for ACDC hospitalizations in our study is 11.1%. Readmission was associated with older age, higher number of diagnoses and chronic conditions, longer length of stay for the index hospitalization, higher severity of illness and having Medicare or Medicaid insurance. While intuitively it makes sense that “sicker” patients are at greater risk, this is the first study to provide empirical evidence that certain co-morbidities correlate with ACDC hospital readmissions. This is also the first study to establish a benchmark for institutions and hospitals to assess their performance using only a small number of eight predictors that are universal and applicable to many hospitals. Model performance from all methods was similar with ANN showing slightly better performance.