Figures
Abstract
Background
Mortality prediction models generally require clinical data or are derived from information coded at discharge, limiting adjustment for presenting severity of illness in observational studies using administrative data.
Objectives
To develop and validate a mortality prediction model using administrative data available in the first 2 hospital days.
Research Design
After dividing the dataset into derivation and validation sets, we created a hierarchical generalized linear mortality model that included patient demographics, comorbidities, medications, therapies, and diagnostic tests administered in the first 2 hospital days. We then applied the model to the validation set.
Subjects
Patients aged ≥18 years admitted with pneumonia between July 2007 and June 2010 to 347 hospitals in Premier, Inc.’s Perspective database.
Results
The derivation cohort included 200,870 patients and the validation cohort had 50,037. Mortality was 7.2%. In the multivariable model, 3 demographic factors, 25 comorbidities, 41 medications, 7 diagnostic tests, and 9 treatments were associated with mortality. Factors that were most strongly associated with mortality included receipt of vasopressors, non-invasive ventilation, and bicarbonate. The model had a c-statistic of 0.85 in both cohorts. In the validation cohort, deciles of predicted risk ranged from 0.3% to 34.3% with observed risk over the same deciles from 0.1% to 33.7%.
Citation: Rothberg MB, Pekow PS, Priya A, Zilberberg MD, Belforti R, Skiest D, et al. (2014) Using Highly Detailed Administrative Data to Predict Pneumonia Mortality. PLoS ONE 9(1): e87382. https://doi.org/10.1371/journal.pone.0087382
Editor: Olivier Baud, Hôpital Robert Debré, France
Received: July 17, 2013; Accepted: December 24, 2013; Published: January 31, 2014
Copyright: © 2014 Rothberg et al. This is an open-access article distributed under the terms of the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.
Funding: The study was funded by the Agency for Healthcare Research and Quality (1 R01 HS018723-01A1). The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.
Competing interests: The authors have the following interests: Dr. Marya D. Zilberberg is employed by EviMed Research Group, LLC. Dr. Zilberberg has received research funding and served as a consultant for Pfizer, Astellas, Cubist, Forest, and Johnson and Johnson. There are no patents, products in development or marketed products to declare. This does not alter the authors’ adherence to all the PLOS ONE policies on sharing data and materials, as detailed online in the guide for authors.
Introduction
Bacterial pneumonia is a leading cause of morbidity and mortality in the United States. Every year, more than 8 million patients are admitted to US hospitals with pneumonia; 8.8% of them will die. [1] Despite the common nature of this condition, there are large gaps in our knowledge regarding how best to care for pneumonia patients. Most recommendations in national treatment guidelines are not based on randomized trials, and there is a paucity of comparative effectiveness research.
Administrative databases derived from billing records are attractive candidates for health services research, as well as for use in hospital profiling initiatives, because the number of patient records is large and the acquisition cost is low. Observational studies using administrative data can be used to assess comparative effectiveness in real world settings, and findings from such studies are sometimes confirmed in randomized trials. One concern, however, is that such studies are often biased by confounding by indication, in which the choice of treatment is influenced by a patient’s severity of illness. This threat can be limited through the use of validated risk prediction instruments that are capable of adjusting for pre-treatment severity of illness, as well as comorbidities.
There exist a number of validated pneumonia mortality prediction instruments for use in clinical care. [2], [3] All of these require clinical data, such as respiratory rate or blood urea nitrogen, which are not generally available in administrative data sets. Others have attempted to construct predictive mortality models from administrative data. International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes assigned at discharge are highly predictive of mortality, in great part because they include complications of hospitalization which often precede death. [4] Such models are not useful for severity adjustment because they incorporate the results of treatment (e.g., complications) as predictors. Models restricted to demographics and comorbidities at the time of admission have much lower predictive accuracy [5].
Highly detailed administrative datasets include a date-stamped record for each item administered during a hospitalization; this allows for differentiation between factors present at the time of hospitalization and those arising during the stay. We used one such dataset to create and validate a mortality risk prediction model that included only tests and treatments administered in the first 2 hospital days along with patient demographics and comorbidities.
Methods
Setting and Patients
We identified patients discharged between July 1, 2007 and June 30, 2010 from 347 US hospitals that participated in Premier, Inc.’s Perspective, a database developed for measuring quality and healthcare utilization that has been described previously. [6]–[8] Member hospitals represent all regions of the US, and are generally reflective of US hospitals; although larger hospitals, hospitals in the South and those in urban areas are over represented. Perspective contains all data elements found in the uniform billing 04 form, such as sociodemographic information, ICD-9-CM diagnosis and procedure codes, as well as hospital and physician information. It also includes a date-stamped log of all billed items and services, including diagnostic tests, medications, and other treatments. Because the data do not contain identifiable information, the Institutional Review Board at Baystate Medical Center determined that this study did not constitute human subjects research.
We included all patients aged ≥18 years with a principal diagnosis of pneumonia, or a secondary diagnosis of pneumonia paired with a principal diagnosis of respiratory failure, ARDS, respiratory arrest, sepsis or influenza (Figure S1). Diagnoses were assessed using International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes. We excluded all patients transferred in from or out to other acute care facilities, because we either could not assess initial severity or could not assess outcomes; those with a length of stay of 1 day or less; patients with cystic fibrosis; those whose attending physician of record was in a specialty that would not be expected to treat pneumonia (e.g., psychiatry); those with a diagnosis related grouping (DRG) inconsistent with pneumonia (e.g., Prostate OR procedure); those with a code indicating that the pneumonia was not present on admission; and any patient who did not have a chest radiograph and did not begin antibiotics on hospital day 1 or 2. For patients with multiple eligible admissions in the study period, 1 admission was randomly selected for inclusion.
Markers of Comorbid Illness and Pneumonia Severity
For each patient, we extracted age, gender, race/ethnicity, insurance status, principal diagnosis, comorbidities, and specialty of the attending physician. Comorbidities were identified from ICD-9-CM secondary diagnosis codes and DRGs using Healthcare Cost and Utilization Project Comorbidity Software, version 3.1, based on the work of Elixhauser. [9] We identified a group of medications, tests, and services that are typically associated with chronic medical conditions (e.g., spironolactone, warfarin, need for a special bed to reduce pressure ulcers), as well as acute medications that may indicate severe illness (e.g., vasopressors, intravenous steroids). We also identified early use of diagnostic tests (e.g., arterial blood gas, serum lactate) and therapies (e.g., mechanical ventilation, blood transfusion, restraints) that are associated with more severe presentations of pneumonia. The complete list of medications, tests, and treatments appears in Table S1. To avoid conflating initial severity with complications of treatment, we limited our analysis to those markers received in the first 2 hospital days. We used the first 2 days because hospital days are demarcated at midnight and the first day often represents only a few hours.
Statistical Analysis
Individual predictors of mortality were assessed using Chi-square tests using the full study cohort. Stratifying by hospital, 80% of the eligible admissions were randomly assigned to a derivation and 20% to a validation cohort, and the two cohorts were compared for differences in potential predictors. Using the derivation cohort, we developed a series of multivariable logistic regression models to predict in-hospital death. Hierarchical generalized linear models (HGLM) with a logit link (SAS PROC GLIMMIX) were used to account for the clustering of patients within hospitals. We grouped predictors into the following categories: demographics, comorbid conditions, and severity markers. We developed separate mortality models for each of these categories, including main effects and significant pairwise interactions. Factors significant at p<0.05 were retained. For each model we calculated the area under the receiver operating characteristic (AUROC) curve, together with 95% confidence intervals. [10] The final model was developed by sequentially adding effects retained in individual category models and evaluating pairwise interaction terms. Main effects that were dropped at earlier stages were re-evaluated for inclusion in the final model.
The purpose of the model was accurate prediction of mortality and risk stratification. We did not attempt to determine which individual factors were associated with mortality or to imply causality. Therefore, we did not require a priori information about the association of the various risk factors or interaction terms with the outcome. Although such an approach may result in spurious associations of individual risk factors, it need not necessarily detract from the model’s accuracy of prediction, which was our primary concern [11].
In order to guard against the possibility of overfitting our model, parameter estimates derived from the model were used to compute individual mortality risk in the remaining 20% of the admissions (the validation cohort). Discrimination of the final model in the validation set was assessed by the c-statistic as well as the expected/observed ratio. Both cohorts were categorized by decile of risk based on the probability distribution in the derivation cohort, and observed mortality was compared to that predicted by the model. We also used the integrated discrimination improvement (IDI) index [12] to measure the improvement of the final model over a basic model including only demographics and ICD-9-CM comorbidities.
We next evaluated model performance in subpopulations of the entire cohort based on hospital and patient characteristics. Specifically, we assessed model performance in strata defined by hospital size, teaching status, patient age, ICU and non-ICU admissions, and pneumonia type [healthcare-associated (HCAP) vs. community-acquired (CAP)]. All analyses were performed using the Statistical Analysis System (version 9.2, SAS Institute, Inc., Cary, NC) and STATA (StataCorp. 2007. Stata Statistical Software: Release 10. College Station, TX: StataCorp LP).
Results
The dataset included 200,870 patients in the derivation cohort and 50,037 patients in the validation cohort. Patient characteristics of the full study cohort appear in Table 1. Most patients were over age 65, 53.3% were female and 68.0% were white. The most common comorbidities were hypertension (46.5%), diabetes (23.8%), chronic pulmonary disease (48.6%), and anemia (22.2%). Patients in the validation cohort were similar (Table S2).
Model Development
Overall in-hospital mortality in the derivation cohort was 7.2%. A large number of patient and hospital factors were associated with mortality (Table 1). Due to the large sample size, even weak associations appear highly statistically significant. Figure 1 shows the model discrimination, as measured by the area under the ROC curve, when subgroups of factors were used to model mortality. Including only patient demographics produced a model with poor discrimination (AUROC = 0.66). Using traditional ICD-9-CM based measures of comorbidity showed greater discrimination (AUROC = 0.71), as did a model that used admission to the ICU in day 1 or 2 as the only predictor (AUROC = 0.73). As an alternative measure of comorbidity, chronic medications were superior to ICD-9-CM codes in predicting mortality (AUROC 0.74 vs. 0.71, p<.001). Combining demographics, comorbidities, and markers of severity of illness on presentation (other infections present-on-admission, admission to ICU, the ability to take oral medications, and acute medications, tests and therapies used in first 2 days) offered excellent discrimination in the derivation cohort (AUROC = 0.85). We also assessed model discrimination using the IDI. Compared to the model including only demographics and ICD-9-CM comorbidities, the full model had an IDI which was 12 percentage points higher (16.6% vs. 4.6%, p<.001).
Factors not significant at p<.05 and interaction terms are not included. All medications, tests and therapies are within the first 2 hospital days. Legend includes area under the ROC curve and 95% confidence intervals.
The final multivariable model included 3 demographic factors, 25 comorbidities, 41 medications, 7 diagnostic tests, and 9 treatments, as well as a large number of interaction terms (Table S3). The strongest predictors were early vasopressors (OR 1.71, 95% CI 1.62–1.81), early non-invasive ventilation (OR 1.55, 95% CI 1.47–1.64), and early bicarbonate treatment (OR 1.70, 95% CI 1.59–1.82). The final model produced deciles of mean predicted risk from 0.3% to 34.5%, while mean observed risk over the same deciles ranged from 0.1% to 34.1% (Figure 2).
Validation
Model discrimination measured by the c-statistic in the validation set was 0.85 (95%CI: 0.844–0.856). Deciles of predicted risk ranged from 0.3% to 34.3% with observed risk over the same deciles from 0.1% to 33.7% (Figure 2). The expected mortality rate according to the model was 7.1% (expected/observed ratio: 1.00 [95% CI 0.97–1.03]).
Performance of the model in subpopulations of the entire cohort is shown in Table 2. The model performed well in all subpopulations tested, but discrimination was poorest among patients in intensive care (c-statistic 0.78) and best among patients aged 18 to 64 years (c-statistic 0.89). In all subgroups the range of predicted mortality extended from ≤0.3% to >90%. Model calibration was also good in all subgroups. The model tended to underestimate the risk of mortality among patients with healthcare-associated pneumonia, and to a lesser extent among patients in teaching hospitals and those outside of the ICU. At the same time, it overestimated the risk of mortality among patients with community-acquired pneumonia.
Discussion
In this retrospective cohort study, we used highly detailed administrative data to derive and validate a pneumonia mortality prediction model for use in observational studies. The model had discriminatory ability comparable to those derived from clinical data, but unlike most other administrative models, it included information on illness severity that would be available in the first 2 hospital days. The model also had excellent calibration and successfully divided patients into mortality deciles ranging from <0.5% to >33%. Interestingly, the 30% of patients with the lowest predicted mortality had an observed mortality of <1%.
At least two clinical prediction tools have been developed for the purposes of risk stratifying patients with community acquired pneumonia–the CURB-65, [3] modified from earlier work by the British Thoracic Society, and the Pneumonia Severity Index (PSI). [2] The CURB-65 consists entirely of exam findings and laboratory values, while the PSI incorporates some historical information as well. At least 3 studies have prospectively compared the predictive abilities of these two measures. [13]–[15] Perhaps due to differences in study population, c-statistics for predicting 30-day mortality ranged from 0.73 to 0.89 across studies; however, within any given study, there were no statistically significant differences between the two scales.
Because the clinical information required for these tools is not available in administrative databases, others have attempted to create models based solely on administrative claims. In general, such models have modest discriminatory ability, unless they are combined with laboratory data. For example, one administrative claims model developed for profiling hospitals’ pneumonia mortality rates, and containing age, sex, and 29 comorbidities (based on ICD-9-CM codes from the index hospitalization and the prior year’s outpatient visits) had a c-statistic of 0.72. [16] Addition of laboratory values to administrative data can substantially enhance discrimination. Tabak et al. demonstrated that laboratory values alone contributed 3.6 times as much explanatory power as ICD-9-CM codes and 2.5 times as much as vital signs to mortality prediction. [17] For example, the c-statistic for a model that only includes laboratory values and age was 0.80. Adding ICD-9-CM codes and vital signs increased the c-statistic to 0.82. [17] Pine et al. also found that ICD-9-CM codes alone produced a c-statistic of 0.78, whereas addition of laboratory values increased the c-statistic to 0.87. [4] Addition of chart-based data (e.g., vital signs) had a small marginal effect on the model’s predictive ability [4], [18].
Our study takes a different approach to overcoming the limitations of administrative data. In brief, our results suggest that it is possible to tell a lot about patients by the tests, medications and treatments they are prescribed. Although others have utilized ambulatory medications to predict outpatient costs and mortality, these generally do not perform better than comorbidity models based on ICD-9-CM codes. [19], [20] In contrast, by assessing medications, tests and treatments administered in the first 2 hospital days, we were able to identify chronic comorbid conditions, as well as factors indicative of the severity of illness on presentation. Indeed, use of chronic medications alone predicted mortality better than ICD-9-CM codes. This could be because billing codes are more sensitive than ICD-9-CM codes, but also because medication use can identify not just the presence of disease, but also provide information about disease severity. For example, among patients with heart failure, spironolactone often signifies severe systolic dysfunction, and nadolol in the presence of liver disease likely indicates portal hypertension. Medications, however, did not capture all the information present in ICD-9-CM codes, and the combination of the two was a more powerful predictor than either one alone. This is likely because some chronic conditions, such as metastatic cancer, may not be associated with any routine medications, but are nonetheless potent predictors of mortality.
The inclusion of certain initial tests and therapies also allowed us to estimate the severity of illness at the time of admission in the absence of laboratory or clinical data. Although it would be helpful to know the results of a blood gas, the simple presence of that test is indirect evidence that the treating physicians were concerned about a patient’s respiratory condition. Similarly, a patient receiving vasopressors is almost certainly hypotensive. More importantly, our model’s predictive ability was comparable to that seen with other administrative models that include laboratory data, as well as those that are based on physiological information obtained from review of medical records. An analogous model, designed for use in sepsis patients, demonstrated that highly detailed administrative data can achieve discrimination and calibration similar to clinical mortality prediction models, [21] with the majority of the additional explanatory power of the model arising from the inclusion of initial treatments [22].
Our study has several limitations. First, our main outcome was in-hospital mortality. Others have modeled 30-day mortality and the factors that are predictive of in-hospital mortality may be different than those which predict 30-day mortality. [23] Second, our study was conducted retrospectively and the model, therefore, may perform differently in a prospective cohort. It would certainly be premature to base treatment decisions on our model, but that is not its intended purpose. Third, our definition of pneumonia was based on diagnosis and charge codes. Some patients may not have had pneumonia and some cases of pneumonia may have been missed. These numbers are likely to be small, as the positive predictive value of an ICD-9 diagnosis paired with an antibiotic description is >95%. [24] Fourth, we excluded patients with pneumonia not present on admission, as well as transfer patients, so our model is not applicable to these groups. Finally, our model derives much of its power from physician assessments of patients’ disease, as represented by physician ordering. To the extent that prescribing thresholds vary by institution, the model may be more or less accurate in certain hospitals, and therefore could not be used for benchmarking purposes. The fact that model discrimination was good across various subgroups of hospitals is reassuring in this regard.
This model could be used in various ways. It could be used for adjustment in observational trials, including comparative effectiveness or epidemiologic studies. Although such studies might also be performed using clinical data, many institutions do not currently have the ability to automatically extract clinical data from electronic medical records and many administrative datasets do not yet contain laboratory data. Our model represents a low-cost yet accurate alternative. In addition, unlike existing clinical models, our model was validated in several different sub-populations, with excellent performance in small and large hospitals, and in teaching and non-teaching institutions.
The model could, for example, be used for severity-adjustment in a study to compare effectiveness of guideline recommended therapies to alternative treatment options in community acquired pneumonia. It could also be used to study the severity of an illness such as healthcare associated pneumonia, in which multiple comorbid illnesses might contribute to poor outcomes. It could have application for studying the methods of hospital profiling for public reporting (e.g., testing alternative definitions of diagnosis), but may not be useful for profiling hospitals per se, because thresholds for treatment might vary across hospitals making patients appear more or less sick. Finally, some aspects of the model–specifically the chronic medications–could be incorporated into clinical prediction rules such as the PSI, in order to improve their accuracy. To avoid showering clinicians with unnecessary complexity, these could be embedded in clinical information systems to provide prognostic information at the point of care. However, prospective validation of such a hybrid model is required before it can be applied in clinical care.
In conclusion, we have created a mortality prediction model based on highly detailed administrative data available in the first 2 days of hospitalization. The performance of the model was comparable to that of models based on clinical data, and the performance was consistent across different patient subpopulations. The model should be useful for comparative effectiveness research using large, administrative databases.
Supporting Information
Figure S1.
Flow Diagram of Patient Selection. PN – Pneumonia; ARDS – Acute Respiratory Distress Syndrome; CXR – Chest X-Ray; CH CT – Chest CT; ABX – Antibiotic; LOS – Length of Stay; MS DRG – Medicare Diagnosis Related Group; POA – Present on Admission.
https://doi.org/10.1371/journal.pone.0087382.s001
(DOCX)
Table S1.
Complete List of Medications, Tests, and Treatments.
https://doi.org/10.1371/journal.pone.0087382.s002
(DOCX)
Table S2.
Patient Characteristics in the Derivation and Validation Cohorts.
https://doi.org/10.1371/journal.pone.0087382.s003
(DOCX)
Table S3.
HGLM Estimates from Multivariable Mortality Model.
https://doi.org/10.1371/journal.pone.0087382.s004
(DOCX)
Acknowledgments
Previous Presentation:
Society for Medical Decision Making Annual Meeting, Phoenix, AZ, October, 2012.
Author Contributions
Conceived and designed the experiments: MBR PSP AP MZ RB DS TL TH PKL. Performed the experiments: MBR PSP AP. Analyzed the data: MBR PSP AP MZ RB DS TL TH PKL. Wrote the paper: MBR AP PSP. Critical revision of the manuscript for important intellectual content: PSP AP MZ RB DS TL TH PKL.
References
- 1. Lindenauer PK, Lagu T, Shieh MS, Pekow PS, Rothberg MB (2012) Association of diagnostic coding with trends in hospitalizations and mortality of patients with pneumonia, 2003–2009. Jama 307: 1405–1413.
- 2. Fine MJ, Auble TE, Yealy DM, Hanusa BH, Weissfeld LA, et al. (1997) A Prediction Rule to Identify Low-Risk Patients with Community-Acquired Pneumonia. N Engl J Med 336: 243–250.
- 3. Lim WS, van der Eerden MM, Laing R, Boersma WG, Karalus N, et al. (2003) Defining community acquired pneumonia severity on presentation to hospital: an international derivation and validation study. Thorax 58: 377–382.
- 4. Pine M, Norusis M, Jones B, Rosenthal GE (1997) Predictions of Hospital Mortality Rates: A Comparison of Data Sources. Ann Intern Med 126: 347–354.
- 5. Pine M, Jordan HS, Elixhauser A, Fry DE, Hoaglin DC, et al. (2009) Modifying ICD-9-CM Coding of Secondary Diagnoses to Improve Risk-Adjustment of Inpatient Mortality Rates. Med Decis Making 29: 69–81.
- 6. Lindenauer PK, Pekow P, Wang K, Mamidi DK, Gutierrez B, et al. (2005) Perioperative beta-blocker therapy and mortality after major noncardiac surgery. N Engl J Med 353: 349–361.
- 7. Lagu T, Rothberg MB, Nathanson BH, Pekow PS, Steingrub JS, et al. (2011) The relationship between hospital spending and mortality in patients with sepsis. Arch Intern Med 171: 292–299.
- 8. Lindenauer PK, Pekow P, Gao S, Crawford AS, Gutierrez B, et al. (2006) Quality of care for patients hospitalized for acute exacerbations of chronic obstructive pulmonary disease. Ann Intern Med 144: 894–903.
- 9. Elixhauser A, Steiner C, Harris DR, Coffey RM (1998) Comorbidity measures for use with administrative data. Med Care 36: 8–27.
- 10. DeLong ER, DeLong DM, Clarke-Pearson DL (1988) Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics 44: 837–845.
- 11. Kattan MW, Gonen M (2008) The prediction philosophy in statistics. Urol Oncol 26: 316–319.
- 12. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS (2008) Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med 27: 157–172 discussion 207–112.
- 13. Aujesky D, Auble TE, Yealy DM, Stone RA, Obrosky DS, et al. (2005) Prospective comparison of three validated prediction rules for prognosis in community-acquired pneumonia. Am J Med 118: 384–392.
- 14. Man SY, Lee N, Ip M, Antonio GE, Chau SS, et al. (2007) Prospective comparison of three predictive rules for assessing severity of community-acquired pneumonia in Hong Kong. Thorax 62: 348–353.
- 15. Capelastegui A, Espana PP, Quintana JM, Areitio I, Gorordo I, et al. (2006) Validation of a predictive rule for the management of community-acquired pneumonia. Eur Respir J 27: 151–157.
- 16. Bratzler DW, Normand SL, Wang Y, O’Donnell WJ, Metersky M, et al. (2011) An administrative claims model for profiling hospital 30-day mortality rates for pneumonia patients. PLoS One 6: e17401.
- 17. Tabak YP, Johannes RS, Silber JH (2007) Using automated clinical data for risk adjustment: development and validation of six disease-specific mortality predictive models for pay-for-performance. Med Care 45: 789–805.
- 18. Pine M, Jordan HS, Elixhauser A, Fry DE, Hoaglin DC, et al. (2007) Enhancement of Claims Data to Improve Risk Adjustment of Hospital Mortality. JAMA: The Journal of the American Medical Association 297: 71–76.
- 19. Fishman PA, Goodman MJ, Hornbrook MC, Meenan RT, Bachman DJ, et al. (2003) Risk adjustment using automated ambulatory pharmacy data: the RxRisk model. Med Care 41: 84–99.
- 20. Perkins AJ, Kroenke K, Unutzer J, Katon W, Williams JW Jr, et al. (2004) Common comorbidity scales were similar in their ability to predict health care costs and mortality. J Clin Epidemiol 57: 1040–1048.
- 21. Lagu T, Lindenauer PK, Rothberg MB, Nathanson BH, Pekow PS, et al. (2011) Development and validation of a model that uses enhanced administrative data to predict mortality in patients with sepsis. Crit Care Med 39: 2425–2430.
- 22. Lagu T, Rothberg MB, Nathanson BH, Steingrub JS, Lindenauer PK (2012) Incorporating initial treatments improves performance of a mortality prediction model for patients with sepsis. Pharmacoepidemiol Drug Saf 21 Suppl 244–52.
- 23. Borzecki AM, Christiansen CL, Chew P, Loveland S, Rosen AK (2010) Comparison of in-hospital versus 30-day mortality assessments for selected medical conditions. Med Care 48: 1117–1121.
- 24. Drahos J, Vanwormer JJ, Greenlee RT, Landgren O, Koshiol J (2013) Accuracy of ICD-9-CM codes in identifying infections of pneumonia and herpes simplex virus in administrative data. Ann Epidemiol 23: 291–293.