Report of the STS Quality Measurement Task ForceQuality Measurement in Adult Cardiac Surgery: Part 2—Statistical Considerations in Composite Measure Scoring and Provider Rating
Section snippets
Executive Summary
There is increasing interest among payers, patients, regulators, and providers to measure and compare cardiac surgery quality. The Society of Thoracic Surgeons (STS) Quality Measurement Task Force (QMTF) was established to develop comprehensive, summary performance measures encompassing multiple domains of quality. This report describes statistical considerations relevant to combining multiple measures into an overall composite score and then using such scores to rate providers.
The QMTF
Performance Measure Definitions
The QMTF selected 11 individual CABG performance measures (five process and six outcome), all of which were endorsed by the National Quality Forum (NQF) and are available in the STS NCD (Table 1). Specific NQF inclusion/exclusion criteria were applied to these measures to the extent possible. The internal mammary artery (IMA) measure excluded patients undergoing repeat CABG surgery, the permanent stroke measure excluded patients with a previous cerebrovascular accident, and the three discharge
Distribution of Individual Performance Measures
The distribution of provider-specific performance on NQF measures was investigated by using random-effects modeling to identify overall performance levels and to quantify between-provider variation. In general, measures that vary widely yield high statistical power for discriminating among providers. Care is needed to ensure that the less variable measures contribute additional statistical information when they are combined with more variable measures in a composite.
The estimated distribution
Individual Performance Measure Correlation
The estimated correlation between pairs of NQF measures (true process usage rates and true risk-standardized outcome rates) is summarized in Table 2, Table 3. For process measures, the estimated Pearson correlation between pairs ranged from 0.10 (IMA versus discharge antiplatelets) to 0.50 (preoperative β-blockers versus discharge β-blockers). For risk-adjusted outcome measures, the estimated Pearson correlation between pairs ranged from 0.15 (prolonged ventilation versus permanent stroke) to
Composite Scoring Methodologies
The QMTF selected four quality-of-care domains for CABG, represented by 11 NQF-endorsed process and outcomes measures: (1) Perioperative Medical Care (preoperative and discharge β-blockers, discharge antiplatelets, and discharge antilipids), (2) Operative Care (IMA usage), (3) Risk-Adjusted Operative Mortality, and (4) Postoperative Risk-Adjusted Major Morbidity (stroke, prolonged ventilation, renal insufficiency, reexploration for any cause, and deep sternal wound infection). The QMTF
Performance Tier Determination
Having selected a methodology for composite scoring, the QMTF considered various options for assigning providers to performance tiers. For example, a number of different approaches have been used in the United Kingdom, including confidence intervals to determine high and low outliers, ranks based on percentiles, and absolute thresholds, which, unlike the first two options, is not a reflection of performance relative to other providers [19, 20].
Using actual STS data, the QMTF pilot tested the
Comment
Composite indicators are useful for summarizing and comparing the quality of care delivered by healthcare providers. In many areas of medicine, the number of acceptable quality indicators is large, and there is a need for summary measures that combine performance on multiple end points. Although quality improvement requires attention to each individual aspect of quality, there are many settings in which the users of quality measures are most interested in the bottom line. The comprehensive
Conclusion
This two-part report by the STS QMTF describes the development of a multidimensional CABG composite quality score that is scientifically rigorous, uses NQF-endorsed measures from the STS NCD, and is consistent with relevant national guidelines for performance measurement. The QMTF regards this as the first step in a process that will constantly evolve as new quality measures, statistical methodologies, and health care policy objectives are developed.
References (25)
- et al.
Quality measurement in adult cardiac surgeryPart 1—Conceptual framework and measure selection
Ann Thorac Surg
(2007) - et al.
Cardiac surgery report cards: comprehensive review and statistical critique
Ann Thorac Surg
(2001) Performance measurement: accelerating improvement
(2006)- et al.
Statistical methods for profiling providers of medical care: issues and applications
J Am Stat Assoc
(1997) - et al.
Improving the statistical approach to health care provider profiling
Ann Intern Med
(1997) - et al.
League tables and their limitations: statistical issues in comparisons of institutional performance
J R Stat Soc (series A)
(1996) - et al.
Multilevel modelling of health statistics
(2001) - et al.
The Society of Thoracic Surgeons: 30-day operative mortality and morbidity risk models
Ann Thorac Surg
(2003) - et al.
Bayesian approaches to clinical trials and health-care evaluation
(2004) - et al.
Bayes and empirical Bayes methods for data analysis
(2000)
Bayesian data analysis
Combining HEDIS indicators: a new approach to measuring plan performance
Health Care Financ Rev
Cited by (201)
Measuring and reporting cardiac surgery quality: A continuing evolution
2023, Journal of Thoracic and Cardiovascular SurgeryPerioperative Optimization of the Cardiac Surgical Patient
2023, Canadian Journal of CardiologyChanges in Controllable Coronary Artery Bypass Grafting Practice for White and Black Americans
2023, Annals of Thoracic SurgeryComposite Performance Measures: The Foundation of The Society of Thoracic Surgeons Quality Measurement Program
2022, Annals of Thoracic SurgeryThe STS Participant-Level, Multiprocedural Composite Measure for Adult Cardiac Surgery
2022, Annals of Thoracic SurgeryFailure to Rescue: A New Society of Thoracic Surgeons Quality Metric for Cardiac Surgery
2022, Annals of Thoracic Surgery
- †
Dr Shahian is the Quality Measurement Task Force Chair and Writing Group Leader.