INTRODUCTION

Healthcare value is realized when patients can navigate healthcare systems to obtain needed care, collaborate with providers in treatment planning, and self-manage conditions between visits. The challenge lies in effective implementation of strategies to promote patient engagement. This may be especially true for safety net settings, as hospitals that serve patients with complex needs are more likely to be penalized under alternative payment models.1,2 Yet healthcare users differ in the extent to which engagement opportunities are experienced as burdensome, and often find it difficult to articulate such challenges and obtain support.3,4 Measurement of patient and family engagement is a priority in the Centers for Medicare and Medicaid Services (CMS) Meaningful Measures framework,5 and cited as an essential healthcare infrastructure need by the National Academy of Medicine Vital Directions initiative.6,7 The ability to measure engagement is pivotal to effectively personalize care, calibrate organizational efforts to promote patient engagement, and evaluate quality improvement efforts.

Healthcare engagement is operationalized as the patient behaviors required to optimize benefit from healthcare.8 We posit that the propensity to engage is a function of an individual’s cumulative self-efficacy for engagement behaviors.3 Self-efficacy is a dynamic, context-sensitive belief in one’s capacity to execute a behavior for a specific goal.9 The propensity to execute engagement behaviors depends only partially on individual resources (e.g., activated role orientation, health literacy, social determinants). Because engagement is an interactive process, the propensity to engage is also shaped by the ecological barriers and facilitators within healthcare systems, clinics, and provider relationships.3

As a new cross-cutting measurement goal, efforts to measure patient engagement have been bolstered by innovation in patient-reported outcome measures (PROMs) and their application as PROM-based performance measurement (PRO-PMs). PROMs can enhance patient-centeredness by amplifying the patient voice, but implementation of PROM-PMs requires additional considerations.10 PRO-PMs must be especially psychometrically sound if systems are accountable for incremental changes in scores. They must also be pragmatic: sufficiently brief, clear, and comprehensible to be completed independently at the point of care, integrated into clinical workflows, or cost-effectively administered to populations of users.11 Finally, PRO-PMs must be compatible with risk adjustment strategies, and, most importantly, demonstrate associations with actionable elements of the structure, process, or outcomes of care.12,13

The current study is the initial validation of the 8-item Patient-Reported Outcomes Measurement Information System (PROMIS®) Healthcare Engagement short form (PHE-8a). The PHE-8a is a subset of the 23-item PROMIS Healthcare Engagement (PHE) item bank v1.014, which has excellent psychometric properties and measures an individual’s propensity to engage with healthcare. The PHE items cover three interrelated domains of behavior: healthcare navigation, collaborative communication, and self-management.14

We constructed a Healthcare Engagement short form to offer a brief measure that can be administered in routine clinical care to assess engagement across diverse populations and comorbidities. The goals of the present study were to (1) establish reliability and concurrent validity of the PHE-8a against proximal indicators of patient engagement and patient experience; and (2) establish predictive validity of the PHE-8a against actionable elements of care that are plausibly linked to patient engagement.

METHODS

For this observational cohort study, we conducted a nationwide mail survey of 25,500 individuals receiving care in 136 Veterans Health Administration (VHA) medical centers in the continental US. Eligible Veterans were identified from administrative data. Surveys were administered from October to December 2018 and responses were linked to VHA administrative data. Concurrent validity was examined via cross-sectional relationships among variables obtained from the survey; predictive validity was examined using a prospective design, aggregating administrative data over the year following the survey. Participants received a $1 non-contingent incentive with the initial mailing, and $10 following completion of the survey.14 The study was approved by the Stanford University School of Medicine Institutional Review Board.

Participants

We sampled from Veterans receiving VHA care for diabetes, hypertension, depression, and/or post-traumatic stress disorder to evaluate the PHE-8a among individuals with a range of treatment demands and to assure applicability for both mental health and chronic conditions. Diagnostic criteria were one past-year inpatient stay or two outpatient visits on different dates. Inclusion criteria were Veteran, age 18–80, White or Black race (Other racial categories each represent very small proportions of VHA users and are often misclassified in administrative data)15, a valid mailing address in VHA administrative data, and assignment to a primary care panel with at least one associated visit in the past year. Exclusion criteria were active diagnoses of dementia or psychotic disorders, or Care Assessment Needs risk scores indicating top 5th percentile risk for hospitalization or death.16 Sampling was stratified within each condition cohort: Because VHA serves a predominantly White and male population, we oversampled women and Black Veterans to 20% of the sample. We balanced the sample on Nosos risk adjustment scores17 to assure variability in the burden of illness.

Data Sources and Measures

We used survey self-report to obtain demographic information and patient-reported measures. The PHE-8a (Supplement 1) is a subset of the 23-item PHE item bank v1.0. (The construct3 and psychometric development14,18 are described elsewhere). The items of the PHE-8a were chosen using computerized adaptive testing (CAT) simulation targeting measurement precision at 1 standard deviation (SD) below the mean, to ensure discrimination at lower levels of engagement (Supplement 2). Consistent with other PROMIS measures, the PHE-8a uses item response theory–based scoring converted to standardized T-scores. Several measures were included to assess cross-sectional convergent validity with related constructs: an indicator variable for a usual source of care adapted from the Behavioral Risk Factor Surveillance System Survey;19 a screen for inadequate health literacy,20 and the 4-item short form of the PROMIS Self-Efficacy for Self-Managing Symptoms.21 We also assessed concurrent validity with composite scores for the Consumer Assessment of Healthcare Providers and Systems (CAHPS)22 communication (CAHPS 5.0) and self-management support scales (CAHPS Patient Centered Medical Home 3.0)23.

Prospective outcomes were ascertained via VHA administrative data. No-show rates were calculated as the proportion of all appointments (no-show or completed) that were no-shows for Veterans with ≥ 3 visits in primary care and mental health, using data from the Managerial Cost Accounting System. Data indicating use of the VHA patient portal, My HealtheVet, and initiation of one or more secure message threads were obtained from the VA Corporate Data Warehouse (CDW). We obtained the following Healthcare Effectiveness Data and Information Set (HEDIS) electronic quality measures from the VHA office of Reporting, Analytics, Performance, Improvement & Deployment: HbA1c poor control (HbA1c ≥ 9 or not tested); controlling high blood pressure (BP < 140/90, age 18–85); and statin therapy adherence (proportion of days covered ≥ 80%). We obtained covariates for rurality24 and Nosos risk adjustment scores from the CDW.

Statistical Analysis

We conducted analyses using Stata 17.0 (StataCorp, LLC, TX, USA). We examined each outcome as a function of patient engagement using mixed effects generalized linear models with a random intercept to account for clustering within VHA facility. Models were adjusted for age group, race/ethnicity, rurality, and Nosos quartile. Binary outcomes and proportions were modeled using mixed effects logistic regressions and mixed effects probit regressions, respectively. Estimated marginal means were plotted at each 1/2 SD in PHE-8a scores to illustrate effects across the continuum of engagement. All analyses were survey-adjusted to account for stratified sampling.

RESULTS

Sample Characteristics

The sample included 9552 Veterans (Table 1). The response rate for the mail survey was 38%.14 We assessed nonresponse bias using a health services research decision framework,25 and found little evidence of bias. The proportion of active non-responders (e.g., opt-outs) was low (6.6%). Comparisons on substantive study variables for responders vs. non-responders (Supplement 3) and early vs. late responders revealed only very small differences (effect sizes ≤ 0.05) with the exception of a small effect (Cohen’s h = 0.2) for a lower likelihood of a primary care no-show in the year prior to the survey among responders relative to non-responders. Responders were moderately older (mean [SD] age = 62 [12] years), though this yielded a better correspondence to the VHA population (mean [SD] age = 62 [17] years).26 Because age was associated with small differences in engagement scores (Table 1), but not consistently associated with primary outcomes, we addressed age differences via covariate adjustment rather than weighting.

Table 1 Risk-adjusted Differences in Healthcare Engagement Scores, by Demographic Characteristics (N = 9552)
Full size table

Engagement T-scores ranged from 23 to 66 and did not meaningfully differ by gender, race/ethnicity, education or rural residence (Table 1). Engagement scores were lower among younger Veterans, and Veterans reporting more financial strain. As expected, engagement scores were higher among Veterans who reported a usual source of care and those with adequate health literacy.

Reliability

The PHE-8a demonstrated good internal consistency with a Cronbach’s alpha of 0.88, and marginal reliability ≥ .85 from approximately −2.5 SD to +1.5 SD. A subset of 428 participants completed the PHE-8a via web-based survey on two occasions (median interval 2 days; interquartile range: 2–4 days) yielding a good test-retest intraclass correlation coefficient of 0.89 (95% CI: 0.87–0.91).

Concurrent Validity

Engagement scores were positively associated with better self-efficacy for self-management (β = 0.48; 95% CI: 0.47–0.50; p < 0.001; N = 9399), and higher ratings of provider self-management support (OR = 1.08; 95% CI: 1.08–1.09; p < 0.001; N = 9195). Higher PHE-8a scores were also associated with higher ratings of provider communication (OR = 1.19; 95% CI: 1.18–1.20; p < 0.001; N = 9209). Statistically significant increases in each of these correlates were observed at each 1/2 SD (5 points) increase in engagement scores (Fig. 1).

Figure 1
figure 1

Proximal correlates: self-efficacy for self-management and ratings of provider communication and self-management support, by healthcare engagement scores. Error bars are 95% confidence intervals. Scores and predicted probabilities are adjusted for age group, race/ethnicity, rurality, and Nosos risk scores. PROMIS, Patient-Reported Outcomes Measurement Information System; Pr, probability; CAHPS, Consumer Assessment of Healthcare Providers and Systems; SM, self-management.

Full size image

Predictive Validity

Higher engagement scores predicted lower outpatient no-show rates over the following year for primary care and mental health appointments. With each 1-point increase in engagement T-scores, the rate of primary care no-shows decreased (b = −0.008; 95% CI: −0.012 to −0.004; p < 0.001; N = 6926), as did the rate of mental health no-shows (b = −0.007; 95% CI: −0.011 to −0.003; p < 0.001; N = 4955). Figure 2 illustrates the 37% reduction in primary care and 24% reduction in mental health no shows across the continuum of low to high engagement scores, with statistically significant differences at each 1/2 SD (5-point) increment.

Figure 2
figure 2

Primary care and mental healthcare no-show rates, by healthcare engagement scores. Error bars are 95% confidence intervals. Rates are adjusted for age group, race/ethnicity, rurality, and Nosos risk scores.

Full size image

Better healthcare engagement also predicted use of the My HealtheVet patient portal and, among users, secure messaging with a care team member. Each 1-point increase in scores predicted a greater likelihood of using the patient portal in the following year (OR = 1.01; 95% CI: 1.00–1.01; p = 0.002; N = 9539), and among My HealtheVet users, a greater likelihood of initiating secure messaging with a provider (OR = 1.01; 95% CI: 1.00–1.01; p = 0.004; N = 4359). Figure 3 illustrates the continuum of engagement scores and the 15.4% increase in My HealtheVet use and the 22% increase in secure message use among individuals with high engagement relative to low engagement. Statistically significant increases are shown at each 1/2 SD.

Figure 3
figure 3

Any patient portal use and initiation of secure messaging among users, by healthcare engagement scores. Error bars are 95% confidence intervals. Predicted probabilities are adjusted for age group, race/ethnicity, rurality, and Nosos risk scores. Pr, probability.

Full size image

Higher healthcare engagement scores also predicted better quality of care for chronic conditions (Fig. 4). Each 1-point increase in engagement scores was associated with lower odds of HbA1c poor control (OR = 0.98; 95% CI: 0.97–0.99; p < 0.001; N = 3705), higher odds of controlling high blood pressure (OR = 1.01; 95% CI: 1.00–1.01; p = 0.011; N = 5876), and higher odds of statin adherence (OR = 1.01; 95% CI: 1.00–1.02; p = 0.022; N = 3280). Figure 4 illustrates the 44% reduction in the likelihood of poor diabetes control, 13.4% increase in hypertension control, and the 12.7% increase in statin adherence across the continuum of low to high engagement scores, with statistically significant differences at each 1/2 SD.

Figure 4
figure 4

HEDIS quality measures: diabetes poor control, controlling high blood pressure, and statin adherence, by healthcare engagement scores. Error bars are 95% confidence intervals. Predicted probabilities are adjusted for age group, race/ethnicity, rurality, and Nosos risk scores. HEDIS, Healthcare Effectiveness Data and Information Set; Pr, probability; BP, blood pressure.

Full size image

DISCUSSION

In this evaluation of a brief, patient-reported measure of healthcare engagement, engagement was prospectively associated with better quality of care among a large, national sample of VHA users with chronic conditions. Previously, evidence for the impact of healthcare engagement was largely triangulated from related constructs, such as self-management/activation and patient-provider communication.27,29,29 The ability to directly measure healthcare engagement charts a clearer course towards healthcare value, with potential to identify points of intervention for personalized care, to guide organizational and systems-level efforts to promote engagement, and to evaluate such quality improvement interventions.

As expected, the PHE-8a was strongly correlated with related constructs, where higher engagement scores were associated with increasing likelihood of maintaining a usual source of care, adequate health literacy, and better self-efficacy for self-management. Our data also indicate that Veterans with better healthcare engagement showed the expected robust increases in positive experiences with provider communication and perceptions of self-management support. These are meaningful correlates of engagement, as patient-provider communication is central to healthcare access and outcomes, especially within public sector systems.30 Taken together, these results lend strong criterion validity evidence for operationalization of the healthcare engagement construct by the PHE-8a.

Prospective analyses revealed that higher levels of healthcare engagement demonstrated a strong dose-response relationship predicting lower no-show rates for primary care and mental health visits over 1 year. Missed appointments are costly, inefficient, and detract from the regularity and continuity of care that promotes better outcomes.31 The primary care no-show rate observed among the more highly engaged Veterans in our sample was commensurate with rates observed for resource-intensive interventions employing reminder calls, enhanced provider continuity, and direct scheduling phone lines.32 Our results suggest that elements of these interventions that require more staff time, such as reminders, could be targeted towards individuals with lower healthcare engagement. In mental health settings, no-show rates were substantially higher, but the pattern of meaningful reductions in no-show rates among users with higher engagement scores was similar to that observed for primary care. Veterans with lower levels of engagement may benefit from shorter wait times for appointments,33 telehealth,34 or enhanced reminders, which in turn could ameliorate some of the negative consequences of high no-show rates, such as difficulties ensuring a minimally therapeutic dose of treatment.35,36

Engagement scores also demonstrated prospective associations with patient portal use. Higher engagement scores predicted a higher likelihood of using the My HealtheVet portal during the following year, and portal users with higher engagement scores were proportionately more likely to contact a provider via secure messaging. Patient portal use is broadly considered an indicator of healthcare engagement behavior from early electronic health record incentive programs to CMS alternative payment models. In VHA, secure messaging through the portal facilitates engagement behaviors, such as medication fills, scheduling, and health symptom discussions.37 The association between engagement and portal use provides convincing evidence for the predictive validity of the PHE-8a, as evidence for this association has been mixed in prior research using activation as a proxy for engagement.38,39

Finally, as healthcare engagement scores increased, there was a proportional increase in the likelihood of better performance on HEDIS quality measures for diabetes and hypertension control and for statin adherence. Prior research has demonstrated improvements in these engagement-sensitive outcomes after VHA primary care redesign,40 which included organizational and systems-level strategies designed to promote engagement through better access, continuity, and communication. Conversely, persistently poor scores on these measures are associated with more complex treatment plans that make engagement more challenging.41 The PHE-8a may be valuable in primary care settings to identify patients at risk for poor outcomes who may benefit from proactive, adjunctive intervention.

We demonstrated strong dose-response relationships of healthcare engagement with better patient experience, engagement behaviors, and outcomes. We observed these associations even after risk adjustment, which is notable as VHA uses a tailored risk-adjustment model that incorporates additional information relevant for vulnerable populations, such as pharmacy data and comprehensive data for mental health conditions.17 The current results establish the PHE-8a as a reliable, valid, and predictive PRO with excellent potential for evaluating individual or systemic interventions to promote engagement, or for tailoring individual care as a part of population health management. The PHE-8a demonstrates precision where statistically significant and clinically meaningful differences were apparent at 5-point differences, or 1/2 SD, a common benchmark for the minimally important clinical difference. The PHE-8a is brief, feasible to administer at the point of care or electronically, and the sixth grade reading level made the PHE accessible to patients with low literacy. Finally, the PHE is freely available, which may be an important advantage for under-resourced or public-sector settings (see https://www.healthmeasures.net/search-view-measures).

Psychometric evaluation is an iterative process,42 and these results are a comprehensive but first step in establishing the clinical validity of the PHE-8a. Results provide strong support for the utility of the PHE-8a due to the large heterogeneous sample and medical record-based outcomes, which enhances ecological validity and limits bias due to method variance. Development in a public sector setting like VHA may enhance applicability to other public or safety-net settings, cross-validation outside VHA will be important. In particular, VHA is a large integrated system, and the measure’s robustness to differences regarding care fragmentation, access, or third-party payors is unknown.43,44 Burgeoning research into the structural and cultural determinants of engagement behavior must guide ongoing evaluation of the PHE. Additional validity research on the measure’s sensitivity to change and precision in characterizing facilities or clinics will also be important to fully determine its value for performance measurement.

Measurement of healthcare engagement and related constructs requires some caution to ensure that these foster equitable healthcare improvement.45 Previously, patient-reported and patient-centered performance measures have been proposed to refine value-based payment models that inadvertently penalize providers or systems that serve complex or vulnerable patients.1,46 Concerns remain that financial accountability incentivizes use of these measures to select healthier or less socially vulnerable patient populations. Implementation must guard against pressure to achieve efficiencies by targeting patients who appear to be easier to engage, as systems should not be penalized for serving populations that struggle to engage with care. In contrast, the PHE-8a items were selected for measurement sensitivity below the population mean to better serve such patients. With the emergence of evidence-based strategies that improve care for individuals who struggle to engage, reimbursement or performance incentives could be applied so that systems are appropriately compensated for the additional time, resources, or community partnerships that such strategies require relative to engagement levels.47 Ultimately, use of measures such as the PHE can support the achievement of high value care through a patient-centered approach that equitably promotes engagement rather than through risk selection.