Background

The pandemic outbreak of the novel coronavirus SARS-CoV-2 and its associated disease, COVID-19 in 2019 [1] has disproportionately impacted racial and ethnic minority populations [2]. Consistent disparities in disease prevalence, severity, and mortality have been reported across United States geographic regions with available race/ethnicity data. COVID-19 deaths in early April 2020 were disproportionate among Black residents of New York City, the initial epicenter of the disease (22% of population but 28% of deaths) and of the larger New York state (9% of the population but 18% of deaths) [3]. In North Carolina, Black residents account for 22% of the population but comprised 36% of COVID-19 related deaths as of May 2020 [4]. Preliminary research published in May 2020 indicated that 22% percent of United States counties that are disproportionately Black accounted for 52% of COVID-19 diagnoses and 58% of COVID-19 related deaths nationally [5]. The age-adjusted rate of COVID-19 hospitalization among non-Hispanic Black individuals (178.1 per 100,000 persons) and Hispanic or Latino (160.7 per 100,000 persons) individuals was over three times higher than that of non-Hispanic White individuals (40.1 per 100,000 persons), between March and June 13, 2020 according to the Centers for Disease and Control (CDC) [6]. These disparities represent a confluence of structural racism [7, 8] with other social and economic factors [9,10,11] that increase the risk of COVID-19 exposure, social-network transmission, and disease-related complications among demographically vulnerable populations.

While there has been an increase in COVID-19 related health care, preliminary evidence also indicates that individuals are delaying other types of care during the pandemic, including care for chronic disease and other acute emergencies. Hospital admissions for heart attacks and strokes have plummeted [12, 13], which may be an indication that patients are afraid to connect to the health care system, even for emergency medical services [14]. Ambulatory care visits declined nearly 60% in mid-March 2020 and remained low through April 2020 according to an evaluation of data from over 50,000 providers [15]. Evidence from the initial 2002–2004 SARS-CoV outbreak showed that chronic-care hospitalizations for diabetes dropped significantly during the crisis and rose dramatically during the post-crisis time period [16]. Because the prevalence of comorbid risk factors for COVID-19 complications including hypertension, diabetes, obesity, and cardiovascular disease is disproportionately higher among Black and Hispanic/Latino populations [17, 18], delaying care for chronic disease and emergency health can further exacerbate health disparities in COVID-19 related complications as well as non-COVID-19 related mortality.

As part of a system-wide rapid response to health equity challenges highlighted by COVID-19, Atrium Health recognized the need to increase awareness of COVID-19 resources and to prevent care delays for general health and social service needs. To address this need, Atrium Health designed and implemented the COVID-19 Patient Engagement Messaging campaign (PEM campaign), a multi-phased engagement outreach to patients in 13 community and rural clinics in a North Carolina county, which serve patients with limited access to health care. The campaign comprised of text and recorded voice messages that encouraged patients to call a dedicated call-center resource. Targeted text and voice messaging campaigns have been used in prior public health and health system initiatives to address information and education gaps among vulnerable populations and to promote behavioral responses including medication adherence [19] and health screenings [20]. A recent call-to-action paper by the National Institute of Health researchers highlighted the need for more digital and community-based interventions to bridge the gap in health care access resulting from the COVID-19 pandemic [21].

Objective

The objective of this manuscript is to describe the development, implementation, and results from the initial deployment of the PEM campaign in a North Carolina county. A descriptive analysis was conducted to evaluate message delivery and call-center calls among the total population of community and rural clinic patients identified for the campaign. The effectiveness of the campaign in promoting calls to the call-center and health care seeking behaviors, defined as scheduling a health care appointment and receiving a COVID-19 test, was assessed among patients that received a message and had no recent contact with Atrium Health prior to the start of the campaign.

Methods

Patient Engagement Messaging Campaign Overview

The PEM campaign was a broad, population-based outreach, delivered as part of a larger health system rapid-response strategy for COVID-19 health disparities. In early April 2020, a multi-disciplinary team of clinicians, researchers, and health equity experts aligned to develop an engagement campaign for patients of community and rural clinics that would help them connect to COVID-19 testing and educational resources, and prevent care delays for other health care needs. The campaign was operationalized into a multi-phase series of informational messages delivered through text (Phase 1) and recorded voice messages (Phase 2a and Phase 2b) using an automated technology solution. Messages were designed and implemented using an expert review approach that engaged key stakeholders across multiple departments. Primary care providers from the targeted community and rural clinics were specifically included as key stakeholders and experts in all stages, from message development and evaluation design to interpretation of results.

The PEM campaign messages promoted a call-center service available to Atrium Health patients to connect with a trained representative that would answer COVID-19 questions, provide information about COVID-19 testing sites in their local community, and assist with scheduling health care appointments in compliance with current safety protocols. The content for Phase 1 and Phase 2a messages focused on COVID-19 education and access to testing, (e.g., “If you are having symptoms of coronavirus, like shortness of breath, cough, fever, loss of sense of taste or smell or have general questions and would like to speak to your health care team please call us at [call-center phone-number]”). The Phase 2b message focused on connecting patients to primary care and preventing care delays (e.g., “We want to make sure you know how to reach your doctor for all your medical, social and emotional needs. Don’t put off getting the care you need. To find out exactly what to do, call us at [call-center phone number].” Messages were provided in both English and Spanish. The language of messages was aligned with the preferred language indicated in each patient’s electronic medical record (EMR). See Appendix A for full message transcripts.

Implementation

The PEM campaign was delivered over a 3-week period. Messages were sent in hourly batches for two consecutive days starting at 9am on the first day of each phase and ending at 4 pm on the second day. Phase 1 started on Tuesday April 21, 2020, followed by Phase 2a on Tuesday April 28, 2020 and Phase 2b on Wednesday May 6, 2020. The automation logic for the text message (Phase 1) delivery system required the recipient to have provided a cell-phone number. The recorded voice message (Phase 2a and Phase 2b) delivery system prioritized finding a home phone number, but it also sent a message to a cell-phone if a home phone number was not available. A phone number attributed to each community clinic was provided in all messages that connected to a regional operation call center (ROC) operated by staff with specialized training to answer COVID-19 questions, connect patients to local COVID-19 testing sites, and schedule appointments for other health care needs.

Campaign Population

The PEM campaign was designed to target established patients, defined as those who were attributed to and visited one of the 13 community and rural clinics in the past 18 months. The campaign population included patients who identified English or Spanish as their preferred language, had at least one phone number on record, did not opt-out of text messaging communication, and were living at the time of the data extraction (n = 48,063). The campaign population was sent messages during all phases (Phase 1, Phase 2a, and Phase 2b) of the PEM campaign.

Campaign Evaluation Design

The evaluation design was part of a rapid-response strategy that included multiple simultaneous workflows to design, implement, and test the PEM campaign. Workflows were led by domain experts and included cross-department participation that supported an agile, solution-focused process. The message development and evaluation design teams were supported in two distinct yet parallel workflows that included providers from the community and rural clinics to help anchor the impact goals of the campaign and design a responsive evaluation strategy. The evaluation was designed to answer the following stakeholder questions: “Did the campaign reach vulnerable populations?” and “Did the campaign help patients that were disconnected from the health care system access care?”.

To address these impact questions, the campaign evaluation was conducted using a standard cross-sectional design approach and comprised of: i) descriptive analysis of message delivery among the campaign population; ii) descriptive analysis and regression analysis of determinants of calls to the ROC-line; and iii) regression analysis assessing associations between calling the ROC-line and health care seeking behaviors. To focus the analysis on patients who were disconnected from the health care system, samples of patients without recent Atrium Health contact were extracted from the total campaign population (n = 48,063). Recent contact was defined as attending any health care appointment at any Atrium Health facility, receiving a COVID-19 test at any Atrium Health facility, or calling the ROC-line attributed to any of the 13 community and rural clinics between March 1, 2020 and April 20, 2020. A total of 4 unique samples were extracted that included all patients from the campaign population that successfully received a message during each phase (i.e., Phase 1 sample, Phase 2a sample, Phase 2b sample), and during the overall campaign period (i.e., combined analytic sample). Patients that met the exclusion criteria for having recent health care contact were excluded from samples. The ROC call-center supporting a teenager-focused health clinic was not able to collect identifiable call data, and therefore patients attributed to that clinic were also excluded from samples.

A patient selection flow diagram for all samples is presented in Fig. 1. The samples size for each extraction was determined as follows. The Phase 1 sample (n = 22,460) excluded those for whom the text message was not delivered during Phase 1 (n = 14,471) and those meeting one or more of the exclusion criteria (n = 11,321). Phase 2a (n = 28,466) and Phase 2b (n = 27,298) samples also excluded those without message delivery during each phase (5210 and 5241 respectively) that met one or more of the exclusion criteria (14,387 and 15,524, respectively). A combined analytic sample (n = 29,214) was extracted that excluded those without any successful message delivery in any of the campaign phases (n = 2195), met at least one of the exclusion criteria (n = 16,654), and for whom gender was unknown (n = 1). Individuals with “other” insurance (n = 107) reflected less than 1% of sample and included those with governmental insurance and other program-specific options that did not conceptually align with larger insurance categories. As this sample size was too small to produce reliable regression model estimates, they were also excluded. The evaluation protocol was approved by Atrium Health Institutional Review Board (IRB) as a quality improvement effort that was exempt from human subjects’ oversight by the IRB and did not require participant consent.

Fig. 1
figure 1

Patient selection flow diagram. Note: All samples represent those who received a message and did not have recent contact with Atrium Health; *Combined Analytic Sample was used in regression analysis to estimate outcomes: calling the ROC-line, scheduling a health care appointment, and receiving a COVID-19 test during the campaign

Full size image

Measures

Message Delivery

For Phase 1 (text), messages that were successfully sent were coded as delivered, and individuals with an invalid phone number (e.g., land-line phone number) or other carrier errors (e.g., phone does not receive text messages) were coded as not delivered. For Phase 2a and 2b (recorded voice message) calls that were answered or recorded on an answering machine were coded as delivered, and calls with an invalid number, no answer, out of order, or busy signal were coded as not delivered.

Outcome: Call to ROC-Line

The first outcome of interest, calling the ROC-line was defined separately for each phase of the PEM campaign, Phase 1 (Tuesday, April 21 to Monday, April 27), Phase 2a (Tuesday April 28–Monday May 5), and Phase 2b (Wednesday May 6–Tuesday May 12). Individuals that called the ROC-line at least one-time during Phase 1, Phase 2a, and Phase 2b windows were coded as having called during that phase, respectively, as binary variables. Calling the ROC-line was not mutually exclusive between phases. Individuals that called at least once at any time during the campaign (Tuesday, April 21–Tuesday May 12) were coded as having called the ROC-line during the campaign, using a binary variable.

Outcomes: Health Care Seeking Behaviors

The outcomes were defined as (i) scheduling any healthcare appointment at any Atrium Health facility (binary variable) during the campaign, and (ii) completing a COVID-19 test at any Atrium Health testing location (binary variable) during the campaign.

Covariate: At-Risk Health Status

An expert panel of Atrium Health physician leaders and researchers developed a definition of “at-risk” for COVID-19 health complications using the Centers for Disease and Control criteria for moderate comorbidities [22] and provider feedback for the health system specific exclusions. Using this definition, patients were defined as having an “at risk” health status if they were elderly (65 years-old or more) or were an adult (18 years-old or more) with one or more of the following comorbidities: severe obesity, diabetes, chronic obstructive pulmonary disease /emphysema, asthma, hypertension, coronary artery disease, congestive heart failure, lupus, sarcoidosis, immunodeficiency. Based on expert panel recommendation, patients currently on dialysis, had a history of a left ventricular assist device procedure, were active or recent transplant patients, or have malignancy, HIV/AIDs, or sickle cell disease were excluded from the “at risk” definition because they have unique health care access pathways and/or are already connected to and receiving specialty care.

Covariates: Patient Demographic Factors

Gender was defined as a binary variable (Male or Female). Race and Ethnicity were defined using a combined variable as Hispanic/Latino and among those who were either non-Hispanic/Latino or unspecified ethnicity, race was categorized as Black, White, and Other/Unknown. Insurance status was measured using the primary source of payment indicated for an individual’s most recent health care visit to a Atrium Health facility using the following categories: Medicaid, Medicare, private, or uninsured. Medicare included both Advantage (commercial) and non-Advantage (public) members. Private represented all commercial insurance categories. For the purpose of this project, patients indicating “self-pay” were considered uninsured. Patient age was incorporated into the definition of “at-risk health status” and was not included as a separate covariate to reduce collinearity. The County Public Health Department identified six ZIP codes, with disproportionately low educational attainment and high proportion of the population living below the poverty threshold, as public health priority areas (PHPAs) as: 28217, 28208, 28216, 28206, 28205, and 28212. The PHPA residency of each patient’s home address ZIP code was defined as a binary variable (PHPA or non-PHPA).

Data Collection

Data were collected from three distinct sources within the Atrium Health to complete the evaluation. Message delivery data was collected from the TeleVox system (Intrado Corporation; Omaha, NE). Patient demographic, health history, and COVID-19 test history data were collected from patient EMRs (Cerner Corporation, Kansas City KS). The ROC-line call data were collected using Cisco Unified Contact Center Enterprise Software (Cisco Corporation, San Jose, CA), as the phone number and time stamp for all calls to community clinic- attributed lines during the data collection window.

Analysis

Descriptive Analysis of Message Delivery Among Campaign Population

The total counts and percentages of messages delivery results during each phase of the campaign, and during the overall campaign, were calculated along with descriptive statistics (chi-squared test) to evaluate differences in the proportion of message delivery results among race/ethnicity and at-risk health status categories.

Effectiveness Analysis of Calling ROC-Line and Health Care Seeking Behaviors Among Samples

The total count and percentage of patients that called the ROC-line during each phase of the campaign was calculated for the Phase 1a, Phase 2a, and Phase 2b samples, and during the overall campaign for the analytic sample. The analysis described hereafter was conducted on the analytic sample constructed by combining Phase 1a, Phase 2a, and Phase 2b samples with additional exclusion criteria. Descriptive statistics (chi-squared test) were calculated to characterize the analytic sample. Total counts of each outcome by day were plotted using a grouped bar chart for the duration of the campaign. Calling the ROC-line during the campaign was estimated using logistic regression as a function of patient health risk status, gender, race/ethnicity, insurance, and PHPA residency. Associations between calling the ROC-line and secondary health care seeking behavior outcomes, scheduling a health care appointment and receiving a COVID-19 test, were modeled using logistic regression, controlling for patient health risk status, gender, race/ethnicity, insurance status, and PHPA residency. To evaluate if vulnerable populations were differentially impacted by calling the ROC-line, interaction terms were included between the ROC-line, and patient health risk status, race/ethnicity, insurance status, and PHPA residency variables. Stratified samples were created for interaction terms that were significant at the 5% level, and health care seeking behaviors were estimated separately for each stratified sample. Estimates were exponentiated and interpreted as odds ratios. All analyses were performed using R version 4.0.1 [23].

Results

Message Delivery Among Campaign Population

The message delivery results among the overall campaign population are described in Table 1. Of the 48,063 patients attributed to community and rural clinics that were sent messages during the PEM campaign, 95.4% (n = 45,868) had at least 1 message delivered. Approximately 69.9% of patients had the Phase 1 (n = 33,592) text message delivered, followed by 89.2% of the Phase 2a (n = 42,853) and Phase 2b (n = 42,822) recorded voice messages. Black patients (vs. White) had the lowest rates of message delivery in all phases, with the largest gap observed for the Phase 1 text message (65.6% vs. 77.5%), compared to the Phase 2a (87.6% vs. 90.6%) and 2b (87.3% vs. 90.7%) recorded voice messages. A smaller gap in message delivery was observed among Hispanic/Latino (vs. White) patients for the Phase 1 text message (69% vs. 78%) and almost no differences were observed for Phase 2a (90.7% vs. 90.6%) and 2b (90.9% vs. 90.7%) recorded voice messages. Phase 1 text messages were equally delivered to patients at risk for COVID-19 health complications (vs. not at risk) (69.9% vs. 69.9%). However, the Phase 2a and 2b messages had slightly higher delivery rates among patients at risk (~ 90%) compared to patients not at risk (~ 89%) (Table 1).

Table 1 Campaign population message delivery results by patient characteristics (n = 48,063)
Full size table

ROC-Line Calls

The first outcome of interest, calling the ROC-line, was examined among samples aligned with each phase of the campaign that met the exclusion criteria of not having recent contact with Atrium Health. Approximately 1.5% (n = 333) of the Phase 1 sample called the ROC-line within 1 week of receiving a text message, followed by 4.5% (n = 1,269) of the Phase 2a sample that received a recorded message, and 3.2% (n = 880) of the Phase 2b sample that received a recorded voice message. The combined analytic sample consisted of 29,214 patients that received at least 1 message during the campaign. Approximately 7.4% (n = 2156) of the combined analytic sample called the ROC-line during the campaign.

All outcomes described hereafter reflect evaluation of the combined analytic sample. By the definition of the sample, only patients without recent contact were included in the sample and thus calls to the ROC-line were zero prior to start of the campaign. The proportions of patient characteristics among the combined analytic sample are described below for those who called the ROC-line at any time during the campaign vs. those who didn’t call. Patients who called the ROC-line (vs. no call) were disproportionately at-risk for COVID-19 health complications (45.4% vs. 40.6%), Black (61.7% vs. 39.3%), Medicaid-insured (44.3% vs. 38.8%), Medicare-insured (16.4% vs. 13.9%), uninsured (21.3% vs. 15.6%), and living in a PHPA (47.2% vs. 30.3%). Difference in proportions were statistically significant at the 1% level. The total counts of outcomes are presented in Fig. 2 by day, starting with the Phase 1 message push on April 21, 2020. A noticeable spike in ROC-line calls is observable on the 2 consecutive days of message delivery for each phase, with a similar pattern for health care appointments scheduled (Fig. 2).

Fig. 2
figure 2

Total outcome counts by campaign day among combined analytic sample (n = 29,214). ROC regional operation call center. Note: Analytic Sample represents a subset of the campaign population defined as those meeting the exclusion criteria including those without Healthcare System contact (ROC-line call, health care appointment, or COVID-19 test) prior to the start of the campaign, therefore outcomes counts were zero prior to 4/21/202

Full size image

Results from the regression analysis are described in Table 2, and showed higher odds of calling the ROC-line among those at-risk for COVID-19 health complications compared to those not at-risk (Odds Ratio [OR]: 1.18; 95% Confidence Interval [CI] 1.07–1.30). Higher odds of calling the ROC-line were observed among Black patients (OR: 2.77; 95% CI 2.40–3.21) and Hispanic/Latino patients (OR: 1.28; 95% CI 1.08–1.53) compared to White patients. Within insurance categories, those who were Medicaid-insured (OR: 1.65; 95% CI 1.45–1.88), Medicare-insured (OR: 1.82; 95% CI 1.56–2.12), or Uninsured (OR: 2.19; 95% CI 1.89–2.53) had higher odds of calling the ROC-line compared to those who were privately-insured. Patients living in a PHPA had higher odds of calling the ROC-line (OR: 1.57; 95% CI 1.43–1.72) compared those living in the larger county (Table 2).

Table 2 Odds of ROC-line call among combined analytic sample (n = 29,214)
Full size table

Scheduling a Health Care Appointment

Overall, 12.1% (n = 3,540) of the combined analytic sample scheduled a health care appointment during the campaign period. The odds of scheduling a health care appointment at any Atrium Health Facility during the campaign were 314% higher among patients who called the ROC-line during the campaign (OR: 4.14; 95% CI 2.93–5.80) compared to those who didn’t, after adjusting for at-risk health status, race/ethnicity, insurance, gender, and PHPA residency (Table 3).

Table 3 Odds of health care appointment scheduled among combined analytic sample (n = 29,214)
Full size table

The interaction term between calling the ROC-line and insurance was significant at the 5% level, indicating that scheduling a health care appointment varied based on an individual’s insurance type. Thus, the analytic sample was stratified by insurance type and scheduling a health care appointment was estimated separately for each insurance stratum. The magnitude of association between scheduling a health care appointment and calling the ROC-line was largest among the Privately-insured stratum (OR: 4.50; 95% CI 3.58–5.64), followed by the uninsured stratum (OR: 3.13; 95% CI 2.48–3.94), the Medicaid-insured stratum (OR: 2.77; 95% CI 2.31–3.31), and the Medicare-insured stratum, (OR: 2.48; 95% CI 1.95–3.13). We found no other statistically significant differences in the relationship between calling the ROC-line and scheduling a health care appointment by at risk health status, race/ethnicity, or living in a PHPA (Table 3).

COVID-19 Test

Overall, 0.8% (n = 233) of the combined analytic sample received a COVID-19 test during the campaign period. The odds of receiving a COVID-19 test during the campaign were 139% higher among patients who called the ROC-line during the campaign (OR: 2.39; 95% CI 1.64–3.39) compared to those who didn’t, after adjusting for at-risk health status, race/ethnicity, insurance, gender, and PHPA residency (Table 4). We found no statistically significant difference in odds of being tested between minorities and White patients, regardless of ROC-line call. The proportion of the sample that received a COVID-19 test during the campaign was too small to support evaluation of interaction effects as data were too sparse to populate all interaction group fields for estimation (Table 4).

Table 4 Odds of COVID-19 test among combined analytic sample (n = 29,214)
Full size table

Discussion

The PEM campaign was a rapid-response effort to reduce care-delays among vulnerable and underserved populations during the COVID-19 pandemic. The overarching goal was to engage patients attributed to Atrium Health’s community and rural clinics in calling the ROC-line to learn about the system’s COVID-19 testing and other resources, and to schedule needed health care appointments that included new virtual options. Our strategy aligned with other thought leaders in the research, medical and health equity fields [24,25,26] and applied a cross-department, process using parallel workstreams to reduce bureaucracy and facilitate an immediate response to health care needs of the local community. The campaign evaluation was designed to answer two stakeholder questions (1) “Did this campaign reach vulnerable populations?” and (2) “Did the campaign help patients that were disconnected from the health care system access care?”.

Addressing stakeholder question (1), we found that the majority of patients (95%) received at least one of the campaign messages, with voice messages (89%) being more successful than text messages (70%). The overall delivery was lower for Black (94.1%) and Hispanic/Latino (95.9%) than White patients (97.4%), with a striking difference for text messages, delivered to only 65.5% of Black and 68.8% of Hispanic/Latino patients, compared to 77.5% of White patients. These results provide a few insights for future campaigns. First, since text messages can be delivered only to cell-phone numbers, while voice messages to either a home number or cell-phone number, the voice message delivery success implies that minority patients are less likely to be successfully contacted through a cell-phone. Second, it is unclear from our analysis if this is due to a lack of self-reporting of correct cell-phone numbers or if patients simply do not have a cell phone. As these were not aims of the current evaluation, we can only hypothesize, based on literature and perceptions of the system’s providers that having incorrect or incomplete contact information may be influenced by the following challenges among low-income populations: (i) greater residential mobility [27], (ii) reduced cell phone access and higher frequency of broken, disconnected, and intermittently connected technology [28], lack of trust or desire not to be contacted out of experiences of discrimination [29], or fear of not being able to pay for services [30, 31].

Addressing stakeholder question (2), we found that while prior to the campaign, 0% of the study population used any Atrium Health services (by definition), after the campaign 7.4% called ROC, 12.1% scheduled a health care appointment and 0.8% got tested for COVID-19. Calling the ROC-line provided two types of value. First, the patients were able to have a conversation with a trained representative about COVID-19 and any other health related issues they were experiencing, which likely provided an unmeasured interpersonal support as a compassionate listening ear, regardless of further connections to resources and information [32]. Second, patients were connected to appropriate health care resources based on their needs. We found that patients who called the ROC-line had higher odds of scheduling a health care appointment and getting a COVID-19 test compared to patients who didn’t call. The magnitude of the effect on COVID-19 testing was smaller than on scheduling the appointment, which may be due to the fact that at the time of this campaign testing was subject to strict screening for symptoms and not widely available. Thus, our results imply that the campaign was successful in engaging disconnected patients to call the ROC-line and by connecting patients who called to health care appointments and COVID-19 tests as available. It is important to note that the outcome of COVID-19 testing is likely tied to having symptoms or recent exposure to someone with symptoms, both of which are not included in the models, and patients that are considered at-risk may not venture out to get tested if they don’t have symptoms. The positive associations between calling the ROC-line and health care seeking behaviors can be interpreted as a function of the ROC-line call, but it could also be reflecting self-selection bias among patients that chose to call. If patients more likely to call the ROC-line are also more likely to get tested and schedule an appointment due to some unobserved personal characteristics, such as health problems not captured by EMR or self-efficacy, then the ROC-line effect is overestimated (i.e., someone randomly selected to talk to the ROC would not have such high odds of being tested and scheduling an appointment). But even with presence of selection bias the overall intervention can still be considered successful because it succeeded in encouraging the impactable group of patients to call the ROC-line, get tested and schedule an appointment, thus fulfilling its objective to engage patients.

We found no evidence of difference in the effect of ROC-line call on scheduling a health care appointment by race. However, we did find a difference by insurance status, with commercially insured patients having higher odds of scheduling an appointment than Medicaid-insured, uninsured and Medicare-insured patients who called the ROC-line. This result highlights limitations of outreach campaigns in engaging patients. Campaigns can encourage individuals to seek care, but for the patient to proceed with making the appointment they must be willing and able to pay for the service, as well as non-medical costs of the visit, such as transportation. Patients with commercial insurance are more likely to have higher income and lower out-of-pocket costs than patients without insurance [33] and thus may be more likely to be engaged by similar campaigns.

These results should be interpreted with respect to several limitations. The evaluation was structured using an observational study design without randomization or comparison groups. Thus, results should be interpreted with caution and can only imply associations between variables, and not direct cause and effect relationships. There may also be some systematic differences between those who called and those who didn’t call the ROC-line that are not fully captured by the model covariates. When considering the outcome of scheduling any health care appointment, this analysis did not account for the type of health care appointment, and those who called may have specific health concerns that are not captured by the included covariates. It is also possible that other, simultaneous interventions were taking place at the same time as the campaign that affected the likelihood of receiving a COVID-19 test and scheduling a health care appointment. Other rapidly changing elements at the county-level, including other mass educational efforts and simply the fact that the supply of COVID-19 tests and access to testing locations increased during the campaign, could not be accounted for in this evaluation. Thus, in this report it is not possible to isolate the effect of the PEM intervention from other structural factors or to assess the magnitude of the self-selection bias and thus the size of the ROC effect. Also, the evaluation did not include a patient-level qualitative assessment to inform interpretation of behavioral associations.

Conclusions

The campaign was successful in reaching a majority of patients served by community and rural clinics through voice and text messages. A small but meaningful portion of those patients engaged in care seeking behavior through calling the ROC-line, scheduling health care appointments, and getting tested for COVID-19. Targeted messaging campaigns can be successfully applied as part of a broad, rapid-response, healthcare system strategy for reducing care-delays and racial and ethnic health disparities, but are also impacted by structural and individual-level barriers that are challenging to overcome in a population-focused approach. As new technology solutions for mass communication continue to emerge, healthcare systems should continue to focus on improving the central patient-provider dynamic and encouraging accurate contact information as necessary for the communication of health information. Future iterations of the PEM campaign should consider more robust, mixed-methods evaluation strategies that include a randomized controlled trial design if possible. Additional research is needed to better understand the associations identified in this analysis, and explore the effectiveness of specific campaign elements.