INTRODUCTION

Patients experiencing disadvantage, such as the socioeconomically disadvantaged, seriously ill, racial/ethnic minorities, and those with limited English proficiency, are historically underrepresented in research, especially concerning potentially sensitive topics such as advance care planning (ACP).1,2 The ability to generalize research findings, particularly to underrepresented subgroups, is limited when participants in a sample do not reflect the target population.3 These knowledge gaps hinder efforts to address health disparities that persist for underrepresented patients.4,5 Despite mandates to include historically underrepresented populations, it remains a challenge for researchers to recruit representative samples, particularly for large pragmatic trials.6 Documenting techniques that improve the recruitment of underrepresented patients is needed.

Many clinical research studies do not recruit their target sample size.7 To improve recruitment and representation, a variety of techniques can be used. Mailing, phone calls, media marketing, and community outreach enhance research participation and sample diversity with varying degrees of success.8 In clinical trials and population-based survey studies, mail and phone reminders increase study participation, with phone reminders being more effective.9,10 However, it is unknown whether phone outreach enhances the recruitment of underrepresented patients, particularly for large pragmatic trials and research on sensitive topics, such as ACP.

Therefore, we assessed the recruitment efforts of seriously ill primary care patients for a population-based ACP pragmatic trial at three academic health systems in California.11 Because underrepresented groups are less likely to engage in ACP, it was critical to enroll and survey a diverse sample of individuals that included historically underrepresented patients.12 We examined the effect of phone outreach in addition to standard mail recruitment on the inclusion of underrepresented patients in the research sample and whether it better reflected the diversity of the baseline patient population of seriously ill patients compared to mail alone.

METHODS

We analyzed response rates to a survey distributed to all seriously ill primary care patients at three University of California health systems and the characteristics of respondents. As part of a pragmatic ACP trial, ACP mailings and electronic health record (EHR) messages were sent out to every patient 18 years and older identified as having serious illness based on a medical record–validated EHR algorithm and lacking an advance directive or Physician Orders for Life Sustaining Treatment (POLST) documented in the EHR within the past 3 years.11 In addition to the population-based rollout of the ACP intervention, the project aimed to survey a representative sample of this seriously ill population to understand attitudes toward and readiness to participate in ACP, care preferences, reports about communication by providers, and health status. This study was approved by the Institutional Review Board at UCLA (18-001612) with delegation from the other two health systems.

English- and Spanish-language-preference patients were mailed a survey in their preferred language. Across the three health systems, surveys were mailed in the latter half of 2019 and beginning of 2020. Patients who did not respond to the mailed survey were called by staff up to three times to remind them of the mailed survey or to help complete the survey over the phone. Patients were called in order of ascending medical record number at each site. Because the number of patients not returning surveys exceeded the available research assistant time for phone calls, not every patient received a phone call. After recruitment was completed, we examined the demographic characteristics and social vulnerability of patients who were recruited with and without phone outreach.

Research staff recorded the dates and outcomes of survey mailing and phone outreach. They also recorded dates and reasons for patient refusal, and dates and methods of survey completion. We developed computer code to identify the timing of logged phone calls across the three sites and merged these with survey outcome data.

Patient demographic characteristics for the baseline seriously ill primary care population and demographics of the individuals recruited by mail alone and by phone were obtained from the EHR at each health system. These included gender, race, ethnicity, preferred language, home street address, and date of birth. Age was calculated at the date that the survey was mailed to the patient.

We measured social vulnerability using the Social Vulnerability Index (SVI), which is based on 15 US Census variables, such as income, educational level, employment, crowding, and vehicle access.13,14 SVI is an indicator of potential negative effects from external stress on health. An SVI score is attributed to each census tract ranging from 0 to 1, with 1 being the most vulnerable. Patient home addresses in our population were geocoded using ArcGIS Pro to obtain geographical coordinates. The coordinates were then binned into census tracts, which allowed SVI scores to be matched to each patient.

Statistical Analysis

Patients were categorized as recruited by mail alone if they completed the survey without the need for a call. Patients were categorized as recruited after phone outreach if a call was made before the survey was completed. Phone calls were considered valid if the call resulted in either verbal contact with the patient or a voicemail message.

We compared demographic characteristics, including SVI, between patients recruited after mail alone and after phone outreach using chi-square tests for categorical variables and t tests for continuous variables. We coded racial/ethnic minority as Hispanic/Latinx, Black/African American, Native Hawaiian/other Pacific Islander, American Indian/Alaska Native, and other race/ethnicity.

The outcomes of phone calls where verbal contact was made with patients were displayed graphically by time and day of week. Phone calls resulting in voicemails, busy lines, or other non-contact outcomes were not included in the graphical analysis. Verbal contact outcomes were categorized into “completed survey” if the patient completed the survey either during the phone call with research staff or by mail after the phone call; “opted out” if patients actively opted out of the survey during the phone call with staff or by leaving a voicemail after the phone call; and “other (no survey)” if patients were contacted by phone, agreed to consider completing the survey, but never completed the survey.

We used descriptive statistics, and associations between participation and demographic variables were assessed with chi-square and t tests. Analyses were performed using Python version 3.9.1.

RESULTS

Across the three academic health systems, 5998 seriously ill patients were mailed surveys. The mean age of the population of seriously ill patients eligible for survey was 71 years and mean SVI was 0.38. The patients were 52% male, 85% English speaking, 53% White, 19% Hispanic/Latinx, 11% Asian/Pacific Islander, 9% Black/African American, and 9% other race/ethnicity. We obtained completed surveys from 1215 patients (20% response rate). The mean age of survey respondents was 69 years and mean SVI was 0.37. Responding patients were 52% male, 88% English speaking, 61% White, 18% Hispanic/Latinx, 9% Asian/Pacific Islander, 7% Black/African American, and 5% other race/ethnicity (Table 1).

Table 1 Comparison of Characteristics of Patients Who Completed Surveys After Mail Alone Vs After Mail and Then Phone Outreach, and Description of Patients in Resultant Full Survey Sample and Total Population of Seriously Ill Patients
Full size table

Of the 1215 seriously ill patients who completed the survey, 787 (13%) did so after mail recruitment alone and 428 (7%) after phone outreach (representing 35% of the total surveys completed; see Fig. 1). Among the 5096 patients who had not completed a survey by mail alone or opted out by mail, 3948 (77%) received a first phone call from staff and 318 additional surveys were obtained. Of the 2949 patients who had not responded to the first phone call or opted out, 1214 (41%) received a second phone call and 66 additional surveys were obtained. Of the 1017 patients who had not responded to the second call or opted out, 624 (61%) received a third phone call and 44 additional surveys were obtained.

Fig. 1
figure 1

Sankey diagram showing proportion of survey completion, phone outreach, and opt outs among the 5998 seriously ill patients who were mailed surveys.

Full size image

Compared to patients recruited by mail alone, patients recruited only after phone outreach were more socially vulnerable (SVI 0.41 phone v 0.35 mail alone, P < 0.001), and included a greater proportion of non-White, non-Asian racial/ethnic minorities (35% v 28%, χ2 (1 df) = 7.567, P = 0.006) and those with a non-English language preference (16% v 10%, χ2 (1 df) = 8.343, P = 0.005) (Table 1). There was no statistically significant difference in age or gender.

The inclusion of patients recruited after phone outreach resulted in a sample that better represented the seriously ill survey eligible population in terms of non-White, non-Asian racial/ethnic minorities (28% mail alone, 30% including phone outreach, 36% baseline population), non-English language (10%, 12%, 15%, respectively), and SVI (0.35, 0.37, 0.38, respectively) (Table 1).

In terms of timing and efficacy, phone calls completed between 8 a.m. and 10 a.m., and 1 p.m. and 4 p.m., reached more patients; however, survey completion rates were similar throughout the day. Opt outs appear to be higher for phone calls placed between 7 p.m. and 9 p.m. (Fig. 2). Phone calls were made by research staff between 8 a.m. and 8 p.m. on Monday through Friday.

Fig. 2
figure 2

Upper plots show responses to phone calls by time of day and day of week. Note that only telephone calls resulting in verbal contact with the patient are shown. Lower plots show the frequency of all phone calls by time of day and day of week.

Full size image

DISCUSSION

Recruiting a sample representative of the baseline clinical population is important for the generalizability of research findings. In this multisite pragmatic trial of ACP, we evaluated the effectiveness of our phone outreach to capture a representative research sample. We found that phone outreach after a survey mailing increased participation by more socially vulnerable, racial/ethnic minority, and non-English speaking seriously ill patients. The inclusion of these patients resulted in a research cohort that more closely represented the baseline clinical population of seriously ill primary care patients.

As shown in previous research, survey response rates can vary significantly between demographic groups.15 Digital media, such as phone calls, text messages, and email, have been used successfully in the recruitment of underrepresented older adults.16,17 Our findings on the effectiveness of phone outreach align with these results. This is a critical step in carrying out population-based research on vulnerable patients such as those who have serious illness. Our findings show that phone outreach not only increases recruitment and enhances diversity, but also can be used for studies that focus on potentially sensitive topics (e.g., ACP). This has implications for the allocation of resources in population-based survey efforts: resources for outreach should be prioritized in order to enhance respondent representativeness. This study also demonstrated that exploring comparisons between enrolled research patients and the baseline clinical population is feasible. Using geocoded data can enhance understanding about whether outreach affects the representativeness of the sample.

In terms of timing of calls, it appears that patients in this analysis were more responsive to phone outreach in the early morning and early afternoon. This could be an area for further research with a more robust prospective data collection process than what was used in this study. In addition, more information would be needed to understand whether the patients in the cohort are still working outside the house and/or have caregiving duties during the day or after hours. We were unable to conduct calls on weekends, which may enhance recruitment or increase the number of opt outs. Our data demonstrate that there appears to be a “bewitching” hour at 7 p.m. as calls made after that time resulted in higher refusal rates.

This study has several limitations. The project was conducted in three academic health systems in California, and these findings may not apply to community or rural health systems. The survey topic of ACP is potentially sensitive, especially for seriously ill patients, and our findings may not apply to research in other areas. The population of patients eligible for the survey was composed of seriously ill patients without ACP documentation in the EHR and as such was more vulnerable (older; more likely to be Black/African American, Hispanic/Latinx or other; and higher SVI) than the seriously ill patients who had completed ACP documentation. In addition, there were limitations in phone outreach to our large sample. Because the number of patients not returning surveys exceeded available research assistant time, not every patient received phone call follow-up. Follow-up of additional patients likely would have yielded a higher survey completion rate. This study was not designed to capture the cost of research assistant phone call follow-up. The cost-effectiveness of phone calls versus other mechanisms of improving response, and other methods of enhancing generalizability, such as employing survey non-response weights, should be evaluated in future studies.

In a pragmatic trial concerning the potentially sensitive topic of ACP, we found that phone outreach significantly enhanced the recruitment of underrepresented seriously ill primary care patients. Identifying additional techniques to improve underrepresented patient recruitment is paramount to ensuring inclusion and representativeness and decreasing health disparities.