INTRODUCTION

Patient-centeredness is an important characteristic of high-quality medical care1,2,3,4 and is associated with decreased health care utilization,5, 6 improved continuity of care,7 and improved clinical outcomes.8 The delivery of patient-centered care requires engaging patients, their families, and their communities in decision-making across levels of health systems: at the level of direct care, organizational design, policy-making, and research.9, 10 Mechanisms by which patient engagement produces patient-centered care and better health include policy, research, and practice strategies that incorporate patients’ ideas or address their concerns, as well as improved implementation of research findings.11,12,13,14

One approach for engaging patients in health systems, endorsed by several national health care quality agencies, is the formation of patient/family advisory boards or councils (PFACs).15,16,17,18 These are groups of individuals who use services in that health care system (or family members of those that do). They collectively become involved in processes in that system that pertain to education and orientation for staff, ethics, quality improvement, research initiatives, or patient safety.18 Evidence for the effectiveness of PFACs, however, is lacking.19, 20 A 2006 systematic review on mechanisms of patient engagement found little evidence from randomized controlled trials on the impact of PFACs and similar organizations on health care decisions at the population level other than in the development of patient information material, which is more relevant and understandable when PFACs were involved.11 Reviews of the effects of PFACs on ambulatory adult clinics,21 engagement in research only,22,23,24 or the impact of such programs on the people involved25 have also shown that transparency of process, compensation of patients, stakeholder engagement, and contextual factors impact patient engagement, but that gaps in research on impact on health systems persist.

As patient engagement has become more compulsory for reimbursement and accreditation purposes, PFACs are increasingly prevalent.12 Yet, evidence-based approaches to PFACs, such as recruitment methods, organizational makeup, and implementation strategies, have not been systematically aggregated and analyzed. We therefore conducted a systematic review with a primary aim of characterizing the impact of PFACs on health systems. We used the framework of Carman, et al., which defines three levels where patient engagement is needed: direct care, organizational design, and policy-making,10 and due to the growing recognition of the value of community-engaged research, we added a fourth level, engagement in research.26,27,28,29 As earlier systematic reviews have not engaged patients and/or PFAC members in their analysis teams,21,22,23,24,25 a lack of understanding of the feasibility or impact of engaging the public in systematic reviews persists.30 Given this gap, and because multiple stakeholders are often needed for an ecological perspective that attends to a complex problem in health care,31 a secondary aim of this review was to test the feasibility of having a PFAC member participate in the conduct of a systematic review.

METHODS

We followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) standards of quality for reporting systematic reviews.32 We developed a protocol for study eligibility a priori and registered it in the PROSPERO database of systematic reviews (registration number CRD42016052817). Our data synthesis process drew from realist synthesis, an analytic approach driven by realist theory that considers the interaction between context, mechanism, and outcome in evaluating an intervention.33, 34 This study was not considered human subjects research by the Yale School of Medicine Human Investigation Committee.

Analysis Team

We crafted a research team of diverse stakeholders because our topic involves collaborations among patients, families, community members, clinicians, and policymakers, and due to recent calls to involve the public in systematic review processes.35,36,37 Our team included clinician-investigators who conduct community-engaged research (BO, MR),31 a clinician-administrator who organized a PFAC in an academic clinic (IG), a community research liaison for a research fellowship (AG), a medical librarian with experience in systematic reviews (JG), a patient with leadership roles on several PFACs in an academic hospital (MEP), and a post-baccalaureate student (MH).

Study Selection

We defined our intervention of interest using the Agency for Healthcare Research and Quality’s classification of a PFAC: a group that engages “patients and families directly into the planning, delivery, and evaluation of care.”38 We limited our search to those that involved an organization of patients, family members, or community members (possibly including clinicians or hospital staff) that advised a health care system on the level of direct care, organizational design and quality improvement, policy-making, or research, but did not require that the term PFAC be used (eTable 1 in Supplement).10 Because we anticipated heterogeneity of outcomes, we did not limit our search to particular outcome measures. We sought studies that included a comparison group, including a randomized control, a parallel cohort, or a historical control (i.e., before-and-after evaluations).

Data Sources and Searches

We searched MEDLINE (Ovid MEDLINE 1946 to September Week 3 2016), Embase (Ovid Embase 1974 to 2016 September 30), PsycINFO (Ovid PsycINFO 1967 to September Week 4 2016), CINAHL (Ebsco), Scopus (Elsevier), and Social Science Citation Index (Web of Science, Thomson Reuters). We identified additional studies by scanning other systematic reviews. We also searched the websites of the following pre-identified organizations for appropriate studies: Institute for Patient and Family-Centered Care, RAND, Patient-Centered Outcomes Research Institute, Agency for Healthcare Research and Quality, and the Robert Wood Johnson Foundation. We conducted all searches on October 3, 2016.

To produce relevant controlled vocabulary and keyword terms, we analyzed 12 previously identified key articles using the Yale MeSH Analyzer (http://mesh.med.yale.edu/). In each database, we ran scoping searches and used an iterative process to translate and refine the search strategies. We used the previously identified articles to validate the success of our searches (eSearch 1 in Supplement). We limited our search to English-language articles but not to any time period.

Data Screening and Abstraction

Two authors independently screened titles and abstracts, and conflicts were resolved by consensus. For each screened article, two authors independently abstracted information about the context, intervention, and outcomes into a standardized form. If desired information was not published, we contacted the first author of the article by email to inquire. To pool the varied expertise on our team, at least two team members read all studies, one of whom had health services research experience. Disagreements were resolved by consensus with the input of the first author.

Intervention and outcome heterogeneity precluded meta-analyses. Instead, we drew from a realist synthesis strategy in which reviewers delineated the contextual influences (C) that are hypothesized to have triggered the relevant mechanisms (R) to generate the outcomes (O) of interest.34, 39 We arrived at consensus for C-R-O sequences during iterative consultations among the research team, with monthly meetings between the lead investigator (BO) and the PFAC member (MEP), and semi-monthly meetings among the entire team from December 2016 through December 2017. We consulted the PFAC of Yale-New Haven Hospital twice (December 2016 and January 2018) for in-person feedback on study design, terminology for search strategies, and interpretation of the three realist synthesis elements.

Methodological Quality Assessment

Two reviewers independently completed the quality assessment of each study using the Cochrane Risk of Bias Tool for randomized trials40 and the Newcastle-Ottawa Scales for observational studies.41

PFAC Classification

We classified PFACs as those that either informed direct care, organizational design, policy-making, or research, and assessed the outcomes at each of these levels in the health care system separately. We adapted the classification of Carman, et al.,10 adding engagement in research because of growing recognition of the value of community-engaged research.26,27,28,29 As themes emerged during the abstraction process, we considered alternative ways to classify programs, but found these categories were the most effective and consistent with frameworks that ground other studies.42, 43

RESULTS

Our search yielded 4389 articles, with 3718 remaining after removal of duplicates. We screened 940 abstracts and identified 18 articles that met criteria for inclusion (Fig. 1). On two occasions, we identified two articles that present data on the same study44,45,46,47, for a total of 16 unique studies (Table 1); these included 13 interventional studies (Table 2)44,45,46,47,48,49,50,51,52,53,54,55,56,57,58 and three observational studies (Table 3).59,60,61 Studies addressed the four levels of the health care system and engaged patients (n = 7), community members (n = 8), and family members (n = 1). They employed heterogeneous terminology to describe organizations and their components: “patient,” “consumer,” or “community” to describe those engaged in organizations called “boards,” “councils,” “committees,” “patient groups,” or “voting panels.” For interventional studies, recruitment to PFACs or similar organizations occurred by nomination (n = 9) or recruitment was not described (n = 4); none involved election processes.

Figure 1
figure 1

PRISMA flow diagram of screened articles.

Full size image
Table 1 Characteristics of the 16 Reviewed Studies
Full size table
Table 2 Organizational Type, Systems Addressed, and Key Findings from Interventional Studies (n = 13)
Full size table
Table 3 Health Systems Studied, Measures of Participation, and Key Findings from Cross-sectional Studies (n = 3)
Full size table

Risk of bias among randomized trials was moderate to high (Table 4) and observational studies were of poor or fair quality (Table 5). For randomized trials, the most common sources of biases were that participants and study personnel were not blinded nor was outcome assessment blinded (four of five studies). For observational trials, the most common reasons for low-quality assessment were a low-quality selection of the non-exposed cohort (10 of 11 studies) and limited demonstration that the outcome of interest was not present at the start of the study (11 of 11 studies).

Table 4 Risk of Bias for Randomized Studies (Cochrane Risk of Bias Tool)
Full size table
Table 5 Quality Assessment for Observational Studies (Newcastle-Ottawa Scale)
Full size table

Organizational Makeup, Recruitment Strategies, and Levels of the Health Care System Addressed

Engagement took the form of groups of patients, family members, or community members only44, 50, 52, 56, 57; or groups that also involved staff or clinicians53, 55, 58; in four studies, these two strategies were compared.46,47,48,49, 51 Two studies engaged groups that speak non-dominant languages: one involved indigenous communities in India44 and one involved Spanish-speaking immigrants in the USA.58

PFACs consisted of collective engagement of patients or community members and in one case focused on engaging family members (parents of pediatric patients).58 Recruitment of individuals tended to occur through nomination, such as asking for representatives from local organizations. No studies used democratic election for recruiting, and five studies did not report on recruitment strategies.51,52,53, 59, 60

Engagement that Informs Direct Care Practices

Three studies evaluated interventions that inform direct care practices.44, 49, 58 These included two in which patient or parent groups developed their own care strategies or educational materials44, 58 and one in which patient groups made recommendations for the development of disease-specific practice guidelines.49 These were considered effective strategies that enhanced the care experience58 or equitably improved clinical outcomes by having greater effect in more socioeconomically marginalized communities.44 The patient-derived clinical practice guidelines were similar to those derived by expert groups, with differences predominately due to patients having a different conceptualization of the balance between risks and benefits than physicians.49

Engagement that Informs Organizational Design

Three studies evaluated interventions that inform organizational design of health systems.53, 56, 58 They targeted organizations that are disease-specific (diabetes centers)53 or population-specific (pediatrics clinic for families with limited English proficiency)58 as well as primary care clinics.56 All studies used multiple forms of data collection, including interviews, surveys, and participant observations, to triangulate perceptions and project-based outcomes. When comparing strategies for engaging community members, the Community Action on Health committee in the UK found greater satisfaction among participants and greater effectiveness of deliberation when meetings were held in community settings and facilitated by existing community organizations than when they occurred on the grounds of clinics or health systems.56

Engagement that Informs Policy-Making

Five studies (six articles) evaluated interventions that inform policy-making.46,47,48,49, 53, 54 Three studies in this group compared the recommendations made by organizations that contained patients with those that contained only “experts,” such as clinicians or organizational leadership.46,47,48,49 Those that contained patients produced similar recommendations to those containing clinicians and organizational leadership only,49 but the process of producing recommendations was costlier and achieving consensus took longer when patients were involved.46, 47 Also, when patients were involved, policy priorities and plans were deemed to be of lower quality when scored by content experts.48 Two studies in this group compared mechanisms of engaging patients and community members, and found that in-person, collective deliberation is more effective than using mailed surveys or telephone calls,54 and that patients with greater community credibility (such as leaders in community organizations) were more effective participants in PFACs than those without.46, 47

Engagement that Informs Health-Related Research

Five studies evaluated interventions that inform health-related research.50,51,52, 55, 57 In four of these studies, patient groups assisted in the recruitment and retention of research participants among marginalized populations, and produced greater recruitment numbers and retention.50, 52, 55, 57 Only one study suggested that these individuals provided guidance on other dimensions of research besides participant recruitment and retention, such as study design or dissemination of findings, but the nature of involvement was not elaborated upon.51

Cross-sectional Surveys of Patient, Family, and Community Engagement Across Systems

Three cross-sectional studies examined engagement across systems (Table 3).59,60,61 All three used different measurement tools for assessing levels of engagement, only one60 of which tools had been validated in previous research.62 Higher levels of engagement were associated both with strong patient leadership and with patient deference to providers.61 Engagement more often occurred at the hospital level than at departmental levels, and more often focused on implementation of quality improvement projects than project design.60

Feasibility of PFAC Member Engagement in the Review Process

Concerning our secondary aim of assessing the feasibility of engaging a PFAC member in the review process, the PFAC member contributed to the study’s search terms (as she had networked with PFACs at other institutions that use other terminology, such as “voting panel” or “community advisory board”) and to the interpretation of findings, particularly gaps in the studies they found to be relevant yet missing, such as descriptions of recruitment and training processes for PFAC members. The PFAC member did not have formal research experience and was unable to perform certain tasks required of the systematic review: identification of study design or comparison groups and assessments of bias or quality of studies. Interpreting the abstracted data in a realist synthesis framework—delineating context, mechanisms, and outcomes of individual studies—was feasible for the PFAC member and formed the basis our approach in analysis team meetings.

DISCUSSION

We identified 16 studies that addressed engaging patients, family members, and community members in various levels of the health care system, including health-related research. In-person deliberation with health system leadership was most effective, and studies that involved patient engagement in research focused on increasing study participation but not in research design or the dissemination of research findings. Programs varied in their structural makeup, the terminology employed to describe organizations and their components, strategies of recruitment of individuals, and types of outcomes measured. All studies suggested important benefits of community engagement to health systems and research, but some described drawbacks, including greater costs, longer time to achieve consensus among groups, and the generation of lower quality plans when patients are involved.46,47,48

While there has been increasing interest in patient engagement1, 18, 38, 63, 64 and our search strategy captured 3718 documents, only 16 met our inclusion or study design criteria. Most were limited by study design. We excluded many studies that described programs but did not evaluate them with a control group, including a historical control, but instead narrated the formation of a project and summarized the lessons learned. The preponderance of articles structured this way represents a challenge in measuring the impact of patient engagement in projects that may not be well defined at the initiation of a partnership between patients and clinicians, staff, or researchers. Experts in community-engaged research have proposed various mechanisms for evaluating these sorts of partnerships, which highlight qualitative methods,31 “contextual” variables that describe historical context or the cohesion among community members,24, 65 or the realist synthesis framework we employed.33, 34, 39

This review builds on the work of previous reviews11,12,13, 23,24,25, 66,67,68 by including studies across various levels of health care systems, focusing on the organizational makeup of PFACs and similar organizations, assessing standardization in measurement tools of patient engagement, and describing recruitment strategies. We found that studies that compared different mechanisms of patient and community engagement suggested three guiding principles. First, in-person and collective deliberation, particularly in community-based settings (such as community organizations), is more effective than using mailed surveys or telephone calls, or one-on-one meetings.54, 56 Second, patients with greater community credibility, such as leaders of community organizations, are more effective participants in PFACs than those without.46, 47 And third, while organizational or policy recommendations made by groups that included patients were similar to those made by experts,49 this process takes more time and resources46, 47 and may produce plans of lower quality.48

The heterogeneity of outcomes and recruitment processes described in our studies of interest, as well as the three guiding principles, suggest three opportunities for improvement in the conduct and evaluation of PFACs and related organizations. First, we identify a need for standardized measurement tools for patient, family, or community engagement, such as the patient-centeredness tool used by Groene, et al., the only tool we identified that is used in multiple studies and that is validated.60, 62 Second, the process of recruiting members to these organizations should be transparent and evaluated. Third, because community members who already had leadership experience tended to be more effective PFAC members, trainings should be available for those members without such experience to allow their voices to be raised effectively on PFACs. Specifically, those who experience health care inequities, including those from traditionally marginalized groups, may offer insights in PFACs that others do not.18, 58, 69 Because priorities and plans developed by patient-containing groups can be of poorer quality and may take more time, PFACs should evaluate and refine their processes of training new members, particularly those with little organizational leadership experience. Programs for training members of the public to effectively participate in other civic organizations and democratic processes, such as “Democracy Schools” to prepare citizens for participation in city governments, could serve as models.70

Engaging those who are not trained in medicine or research, such as PFAC members, in systematic reviews is feasible and produces important dividends to project design (in the case of this project, search terminology) and to the realist synthesis of abstracted information (in this project, identifying gaps in the literature regarding recruitment and training of PFAC members). Manualized strategies for involvement of stakeholders such as PFAC members in systematic reviews are needed, particularly for the assessment of studies’ risk of bias or quality. This warrants further exploration as leaders in this methodology such as the Cochrane Collaboration have considered patient involvement as an integral organizational attribute since inception35 and yet there are few examples of active patient involvement in the review process.71 An ongoing systematic review is aiming to further characterize optimal strategies for involving the public and other stakeholders in the conduct of systematic reviews.30

Our systematic review has several limitations. Abstracted data from observational studies, which made up the bulk of our screened studies, may lead to conclusions that would differ from data obtained via randomized studies.72 The bias and low quality of the studies selected decrease the generalizability of the conclusions of each and heterogeneity of outcomes prohibits meta-analyses. The exclusion of many appropriate interventions due to weak study design might have excluded those interventions that engaged patients and community -members in ways that made the development of a comparison group difficult or impossible. Our search required an iterative process of identifying articles of interest, analyzing their content and indexing, and building a strategy constituted a process by which important studies may have not been included. Finally, we captured studies across four decades and several continents, and such variability in time and space complicates the ability of our findings to inform practice in any one health system.

Despite these limitations, we have identified three guiding principles and opportunities for improvement for the conduct of and evaluation of mechanisms of collective patient engagement. In performing this review, which engaged multiple stakeholders in local PFACs, we demonstrated that individuals with little to no research experience, but with important perspectives on the topic, can be successfully engaged in performing a systematic review on a complex intervention when using a realist synthesis to analyze abstracted data.36, 37 More data and metrics are needed to guide the development of PFACs, particularly focused on standardizing evaluation measures as well as evaluating training and recruitment strategies. Future systematic reviews on patient engagement or patient-centered outcomes should involve patients, family members, and/or community members in the analysis processes.