Definition of evidence

In this study, we understand evidence as “robust information, data, and research” and use these terms synonymously with “evidence” in reporting and discussing the studies’ findings. Based on existing research [34], we applied a more extended and illustrative definition for the capacity assessment:

Systematically and transparently conducted and reported analyses. These may originate from academic literature, monographs, books, or gray literature, and include internal studies and evaluations. In this sense, robust information, data, and research is not limited to the work of academics from universities but may include findings/studies from other research organizations, e.g., independent research institutes, competence centers, and evaluation and consulting firms.

Study design

This mixed-method interview study investigated policymakers in German-speaking Swiss cantonal health administrations. Here, policymakers are civil servants (we use the two terms interchangeably) working either as secretary-general or person in charge of HSMP. The sample was limited to German-speaking cantons to ensure that the first researcher to conduct the interviews was able to fully apprehend the opinions and experiences expressed by respondents, both culturally and linguistically.

We quantitatively assessed the organization-level capacity for EIPM by interviewing health administrations’ secretaries-general, applying a measurement tool. To embed these findings in the practical context and investigate individual-level needs and perspectives regarding EIPM, we purposefully selected civil servants responsible for HSMP for in-depth interviews that were subjected to qualitative analysis. Participants were recruited and data collected between October 2020 and May 2021.

Study setting

The supreme governing body of each health administration, politically and organizationally, is the executive, i.e., one of the five to seven members of the cantonal executive council (the government), a politician elected by citizens. The organization of the administrations differs among cantons. While some executive councils head distinct health departments, others head departments responsible for multiple areas, such as health and social affairs, with a specific unit or section dedicated to health. General secretariats are the staff units and support the executive council in political and operational management of their (health) administration. They are the central interface between politics and administration and are often responsible for areas such as finance and controlling, legal services, human resources, communication, or IT. While every administration has one secretary-general, the number of civil servants concerned with HSMP varies considerably, depending on the canton size, and ranges from a handful to several dozen civil servants. The population of Swiss cantons is between 16,000 and 1.5 million.

Quantitative capacity assessment

Qualitative in-depth interviews

Participant selection and data collection.

We purposefully sampled civil servants from the higher hierarchical level responsible for HSMP from all German-speaking cantons (n = 21). We have not selected secretaries-general for the in-depth interviews because their scope of work is not limited exclusively to the health sector (e.g., for departments of such as “health and social affairs”) and because their overarching function places them too far away from HSMP practices and decisions.

Interview candidates were identified through health administration websites and ORACLe interviewees’ (i.e., secretaries-general) nominations. A summary of interview topics was provided on request. All interviews started with the provision of key definitions (i.e., evidence, policymaking), followed by the questions of the semi-structured interview guide, and ended with capturing sociodemographic data.

Face-to-face or video interviews were conducted in Swiss German, audio was recorded in both cases, and notes were taken. Due to the exploratory character, we did not strive for complete thematic saturation [38] but expected to reach this point between ten and fifteen interviews, as the target audience was judged relatively homogeneous [39].

In total, we performed 12 interviews with civil servants from 10 cantons (see Table 1) responsible for HSMP, with one additional respondent working in the area of prevention (length range: 40–80 min, mean length: 55 min). Civil servants who were not available for interview participation stated the high workload due to the COVID-19 pandemic as their reason not to participate.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Table 1. Qualitative in-depth interview participant characteristics.

https://doi.org/10.1371/journal.pone.0302864.t001

Ethics

This study underwent an ethical review and clearance (Req-2018-00460, response from July 5, 2018) with the Ethics Commission of Northwestern and Central Switzerland. Participants were informed about the study’s purpose and objectives. Written informed consent for study participation and verbal agreement to interview recording were obtained before conducting the interviews. The first author was able to identify individual participants during and after data collection. Interview recordings and transcripts were stored on a password-protected computer and server. The data will be retained for five years after publication of the study results.

Moderate evidence-use capacity in health administrations

The overall capacity (scale: 0−9), as assessed by the ORACLe measurement tool, was moderate and similar among the administrations. The mean overall capacity in numbers was 5.1 (range: 4.4−5.8), whereas zero means no and nine means high capacity. For a visual representation, see S1 Fig. Calculation of the unweighted total (i.e., the sum of the domain scores, making no assumptions about the relative importance of domains) resulted in a similar pattern, though with a slightly different rank order. Details are presented in S2 Fig.

Regarding the administrations’ capacity on the level of the seven ORACLe domains, administrations expressed a similar pattern with high and low capacity, generally with considerable variation among domains within an administration (Fig 1). Most administrations had the highest capacity scores for their research generation activity (e.g., regarding health services demands and prognoses) or their relationships with researchers and academic partners. Secretaries-general typically reported limited capacity in the documentation of processes, availability of tools and programs for leaders, and staff support with training. As such, administrations scored lowest in domains two to four, which were weighted strongest in calculating the overall capacity score [35]. The following paragraphs inspect the administrations’ domain capacity in more detail.

Download:

• PPT
  PowerPoint slide
• PNG
  larger image
• TIFF
  original image

Fig 1. Domain capacity scores by health administration.

https://doi.org/10.1371/journal.pone.0302864.g001

Administrations indicated uniformly low capacity regarding the documentation of processes that mandate or encourage evidence-use to develop policies (domain one). Documentation was generally limited to formal aspects of politico-administrative processes, such as legislative procedures. There were no examples of written guidance to develop policy or program content. Most explicit requirements were described as being ad hoc on a case-to-case basis. At the same time, a larger scope of the policy or program (i.e., its costs, duration, and extent) increased the relevancy of consulting and demonstrating evidence use, often through the involvement of external partners such as academic institutions or consultancy agencies.

Administrations also exhibited little capacity concerning tools and programs that assist leaders in supporting evidence use (domain two). In most administrations, there were no specific programs for leaders to enhance confidence or expertise in evidence use, and neither job descriptions of collaborators nor performance evaluations/targets entailed expertise in evidence use.

In contrast to the first two domains, capacity in programs enabling staff to learn and maintain evidence-use skills (domain three) varied more strongly among the respondents. Three administrations showed moderate capacity scores, two minimal, and one no capacity at all. Administrations generally provided regular access to training opportunities, and applying evidence to policymaking was sometimes part of the offering. Participation in such training was frequently not considered in staff performance management. Administrations generally reported continuing education and training to be expected but rely mainly on the staff’s initiative.

Administrations showed a similarly low capacity for systems and tools that support staff using evidence (domain four). All but one administration reported regular internal events for the dissemination of evidence. No resources were available that guided access to, appraisal of, or application of evidence. Most administrations indicated having staff expertise in evidence use. While the expertise was required or assumed for certain functions, it was not tied to a specific role. In general, this expertise was described as being accessible to other employees. Secretaries-general also stated that administrations had access to most or all relevant academic journals. They also indicated that needs and, thus, access differed significantly among individuals and different fields of work. On the other hand, databases with research literature were available only in some administrations. About half the administrations had an easily accessible and cataloged library, and none systematically provided licenses for literature-management software. There were no established methods to commission literature reviews or knowledge-management systems specifically applied for evidence management.

Secretaries-general indicated the highest capacity scores for efforts to generate new evidence (domain five). While four of six administrations showed moderate to high capacity, two indicated none. Most research projects were carried out with the involvement of external partners. Half the administrations reported recently having commissioned one or more research projects.

Capacity regarding processes for evidence-informed policy evaluations (domain six) was uniformly low among all administrations. Although evaluations for measures of certain scope were expected and carried out regularly or frequently, a requirement for undertaking such evaluations was not explicitly documented.

All administrations showed moderate to high capacity regarding their relationships with researchers and research organizations (domain seven). Administration staff generally participated in research fora or conferences, and most administrations had several formal and informal relationships with academic organizations. There were no administration-staff appointments from research organizations in the area of health services. However, there was regular consultation and involvement of external experts in administrations’ work, often service providers, such as physicians, with double affiliations to practice and academia.

Potential for increased organization-level, structural capacity in health administrations

When asked about the evidence base for HSMP, civil servants frequently criticized data and information gaps in some areas of work along with challenges in accessing data (“we really have to scrape it together”—J1; “there we are groping around completely in the dark”—A1). While the availability of inpatient care data is typically considered satisfactory, the “black box” of outpatient data presents a challenge due to poor comprehensiveness and quality (F1), making it intricate to assess important trends, such as the shift of inpatient care services to the outpatient sector. Similarly, outcomes data related to the quality of care would be necessary for civil servants’ work in governing health care but hardly exist. Individual voices criticized “statistical silos” and emphasized the potential value of better linking data collected through different systems—for example, health and social care data (D1).

There is not only a lack of data but also of personnel, expertise, and time to analyze available evidence. Due to limited resources, civil servants felt they did not have enough time to deal with specific problems in greater depth or study the academic literature.

Lean management means efficient reduction [of activities] to the core business […] not prospective planning and projects … that is a bonus. So, the more you are under pressure with resources, the more you have to reduce [the time you devote] to the day-to-day business. And I think that’s not just in our office, but that’s the situation in the administration in general. D1

In this context of scarce resources, civil servants are forced to use evidence pragmatically. Data and information are sought and processed when they are available and “accessible at a reasonable cost,” and their contribution is judged to be meaningful (H1). Consequentially, HSMP decisions rely on minimal information and intuition. Where the administration can manage a task effectively, there is little incentive from an organization-level perspective to examine the validity of underlying data more closely.

Apart from the lack of general resources, the pragmatic engagement with evidence is a consequence of the low institutionalized EIPM-specific support and guidance in the administration. Despite leadership typically supporting EIPM, administrations place the focus and responsibility for engagement with evidence at the level of individual civil servants that “should be able to work as freely as possible” (H1). Thus, administrations show limitations in structural capacity.

While the administrations’ prevailing informal culture towards EIPM is considered relevant for individual civil servants’ evidence-use behavior, this behavior is perceived as being influenceable only to a limited extent, as one secretary-general outlines:

People just have different ways of doing things. Some find it exciting and like to read such documents; others do it less [gladly]. You can’t enforce this very well. You can’t tell someone that they have to read three academic publications on a certain topic every month. You can’t do that at this level. It’s more about having that culture and also keeping the curiosity to know what’s happening in the specialty and what exciting things have come out of the research. A-SG

When asked about their needs for the administration’s support in evidence management, the civil servants provided few specifics, indicating that there is no conception of potential targeted EIPM support measures. Apart from the desire for research databases access in two administrations, civil servants said that they require resources such as time and additional staff, more orientation to and overviews of existing data, health service statistics, academic publications, and health system guidelines.

The administrations depend on external resources to compensate for their limited internal capacity to produce and engage with evidence. An essential such resource for health care planning, especially for medium-sized and smaller cantons, is the government-supported competence center Swiss Health Observatory (OBSAN; https://www.obsan.admin.ch/en). The OBSAN offers analytical expertise, provides access to otherwise difficult-to-access data, and produces valuable evidence for policymaking. It serves, to some extent, as a knowledge broker (D1) that “can reconcile the balancing act between” politico-administrative needs and scientific demands (H2) and helps administrations that are “overwhelmed” (H1) with data use in dealing with their limited internal capacity.

As also noted in the capacity assessment, the in-depth interviews confirmed collaboration with research organizations and research-related consulting firms to be relevant to the administration. Nevertheless, such collaboration is sporadic, with little direct contact overall. While some civil servants emphasized that research and evaluation assignments are “horribly expensive” (F1), others regretted that “you just can’t spend money if you don’t have time” (A1) to manage an evaluation mandate.

Besides that, health care providers—first and foremost, hospitals—are vital to the administration in building a “bridge between practice, research, and health authorities,” providing access to evidence and supporting the administration in its appraisal (A-GS). These “experts who are on the front lines” (G1) are essential partners in aligning, planning, and implementing health care policies. This relationship constitutes a knowledge asymmetry in favor of the service providers and is thus not without problems. Especially since service providers do not primarily think in terms of a networked and integrated “health care logic for the population” (A1). An interviewee from a large canton expressed the hypothesis that in the administration’s efforts to manage the service providers better, the need for evidence in the administration has increased (G1).

Evidence is strongly valued for health service management and planning

Interviews have demonstrated that evidence gives civil servants confidence in and orientation for actions, helps them understand the current healthcare situation, identifies areas in which action is needed, allows forecasting of future needs, and drives planning. Overall, evidence is becoming increasingly crucial to the administration’s work to shape health care. External demands, such as those from the government, parliament, and citizens, require the administration to employ evidence, as outlined by the civil servants. Some institutionalized political processes, such as reporting to parliament or legal requirements, even oblige using evidence.

I think evidence is very important for us in our daily work but also in health care planning. If we have to assess the health care situation, we need information, data, evidence, we need to know if access is guaranteed—if they [the patients] can see a general practitioner within such and such a time, for example, or a hospital, and so on. K1

Besides the relevance of evidence for effective HSMP from an organization-level perspective, civil servants consider the employment of evidence important and situations with little or unclear evidence unsatisfactory. They wish existing evidence to be consulted more frequently within decision processes and desire opportunities to perform more in-depth analyses of topical areas they are tasked with. Civil servants’ narrative suggests that efforts to incorporate evidence into the work “as well as possible” (F1) is a consequence of their self-image in the sense of a “professional self-expectation” (D1) and “attitude” (A1). Thus, accounts of using evidence as working “well and carefully” (A1) highlight the internalization of evidence’s fundamental importance and its normative meaning as the right guide to make decisions about HSMP.

The area of hospital planning is perceived to be particularly evidence-informed, and for care planning in general, the medical statistics of the hospitals to represent the “basis of the whole” (E1). Population, hospital, cost, social security data, and other forms of evidence, such as academic publications and survey data, also feed into policy- and decision-making. However, the academic literature appears less relevant for HSMP—it is used to answer specific questions, such as how health care for medical conditions like strokes should be provided or how minimum case numbers should be used to steer care. Regarding the application to health care governance in Switzerland, the international research literature is often considered “not useful for our conditions or our problem” (H2).

Only studying the literature doesn’t help that much either, so if you read any [studies] from Germany or England … we don’t have an NHS (National Health Service), and we don’t have the same underlying circumstances in terms of funding as Germany or so. Thus, certain things you just can’t realize. A1

The same argument is made about the transferability of evidence or policy solutions between cantons, which is considered limited due to contextual situations and environmental differences, such as geography.

Other civil servants emphasize the potential of intensified evidence and tacit knowledge exchanges between the cantons. In health care planning, such interactions are described as close in isolated cases but limited overall. Essentially, they depend on individuals in the administration, predominantly involve the nearest neighboring cantons, and hardly exist across language borders in Switzerland.

From a theoretical point of view, the conceptual use of evidence [43]—for example, to develop new ideas for health services and their regulation—seems to be less predominant than, for instance, symbolic uses to legitimize preexisting positions [44]. Indeed, civil servants highlighted that evidence supports their arguments within the administration and discussions with external policy actors, such as service providers or professional associations, supports the justification actions, and helps convince stakeholders or enforce plans.

High individual-level motivation to engage with evidence despite a challenging politicized context

The administration work occurs in a “political environment” (B1). Therefore, administrations can only shape EIPM to some extent, for example, by providing information and creating framework conditions to promote the integration of evidence through their role in managing service providers. The role of the administration was described as “to do a balancing act” between “those who are concerned, science and politics” (H2) and trying to “promote evidence-based policy” (J1). As individuals who help the administration carry out this role, they see it as their duty to alert when policy proposals conflict with evidence. In some cases, this conception of the role goes so far that civil servants bring evidence into the political processes beyond the administration’s management level if it is in danger of being withheld there. Here is how one civil servant describes such actions:

By working on it and making the evidence available to different political stakeholders … and with that, the possibility was actually no longer there to just let the [evidence] disappear into the drawer. That was not always without its problems, that conduct. H1

Civil servants understood and accepted political rationales but sometimes described being dissatisfied with the limited inclusion of evidence in political processes. They identify patterns that resembled a dichotomy [45] between substantially evidence-informed work on the part of the administration, at least up to the political-strategic level, versus the political decision-making arena, in which evidence does not play an essential role and policymaking is often driven by ideology and expected short-term benefits.

At the same time, respondents provided several examples where evidence significantly influenced or shaped policymaking. Still, their discussion of health policy was dominated by a narrative on the limited impact of evidence on the political rationales. Local and regional policy context was generally described as more ideology-based and conflicting with EIPM.

The more local [the policy issue] the less, how should I say … evidence-based, data-based it is, because those data are not available in studies or anything the like. D1

In this politicized context, using evidence to advocate for a cause can result in negative consequences. A few civil servants reported being verbally attacked professionally or even personally for ideological reasons in advocating for evidence, given “a roasting” (A1) or “finished off” (H2).

The discrepancy between political motives and evidence becomes particularly apparent in the case of recommendations to discontinue ineffective programs, such as disease screening, or cuts to oversupplied services and infrastructure, such as hospital closures.

You could say that health care, accessibility, remains just as good, but the quality could increase, and the costs are better controlled. These are not always the arguments that work when it comes to a local vote on whether a population wants a hospital close to home or not. There are completely different emotional and, economic, local aspects involved, which then prevent this. G1

Limitations

This study targeted a specific, comparatively small group of policymakers from German-speaking state-level health administrations and focused mainly on HSMP at the intermediate to high managerial level. The specificity of the sample might limit the transferability of findings to other work areas within health administrations and language regions in Switzerland and beyond. For example, French-speaking Swiss cantons lean toward a stronger role of the public sector, with the governmental services at the forefront in areas such as public health [84,85]. Such cultural differences among the language regions might manifest in the extent administrations support EIPM and provide respective resources [but see 30]. However, as participants were from administrations representing diverse cantonal characteristics, we consider central issues for Swiss health administrations to be captured. In addition, the descriptions provided by the interview participants correspond broadly with findings from the international literature, suggesting that the results of this study are also relevant to other countries and contexts.

It must be noted that the capacity data presented is based on information provided by six health administrations, with each participant representing one administration. It is possible that the secretaries-general interviewed might not have been aware of all details concerning evidence in specific administration areas [86]. Moreover, due to the exclusion of very small cantons with less than 50,000 inhabitants (n = 5) from the capacity assessment, an underestimation of the mean overall capacity cannot be excluded. Therefore, the general validity of the results should be interpreted with caution. Future studies should verify and extend these results with a broader target group and a larger sample, preferably with quantitative surveys in written or electronic form [e.g., 87].

Several capacity-measurement tools are available [e.g., 88–90]. We selected ORACLe because its development was strongly guided by academic literature and extensively informed by policymakers and knowledge-translation experts. The tool provides clear operationalization of capacity magnitude and tool availability, targeted toward health-policy organizations. Moreover, ORACLe was developed for and tested in a high-income context and found helpful [12,37]. However, the application of the capacity-measurement tool showed the potential for improvement, as recently confirmed by other scholars [91]. For example, we found that interview questions might benefit from a more detailed operationalization of the concepts surveyed. Furthermore, the specificity of the scoring guides’ categories could be enhanced, as assigning interviewee responses to the categories proved challenging in some instances. While these issues should be addressed in future applications of ORACLe, we mitigated shortcomings in the measurement and improved consistency by consulting a second independent rater who double-coded all interviews.

Finally, due to the heavy workload of health administrations in responding to the COVID-19 pandemic, such a quantitative approach involving a large group of civil servants was not considered ethical and operationally feasible. Indeed, this study was conducted in a pandemic context that strongly influenced the operations of cantonal health administrations. As data collected were self-reported, it cannot be ruled out that the salience of research in the pandemic context has impacted current perspectives on the relevance of evidence and administrations’ resources to engage with it. Similarly, social desirability bias cannot be ruled out but seems unlikely, as interview participants felt generally comfortable expressing criticism of their administration, shortcomings in their performance, or its resources.