Applicability and transferability of interventions in evidence-based public health

Abstract

The context in which public health programmes operate can play an important role in influencing their implementation and effectiveness. An intervention that has been shown to be effective in one setting may turn out to be ineffective somewhere else, even supposing it can be implemented there. Therefore, systematic reviews of public health interventions should appraise the applicability of the intervention process and the transferability of the intervention effectiveness to other localities. However, applicability and transferability appraisal is seldom reported in systematic reviews of public health and health promotion interventions. This paper aims to introduce an innovative approach to bridging this gap. A list of attributes that may impact on applicability and transferability can be developed, based on knowledge of the proposed intervention. Then the applicability and transferability of the intervention to the local setting can be rated, and given a score, based on knowledge of the local setting. This approach provides a useful tool for evaluating public health interventions and provides a reliable basis for informed decision making in resource-poor settings, where rigorous primary studies are lacking and where very limited resources put a high demand on evidence-based approaches to health promotion.

INTRODUCTION

Evidence-based public health requires that health care decisions should be based on the best evidence available. To determine the effectiveness of a public health intervention in a specific local setting, a well-designed and well-conducted randomized controlled trial (RCT) may provide the best evidence (Rychetnik et al., 2002). Besides RCTs, observational studies can also render useful information about public health interventions. The outcomes of a public health intervention can be measured with a wide range of indicators, ranging from health literacy and healthy lifestyles to final health outcomes, such as quality of life and mortality or morbidity (Nutbeam, 1998). However, with limited resources, many countries, especially developing countries, will be unable to afford to test all public health interventions that show promise in the outcome measures of interest. Therefore, they may have to rely on evidence obtained elsewhere.

In addition, for public health problems that have a particularly large impact on population health, such as the acquired immunodeficiency syndrome (AIDS) in many developing countries, it may be unethical to wait until evidence from the actual local setting becomes available. Policy makers have to make a decision based on currently available evidence. Delaying until better evidence becomes available is itself a decision having its own costs and benefits. However, as public health interventions are dependent on the context where they are implemented, an intervention that has been shown to be effective in one setting may turn out to be ineffective in another, even supposing it can be implemented there. By context, we mean the particular social and cultural environment and the particular political and organizational system in a society. Therefore, the applicability and transferability of public health interventions from the study setting to the local setting should be assessed whenever there is a substantial uncertainty.

For the purpose of this paper, we define ‘applicability’ as the extent to which an intervention process could be implemented in another setting; and transferability as the extent to which the measured effectiveness of an applicable intervention could be achieved in another setting. Applicability and transferability are compared and contrasted in Table 1.

To illustrate the concept of applicability, consider the cervical cancer screening programmes that are widely deployed in developed nations and have been demonstrated to be effective in reducing the risk of cervical cancer (Sasieni et al., 2003). Such an intervention may not be readily applicable to a setting in which conservative social norms about sexuality and fertility do not allow women to undergo a Pap smear. Under such circumstances, interventions encouraging the uptake of such a test perhaps should be introduced first in order to relieve the embarrassment associated with Pap smear taking. Another example is a self-help intervention using written materials in promoting smoking cessation (Secker-Walker et al., 2002). If most of the target population is illiterate, this intervention would be inapplicable and alternative interventions such as a health-worker-delivered intervention should be adopted.

In examining the transferability of the measured effectiveness of an intervention from the study setting to a local setting, we are again focusing on the impact of context. One of the possible contextual factors that may have an impact on whether a public health programme achieves its intended effect is the skill of the proposed providers. If a programme is quite new to the local setting and the staff are unfamiliar with its approach, the effectiveness originally demonstrated in the study setting may not be achievable in the local setting. Another possible contextual factor is whether the target population has access to requisite facilities. Imagine a malaria control programme promoting insecticide-treated bed-nets and curtains in a poverty-stricken community. If bed-nets and curtains are not widely available and no measures are taken to increase their accessibility, then the effects of such an intervention would be compromised by its limited coverage. We also accept that strictly biological differences between populations may also affect transferability, although such a discussion is beyond the scope of this paper.

Applicability and transferability can mean different things to different commentators. We tried to find related terms for ‘applicability’ and ‘transferability’ and methods for assessing them in a systematic review by searching A Dictionary of Epidemiology (Last, 2001), the Oxford Textbook of Public Health (Detels et al., 2001), Evidence-based Public Health (Brownson et al., 2003), Annual Review of Public Health (1990–2003), the Cochrane Library, the Australian National Health and Medical Research Council (NHMRC) Handbook Series on Preparing Clinical Practice Guidelines (Australian National Health and Medical Research Council, 2000), and A Schema for Evaluating Evidence on Public Health Interventions (Rychetnik and Frommer, 2002). We did not find explicit definitions for these two terms. In some cases ‘applicability’, ‘transferability’ and ‘generalizability’ were used interchangeably, and implicitly meant the capacity of primary study results to be useful beyond original study populations. To make the difference between these two terms clearer for readers, we explicitly defined them for the purpose of this paper (see above).

The Grade Working Group (Grade Working Group, 2004) and The Cochrane Applicability and Recommendations Methods Group (Glasziou, 2004) aim to establish guidelines for grading quality of evidence and strength of recommendations for practice and policy. Because their primary focus is clinical practice, the underlying principle of their approach is to use knowledge of statistics and epidemiology to determine the generalizability of review results, by assessing the internal and external validity of the primary studies. For evaluation studies of public health interventions, not only should quality and generalizability be assessed using similar tools to those mentioned above, but also the impact of context on the applicability of the intervention process and the transferability of intervention effectiveness to the local setting should be examined.

For public health interventions, the appropriateness of an intervention adopted from one setting to another has been discussed and the need to assess the applicability and transferability has been proposed (Rychetnik and Frommer, 2002; Rychetnik et al., 2002). In order to meet this need, we have explored an innovative approach to assessing applicability and transferability.

This work has been inspired by A Schema for Evaluating Evidence on Public Health Interventions (Rychetnik and Frommer, 2002) in which the role of intervention context and evaluation context in public health interventions is pointed out and has been motivated by the need to assess the applicability and transferability of public health interventions, particularly for decision-making in developing countries, where primary evaluation studies of public health interventions are performed less often than they are needed. We hope our method can provide a useful tool for evaluating public health interventions and provide a reliable basis for informed decision-making in resource-poor settings.

APPROACHES TO ASSESSMENT

The implementation of any public health intervention places a set of demands on the context, for example, a supportive political environment and/or accord with the social norm. Therefore, for an applicability appraisal, we need to compile a list of possible attributes of applicability, which could only be developed on the basis of a sound knowledge of public health interventions. These attributes would then be scored according to their presence in the local setting by raters with sound local knowledge.

Before commencing a transferability assessment, the internal and external validity of each primary study should be appraised, as would happen in any systematic review. If the primary study has an acceptable quality, then transferability appraisal can begin. Transferability appraisal focuses on the outcomes of the intervention. Since a public health intervention could be measured across a broad range of outcomes (Nutbeam, 1998), the transferability should be appraised for each outcome. In a manner akin to the assessment of applicability, during transferability appraisal a list of factors that could potentially influence the intervention effectiveness should be developed first, and then the similarity of these factors between the original study setting and the local setting would be rated.

There are many external and internal contextual factors that may have an impact on the effectiveness of an intervention. These various factors include: the epidemiological situation (including the stage of the epidemic, the incidence or prevalence of the disease in question and its distribution across geographical areas and/or population groups), the capacity to implement the intervention (including resource availability, the skills of local people, organizational factors, and the social and political environment), and the characteristics of the target population (including cultural practices and the level of literacy) (Rychetnik and Frommer, 2002). One or more of these factors may have an impact on the effectiveness of an intervention, depending on the substance of the intervention and the approach to programme delivery. For example, if an intervention is delivered mainly by casual conversation in a friendly environment, which allows questions and discussions, then the literacy of the participant may not have a noticeable impact on the effectiveness.

Sample questions to ask in determining applicability and transferability have been listed in Table 2. Since the attributes of applicability and transferability are intervention-specific, the primary evaluation study should provide adequate process and contextual information to inform about the factors that may contribute to the implementation and effectiveness of the intervention. Otherwise, the relevance of this evaluation study to other settings will be unable to be determined. Therefore, if the process and contextual information are too lengthy to fit in the published paper, as frequently happens (Rychetnik and Wise, 2004), a linked source should be provided for such information.

There are at least two alternative approaches for obtaining lists of attributes for applicability and transferability and for rating them. One approach is to employ a Delphi technique. Taking the whole process of implementing an intervention into account, from design to outcome evaluation, individuals with diverse expertise (including experts in this particular health problem, epidemiologists, sociologists, health economists, experienced public health practitioners and representatives of the target population) could develop a list of attributes of applicability and transferability (respectively), in terms of political, social/cultural and organizational environments. Then others who know the local context could decide whether or not, and to what extent, the intervention could be applied to the local setting and whether the evidence of intervention effectiveness detected in the study setting could be transferred to the local setting. The second group of individuals should have the same expertise as the first expert panel, besides the additional requirement of demonstrated local knowledge.

An alternative approach is literature-based. Investigators undertaking a systematic review make decisions based on available evidence from the literature as well as their own experience and knowledge about the relationship between the proposed public health intervention and the local setting. First, literature regarding the possible attributes of applicability and transferability would be collected using a validated search algorithm. Systematic reviewers would develop a list of parameters for applicability and transferability, building on available information from the literature and their own knowledge of that public health intervention. Next, literature about the local social, cultural, economic and political context and epidemiological studies about the target health problem would also be collected, again using a validated search algorithm. Evidence on intervention process and contextual information is often drawn from research that uses a combination of different types of studies, including descriptive, qualitative and observational studies. A collection of example guides to evaluating such evidence is available (Rychetnik et al., 2002). After carefully studying the literature, systematic reviewers could rate the level of applicability and transferability for each attribute, based on both the literature and their understanding of the local setting. Comparing the local context with the primary evaluation context and rating the applicability and transferability is a matter of judgement that builds on the best available evidence. We suggest making explicit and transparent the factors underlying this judgement, so that local decision-makers can form their own opinion about its validity.

Since we are dealing with evidence-based decision-making, the question arises as to what extent applicability and transferability appraisal can themselves be evidence-based. For this, we have the following suggestions. First, authors of primary studies should be encouraged to publish information about the context in which their research was conducted. Secondly, steps should be taken to assure the validity of this contextual information by using demographic, epidemiological and/or qualitative research methods that are recognized by standard textbooks and methodological guidelines. Similarly, steps should be taken to ensure that the local contextual information is valid by using the best available evidence, for example from the national census, the national registration of vital statistics and international comparisons such as that are published by the World Health Organisation. Finally, it is inherently a matter of judgement as to how much resemblance there is between the original study context and the local context, but the probity of the judgement can be enhanced by making the reasons behind it explicit and public.

POSSIBLE RESULTS OF APPLICABILITY AND TRANSFERABILITY APPRAISAL

There are four possible results of an applicability and transferability appraisal. For each attribute, one extreme is that no important difference can be found between the local setting and the study setting, and thus the intervention can be expected to generate useful health outcomes in the new setting. Such congruence may be uncommon if there is a big difference in context between study setting and target setting. The other extreme is that there emerges a considerable difference between the local setting and the study setting, and thus the intervention cannot be applied to the local setting. Once an intervention is regarded as virtually inapplicable to the local setting even on only one relevant attribute, there is little point in embarking on a transferability appraisal. Should this be the case, the local health planners need to consider another approach to this health problem or to work towards changing their local circumstances so they become receptive.

Between these two extremes may lie a situation where the local setting can meet the basic requirements with extra effort. Planners may need to either adjust the intervention to suit the local setting or to take measures to make the local setting more receptive to the intervention. Training local policy makers and interventionists, advocating for a supportive political and social environment and educating the public to change their attitude toward a particular aspect of the intervention may all be needed. It is also possible that making a judgement about the degree of applicability and/or transferability may not be feasible, because of a lack of evidence. If so, specific further research should be recommended to generate the evidence needed (Grade Working Group, 2004).

AN EXAMPLE OF APPLICABILITY AND TRANSFERABILITY APPRAISAL

Table 3 illustrates an appraisal of the applicability of and transferability to the Chinese setting of behavioural interventions for the prevention of HIV/AIDS among men who have sex with men (MSM).

The attributes listed in this table were developed by the authors, who are undertaking a systematic review of HIV behavioural interventions among MSM for the Chinese setting. In this table we use a five point Likert scale to rate the applicability and transferability of several possible interventions to the local setting. The ratings were primarily based on a review of HIV behavioural interventions among MSM and available literature about the Chinese political, cultural and socio-economic environment related to sex and homosexuality, as well as literature on the HIV epidemic and its prevention in China.

Three types of interventions are included in our systematic review. One-to-one counselling and education delivered face-to-face or over the telephone by a relevant expert is described as an individual intervention. Small group interventions are delivered to a group of eligible participants by qualified AIDS educators. Community interventions (in this review) are usually delivered in gay bars by peer educators.

From Table 3, we can see that for one attribute of applicability—social acceptability—small group intervention is rated as ‘very unfavourable’. Therefore, small group interventions for MSM are judged to be unsuitable for the current Chinese setting. In contrast, for individual and community interventions, all attributes of applicability are rated as either ‘favourable’ or ‘very favourable’, and thus they could be implemented in China at least to some extent. Because of the traditional restrictive environment for MSM in China, most are reluctant to acknowledge their sexual orientation (Pan et al., 1994; Zhang et al., 2000; The UN Theme Group on HIV/AIDS in China, 2002). Small group interventions that need to recruit a group of MSM to receive education and training outside their gay community therefore face barriers to acceptance. Our judgement is that only a limited number of gay men would be willing to make themselves known for recruitment. These men may provide an opportunity to implement a peer-led intervention within the gay community, although there will be many other challenges in the process of implementation, such as continued training and support for peer educators (Zhang, 2002). Additionally, because of changing attitudes toward sex, emerging tolerance to MSM and the availability of gay gathering places (such as gay bars) (The US Embassy in Beijing, 1997; Zhang et al., 1999; Zhang, 2002), we believe that a peer education programme based on a community and individual counselling service could be possible in the Chinese setting, although great efforts still need to be made, such as campaigning for a more tolerant environment, and training of counsellors to ensure their client's privacy.

In regard to transferability appraisal, since small group interventions are regarded as inapplicable, it is pointless to assess transferability. For individual and community interventions in turn, we can see that at least one attribute of transferability is rated as ‘uncertain’, and thus the transferability of the effectiveness of these two interventions to the Chinese setting has not been demonstrated.

DISCUSSION

In order to remedy the absence of applicability and transferability assessment in systematic reviews and to facilitate informed public health decision-making, we propose this innovative approach to assessing applicability and transferability from a study setting to a local setting using evidence about both the local setting and the public health intervention of interest. To our knowledge, this is the first methodological description of such an approach for developing countries.

A good understanding of both the proposed intervention and the local context is the key to reaching a sound decision on applicability and transferability. Public health interventions depend very much on the context. Without applicability and transferability appraisal in a systematic review of public health interventions, the results of a review may be irrelevant for a particular setting.

Appraisal of applicability and transferability has been one of the areas of interest of the Cochrane Health Promotion and Public Health Field (Frommer et al., 2003). The Field has been developing ‘Guidelines for systematic reviews of health promotion and public health interventions’ that provide recommendations to reviewers on the assessment of applicability. These documents will become available from their website (personal communication).

If an intervention scores well on applicability and transferability, then local policy makers should be able to readily adapt it to their local setting. Thus, this method should contribute to disease prevention and health promotion in this setting with the least feasible amount of resource use. If the intervention is inapplicable at the moment, this method could help to identify the gap between an enabling context and the current local context, and shed light on possible measures for bridging the gap. Strengthening capacity building at the policy and organizational level, advocating an environment favourable to the intervention of interest and extensive training of the interventionists are possible approaches for rendering an intervention applicable. If the intervention is applicable but not yet demonstrated to be transferable, rigorous evidence such as that obtainable through a randomized controlled trial is recommended to determine the effectiveness of this intervention, before it is scaled up to a national level.

Improving the assessment of applicability and transferability per se depends on developing an enhanced understanding of the factors that actually have an impact. The attributes of applicability and transferability may vary from intervention to intervention, from time to time, and from place to place. Developing a validated list of these attributes is critical. Thus the availability of contextual information about the public health intervention is crucial in applicability and transferability appraisal.

We acknowledge that there are both strengths and weakness in our proposed method. The strength of the involvement of experts could be their richness of perspective, but this approach could be costly and would require a considerable effort in the selection of experts and their retention on the panel. The longer the time-span of the Delphi survey, the greater the likelihood of attrition of personnel. The second alternative of a literature review could be less expensive. Most effort would be devoted to literature collection and interpretation. However, this approach is demanding of the skill and knowledge of reviewers, who must have a very sound understanding of the local setting and an ability to read local literature.

Appraisal of applicability and transferability is inevitably subjective, and value judgements are inherent. However, the soundness of such judgements could be enhanced by ensuring a thorough knowledge of the proposed public health intervention and of the local setting, by basing the appraisal on a thorough search of the published literature, and through an open declaration of the appraisers' backgrounds. This approach could contribute to the field of systematic review by providing a practical and feasible method of enhancing the usability of findings.

CONCLUSION

Applicability and transferability are to be judged in a systematic way based on a wide range of evidence. Although it has some limitations, this method introduces a practical approach to appraising the applicability and transferability of public health interventions from their original study setting to a particular local setting. The novelty is in the systematic comparison of the original and the target setting. It bridges the gap of applicability and transferability appraisal in the systematic review of public health interventions, particularly for developing countries, where rigorous primary studies are lacking and where the very limited resources constrain evidence-based approaches to health promotion.

The authors wish to thank Ms Nicki Jackson from the Cochrane Health Promotion and Public Health Field and two anonymous referees for their helpful comments on earlier drafts.

REFERENCES