Setting priorities for the evaluation of health interventions: when theory does not meet practice

Abstract

Priority setting is a key component of the process of evaluating health interventions. This has traditionally been an informal process led by power and influence, but a number of explicit criteria and systematic models have been developed since the late 1980s. This paper presents a review and appraisal of these conceptual models and examines how they have influenced the practice of priority setting in the United States and Europe. The main conclusion is that a significant gap exists between theory and practice. Most models have been developed for the purpose of maximising health gains through an efficient allocation of resources. However, they present at least three important limitations that need to be removed if formal models are to play a more substantial role in decision making: they tend to prioritise interventions for evaluation, rather than evaluations themselves; they fail to address priority setting in a research portfolio perspective; and they fail to adopt an incremental perspective. Existing prioritisation models are not suitable for supporting cost-containment or distributional objectives.

Introduction

The effectiveness and the efficiency of a system for evaluating health interventions depend critically on the setting of evaluation priorities, as well as on the conduct of appropriate evaluations and on the translation of evidence into policy. During the second half of the past century, investments in evaluative research have grown steadily in many industrialised nations, along with calls for the adoption of transparent and systematic approaches to the allocation of resources for evaluations. However, the nature of the political processes through which a large part of such resources is allocated is often at odds with the aspiration to a transparent and systematic setting of evaluation priorities. The multitude of values, interests, and possible criteria for priority setting at play often inhibits the adoption of fully systematic approaches and continues to generate tensions between and within groups of stakeholders.

Advocates of priority setting for health technology assessment research argue that this may improve the efficient use of scarce research resources, and the consistency of research with the aims of health care policy makers. Additionally, priority setting may improve transparency and accountability in the health system. On the other hand, there is no evidence that the application of formal priority setting methods may lead to more desirable outcomes than those achieved through largely informal processes of allocation of research resources.

One of the earliest attempts to investigate opportunities to make research priority setting a systematic activity was made by the U.S. Congressional Office of Technology Assessment (OTA, now no longer in existence). Its conclusion was that ‘the factors that need to be taken into account in research planning, budgeting, resource allocation, and evaluation are too complex and subjective; the payoffs too diverse and incommensurable; and the institutional barriers too formidable to allow quantitative models to take the place of mature, informed judgement’ [1]. An important question is whether the OTA's conclusion still stands in a world that has substantially changed.

The purpose of this paper is to review and appraise the theory and the practice of priority setting for the evaluation of health interventions after the publication of the OTA report. First, the conceptual models developed to make the prioritisation process transparent and systematic are analysed and assessed. The ability of these models to serve different policy aims is discussed. Second, an overview of priority setting used by public and private organisations in Europe and in the U.S. is provided. Finally, policy implications are drawn.

The analysis contained in this paper is based on a comprehensive review of the existing literature and on interviews with health policy makers, research and health care organisations in Europe and in the U.S.