Introduction

Who should be asked to value health states, defined in terms of generic health dimensions, is an ongoing debate among economists, philosophers and other researchers in the field of health outcomes research [1,2,3,4,5,6,7,8,9,10]. There are two main approaches to eliciting health state valuations: individuals’ valuations based on their experience of the health state or individuals’ valuations based on a description of the health state. The choice of whose values to use for health state valuations has generally been described as a choice between ‘patient values’ versus ‘general public values,’ i.e., between those experiencing and those not experiencing the health state to be valued [7, 10]. The former has also been referred to as ‘individual values’ and the latter as ‘social’ or ‘hypothetical values’ [5, 11,12,13,14,15].

The need for clarification and interpretation of the terminology used in health state valuations has been raised [10, 16]. For instance, ‘experience-based values’ has been referred to as the value of the individual’s currently experienced health state [17]. However, experience might be based not only on current experience of the health state to be valued, but also on previous personal experience of the health state, experience of another health state similar to that to be valued, or experience based on relatives’ or other persons’ ill health [18]. When patients are asked to value health states, they are often valuing their own current health state, but patients may also value other health states described to them. Furthermore, the valuation process to elicit experience-based values might involve the imagined states ‘full health’ and ‘being dead.’ When the source of values is the general public, the values are hypothetical in the sense, and to the extent, that usually none of the health states to be valued is experienced by the individual. However, by chance, some of the respondents in the general public might currently experience or have experience of the health state to be valued. When patient values are used, then usually only one health state (their own) is valued by the respondent. When general public values are used, several health states are usually valued by the respondent.

In the literature, it has commonly been assumed that experience-based values must be obtained from patients (i.e., patients are the only kind of raters/respondents thought of as having experience of a health state) and that general public values must be values for described health states only. However, in some recent studies, the general public has been the source of values for valuing experience-based health states; i.e., the general public has been asked to value their own current health state [17, 19,20,21].

In economic evaluation of health technologies and health care interventions, the Quality-Adjusted Life Year (QALY), which combines length of life and health-related quality of life (HRQoL) into one measure, is commonly used [7]. In the QALY context, the quality of life component is usually based on preferences for health states. The Washington Panel on Cost-effectiveness in Health and Medicine recommended in 1996 that “for purposes of resource allocation, the relevant preferences are those of the general public” [22]. Such values, i.e., individuals’ valuations based on a description of the health state obtained from a sample of the general public, have been the most common. However, there have also been arguments in support of obtaining valuations based on individuals’ experience of the health state [4, 5, 19, 23,24,25,26].

The National Institute for Health and Care Excellence (NICE) in England and Wales [27] recommends the use of general public preferences, i.e., valuations of described states, in their guidelines for economic evaluations. However, in Sweden, the Dental and Pharmaceutical Benefits Agency (TLV) states that “QALY weightings based on appraisals of persons in the health condition in question are preferred before weightings calculated from an average of a population estimating a condition depicted for it (e.g., the ‘social tariff’ from the EQ-5D)” [28, 29].

Strengths and weaknesses with these main approaches to health state valuations have been discussed in the scientific literature, but to our knowledge, no systematic review of the arguments has so far been published. The aim of this paper was to determine and critically evaluate the arguments in the published literature regarding the most accurate source of information for valuation of health states: values based on experienced health states or described health states.

When critically evaluating arguments in the literature, we try to take as little as possible for granted when it comes to theoretical starting points. For instance, we leave it open whether the value of a health state should be understood primarily in terms of preferences for the state in comparison with other states, in terms of what it is like to be in the state in question, or some other option [3, 8, 9, 16, 30]. We are, furthermore, open also to the view that the value of a health state can be understood independently of the addition ‘according to whom,’ i.e., that the value of a health state is something intrinsic to that state rather than a relation between the state and one or another group of evaluators. Besides, we are aware that ‘value’ can be used to denote either the goodness/badness of the health state or the result of the valuation of the goodness/badness of the health state, which is something else.

In the methods section, we describe the search strategy, publication selection, data extraction, and process of synthesis. The two sets of arguments and our analysis of the data are presented in the results section. The paper ends with a discussion of the arguments and their relevance for the question regarding the most accurate source of information for valuation of health states.

It is outside the scope of this paper to address how the different approaches influence cost-utility ratios, as well as to address which method (such as time trade-off, standard gamble, or visual analogue scale) to use for valuation of health states. Also the paper does not rest on any elaborated views on how to understand ‘experience’ and ‘described’ health state, since that would go beyond the preciseness of most of the literature reviewed.

Material and methods

This paper is based on a systematic review of arguments in the published literature regarding the most accurate source of information for valuation of health states. The two main approaches to eliciting health state valuations, values based on experienced health states and values based on described health states (hereafter labeled ‘patient values’ and ‘general public values,’ respectively), have been expressed in a variety of ways in the literature.

Search strategy

A systematic search for relevant literature was conducted in October 2015 in three electronic databases: Scopus, Ovid Medline, and Econlit. The search was later updated to include publications until October 2017. The index terms were Medical Subject Headings (MeSH) and free text. The search strategy is presented in Online Appendix.

Selection of publications

All identified publications were independently screened by two readers (second and third author). Potentially relevant publications were first identified through screening of title and abstract. They were considered relevant if they contained (1) health state valuations comparing patient values and general public values, or (2) a discussion of perspective when valuing health states. Publications were then read in full text if at least one of the reviewers considered it relevant for the review, applying the same inclusion criteria as above. In this preliminary reading of the full texts, publications were removed either because they were not written in English or because they were deemed as clearly not relevant for this paper (it was easily determined that they did not contain any arguments of the kind relevant for this paper).

As our aim of this paper was to determine and critically evaluate the arguments regarding the most accurate source of information for valuation of health states, when the remaining publications were read in full text after this preliminary screening, only those publications that contained arguments for one or the other of the two alternatives (using patient values or general public values) were included in the final analysis. For a flowchart on data search, see Fig. 1.

Fig. 1
figure 1

Flow chart on data search

Full size image

Our analysis did not concern arguments regarding choice of other sources for valuation of health states, such as arguments in favor of using valuations of politicians or medical experts.

Data extraction and process of synthesis

A portion (n = 31) of the publications considered eligible for analysis (n = 163) was first read through independently by two readers (first and last author) in order to test the procedure of identifying arguments in favor of any of the two alternatives. Notes were taken according to a scheme where each argument in favor of one or the other of the two alternatives was stated schematically (“A is better than B since … [argument],” or vice versa, where “A” stood for general public values and “B” stood for patient values), exemplified by a quotation from the paper (with page and paragraph reference), and given a preliminary label (such as “The patients-are-better-informed argument”). The procedure was further tested at an early stage by including a third independent reader where all three readers read two papers and compared the outcomes in terms of identified arguments.

When the 31 selected publications had been read by the two readers, a full comparison of the lists of extracted arguments was made. When similar outcomes were noted, the readers discussed and agreed upon the best way to phrase the argument, the most adequate quotation to support it, and the best way to label the argument. Here existing labels from the literature were considered and used when found adequate. When different outcomes were noted—either that one reader had identified an argument that the other reader had not, or that there was disagreement about the content and point of a stated argument identified by both readers—then there was a discussion about what would be the most reasonable way to interpret the relevant passage. When agreement was reached, the best way to phrase the argument, the most adequate quotation, and the best label for it was discussed and agreed upon. It was agreed on beforehand that if agreement could not be reached regarding a certain presumed argument, then that argument would be disregarded (this did not in fact occur).

The rest of the publications (n = 132) considered eligible for analysis were divided between the two readers, who read them, discarded some of them as not containing any relevant arguments, and classified the arguments found as above, and consulted the other reader when needed.

Synthesis

The outcome of this analysis (lists of arguments, sources for the arguments, and labels for arguments for each included publication) formed the basis for identification of first-, second-, and third-level themes, in accordance with common practice in descriptive qualitative content analysis [31, 32]. The grouping into themes was handled separately for the two approaches: using patient values or using general public values.

Preliminary first-level themes were identified by bringing together identical or very similarly stated arguments from the different publications where they occurred. This list of themes was discussed and successively shortened during the process of comparing the variation in expressions (“Is this the same argument or is it another argument?”). The purpose was to eliminate any duplications and overlaps, but without losing any distinct argument in the process. This operational process was complicated as the initial list of themes included variants for most of the themes. The reduced combined list formed the set of first-level themes.

In the next step, related first-level themes were distilled into second-level themes. This process was in some parts simple and straightforward, but in others a fairly complex and dynamic process where themes were arranged and re-arranged until the emerging second-level themes were seen as clearly distinct and adequately stated. The process of distilling second-level themes from first-level themes was first carried out independently by the two readers. Then, the two sets of suggestions for second-level themes were compared. The discrepancy between the suggestions was minor and consensus was reached through discussion. The third-level themes emerged from the readers’ individual examination of the second-level themes and a discussion to reach consensus (in one case). The first-, second-, and third-level themes were then critically discussed with the second and third author to get their input on the categorization and the labeling of the themes.

Results

Through the database search, 3651 articles were identified after removing duplicates (Fig. 1). Of these, 223 articles were considered potentially relevant and the publications were read in full text. Nine articles were excluded as the full text was in another language than English, and 51 articles were considered not relevant as they did not meet the inclusion criteria. As a result of this full-text screening, 163 articles were left to be thoroughly assessed for arguments. Based on this assessment, another 81 articles were excluded as not relevant (i.e., not containing any arguments for either of the approaches). Finally, 82 articles were included because they contained arguments for either of the approaches.

The analysis of arguments favoring either values of experienced health states (patient values) or values of described health states (general public values) as a basis for valuation of health states resulted in two structured sets of arguments, one for each position. The analysis of arguments in favor of valuations based on patient values resulted in 16 first-level themes (Table 1), five second-level themes, and three third-level themes: ‘failures of the general public to value health states,’ ‘effects of measuring health states’ and ‘theoretical reasons’ (Table 2). The analysis of arguments in favor of valuations based on general public values resulted in 19 first-level themes (Table 3), five second-level themes, and three third-level themes: ‘patient failures to value health states,’ ‘ideas regarding how to reach the most acceptable societal outcome,’ and ‘possibilities and effects of measuring health states’ (Table 4). References to the articles included in the analysis are listed in Table 5 (see Online Appendix). Arguments to the effect that patient values are more accurate to a great extent concerned the idea that patients are better informed about the health state than the general public, circumstances regarding adaptation that the general public is not aware of and which underlie misjudgments, and valuation difficulties, like focusing and contrast effects, leading the valuations of the general public astray. Arguments to the effect that general public values are more accurate also focus on adaptation issues and valuation difficulties, this time as a particular problem for patient values. Another set of arguments, often referring to ‘social values,’ such as the tax payer/insurance payer argument and the argument about vested interests, uniquely supports general public values.

Table 1 Arguments favoring patients as the most accurate
Full size table
Table 2 Arguments favoring patients as the most accurate source for valuing health states sorted in themes
Full size table
Table 3 Arguments favoring the general public as the most accurate
Full size table
Table 4 Arguments favoring the general public as the most accurate source for valuing health states sorted in themes
Full size table

Discussion

This review of arguments regarding the most accurate source of information for valuation of health states—individuals’ valuations based on their experience of the health state or individuals’ valuations based on a description of the health state, usually framed in the debate as patient values versus general public values—shows that there is an intense discussion in the literature with several arguments for each position (see Tables 1, 3). There is a structural similarity between the two sets of arguments. The arguments supporting the use of valuations made by the general public point out that patients have difficulties making the valuations, partly relating to adaptation [33,34,35], partly relating to focusing effects and distortions due to perspective (reference point effects), such as that being in a health state very far from full health may lead to difficulties in estimating the distance [2, 3, 16, 36]. This is exactly what the opposing arguments say about the general public, as a reason to favor patient values (although the details of the arguments differ) [3, 16, 36, 37]. In both sets of arguments, there are also arguments to the effect that their position is best when it comes to protecting the interests of patients in need [2, 11, 16, 38]. What distinguishes them is that theoretical reasons based on welfare economics are frequently and explicitly appealed to in support of patient values [2, 21, 30, 39], while arguments relating to social values [36, 40, 41], absence of bias [24, 35, 37, 42], and research advantages [24, 43, 44] are repeatedly brought up in support of using general public values.

No flawless position

When it comes to evaluating the strength of the two positions, it becomes clear that neither position is flawless. That patients have superior knowledge about the health state they experience, compared to the general public, seems to be unquestionable in many cases. We take this to be the most significant difference between the two approaches. It is argued that the general public also tends to be considerably misled by focusing effects (effects of focusing on some aspects of ill health while not considering unaffected life domains) [3, 42] and to have difficulties understanding the degree to which the individual can adapt to a new health state, even if the change from a healthy state to ill health is dramatic, shocking, and deeply saddening when it occurs [31, 45, 46]. However, there are also convincing arguments that patients in various ways may misrepresent their new situation to themselves. Their own view of what their life is like may be considerably distorted, perhaps because of a need to see it as better than it is [3, 47, 48].

Arguments relating to welfare economics do not put an end to the discussion. As put by Brazier and colleagues [2], p. 204:

An important normative position (or value judgement) in welfare economics is that the well-being of a society is simply the aggregation of the utility of individual members of society. In other words, it asserts the supremacy of an individual’s valuation of their own well-being. This implies that it is the preferences of the losers and gainers from a public programme that should be elicited, and not a sample of the general population who will be unaffected by the change; this would seem to suggest that patient values should be used. One response to this is that current patients do not represent all those likely to be affected by the set of decisions being made. It could be argued that every citizen in a public system has an option to use the service and so may be gainers and losers, meaning that the general population would be a good proxy. However, this presumes that all citizens have an equal chance of receiving all forms of care, which is unlikely. Therefore, welfare economics does not seem to offer a clear argument for either approach.

All this means that there are good reasons not to trust any of the alternatives without reservations. This being said, our review of the arguments found in the published literature suggests that the overall most accurate source of information for valuation of health states is that based on experience (patient values), mainly because those with own experience of a health state are better informed about it. This suggestion is not conclusive, but what would be required of new empirical input to change the balance would be to show that patients as a matter of fact are not better informed about their own health state than those getting the state described to them.

Irrelevant arguments

One of the most striking features of the debate is that some of the arguments used are entirely irrelevant to the issue at stake here: what is the most accurate source of information for valuation of health states? One example, found in both sets of arguments, is the argument that making the valuation as proposed is advantageous to the patient group [2, 16]. Advantageous or not—this has nothing to do with the question at hand.

A large set of equally irrelevant arguments to the question at hand are found among those supporting the use of the general public valuations of described health states, gathered under the third-level theme ‘ideas regarding how to reach the most acceptable societal outcome.’ The identified first-level themes were referring, for instance, to the societal perspective and tax payer (including insurance payer) arguments. It has been suggested that valuations from the general public for described health states are to be preferred with arguments like “[T]he best articulation of society’s preferences for a particular state would be gathered from a representative sample of fully informed members of the community” [22], “The public, and not patients, should make distributive decisions about who should gain health-related utility as a result of public programmes” (p. 133) [33] and “The social perspective for publicly funded services can also be supported on the grounds that it is tax funded” (p. 204) [2].

Notice, however, that the defense concerns something else than being the most accurate source of information for valuation of the health state; the arguments rather focus on another matter, namely who should exert influence over health policy decisions. There is indeed a fundamental difference between what is the most accurate source of information for valuation of health states and who should get a say regarding policy decisions (which certainly also is worth discussing). As commented by Wolff et al. (p. 461) [26], “Arguably, the political concern about how the public likes to see its money spent should not impinge on the empirical question of how best to measure the benefits of health interventions.” Generally, what is a good answer depends on what is the question. If you want to know the value of a certain health state (which was our question), then that is what you should examine. If you instead want to know how the health state should be handled, what resources should be allocated and how it should be prioritized in relation to other health states in need of attention, then that is what you should investigate. There is no reason to assume beforehand that the answer is the same to both questions. This needs to be argued rather than assumed. Hence, if one believes that prioritization in health care should ultimately be decided by the general population, or should be influenced by their preferences, it still remains to be shown that the general population is also the best source of health state valuations.

This perhaps comes more naturally for those who are not used to understand value in terms of preferences, but rather see it as something supervenient on characteristics of that which is valued. For the latter, the value of a health state is more obviously something distinct from what desires people might have towards that state (i.e., value of the health state means something else than preferences for or proattitudes towards the health state). Policy-makers might, of course, want several kinds of input, including (i) an accurate valuation of concerned health states and (ii) a description of perceptions of different health states among the general public [16].

Arguments about ‘social values’ can also be regarded as irrelevant in another way. People in the debate seem to assume that individuals in the general public generate social values, while patients generate individual values. However, there is nothing more social about the values of individuals from the general public than the values of some patient group. In the words of Brazier et al. (p. 201) [2]:

The conventional approach to valuing health states essentially asks respondents for their ‘off the cuff’ ex ante valuation of the states. The resultant values have sometimes been described as social values but this is a misleading term since respondents are being asked to value the states from the perspective of being in the states. The resultant set of health state values is really an average of the individual valuations of those states.

More relevant under this third-level theme, since they do concern the correct valuation of health states, are the veil of ignorance and the vested interests arguments, both stressing the risk that patients valuing their present health states present partisan valuations in order to serve their own interests. In theory, this could present a real problem. However, in practice, these misgivings seem to be unjustified since empirical research shows that patients often provide higher health state values than the general public, which would rather be to their disadvantage in the context of health improving treatments [2].

Subjective and objective aspects of health states

There are genuine difficulties with the core issue—what should be meant by ‘a health state’? Both subjective and objective considerations might come in here. One way to understand ‘health state’ is that a health state consists of nothing but what it is like to be in that state—health is what it appears to be “from within”—that is, it is an entirely subjective concept. However, if a health state is fully characterized by what it is like to be in that state, then the valuation errors identified in the debate regarding experience-based values—for instance, relating to shifting value scales, focusing effects and reference point effects—are, after all, not errors. More specifically, there will be no external standard (i.e., based outside the patient) by which to label patient valuations as accurate or inaccurate. The obvious implication would be that asking those being in the health state to value it is superior to any other alternative.

However, it can be contested whether a health state exclusively consists of its subjective aspects. For instance, being in a certain health state might restrict the opportunities to do things, a fact stressed both by Norman Daniels [49, 50] and in the discussion of capabilities [8, 26, 51,52,53]. Whether the individual perceives such limitations as genuine loss is one thing, and whether opportunities are in fact lost, missed or not, is another. One might argue that loss of opportunities or capabilities is a genuine loss whether missed by the individual or not, putting the individual in another health state compared to the state before the loss occurred. Opportunities and capabilities, in other words, bring in an objective (i.e., non-subjective) aspect to the discussion of what constitutes a health state. Only if we allow considerations other than what it is like to be in that state can we question the superiority of experience-based valuations of health states. Although it goes beyond the ambitions of the present study to compare with an opportunity- or capability-based approach, it can be noted that in cases where those being in a certain health state do not see that they have lost anything by ending up in that state, members of the general public may see it because they note opportunities and capabilities lost [26].

So we conclude that if the main problem with using values from the general public for described health states is that they do not know enough about the health states they are valuing and become victims of all kinds of misleading focusing effects, which would most likely remain even if they would receive more information, the main problem for patients as evaluators is that they may be overly focused on what it is like to be in the health state as perceived from their new perspective, while that may not be all there is to say about it.

Strengths and limitations

The main strength of this paper is that it provides a structured overview of the arguments occurring in the debate regarding most accurate source of information for valuation of health states, and that it identifies some common arguments in the debate as irrelevant for this specific matter. Hopefully it can contribute to an increased focus on the quality of the relevant arguments. We also take it to be a strength of the paper that it is the outcome of collaboration between health economics and ethics.

Although the ambition of this review has been to cover the wide variety of arguments relating to the most accurate source for valuations of health states, patient values or general public values, we cannot exclude that limitations of the search terms or data bases used have made us miss some articles that would have been relevant to our paper. The exclusion of books from the systematic review might have had similar effects, although we suggest that what appears from an author in the form of books often have turned up in print before that in the form of journal papers.

A built-in difficulty with the approach of the present paper is the identification and interpretation of arguments. Many of the arguments identified in the literature are not clearly expressed and quite a few are just briefly sketched rather than thoroughly formulated. Sometimes, it is not clear whether the authors have intended a passage as an argument or if they just point to differences in outcomes depending on the chosen source for health state valuations. It cannot be excluded that we have occasionally failed to identify relevant articles or relevant arguments in the articles reviewed—e.g., because they have been embedded in arguments explicitly concerning something else—or that we have interpreted passages as containing arguments where no argument in favor of the one or the other alternative was intended. It was not part of our review to systematically improve the arguments found in the literature into their best possible version.

Another potential limitation is the organization and structure of the paper. The analysis of arguments has taken its starting point in the existing literature, including the way the topic has been framed, which means that some of the messiness of the discussion, not least the terminological variations, has affected the paper. Furthermore, it is not entirely clear that central terms, like “general public” and “patient values,” are used in the same way by different authors and over time. From an analytical perspective, it might have been more fruitful to use some alternative approach that would have distanced itself more from the existing discourse. But then one would, of course, easily have missed the point of showing what the discussion has looked like. Furthermore, some lack of clarity cannot be circumvented by introducing a new terminology (this would, for instance, not solve potential lack of clarity as to what authors mean by “general public” and “patient values”). The need for clarification and interpretation of the terminology used in health state valuations remains [9, 16, 18]. The extensive search strategy reduced the risk of not finding relevant publications due to the variety in terminology used in health state valuations.

Finally, it cannot be excluded that preconceptions of the authors may have influenced both identification and interpretation of arguments in the debate, although awareness of this risk throughout the work has hopefully tempered such tendencies. The last author has published a paper on experience-based value sets for EQ-5D health states [17].

Conclusion

This paper has provided a structured overview of the arguments in the discussion in the published literature of the most accurate source of information for valuation of health states—individuals’ valuations based on their experiences of the health state or individuals’ valuations based on a description of the health state. Our review shows that none of these approaches is flawless and that both positions are partly backed up by irrelevant arguments. This is particularly so regarding support for the described health states position, where ideas regarding social values dominate the argumentation while being beside the point for the question of the most accurate source of information for health state valuation, which was the focus of our study. Arguments for both approaches rightly point out that the other approach has difficulties in relation to adaptation: the general public valuing described states underestimates the ability of those in ill health to adapt, while those valuing the health state based on their experience may have adapted in ways that distort their valuation. Both approaches also have difficulties relating to focusing and reference point effects. Nevertheless, our review of the argumentation in the published literature suggests that the most accurate source of information for valuation of health states is that based on experience, mainly because those with own experience of the health state are better informed about it. This suggestion is not conclusive, but what is required of new empirical input to change the balance is to show that those with experience of the state are as a matter of fact not better informed about their own health state than those getting the state described to them.