Meta-analysis of data on costs from trials of counselling in primary care: using individual patient data to overcome sample size limitations in economic analyses
BMJ 2003; 326 doi: https://doi.org/10.1136/bmj.326.7401.1247 (Published 05 June 2003) Cite this as: BMJ 2003;326:1247- Peter Bower, senior research fellow (peter.bower{at}man.ac.uk)1,
- Sarah Byford, senior lecturer2,
- Julie Barber, lecturer in medical statistics3,
- Jennifer Beecham, senior lecturer2,
- Sharon Simpson, research associate4,
- Karin Friedli, principal lecturer5,
- Roslyn Corney, professor of psychology6,
- Michael King, professor of primary care psychiatry8,
- Ian Harvey, professor of epidemiology and public health7
- 1 National Primary Care Research and Development Centre, University of Manchester, Manchester M13 9PL
- 2 Centre for the Economics of Mental Health, David Goldberg Centre, Institute of Psychiatry, London SE5 8AF
- 3 Research and Development Directorate, University College London Hospitals NHS Trust, London NW1 2LT
- 4 Centre for Occupational Psychology, University of Cardiff, Cardiff CF10 3YG,
- 5 Centre for Research in Primary and Community Care, Faculty of Health and Human Sciences, University of Hertfordshire, Hatfield AL10 9AB,
- 6 University of Greenwich, Bronte Hall, London SE9 2UG,
- 7 School of Medicine, Health Policy and Practice, University of East Anglia, Norwich NR4 7TJ,
- 8 Department of Psychiatry and Behavioural Sciences, Royal Free and University College Medical School, London NW3 2PF
- Correspondence to: P Bower
- Accepted 1 May 2003
Abstract
Objective To assess the feasibility of overcoming sample size limitations in economic analyses of clinical trials through meta-analysis of data on individual patients from multiple trials.
Design Meta-analysis of individual patient data from trials of counselling in primary care compared with usual care by a general practitioner.
Setting Primary care.
Patients People with mental health problems.
Main outcome measures Direct treatment costs, depressive symptoms, and cost effectiveness.
Results Meta-analysis of individual patient data proved feasible. The results showed that the previous analyses of individual trials were underpowered to provide useful conclusions about the cost comparisons. The results are sensitive to assumptions made about the costs of sessions with a counsellor and the management of patients by a general practitioner.
Conclusions Meta-analysis of individual patient data may assist in overcoming sample size limitations in economic analyses. Although feasible, such analysis has shortcomings that may limit the validity of the results. The relative costs and benefits of this method, as opposed to further collection of primary data, are as yet unclear.
Introduction
Economic evaluation is an increasingly important element of clinical trials. Several aspects of economic evaluations can make their addition to clinical trials problematic, but of particular importance is the sample size requirement.1 2 Trials are generally powered to detect clinically significant differences in outcomes, but since costs are often characterised by higher variability and skewness than clinical outcomes,3 economic analyses may be underpowered, resulting in type II errors— that is, failure to reject the null hypothesis that there is no difference in costs between treatments.
A recent example of this concerns counselling in primary care. Four trials reported no evidence of differences in the total costs of patients treated by counsellors compared with those who remained under the usual care of a general practitioner.4–7 None of these trials were powered on costs, however, so that firm conclusions about costs cannot be drawn. An adequately powered trial could be conducted, but this would need to be large, would be costly, and would delay access to relevant information for several years.
An alternative solution is to pool existing data in a meta-analysis. Analyses of data on individual patients are considered the gold standard for this technique and have several advantages, such as ensuring quality of randomisation and intention to treat analyses,8 but in the present context the primary advantage would be in increasing the sample size for economic evaluation and overcoming the limitations of previous analyses. We examined the feasibility of meta-analysis of economic data, using individual patient data on healthcare utilisation from trials of counselling in primary care.
Methods
Studies were identified from a Cochrane systematic review. Seven trials met the Cochrane eligibility criteria.9 Three studies were excluded after examination; two reported data on healthcare utilisation, but neither provided adequate detail.10–12 The randomisation procedures of these studies were of poor quality and likely to be associated with bias.9 The third trial is yet to report economic data but also used a comparison group of antidepressant treatment prescribed by a general practitioner rather than usual care by a general practitioner.12 This trial was also excluded, given that management by a general practitioner (and associated costs) were likely to be significantly different. Table 1 lists the included studies.4–7
Pooling data using meta-analysis requires studies to be broadly comparable, but the degree of legitimate difference is contentious.13 All four studies used counsellors meeting the criteria for accreditation by the British Association for Counselling and Psychotherapy, although the exact models of counselling differed (for example, non-directive, psychodynamic). In three studies, general practitioners recruited patients, whereas in the fourth patients were screened, and the trial was restricted to patients with chronic problems of six months or more. The time of follow up varied from three to 12 months.
Identification of comparable data
Costs
The perspective taken for the analysis was that of the healthcare system, and only direct healthcare costs were analysed. Available data on the utilisation of health care varied, but all trials included consultations with a general practitioner, prescriptions for psychotropics, and mental health referrals, which are the sources of cost most likely to be influenced by the provision of counselling (see table 1). Patient specific data were collected from medical records and patient self report.
Consultations—Not all trials collected data on all types of consultations in primary care, so consultations had to be limited to surgery attendances only, whereas other types of consultations, such as home visits or appointments with a practice nurse, were excluded.
Drugs—Recording of data on drugs varied. Two studies reported exact dosage and duration of use of all drugs, which allowed calculation of number of tablets prescribed and a cost per tablet to be applied. The remaining two trials recorded only whether a prescription for relevant psychotropics had been given. Standard duration of use and dosage for tricyclics, selective serotonin reuptake inhibitors, and anxiolytics were therefore imputed. Data on other drugs were excluded.
Mental health referrals—Data on mental health referrals also varied and were recoded as the number of attendances at four categories of services: inpatient psychiatry, outpatient psychiatry (including community mental health teams and psychotherapy), practice based psychological therapy (for example, psychologists and counsellors in primary care), and other providers of psychological therapy in community and voluntary groups, such as Relate and self help groups. Practice based psychological therapy (provided to patients in both groups as part of usual care by a general practitioner) was differentiated from protocol therapy (provided to the experimental group only). It was assumed that all protocol therapy sessions were completed in the first six months of the trial.
Data collection periods—The time periods covered by data collection varied (see table 1). Where required, costs were adjusted to represent standard periods of six months, allowing comparative analysis in the short term (six months after baseline) for all four trials and in the long term (12 months after baseline) for three trials. These adjustments assumed that service use remained constant over time and were applied to all variables apart from protocol therapy, since the length of such therapy in all trials was fixed.
Effectiveness In three studies the Beck depression inventory was the primary outcome measure.4 5 7 14 As with the cost data, clinical outcome data were collected at different periods (three, four, and six months for short term data and nine and 12 months for long term data). However, no attempts were made to adjust these data to standard periods of six months. The results may thus be subject to bias if differential outcomes between treatments change over time. Only one trial included a generic health status preference instrument.4
Calculation of costs
We applied standardised national unit costs for the financial year 1999–2000 to the data on utilisation of healthcare services (table 2). The costs of drugs were based on the British National Formulary, using the cheapest alternative available.15 Costs of protocol therapy and practice based counselling services were based on salary figures provided by a relevant national organisation (Counsellors and Psychotherapists in Primary Care). All relevant costs incurred by employers were included, plus capital and direct overheads (data not shown). All other costs were based on figures from the Personal Social Services Research Unit.16 Discounting was unnecessary as costs were all within the same time period and were not recorded beyond 12 months.
Data analysis
Not all trials reported long term data or usable effectiveness outcomes, and there were missing data in all studies. Therefore, the data available varied for analyses of costs alone, effectiveness alone, and cost effectiveness. Missing data on costs in the long term were not associated with randomised group, sex, or employment status in any trial but were associated with baseline score on the Beck depression inventory, age, and marital status in one trial in each case. Missing data on effectiveness were imputed using the last observation carried forward. Figure 1 shows the data available for the analyses.
Data on costs and outcomes were pooled separately in a fixed effects meta-analysis.17 This combines estimates of treatment effects from the individual trials into an overall weighted average, with weights given by the inverse of the variance of the difference in means for each trial. A fixed effects analysis was preferred because of the small number of trials available; however, these results were compared with those from random effects models. A test for heterogeneity was also conducted, although this may lack power given the small number of trials available. Analyses were conducted separately for the total direct costs and the total primary care costs, to enable primary care commissioners to assess the economic impact on their sector alone. All analyses were carried out with Stata (Release 7).
Meta-analysis assumes that the overall treatment effect is normally distributed. Costs were not normally distributed, but the validity of the results was confirmed using non-parametric bootstrapping.18 Such an approach was preferred to transformation or non-parametric tests, as it allows inference about the population mean.19
Relative cost effectiveness was described using incremental cost effectiveness ratios, the ratio of differential average costs of the two interventions to the differential average effects.20 21 Cost effectiveness acceptability curves were also calculated for both the short term and the long term. These plot the probability that counselling is cost effective compared with usual care by a general practitioner for a range of possible maximum values (the ceiling ratio) a decision maker might be willing to pay for an improvement in the Beck depression inventory score. The curves are based on an estimate of the joint distribution of difference in mean cost and difference in mean Beck depression inventory score obtained using bootstrapping.20–22 These curves incorporate the uncertainty that exists around the estimates of mean costs and effects as a result of sampling variation and uncertainty on the maximum cost effectiveness ratio that a decision maker would consider acceptable.
Results
Cost outcomes
The main analysis of costs over the long term indicated that counselling was associated with significantly greater total direct costs per patient (weighted difference in means £110 ($177; €154), 95% confidence interval £38 to £182; fig 2) and primary care costs per patient (£146, £110 to £183) than usual care by a general practitioner. No evidence was found of heterogeneity (total costs P=0.70; primary care costs P=0.83), and random effects analysis gave similar results.
In the short term, counselling was again associated with significantly greater total direct costs per patient (£92, £57 to £126; fig 3) and primary care costs per patient (£135, £114 to £156) than usual care by a general practitioner. Some evidence was found of heterogeneity in the analysis of primary care costs (χ2=7.2, df=3, P=0.07). Interpretation is not substantially different if based on the results of random effects analysis (£133, £99 to £167).
Sensitivity analysis
Examination of the cost components indicated that the costs of consultations with a general practitioner and costs of protocol therapy were the main drivers of direct costs. Therefore, post hoc sensitivity analyses were conducted, systematically changing these variables.
Duration of general practitioner consultations—It has been suggested that referral to a counsellor may reduce the time general practitioners spend consulting with patients, as well as the overall rate of consulting.23 Therefore the effect of selectively increasing the duration of consultations in the group receiving usual care was tested in a threshold analysis. The significant differences found in total direct costs in the long term became non-significant (£72, -£1 to £145) when the duration of a consultation for the control group was increased to 11.75 minutes, equivalent to a 26% increase in the cost of a consultation.
Cost of protocol therapy—The cost of protocol therapy was based on the average of three different grades, ranging from standard counsellors to those in more senior management positions. It is likely that more counsellors in primary care are paid at the lower grade, in contrast to the counsellors employed within the trials included in our study. When costs for counselling sessions were reduced to those of the least expensive counsellor, the differences between the groups in total direct costs in the long term became non-significant (£69, -£1 to £139).
Effectiveness outcomes
Counselling produced superior scores on the Beck depression inventory in the short term (1.93, 0.14 to 3.71) but not in the long term (1.16, −0.65 to 2.97). Some evidence was found of heterogeneity in the short term data (χ2=7.3, df=2, P=0.03). The results using random effects analyses were similar.
Cost effectiveness analysis
The analysis of cost effectiveness was based on patients with data available on both cost and effectiveness. The incremental cost effectiveness ratio for counselling compared with usual care by a general practitioner over the long term was £196 per one point improvement on the Beck depression inventory (counselling minus usual care incremental mean cost £110, incremental mean effect 0.56). Figure 4 illustrates the uncertainty associated with the costs and effects of the two treatments in the long term, and shows that for willingness to pay values above £196, counselling has a greater than 50% probability of being cost effective compared with usual care by a general practitioner. The probability of counselling being more cost effective than usual care stabilises at about 69% for willingness to pay ratios greater than £2000.
The incremental cost effectiveness ratio for counselling compared with usual care by a general practitioner in the short term was £50 per one point improvement on the Beck depression inventory (counselling minus usual care incremental mean cost £109, incremental mean effect 2.16).
Discussion
Meta-analysis of individual patient data can be used to overcome the sample size limitations often associated with economic analyses. The analyses may, however, be limited in several ways. The need to identify comparable data and apply standardised costs means that detail in individual studies may be lost. For example, non-psychiatric referral and drug costs were ignored, as were distinctions between particular services, such as between outpatient psychiatry and community mental health teams. The lack of data on costs of lost production meant that advantages associated with either treatment in returning people to work had been ignored. However, none of these issues may be as central to the effects of counselling (or as likely to show effects) as the measures of depressive symptoms and utilisation of mental health services.
We found significant variation between trials in standard deviations of costs. This may reflect more comprehensive estimates of service use in some trials, or differences between trials in the availability of certain services (for example, treatment for mental health outside the trial). Such differences have important implications for standard meta-analysis approaches, where estimates are weighted inversely by their assumed precision. Cost sources omitted or imputed in the current analysis would have also increased variance further if they had been measured. Adjusting costs to represent common time periods assumes service use is linear over time, when it may be that service use (such as consultation rates and drug use) reduces when an effective therapy is provided.
Such methodological compromises mean that the present analysis can never approach the precision of primary data collection. However, it is a matter of debate (and economics) whether this additional precision is worth the extra cost and time delay required to fund an adequately powered economic analysis, or whether secondary analysis provides a reasonably accurate estimate to inform policy and practice.
One compromise in future research is to attempt to ensure greater comparability in methodology in studies examining common cost effectiveness questions, through the use of the same outcome measures and the collection of data on service use on the same range and type of services. The prospective registration of trials might facilitate this. Researchers in the United States have gone further in explicitly preplanning meta-analyses of different interventions, deliberately standardising across studies where possible and identifying methods to deal with methodological heterogeneity.24
Several statistical issues also arise in meta-analyses of cost data. Specific features of economic outcomes (for example, skewness in the distribution of cost data) mean that further investigation into appropriate statistical methodology is needed.
Costs are affected particularly by missing data.25 Analysis of individual patient data allows imputation of missing values through several methods. Recent approaches that more accurately reflect the uncertainty associated with imputation of missing data have been described but were beyond the scope of our study.25 Data were also lost for the cost effectiveness analysis owing to use of different effectiveness measures in trials. In cases where outcomes are conceptually the same but measured on different scales, the meta-analysis approach of standardised differences might be used, allowing all trials to be included in the analysis.
Cost effectiveness of counselling
Caution should be exercised in drawing specific conclusions about the cost effectiveness of counselling, given that the validity of the current methodology is unclear. However, the main results indicate that the costs associated with counselling were higher than those with usual care by a general practitioner, which supports the argument that previous analyses were underpowered to detect these effects.2 The interpretation of figure 4 is complex, because typical ceiling ratios for a one point change in scores on the Beck depression inventory are not known. Such ratios are often assigned to quality adjusted life years (QALYs), but methods for translating Beck depression inventory scores into QALYs are crude at present.26
What is already known on this topic
Issues of cost effectiveness are an increasingly important part of clinical trials
Trials powered on clinical outcomes may be underpowered to detect important economic effects because of variability associated with data on costs
What this study adds
It was feasible to pool cost data from several trials to overcome the sample size limitations of previous studies on counselling in primary care
Previous studies may have been unable to detect differences in costs between counselling and usual care by a general practitioner
Results depended on assumptions made about costs of counsellor sessions and management of patients by a general practitioner
Given these limitations, the main analysis of differences in costs alone may be easier to interpret. These differences were found to be sensitive to increases in the duration of consultations with a general practitioner in patients not referred for counselling. However, no trials reported objective measures of duration of consultation, and thus this effect is speculative.
Results were also sensitive to the costs of counselling sessions. Tension exists between the desire to provide high quality counselling services by employing experienced (and thus more expensive) counsellors and the need to ensure that session costs do not jeopardise cost effectiveness. This has implications for other psychological therapies in primary care, such as cognitive behavioural therapy, where session costs may be even higher.4 Our findings may provide an additional stimulus to the development of “minimal” interventions, which reduce or remove the need for the therapist, if effectiveness can be maintained.26 27
Footnotes
-
Characteristics of included patients appear on bmj.com We thank Brendan Delaney, the referee, for identifying an important technical error in the first submission of the paper.
-
Contributors PB devised the study and wrote the paper and all authors commented on drafts; PB will act as guarantor for the paper. PB and SB wrote the protocol and performed the initial analysis, with additional economics advice from JBe. JBe, SS, RC, KF, MK, and IH provided data for the study and assisted in the development of the final database. JBa provided statistical advice and performed the bootstrapping and cost effectiveness acceptability curve analyses.
-
Funding PB is funded by the National Primary Care Research and Development Centre at the University of Manchester, through the Department of Health.
-
Competing interests None declared.