1

BISCUIT FINAL REPORT

December 2012

Dr Lesley Wye

Research Fellow

Centre for Primary Health Care

School of Social and Community Medicine

University of Bristol

lesley.wye@bristol.ac.uk

2

Executive summary

Increasingly, NHS commissioners need information about the costs of services to

help inform their decision making. Currently, commissioners rely almost entirely on

crude cost data to evaluate the economic impact of services, if any cost data are

available at all.

The aim of the BISCUIT study was to pilot approaches to economic evaluation of

homeopathic packages of care delivered at Bristol Homeopathic Hospital. We

wanted to test the methods and outcome tools to facilitate the development of a

larger study that could investigate the differences in resource use between those

who use NHS homeopathic packages of care and those who do not. Our research

question for this feasibility study was:

What are the best methods and best outcome tools for undertaking a large

scale economic evaluation of a cohort of homeopathic patients compared with

a matched control group not using NHS homeopathy?

To answer this, we used a cost utility model and carried out the first steps of a

discrete choice experiment.

Using a prospective matched controlled cohort design, we recruited 29 Bristol

Homeopathic Hospital patients from 315 possible candidates, but only received the

relevant permissions to access medical records and subsequently found matched

controls for nine. Nineteen matched controls returned data.

Qualitative interview data was collected to learn about resource use, experiences of

the study and views on the outcome tools for 23 participants. Quantitative data was

collected through GP medical record extraction of data such as GP and practice

nurse consultations, medications, hospital consultations, tests and investigations. In

addition, participant questionnaires were administered to capture wellbeing (Warwick

Edinburgh Wellbeing tool), quality of life (SF-36) and resource use (Client Service

Resource Inventory). This latter tool collected data on personal costs such as private

healthcare, over the counter products, travel and lost earnings. Qualitative data were

analysed using qualitative content analysis and a framework approach. Descriptive

analyses were carried out on the quantitative data by an independent statistician.

Key findings around feasibility were:

1. There were two major challenges in this feasibility study. The first was the low

level of recruitment of case and control participants. The approach adopted

whereby case participants were recruited directly from the Bristol Homeopathic

Hospital without prior recruitment of their GP practice meant that the pool of

potential candidates was substantially reduced when GP practice permission was

3

not forthcoming. The second challenge was achieving good enough matching of

cases and controls, which also affected the recruitment of controls.

2. Because only 9 of the original 315 potential candidates from the Bristol

Homeopathic Hospital were matched with controls, representativeness is

questionable.

3. The outcome tools were feasible, acceptable, sensitive to change and

appropriate.

4. Cases and controls were well matched at baseline for quality of life and

wellbeing, but not for resource usage.

5. Quality of life for case participants improved compared to controls. Wellbeing for

case participants also changed significantly for cases compared with controls.

However because this was not a randomised controlled study, we do not know if

the improvements seen were due to the Bristol Homeopathic Hospital service.

6. At baseline, costs were substantially higher for case participants. There was no

difference in resource utilisation in the 12 months after baseline between cases

and controls.

7. If these study results were to apply to other homeopathic case and matched

control populations, then 90 participants (1:1 matched) would be needed in a

larger study to demonstrate that the quality of life and wellbeing improvements

generated by homeopathic packages of care were worth their cost. But because

case and control participants were not well matched at baseline for resource use,

we do not have complete confidence in this figure.

8. Individual variability in resource utilisation for both cases and controls was

substantial.

Given these findings, results should be taken with caution.

In answering our research question, we found that the prospective controlled cohort

design produced difficulties in recruitment and matching case and control

participants. But the outcomes tools of the SF-36, Warwick Edinburgh Wellbeing and

Client Service Resource inventory were acceptable and useful.

The other aspect of this study was to identify key attributes of homeopathic

packages of care for a discrete choice model. From interviews with the Bristol

Homeopathic Hospital participants, we found the most valued aspects of the

treatment were:

positive impact on health

the questions that the homeopaths asked

personalised approach

The most valued aspects of the service were:

extended length of the consultation

convenient location of the clinic

not having to pay for the service

4

In thinking about designing a larger study, we considered the use of a traditional

randomised controlled trial design, which would address known and unknown

confounders arising in through matching and lead to the possibility of uncovering a

causal connection between the Bristol Homeopathic Hospital service and any

changes brought over time. But this type of methodology is difficult to use in

evaluating a long-standing service (rather than a treatment) as it does not allow for

real life decision-making on behalf of patients. There are also ethical issues about

denying patients access to an existing, publically funded service, once patients have

requested referral.

An alternative could be the use of a Zelens design. In this type of design, the top 20

practices referring into the Bristol Homeopathic Hospital would be recruited first,

thereby avoiding the experience in this study of the loss of 40% of participants from

GP practice refusal. Then, we would recruit a cohort of patients experiencing the top

five conditions treated by the Bristol Homeopathic Hospital from these 20 practices.

As patients are referred into the Bristol Homeopathic Hospital and recruited, they

would be matched with community controls from any of the 20 practices on the basis

of condition, symptom, age within 5 years and a resource use proxy measure of GP

consultations (within 2 consultations) in the past year.

The advantages of this approach is that it reflects real life by not tampering with

patient decision-making and would hopefully lead to much closer matching at

baseline between case and control participants. The disadvantages are that without

randomisation, known and unknown confounders could affect any observed changes

over time and there is no blinding. Moreover, we know that individual doctors do vary

substantially in their referral patterns, which would impact on patient resource use,

and this needs to be taken into account in this design. Nonetheless, this Zelens

approach would be a step closer to developing a robust study of economic

evaluation of the Bristol Homeopathic Hospital.

Although undeniably challenging, the importance of designing good quality economic

evaluations for generalist services like the Bristol Homeopathic Hospital is important.

Therefore, we would encourage future researchers to work through the

methodological difficulties of conducting economic evaluations of services to ensure

NHS commissioners - and the wider public - have the information necessary to make

appropriate judgments.

5

Acknowledgements

We thank our funders, Blackie Foundation, the British Homeopathic Association,

South West GP Trust and United Hospitals Bristol Above and Beyond Fund.

We thank all of the case and control participants who have taken the time to respond

to questionnaires. We also would like to thank the GP practice staff who gave

permission to access patient records and those who carried out the searches. In

addition, Susan Barron, departmental manager at the Bristol Homeopathic Hospital,

and the Homeopathic Hospital staff were instrumental in recruiting case participants.

Vicky Burrows, a fifth year medical student, dedicated a period of self-study to help

set this project up. Patricia Martens continued with the administrative tasks. Hallam

Wye checked data entry. Tom Griffin was the independent statistician. Lorna Duncan

conducted the majority of participant interviews and acted as research assistant.

Thanks also go to our steering group and wider advisors. The steering group

consisted of Dr Elizabeth Thompson, lead clinician at the Homeopathic Hospital,

Professor William Hamilton (Peninsula Medical School) and Professor Debbie Sharp

(University of Bristol). Many economists contributed their time and expertise

including Dr