dr. stavros petrou presentation to nottingham ctu 19 th june 2008
DESCRIPTION
Economic evaluations alongside clinical trials: what they contribute, how they are performed and their limitations. Dr. Stavros Petrou Presentation to Nottingham CTU 19 th June 2008. What is economic evaluation?. Premise: scarce (health care) resources - PowerPoint PPT PresentationTRANSCRIPT
Economic evaluations alongside clinical trials: what they contribute, how they are performed and their
limitations
Dr. Stavros Petrou
Presentation to Nottingham CTU
19th June 2008
What is economic evaluation?
• Premise: scarce (health care) resources• Aim: to maximise health gain with the available
resources• Method: compare cost and consequences of
interventions• Balance: about costs and consequences, inputs
and outputs• Economic evaluation: explicit criteria for making
choices.
Definition of economic evaluation
• Definition of economic evaluation:
“The comparative analysis of alternative courses of action in terms of both their costs and their consequences” (Drummond et al, 1997)
• Requires:– a comparison of two or more alternatives– examination of both costs and
consequences
Types of economic evaluation
NOCosting studyYESIs effectiveness of interventions equal?
YESNOCost minimization study
Can all outcomes be valued in monetary terms ( e.g. willingness to pay)?
YESCost benefit analysis
NO
Can outcomes be measured as quality adjusted life years? YES
NOCost-effectiveness analysis
Cost-utility analysis
Is there good evidence on effectiveness of interventions being compared?
Economic evaluation alongside trials
Two independent groups
Control group Treatment group
Patient (Cost,
Effect)
1 ( CC1, EC
1 )
2 ( CC2, EC
2 )
3 ( CC3, EC
3 )
.
.
nC ( CCn, EC
n )
Mean: ( CC, EC )
Patient (Cost,
Effect)
1 ( CT1, ET
1 )
2 ( CT2, ET
2 )
3 ( CT3, ET
3 )
.
.
nT ( CTn, ET
n )
Mean: ( CT, ET )
CT - CC
ET - EC
Economic evaluation alongside trials
Two independent groups
Control group Treatment group
Mean: ( CC, EC ) Mean: ( CT, ET )
Incremental cost-effectiveness ratio
The cost-effectiveness plane
C
New treatmentmore costly
New treatment more effective
New treatmentless effective
New treatmentless costly
NENW
SW SE
Existing treatmentdominates
New treatment dominates
New treatment less costlybut less effective
New treatment more effectivebut more costly
New treatment cost-effective
New treatmentcost-ineffective
The cost-effectiveness plane
C
New treatmentmore costly
New treatment more effective
New treatmentless effective
New treatmentless costly
NENW
SW SE
Maximum acceptable ICER
NICE: method of operation
• Preferred measure of cost-effectiveness:– Quality-adjusted life year (QALY)– Alternatives - e.g. cost per life year gained -
acceptable
• No absolute threshold for level of acceptability:– no empirical basis for setting a value– may in some circumstances want to ignore threshold– A set threshold implies efficiency has absolute priority
over other objectives (e.g. fairness)– many technology suppliers are monopolies; a
threshold would discourage price competition
Cost per QALY results from NICE
Those in bold: rejected
Source: N Devlin, D Parkin. Does NICE have a cost-effectiveness threshold and what other factors influence its decisions? A binary choice analysis. Health Economics 2004: 13(5):437-52.
Why QALYs as a measure of outcome?
• To use cost-effectiveness as a guide to decision-making, we need to compare the c-e of different uses of resources
• Therefore we need an effectiveness measure that can be used in a wide range of settings:
• Life-years gained– but only where survival is main outcome
• Quality adjusted life years (QALYs)– Composite of survival and quality of life
Time in years t
01 2 3 4 5 6
1.0
Health profile with intervention
Health profilewithout intervention
QALYs gained
Qualityof life
valuationu
The EuroQol EQ-5DThe following questions are designed to tell us about your state of health today. Please look at each group of questions, and then tick the statement which best describes youown health state today. You should make five ticks in all, one for each group.
MobilityI have no problems walking about ____I have some problems walking about ____I am confined to bed ____
Self-careI have no problems with self-care ____I have some problems washing or dressing myself ____I am unable to wash or dress myself ____Usual activitiesI have no problems performing my usual activities ____I have some problems with performing my usual activities ____I am unable to perform my usual activities ____Pain/discomfortI have no pain or discomfort ____I have moderate pain or discomfort ____I have extreme pain or discomfort ____Anxiety/depressionI am not anxious or depressed ____I am moderately anxious or depressed ____I am extremely anxious or depressed ____
Tariffs for the EuroQol EQ-5D Coefficient
Constant 0.081
Mobility
- Some problems 0.069
- Confined to bed 0.314
Self care
- Some problems 0.104
- Unable to wash/dress 0.214
Usual activities
- Some problems 0.036
- Unable to perform 0.094
Pain/discomfort
- Moderate 0.123
- Extreme 0.386
Anxiety/depression
- Moderate 0.071
- Extreme 0.236
N3 (Level 3 at least once) 0.269
Why randomised trials?
• Most treatments do not have large effects; reliably detecting moderate effects requires studies that simultaneously avoid:– moderate bias
• proper randomisation• intention-to-treat analysis• avoidance of inappropriate sub-group analysis
– moderate random error• adequate size
Why economic assessment in clinical trials?
• Many health economists advocate models using lots of data sources: trial, non-trial, summary data etc
• But...issues of bias and random error also affect incremental resource use and health outcomes
• And, trials provide patient-level data, useful for:– Dealing with patient heterogeneity– Examining covariance, e.g. between costs and outcomes – Building and validating models, eg to extrapolate
• Trials allow prospective measurement of resources & outcomes of interest
• Incremental cost of economic evaluation alongside trials is low
UK Collaborative ECMO Trial
• Pragmatic RCT• 185 mature (35 weeks, 2kgs) infants with
severe respiratory failure (ox. index 40) • Infants recruited from 55 centres in 1993-5• Randomisation to ECMO: 1 of 5 specialist
centres, cannulated, ECMO support• Randomisation to CM: conventional care• Outcomes: survival without severe disability up
to 7 years of age
Design and analytical issues
• Costs: measurement and valuation
• Consequences: measurement and valuation
• Analytical issues: within and beyond RCTs
Perspectives and types of costs • Direct costs
– Health care system– Other care inputs, e.g. social services– Patient, family, carer expenses
• Informal care costs– Opportunity cost of unpaid informal care
• Indirect costs– Time off work, reduced productivity– Early retirement– Premature mortality
• Transfer payments– Payments such as social security benefits that
redistribute output with no exchange of goods or services
Three elements of cost
• Resource use (cost generating event)– a day in hospital– a GP consultation
• Unit cost– cost per in-patient day / per hospitalisation– cost per GP consultation / per GP minute
• Cost– the product of resource use and unit costs
Measurement of resource use
• All significant resource inputs during first 7 years of life• Trial data collection forms:
– transport (mode, distances)– duration and intensity of neonatal care
• Observational research for infant death• Postal questionnaires, validated by information from
hospital records:– post-discharge hospital and social service
utilisation• GP records:
– community service utilisation
Valuation of resource use
• Unit costs employed to value resource use
• Neonatal care – top down methodology
• Readmissions - Reference cost schedules
• Community service utilisation - Published cost data (previous studies, PSSRU, etc.)
• Drugs – BNF
• £, 2002-3 prices
Time horizon and discounting
• Should extend far enough into the future to capture all costs and consequences of interventions being evaluated
• ECMO Study – time horizon initially reflected that of RCT
• Costs (and consequences) occurring beyond the first year of life must be reduced to present values
• Rationale for discounting- Time preference- Opportunity cost – market basis
Mean costs and mean cost differences
Cost category ECMO (n=93) CM (n=92)
Mean (SD) Mean (SD) Meandifference
P value
Death 434 (665) 846 (714) -412 <0.0001
Transport 2147 (3080) 296 (777) 1851 <0.0001
Initial hospitalisation 22996 (21618) 6143 (13144) 16853 <0.0001
Hospital readmissions 1518 (3868) 1127 (3063) 391 0.447
Outpatient hospital care
970 (1592) 496 (1008) 474 0.017
Community care 1976 (2882) 1247 (2647) 792 0.075
Other costs 229 (1255) 74 (182) 155 0.241
Total costs 30270 (24380) 10229 (18356) 20041 <0.0001
Source: Petrou S, Bischof M, Bennett C, Elbourne D, Field D, McNally H. Cost-effectiveness of neonatal ECMO based on seven year results from the UK Collaborative ECMO Trial. Pediatrics 2006; 117(5): 1640-1649.
Bootstrap mean cost differences
Cost category Mean cost difference
(ECMO – CM)
Bootstrap mean cost difference (95% CI)
Death -412 -406 (-614, -209)
Transport 1851 1849 (1251, 2497)
Initial hospitalisation 16853 16,826 (11,775, 22,044)
Hospital readmissions 391 371 (-603, 1334)
Outpatient hospital care 474 481 (97, 875)
Community care 792 730 (-62, 1549)
Other costs 155 156 (-13, 443)
Total costs 20041 20057 (13690, 26318)
Source: Petrou S, Bischof M, Bennett C, Elbourne D, Field D, McNally H. Cost-effectiveness of neonatal ECMO based on seven year results from the UK Collaborative ECMO Trial. Pediatrics 2006; 117(5): 1640-1649.
Measurement of outcomes at 7 years
• Survival period → life years gained
• Neurodevelopmental assessments performed by developmental psychologist across 6 domains:– cognitive ability - behaviour– neuromotor skills - hearing– general health - vision
• Each domain defined as normal, impaired or mild, moderate or severely disabled
• Overall status defined by highest degree of impairment or disability in any domain
→ disability-free life years gained
• Limitations of QALYs in childhood context
ECMO(n=93)
CM(n=92)
Deaths
Before discharge 28 (30.1%) 54 (58.7%)
Between discharge and one year 2 (2.2%) 0 (0.0%)
Between one years and four years 1 (1.1%) 0 (0.0%)
Between four years and seven years 0 (0.0%) 0 (0.0%)
Total 31 (33.3%) 54 (58.7%)
Loss to follow-up
Between discharge and one year 1 (1.1%) 1 (1.1%)
Between one year and four years 2 (2.2%) 2 (2.2%)
Between four years and seven years 3 (3.2%) 1 (1.1%)
Total 6 (6.5%) 4 (4.3%)
Final assessment at seven years of age
Severe disability 3 (3.2%) 0 (0.0%)
Moderate disability 9 (9.7%) 6 (6.5%)
Mild disability 13 (14.0%) 11 (12.0%)
Impairment only 21 (22.6%) 15 (16.3%)
No abnormal signs or disability 10 (10.8%) 2 (2.2%)
Known survivors with no disability 31/56 (55.4%) 17/34 (50.0%)
Cost-effectiveness of neonatal ECMO
Incremental cost-effectiveness ratio = C / E
= £13,385 per life year gained
or
= £23,566 per disability-free life year gained
NB: Natural or physical unit of outcome, which ignores full range of consequences.
Cost-effectiveness plane, life years gained
-£40,000
-£30,000
-£20,000
-£10,000
£0
£10,000
£20,000
£30,000
£40,000
-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0
Incremental life years gained
Incr
emen
tal c
osts
ECMO more effective, more costly
ECMO more effective, less costly
ECMO less effective, less costly
ECMO less effective, more costly
maximum ICER
Cost-effectiveness plane, disability-free life years gained
-£40,000
-£30,000
-£20,000
-£10,000
£0
£10,000
£20,000
£30,000
£40,000
-4.0 -3.0 -2.0 -1.0 0.0 1.0 2.0 3.0 4.0
Incremental disability-free life years gained
Incr
emen
tal c
osts
ECMO more effective, more costly
ECMO more effective, less costly
ECMO less effective, less costly
ECMO less effective, more costly
maximum ICER
Cost-effectiveness acceptability curves, probability that neonatal ECMO is cost-effective after seven years
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
£0 £10,000 £20,000 £30,000 £40,000 £50,000
Willingness to pay threshold (£)
Prob
abili
ty co
st-e
ffect
ive
Life year gained Disability-free life year gained
0.69
0.98
Mean net benefits of neonatal ECMO
Mean net benefits = Rc.E - C
Rc Life year gained Disability-free life year gained
Mean net benefit
95% CI Mean net benefit
95% CI
0 -20,130 (-26,027, -13,522) -19,949 (-25,997, -13,375)
10,000 -5,299 (-14,475, 3,750) -11,387 (-20,394, -244)
20,000 9,532 (-6,691, 25,351) -2,825 (-18,377, 13,804)
30,000 24,362 (1,474, 47,314) 5,737 (-16,245, 29,393)
40,000 39,193 (8,926, 69,513) 14,299 (-14,863, 44,781)
50,000 54,024 (15,496, 91,567) 22,861 (-13,246, 60,319)
Are trial-based economic evaluations sufficient?
• Under the right circumstances, Yes
• Like clinical evidence, economic evidence can stand alone or be synthesised
• Requirements: reasonable comparators, adopts final outcomes, collects data on a sufficiently broad set of services, adequately powered, sufficiently long follow-up, representative patient population
• Problems: Use of inappropriate statistical tests, lack of power, failure to handle missing data, lack of intention-to-treat analysis, follow-up too short, lack of transferability
Was the ECMO trial-based economic evaluation sufficient?
• Reasonable comparator • Representative patient population • Intention to treat analysis • Adequately powered • Appropriate perspective • Sufficiently long follow-up x• Adopts final outcomes ?• Appropriate statistical tests • Handled missing data
Longer-term cost-effectiveness
• Simple decision-analytic model• Assumptions:
– restricted to first 18 years of life– survivors to 7 years survive to 18 years– disability status at age 7 does not vary– excess annual costs during years 4-7 continue
during years 8-18
ICER: £7344 per life year gained £11802 per disability-free life year
gained
Vehicles for economic evaluation• Prospective collection of data alongside randomised
controlled trials– Least subject to bias, control over instruments, low incremental
cost. May need to supplement.
• Prospective collection of data alongside non-randomised studies– More subject to possible bias, control over instruments, low cost.
• Retrospective analysis of available data– Low control over design and data, subject to bias.
• Modelling study– Data from different sources, combined in decision-analytic
models. Hard to validate. Useful and often unavoidable adjunct to trials.
Limitations of trials as a vehicle for economic evaluation
Trial limitations
Inappropriate or partial comparisons
More than one trial
Partial measurement
Unrepresentative practice
Intermediate outcomes
Limited follow-up
No trials
NICE Examples
Temozolomide (recurrent malignant glioma)
Drugs for Alzheimer’s
Riluzole (resource use)
Glycoproteins
Beta interferon (MS)
Implantable cardioverter defibrillators
Liquid-based cytology
Modelling
• Hence modelling aims to address all these questions and can be used instead of or as a complement to trial evidence– Structure the economic question– Extrapolate beyond observed data– Links intermediate and final endpoints– Generalizes results to other settings/patient groups– Synthesises evidence and can create head-to-head
comparisons where RCTs don’t exist– Can indicate the need for further research
Models should:
• Represent a simplification of the real world• Encourage decision-makers to be explicit• Reflect current clinical practice and use
appropriate comparators• Be based on the best quality data available• Cover the appropriate time period• Include sensitivity analysis to explore uncertainty
of data inputs and model structure• Be transparent and reproducible• Have internal and external validity
The realities of research funding• A large proportion of funding for economic
evaluation is attached to trials• Three options available to analysts:
– Satisfy expectation of standard trial-based economic evaluation and risk misleading results
– Refuse to collaborate
– Pragmatic collaboration:• Seek opportunities to use modelling to inform trial
design• Ensure sufficient budget for analysis which includes
synthesis and modelling
A pragmatic way forward?
Synthesis andmodelling with
updatedevidence
Setting ofresearchpriorities
Primaryresearch
(e.g. RCTs)
Identifydecisionproblems
Synthesis and modelling
givenavailableevidence
Suggested checklist for assessing economic evaluationsSuggested checklist for assessing economic evaluations
1)Was question well-defined?
2)Were alternatives clearly described?
3)What evidence on effectiveness was used?
4)Were resources measured and valued fully & appropriately?
5)Was discounting necessary and was it performed?
6)Were incremental costs and outcomes analysed?
7)Was an adequate sensitivity analysis performed?
8)Are the results adequate to inform purchasing?
9)Are the conclusions justified?
10) Are the results applicable to the local population? Source: Drummond MF, O’Brien, B, Stoddart GL, Torrance GW. Methods for the economic evaluation of
health care programmes. 2nd edition. Oxford: Oxford University Press, 1997.
Drummond MF, Jefferson T, et al. Guidelines for authors and peer reviewers of economic submissions to the BMJ. BMJ 1996; 313:275-83.