Identifying evidence for decision-analytic models

Suzy PaisleyDoH Research Scientist in Evidence Synthesis

Consensus Working Group on the Use of Evidence in Economic Decision Models : Establishing the Current Situation

MRC HSRC Workshop 25 July 2005

Consensus Working Group on the Use of Evidence in Economic Decision Models

DoH Award

To develop a systematic approach to searching for evidence to inform decision-analytic models

Characteristics and sources of evidence to inform parameter values

Use of evidence in informing other parts of the modelling process

Consensus Working Group on the Use of Evidence in Economic Decision Models

Today’s presentation

Why an issue Current practice

– Identification of evidence– Use of evidence

Implications of current practice

Consensus Working Group on the Use of Evidence in Economic Decision Models

Why an issue?

Environmental issues Own experience

– Undertaking HTA searches– Peer review of HTA reports

Consensus Working Group on the Use of Evidence in Economic Decision Models

‘Typical’ HTA

Decision problem / research question Syntheses of existing evidence Systematic review of clinical-

effectiveness Decision-analytic model of cost-

effectiveness

Consensus Working Group on the Use of Evidence in Economic Decision Models

Decision problemsearch question

P = colorectal cancer

I = irinotecan

C = any relevant

O = survival

#1 colorectal

#2 irinotecan

#3 rct

#4 #1 and #2 and #3

Consensus Working Group on the Use of Evidence in Economic Decision Models

Modelling decision problems

Effectiveness Costs Resource use / activity Health states Utility values Indirect comparators Longer term outcomes ‘Other’ interventions Natural history Epidemiology

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice

Analysis of 27 technology assessment reports (TARs)

Review of HTA guidelines Content analysis of modelling guidelines

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:

Reports of models

Use evidence which cannot be identified systematically by PICO focused searches(21/27 (78%) assessments for NICE Paisley, 2001)

Report PICO focussed searches(4/21 (19%) reported additional ‘ad hoc’ searches Paisley, 2001)

Issues of transparency

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:

Reports of modelsTypes

EffectivenessCostsResource use / activityHealth statesUtility valuesIndirect comparatorsLonger term outcomes‘Other’ interventionsNatural historyEpidemiology

Sources

‘Published papers’Routine dataReference sourcesLocal / clinical /

expert opinionSponsor submissions

Uses

Parameter valuesModel structureSensitivity analysisValidation / consistency / calibration

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:

Methodological guidance

HTA Guidelines Modelling guidelines

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:

HTA guidelines (n=32)

Identification of evidence:

No mention 13 (41%)Systematic review approach 13 (41%)Broad approach 2 (6%)Approach unclear 2 (6%)Not available 2 (6%)

Name data sources 21 (66%)

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:

Modelling guidelines

ISPOR Principles of Good Practice (Weinstein et al, 2003)

1 explicit reference to systematic reviews of the literature

41 implicit references to a process of seeking evidence

30/41 (73%) references form statements within recommendations

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:

Modelling guidelines

Model structure 10 (33%)

Data identification 7 (23%)

Data modelling 7 (23%)

Data incorporation 0 (0%)

Model validation 6 (20%)

Consensus Working Group on the Use of Evidence in Economic Decision Models

Current practice:

Conclusions

Reports of models– Report PICO focussed searches– Use a wide range of evidence from a wide

range of sources throughout the modelling process

Modelling and HTA guidelines– Give limited explicit guidance– Give an implicit indication that range of

sources should be used throughout the modelling process

Consensus Working Group on the Use of Evidence in Economic Decision Models

Implications for searching:

Types of evidence

Effectiveness Costs Resource use / activity Health states Utility values Indirect comparators Longer term outcomes ‘Other’ interventions Natural history Epidemiology

Defining search questions

Defining the evidence base

Transferable / ‘proxy’ data

Grouping questions

Consensus Working Group on the Use of Evidence in Economic Decision Models

Implications for searching:

Sources of evidence

‘Published papers’ Routine data Reference sources Local / clinical /

expert opinion Sponsor

submissions

Prioritising appropriate sources

Consensus Working Group on the Use of Evidence in Economic Decision Models

Implications for searching:

Uses of evidence

Parameter values Model structure Sensitivity analysis Validation /

consistency / calibration

Iterative nature of modelling process

Matching retrieval of evidence to use of evidence

Consensus Working Group on the Use of Evidence in Economic Decision Models

Other issues

Aim of the search– What evidence?– What for?– How much?

Methods of selecting evidence Alternative search methods Reporting standards

Consensus Working Group on the Use of Evidence in Economic Decision Models

Conclusions

The process of identifying evidence for models needs to take account of the following issues:

The range of types, sources and uses of evidence The process of identifying relevant evidence bases

and defining search questions The iterative nature of model development How much evidence is required Reporting standards

Consensus Working Group on the Use of Evidence in Economic Decision Models

Conclusions

Identifying evidence for models is different to identifying evidence for systematic reviews.