decision and cost-effectiveness analysis: understanding sensitivity analysis training in clinical...

Decision and Cost-Effectiveness Analysis: Understanding Sensitivity Analysis

Training in Clinical ResearchDCEA Lecture 5

UCSF Dept. of Epidemiology & Biostatistics

Jose Luis Burgos, MD, MPH, AAHIVS

March 4, 2009

Project Map

• Think through your research question• Sketch your analysis • Collect data for your model• Adjust model• Run-test your model• Conduct Sensitivity Analysis • Write it up.

Objectives

To understand the purposes of sensitivity analysis.

To understand techniques used for sensitivity analysis.

Why do sensitivity analyses?

• All CEAs have substantial uncertaintyAll CEAs have substantial uncertainty

• Sensitivity analyses deal with that Sensitivity analyses deal with that uncertainty systematicallyuncertainty systematically

• Convince audience results are ‘robust’ – Convince audience results are ‘robust’ – qualitativequalitative findings don’t findings don’t change with small changes in inputschange with small changes in inputs

Sensitivity analysis

Prior lectures reviewed how inputs are determined, Prior lectures reviewed how inputs are determined, plus a few simple sensitivity/threshold analyses.plus a few simple sensitivity/threshold analyses.

This lecture will cover four topics:This lecture will cover four topics:

1. Types of uncertainty 1. Types of uncertainty

2. Deterministic sensitivity analyses (one-way, 2. Deterministic sensitivity analyses (one-way,

multi-way, scenario) multi-way, scenario)

3. Probabilistic sensitivity analysis (Monte Carlo)3. Probabilistic sensitivity analysis (Monte Carlo)

4. Uses of sensitivity analysis4. Uses of sensitivity analysis

Types of uncertainty

1.1. Parameter Uncertainty Parameter Uncertainty • What’s are the correct input values?What’s are the correct input values?

2.2.Methodological uncertaintyMethodological uncertainty

a)a) Model Structure Model Structure

• How values are combined How values are combined

or modeledor modeled

b)b) Model Process Model Process

a)a) Implicit decisions made by the analyst such as Implicit decisions made by the analyst such as

viewpoints or effects consideredviewpoints or effects considered

Addressing Methodological Uncertainty

• Sensitivity Analysis– Scenario analysis with different models to

combine costs and estimate effects

• Statistical Analysis – Where multiple parameter sets available, can test

the fit of different models

Addressing Model Process Uncertainty• Standardized CE analysis – difficult and different panels give

different recommendations, but key common components are (from Drummond): 1. The background of the question2. The viewpoint for the analysis3. The reason for selecting a particular form of analysis4. The population to which the analysis applies5. The comparators being assessed6. The source of the medical evidence and its quality 7. The range of costs considered and their measurement8. The measure of benefit in the economic study (e.g. LY gained, QALYs

gained) 9. The methods for adjusting costs and benefits according to their

timing10. The methods dealing with uncertainty 11. The incremental analysis of costs and benefits12. The overall results of the study and its limitations

Deterministic Sensitivity AnalysesHow does assigning specific different values to inputs change

output?

One-way (‘univariate’): Vary 1 input at a time Multi-way (‘multivariate’): Vary 2+ inputs at a time Scenario (variant of multi-way): Tests set of relevant

conditions. Threshold analysis (one-way or multi-way): Input values

beyond which cost-effectiveness achieved (or lost).

Univariate Sensitivity Analysis

• Examine robustness of ICERs to changes in a single parameter: – ‘best’, ‘high’ and ‘low’ estimates (but experts

consistently underestimate true variability)– Value +/- 1 SD– 95% CI (based on observed or assumed distribution)– ‘clinical meaningful range– Extremes– Threshold analysis

One-way SA – Aneurysm managementOne-way SA – Aneurysm management

Sensitivity analysis (1-way),aneurysm management CUA

$0

$100,000

$200,000

$300,000

$400,000

0 0.005 0.01 0.015 0.02 0.025

Annual risk of rupture

$ / Q

ALY

, clip

ping

Base case estimate

One-way SA

Muennig, 2008

13

One-way SA – LTBI programOne-way SA – LTBI program

Peginterferon Model Inputs: Estimated “base case” and rangePeginterferon Model Inputs: Estimated “base case” and range

European Journal of Gastroenterology & Hepatology. 2007;19:631-638.

LTBI Model Inputs: Estimated “base case” and rangeLTBI Model Inputs: Estimated “base case” and range

Int Jour Tuberc Lung Dis 2008

Univariate Sensitivity Analyses: Univariate Sensitivity Analyses: Base case and range of outcomes for 1,000 IDUsBase case and range of outcomes for 1,000 IDUs

Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008

Automating one-way SAs: Automating one-way SAs: Male circumcision for HIV prevention in South AfricaMale circumcision for HIV prevention in South Africa

$100

$200

$300

$400

Co

st p

er H

IV c

ase

aver

ted

Percentiles of the variables

Protective ef fect

Cost per male circumcision

Multiplier due to epidemic ef fects

Proportion HIV-uninfected

Disinhibition impact on protective ef fect

Frequency of short-term adverse events (outpatient)

Kahn JG, et al. PLoS Med 3(12): e517. doi:10.1371/journal.pmed.0030517

Male circumcision for HIV prevention in South AfricaMale circumcision for HIV prevention in South AfricaConsidering HAART cost avertedConsidering HAART cost averted

Kahn JG, et al. PLoS Med 3(12): e517. doi:10.1371/journal.pmed.0030517

Tornado Diagram

Two-way SA:CE of Empowerment program

Figure 1. Sensitivity analysis: cost per HIV infection averted by reduction in risk behavior, 5 years

$0

$10,000

$20,000

$30,000

$40,000

0% 10% 20% 30% 40% 50%

Reduction in risk behavior

Cos

t per

HIV

infe

ctio

n av

erte

d

Steady state

Pre-steady state

Two-way sensitivity analysis for changes in HIV risk and average cost for managing active TB cases

Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008

Three-Way SA

Kahn JG, et al. PLoS Med 3(12): e517. doi:10.1371/journal.pmed.0030517

Scenario SA

Burgos JL, Kahn JG, et al. Int Jour Tuberc Lung Dis 2008

Low HIV prevalence setting: 25% in CSWs; 16.4% in clients

Low HIV/AIDS treatment cost: $1,433 Med. HIV/AIDS treatment cost: $2,507 High HIV/AIDS treatment cost: $3,582

Low FC cost: $0.33

Medium FC cost:

$0.66

High FC cost: $1.32

Low FC cost: $0.33

Medium FC cost:

$0.66

High FC cost: $1.32

Low FC cost: $0.33

Medium FC cost:

$0.66

High FC cost: $1.32

($3,864) ($1,824)$2,076 [$509]

($7,426) ($5,386) ($1,486) ($10,989) ($8,949) ($5,059)

Medium HIV prevalence setting: 50.3% in CSWs; 33.0% in clients

Low HIV/AIDS treatment cost: $1,433 Med. HIV/AIDS treatment cost: $2,507 High HIV/AIDS treatment cost: $3,582

Low FC cost: $0.33

Medium FC cost:

$0.66

High FC cost: $1.32

Low FC cost: $0.33

Medium FC cost:

$0.66

High FC cost: $1.32

Low FC cost: $0.33

Medium FC cost:

$0.66

High FC cost: $1.32

($6,023) ($3,983) ($83) ($11,203 ($9,163) ($5,263) ($16,386) ($14,346) ($10,446)

Multivariate SA on female condom promotion: Net costs by HIV prevalence and key cost inputs for 1,000 CSWs

Threshold Analysis: NVP for prevention of vertical transmission of HIV in sub-Saharan Africa

Input values needed for $50/DALY

15% HIV prevalence

30% HIV prevalence

Regimen efficacy (47%)18.0% 10.6%

VCT cost ($7.30) $18.50 $36.00

HIV transmission (25.1%) 9.6% 5.6%

HIV prevalence for $50/DALY

4.5%

Threshold Analysis: NVP $ for prevention of vertical transmission of HIV in sub-Saharan Africa

Marseille E, Kahn JG, Saba J. AIDS 1998; 12:939-948

Clipping, asympt, <10 mm, SAH hx Coiling, asympt, <10 mm, SAH hx Rupture risk/yr 0.0050 Rupture risk/yr 0.0050 RR rupture w/ surgery 0 RR rupture w/ surgery 0.1 Surgical mortality 0.023 Surgical mortality 0.004 Surg morb (disability) 0.075 Surg morb (disability) 0.037 Cost of surgery 25,150 $ Cost of surgery 20,660 $

Δ QALYs -0.58 Δ QALYs 0.40

Δ $ $34,324 Δ $ $22,492

Scenario Analyses Aneurysm management CUA

$ / QALY Dominated Strategy

$ / QALY $56,230

Addressing Parameter Uncertainty:Multivariate Sensitivity Analysis

Types of Multivariate Sensitivity Analysis:

• Repeat bivariate• Maximize / minimize CE ratio for different

parameter combinations• Scenario analysis • Monte Carlo simulations under different

assumed distributions for parameters (probabilistic sensitivity analysis’ Gold)

Probabilistic sensitivity analysis

What is it?

What is it good for?

The problem with deterministic SAs

No estimate of the probability of achieving a particular outcome

(Probabilistic SAs are the remedy)

Probabilistic sensitivity analyses

Value •Returns the likelihood of attaining particular outcome or outcome range.• Everything known about each input expressed all at once.•Particularly valuable when many inputs important.

 DrawbackNeed to know, or be able to make decent estimates of,

the underlying probability distribution.

From empirical data to PD (1)

# of clients # of Subjects1 to 19 320 to 39 1540 to 59 3060 to 79 2580 to 99 12

100 to119 7120 to 139 5140 to 159 2

≥160 1

Frequency distribution of # of clients reported by 101 FSWs

Variable | Obs Mean Std. Err. SD [95% Conf. Interval]-------------+---------------------------------------------------------------# of Clients | 100 67 3.22 32 60.6 - 73.4

From empirical data to PD (2)

Graphical Representation of the # of clients reported by 101 FSWs

# of clients # of Subjects1 to 19 0.0320 to 39 0.1540 to 59 0.360 to 79 0.2580 to99 0.12

100 to119 0.07120 to 139 0.05140 to 159 0.02

≥160 0.01

Probability distribution of # of clients reported by 100 FSWs

From empirical data to PD (3)

From empirical data to PD (4)Graphical Representation of the Prob dist. of clients reported by 101 FSWs

From empirical data to PD (5)Probability Distribution of # of clients among 10,000 FSWs

Variable | Obs Mean Std. Err. SD [95% Conf. Interval]-------------+----------------------------------------------------------------------------------- # clients | 10,000 68 0.36 33 66.7 - 67.7

Triangular Distribution

Muennig, 2008

Published distributions

Henson SJ. Estimating costs of acute gastrointestinal Illness in BC. Int Jou Food Microb. 2008; 127: 43-45

Frequency Chart

%

Mean = 85.7%.000

.007

.014

.021

.029

0

14.25

28.5

42.75

57

79.5% 82.6% 85.7% 88.7% 91.8%

2,000 Trials 23 Outliers

Forecast: Percent reduction in mortality

Monte Carlo simulation output

Crystal Ball output

$226-$504

SA For QALYs Gained

Treeage output

Other uses of sensitivity analysis

(the inner teachings)(the inner teachings)• Planning the analysis

• Debugging the model

• Documenting relationships between inputs and outputs

• Identifying thresholds

• Influencing policyInfluencing policy

Other uses: Planning the analysis

• Program software to permit SAs on likely SA variables.

• SA curves provide a check on integrity of model.

• Identify candidates for more data collection early.

Other uses: Debugging the model

Tricks of the trade

• One-ways best because simple and intuitive. • Plug in extreme values. • Separate diagnosis of numerator from denominator.

• Break outputs down further if necessary (intervention versus control arms).

Other uses: Documenting relationships between inputs and outputs

Distinguish between ‘bugs’ and insights.

Examples of insights: • Slowing disease progression can increase costs.

• Higher disease prevalence can mean lower benefits.

• Benefits decrease with age - competing mortality risks.

Unexpected dynamic uncovered Unexpected dynamic uncovered by SA: Female condoms studyby SA: Female condoms study

Figure 2: HIV cases averted per 1,000 CSWs by HIV prevalence in CSWs and partners

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100%

HIV prevalence in CSWs (Prevalence in clients = 67% of CSW value)

HIV

cas

es a

vert

ed

Marseille et. al. Soc Sci Med. 2001 Jan;52(1):135-48.

Other uses: Identify thresholds – Influence Policy

Preventing HIV vertical transmission in sub-Saharan Africa

• Cost of ARVs to prevent vertical transmission.

• Universal versus targeted provision of NVP.

Cost per DALY of HIVNET 012 NVP regimen as function of HIV

seroprevalence and type of counseling/testing regimen

0

5

10

15

20

25

30

35

40

45

50

0% 5% 10% 15% 20% 25% 30%

HIV-1 seroprevalence

Cos

t per

DA

LY (U

S$)

UniversalTargeted

Pegynterferon CE acceptability curve

European Journal of Gastroenterology & Hepatology. 2007;19:631-638.

C-E Acceptability Curve (QALYs)WTP

$0 0.4$50 0.4$100 0.5$150 0.5$200 0.6$250 0.6$300 0.7$350 0.7$400 0.8$450 0.8$500 0.8

WTP

$550 0.8$600 0.9$650 0.9$700 0.9$750 0.9$800 0.9$850 0.9$900 0.9$950 0.9$1,000 0.9

50

Summary

• SA is a set of techniques for the explicit management of uncertainty.

• Essential part of establishing key findings.

• Indispensable for convincing your audience that your results are technically sound and policy-relevant.

Practicum