dealing with uncertainty in decision models · decision models susan griffin centre for health...
TRANSCRIPT
![Page 1: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/1.jpg)
Dealing with uncertainty in
decision models
Susan Griffin
Centre for Health Economics, University of York
![Page 2: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/2.jpg)
Economic evaluation in health care
• Decision problem
– Which interventions to provide given resource
constraints?
• Assess health gains and costs associated with
alternative interventions
– Utilise available evidence
– Attribute differences to use of particular interventions
– Reimburse set of interventions that maximises net
health benefit
![Page 3: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/3.jpg)
Evidence base
Study
Costs
Health
Administrative
database
Observational
study
Price list
Decision
model
![Page 4: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/4.jpg)
Cost-effectiveness plane
• True value of
health gains and
costs of new
alternative
Cost
Health
Rate at which
existing activities
generate health
![Page 5: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/5.jpg)
Uncertainty
• Stemming from incomplete knowledge
– Which sources of evidence are relevant
– Relationship between inputs
– True/population values
• Reducible through further research
• Resolvable over time
![Page 6: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/6.jpg)
Cost-effectiveness plane
• Possible values for
health gains and
costs for new
alternative
Cost
Health
Rate at which
existing activities
generate health
![Page 7: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/7.jpg)
Sources of uncertainty
• Which sources of evidence
– Internal validity, external validity, bias
– Missing observations and outcomes
– Sample size
• Relationship between inputs
• Value judgements
![Page 8: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/8.jpg)
Relevant evidence for treatment effects
• Causal effects of interventions
• Internal validity
– Impossible to directly observe health gains with
intervention and health gains without
– Ability to approximate counterfactual
• E.g. RCT versus observational study
• External validity
– Extent to which results in studied population hold true
for target population
• E.g. trial setting versus general practice
![Page 9: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/9.jpg)
Relevant evidence for treatment effects
• Lack of validity indicates bias
– Systematic difference between estimate and true value
– Bias is source of uncertainty
• Missing observations
– If not missing completely at random, complete case
analysis will be biased
![Page 10: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/10.jpg)
Cost-effectiveness plane
• Biased estimates
of health gains and
costs for new
alternative
Cost
Health
Rate at which
existing activities
generate health
![Page 11: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/11.jpg)
Dealing with bias
• Eliminate or minimise
– E.g. rely on RCTs for treatment effect if suitable
– Adjust for selection bias in analysis
• E.g. regression model, propensity score, IV, selection model
– Utilise imputation for missing observations
• E.g. multiple imputation
• Characterise as additional parameter
– E.g. elicitation, informative prior
![Page 12: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/12.jpg)
Relationships between inputs
• Several studies reporting same information
– Meta-analysis
– Generalised evidence synthesis
• Missing outcomes
– Required for CEA, not measured directly
– Expected survival: extrapolation
– HRQL: cross-walks/mapping
• Decision model
– Explicit framework
– Assumptions, logical relationship between inputs
![Page 13: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/13.jpg)
Structural uncertainty
• Modelling or structural uncertainty
– Alternative model structures or assumptions could
generate different results
• Model validity
– Assess how accurately available info characterised
– Typically no source for external validation
• Value judgements
• Can identify some models as invalid, but may not identify
single best structure
![Page 14: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/14.jpg)
Sampling uncertainty
• Inputs informed by sample data
– Underlying population values estimated with uncertainty
– Evidence supports a range of plausible values with
varying degrees of likelihood
• Direct data unavailable
– Cannot omit important variable from analysis
– Elicitation
![Page 15: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/15.jpg)
Dealing with uncertainty
• Describe range of
– True values of inputs
– Possible relationships between inputs
– Value judgements
• Describe outputs from alternative values deterministic
sensitivity analysis
• Also describe likelihood of particular values
– Probabilistic sensitivity analysis for parameters
– Bayesian model averaging
![Page 16: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/16.jpg)
Probabilistic sensitivity analysis
• Produces distribution of model outputs
– Best estimates of mean costs and health outcomes non-
linear model
• Estimate decision uncertainty
– How likely is the decision to be in error?
– What are the consequences of that error?
– Attributable to uncertainty characterised for parameters
![Page 17: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/17.jpg)
Characteristics
of parameterMean and
standard errorData generating
process
Repeatedly take randoms draws from assigned distributions and
calculate expected costs and outcomes for each
Assign distribution
Generate distribution of
expected costs and outcomes
![Page 18: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/18.jpg)
Why uncertainty?
• Non-linear model
• Value of evidence
– Is current evidence sufficient?
– Is further research valuable?
• Consequences of uncertain decision
– New evidence emerge suggesting change in decision
– Resource implications: Investment/reversal
• Dependence between reimbursement and research
– Value of information forgone
– OIR, coverage with evidence development
![Page 19: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/19.jpg)
Is evidence sufficient?
• Additional evidence expected to reduce decision
uncertainty
– Reduce probability of error
– Reduce opportunity cost of uncertainty
• Compare expected improvement in health gains
with reduced uncertainty to cost of research
– Perfect information: EVPI, EVPPI
– Imperfect information: EVSI, ENBS
![Page 20: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/20.jpg)
Impact of uncertainty
• Investment cost
– Sunk costs, irrecoverable if decision changed
– Gains from new technology must be sufficient to
outweigh investment cost
• Reversal cost
– Incurred only when decision revised
• Characterising uncertainty helps estimate likelihood
of change
– Additional info on when new evidence could emerge
![Page 21: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/21.jpg)
![Page 22: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/22.jpg)
Interaction between research and
reimbursement
• Some research not possible once technology in
widespread use
– Approval removes option to collect further evidence
– Value of information forgone
– Compare to opportunity cost of delaying access (OIR)
• Some research easier after approval (AWR)
• Reimbursement decision conditional on uncertainty
– E.g. Patient access schemes, risk sharing
![Page 23: Dealing with uncertainty in decision models · decision models Susan Griffin Centre for Health Economics, University of York. Economic evaluation in health care •Decision problem](https://reader034.vdocuments.net/reader034/viewer/2022042923/5f731970014b4c18ce29bd3e/html5/thumbnails/23.jpg)
Summary
• Uncertainty inherent to resource allocation
decisions
– Regardless of whether based on formal or informal
analysis
• Characterisation of uncertainty essential to inform
reimbursement decisions
– Appropriate response to uncertainty required to achieve
best possible health outcomes
– Even for decision maker with remit for reimbursement
not research