exploring uncertainty in cost-effectiveness analysis · exploring uncertainty in cost ... ....

Pharmacoeconomics 2008; 26 (9): 781-798BRIEFING PAPER 1170-7690/08/0009-0781/$48.00/0

© 2008 Adis Data Information BV. All rights reserved.

Exploring Uncertainty inCost-Effectiveness AnalysisKarl Claxton

Centre for Health Economics, Department of Economics and NICE Decision Support Unit,University of York, Heslington, York, UK

This paper describes the key principles of why an assessment of uncertaintyAbstractand its consequences are critical for the types of decisions that a body such as theUK National Institute for Health and Clinical Excellence (NICE) has to make. Indoing so, it poses the question of whether formal methods may be useful to NICEand its advisory committees in making such assessments. Broadly, these includethe following: (i) should probabilistic sensitivity analysis continue to be recom-mended as a means to characterize parameter uncertainty; (ii) which methodsshould be used to represent other sources of uncertainty; (iii) when can computa-tionally expensive models be justified and is computation expense a sufficientjustification for failing to express uncertainty; (iv) which summary measures ofuncertainty should be used to present the results to decision makers; and(v) should formal methods be recommended to inform the assessment of the needfor evidence and the consequences of an uncertain decision for the UK NHS?

1. Background A discussion of appropriate methods to charac-terize uncertainty requires first considering how de-

When the UK National Institute for Health and cisions can be made based on evidence of costClinical Excellence (NICE) makes recommenda- effectiveness and why an assessment of uncertaintytions on the use of technologies in the UK NHS, might be important for the types of decisions that ainformation about cost effectiveness is critical. body such as NICE has to make.However, any assessment of effect and cost is uncer-tain and, thus, any decision based on cost effective- 1.1 Making Decisions Based onness will also be uncertain. This uncertainty arises Cost Effectivenessfrom a number of sources: not only the uncertainty

NICE appraisal requires an estimate of the costin the estimates of parameters of the decision mod-effectiveness of the technology, often summarizedels commonly used to estimate costs and effects, butas an incremental cost-effectiveness ratio (ICER),also a wide range of other sources of uncertainty,i.e. the additional cost for each additional QALYe.g. the potential bias or relevance of evidence andgained. This must be compared with some thresholdthe assumptions required in extrapolating effectsfor cost effectiveness, representing the health expec-and costs over time. In the face of this uncertaintyted to be forgone elsewhere in the NHS because of(only some of which is often explicitly characterizedthe additional costs.[1,2]and presented), NICE must come to a view about

whether the benefits of allowing access to an appar- For example, a threshold of £20 000 means thatently cost-effective technology are greater than the 1 QALY is expected to be lost for every £20 000 thepotential consequences of an uncertain decision for NHS must find by curtailing other activities to ac-the NHS. commodate the use of a more costly technology. If a

782 Claxton

technology costs the NHS an additional £12 000 and 2. Why Does Uncertainty Matter?provides 0.8 additional QALYs, the ICER of

Although the importance of cost effectiveness is£15 000 per QALY gained (£12 000/0.8) is less thannow well established, the reasons why uncertaintythe threshold of £20 000 and the technology can bematters are less so. There are strong arguments forregarded as cost effective.basing decisions about use of a technology on the

Alternatively, and entirely equivalently, this can expected incremental effects and costs rather thanbe expressed as a positive net health benefit of applying traditional rules of statistical significance0.2 QALYs – the 0.8 QALYs gained more than to estimates of cost effectiveness. Indeed, previousoffsets the 0.6 QALYs forgone elsewhere (£12 000/ guidance indicates that it is evidence about the ex-£20 000). This can also be expressed as positive net pected effects and costs that is the primary basis formonetary benefit of £4000 – the monetary value of the guidance issued.[6] However, making decisionsthe health benefits of £16 000 (0.8 × £20 000) is based only on expected effects and costs in no waygreater than the monetary costs of £12 000.[3] The implies that the uncertainty surrounding such a deci-relationships between these entirely equivalent ways sion is unimportant. Indeed, an assessment of theto express cost effectiveness are particularly impor- implications of decision uncertainty is an essentialtant to appreciate when considering uncertainty. part of any decision-making process that is con-This is because, although ICERs are a useful and cerned with improving health outcomes, given theintuitive way to summarize cost effectiveness, they resource constraint the NHS faces. There are threealso have very unhelpful statistical properties.[4]

reasons why the uncertainty surrounding effects andOver recent years, almost all analyses of uncertainty costs matters: (i) to provide correct evaluation ofin cost-effectiveness analysis have utilized measures expected effect and cost; (ii) to consider whetherof net health or net monetary benefit (of course, the existing evidence is sufficient; and (iii) to assess theresults can and often are summarized using ICERs). possible consequences of an uncertain decision for

the NHS.

1.2 Variability, Heterogeneity2.1 Evaluating Expected Effects, Costs andand UncertaintyNet Benefit

It is useful to distinguish between variability, When effects and costs are evaluated using aheterogeneity and uncertainty at the outset. Variabil- decision model in which there is a nonlinear rela-ity refers to the natural variation between patients in tionship between inputs and outputs, e.g. Markovtheir response to treatment and the costs they incur – process,[7] the correct calculation of expected effectseven when they have the same observed characteris- and costs will require the uncertainty around all thetics. Variability is irreducible – additional evidence inputs (the parameters) to be expressed. This issue iscan not reduce it. Heterogeneity refers to differences illustrated in figure 1, where the value of parameterbetween patients who have different observed char- θ has a nonlinear relationship with net benefit (NB).acteristics – for which, in principle, different gui- Simply using the expected value of θ (θ1) in thedance could be issued.[5] Uncertainty refers to the model would provide NB1. However, the value of θfact that we can never know for certain what the is uncertain and has a probability distribution. It ismean (expected) costs and effects would be if the possible that θ will take a value of θ2, generatingtreatment is provided for a particular population of NB2 or a value of θ3 generating NB3. Both valuespatients, even if they have the same observed char- are equally likely (they could represent the 95%acteristics. Additional evidence can reduce uncer- confidence interval for θ) but generate very differenttainty and provide more precise estimates. If the values of NBs. When NB is evaluated over thepurpose of the NHS is to maximize health gains possible values of θ, the expected NB [averagedfrom its budget, then it is these expected effects and over the possible values of θ or Eθ(NB,θ)] willcosts, and the uncertainty in estimating them, that clearly be less than NB1. An analysis that only usedare of primary interest. the expected values of the parameters to estimate

© 2008 Adis Data Information BV. All rights reserved. Pharmacoeconomics 2008; 26 (9)

Exploring Uncertainty in CEA 783

the probability that B is cost effective is 0.6. Forsome realizations (two of five), treatment A wouldhave been better. Therefore, a decision to approve Bbased on current evidence is associated with an errorprobability of 0.4. This is substantially greater thantraditional benchmarks of statistical significancesuch as 0.05. Whether or not this level of uncertainty‘matters’ depends on the consequences, i.e. whatimprovement in NB (or avoidance of harm) couldhave been achieved if this uncertainty had beenresolved.

If uncertainty could be completely resolved (i.e.through complete evidence about effect and cost),then we would know the true value of NB beforechoosing between A and B (column 5 of table I). Ofcourse we can not know in advance which of thesevalues will be realized but on average (over the 5th

θ2θ3 θ1

Net

ben

efit

NB1

NB2

NB3

Eθ[NB(θ)]

θ

θ

Fig. 1. Uncertainty and nonlinearity. The parameter θ has a non-linear relationship with net benefit (NB). The value of parameter θ isuncertain and has a probability distribution. E = expected.

column) we would achieve 13.6 rather than13 QALYs. Therefore, the cost of uncertainty or theexpected effects, costs and NB could be seriouslymaximum value of more evidence is 0.6 QALYs ormisleading. Therefore, uncertainty matters, even if a£12 000 per patient (at a threshold of £20 000 “perdecision maker only wishes to consider expectedQALY”) – more than half the value of the technol-cost effectiveness in their decisions.ogy itself. It should be clear that the cost of uncer-tainty or the value of evidence is just as ‘real’ as2.2 Is the Evidence Sufficient?access to a cost-effective treatment, as both aremeasured in terms of improved health outcomes forSimply basing decisions on expected cost effec-NHS patients. In principle, evidence can be just as,tiveness will ignore the question of whether theor even more, important than access to a cost-effec-current evidence is a sufficient basis for guidance. Ittive technology.[8]

would fail to address the question of whether furtherresearch is needed to support NHS practice. The

2.3 Consequences of an Uncertain Decisionvalue of evidence or the health costs of uncertaintyfor the UK NHScan be illustrated using a simple example in table I.

Each row represents a realization of uncertainty, i.e.Almost all decisions faced by NICE will imposethe net health benefit (QALYs) that results from

some costs on the NHS which, if the guidance istaking a sample (a possible value) of each of thereversed at a future date, cannot be recovered. Forinput parameters to the model. Therefore, each rowexample, investment in equipment, the training ofcan be thought of as representing one of the waysspecialist staff or simply engaging in activities to‘things could turn out’ given our current uncertainty.

The expected NB for treatment A and B is theaverage over all these possibilities (in this example,the range of potential values is simplified to onlyfive possibilities).

On the basis of current evidence, we would con-clude that treatment B was cost effective and onaverage we expect to gain an additional 1 QALY perpatient treated compared with treatment A. How-ever, this decision is uncertain and treatment B is notalways the best choice (only three of five times), so

Table I. Why is evidence valuable?

How things Net health benefit Best we

could turn treatment treatment best could doout A B choice if we knewPossibility 1 9 12 B 12

Possibility 2 12 10 A 12

Possibility 3 14 17 B 17



Average 12 13 13.6


784 Claxton

ensure the guidance is implemented at a local level. 2.4 Current and DevelopingPolicy EnvironmentThese types of cost are often described as ‘sunk’ and

are irreversible. When a decision is uncertain and Since the 2004 Guide to the Methods of Technol-there is a prospect that guidance may be reversed at ogy Appraisal[13] was issued, a number of policysome point in the future (e.g. if new evidence sug- initiatives have occurred that pose more sharply thegests that the technology is not cost effective or if questions whether current evidence is sufficient toanother technological development makes it obso- support NHS practice and what are the conse-lete) then it may be better, in terms of NBs for the quences of an uncertain decision? Importantly,NHS, to wait until these uncertainties are resolved NICE increasingly issues guidance on technologiesover time or until additional research is reported.[9,10] close to launch when the evidence base is inevitably

least mature. However, NICE does not have theThere is another type of (opportunity) cost orremit to commission or fund research as part of itsirreversibility that is often more significant for theTechnology Appraisal Programme, but it does makeNHS and common to many NICE decisions. Aresearch recommendations, and can issue guidancedecision to approve a technology will inevitablyconditional on additional evidence being provided,have an impact on the prospects of acquiring evi-or restrict use within a clinical trial (an ‘only indence to support its use in the future. This is becauseresearch’ recommendation).the incentives on manufacturers to conduct evalua-

2.4.1 Research Recommendationstive research (for the claimed indication) are re-If evaluative research priorities are to be moremoved once positive guidance or coverage has been

closely related to the needs of the NHS and thegranted. Furthermore, the clinical community is un-guidance issued by NICE, as envisioned in the re-likely to regard further experimental research to becent reforms to UK health research funding,[14] thenethical and, even if ethical approval was granted,a means to prioritize and effectively communicateany randomized controlled trial is unlikely to re-the importance of additional evidence to those re-cruit. This is particularly so when access to thesponsible for commissioning research will be re-technology is mandatory, as is the case with NICEquired. To fulfil its remit, some form of assessmentguidance.(whether informal or informed by explicit analysis)

If the type of evidence the NHS needs can not be of the uncertainty in cost effectiveness, its conse-provided once positive guidance is granted, then the quences for the NHS and the importance of particu-decision to issue guidance should take account of lar types of evidence, will be needed at some pointboth the value of the technology (the expected NBs) during the NICE appraisal process.and the value of the evidence that may be forgone

2.4.2 Guidance with Evidence and Risk Sharing(which can also be measured in terms of NB; seeIn some circumstances, it might be possible to

section 2.2). A technology may be regarded as costapprove the use of a technology that appears to be

effective when only considering the expected net cost effective based on existing evidence, while athealth benefits of the technology itself. However, the same time additional evidence is obtained fromthe value of the evidence (also expressed in terms of the manufacturer or by publicly funded research. Inpopulation health) that would be forgone for future principle, this seems an efficient solution to thepatients (the opportunity cost of approval) may ex- problem – some patients get early access to anceed the NBs of approval. In these circumstances, it apparently cost-effective technology but the evi-would be better for the NHS if this evidence can be dence is also generated so that this decision can beacquired by withholding approval, despite the tech- reconsidered at a later appraisal when cost effective-nology having an ICER below the cost-effectiveness ness as well as potential harm can be reassessed.threshold. This can be achieved by NICE issuing an This question of ‘coverage or guidance with evi-‘only in research’ recommendation and reconsider- dence’ or ‘risk sharing’ seems to be envisaged with-ing approval once sufficient evidence is avail- in the recent reforms to research funding and pro-able.[11,12] posed reforms to UK pharmaceutical pricing.[14-16]



If ‘coverage with evidence’ is to become a useful • Which methods should be used to characterizepolicy tool, an assessment of three questions will be parameter uncertainty?needed: (i) is additional evidence needed; (ii) if so, • How should other sources of uncertainty be rep-what type of evidence is required; and (iii) will the resented?type of evidence that can be gathered with concur- • How can the computational challenges be over-rent approval be valuable and meet these needs? For come?example, if the key uncertainty is evidence about • How should uncertainty in cost effectiveness berelative effects, then the type of observational regis- presented?try data that are often envisaged will not be able to • How should the need for evidence and the conse-provide more reliable and precise estimates because quences of an uncertain decision be assessed?a comparable control group will not be available. These key questions are discussed in turn in theEarly access will have been allowed, but the promise following sections.of the type of evidence needed will not be fulfilled.In these circumstances, early guidance would mean 3. Characterizing Parameter Uncertaintythat the evidence base for future NHS practice will

The 2004 Methods Guide[13] recommended thatbe undermined. Therefore, the question of how suchprobabilistic sensitivity analysis (PSA)[17] should beassessments might be made by NICE is critical.conducted to reflect the combined implications ofuncertainty in the parameters (inputs), and to quanti-2.4.3 Only in Researchfy the uncertainty in cost effectiveness (outputs) andThere are a number of circumstances in which itthe decision to approve or reject a technology. Thismight be better for the NHS if approval of a technol-parameter uncertainty is only one source of uncer-ogy that appears to be cost effective is withheld untiltainty; nevertheless, it is an important component ofthe uncertainty surrounding the decision is resolveddecision uncertainty, so the question of which meth-(i.e. an ‘only in research’ recommendation).[12] Thisods should be used to characterize it is important.includes situations in which (i) the NHS must make

a significant and irreversible investment; (ii) new3.1 Probabilistic Sensitivity Analysisand potentially cost-effective technological devel-

opment is imminent; and (iii) new and significant The principles of PSA are very intuitive. Distri-evidence is likely to become available. It also in- butions are assigned to each of the model para-cludes situations in which the type of evidence meters. These are sampled (often using Monte Carloneeded to reach a well informed decision about simulation, which samples at random), and the out-possible NHS use of the technology cannot be pro- put of the model (estimates of expected costs, effectsvided once it has been approved for NHS use. In all and NB) is recorded for each set of samples from allthese circumstances, the decision to issue guidance the parameters. This process of sampling inputs andought to take account of both the value of the recording output is repeated (e.g. 10 000 times) sotechnology (the additional expected NBs) and the that the range of values that the parameters are likelycosts of the uncertain decisions – including the to take are represented in the range of outputs.[18] Asevidence that may be forgone and the likelihood that well as providing the correct estimate of expectedthe investment required will ultimately be wasted. effects, costs and NB in nonlinear models (see figure

1), the output of this process also provides the2.5 Summary proportion of times (the probability) that each alter-

native is cost effective and all the information re-All of the considerations described above requirequired to calculate the maximum value of additionalsome assessment of the uncertainty surrounding aevidence, as illustrated in table I and discussed indecision based on cost effectiveness and its conse-section 2.2.quences. Whether or not formal methods can assist

3.1.1 Distributional Formsin making these assessments is central to the devel-opment of appropriate methodological guidance. In It may appear that PSA introduces further as-particular: sumptions about the choice of distribution to re-


786 Claxton

3.1.2 Correlationpresent the uncertainty in the model inputs. How-Model parameters may be correlated. For exam-ever, the choice of distribution is not at all arbitrary

ple, if a regression analysis is used to estimate modelif appropriately guided by the following three prin-parameters, the relationship between them can beciples, as well as explicit judgments about the poten-estimated from the co-variance matrix so that thetial bias and relevance of the available evidence:correlations between regression coefficients can be(i) the nature of the parameter itself; (ii) the way theincluded in PSA in the multi-variate normal case.[8]

parameter was estimated, so that the distribution Similarly, methods of evidence synthesis[21,22] gen-assigned reflects the statistical uncertainty (and any erate correlated outputs that can be fully captured incorrelations) in its estimation; and (iii) the decision PSA by sampling from or directly using the outputcontext, e.g. is it the average patient or the average of such synthesis in the decision model. If there is noof patients with a particular set of characteristics that evidence from statistical analysis that parametersis of interest? are correlated, it is generally not necessary to im-

pose one. Of course, any logical or latent structuralApplication of these general principles meansrelationship between parameters should be reflectedthat there will only be a very limited choice ofin the model structure rather than imposing correla-appropriate distributional forms. For example,tion.probability parameters are bounded by 0–1, so it

would be inappropriate to specify a distribution that3.2 Deterministic Sensitivity Analysisgave a probability to obtaining values outside this

range. Where a probability is estimated from a pro-In deterministic sensitivity analysis (DSA), eachportion, the beta distribution is the natural choice.

parameter in turn is set at an extreme but plausibleHowever, if the probability parameter is estimatedvalue and the decision is described as sensitive if afrom a logistic regression, then the parameters ofdecision based on cost-effectiveness changes. Alter-

interest are the coefficients on the log-odds scale,natively, a threshold value for a parameter is found

and multi-variate normality on this scale would be at which the decision changes. Although DSA isappropriate. For probabilities estimated from time- very simple to understand and generally easy toto-event data, the parameters would be the coeffi- implement, there are a number of problems.cients from a survival analysis estimated on the log • The estimates of expected effects, costs and NBhazard scale and, again, the appropriate assumption will be biased in nonlinear models.would be multi-variate normality on this scale.[8] • What is an ‘extreme but plausible’ value for the

parameter is not clear as there is an implicitThere is often insufficient evidence to applyassumption about the distribution of the para-standard statistical estimation and/or the evidence ismeter.limited in some other respect, e.g. potential bias and

• Neither is it clear whether a threshold value for arelevance. In these circumstances it becomes moreparameter is likely or extremely unlikely to oc-apparent that the distributions assigned in PSA arecur.really an explicit judgement (a reasonable beliefTherefore, any useful interpretation of one-wayoften informed by evidence) about value of the

sensitivity analysis requires exactly the same infor-parameter and its uncertainty.[19] Formal methods

mation about the distribution of the parameter asare available that can be used to elicit the views of PSA. The difference is that this information is notexperts about the likely value of a parameter and its presented, so the judgments required remain implicitdistribution.[20] Of course, parameter uncertainty is and opaque rather than explicit and transparent.only one source of uncertainty. Alternative but cred- Indeed, where parameters are based on statisticalible and plausible views of the quality and relevance estimates (e.g. meta-analysis, evidence synthesis orof evidence as well as other assumptions are often regression analysis), then the information about thepossible. These other sources of uncertainty are uncertainty and correlation in these estimates is indiscussed more fully in section 4. essence thrown away.[23]



A decision may not seem sensitive to any plausi- sumptions that are possible, and some may be re-ble parameter values in a series of one-way sensitiv- garded as plausible and credible. In these circum-ity analyses, but when taken in combination (para- stances, the 2004 Methods Guide[13] recommendedmeters are simultaneously uncertain) there may be that probabilistic scenarios[25] (a revision and rerunconsiderable uncertainty. Therefore, even if inter- of the probabilistic model) should be presented, withpreted correctly, one-way sensitivity analysis will the expected effects, costs and measures of uncer-commonly (in the absence of correlation) underesti- tainty reported for each. It is then for the decisionmate uncertainty, making it particularly vulnerable maker to take these into account in the light of otherto false claims that results are robust. Of course, a evidence (e.g. views of clinical experts).poorly and inadequately conducted PSA can be used There are broadly two possibilities. First, onlyto make the same false claims. The question is one scenario is regarded as credible and all otherswhether an analysis that is more explicit and trans- can be disregarded – the parameter uncertainty cap-parent about distributions assigned to parameters, tures all the uncertainty surrounding the decision.and that is directly interpretable, makes such claims More commonly, one scenario may be the mosteasier to detect. credible but others are also possible and cannot be

Best- and worst-case scenarios are very difficult disregarded. In this case, there is uncertainty aboutto interpret correctly. The probability of all para- effects and costs given a particular set of judgmentsmeters taking extreme but plausible values simulta- and also uncertainty about which set of judgmentsneously will be very small indeed, so if the decision might be realized. Parameter uncertainty no longeris sensitive to such extremes it is not clear whether represents all the uncertainty surrounding the deci-the decision should be regarded as uncertain or not. sion, and the decision maker must implicitly weighMulti-way sensitivity analysis with more than two these different scenarios to come to a view aboutor three parameters is very difficult to present. More effects, costs and overall uncertainty. When differ-importantly, it is generally even more difficult to ent scenarios suggest different decisions, the struc-interpret correctly and becomes impossible if some tural uncertainty clearly matters. However, evenparameters are correlated, because it is impossible to when this does not occur, it will affect decisionlocate a fixed (set of) parameter value(s) that can be uncertainty too, in ways that are difficult to assessregarded as ‘extreme’.[23] implicitly and intuitively.[24] There are alternatives

to this implicit weighting of probabilistic scenarios,4. Representing Other Sources which require the formal specification of probabilityof Uncertainty distributions across scenarios or across structural

parameters.The uncertainties surrounding the parameters of

the model are only one source of uncertainty. Other 4.2 Model Averagingsources of uncertainty include the different types of

The weighting of scenarios can be made explicitscientific judgments that have to be made whenby assigning probabilities to represent how credibleconstructing a model of any sort such as decisioneach is believed to be.[24,26] The weighted average ofmodel, statistical model or even standard meta-ana-costs and effects from each scenario can easily belysis. These types of uncertainties have been classi-calculated but overall uncertainty and its conse-fied in a number of different ways (methodological,quences requires the simulated values from the sce-model, clinical, etc.) but can be referred to collec-narios to be merged, based on these probabilities,tively as structural uncertainties.[24]

before calculating a new estimate of the errorprobabilities and value of information. This does not4.1 Probabilistic Scenariosrequire additional simulation and is quick, easy toimplement, and explicit. However, it does pose aIt is important to recognize that the results of anynumber of questions:type of model are correct if the necessary judge-

ments and assumptions are acceptable. However, • Who should be responsible for choosing thethere are always other sets of judgments and as- weights?


788 Claxton

• If there is no consensus, how can differing views • it does not implicitly assign extreme and general-be combined? ly unlikely values (0 or 1) to the parameter;

• it indicates who might be best placed to provide• Which methods should be used to elicitinformed and credible judgments;probabilities (see section 4.4)?

• when there is more than one difference between• How many scenarios should be used to representscenarios it allows the most important source ofthe possibilities?uncertainty to be identified;Despite these practical issues, this approach

• the impact on decision uncertainty and the appli-would provide measures of overall decision uncer-cation of value of information methods allowstainty and its consequences. Importantly, it wouldconsideration of whether further evidence aboutdo so even when the different scenarios did notthe parameter is needed.suggest different decisions based on cost effective-

ness. However, it remains difficult to assess which4.4 Elicitationuncertainties matter most and what type of addition-

al evidence might be most useful.These more formal approaches to structural un-

certainty described in sections 4.2 and 4.3 require4.3 Parameterizing Structural Uncertainty explicit judgments about the value of, and the uncer-

tainty surrounding, a parameter for which no directevidence exists.[28,30] Of course, similar issues areIn almost all cases, structural uncertainty can beraised when selecting distributions for parameters inthought of as a missing and uncertain parameter inPSA but they are posed more sharply in this context.the model, i.e. different scenarios are special casesAny judgement made will need to be credible andof a common ‘meta-model’ but with missing para-defensible. Well established methods for this type ofmeters taking extreme values.[24,27] For example, in aelicitation are available and documented.[20] How-progressive disease where a treatment may or mayever, they do take time and their use poses a numbernot be disease modifying, two alternative scenariosof detailed questions.of rebound following treatment failure could be• Who should provide the judgements – whichconstructed: (i) rebound is equal to initial improve-

experts or decision makers?ment on treatment; or (ii) rebound back to where the• Which particular methods of elicitation should bepatient would have progressed. This structural un-

used?certainty is really an uncertain and missing para-meter – the magnitude of rebound relative to disease • How should the quality of judgments be cali-progression. Rather than present two scenarios, this brated (tested) and weighted?parameter could be included in the model despite • Should more than one expert be used? If so, thenthere being no evidence available to estimate it.[28] how should judgments be combined and if notThe distribution assigned to it should represent an combined, how should we choose which one isinformed and credible view about its value and most credible?uncertainty. Similarly, differing views about thequality or relevance of trial evidence could be repre- 5. Dealing withsented as scenarios where some trials are either Computational Challengesincluded (they are relevant and unbiased) or exclud-ed (they are irrelevant and so biased that the evi- Some types of model structures are particularlydence they provide is worthless). The alternative to computationally expensive, and expressing uncer-these extremes is to assign uncertain parameters tainty using PSA becomes particularly burdensome.(weights) for bias and relevance.[29] The latter ap- There are three key questions: (i) when can com-proach has a number of advantages: putationally expensive models be justified; (ii) can• it represents the source of uncertainty as a real the computational expense be overcome; and (iii) if

and measurable parameter that is amenable to not, can computational expense justify a failure tofurther investigation; adequately express uncertainty?



5.1 When Can Computationally Expensive • Sampling patients with different observed char-Models be Justified? acteristics provides the expected effects and costs

averaged over this heterogeneous population.However, NICE needs separate estimates of ef-The choice of model structure and complexity isfects and costs for identifiable groups of patientsalways a trade-off between descriptive realism,with different characteristics.[5] If NICE requirescomputational burden and evidence requirements.estimates for a number of combined patientThe key issue is whether the model is sufficientlygroups, this can be based on the weighted aver-realistic to inform the decision, not whether it is aage of the expected effects and costs for eachfine description of reality. There are two types ofgroup, using a standard analysis.common model structure that can be particularly

• These types of model necessarily sample vari-computationally expensive: patient-level simulationability. However, the primary interest for NICE(PLS; which includes micro or discrete-event simu-is expected (population mean) costs and effects,lation) and dynamic models.not their variability. In addition, for most com-mon model structures, variability will have no5.1.1 Patient-Level Simulationeffect on expected effects and costs (e.g. inPLS requires simulation to obtain a single esti-Markov models). There are some circumstancesmate of expected effect and cost, so expressingwhen expected effects and costs do depend onuncertainty using PSA can become particularly bur-both the uncertainty and the variability. In thesedensome. However, a single estimate of expectedcases, both need to be expressed, but computa-effect and cost based on a single set of (mean) valuestionally less expensive methods can be used. Infor the parameters will be biased because such com-the calculation of expected effects and costs thereplex models are inevitably nonlinear. Furthermore,are never circumstances when variability in ef-uncertainty and its consequences will not be ex-fects and costs matters but the uncertainty doesplored. PLS models have increasingly been used innot.NICE appraisals but only very rarely has PSA been

• PLS is often used to switch patients betweenconducted to capture uncertainty.[31] Therefore, thetreatments to generate a sequence of treatmentscircumstances in which PLS may be justified needfor each patient. The result of such analysis pro-to be considered.vides the effects and costs averaged over theThis type of model can be particularly useful formany different sequences that might be experi-modelling time and state dependence. Althoughenced. However, NICE often wishes to issuesimple time dependence is easily handled inguidance on which sequence is cost effective.standard analysis, it becomes more difficult whenThis requires estimates of the effects and costs ofthe probability of, or the time until, future eventseach possible sequence or, when there are a vastdepends on a patient-accumulated history, e.g. thenumber of possible sequences, the most commonsequence and time spent in different states. In someor clinically relevant ones. Identifying the mostcircumstances, this can be handled in standardcost-effective sequence when there are a vastMarkov models and in some others by implementingnumber of possibilities is challenging for allsemi-Markov processes using efficient program-modelling approaches, but more so for those thatming platforms.[8] A model structure that is unableare more computationally expensive.to take account of these issues may well lead to

biased estimates of expected effects and costs.5.1.2 Dynamic ModelsTherefore, the existence of time and state depend-

ence itself is not necessarily a justification for adopt- Dynamic models are used to model infectiousing PLS, but clearly there are circumstances where diseases, where the effective treatment of one indi-such models are justified for this reason. vidual can have an impact on others through reduced

Other reasons given for using PLS include the infection and herd immunity.[32] Where a technologyability to sample patient characteristics, integrate is designed or likely to have a significant impact onvariability and treatment switching. an infectious disease and the impact of treatment on


790 Claxton

the infection of others is a consideration, then mod- gradients may perform badly. A number of mathe-els failing to account for this may underestimate matical approximations are possible and have beenbenefits and overestimate costs. Dynamic models used in health technology assessment and NICEare often implemented using PLS and are computa- appraisals.[35]

tionally expensive. However, this is not always ne- In summary, computationally expensive modelscessary, and other programming platforms can be may be justified in some circumstances, and meth-used to solve the differential equations analytically ods are available to ease this burden. A failure toor numerically. adequately express uncertainty means that the esti-

mates of expected cost and effect will commonly be5.2 Can Computational Expense biased, and uncertainty and its consequences will bebe Overcome? inadequately explored. Therefore, the computational

expense of the chosen modelling approach and plat-Clearly there are circumstances in which com- form can not reasonably justify a failure to ade-

putationally expensive models may be justified. In quately express uncertainty.these circumstances, computational burden can oft-en be eased in broadly three ways: more efficient 6. Presenting Uncertainty in Estimatessimulation, emulators and linear approximations. of Cost Effectiveness

5.2.1 More Efficient SimulationThe reasons why uncertainty matters for the NHS

Faster processing or more efficient simulation suggest that it might be useful for decision makers tocan ease this burden. Recently, it has been shown have some information about (i) the probability thatthat substantial improvement can be made in the a decision based on expected effects and costs willbasic Monte Carlo approach to computing PSA in be correct (p) and the probability it will turn out totypical PLS models. This method is simple to apply be wrong – the error probability (1 – p); andand makes many otherwise intractable models ame- (ii) some indication of the consequences of error, i.e.nable to PSA (for further information, see O’Hagan what health and/or resource costs will be incurred byet al.[33]). patients and the NHS if the decision turns out to be

wrong. This information is useful because it contrib-5.2.2 Emulatorsutes to an assessment of whether uncertainty mat-An emulator is essentially a model of a modelters. For example, a decision may be uncertain (veryand can dramatically reduce the computational bur-high error probability), but the consequences (costsden of any type of computationally expensiveto the NHS or NB forgone to the NHS patientmodel. Not all emulators perform well on nonlinearpopulation) might be limited. Conversely, a decisionmodels (e.g. simple linear regression), but there is athat has a low error probability (is more certain)class of emulator (the Gaussian process) that canmight be more important if the costs and NB forgonedeal appropriately with the types of nonlinear mod-are very large. Therefore, when considering alterna-els often used in appraisal. This approach is consid-tive summary measures of uncertainty it is importanterably more complex than PSA but the mathemati-to consider whether they can be interpreted in a waycal properties of these emulators are well establishedthat might help to assess the consequences of uncer-and have been demonstrated through their applica-tainty.tion in health technology assessment.[34]

5.2.3 Linear Approximations 6.1 Confidence Ellipses and Scatter PlotsLinear models provide unbiased estimates of ex-

pected effects and costs without fully expressing When there are only two alternatives (e.g. a sin-uncertainty or variability in the parameters. There- gle active treatment), a joint probability distributionfore, if an unbiased linear approximation to a non- can be represented in the incremental cost-effective-linear model can be found, this will reduce the ness plane. This can be illustrated graphically incomputational burden. However, it is not clear when several equivalent ways, two of which are ellipsesthis will work. In particular, linearization based on and scatter plots.[36] Scatter plots record each simu-



lated estimate of the expected additional effects and • it is difficult to visualize alternative cost-effec-tiveness thresholds in this cost-effect space;additional costs. This is illustrated in figure 2a using

• when represented in two dimensions (or even insimulated output from a probabilistic model of twothree), it is difficult to accurately assess the pro-alternatives, B compared with A.portion of points that lie below the line to provide

The threshold (λ) is represented by the dashed an intuitive estimate of the probability that B isline, with slope of £20 000 per QALY and the circle cost effective compared with A.indicating the expected incremental cost effective- When there is more than one active treatment,ness of B compared with A. This is above the dashed which is commonly the case, scatter plots or confi-line, so B is not regarded as cost effective dence ellipses become impossible to interpret cor-(ICER = £61 680 per QALY). Consideration of the rectly. Figure 2b illustrates a scatter plot when alter-

native C is also considered alongside A and B. Nowuncertainty surrounding a decision to reject B onthe expected effect-cost pairs, rather than the incre-these grounds requires an assessment of the propor-ments, are plotted.tion of points that lie below the dashed line (where B

Intervention A remains cost effective at a thres-is cost effective). Even with only two alternatives tohold of £20 000 per QALY. But consideration ofconsider, there are a number of problems:uncertainty requires some assessment of the propor-tion of points lying below the line for interventionsB and C compared with A. This is difficult enough,but a critical piece of information is also missing:each of the cost-effect points for A, B and C will becorrelated by the structure of the model itself. Forexample, if A has a higher than expected cost, thenB or C might also have a higher than expected cost,etc. Without this information it is impossible toassess the probability that A is cost effective and theerror probability associated with adopting A. This isalso true for other proposed summary measures suchas confidence ellipses, confidence intervals or thefull distribution of NB of each alternative.

6.2 Cost-Effectiveness Acceptability Curves

Cost-effectiveness acceptability curves (CEACs)are constructed by recording the number of timeseach alternative has the highest NB (i.e. is costeffective) from the simulated output of a model. Theprobability (proportion of times) that each is costeffective is plotted for a range of possible cost-effectiveness thresholds. Figure 3 illustrates theCEAC associated with the scatter plot in figure 2b.At first sight, the CEAC seems easier to interpretsince the probability that A, B or C is cost effective(p) and the associated error probability (1 – p) cansimply be read off for any particular threshold.[37]

However, it is important to note that the alterna-tive with highest probability of being cost effectivemay not be the most cost-effective alternative (hav-

0

1

2

3

4

5

6

7

8

9

−0.4 −0.3 −0.2 −0.1 0 0.1 0.2 0.3 0.4

Incremental QALYs

Incr

emen

tal c

osts

(£

× 10

00)

B compared with A

0

2

4

6

8

10

12

14

16

18

0 0.2 0.4 0.6 0.8 1 1.2 1.4

QALYs

Cos

t (£

× 10

00)

Comparing A, B and C

ABC

a

b

Fig. 2. Scatter plots of hypothetical treatment options: (a) B vs A(incremental cost effectiveness); (b) with multiple alternatives (ef-fect-cost pairs). The dashed line represents the threshold (λ) of£20 000 per QALY (0.2 QALYs = £4000). Red circles indicate theexpected incremental cost effectiveness.


792 Claxton

and other alternatives – they provide the probabilitythat a decision based on expected costs and effectswill be correct (p) and the error probability (1 – p).However, they do not provide information about thedifferences in the NBs of the uncertain alternatives,so it is difficult to assess whether the uncertainty‘matters’.

6.3 Other Ways to Present Uncertainty

NICE is primarily interested in thresholds be-tween £20 000 and £30 000 per QALY. A simplealternative to the CEAC and CEAF is to report

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 10 20 30 40 50 60Cost-effectiveness threshold (£ × 1000 per QALY)

Pro

babi

lity

cost

effe

ctiv

e

ABC

A is cost effective C is cost effective

Fig. 3. Cost-effectiveness acceptability curve for hypothetical treat-ment options A, B and C. The dashed line represents the cost-effectiveness frontier.

expected effects, costs, ICERs, the expected NB(expressed in health or money terms) and probabili-ing the highest expected net-benefit) – this is illus-ties for each alternative, as well as the errortrated in figure 3 for thresholds between £24 628probability for a decision based on expected costand £34 000 per QALY. This may seem countereffectiveness. This is illustrated in table III.intuitive, but is easily illustrated in table II, which is

very similar to table I. B is cost effective and has the This indicates that A is cost effective at a thres-hold of £20 000 per QALY, but with a probability ofhighest expected NB, but the probability that it iserror >0.2. This uncertainty is primarily in thecost effective is only 0.4 (two of five), which is achoice between A and C – there is little chance thatlower probability than A (0.6; three of five). TheB will be cost effective. At a threshold of £30 000reason is that when B is better than A it is ‘muchper QALY, C is expected to be cost effective, butbetter’, but when A is better than B it is only ‘a littlethis decision is more uncertain (>0.6 error probabili-bit better’.ty). Again the uncertainty is primarily between A

Therefore, presenting a CEAC alone is not and C. The difference in expected NB between theenough. It is important to indicate which of the uncertain alternatives also gives some indication ofalternatives is expected to be cost effective as well the consequences of error. At £30 000 per QALY,as its probability. This is indicated by the dashed the decision is more uncertain and the consequencesline on figure 3, or by the cost-effectiveness accepta- might be more serious (the difference in expectedbility frontier (CEAF), which is a plot of the NB between C and either A or B is greater) than atprobability that the most cost-effective alternative £20 000 per QALY (between A and either C or B).(with the highest expected NB) is cost effective.[38]

The information in table III can also be presentedThe type of CEAC in figure 3 and the associated graphically as in figure 4. This visual representation

CEAF have significant advantages over scatter plots can be particularly useful for large numbers of alter-natives;[39] indicating which appears to be cost effec-tive, the associated uncertainty, how this uncertaintyis ‘distributed’ over the other alternatives, and theconsequences of uncertainty.

The consequences of an uncertain decision cannot be directly calculated from table III or figure 4 asthey only report the expected NB of the uncertainalternatives. However, this type of information isindicative of the relative importance of uncertaintyin many common circumstances.

Table II. Probability and cost effectiveness

How things Net health benefit Best

could turn treatment treatment best we could doout A B choice if we knewPossibility 1 10 14 B 14





Average 12 13 13.8



Table III. Presenting uncertainty and cost effectiveness

Option Cost Effect ICER λ = £20 000 per QALY λ = £30 000 per QALY

(£) (QALY) (£/QALY) NB (£) probability p(error) NB (£) probability p(error)A 4147 0.593 7722 0.792 0.208 13 656 0.465

B 8363 0.658 ED 4794 0.054 11 373 0.186

C 8907 0.787 24 628a 6827 0.154 14 695 0.348 0.652a Option C versus option A.

ED = extended dominance; ICER = incremental cost-effectiveness ratio; NB = net benefit.

7. Assessing Need for Evidence and the ular groups of parameters can be used to identifyConsequences of an Uncertain Decision where such research should be focused.

7.1.1 Expected Value of InformationAddressing why uncertainty surrounding expec- The expected value of information (EVI) com-

ted costs and effects matters includes asking wheth- bines both the probability of error and the conse-er existing evidence is sufficient and assessing the quences of error in terms of the NBs forgone. It canpossible consequences of an uncertain decision for be expressed in health or monetary values or canthe NHS. Whether more explicit methods might simply be regarded as a metric of the importance ofhelp NICE make these assessments is considered uncertainty.[8,35,40] The principles of EVI have al-below. ready been illustrated in table I and discussed in

section 2.2. As demonstrated, the additional expec-ted net health benefits offered by the technology was7.1 Is the Evidence Sufficient?1 QALY but the expected value of resolving theuncertainty would contribute 0.6 QALY, or £12 000The Methods Guide of 2004[13] suggested thatper patient (at a threshold of £20 000 per QALY).research priorities can be identified on the basis ofThis represents the value of resolving all the uncer-evidence gaps identified by systematic review oftainty for an individual patient. It also represents theeffects and cost-effectiveness analysis. It suggestedmaximum value of acquiring additional evidence,that these may be best prioritized by considering thei.e. the expected value of perfect informationvalue of additional information in reducing the de-(EVPI). Of course, any additional research will nev-gree of decision uncertainty. It also recommendeder resolve all uncertainties but EVPI does place anthat the value of information associated with partic-upper bound on the value of additional evidence. Aninformal indication of the importance of uncertaintycan be made by comparing the per-patient value ofaccess to the technology (the expected additionalNB) to the maximum value of additional evidenceabout its use.

7.1.2 Population Expected Value ofPerfect InformationAdditional evidence can be used to guide the

treatment of all other current and future NHS pa-tients. Therefore, the maximum value of evidence tothe NHS as a whole requires estimates of this currentand future patient population (where the populationEVPI is the discounted sum). This requires a judg-ment to be made about the time over which addition-al evidence that can be acquired in the near future islikely to be useful and relevant. Generally, fixedtime horizons of 10, 15 and 20 years have common-

ABC

(0.792)

(0.154)(0.054)

NB = £7722

NB = £6827NB = £4794

0123456789

10

0.4 0.5 0.6 0.7 0.8 0.9QALYs

Cos

ts (

£ ×

1000

)

Fig. 4. Uncertainty and cost effectiveness for hypothetical treat-ment options A, B and C. The numbers in brackets are theprobabilities of cost effectiveness for a threshold (λ) of £20 000 perQALY (0.2 QALYs = £4000). NB = net benefit.


794 Claxton

ly been used in the health literature as well as the 7.2 What Type of Evidence is Needed?environmental risk and engineering literature.[41]

There is some empirical evidence that suggests thatIf further research is potentially worthwhile, it

clinical information may be valuable for much long- would be useful to have an indication of what typeer (a half-life of 45 years).[42] However, any fixed of additional evidence might be most valuable. Thistime horizon is really a proxy for a complex and can inform the decision of whether guidance withuncertain process of future changes, all of which evidence, or an ‘only in research’ recommendationimpact on cost effectiveness and the future value of might be more appropriate. There are a number ofevidence.[43] In health, some future changes can be alternative measures that can be considered.anticipated (a new technology will be launched, atrial that is recruiting will report or a branded drug

7.2.1 Expected Value of Perfectwill go generic), and differing judgments about timeParameter Information

horizons for different appraisals might be appropri-The analysis of the value of information asso-ate.

ciated with different (groups of) parameters is, inAs well as a simple metric of the relative impor- principle, conducted in a very similar way to the

tance of uncertainty across different appraisals, the EVPI for the decision as a whole. The EVPI for apopulation EVPI can be compared to the expected parameter or group of parameters (EVPPI) is simplycost of additional research, which includes the NB the difference between the expected NB when theirforgone if conducting research requires delaying uncertainty is resolved (and a different decision canapproval of a technology that appears to be cost be made) and the expected NB given existing uncer-effective based on current evidence. If these expec- tainty.[8,35,44]

ted opportunity costs of research exceed the popula- EVPPIs can be used as a simple metric of thetion EVPI (maximum benefits) then the research is relative importance (sensitivity) of different types ofnot worthwhile – the resources could generate more parameters and sources of uncertainty in contribut-health improvement by being used elsewhere in the ing to the overall EVPI.[45] As a simple measure ofNHS, and guidance should be based on current sensitivity it has a number of advantages: (i) itestimates of expected cost effectiveness. combines both the importance of the parameter

(how strongly it is related to differences in NB) andits uncertainty; (ii) it is directly related to whether7.1.3 Computationthe uncertainty matters (whether the decision

The per-patient EVPI is easily calculated from changes for different possible values); and (iii) itthe simulated output of a probabilistic model (see does not require a linear relationship between inputstable I and section 3), which may include a formal and outputs. In addition, it can be expressed inassessment of other sources of uncertainty (see sec- health or monetary values and either per patient, ortion 4). Therefore, the time and effort required to for the population of NHS patients.calculate EVPI is negligible once all the evidence When population EVPPI is expressed in moneta-has been assembled and judgments have been made ry terms it can be directly compared with the expec-in constructing a model, synthesizing evidence and ted opportunity costs of the type of research thatassigning probability distributions to all the sources might be needed to provide the evidence. This isof uncertainty. Calculation of the population EVPI important because some uncertainties are relativelyis also trivial given that estimates of incidence and cheap to resolve (in terms of time and resource)prevalence are already required in the evidence re- compared with others (e.g. an observational study toview and the appropriate discount rate is detailed in link a clinical endpoint to quality of life comparedNICE guidance. However, a judgment about the with an RCT of long-term relative treatment effect).time horizon is required. The uncertainty in such Which source of uncertainty is most important re-judgments can be incorporated in the analysis and quires a comparison of these benefits and opportuni-will influence the ‘expected’ population EVPI.[43] ty costs.



7.2.2 Computation 7.2.3 Other Measures of Sensitivityand ImportanceEvaluating EVPPI does come at a computationalOther measures of the importance of parameterscost. For linear models, each estimate of EVPPI

are also available. For example, there is a large bodyrequires some additional computation (the manipu-of literature outside health on the use of conditionallation of the simulated values rather than repeatedPSA.[46] This can provide a variety of different mea-simulations). When the model itself is not computa-sures of sensitivity, e.g. the contribution that a para-tionally expensive it is a generally manageable ex-meter makes to the overall uncertainty in modelpense. However, if the model is nonlinear, EVPPIoutputs (contribution to variance in expected NB).may require many repeated runs of the same proba-Unfortunately, these approaches are often just asbilistic model, which can become prohibitively ex-computationally demanding as EVPPI, are not al-pensive given the time constraints of NICE apprais-ways related to whether the uncertainty mattersal. Therefore the computational expense of EVPPI(whether the decision changes) and do not provide aneeds to be justified. For example, estimates ofmeasure of value that can be compared with theEVPPI may be justified if the analysis of populationopportunity costs of research. They offer no realEVPI suggests that additional evidence might beadvantage when parameters are correlated.required (population EVPI is greater than the expec-

ted opportunity costs). If justified, there are a num- Much simpler approaches are also possible. Onceber of ways to reduce computation expense:[44]

parameter values and associated expected NBs have• It is more efficient and more informative to first been simulated and recorded, it might be tempting to

consider a limited number of groups of para- summarize the correlation between them. However,meters, informed by the types of research re- simple correlation can be high, even when the con-quired, e.g. RCT, survey of QALYs, or an obser- tribution to variance in NB is low. Another approachvational epidemiological study. If there is sub- would be to use analysis of co-variance methodsstantial EVPPI associated with a particular (ANCOVA), which can summarize the proportiongroup, only then conduct additional analysis to of the variance in the output ‘explained’ by variationexplore which particular source of uncertainty in the input parameters.[8] However, in its simplewithin the group matters most. form, this assumes a linear relationship between

inputs and outputs. Therefore it offers little advan-• The number of simulations to conduct and wheretage in terms of computation (i.e. if the model isto take them (inner and outer loops) can be madelinear, EVPPI is not particularly expensive anyway).more efficient.They also have some disadvantages: they are not• In addition, computational burden can be reduceddirectly related to whether the uncertainty mattersby faster processing and more efficient sampling;(whether the decision changes) and do not provide afinding mathematical linear approximations tomeasure of value that can be compared with thethe model itself or to nonlinear relationshipsopportunity costs of research.within it; and using emulators (see section 5).

There are other considerations that also impact These simpler methods might provide some indi-on computational expense. First, EVPPI can be eval- cation of where uncertainty matters in a nonlinearuated for a group of correlated parameters (often model – even though they would be biased. The biaswithout additional computation expense). However, will depend, among other things, on how nonlinearthe EVPPI of each, individually, will not capture the the model really is. However, evaluating EVPPI asrelationship to others and may under/overestimate if the model was linear would only require similarthe value of more evidence. When certain groups of computation, but have the advantages describedparameters are correlated, the type of repeated sam- above. It is not clear whether a biased estimate ofpling can be required even in a linear model.[35] sensitivity or EVPPI is better than none at all, orSecond, when considering the order in which re- how much nonlinearity or correlation and thereforesearch studies might best be conducted, measures of potential bias would render them of no value. Thisthe EVPPI of a sequence of studies is possible but question is very difficult to answer, as the appropri-generally requires more computation.[46] ate comparison ought to be between the quality of


796 Claxton

the assessment of the need for evidence with and nomics (real options) to evaluate this complex andwithout such imperfect information. But this is a uncertain prospect.[10]

common question for NICE when considering im-7.3.2 Value of Evidence Forgoneperfect evidence of cost effectiveness.All decisions are uncertain, particularly when

they are made close to launch of a product, and a7.3 Assessing the Consequences of andecision to approve a technology will inevitablyUncertain Decision for the NHShave an impact on the prospects of acquiring evi-dence to inform judgements about its use in theAlmost all decisions faced by NICE will imposefuture. The upper bound on the value of evidencesome costs on the NHS which, if the guidance isthat may be forgone by issuing positive guidance isreversed at a future date, cannot be recovered. Im-the population EVPI. This can be compared with theportantly, a decision to approve a technology willadditional expected NBs of early access to the tech-inevitably have an impact on the prospects of ac-nology. If the benefits of early access exceed thequiring evidence to support its use in the future.maximum value of evidence forgone, immediate

7.3.1 Costs of Changing Clinical Practice positive guidance is worthwhile (even though someThe sunk or irreversible costs of changing clin- evidence will be forgone). Where the maximum

ical practice may include investment in equipment (evidence) costs exceed the benefits of early access,and the training of specialist staff or simply engag- a judgment will be necessary about when and whating in activities to ensure the guidance is implement- type of evidence might become available by issuinged at a local level. The importance of these costs for an ‘only in research’ recommendation. Of course,early adoption of a technology requires, first, some these judgements can, in principle, be explicitlyassessment of the probability that guidance will incorporated in the analysis and can be linked to thechange (at some point in time), and, second, the value of particular types of evidence that might bevalue of the sunk cost forgone (over time). If the forgone through estimates of EVPPI and EVSI forsunk costs expected to be forgone exceed the expec- particular research designs.[35,47,48] It may not beted additional net health benefits of immediate adop- possible or desirable to make explicit in formaltion, then it may be worth withholding approval analysis all the judgments required for this type ofuntil more secure guidance can be issued.[9,10]

assessment. However, the question is whether theThe uncertainty surrounding cost effectiveness type of explicit analysis that can be made available

gives some indication of how vulnerable the gui- is a useful starting point for deliberations arounddance may be to new evidence. In addition, judg- ‘only in research’ recommendations?ment about whether research is likely to be conduct-ed and whether the range of possible results is likely 8. Conclusionsto change guidance is also needed, which can bebased on estimates of predicted posteriors and the This paper has described the key principles ofexpected value of sample information (EVSI).[35,40] why an assessment of uncertainty and its conse-It is also possible to evaluate the maximum sunk quences is critical to the decisions that NICE mustcosts that the NHS can afford while still adopting the make. In doing so, it poses the question of whichtechnology,[6] or integrating estimates of sunk costs formal methods may be useful in helping NICE andto consider the overall NBs of the decisions to adopt its advisory committees make such assessments. Innow (no research), adopt and conduct research, or coming to a view about whether more explicit meth-conduct research and withhold adoption until it re- ods of assessment are required, it should be recog-ports.[9] However, research and new evidence is only nized that no formal analysis can capture everythingone source of change that might lead to a change in that might be important to the NHS. The question isguidance. Others include changes in technologies, ‘do they capture enough to be useful to the delibera-prices or evidence from other sources.[43] Attempts tions that take place in the appraisal process?’ Forhave been made to explicitly model these types of example, few would argue that cost-effectivenessfuture changes or use methods from financial eco- analysis can capture everything of importance, but it



13. National Institute for Health and Clinical Excellence. NICEis generally accepted that if it is based on wellguide to the methods of health technology appraisal. London:

conducted analysis it captures enough to be at least a NICE, 200414. Cooksey D. A review of UK health research funding. London:useful starting point for deliberation. The same prin-

Stationery Office, 2006ciples ought to guide the choice of methods for the15. Office of Fair Trading. The pharmaceutical price regulation

characterization and presentation of uncertainty and scheme: an OFT market study. London: Office of Fair Trading,2007its consequences.

16. Claxton K, Briggs A, Buxton MJ, et al. Value-based pricing forNHS drugs: an opportunity not to be missed? BMJ 2008; 336:

Acknowledgements 251-417. Claxton K, Sculpher MJ, McCabe C, et al. Probabilistic sensitiv-

This paper was initially prepared as a briefing paper for ity analysis for NICE technology assessment: not an optionalextra. Health Econ 2005; 14: 339-47NICE as part of the process of updating the Institute’s 2004

18. Thompson K, Graham J. Going beyond the single number: usingGuide to the Methods of Technology Appraisal. The workprobabilistic risk assessment to improve risk management.was funded by NICE through its Decision Support UnitHuman Ecologic Risk Assess 1996; 2: 1008-34(DSU), which is based at the universities of Sheffield,

19. O’Hagan A, Luce BR. A primer on Bayesian statistics in healthLeicester, York, Leeds and at the London School of Hygiene economics and outcomes research. Bethesda (MD): Bayesianand Tropical Medicine. initiative in health economic and outcomes research,

The author has no conflicts of interest that are directly MEDTAP International, 200320. O’Hagan A, Buck CE, Daneshkhah A, et al. Uncertain judge-related to the contents of this article.

ments: eliciting expert probabilities. Chichester: John WileyThe author thanks members of the DSU who commentedand Sons, 2006on the briefing document that forms the basis of this paper as

21. Sutton A, Ades AE, Cooper N, et al., on behalf of the NICEwell as Iain Chalmers, Alex Sutton, Alan Brennan, Louise Decision Support Unit. Use of indirect and mixed treatmentLongworth and Carole Longson, who provided helpful com- comparisons for technology assessment. Pharmacoeconomicsments on earlier drafts of this paper. All errors and omissions 2008; 26 (9): 753-67are the responsibility of the author. 22. Ades AE, Sutton AJ. Multiparameter evidence synthesis in

epidemiology and medical decision-making: current approach-es. JRSS (A) 2005; 16 (1): 5-35

References 23. Ades AE, Claxton K, Sculpher MJ. Evidence synthesis, para-1. Culyer AJ, McCabe C, Briggs A, et al. Searching for a thres- meter correlation, and probabilistic sensitivity analysis. Health

hold, not setting one: the role of the National Institute for Econ 2006; 15: 373-81Health and Clinical Excellence. J Health Serv Res Pol 2007; 24. Bojke L, Claxton K, Palmer S, et al. Defining and characterising12: 56-8 structural uncertainty in decision analytic models [CHE re-

2. McCabe C, Claxton K, Culyer AJ. The NICE cost-effectiveness search paper 9]. York: Centre for Health Economics, Univer-threshold: what it is and what that means. Pharmacoeconomics sity of York, 20062008; 26 (9): 733-44 25. McCabe C, Dixon S. Testing the validity of cost-effectiveness

3. Phelps CE, Mushlin AI. On the (near) equivalence of cost- models. Pharmacoeconomics 2000; 17 (5): 501-13effectiveness and cost-benefit analyses. Int J Technol Assess 26. Draper D. Assessment and propagation of model uncertainty.Health Care 1991; 7 (1): 12-21 J Royal Stat Soc 1995; 57: 45-97

4. Stinnett A, Mullahy J. Net health benefits: a new framework for 27. Chatfield C. Model uncertainty, data mining and statisticalthe analysis of uncertainty in cost-effectiveness analysis. Med inference (with discussion). J Royal Stat Soc 1995; 158:Decis Making 1998; 18: S68-80 419-66

5. Sculpher MJ. Subgroups and heterogeneity in cost-effectiveness28. Bojke L, Claxton K, Bravo Vergel Y. Using expert elicitation toanalysis. Pharmacoeconomics 2008; 26 (9): 799-806

resolve issues of structural uncertainty in decision analytic6. Claxton K. The irrelevance of inference: a decision making models. Med Dec Making 2007; 27 (1): E8

approach to the stochastic evaluation of health care technolo-29. White IR, Pocock SJ, Wang D. Eliciting and using expertgies. J Health Econ 1999; 18: 342-64

opinions about dropout bias in randomised controlled trials.7. Briggs A, Sculpher M. An introduction to Markov modelling forClin Trials 2007; 4 (2): 125-39economic evaluation. Pharmacoeconomics 1998; 13: 397-409

30. Leal J, Wordsworth S, Legood R, et al. Eliciting expert opinion8. Briggs A, Claxton K, Sculpher MJ. Decision analytic modellingfor economic models. Value Health 2007; 10 (3): 195-203for the evaluation of health technologies. Oxford: Oxford

31. Griffin S, Claxton K, Hawkins N, et al. Probabilistic analysisUniversity Press, 2006and computationally expensive models: necessary and re-9. Eckermann S, Willan A. Expected value of information andquired? Value Health 2006; 9: 244-52decision making in HTA. Health Econ 2007; 16: 195-209

32. Edmunds WJ, Medley GF, Nokes DJ. Evaluating the cost-10. Palmer S, Smith PC. Incorporating option values into the eco-effectiveness of vaccination programmes: a dynamic perspec-nomic evaluation of health care technologies. J Health Econtive. Stat Med 1999; 18 (23): 3263-822000; 19: 755-66

33. O’Hagan A, Stevenson M, Madan J. Monte Carlo probabilistic11. Griffin S, Claxton K, Palmer S, et al. Dangerous omissions: thesensitivity analysis for patient level simulation models: effi-consequences of ignoring decision uncertainty. Med Dec Mak-cient estimation of mean and variance using ANOVA. Healthing 2007; 27 (1): E8Econ 2007; 16 (10): 1009-2312. Chalkidou K, Hoy A, Littlejohns P. Making a decision to wait

for more evidence: when the National Institute for Health and 34. Stevenson M, Oakley J, Chilcott J. Gaussian process modellingClinical Excellence recommends a technology only in the in conjunction with individual patient simulation modelling: acontext of research. J R Soc Med 2007; 100 (10): 453-60 case study describing the calculation of cost-effectiveness


798 Claxton

ratios for the treatment of established osteoporosis. Med Dec 43. Philips Z, Claxton K, Palmer P. The half-life of truth: what areMaking 2004; 24: 89-100 appropriate time horizons for research decisions? Med Dec

Making 2008; 28: 287-9935. Ades AE, Lu G, Claxton K. Expected value of sample informa-tion in medical decision modelling. Med Dec Making 2004; 44. Brennan A, Kharroubi S, O’Hagan A, et al. Calculating partial24: 207-27 expected value of perfect information via Monte Carlo sam-

pling algorithms. Med Dec Making 2007; 27: 448-7036. Van Hout BA, Al MJ, Gordon GS, et al. Costs, effects and c/e-ratios alongside a clinical trial. Health Econ 1994; 3: 309-19 45. Felli JC, Hazen GB. Sensitivity analysis and the expected value

of perfect information. Med Dec Making 1998; 18: 95-10937. Fenwick E, Claxton K, Sculpher MJ. Representing uncertainty:the role of cost-effectiveness acceptability curves. Health Econ 46. Griffin S, Claxton K. Interpreting the expected value of perfect2001; 10: 779-89 information about parameters. Med Dec Making 2007; 27 (1):

38. Fenwick E, O’Brien B, Briggs AH. Cost-effectiveness accepta- E8bility curves: facts, fallacies and frequently asked questions. 47. Brennan A, Kharroubi S. Expected value of sample informationHealth Econ 2004; 13: 405-15

for Weibull survival data. Health Econ 2007: 16: 1205-2539. Colbourn T, Asseburg C, Bojke L, et al. Preventive strategies 48. Conti S, Claxton K. Dimensions of design space: a decisionfor group B streptococcal and other bacterial infections in early

theoretic approach to optimal research design [CHE researchinfancy: cost effectiveness and value of information analyses.paper 38]. York: Centre for Health Economics, University ofBMJ 2007; 335: 655-62York, 2008

40. Pratt J, Raiffa H, Schlaifer R. Statistical decision theory. Cam-bridge (MA): MIT Press, 1995

41. Yokota F, Thompson KM. Value of information literature ana- Correspondence: Professor Karl Claxton, Centre for Healthlysis: a review of applications in health risk management. Med Economics and Department of Economics and RelatedDec Making 2004; 24: 287-98

Studies, University of York, Heslington, York, YO10 5DD,42. Poynard T, Munteanu M, Ratziu V, et al. Truth survival in

UK.clinical research: an evidence-based requiem? Ann Intern Med2002; 136: 888-95 E-mail: [email protected]


exploring uncertainty in cost-effectiveness analysis · exploring uncertainty in cost ... ....

Documents