EVIDENCE-BASED? CAVEAT EMPTOR II!

Presented to the NAMI Symposium: “What Do We Mean By Evidence-based Practices?”

byBryan R. Luce, Ph.D., MBA

The MEDTAP Institute at UBCMonday, June 20 2005

Austin, Texas

Caveat Emptor* “I”: Premise

Today, being evidence-based is “de rigueur”.

Steinberg, Luce, Health Affairs 2005

Selected Organizations Using “EBM”

• BCBS’s Technology Evaluation Center

• US Preventive Services Task Force

• Clinical Practice Guidelines

• CMS Medicare Coverage Advisory Committee (MCAC)

• AHRQ’s Evidence-Based Practice Centers

Organizations Using EBM (Cont.)

• UK’s National Institute for Clinical Excellence (NICE)

• AMCP’s Format for Formulary Submission – Multiple MCOs

• OHSU Drug Effectiveness Review Project – 13 Medicaid Agencies – Consumers’ Union BestBuyDrugs website– AARP’s ResearchRx website

Organizations Using EBM (Cont.)

• CMS’s MMA: Comparative Effectiveness

• CMS Interim Coverage: PCT/Registries

• Institute of Medicine: EBM/Comparative Effectiveness Private-Public Initiative

Evidence-Based?Caveat Emptor!

Earl P. Steinberg, MDBryan R. Luce, Ph.D.

Health AffairsJanuary, February 2005

Caveat Emptor I: Conclusions

• Rigorous methods for evaluating clinical studies now exist, but– Methods not applied consistently– Results not always interpreted properly

Thus…

– Much variation in validity of “EB” decisions

Also…

– Evidence may be available for some but not all issues Or…

Caveat Emptor I: Conclusions (cont.)

Available evidence may not be applicable to decision at hand

Thus…

– Decision-makers should not blindly assume the evidence-based label truly applies

And, finally…

– EBM analysts should guide decision-makers as to applicability to populations of interest and conditions to which evidence applies.

Now…Caveat Emptor “II”

• Poor agreement…

– Definition of “EBM”

– Definition of “systematic review”

– The relevance and scope of evidence

• Other issues to be resolved

• Recommendation for evidence-based

evaluations for policy decisions

Definitional Clarity of EBM

• EBM is not a unitary concept*– Evidence based individual physician-patient

decisions– Evidence-based policy/group-focused decisions

• Each application has/should have own unique definition and designation

(Eddy, 2005)

EBM’s Early Concept

• Based on systematic review (meta-analysis) of high quality RCT evidence as proposed by Archie Cochrane and others.

• Early definition: “The conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients.”

• Debate in UK literature: “Leads to cookbook medicine”?

Sackett’s Evolving Definition*

Countering cookbook charge, Sackett et al added: “The practice of EBM means integrating individual clinical expertise with the best available external clinical evidence from systematic research [and] individual patients’ predicaments, rights and preferences in making clinical decisions about their care”.

*Sackett et al., EBM, What it is and what it isn’t. 1996, BMJ

Thus….

Sackett et al sought to expand the concept of EBM to include clinical judgment and patient values. This could (and probably should) include real world clinical practice issues such as convenience, adherence factors, even economic effects as they may pertain to patient adherence.

Result: EBM for individual decision-making no longer prisoner of RCT; also, is a much less precise process & concept.

What About Policy/Group EBM?

• Terminology: Since policy/group EB decision-making is different concept, we need different term:– How about “EBG”? (Eddy, 2005)

• Second, definition?– We could find none in literature

– So let’s look at practice

EBG in Practice: Three Groups

• The “efficacy” group

• The “effectiveness” group

• The “cost-effectiveness” group

More Definitions

• Efficacy: A healthcare intervention is considered “efficacious” when there is evidence that the intervention is beneficial when administered by experts in a research setting.

• Effectiveness: An intervention is considered “effective” when there is evidence of benefit when administered by community-based physicians to real-world patients in real-world clinical settings.

Steinberg, Luce, 2005

The EBG “Efficacy” Group

• Mantra is “minimize bias”

– Opt for maximizing internal validity of studies

at expense of generalizability

• Examples of organizations:

– Cochrane Collaboration

– OHSU’s Drug Effectiveness Review Project

• Medicaid agencies (?), Consumers Union, AARP

The EBG “Effectiveness” Group

• Mantra: “Is it effective in the real world?”– Opt for generalizability to populations and clinical

settings of interest (incl non-experimental evidence)

• Examples of organizations:– CMS (MCAC, MMA, PCT/Registry Interim Coverage

Policy)

– AHRQ’s EPCs

– BCBS TEC

– Clinical Practice Guidelines

– IOM’s Effectiveness Initiative

The EBG Cost-Effectiveness Group

• Mantra is “real world effectiveness and real world value for money”– Opt for generalizability often over long haul and

accuracy over precision

• Examples of organizations:– Academy of Managed Care Pharmacy

– U.K.’s National Institute for Clinical Excellence (NICE)

– AHRQ’s EPCs

Possible EBG Definition

“The conscientious, explicit and judicious use of current best evidence in making health policy decisions affecting groups of patients.”

But this begs the question, do we care about efficacy, effectiveness or cost-effectiveness?

My answer: Not Efficacy! Thus….

Suggested EBG Definition

“The conscientious, explicit and judicious use of current best evidence in making health policy decisions affecting groups of patients. The practice of ‘EBG’ means integrating evidence from real world practice with the best available external (RCT) clinical evidence from systematic research. EBG may or may not include economic evaluation.”

Next… Systematic Review

• How defined?

• How performed?

Sackett’s Definition

“A summary of the medical literature that uses explicit methods to perform a thorough literature search and critical appraisal of individual studies and that uses appropriate statistical techniques to combine these valid studies.”

Sackett, et al. 2001

Does Practice Measure up to Sackett’s Definition?

• Mainly pertains to meta-analysis of similar RCTs, thus– Pertains to efficacy group– Meta-analysis portion of effectiveness and CE groups

• But, non-efficacy groups also speak of using systematic review process– Using term descriptively– Do not imply formal statistical meta-analysis

However (as aside)…

• If formal statistical approach is desired, it is possible to employ a formal Bayesian statistical approach to different types of evidence (e.g. RCT & observational data). (Or, as AMCP: decision modeling)

• NICE accepts such studies.

• But, main point is….

When “Systematic Review” is claimed…

• You do not know what is meant.

So….we need– New terminology– With definitions

Suggested Terminology & Definitions

• “Systematic review of trials”, or “SRT” would refer to the structured meta-analysis process and generally result in best estimate of efficacy;

• “Systematic review of evidence” or “SRE” would refer to the broader, less structure review of evidence and generally result in best estimate of effectiveness. Best practice would include SRT.

Relevance & Scope of Evidence for EBG Process

“Best” evidence should include or consider:– All relevant RCTs – All relevant evidence from observational studies (eg

claims data bases)– Estimates of final health outcomes using surrogate

markers when needed – Economic outcomes of affected parties– Patient-reported outcomes including preferences– Possibly case studies, expert opinion, testimonials

Some EBG-Related Issues

• If effectiveness is what is wanted, to what extent and under what circumstances does “efficacy” suffice for “effectiveness”?

• How does one translate efficacy to effectiveness?

Other EBG-Related Issues (cont.)

• Under what circumstances is evidence considered to be inadequate?

• To what extent should inadequate or lack of evidence mean “not effective”?

Wrap Up

• Caveat Emptor I: “Decision-makers should not blindly assume the evidence-based label truly applies”

• Caveat Emptor II: “We need to be explicit with respect to terminology & definitions and we need to agree on what is the appropriate evidence base for E-B policy/group decision-making.”

And…

• Just as Sackett et all responded to “cookbook” criticisms by acknowledging need for clinical judgment and patient preferences/values for EBM…

• EBG requires multiple types of evidence to evaluate real world effectiveness– Modeling will commonly be needed

– Answers will not necessarily be unambiguous, but…

With respect to the primacy of effectiveness over efficacy…

“…an imprecise estimate of the right concept is superior to a precise estimate

of a wrong concept” (Mishan, 1972)

To put it another way…

“It is better to be approximately right than precisely wrong.”*

*L.Read, 1988