oregon epcoregon epc evidence-based practice centers created in 1997; now 13 centers produce...

OREGON EPC

Evidence-based Practice Centers

• Created in 1997; now 13 centers• Produce

– “evidence reports” – systematic reviews– technology assessments– “rapid reviews”– meta-analyses and cost analyses– analysis of large databases

• Work with public and private sector partners

OREGON EPC

Evidence-based Medicine

Mark Helfand, MDDirector

Oregon Evidence-based Practice Center

OREGON EPC

What is the kind and strength of the

evidence you are relying on to make a recommendation?

The Question:

OREGON EPC

What does evidence-based mean?

• A comprehensive, systematic, open minded review of all the evidence

• The evidence determines the conclusion, not vice versa

• Not, the citation of papers supporting a preformed conclusion (and trashing of those that don’t)

• Not, the use of evidence when it is ‘positive’ but judgement when it isn’t

OREGON EPC

Systematic literature reviews

• Are systematic to remove bias in finding and reviewing the literature.

OREGON EPC

Systematic literature reviews

• Are systematic to remove bias in finding and reviewing the literature.– Experts may interpret the data (and

their own experience) differently.

OREGON EPC

How sure are we?Expert estimates of breast

implant rupture rates

0% 0.2% 0.5% 1% 1% 1% 1.5% 2% 3% 3% 4%

5% 5% 5% 5% 5% 5% 5% 5% 6% 6% 6% 8%

10% 10% 10% 10% 13% 13% 15% 15% 18%

20% 20% 20% 25% 25% 25% 30% 30% 40%

50% 50% 50% 62% 70% 73% 75% 75% 75%

75% 80% 80% 80% 80% 80% 80% 100%

Source: Dr. David Eddy

OREGON EPC

Experts estimates of the effect of colon cancer screening on

chance of dying

0% 25% 50% 75% 100%

Source: Dr. David Eddy

OREGON EPC

Experts’ estimates of probability of acute retention in men with

BPH

0

5

10

15

20

25

30

35

0% 20% 40% 60% 80% 100%

Number ofRespondants

Source: Dr. David Eddy

OREGON EPC

Systematic literature reviews

• Are systematic to remove bias in finding and reviewing the literature.– Studies with disappointing results

may get less attention

OREGON EPC

Trial Number Groups RESULTS

114 302 40mg bid 80mg bid

Total improvement at all doses compared with PLACEBO

115 419 20 mg bid60 mg bid 100mg bid

Total improvement at all doses compared with PLACEBO

106 139 20 mg bid, 60mg bid

Borderline improvement at 60 mg dose compared with PLACEBO

104 153* 20 mg bid, 40 mg bid

No improvement compared with placebo at either dose.

303 (32 wks)

294 20 mg bid, 60 mg bid 80mg bid

Lower relapse rate (31% to 36%) vs. PLACEBO (57%)

*Excludes 5 mg bid group

OREGON EPC

Trial 114

OREGON EPC

Systematic literature reviews

• Are systematic to remove bias in finding and reviewing the literature.– Experts may underplay controversy

or select only supportive evidence

OREGON EPC

Simpson et al, 2004

OREGON EPCSimpson et al, 2004

OREGON EPC

OREGON EPC

In a double-blind study vs risperidone…GEODON sustained control of positive symptoms at 1 year

1

OREGON EPC

In a double-blind study vs risperidone…GEODON sustained control of positive symptoms at 1 year

1

OREGON EPC

Systematic literature reviews

• Are systematic to remove bias in finding and reviewing the literature.– Experts may underplay controversy or

select only supportive evidence

• Emphasize the best evidence

OREGON EPC

The best evidence

• Reflects patients’ concerns– By addressing health outcomes

patients, their caregivers, and families care about

OREGON EPC

The best evidence

• Reflects patients’ concerns– By addressing health outcomes

patients, their caregivers, and families care about

•Help you feel similar to other people•Help you feel less lonely and removed from others•Help you feel more hopeful and happy•Allow you to think and express yourself more clearly

OREGON EPC

Selecting questions

• Researchers often use their own curiosity or research interest as the basis for selecting questions.

• They often use “standard” scales and measures instead of seeking a deeper understand of the patient’s well-being and quality of life.

OREGON EPC

Selecting questions

• Our premise is that important questions arise from practice, and from life. “Experts in practice”--and patients--select the populations, interventions, and outcome measures of interest.

OREGON EPC

The best evidence

• Reflects patients’ concerns– By addressing health outcomes

patients, their caregivers, and families care about

– By using simple measures of benefit and risk

OREGON EPC

Example

Relative benefit

PRESS RELEASE

“half as many patients treated with DRUG A experienced dry mouth.”

Absolute benefit

DATA FROM STUDY

1/100 vs. 2/100 or1 in 10 vs. 2 in 10

972

OREGON EPC

• Define the strengths and limits of the evidence.

• Clarify what is based on evidence and what is based on other grounds.

• Do not necessarily tell you what to do when the evidence is limited. Other factors, such as equity, clinical judgment, values, and preferences play a role in using the evidence.

Why use systematic literature reviews?

OREGON EPC

+

= Evidence-based decision-

making

+

OREGON EPC

An evidence-based decision process

• Makes use of an independent, systematic review of the evidence

• Employs rules for linking evidence to recommendations

• Produce explicit, defensible recommendations

OREGON EPC

Oregon ApproachWhat are we after?

• Systematic drug-class reviews should address questions that reflect clinicians’ and patients’ concerns.

• Decision-makers should begin to wrestle with the idea of what is good evidence.

• Manufacturers should gain market share if they produce good evidence of superiority over other drugs in a class.

• Patients, caregivers, payers (and NAMI) should demand better evidence about outcomes that matter !

OREGON EPC

OREGON EPC

Drug Class Review on

Atypical Antipsychotics

OREGON EPC

Included Drugs

Clozapine not posted

risperidone (1993) not posted

olanzapine (1996) not posted

quetiapine (1997) not posted

ziprasidone (2001) posted

aripiprazole (2002) posted

OREGON EPC

Eligible Outcomes

OREGON EPC

Results• 196 studies included overall

– 33 head-to-head – 24 placebo-controlled– 58 active controlled– 63 observational studies– 18 systematic reviews

• 427 study publications excluded

OREGON EPC

SchizophreniaHead to Head Trials

• 3 Effectiveness Trials– 12 month pragmatic trial of olanzapine,

risperidone or continuing typical AP– One 12-month switching study of olanzapine

& risperidone– InterSept trial of clozapine and olanzapine

to prevent suicidality found clozapine superior

• 30 Efficacy Trials

OREGON EPC

Head to head trials in outpatients

OREGON EPC

Summary: Benefits

•Clozapine, olanzapine and risperidone had similar efficacy with two exceptions

–Clozapine > olanzapine in suicidality/suicide prevention

– Olanzapine > risperidone in reducing rates of relapse

•Aripiprazole, quetiapine, and ziprasidone: Evidence too limited to say

OREGON EPC

•Weight gain•Greater risk for olanzapine than risperidone •Results mixed in long-term observational studies

•Diabetes mellitus•Risk greater for olanzapine than risperidone, but studies had mixed results•Risk with clozapine relative to others not clear •Limited evidence on quetiapine

•Other long-term safety•No conclusions about comparative safety can be made

Summary: Harms

OREGON EPC

Other harms

• Movement disorders• Somnolence• Hyperprolactinemia/sexual

dysfunction• Long QT interval• Bone marrow problems

OREGON EPC

Outpatient studies

Better head-to-head comparisons of antipsychotics are needed to discern the relative efficacy and safety profiles of these compounds.

OREGON EPC

What we can do together

1. select and refine questions that puts patients’ and caregivers’ concerns center stage

2. Rely on unbiased reviews to inform patients, families, and clinicians

3. Promote an evidence-based process, not just systematic reviews.

4. Promote higher standards for evidence about treatments for mental illnesses

OREGON EPC

Observational Studies: Long-term Safety

• 48 studies, 6 months in duration• primarily schizophrenia patients• 8 head-to-head cohort studies• 10 AAP versus typical AP cohort studies• 29 descriptive epidemiologic studies

• 1 case-control study • Death: Rates ranged from 0.1% to 3.3% for

clozapine, quetiapine and risperidone (7 uncontrolled studies)

OREGON EPC

Criticism

• “By adhering to rigorous rules of inclusion, the process maximizes the validity of assessing proven treatment efficacy (strength), while it ignores or discards other germane but less statistically rigorous evidence of real-world effectiveness and cost-effectiveness (weakness).

OREGON EPC

Our response

• We agree controlled trials ignore important aspects of effectiveness…

OREGON EPC

Limitations of RCTs

• There aren’t enough of them.

• They test interventions that may or may not fit easily into practice.

• They often don’t tell you about important subgroups.

• They may not extend for a long time.

OREGON EPC

More limitations of RCTs

• Design features are poorly adapted to the purpose of assessing average effectiveness– Populations

• run-in periods• Exclusions

– Comparators and comparisons– Outcome measures

• Followup period

– Feasibility• Implementation costs• Maintenance costs

OREGON EPC

Most common problems with head-to-head trials

• Doses of the different drugs aren’t equivalent.

• Strategies for using the drugs aren’t realistic.

• Usually, focus on efficacy or harms but not on both

• Do not address all important outcomes

OREGON EPC

RCTs & harms

• Design features are poorly adapted to the purpose of assessing harms– run-in periods– exclusions of susceptible people

• Reporting is poor•unreported•Selectively reported•Misleadingly reported•Lack of severity data

OREGON EPC

Applicability: How to bias an efficacy study and still

get a “good-quality” rating

• select compliant patients• dilute the control group

interventions• measure only certain outcomes• cheat

– selective use of cut-off dates– what are the norms?

OREGON EPC

• We agree controlled trials ignore important aspects of effectiveness…

• and agree on what information we’d like to have.

OREGON EPC

Quality of the evidenceat 4 levels

1.Type of study.2.Quality of each study

based on study design.3.Overall quality of the

evidence for a key question.

OREGON EPC

1. Types of studies

• case reports, case series• animal studies• studies of etiology• prospective cohort studies• “open-label” controlled or

uncontrolled studies• randomized trials

OREGON EPC

2. Quality of individual studies

• quality (“good,” “fair,” or “poor”) for each type of study design

• Use of random allocation• Concealed allocation• Double-blind method• Exclusions after randomization

• applicability

OREGON EPC

• Initial assembly of comparable groups

• Maintenance of comparable groups• Minimal loss to follow-up• Measurements: equal, reliable, valid• Clear definition of interventions• All important outcomes considered• Intention-to-treat analysis OHSU EPC

Internal Validity Criteria for RCTs & cohort studies

OREGON EPC

3. Evidence at each linkage

• Aggregate internal validity: Are there any studies with good design (for the question) that were also well-conducted? Is the “best evidence” of good internal validity?

• Consistency/coherence: Do studies conflict in their findings? Is there a body of supporting evidence so that the “best evidence” makes sense?

OREGON EPC

3. Rating each link in the AF

• Quality and consistency of studies– large numbers of patients– consistent results across studies

• Applicability of studies– patient populations, interventions,

outcomes like those of interest to the organization

– “real life” evidence not just “efficacy”– attention to harms

OREGON EPC

• Define the strengths and limits of the evidence.

• Clarify what is based on evidence and what is based on other grounds.

• Do not necessarily tell you what to do when the evidence is limited. Other factors, such as equity, clinical judgment, values, and preferences play a role in using the evidence.

Systematic literature reviews

OREGON EPC

What Does it Mean for Decisions

to be “Evidence-Based”?

• Decisions are based on “best evidence”

• Best evidence:– Is unbiased– Is appropriate for decision at hand– Includes all germane evidence

Luce

OREGON EPC

An evidence-based decision process

• Makes use of an independent, systematic review of the evidence

Employs rules for linking evidence to recommendations

Produce explicit, defensible recommendations

OREGON EPC

Strength of recommendations

Estimate of Net Benefit (Benefit Minus Harms)

Quality of Overall Evidence Substa

ntial Moderate Small Zero/

Negative Good A B C D Fair B B C D Poor I – Insufficient Evidence

OREGON EPC

Strength of recommendations

Estimate of Net Benefit (Benefit Minus Harms)

Quality of Overall Evidence Substa

ntial Moderate Small Zero/

Negative Good A B C D Fair B B C D Poor I – Insufficient Evidence

OREGON EPC

What is evidence-based medicine?

“Evidence-based medicine is the integration of best research evidence with clinical expertise and patient values.”

David Sackett

OREGON EPC

What is evidence-based medicine?

• Where there is evidence of benefit and value, do it

• Where there is evidence of no benefit, harm, or poor value, don’t do it.

• When there is insufficient evidence to know for sure, be conservative

David Eddy

OREGON EPC

OREGON EPC

Evidence-based Practice Centers

• Created in 1997; now 13 centers• Produce

– “evidence reports” – systematic reviews– technology assessments– “rapid reviews”– meta-analyses and cost analyses– analysis of large databases

• Work with public and private sector partners

OREGON EPC

Oregon Evidence-based Practice Center

• USPSTF• Drug class reviews for states• Food claims for FDA• Various other topics

– HBOT for cerebral palsy– Rehabilitation for traumatic brain injury– Treating actinic keratoses– Telemedicine– VBAC– Osteoporosis diagnosis and treatment– Preventing youth violence

OREGON EPC

Oregon Evidence-based Practice Center

EVIDENCE REPORTS FOR DRUG CLASSES:

http://www.ohsu.edu/drugeffectiveness/reports/

USPSTF RECOMMENDATIONS:

http://www.ahrq.gov/clinic/uspstfix.htm

OREGON EPC

Criticism 3. EBM hurts minorities and vulnerable

populations-- “each drug is unique”-- “each patient is unique”-- “doctors should be able to choose

any drug for any patient”

OREGON EPC

Other study designs could be helpful, after the following questions are answered:

• Will our users find them credible enough to use them?

• Can it be identified, introduced into the review in a systematic way?

• Can we tell a good outcomes study from a poor one?

• Can we tell a good economic study from a poor one?

• Can users incorporate it into decisions in a meaningful way?

OREGON EPC

Most common problems with observational studies of adverse

events• Incomplete ascertainment• Few data on severity of the event• Don’t report on efficacy (to

examine trade-offs)• Confounding, bias

OREGON EPC

Level 1: “Would you have this done for yourself or for someone else in your immediate family?”

Influenced by one’s personal experience with the disease and capacity to deal with risk.

Affects few people.Level II: “What would I recommend to my

patient/client?”Physician making a recommendation for his/her

patients. Influenced by prior experience, but the scientific evidence may play a greater role.

Affects possibly hundreds of people.Level III: “What would I recommend to the nation, the

world?”Across-the-board recommendations for a

population. Must be based on rigorous assessment of the

scientific evidence.Affects hundreds of thousands, even millions of

people.

OREGON EPC

1998—First FDA application 2001—FDA approval for schizophrenia2004—Approval in acute maniaAugust, 2004—Warning hyperglycemia and diabetes April, 2005—Warning on “off-label” use in elderly (olanzapine), Abilify (aripiprazole), Risperdal (risperidone), and Seroquel (quetiapine).

June, 2005—Lilly settles Zyprexa suits