understanding comparative effectiveness
DESCRIPTION
How to understand and put into perspective a comparative effectiveness reportTRANSCRIPT
© Cerner Corporation All Rights Reserved 1
Robert W. Dubois, MD, PhDChief Medical OfficerCerner LifeSciences
Demystifying Comparative Effectiveness Research: A Case Study Learning Guide
© Cerner Corporation All Rights Reserved 2
Topics
Opening thoughtsFramework for reviewing a new CER report Case studies and lessons learned
Randomized trial-incomplete answerMeta-analysis-misleading conclusionObservational study-incorrect change in clinical practice
Final thoughts
© Cerner Corporation All Rights Reserved 3
Institute of Medicine Definition*
“…The purpose of CER is to assist consumers, clinicians, purchasers and policy makers to make informed decisions that will improve health care at both the individual and population levels.”
*IOM 2009 Initial National Priorities for Comparative Effectiveness Research
© Cerner Corporation All Rights Reserved
Not All Studies Are Created (or interpreted) Equally
Good study
Well informed
Not so good study
Not so well
informed
Good study that doesn’t answer your
question
Uninformed
© Cerner Corporation All Rights Reserved 6
Proceed With Caution…
© Cerner Corporation All Rights Reserved
When the reports arrive…
© Cerner Corporation All Rights Reserved
It may feel like this…
© Cerner Corporation All Rights Reserved
Categorizing is good…
© Cerner Corporation All Rights Reserved
Understanding is even better…
© Cerner Corporation All Rights Reserved 11
Topics
Opening thoughtsHow to approach CER
© Cerner Corporation All Rights Reserved
How to Approach CER
Step 1 Consider for
whom the findings are applicable
Step 2 Consider whether
aspects of the study design might affect the results
Step 3Consider
whether the findings may change with new research
© Cerner Corporation All Rights Reserved 13
Step 1: Consider for Whom the Findings are Applicable
An H1N1 study in school age kids may (or may not) help an internist with his/her adult population.A study of patients with severe rheumatoid arthritis may (or may not) apply to a population of milder patients.A study of diabetic control after one year may (or may not) apply to the longer term implications of a policy change.
Will the study results generalize to your environment?
© Cerner Corporation All Rights Reserved 14
Questions for Step 1
Does the study resemble your population of interest (e.g., age, gender, SES, disease profile)?Does the clinical setting resemble your own (e.g., primary care, inner city)?Is the study conducted in the “real world” or in a highly controlled environment?Are the outcomes appropriate to your needs?
© Cerner Corporation All Rights Reserved
How to Approach CER
Step 1 Consider for
whom the findings are applicable
Step 2 Consider whether
aspects of the study design
might affect the results
© Cerner Corporation All Rights Reserved 16
There are 3 Common Types of CER
Randomized trialsMeta-analysisObservational studies
Was the right design chosen?
© Cerner Corporation All Rights Reserved 17
StudyQuestion
Study Design
Study Results
© Cerner Corporation All Rights Reserved 18
Does the design match the study question?
RCT: may not meet your “real world” needs.Observational study: may have too many uncontrollable factors to be valid.Meta-analysis: just combining studies because they are “there” may not be appropriate.
© Cerner Corporation All Rights Reserved 19
Step 2: Consider Whether Aspects of the StudyDesign Might Affect the Results
Does the study design match the question being asked?Was it carried out with adequate rigor?
Each study design has its own issues to consider.One size does not fit all.
© Cerner Corporation All Rights Reserved
One size does not fit all…
© Cerner Corporation All Rights Reserved
One size does not fit all…
© Cerner Corporation All Rights Reserved
How to Approach CER
Step 1 Consider for
whom the findings are applicable
Step 2 Consider whether
aspects of the study design might affect the results
Step 3Consider
whether the findings may change with new research
© Cerner Corporation All Rights Reserved
“How sure are we… really”
http://www.hasyudeen.com/2008/02/network-centric-competition-in-flat.html
© Cerner Corporation All Rights Reserved 25
Topics
Opening thoughtsFramework for reviewing a new CER report (a.k.a. “Readers’ Guide”Case studies and lessons learned
Randomized trial-incomplete answer• Step 3-new information• Step 1-revisit the relevant population
© Cerner Corporation All Rights Reserved 26
Can it Work?
Randomized Controlled Trials (RCTs)Can an intervention work under certain controlled conditions?Characteristics:
Planned experimental frameworkDefined treatment optionsSpecified outcomesMay be single or double blinded
© Cerner Corporation All Rights Reserved
RCT Design
© Cerner Corporation All Rights Reserved 28
RCTs Have Strengths and Weaknesses
Strengths:Substantial internal validityReduced likelihood of bias or confoundingGold standard for clinical researchPathway to FDA approval
Weaknesses:May not generalize Modest size reduces ability to observe rare events.Placebo may not be the best “real world” comparator.Some patients leave the trial or “cross over” to the other therapy.
© Cerner Corporation All Rights Reserved 29
RCT Case Study: Cetuximab
Clinical SituationColorectal cancer is the 3rd most common cancer and cause of cancer death.Only an 11% survival if metastaticTreatment is not curative.Understanding of cancer genetics led to biomarker testing and targeted therapies.Cetuximab is an anti-epidermal growth factor receptor agent (EGFR).
© Cerner Corporation All Rights Reserved 30
Cetuximab RCT*: Was This the Full Story?
Cetuximab Comparison0
1
2
3
4
5
6
7
*Jonker N Engl J Med 2007
Survival in Months
572 patients with mCRCNon-blinded RCTCetuximab + supportive care vs. supportive care alone
© Cerner Corporation All Rights Reserved 31
Not the Full Cetuximab Story
Basic science research suggested that the KRAS gene and its mutations could influence efficacy.Subgroup analysis performed based upon testing tissue samples
Those with non-mutated gene had much greater benefit from cetuximab (5 months greater overall survival).
© Cerner Corporation All Rights Reserved 32
Not the Full Cetuximab Story
FDA narrowed the drug’s indications to patients with a non-mutated gene.Recommendations to test patients for the genetic marker
Therapy targeted to patients most likely to respond.Patients unlikely to respond avoid ill effects of the drug.
© Cerner Corporation All Rights Reserved 33
RCT Tips for the Consumer
RCTs assess whether an intervention can work in a controlled environment.Although RCTs are viewed as the “gold standard,” the initial impression of the cetuximab study was incomplete.
Multiple studies designed similarly would have likely yielded similar incomplete conclusions.A subgroup analysis based upon genetic markers yielded different results and conclusions.
No results are permanent.
© Cerner Corporation All Rights Reserved 34
Topics
Opening thoughtsFramework for reviewing a new CER report (a.k.a. “Readers’ Guide”Case studies and lessons learned
Randomized trialMeta-analysis-misleading conclusion
Step 2: methods used (and not used)
© Cerner Corporation All Rights Reserved 35
Meta-Analysis
“If one RCT is good, then more RCTs must be better…”
Quantitative and statistical combination of study resultsHighest level of evidenceUseful when different studies have different results
© Cerner Corporation All Rights Reserved
Bagshaw SM, Ghali WA. Acetylcysteine for prevention of contrast-induced nephropathy after intravascular angiography: a systematic review and meta-analysis. BMC Med. 2004;2(1):38.
Metanalysis Clarifies the Benefit
© Cerner Corporation All Rights Reserved
Wilcken, MJA. 2007;186 (7):368-370.
Metanalysis Clarifies the Benefits
© Cerner Corporation All Rights Reserved 38
Meta-Analysis
StrengthsIncreases the effective sample sizeProvides statistically stronger conclusionsDetects lower frequency events and more subtle distinctions
WeaknessesCreates an impression of “truth”Easy to do wrong…If care is not taken, results may be invalid.
© Cerner Corporation All Rights Reserved 39
Meta-Analysis Case Study: Avandia
Clinical SituationBlood sugar control reduces certain diabetic complications.But, oral drugs have been associated with an increased risk of heart disease (tolbutamide).TZDs seemed safer than sulfonylureas.Avandia approved by the FDA in 1999.
© Cerner Corporation All Rights Reserved 40
Original Article Effect of Rosiglitazone on the Risk of Myocardial Infarction and Death from Cardiovascular Causes
Steven E. Nissen, M.D., and Kathy Wolski, M.P.H.
N Engl J MedVolume 356(24):2457-2471
June 14, 2007
© Cerner Corporation All Rights Reserved 41
What was Done and Found?
A meta-analysis examined the impact of Avandia on cardiac events.Out of 116 studies, 42 met the authors’ inclusion criteria.
15,565 Avandia patients12,282 comparison patients
© Cerner Corporation All Rights Reserved
Avandia Adversely Impacts Heart Disease
Compared with the placebo, estrogen plus progestin resulted in:
Increased risk of heart attack
Increased risk of stroke
Increased risk of blood clots
Increased risk of breast cancer
Reduced risk of colorectal cancer
Fewer fractures
No protection against mild cognitive impairment and increased risk of dementia (study included only women 65 and older)
Nissen SE, Wolski K. N Engl J Med. 2007;356:2457-2471.
© Cerner Corporation All Rights Reserved
© Cerner Corporation All Rights Reserved 44
Was This the Right Answer?
“Thou shalt not combine heterogeneous studies…”Some studies had placebo and others active comparators.Some studies observed one arm longer than the other.Excluded “zero” event trials
© Cerner Corporation All Rights Reserved 45
Was This the Right Answer?
The choice of statistical methods impacts the results.
Re-analysis with an alternative approach showed no increased risk.
Inclusion of the “zero event” studies also eliminated any statistical differences.New trial results (RECORD) showed no difference in deaths but an increase in heart failure.
© Cerner Corporation All Rights Reserved 46
Meta-analysis Tips for the Consumer
Results are highly dependent on the studies included and excluded.Statistical methodology can impact study results.Nothing is permanent - emerging data may change the conclusions.
© Cerner Corporation All Rights Reserved 47
Topics
Opening thoughtsFramework for reviewing a new CER report (a.k.a. “Readers’ Guide”Case studies and lessons learned
Randomized trialMeta-analysisObservational study-incorrect change in clinical practice
Step 1: population differed in subtle ways from a more typical oneStep 2: intervention and comparison groups were different
© Cerner Corporation All Rights Reserved 48
Observational Studies
Answers the question: will it (likely) work (not can it work)?Examines the effects of treatment without formal randomizationPerformed prospectively or retrospectively
© Cerner Corporation All Rights Reserved 49
Observational Studies
Framingham Study identified heart disease risk factors.Commonly uses existing “real world” databases
Administrative or billing dataElectronic health records
© Cerner Corporation All Rights Reserved 50
Observational Studies
StrengthsAssess health care in the “real world”Lower cost and faster to performLarge sample sizes feasibleHypothesis generating
WeaknessesAdministrative databases have minimal clinical detail and may contain errors.Uncontrolled design leads to potential bias or confounding.Subject to “data dredging”Cannot prove cause and effect
© Cerner Corporation All Rights Reserved 51
Observational Case Study: Hormone Treatment
Clinical SituationHeart disease is the leading cause of death in women above age 50Estrogen falls after menopause
May account for the postmenopausal heat disease risk
© Cerner Corporation All Rights Reserved 52
What was Done and Found?
Nurses’ Health Study (NHS) collected data from 48,470 women aged 30-55.Focused on the impact of hormones on outcomesOngoing surveys assessed risk factors and health outcomes.
Estrogen use associated with lower heart disease risk
© Cerner Corporation All Rights Reserved
Implications of the Nurses’ Health Study
Estrogen use associated with
lower heart disease risk
© Cerner Corporation All Rights Reserved
Was This the Right Answer?
Estrogen use associated with lower
heart disease risk
Menopausal women
should take hormone
treatment
© Cerner Corporation All Rights Reserved
Was This the Right Answer?
Estrogen use associated with lower
heart disease risk
Menopausal women
should take hormone
treatment?
© Cerner Corporation All Rights Reserved 56
The clinical trials were designed to test the effects of postmenopausal hormone therapy, diet modification, and calcium and vitamin D
supplements on heart disease, fractures, and breast and colorectal cancer.
© Cerner Corporation All Rights Reserved 57
Women’s Health Initiative
RCT with 27,347 post-menopausal womenMean age was 63.6 years.
© Cerner Corporation All Rights Reserved 58
Women’s Health Initiative Muddied the Waters…
RCT with 27,347 post-menopausal womenMean age was 63.6 years.Study halted early.
Increased risk of clots, breast cancer and stroke.Estrogen-progestin users: increased risk of heart disease Estrogen only users: no CV benefits.
© Cerner Corporation All Rights Reserved
Was This the Right Answer?
Estrogen use associated with lower
heart disease risk
Menopausal women
should take hormone
treatment?
© Cerner Corporation All Rights Reserved 60
Why the Different Results?
Theory 1: WHI population was older than the NHS cohort (started hormones much later)Theory 2: NHS women who took hormones were healthier
Higher educationHigher SESLeanerLacked prior cardiac disease
Theory 3: The NHS women who took hormones differed from those who did not
© Cerner Corporation All Rights Reserved 61
The Differences Were Explainable (Partly)
1. NHS data re-analyzed…Less benefit for older women starting HRT later.
2. WHI re-analyzed…Some benefit for younger women starting HRT early.
Steps 1 and 2
With observational studies, make sure…
• The population resembles yours
• The intervention and comparison groups are similar
© Cerner Corporation All Rights Reserved 62
Observational Study Tips for the Consumer
Useful to understand “real world” benefits and harmsLook for confounding factors (Step 2)Nothing is permanent - emerging data may change the conclusions
© Cerner Corporation All Rights Reserved 63
Topics
Opening thoughtsFramework for reviewing a new CER report (a.k.a. “Readers’ Guide”Case studies and lessons learnedFinal thoughts
© Cerner Corporation All Rights Reserved 64
Final Thoughts
1. Different study types can offer different understandings.
Question
Study Design
RCTMetanalysis
Observational study
Results
© Cerner Corporation All Rights Reserved 65
Final Thoughts
1. Different study types can offer different understandings.
2. Results matter.Avandia sales fell after meta-analysis published.Hormone treatment plummeted after results of the WHI.Cetuximab use is growing.
© Cerner Corporation All Rights Reserved 66
Final Thoughts
1. Different study types can offer different understandings.
2. Results matter.3. Consider the potential impact upon policy making.
Influence guidelines?Impact reimbursement?Stable clinical area or advancing rapidly?
© Cerner Corporation All Rights Reserved 67
Demystifying Comparative Effectiveness Research
© Cerner Corporation All Rights Reserved 68
Demystifying Comparative Effectiveness Research
© Cerner Corporation All Rights Reserved 69
Demystifying Comparative Effectiveness Research: A Case Study Learning Guide
Reader’s Guide Check Lists
Robert W. Dubois, MD, PhDChief Medical OfficerCerner LifeSciences
© Cerner Corporation All Rights Reserved
How to Approach CER
Step 1 Consider for
whom the findings are applicable
Step 2 Consider whether
aspects of the study design might affect the results
Step 3Consider
whether the findings may change with new research
© Cerner Corporation All Rights Reserved 71
Use Check Lists With Caution
The lists are not all inclusive.A “yes” to all is not all that you need…All items are not created equally.
© Cerner Corporation All Rights Reserved
Step 1: Are the Findings Applicable?
Assessing External Validity1. Does the study population reflect your population of
interest? ___yes ___no
2. Does the type of clinical environment reflect your own? ___yes ___no
3. Is the study conducted in a “real-world” setting? ___yes ___no
4. Are the outcomes being measured appropriate to your needs? ___yes ___no
5. Is the evidence clinically significant, and the effect size large enough to be relevant? ___yes ___no
6. Does the duration of the study allow you to draw any long-term conclusions? ___yes ___no
7. Have appropriate subgroups been considered? ___yes ___no
© Cerner Corporation All Rights Reserved
Step 2: Design Affect the Results?
Randomized Controlled Trials
1. Is there a clear hypothesis? ___yes ___no
2. Were the inclusion criteria logical? ___yes ___no
3. Was the comparison group appropriate? ___yes ___no
4. Were study methods clearly described and valid? ___yes ___no
5. Are all relevant outcomes reported? ___yes ___no
6. Is there a strong exposure/outcome link? ___yes ___no
7. Are study limitations discussed? ___yes ___no
© Cerner Corporation All Rights Reserved
Step 2: Design Affect the Results?Meta-Analyses
1. Is there a clear hypothesis? ___yes ___no
2. Were the outcomes clearly defined and clinically relevant? ___yes ___no
3. Were the inclusion criteria logical? ___yes ___no
4. Were all relevant studies included? ___yes ___no
5. Do the authors assess the potential for publication bias? ___yes ___no
6. Was the search for relevant studies thorough? ___yes ___no
7. Were multiple searchable databases employed? ___yes ___no
8. Were the included studies of high methodologic quality? ___yes ___no
9. Did the authors test whether their studies were similar (homogeneous) or different (heterogeneous)? ___yes ___no
10. Were the combined studies sufficiently similar? ___yes ___no
11. Did the authors adequately describe and defend the statistical methods they used for analysis? ___yes ___no
© Cerner Corporation All Rights Reserved
Step 2: Design Affect the Results?
Observational Studies1. Does the population seem properly defined for the
stated outcomes of interest? ___yes ___no
2. Are the baseline characteristics similar for the groups being studied? ___yes ___no
3. Were study methods clearly described and valid? ___yes ___no
4. Are all relevant outcomes reported? ___yes ___no
5. Is there a strong exposure/outcome link? ___yes ___no
6. Did the authors adequately describe and defend the statistical methods they used for analysis? ___yes ___no
7. Have potentials for bias and confounding been discussed and accounted for? ___yes ___no
© Cerner Corporation All Rights Reserved
Step 3: Findings Change With New Research?
Innovation and Change*
1. Is this research in an area that is changing rapidly, (such as cancer research)? ___yes ___no
2. Are advances in scientific knowledge and understanding likely to influence how we view the effectiveness of this intervention?
___yes ___no
3. Will these results change clinical guidelines or impact policy? ___yes ___no
4. Is future work in this arena likely to take place from existing databases? ___yes ___no
5. Could subgroup analysis clarify these results? ___yes ___no
*In contrast to the prior tables, a “yes” indicates that the reader should view the study results with caution.