can addiction research be trusted?
TRANSCRIPT
Can Addiction Research Can Addiction Research Be Trusted?Be Trusted?
Stewart B. Leavitt, PhD, EditorAddiction Treatment Forum
Sponsored by an educational grant from Mallinckrodt Inc.
Introducing EBAM*(*Evidence Based Addiction Medicine)
2
EBAM Guides AvailableEBAM Guides Available “Can Addiction Research Be Trusted?” – report Available online at www.atforum.com
Look under Addiction Resources tabSlides from this presentation also available
Called “EBAM AATOD Slides” PowerPoint format
3
““EBAM for Practitioners”EBAM for Practitioners”Available at…
www.atforum.com under “Addiction Resources” tab.
A more technical explanation for persons who need to critically assess research evidence.
4
Can Addiction ResearchCan Addiction ResearchBe Trusted?Be Trusted?
Yes? No? Sometimes?Yes? No? Sometimes? Research must be critically examined
and interpreted for particular clinical needs and patient benefits.
Just because something appears in print does not mean it is worthwhile or valid.
5
Why Should You Care?Why Should You Care?
Research directly or indirectly affects the practice of addiction medicine on a daily basis.
There is an urgent need for good science to overcome stigma, prejudice, and misunderstanding.
Trepanation once was a “scientific cure” for
mental illness.
6
What is EBAM?What is EBAM?Evidence-Based Addiction Medicine Evidence-Based Addiction Medicine
The use of current best evidence in making decisions about the care of patients.
Involves combining clinical expertise with valid research on a topic of concern.
Differentiates between practices based on sound evidence versus those founded more on 1) tradition, 2) outdated beliefs, or 3) “junk” science.
7
Why EBAM?Why EBAM?
EBAM approaches allow you to respond more convincingly to the following questions…. Where did you hear about that treatment? How do you know the information is worthwhile
and true? What do you propose doing and what results do
you expect? What are the costs versus benefits?
8
Essential UnderstandingsEssential Understandings
Medical research is an imperfect science.
Medical writing/reporting is often biased.
Scientific research does not “prove” anything to be true or not true.
9
The Nature of Clinical ResearchThe Nature of Clinical ResearchCompares experimental treatment (intervention)
with standard therapy or placebo.Research subjects represent only a tiny portion of
the population in question – statistical analyses estimate effects in the larger population.
All things are possible (some patients recover without treatment), and chance can play an important role in the results.
10
Interpreting ResearchInterpreting Research Interpretation is a matter of
perception. Not everyone “sees” a
research study in the same way.
Becoming a more critical consumer of research is essential.
11
Problems with Medical Problems with Medical Research Articles OverallResearch Articles Overall
Surveys of research have found startling results… On average, half or more of journal articles contain
errors – from missing data to major design flaws. Roughly, up to 2/3 of article abstracts may contain
data that is either inconsistent with or missing from the main body of the article.
In one investigation, 80% of systematic reviews & meta-analyses examined had serious flaws.
12
Concerns About Addiction Concerns About Addiction Research StudiesResearch Studies
Many articles appear to be written for other researchers, rather than practitioners.
There are often too few subjects and/or inadequate effects for accurately detecting significant group differences.
Subjects are often selected to maximize outcome results, but they are not typical patients.
Cases are presented as if they are evidence of causality or applicable to broader groups.
There may be a biased leap of interpretation from laboratory experiments to potential effects in humans.
13
What Really Matters?What Really Matters?
“Not everything that can be counted counts, and not everything that counts can be counted. What really mattersis not always obvious.” – Albert Einstein
Just because data were collected, analyzed, and reported does not mean they are relevant to the clinical question at hand – but it mattered to somebody.
14
What Are What Are Significant Significant Outcomes?Outcomes?
Statistical Significance – the likelihood (probability) that the results represent true differences between groups.
Clinical Significance – importance of the results for providing meaningful benefits to patients.
Statistical significance does not automatically transfer to clinical significance.
Taking A Closer LookTaking A Closer LookAtAt
Clinical Research ArticlesClinical Research Articles
16
Anatomy of a Research ArticleAnatomy of a Research Article
AbstractIntroduction
MethodsResults
DiscussionConclusion
17
AbstractAbstractPresents overview: who, what, where,
when, why, and how.Structured abstract is a mini-article…
Summarizes major sections of article. Limited to 250 words.
Abstract may be inaccurate & misleading.
18
IntroductionIntroductionStates purpose of study, including:
clinical background, review of previous research, rationale for current investigation.
Identifies questions (hypotheses) to be investigated in present study.
19
MethodsMethodsDefines the patients (subjects) included in
the study and the sample sizes.Describes in detail how the study was
conducted (treatment protocol).Defines outcomes of importance and how
they were measured.Describes how data was analyzed (statistics).
20
ResultsResultsDescribes in detail what was found:
differences in treatment effects (outcomes), their significance (p values), and, confidence limits.
Should show how statistical procedures were performed and any data adjustments (eg, for noncompliant/dropout subjects).
21
DiscussionDiscussion
Recaps results (the “payoff”).Interprets findings in comparison
with other research on the topic.Describes study limitations.Notes other factors that might have
influenced outcomes.
22
ConclusionConclusion
Presents case for “external validity” – how, when, where, and in which patients the results might be applied in clinical settings.
Often provides suggestions for further research on the subject.
23
Hierarchy of EvidenceHierarchy of EvidenceClinical Therapy ResearchClinical Therapy Research
Systematic Reviews / Meta-Analyses of RCTs.
Randomized Controlled Trials (RCTs).
Cohort Studies
Case-Control Studies
Cross-Sectional Surveys
Case Reports
“Perspectives” Articles
24
““Perspectives” ArticlesPerspectives” ArticlesAka* overviews, reports, commentary,
editorials, or interviews. *Also known as…
Most common in addiction field; often cited as evidence.
Summarize or comment on research, and are the most subject to bias.
Usually, favors one viewpoint (perspective) over another.
25
““The Emperor’s NewThe Emperor’s NewPerspective (Clothes)”Perspective (Clothes)”
“He thought it better to continue the illusion that anyone who couldn’t see his clothes was either stupid or incompetent.” – H.C. Anderson
Sometimes, the higher the “authority” of the source, the less its perspective can be trusted as being valid and unbiased.
26
Case ReportsCase Reports
Aka case histories, case series, or anecdotes.Draw upon personal observations or medical
records to report unexpected events.Subject to errors in observation or interpretation,
or inadequate documentation.May arrive at unsupported conclusions.“The plural of anecdote is not evidence.” – Leshner
27
Cross-Sectional SurveysCross-Sectional Surveys
Aka prevalence or epidemiology studies.Examines relationships of medical conditions,
treatments, and/or other factors in a patient population at a particular time.
Can establish associations but not causality. May be problems with recall and extraneous
factors (confounders).
28
Case-Control StudiesCase-Control StudiesAka case-referent, case-comparison, or
retrospective studies.Identifies patients with outcome of interest (Cases)
and Control patients without the outcome.Researchers look back in time to see how the two
groups differed (especially, regarding treatment or exposure received).
Subject to recall bias or inconsistent records.
29
A “Classic” Case-Control StudyA “Classic” Case-Control Study25 persons (Cases) exposed to
Tutankhamen’s tomb in 1923-26.Compared with 11 explorers in Egypt
at that time not exposed (Controls).
Results: No significant differences in mean age at death or in years survival after exposure.
Conclusion: Mummy’s curse was an urban legend.
30
Cohort StudiesCohort StudiesAka followup, incidence, longitudinal,
observational, or prospective studies.Most common trial in addiction medicine.An Experimental group (cohort) receives
treatment of interest; Control group does not.Outcome differences between groups are
compared and contrasted.Nonrandomized, many factors uncontrolled.
31
A First A First (?) (?) Clinical Cohort TrialClinical Cohort Trial
In 1774, a ship’s doctor took 12 sailors with scurvy and treated them daily, 2 at a time, with either……cider, elixir vitriol, vinegar, sea
water, nutmeg, or oranges/lemons.Compared with untreated patients.
Citrus fruit provided rapid relief.
32
Randomized Controlled TrialsRandomized Controlled Trials
RCTs considered “gold standard.”Subjects randomized to Experimental and Control
conditions; then followed for outcomes.Extraneous factors (confounders) are equally
distributed across groups.Subjects/investigators usually “blinded.”May be ethical concerns about Control condition.
33
Systematic ReviewsSystematic Reviews& Meta-Analyses& Meta-Analyses
Gather highest quality evidence on a subject.Compare and contrast studies to reach summary
statements and conclusions.Meta-analyses combine statistical evidence from
multiple studies for greater precision.Conclusions tend to be reliable and accurate.Clinical guidelines often come from these studies.
A Closer LookA Closer LookAtAt
Clinical Research ValidityClinical Research Validity
35
What is Validity?What is Validity?The degree to which the research represents the
most accurate and bias-free treatment effects.Internal Validity – refers to effects that are valid
for the subjects in the particular study. External Validity – refers to the relevance of a
study for typical addiction treatment settings.Validity is determined by looking at a study’s
design and sources of potential bias.
36
What is Bias?What is Bias?
Bias is anything that influences conclusions about the groups under investigation and potentially distorts comparisons.
The question is not if there is bias, but how much and whether it is sufficient to make the study invalid for a particular purpose.
37
Sources of Potential Bias - 1Sources of Potential Bias - 1Publication Bias – Trials with “positive”
results are more likely to be submitted and published more quickly.
Patient Selection – Who was excluded or included? Were groups equivalent?
Subject Management – Were groups treated exactly the same, except for the intervention under study?
38
Sources of Potential Bias - 2Sources of Potential Bias - 2
Randomization – Subjects should have an equal chance of assignment to each study condition.
Blinding (Masking) – Avoids preconceived ideas about treatment effects.
Placebo Effects – Mere expectation of benefit (or harm) may influence outcomes.
39
Sources of Potential Bias - 3Sources of Potential Bias - 3Confounding Factors – Extraneous factors,
known or unknown, may cause certain effects to occur or not to occur.
Compliance/Follow-up – Adherence to protocol and retention in study are critical.
Completeness of Data – Are all or only some subjects analyzed? (ITT vs Per Protocol)
40
Overcoming “Statistophobia”Overcoming “Statistophobia”
When most people hear the word statistics they want to hide.
It is not necessary to be a biostatistician to understand and assess addiction research.
41
Example:Example: How Large Is The How Large Is TheTreatment Benefit?Treatment Benefit?
RCT comparing 8 mg/d buprenorphine (BUP) vs 80 mg/d methadone (M80).
Average illicit-opioid+ urinalyses at 26 weeks.
M80 produced significant 17% reduction. (55% - 38% = 17%; p < .01) Ling et al. Arch Gen Psychiatry.
1996;53:401-407.
42
Example:Example: How Many Patients How Many Patients Need To Be Treated?Need To Be Treated?
Treatment effect = 17% reduction with M80.For every 100 patients treated with M80,
rather than BUP, 17 more would benefit.That is…
…for every 6 patients (100 ÷ 17) treated with M80, one additional patient would remain illicit-opioid abstinent than if treated with BUP.
43
What The Article What The Article Didn’t Didn’t SaySay
In the 8 mg/d BUP vs 80 mg/d Methadone trial – The patients selected were ideal; rather than typical. There was a 95% chance that the “true” treatment
effect was between 7% and 28% (vs 17% reported). Not enough subjects enrolled (n=150) to accurately
detect the relatively small treatment effect of 17%. So many subjects dropped out that the validity should
be questioned. (48% M80, 65% BUP gone at 26 wks)
Why DoesWhy DoesGood ResearchGood Research
Go Bad?Go Bad?
45
Researcher-WritersResearcher-WritersFace ChallengesFace Challenges
Insufficient funding – researchers may choose to do “something” when funds are not available to do it “right.”
Inadequate training in research & reporting. Lack of technical support – many
researchers do not have access to data processing and statistical support staff.
46
“Publish or perish” pressures – influence poor quality in design and reporting.
Political agendas – promote certain lines of research looking for “desired” results.
Journal space limitations – foster inadequate descriptions of analyses and results.
“Too many cooks” – multiple authors and peer reviewers complicate the process.
Researcher-WritersResearcher-WritersFace Challenges Face Challenges (Cont.)(Cont.)
47
Research Writing Run AmokResearch Writing Run Amok
When too many people are involved in a research article, it reminds us that…
“A camel is a horse that was designed by a committee.”
48
Dangers of “Data Dredging”Dangers of “Data Dredging”
Searching for “significant results” after the study is over and the findings were not as expected or desired (post hoc analysis).
Analyzing the same data in different ways until something “important” turns up.
Cranking-out several articles from a single small clinical study.
49
Laboratory vs Clinical ResearchLaboratory vs Clinical Research
Test tube & animal results do not necessarily apply to people.
“Bench work” can be invaluable for guiding clinical trials in humans.
But the ultimate test is in patients “at the bedside.”
©1980, David Sinclair, PhD - used by permission
50
Distortions in the Mass MediaDistortions in the Mass Media“Medical Propaganda”“Medical Propaganda”
Medical journalists usually lack understanding of research methods and shortcomings.
News editors seek sensational health stories to grab the public’s attention.
Research organizations and publications should be more cautious in their reporting of research to the press.
Readers need to become more critical.
PuttingPuttingAddiction ResearchAddiction Research
Into PracticeInto Practice
52
Judging Everyday RelevanceJudging Everyday Relevance
Is the article of high quality and valid?Does the study relate to your clinical needs?Are the participants similar to your own patients?Is the approach used in the study clearly
explained and free of bias?Are the results understandable and significant?Do the conclusions make sense and suggest clear
benefits for patient care?
53
Develop A Healthy SkepticismDevelop A Healthy Skepticism
Learn to trust your educated instincts.
If it sounds like a duck, and walks like a duck, but smells like a fish… …something is probably a bit “fishy.”
54
What Else Can You Do?What Else Can You Do?1. Only support/subscribe to publications
offering high-quality, well-documented research evidence.
2. Lobby NIDA & other funding organizations to provide clinically-relevant research of the best quality in design and reporting.
3. Challenge “fishy” reports in the media or from other sources – write letters.
55
““Bible” for Medical AuthorsBible” for Medical Authors
AMA style book is the “gold standard” for medical writing.
Depicts correct formats for research papers, plus a wealth of other info.
Differs from other styles used in behavioral and social sciences.
56
E-BM GuidebookE-BM GuidebookUsers’ Guides to the
Medical Literature (Textbook with CD-ROM)
Available at: www.usersguides.org www.amazon.com
Presents all aspects of evidence-based medicine.
57
““EBAM for Practitioners”EBAM for Practitioners”Available at…
www.atforum.com under “Addiction Resources” tab.
A more technical explanation for persons who need to critically assess research evidence.
58
Can Addiction Research Be Trusted?Can Addiction Research Be Trusted?
Usually!Usually!IF IF you learn what toyou learn what to
look for.look for.
“We have met the enemy and he is us!” - Pogo (Walt Kelly)
59
Sponsored by an educational grant from Mallinckrodt Inc.