ahcj 2012 atlanta conf. talk
DESCRIPTION
These are my slides from my workshop at the Association of Health Care Journalists national conference in Atlanta on April 19, 2012.TRANSCRIPT
![Page 1: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/1.jpg)
What are your criteria in reporting on health care research?
Gary Schwitzer, Publisher, HealthNewsReview.org
![Page 2: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/2.jpg)
Background on HealthNewsReview.org, what we do, how we do it, why, and what we’ve found
Recurring pitfalls we’ve seen in 6 years of daily monitoring
An editor’s perspective on addressing these issues
We’ll review a story – together – using HealthNewsReview.org criteria.
Plenty of time for Q & A – tapping Ivan’s editor expertise
Handouts: Summary of our 6-year, 1,700 story experience Resources for reporting on studies Story to review & scoresheet
Our plan for next 2 hours:
![Page 3: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/3.jpg)
![Page 4: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/4.jpg)
We review stories that include claims about…
Medical treatments
Tests
Products
Procedures
![Page 5: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/5.jpg)
![Page 6: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/6.jpg)
Site stats
28 reviewers: 19 clinicians or researchers, 7 journalists, 2 breast cancer survivors
Site launched April 17, 2006 – 6th anniversary week!
>1,700 stories reviewed
Blog launched in 2004 - >2,500 posts
Sole funding from the Informed Medical Decisions Foundation, founded by Jack Wennberg et al
![Page 7: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/7.jpg)
![Page 8: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/8.jpg)
= HealthNewsReview.org (U.S.), Media Doctor sites in Australia, Canada, Germany, Hong Kong, Japan plus somewhat similar sites in UK and Austria
= talks/workshops in Beijing, UK, Spain, Brazil, Argentina, Mexico, Puerto Rico only in two years’ time
Global Reach for such efforts
![Page 9: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/9.jpg)
Our criteria: Does the story explain…
What’s the total cost?
How often do benefits occur?
How often do harms occur?
How strong is the evidence?
Is the condition exaggerated?
Is this really a new approach?
Is it available?
Are there alternative choices?
Who’s promoting this?
Do they have a financial conflict of interest?
![Page 10: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/10.jpg)
After 6 years and 1,700 stories
70% fail to adequately discuss costs.
66% fail to quantify benefit – often exaggerating potential benefit
65% fail to quantify harm - often minimizing potential harm
62% fail to evaluate the quality of the evidence
57% fail to compare new idea with existing options
![Page 11: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/11.jpg)
Kid-in-candy-store picture of U.S. health care
Everything is terrificNothing is riskyNo price tags
![Page 12: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/12.jpg)
Recurring themes seen after 6 years of daily
monitoringFailure to explain limitations of
observational studies. Stories that conflate association and causation.
Framing numbers to exaggerate benefit, minimize harms
Stories about screening tests that emphasize only benefits, minimizing or ignoring potential harms
![Page 13: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/13.jpg)
![Page 14: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/14.jpg)
![Page 15: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/15.jpg)
Be skeptical about observational studies
Remind readers that the findings may be more about the people being studied than the "exposure".
They can point to a strong statistical association, but they cannot prove cause and effect.
![Page 16: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/16.jpg)
White rice increases risk of type 2 diabetes
Thursday, March 15, 2012
The risk of type 2 diabetes is significantly increased if white rice is eaten regularly, claims a study published today on bmj.com.
More than 400 words in the news release but NONE about the limitations of such an observational study.
![Page 17: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/17.jpg)
Does your language fit the evidence of observational studies?
![Page 18: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/18.jpg)
Absolute versus relative risk
![Page 19: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/19.jpg)
Following slides courtesy Drs. Steve Woloshin & Lisa
SchwartzDartmouth Medical School
White River Junction, Vermont VA
Part of their syllabus at: NIH Medicine in the MediaMIT Medical Evidence Boot
Camp
![Page 20: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/20.jpg)
Nolvadex (tamoxifen)Reducing breast cancer risk by 50
percent.
![Page 21: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/21.jpg)
For the first time, there is a clinically proven way for many women at high risk of developing breast cancer to significantly reduce that risk.
The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.
![Page 22: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/22.jpg)
48% of what ???
Women who took Nolvadex had 48% fewer breast cancers….
![Page 23: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/23.jpg)
48
Extremely Fancy Store
%OFF
On selected items!
It's like a coupon…
What if selected items were..
TV’s, washing machines?
Things like a pack of gum?
save $100s
save pennies
“48% of what” matters!
Know the REGULAR price!
![Page 24: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/24.jpg)
What is the
48% off coupon really worth?
How much do you save?
"Breast cancer risk"
NOLVADEX???
Placebo
???
Chance of getting breast cancer
![Page 25: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/25.jpg)
The chance of getting breast cancer over 6 years
in the PLACEBO group
3.3%
The REGULAR price!
PLACEBO
The "base rate"
Absolute risk in the placebo group
Event rate in the placebo group
![Page 26: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/26.jpg)
NOLVADEX
The chance of getting breast cancer over 6 years
in the INTERVENTION group
3.3%
The SALES price!The REGULAR price!
PLACEBO
1.7%
![Page 27: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/27.jpg)
Chance of getting breast cancer (over 6 years) with NOLVADEX was 1.6 % points lower than with placebo
NOLVADEX
3.3%
PLACEBO
1.7%
What is the effect of Nolvadex?How good is the sale?
Savings = Regular price – Sales price
How much do you save?
If 100 women took NOLVADEX instead of placebo for 6 years, there would be about 2 fewer cases of breast
cancer.
The SALES price!The REGULAR price!
= 1.6%
3.3% 1.7% Absolute risk reduction
![Page 28: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/28.jpg)
NOLVADEX
3.3%
PLACEBO
1.7%
What is the effect of Nolvadex?How good is the sale?
Relative Risk =Chance of outcome (intervention)
Chance of outcome (control)
![Page 29: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/29.jpg)
NOLVADEX
3.3%
PLACEBO
1.7%
Relative Risk = = 0.52 ???
???
1.7%
3.3%
Describing the effect of NOLVADEX
"% Lower" format =1 - RR =1 - .52 =.48
So finally....this is how you get to
the 48% off sale! It's the relative risk reduction
At 6 years, the chance of breast cancer for women taking NOLVADEX was 48 % lower than that of women taking placebo.
![Page 30: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/30.jpg)
48
Extremely Fancy Store
%
OFF
On selected items!
1.6%
Extremely Fancy Store
SAVINGS
On selected items!How you say it matters!"Framing"
Two ways of saying the same thing:
the benefit of NOLVADEXOne feels
bigOne feels
small
![Page 31: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/31.jpg)
??
Relative vs. absolute risk reductions
Relative Absolute
30% 10%
Chance of death at 1 year
Risk reduction
67%
67%
67%
3% 1%
0.003% 0.001%
??20%
Placebo DRUG (1-[DRUG/Placebo]) (Placebo-DRUG)
2%
0.002%
"% off" "savings"
![Page 32: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/32.jpg)
The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.
”Strokes, cataracts more common with Nolvadex. Most women experience some level of hot flashes and vaginal discharge".
"Nolvadex isn't for every woman…In the study women taking Nolvadex were 2 to 3 times more likely to develop uterine cancer or blood clots in the lung and legs, although each occurred in less than 1% of women".
BenefitHarm
No numbers
![Page 33: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/33.jpg)
Present benefits and harms the same way
![Page 34: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/34.jpg)
210% more uterine cancer and potentially life threatening blood clots in the lung and legs.
210% more
The proof? In a landmark study…women who took Nolvadex had 48% fewer breast cancers than women taking sugar pills.
![Page 35: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/35.jpg)
Harm
6-yr chance of venous thromboembolic event:
Like inflation…doubled the price!
NOLVADEX
0.5%
PLACEBO
1.0%
![Page 36: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/36.jpg)
Over the next 6 years, what happened when women…
Benefits: Nolvadex lowered chance
Getting breast cancer
Harms: Nolvadex increased chance
Having a serious blood clot
3.3% 1.7%
0.5% 1.0%
Net effect of Nolvadex for every 1000 women: 16 fewer women get breast cancer 5 more women get serious blood clots 6 more get uterine cancer
Getting uterine cancer 0.5% 1.1%
NOLVADEX PLACEBO
![Page 37: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/37.jpg)
![Page 38: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/38.jpg)
Take home messagesExtrapolate with caution! Don't tell people what to worry about – or do –
based on very preliminary animal / lab science.
Recognize pseudo-evidence Publication in a medical journal - even "the New England Journal of Medicine" - does not guarantee the findings are true (or even important).
Be wary of inherently weak science Without a comparison group, it is impossible to be sure if the drug was responsible for the findings.
Pay attention to the outcome Surrogate outcomes (like tumor shrinkage) do not reliably translate into clinically meaningful outcomes (longer life).
Be skeptical about observational studies Remind readers that the findings may be more about the people being studied than the "exposure".
Avoid exaggerated numbers Use absolute risks for both benefits and harms.
![Page 39: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/39.jpg)
Pitfalls of a steady diet of journal
stories
PLoS Med 2005; 2(8): e124
![Page 40: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/40.jpg)
The problem begins with the public’s rising expectations of science. Being human, scientists are tempted to show that they know more than they do. The number of investigators—and the number of experiments, observations and analyses they produce—has also increased exponentially in many fields, but adequate safeguards against bias are lacking. Research is fragmented, competition is fierce and emphasis is often given to single studies instead of the big picture. Much research is conducted for reasons other than the pursuit of truth. Conflicts of interest abound, and they influence outcomes.
![Page 41: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/41.jpg)
![Page 42: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/42.jpg)
Spinning results of randomized clinical
trialsBoultron et al, JAMA May 26, 2010, 303 (20): 2058
In this representative sample of RCTs published in 2006 with statistically non-significant primary outcomes, the reporting and interpretation of findings was frequently inconsistent with the results.
![Page 43: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/43.jpg)
Reporting bias in medical research - a narrative
review
McGauran et al. Trials 2010, 11:37
We identified reporting bias in 40 indications comprising around 50 different pharmacological, diagnostic, and preventive interventions. Many cases involved the withholding of study data by manufacturers and regulatory agencies or the active attempt by manufacturers to suppress publication. The ascertained effects of reporting bias included the overestimation of efficacy and the underestimation of safety risks of interventions.
![Page 44: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/44.jpg)
• What Ivan’s project has shown about retractions, research fraud, fabrication, falsification of data
• Unpublished data (BMJ recently published 8 articles in one edition on “the extent, causes and consequences of unpublished evidence”)
• Conflicts of interest – guideline setting
BE AWARE OF…..
![Page 45: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/45.jpg)
• Commercialization of research: contract research organizations, commercial IRBs or institutional review boards, medical education and communication companies (Carl Elliott)
• Ghostwriting of journal articles (The Public Library of Science hosts a “Ghostwriting Collection” on its website.)
• The focus on surrogate markers in many studies may be hurting patient care. (primer in HealthNewsReview.org online toolkit)
![Page 46: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/46.jpg)
News coverage & poor public discussion of screening tests is one of the most concerning public policy issues.
Worst, most biased coverage I’ve seen in 37 years
![Page 47: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/47.jpg)
“The decision to start regular, biennial screening mammography before the age of 50 years should be an individual one and take patient context into account, including the patient's values regarding specific benefits and harms.”
What the Task Force actually wrote:
![Page 48: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/48.jpg)
As 37-year ChiTrib & NYT vet John Crewdson wrote in The Atlantic…
“There are multiple reasons women are ill-informed about breast cancer. The fault lies primarily with their physicians, the cancer establishment, and the news media--especially the news media. Until coverage of breast cancer rises above the level of scary warnings mixed with heartwarming stories of cancer survivors, women are likely to go on being perplexed."
![Page 49: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/49.jpg)
Other examples:
Chicago Sun-Times
Wall Street Journal
Washington Post
New York Daily News
Minneapolis Star Tribune
All TV networks
![Page 50: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/50.jpg)
The ink isn’t even
dry on studies before
marketin
g begins
![Page 51: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/51.jpg)
HealthDay wire service didn’t challenge researcher promoting universal pancreatic cancer screening for everyone over 50.
After study of tissue from 7 people!
![Page 52: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/52.jpg)
We could have given the finger to this story
"Relative finger length could be used as a simple test for prostate cancer risk, particularly in men aged under 60," said one of the researchers.
![Page 53: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/53.jpg)
Screening Madness
Crusading
one-sided
advocacy
![Page 54: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/54.jpg)
Why don’t we deliver this
message?
“All screening tests cause harm;
some may do good.”
But much health journalism consistently emphasizes benefits & minimizes harms
![Page 55: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/55.jpg)
A form of disease-mongering
Selling sickness
Selling the search for weapons of mass destruction inside everyone
Dr. Gil Welch in NYT – If You Feel OK, Maybe You Are OK
“Screening the apparently healthy potentially saves a few lives. But it definitely drags many others into the system needlessly — into needless appointments, needless tests, needless drugs and needless operations. This process doesn’t promote health; it promotes disease. People suffer from more anxiety about their health, from drug side effects, from complications of surgery. A few die. And remember: these people felt fine when they entered the health care system.”
Communication of the evidence is a key health policy issue
![Page 56: AHCJ 2012 Atlanta conf. talk](https://reader035.vdocuments.net/reader035/viewer/2022070314/554ca7e8b4c905c10d8b586b/html5/thumbnails/56.jpg)
“I honestly believe it is better to know nothing than to know what ain’t so.”
Josh Billings (pen name of humorist Henry Wheeler Shaw, 1818 – 1885)