simon thornley meta-analysis: pooling study results
TRANSCRIPT
![Page 1: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/1.jpg)
Simon Thornley
Meta-analysis: pooling study results
![Page 2: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/2.jpg)
Objective• Understand the philosophy of meta-analysis and its
contribution to epidemiology and science.
• Understand the limitations of meta-analysis
![Page 3: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/3.jpg)
Introduction• Systematic quantitative integration of results several
independent studies
• Distinct from a narrative review “expert”
• Synthesis of published information.
• Usually considered only appropriate for RCTs
• Still controversial even in this context.
• Google search on “meta-analysis” 8 million hits!
![Page 4: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/4.jpg)
Criticism • “statistical alchemy” for the 21st Century
• “The intellectual allure of making mathematical models and aggregating collections of studies has been used as an escape from the more fundamental scientific challenges”
• -Feinstein.
![Page 5: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/5.jpg)
Purposes of meta analysis• Inefficiency of traditional narrative reviews.
• Allow researchers to keep abreast of accumulating evidence
• Resolution of uncertainty when research disagrees?
• Increase statistical power, enhances precision of effect estimates – especially small effects
• Allows exploratory analysis (subgroups)
![Page 6: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/6.jpg)
Inadequate sample size? (Deal with type-2 error)• Single trials too small to detect moderate effects
• (low power – high chance of Type-2 error (false-negative))
• Investigators often over enthusiastic about size of treatment effects and sample size
• Meta-analysis doesn’t deal with other threats to study validity (bias, measurement error; in fact, may increase)
• e.g. CVD death vs. total mortality
![Page 7: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/7.jpg)
Accept H0 Reject H0
Statistical Test result
H0
True
False
OK
OK
Type-1 error
Type-2 error
Prob of a type 1 error = alpha a (usually fixed, say 0.05)Prob of a type 2 error = beta b= 1-power
![Page 8: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/8.jpg)
Random error lecture
Average odds ratio is 21?? Consistency??
![Page 9: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/9.jpg)
Which studies?• Need defined question, state MESH terms• Reproducible• Exhaustive search• Unpublished and published studies• Variety of databases.
![Page 10: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/10.jpg)
• Difference in means,
• Standardized differences in means
• Survival measures
• Relative risk • Odds ratio• Risk difference• NNT [=1/RD]• Incidence rate ratios
(person time data)
Typical summary outcome measuresBinary: Continuous:
![Page 11: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/11.jpg)
• Assume distribution of true effects
• Aim is to measure mean of distribution of true effects
• Greater heterogeneity --> greater variation
• Gives greater weight to small studies than fixed effect method of analysis.
• More conservative (wider confidence interval around effect estimate, compared to fixed effect method)
• Mantel-Haenszel method
• treat each trial as a “stratum” take weighted average of effects.
• O-E (Peto) method
• Binary outcome (e.g. death)
• Oi =observed # deaths on treatment in trial i
• Ei=expected # deaths (assuming no treat effect)
• look at average of Oi - Ei over all trials
• Assumes underlying true effect for each study and differences only due to random error
Methods of analysisFixed effect Random effect
![Page 12: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/12.jpg)
Dietary fat and cholesterol
![Page 13: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/13.jpg)
Reduced or modified dietary fat and all-cause mortality
![Page 14: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/14.jpg)
Publication bias
![Page 15: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/15.jpg)
When meta-analysis goes bad…• In CVD drug research, CVD outcomes
often favoured over total mortality• Which would you prefer????
![Page 16: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/16.jpg)
Publication bias: other methods • Ioannidis JPA, Trikalinos TA. An exploratory test for an
excess of significant findings. Clin. Trials 2007;4(3):245-53.
• Calculate expected number of positive studies, given:
• Sample size of individual studies
• Number of events in controls
• Summary effect (assumed true)
![Page 17: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/17.jpg)
Statin meta-analysis
![Page 18: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/18.jpg)
Problems• Combining heterogeneous studies (apples and oranges)
• Combining good and bad studies (good and bad apples) (study quality)
• Publication bias (tasty apples only)
• The "Flat Earth" criticism – reductionism –(Braeburns only)
• Combining data (individual v summary data stewed apples have different character to raw)
• Application to randomized studies only?
• Type-2 error only one problem with epi studies
![Page 19: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/19.jpg)
Meta analysis in observational studies• MA often applied in observational studies
• As often as RCTs (Egger et al)
• …. with controversy ….• Confounding and bias unlikely to “cancel out”
• Publication bias and “research initiation bias” (i.e. studies only done when there is an association)
• Different ways of reporting/analysing result (e.g different outcome measures, confounders, models, exposure levels)
![Page 20: Simon Thornley Meta-analysis: pooling study results](https://reader035.vdocuments.net/reader035/viewer/2022062322/56649ebd5503460f94bc6952/html5/thumbnails/20.jpg)
Summary• Meta-analyses increasingly used
• Logical only for RCTs?
• Summarise medical literature
• Reduce type-2 error by increasing sample size.
• Don’t deal with other types of epidemiological error (confounding/measurement error)
• Prone to unique type of error (Publication bias)
• Can be difficult to detect