introduction to meta-analysis - the university of … to meta...introduction to meta-analysis by...

CAMARADES: Bringing evidence to translational medicine

Introduction to Meta-Analysis by Michael Borenstein, Larry V. Hedges , Julian P. T Higgins, Hannah R.

Rothstein

Summary of

Chapter 1. How a Meta-Analysis Works Chapter 2. Why Perform a Meta-Analysis?

Zsanett Bahor


FOREST PLOT = used to illustrate the relative strength of treatment effects in multiple quantitative scientific studies addressing the same question

Chapter 1. How a Meta-Analysis Works


EFFECT SIZE = unit of currency in a meta-analysis

Magnitude of the treatment

effect Strength of any relationship

between two variables

Line of no effect: Risk

of death or MI same in

both groups

Risk lower in high-dose

group Risk lower in std-dose

group



PRECISION Effect size is bounded by a

confidence interval

Reflects precision with which effect size has been estimated

Narrower CI = greater precision



STUDY WEIGHTS Relationship between

precision & weight of study

Precision driven primarily by sample size

Good precision

Poor precision

More weight

Less weight

Studies with Are assigned



p-VALUES p-value is shown for a test of

the null

p-value will fall under 0.05 if 95% CI does not include null value



THE SUMMARY EFFECT Weighted mean of the

individual effects

Mechanism used to assign weights depends on assumptions about the distribution of effects sizes from which studies were sampled

Fixed-effect model

Assume that all studies in analysis share the same true effect size

Random-effects model Assume that the true effect size varies from study to study



HETEROGENEITY OF EFFECT SIZES If effect size consistent Focus on the summary effect Note this effect is robust across the

domain of studies included in analysis

If effect size varies modestly Might still report summary effect Note that true effect in any study

could be lower or higher than this value

If effect size varies substantially Focus on dispersion itself instead of

summary effect



• To perform a meta-analysis we compute an effect size and variance for each study, and then compute a weighted mean of these effect sizes.

• To compute the weighted mean we generally assign more weight to the more precise studies, but the rules for assigning weights depend on our assumptions about the distribution of true effects.

SUMMARY POINTS



GOAL OF A SYNTHESIS

Understand the results of any study in the context of all other studies.

1. Is effect size consistent across the body of data?

Yes No

2.

Estimate the effect size as accurately as possible

Report that it is robust across the studies

Quantify the extent of the variance

Consider the implications

• Narrative review = qualitative approach, summarizes the conclusions of others into a narrative about the effect of interest

• Meta-analysis = quantitative approach, ignores conclusions drawn by others, looks at evidence that has been collected. It is able to address above issues.

Chapter 2. Why Perform a Meta-Analysis?


STATISTICAL SIGNIFICANCE OF RESULTS

Narrative Review

Looks at p-values from individual

studies

No mechanism for synthesizing p-values from different studies

Must deal with them as discrete pieces of data

Meta-Analysis

Works with effect sizes from each

study

Allows us to combine the effects

Evaluate the statistical significance of the summary effect

Vs.

• Misleading to base conclusions on statistical significance from each study because p-value driven by effect AND size

• Studies might not be statistically significant because of small sample sizes and low statistical power.



CLINICAL IMPORTANCE OF THE EFFECT

Narrative Review

Point of departure is p-values

reported in studies

Review will focus on whether body of evidence allows us to reject the null hypothesis?

-> p-value can only tell us that the effect is not zero

No good mechanism for

discussing magnitude of effect

Meta-Analysis

Point of departure is estimate of

effect size for each study

Clinically relevant, because if need to make decision whether to employ a treatment or not -> want to know the magnitude of the effect treatment has

Vs.



CONSISTENCY OF EFFECT SIZES ACROSS STUDIES

• Implications different for a drug that consistently reduces the risk of death by 50% vs. drug that does this on average

Narrative Review

No good mechanism for assessing the

consistency of effects

Based on p-values, so can interpret non-significant result to mean there is no effect

Small p-value = large effect size/small effect size + large study

Large p-value = small effect size/large effect size + small study

Meta-Analysis

Work with effect sizes to determine

whether effect sizes consistent across studies

Apply methods based on statistical theory to allow that some observed dispersion is due to random sampling variation rather than differences in true effect size

Apply formulas to partition the variance into random error vs. real variance

Vs.



• Since the narrative review is based on discrete reports from a series of studies, it provides no real mechanism for synthesizing the data.

• By contrast, in a meta-analysis we introduce two fundamental changes. First, we work directly with the effect size from each study rather than the p-value. Second, we include all of the effects in a single statistical synthesis. This is critically important for the goal of computing (and testing) a summary effect. Meta-analysis also allows us to assess the dispersion of effects, and distinguish between real dispersion and spurious dispersion.

SUMMARY POINTS



Why do we do meta-analysis of animal studies?

• Preclinical studies are performed to inform human health

• Used in preclinical research to:

– assess the quality and range of evidence – identify gaps in the field – assess for publication bias – try to explain discrepancies between preclinical and clinical trial

results – inform clinical trial design – hypothesis-generating tool

• Used in clinical research to provide summary estimates to inform clinical practice

introduction to meta-analysis - the university of … to meta...introduction to meta-analysis by...

Documents