advanced topics in meta-analysis
Post on 23-Feb-2016
66 Views
Preview:
DESCRIPTION
TRANSCRIPT
1
Wim Van den Noortgate Katholieke Universiteit Leuven, Belgium
Belgian Campbell GroupWim.VandenNoortgate@kuleuven-kortrijk.be
Workshop systematic reviews Leuven June 4-6, 2012
Advanced topics in meta-analysis
2
1. Modelling heterogeneity2. Publication bias
Content
3
1. Modelling heterogeneity
4
Growing popularity of evidence-based thinking:
Decisions in practice and policy should be based on scientific research about the effects of these decisions/interventions
But: conflicting results (failures to replicate), especially in social sciences!
5
1. The role of chance- in measuring variables- in sampling study participants
2. Study results may be systematically biased due to
- the way variables are measured- the way the study is set up
3. Studies differ from each other (e.g., in the kind of treatment, the duration of treatment, the dependent variable, the characteristics of the investigated population, …)
Explanation for failures to replicate?
6
Differences between observed effect sizes due to chance only
Population effect sizes all equal
Fixed effects model
1 2( ... )k
7
0 :H
Assessing heterogeneity
2
1
ˆ( )k
j jj
Q w g
2( 1)k
1 2 ... k :aH at least one differs from an otherj
0 : ~H Q
8
Measuring heterogeneity( )² *100%Q dfIQ
Rough guidelines:0% to 40%: might not be important 30% to 60%: may represent moderate heterogeneity 50% to 90%: may represent substantial heterogeneity 75% to 100%: considerable heterogeneity
Interpretation based on both I² and heterogeneity test!
= percentage of variability in effect estimates due to heterogeneity rather than chance
9
An example in education(Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.)
StudyWeeks prior
contact gj
1.2.3.4.5.6.7.8.9.
10.11.12.13.14.15.16.17.18.19.
Rosenthal et al. (1974)Conn et al. (1968)Jose & Cody (1971)Pellegrini & Hicks (1972)Pellegrini & Hicks (1972)Evans & Rosenthal (1969)Fielder et al. (1971)Claiborn (1969)Kester & Letchworth (1972)Maxwell (1970)Carter (1970)Flowers (1966)Keshock (1970)Henrickson (1970)Fine (1972)Greiger (1970)Rosenthal & Jacobson (1968)Fleming & Anttonen (1971)Ginsburg (1970)
2330033301001233123
0.030.12
-0.141.180.26
-0.06-0.02-0.320.270.800.540.18
-0.020.23
-0.18-0.060.300.07
-0.07
0.130.150.170.370.370.100.100.220.160.250.300.220.290.290.160.170.140.090.17
( )jg
10Q = 35,83, df = 18, I²= 50 %, p = .007
11
Not always wise: make set of studies more homogeneous!
Can help to say something about ‘fruit’ Can help to make detailed conclusions:
Does the effect depend on the kind of fruit?
Mixing apples and oranges?
12
13
Population effect size possibly depends on study category
Differences between observed effect sizes within the same category due to chance only
Fixed effects model with categorical moderator variable
14
An example in education(Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.)
StudyWeeks prior
contact gj
1.2.3.4.5.6.7.8.9.
10.11.12.13.14.15.16.17.18.19.
Rosenthal et al. (1974)Conn et al. (1968)Jose & Cody (1971)Pellegrini & Hicks (1972)Pellegrini & Hicks (1972)Evans & Rosenthal (1969)Fielder et al. (1971)Claiborn (1969)Kester & Letchworth (1972)Maxwell (1970)Carter (1970)Flowers (1966)Keshock (1970)Henrickson (1970)Fine (1972)Greiger (1970)Rosenthal & Jacobson (1968)Fleming & Anttonen (1971)Ginsburg (1970)
2330033301001233123
0.030.12
-0.141.180.26
-0.06-0.02-0.320.270.800.540.18
-0.020.23
-0.18-0.060.300.07
-0.07
0.130.150.170.370.370.100.100.220.160.250.300.220.290.290.160.170.140.090.17
( )jg
15
16
Total variabilityin observed ES’s
A weighted ANOVA
T B WQ Q Q
Variability within groups
Variabilitybetween groups= +
H0: QT ~²k-1
H0: QB ~²J-1
H0: QW ~²k-J
2
1
ˆ( )k
j jj
Q w g
QT : homogeneity test
QB : moderator test
QW : test for within group homogeneity
17
Q total = Q Between + Q within
² 35.83 20.38 15.45
df 18 3 15
p 0.007 0.0001 0.42
18
A second example(using a sorted caterpillar plot)
-1.5
-0.5
0.5
1.5
2.5
3.5
4.5
5.5
6.5
Observed effect sizes for the 3 tasks
Semantic categorization Lexical decision Naming
ES
= Mean ES REM
19
Population effect size possibly depends on continuous study characteristic
e.g.,
After taking into account this study characteristic, differences between observed effect sizes due to chance only
Fixed effects model with continuous moderator variable
0 1 1 ... j j p pjx x
20
Initial effect is moderate (0.41, p < .001), but decreases with increasing prior contact (with -0.16 per week, p <.001)
Conclusions for Raudenbush (1984):
21
Population effect size possibly varies randomly over studies
Differences between observed effect sizes are due to- chance- ‘true’ differences
Random effects model
22
23
Population effect size possibly depends on study category
Differences between observed effect sizes within the same category are due to- chance- ‘true’ differences
Random effects model with categorical moderator variable
24
Population effect size possibly depends on continuous study characteristic
e.g.,
After taking into account this study characteristics, differences between observed effect sizes are due to- chance- ‘true’ differences
Random effects model with continuous moderator variable
0 1 1 ... j j p pj jx x u
25
Random effects model with moderators:
◦ The least restrictive model: allows moderator variables & random variation
◦ Also called a ‘Mixed effects model’
26
FEM REM
Without moderator
Categorical moderator
Continuous moderator
27
1. Is there an overall effect?2. How large is this effect?3. Is the effect the same in all studies?4. How large is the variation over studies?5. Is this variation related to study
characteristics?6. Is there variation that remains unexplained?7. What is the effect in the specific studies?
Basic meta-analytic questions
28
An example in education(Raudenbush, S. W. (1984). Magnitude of teacher expectancy effects on pupil IQ as a function of the credibility of expectancy induction: A synthesis of findings from 18 experiments. Journal of Educational Psychology, 76, 85-97.)
StudyWeeks prior
contact gj
1.2.3.4.5.6.7.8.9.
10.11.12.13.14.15.16.17.18.19.
Rosenthal et al. (1974)Conn et al. (1968)Jose & Cody (1971)Pellegrini & Hicks (1972)Pellegrini & Hicks (1972)Evans & Rosenthal (1969)Fielder et al. (1971)Claiborn (1969)Kester & Letchworth (1972)Maxwell (1970)Carter (1970)Flowers (1966)Keshock (1970)Henrickson (1970)Fine (1972)Greiger (1970)Rosenthal & Jacobson (1968)Fleming & Anttonen (1971)Ginsburg (1970)
2330033301001233123
0.030.12
-0.141.180.26
-0.06-0.02-0.320.270.800.540.18
-0.020.23
-0.18-0.060.300.07
-0.07
0.130.150.170.370.370.100.100.220.160.250.300.220.290.290.160.170.140.090.17
( )jg
29
Parameter REM
Fixed
Intercept 0.084 (0.052)
Between study variance 0.019 (0.023)
0
2u
30
Parameter REM MEM
Fixed
Intercept 0.084 (0.052) 0.41 (0.087)
Weeks -0.16 (0.036)
Between study variance 0.019 (0.023) 0.00 (-)
0
1
2u
31
1. Models can include multiple moderators2. REM assumes randomly sampled studies3. REM requires enough studies4. Association (over studies) ≠ causation!
Be aware of potential confounding moderators (studies are not ‘RCT participants’!)
Remarks
Dependencies between studies◦ E.g., research group, country, …
Multiple effect sizes per study◦ Several samples◦ Same sample but, e.g., several indicator variables
32
Note: Sources of dependencies
Ignoring dependence? NO! Avoiding dependence
◦ (Randomly choosing one ES for each study)◦ Averaging ES’s within a study◦ Performing separate meta-analyses for each kind
of treatment or indicator Modelling dependence
◦ Performing a multivariate meta-analysis, accounting for sampling covariance.
◦ Performing a three level analysis
33
How to account for dependence?
34
2. Publication bias
35
Example 1: advanced ovarian cancer: monotherapy alkylating agent vs. combination chemotherapy? International Cancer Research Data Bank
(Egger, M. D., & Smith, G. (1998). Meta-analysis. Bias in location and selection of studies. British Medical Journal, 316, 61-66. http://www.bmj.com/cgi/content/full/316/7124/61).
36
Proportion of publication within 5 years after conference:
81 % (of 233 trials) for significant results 68 % (of 287 trials) for nonsignificant results
Example 2: 510 large trials presented at conferences of the American Society of Clinical Oncology (ASCO)
(Kryzanowska, M. K., Pintilie, M., & Tennock, I. F. (2003). Factors associated with failure to publish large randomized trials presented at an oncology meeting. Journal of the American Medical Association, 290, 495-501).
37
-1 -0.5 0 0.5 1 1.50
100
200
300
400
500
Observed effect sizes
Sam
ple
size
The funnel plot
38
-1 -0.5 0 0.5 1 1.50
100
200
300
400
500
Observed effect sizes
Sam
ple
size
Example: Raudenbush, 1984
39
Þ Thorough search for all relevant published and unpublished study results
a) Articlesb) Booksc) Conference papersd) Dissertationse) (Un)finished research reportsf) …
40
- outliers- detection using graphs (or tests)- conduct analysis with and without outliers
- calculation effect sizes : several analyses- publication bias: analysis with and without
unpublished results- design & quality: compare results from studies
with strong design or good quality, with those of all studies
- researcher: literature search, effect size calculation, coding quality, …, done by two researchers
- …
Note: sensitivity analysis: how robust are conclusions?
41
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99100101102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131
-1.5
-0.5
0.5
1.5
2.5
3.5
4.5
5.5
6.5
Observed effect sizes
Experiment
ES
Outliers
42
Spreadsheets (e.g., MS Excel, …) Some general statistical software (note:
often not possible to fix the sampling variance)SAS Proc Mixed, Splus, R Metafor package, …
Software for meta-analysis (note: often not MEM; often only one moderator!)CMA (http://www.meta-analysis.com/), RevMan, …
Software for multilevel/mixed modelsHLM, MLwiN, …
Software for MA
43
Software Excel SAS R CMA RevManCalculation of effect sizes X X √ √√ XNumber of moderators X ∞ ∞ 1 1 (cat.)Funnel X X √ √ √Trim & Fill X X √ √ XForest X X √ √ √Max. nr of levels 2 ∞ 2 2 2Flexibility √ √√ √√ X XPrice Expensive Free Expensive Free
(but student (but limitedversion) trial vers.)
Complexity √ X X √√ √√
44
Cooper, H., Hedges, L. V., & Valentine, J. C. (Eds.) (2009). The handbook of research synthesis and meta-analysis. New York: The Russell Sage Foundation.
Lipsey, M. W., & Wilson, D. B. (2001). Practical meta-
analysis. Thousand Oaks, CA: Sage.
Van den Noortgate, W., & Onghena, P. (2005). Meta-analysis. In B. S. Everittt, & D. C. Howell (Eds), Encyclopedia of Statistics in Behavioral Science (Vol. 3 pp. 1206-1217). Chichester, UK: John Wiley & Sons.
Recommended literature:
45
Site of David Wilsonhttp://mason.gmu.edu/~dwilsonb/ma.html
Site of William Shadishfaculty.ucmerced.edu/wshadish/
Recommended sites
top related