publication bias in meta-analysis selection method approaches

Publication Bias in Meta-Analysis:Selection Method Approaches

Michaela Paul

Biostatistics UnitInstitute for Social and Preventive Medicine

University of Zurich

PhD seminar, 26 May 2009

Introduction Selection models using weight functions Copas selection model Example Summary

Outline

1 Introduction

2 Selection models using weight functions

3 Sensitivity approach of Copas

4 Example: The effects of environmental tobacco smoke

5 Summary

2 / 26


Introduction

Effect size model

specifies what the distribution of the effect size would be if therewere no selection

Selection model

specifies the mechanism by which effect estimates are selected tobe observed

The selection model involves unknown parameters that govern theselection process:

estimate from observed effect size data (if possible)

assume specific values and carry out a sensitivity analysis

3 / 26


Effect size model

Any model can be used, e.g. the random effects model

θi = θi + σiεi , θi ∼ N(θ, τ2), εi ∼ N(0, 1)

with parameters

θi estimated effect size

θ overall (true) treatment effect

σ2i within-study sampling variance

τ2 between study variance

Usually, the within-study sampling variance is assumed to beknown as σ2

i ≈ s2i , where si is the estimated standard error (SE).

4 / 26


Selection models using weight functionsDenote

T ? random variable representing the effect estimateirrespective of selection

f (t|θ) density function of T ? (with true effect size θ)

w(t|ω) non-negative weight function depending onparameters ω

The weighted density of the observed effect size T is given by

g(t|θ, ω) =w(t|ω)f (t|θ)∫∞

−∞ w(t|ω)f (t|θ)dt

When w(t|ω) 6= const and a monotonic increasing function of t,the sampling distribution of T differs from that of T ?:

E[T ] = θ + bias︸︷︷︸>0

5 / 26


Suggested weight functions

Idea

Decisions about the conclusiveness of research results are oftenbased on statistical significance.

⇒ The chance of a study being included in the meta-analysisdepends only on the p-value, i.e. on the ratio θi/si

Examples: (see Sutton et al.; 2000)

simple dichotomised weight function with p-value p = 0.05as cutpoint

parametrically decreasing weight function with cutpointp = 0.05

step function with (psychologically motivated) prespecifiedcutpoints (Hedges; 1992; Hedges and Vevea; 2005)

. . .

6 / 26


Example: Hedges’ stepped weight functions

0.0

0.2

0.4

0.6

0.8

1.0

p−value

Pro

babi

lity

of o

bser

ving

effe

ct

0 0.1 0.25 0.35 0.5 0.65 0.75 0.9 1

weak 1−tailedweak 2−tailed

strong 1−tailedstrong 2−tailed

7 / 26


Hedges’ stepped weight functions

If selection is a function of the one-tailed p-value pi = Φ(− θi

si

),

then

w(·) =

ω1 if 0 < pi < a1 ⇔ −siΦ−1(a1) < θi ≤ ∞

ωj if aj−1 < pi < aj ⇔ −siΦ−1(aj) < θi ≤ −siΦ

−1(aj−1)

ωk if ak−1 < pi < 1 ⇔ −∞ < θi ≤ −siΦ−1(ak−1)

where Φ is the standard normal cdf.

Set ω1 = 1: Weights must be relative rather than absolutebecause number of studies before selection occurs is unknown.

Parameter estimation via ML (Newton-Raphson).

Test for publication bias based on likelihood ratio testH0 : ωi = 1.

8 / 26


How well do weight functions work?

Number and location of cutpoints

locations not completely arbitraryat least one observed p-value must be within each interval

Performance of estimation procedure

reasonable starting values requirednumerical problems if number of studies is smallstandard errors of estimates can be very large

Alternative:Specify several weight functions and carry out a sensitivity analysis.

9 / 26


Why go to the trouble of implementing somethingthat is complex and may not even work?

Simulation studies have shown (Hedges and Vevea; 2005):

Even when the selection model is poorly estimated, the associatedadjustment to the effect estimate can be quite accurate(provided the effect size model is well specified).

10 / 26


Copas selection model

Idea (Copas and Shi; 2000a, 2001)

Keep as close as possible to usual ML random effects model,but add two parameters which describe study selection.

1 effect size model

θi = θi + σiεi , θi ∼ N(θ, τ2), εi ∼ N(0, 1)

2 selection model

Zi = γ0 +γ1

si︸︷︷︸ui

+δi , δi ∼ N(0, 1), corr(εi , δi ) = ρ

where residuals (εi , δi ) are assumed to be jointly normal andindependent across studies

θi is observed only when latent variable Zi > 0.

11 / 26


Copas selection model

The observed treatment effects are modelled by the conditionaldistribution of θi in 1) given that Zi > 0.

E(θi |Zi > 0, si ) = θ + ρσiλ(ui ) where λ(·) = ϕ(·)Φ(·) is Mill’s ratio

Interpretation of parameters:

γ0, γ1(> 0) Inestimable parameters that control the marginalprobability that a study with within-study SE si ispublished.

ρ = 0 No publication bias: θi and Zi are independent.

ρ > 0 Selected studies will have Zi > 0⇒ δi , εi , θi are morelikely to be positive, overestimating the true mean θ.

12 / 26


Estimation

The log-likelihood of Copas selection model is given by

l(θ, ρ, τ, γ0, γ1) =∑

i

log(P(θi |Zi > 0, si ))

=∑

i

log

(P(Zi > 0|θi , si ) P(θi )

P(Zi > 0|si )

)= . . .

where P(Zi > 0|θi , si ) = Φ

ui + ρσiθi−θσ2

i +τ2√1− ρ2σ2

i /(σ2i + τ2)

The nuisance parameters σ2

i = Var(θi |θi ) are replaced by their

sample estimates based on s2i = Var(θi |Zi > 0)

Asymptotic inference about θ for given γ0, γ1

13 / 26


Goodness of fit test for residual publication bias

For a specific pair (γ0, γ1) for the selection model, we need tocheck that the resulting model gives a reasonable fit.

The expected values give a good fit to the data if theysatisfactorily predict any observed trend in the funnel plot.

This can be tested using an extended model for the treatmenteffect

θi = θi + σiεi + βsi

Likelihood ratio test H0 : β = 0.

14 / 26


Sensitivity analysis

Marginal selection probability of a typical study with SE s:

Ps = P(Z > 0|s, γ0, γ1) = Φ(γ0 +γ1

s)

> # Convert selection probabilities into values (gamma0, gamma1)> tran <- function(p, se){+ q1 <- qnorm(p[, 1]); q2 <- qnorm(p[, 2])+ gamma1 <- (q1 - q2)/(1/min(se) - 1/max(se))+ gamma0 <- q1 - gamma1/min(se)+ return(cbind(gamma0, gamma1))+ }> (ps <- matrix(c(.99,.8,.6,.4,.2,.8,.5,.3,.1,.01), ncol=2))

[,1] [,2]

[1,] 0.99 0.80

[2,] 0.80 0.50

[3,] 0.60 0.30

[4,] 0.40 0.10

[5,] 0.20 0.01

15 / 26


Sensitivity analysis

> hackshaw <- read.table("../data/data2_hackshaw.txt",header=T)> gamma01 <- tran(ps, se = hackshaw$se)

0.1 0.2 0.3 0.4 0.5 0.6 0.7

0.0

0.2

0.4

0.6

0.8

1.0

se

sele

ctio

n pr

obab

ility

For any pair (γ0, γ1), θ can be estimated by ML.16 / 26


Epidemiological evidence on lung cancer and passivesmoking

Meta-analysis reported by Hackshaw et al. (1997), reanalysed inCopas and Shi (2000b):

meta-analysis consists of 37 published epidemiological studies

each study provided an estimate of the odds ratio

Objective:Asses the epidemiological evidence for an increase in the risk oflung cancer resulting from exposure to environmental tobaccosmoke

Outcome measure:Relative risk of lung cancer among female lifelong non-smokers,according to whether her partner was a current smoker or a lifelongnon-smoker

17 / 26


Funnel Plot

0.5 1.0 2.0 5.0

0.6

0.4

0.2

0.0

Odds Ratio

Sta

ndar

d er

ror

●●

●●●

●●● ●●●

●

●●●

●●●● ●● ●

●

●●

●●●

●●●

●

●●

●

●

●

18 / 26


Analysis using copas

Copas selection model is implemented in the R package copas(Carpenter et al.; 2009).

> # loading library, requires library meta> library(copas)> # fit model> metaH <- metagen(lnOR,se,data=hackshaw, sm="OR")> copH <- copas(metaH,+ gamma0.range = NULL,+ gamma1.range = NULL,+ ngrid = 20, # grid for contourplot+ levels = NULL, # levels for contourplot+ slope = NULL # slope of ’orthogonal’ line+ )

19 / 26


Explore sensitivity of conclusions for selection mechanismsof varying strength

> plot(copH, which = 2)

0.06 0.08

0.1 0.12

0.14

0.14

0.16

0.1

8

0.2

−0.45 0.04 0.53 1.02 1.51 2

00.

030.

060.

090.

120.

15

Values of gamma0

Val

ues

of g

amm

a1●

●●

●

●

●

Contour plot

20 / 26


How does treatment effect change with varying selection?


1 0.9 0.8 0.7 0.6 0.5

00.

050.

10.

150.

20.

250.

3

Probability of publishing the trial with largest sd

log

OR

●

●

●

●

●

●

●

Treatment effect plot

21 / 26


Which probabilities of publishing the trial with largest SEare most consistent with the model?


●

●

●

●

●

●

●

1 0.9 0.8 0.7 0.6 0.5

00.

20.

40.

60.

81

Probability of publishing the trial with largest sd

P−

valu

e fo

r re

sidu

al s

elec

tion

bias

P−value for residual selection bias

22 / 26


Results

> summary(copH)

Summary of Copas selection model analysis:

publprob OR 95%-CI pval.treat pval.rsb N.unpubl

1.00 1.2416 [1.1256; 1.3697] < 0.0001 0.037 0

0.93 1.2218 [1.1072; 1.3482] < 0.0001 0.0633 1

0.86 1.1973 [1.0863; 1.3197] 0.0003 0.1145 3

0.81 1.1737 [1.0678; 1.2902] 0.0009 0.167 5

0.69 1.1504 [1.0248; 1.2914] 0.0176 0.2452 11

0.59 1.1271 [0.9972; 1.2740] 0.0555 0.3583 17

0.48 1.1047 [0.9666; 1.2625] 0.1438 0.4975 27

Copas model (adj) 1.1973 [1.0863; 1.3197] 0.0003 0.1145 3

Random effects model 1.2378 [1.1294; 1.3566] < 0.0001

Legend:

publprob Probability of publishing the study with the largest SE

pval.treat Pvalue for hypothesis that treatment effect is equal in both groups

pval.rsb Pvalue for hypothesis that no further selection remains unexplained

N.unpubl Approx. number of studies the model suggests remain unpublished

23 / 26


Conclusion

A modest degree of publication bias leads to a reduction inthe relative risk.

Although the number of unpublished studies is unlikely to belarge (see Copas and Shi; 2000b),the possibility of publication bias cannot be ruled out.

The published estimate of the increased risk needs to beinterpreted with some caution.

24 / 26


Summary

Simpler methods such as Trim and Fill can give misleadingresults if funnel plot asymmetry is due to other factors thanpublication bias.

Selection method approaches can account for such factorsand provide considerable insight into the problem ofpublication bias.

However, they are rarely used in practice partly due to thecomplexity of the methods and the rareness of user-friendlysoftware.

Carpenter et al. (2009) showed the practical utility of theCopas selection model and recommend using the modelroutinely for systematic reviews.

25 / 26


References

Carpenter, J. R., Schwarzer, G., Rucker, G. and Kunstler, R. (2009). Empiricalevaluation showed that the Copas selection model provided a useful summary in80% of meta-analyses, J Clin Epidemiol 62(6): 624–631.

Copas, J. B. and Shi, J. Q. (2000a). Meta-analysis, funnel plots and sensitivityanalysis, Biostatistics 1(3): 247–262.

Copas, J. B. and Shi, J. Q. (2000b). Reanalysis of epidemiological evidence on lungcancer and passive smoking, BMJ 320(7232): 417–418.

Copas, J. B. and Shi, J. Q. (2001). A sensitivity analysis for publication bias insystematic reviews, Stat Meth Med Res 10(4): 251–265.

Hackshaw, A., Law, M. and Wald, N. (1997). The accumulated evidence on lungcancer and environmental tobacco smoke, BMJ 315(7114): 980–988.

Hedges, L. V. (1992). Modeling publication selection effects in meta-analysis, Statist.Sci. 7(2): 246–255.

Hedges, L. V. and Vevea, J. L. (2005). Selection method approaches, in H. Rothstein,A. Sutton and M. Borenstein (eds), Publication Bias in Meta-Analysis: Prevention,Assessment Adjustments, Chichester, West Sussex: Wiley, pp. 145–174.

Sutton, A. J., Song, F., Gilbody, S. M. and Abrams, K. R. (2000). Modellingpublication bias in meta-analysis: a review, Stat Meth Med Res 9(5): 421–445.

26 / 26

publication bias in meta-analysis selection method approaches

Documents