computing effect sizes for clustered randomized trials€¦ · in cluster randomized trials, smd...

1

The Campbell Collaboration www.campbellcollaboration.org

Computing Effect Sizes for Clustered Randomized Trials

Terri Pigott, C2 Methods Editor & Co-Chair Professor, Loyola University Chicago

[email protected]

Campbell Collaboration Colloquium – August 2011 www.campbellcollaboration.org

Computing effect sizes in clustered trials •  In an experimental study, we are interested in the difference

in performance between the treatment and control group •  In this case, we use the standardized mean difference, given

by

T C

p

Y Yds−

= g g

Treatment group mean

Control group mean

Pooled sample standard deviation

2


Variance of the standardized mean difference

22 ( )

2( )

T C

T C T C

N N dS dN N N N+

= ++

where NT is the sample size for the treatment group, and NC is the sample size for the control group


TREATMENT GROUP CONTROL GROUP

1 2, ,..., TT T T

NY Y Y

TYgCYg

Overall Trt Mean Overall Cntl Mean

1 2, ,..., CC C C

NY Y Y

2TrtS 2

CntlS2pooledS

3


In cluster randomized trials, SMD more complex •  In cluster randomized trials, we have clusters such as

schools or clinics randomized to treatment and control •  We have at least two means: mean performance for each

cluster, and the overall group mean •  We also have several components of variance – the within-

cluster variance, the variance between cluster means, and the total variance

•  Next slide is an illustration



{ } { }11 1 1,... ,...,T T

T T T Tn m m n

Y Y Y YgggTrt Cluster 1 Trt Cluster mT

{ } { }11 1 1,... ,...,C C

C C C Cn m m n

Y Y Y YgggCntl Cluster 1 Cntl Cluster mC

1TY g T

TmY

g 1CY g C

CmY

g

Trt Cluster 1 Mean Trt Cluster mT Mean Cntl Cluster 1 Mean Cntl Cluster mC Mean

TYgg

••• •••

CYggOverall Trt Mean

Overall Cntl Mean

Assume equal sample size, n, within clusters

4


The problem in cluster randomized trials •  Which mean is the appropriate mean to represent the

differences between the treatment and control groups? –  The cluster means? –  The overall treatment and control group means averaged

across clusters? •  Which variance is appropriate to represent the differences

between the treatment and control groups? –  The within-cluster variance? –  The variance among cluster means? –  The total variance?


Three different components of variance •  The next slides outline the three components of variance that

exist in a cluster randomized trial •  These components of variance correspond to the variances

that we would estimate when analyzing a cluster randomized trial using hierarchical linear models

5


Pooled within-cluster sample variance •  One component of variance is within-cluster variation •  This is computed as the pooled differences between each

observation and its cluster mean

2 2

1 1 1 12

( ) ( )T Cm n m n

T T C Cij i ij i

i j i jW

Y Y Y YS

N M= = = =

− + −

=−

∑∑ ∑∑g g

M is number of clusters: M = mT + mC

N is total sample size: N = n mT + n mC = NT + NC with equal cluster sample sizes, n



{ } { }11 1 1,... ,...,T T

T T T Tn m m n


{ } { }11 1 1,... ,...,C C

C C C Cn m m n


1TY g T

TmY

g 1CY g C

CmY

g


TYgg

••• •••


Overall Cntl Mean

2WS

Within-cluster variance

6


Pooled within-group (treatment and control) sample variance of the cluster means •  Another component of variation is the variance among the

cluster means within the treatment and control groups •  This is computed as the pooled difference between each

cluster mean and its overall group (treatment and control) mean

2 2

2 1 1

( ) ( )

2

T Cm mT T C Ci i

i iB T C

Y Y Y YS

m m= =

− + −=

+ −

∑ ∑g gg g gg



{ } { }11 1 1,... ,...,T T

T T T Tn m m n


{ } { }11 1 1,... ,...,C C

C C C Cn m m n


1TY g T

TmY

g 1CY g C

CmY

g


TYgg

••• •••


Overall Cntl Mean

2BS Variance between cluster

means

7


Total pooled within-group (treatment and control) variance •  The total variance is the variation among the individual

observations and their group (treatment and control) overall means

2 2

1 1 1 12

( ) ( )

2

T Cm n m nT T C Cij ij

i j i jT

Y Y Y YS

N= = = =

− + −

=−

∑∑ ∑∑gg gg



{ } { }11 1 1,... ,...,T T

T T T Tn m m n


{ } { }11 1 1,... ,...,C C

C C C Cn m m n


1TY g T

TmY

g 1CY g C

CmY

g


TYgg

••• •••


Overall Cntl Mean

2TS Total pooled within-group variance

(treatment and control)

8


Just a little distributional theory •  As usual, we will assume that each of our observations

within the treatment and control group clusters are normally distributed about their cluster means, µT

i and µCi with

common within-cluster variance, σ2W, or

2

2

~ ( , ), 1,..., ; 1,...,

~ ( , ), 1,..., ; 1,...,

T T Tij i W

C C Cij i W

Y N i m j n

Y N i m j n

µ σ

µ σ

= =

= =


Just a little more distributional theory •  We also assume that our clusters are sampled from a

population of clusters (making them random effects) so that the cluster means have a normal sampling distribution with means µT

• and µC• and common variance, σ2

B

2

2

~ ( , ), 1,...,

~ ( , ), 1,...,

T T Ti BC C Ci B

N i mN i m

µ µ σ

µ µ σ

=

=g

g

9


The three component s of variance are related:

2 2 2T W Bσ σ σ= +


The intraclass correlation, ρ •  This parameter summarizes the relationship among the three

variance components •  We can think of ρ as the ratio of the between cluster variance

and the total variance •  ρ is given by

2 2

2 2 2B B

B W T

σ σρ

σ σ σ= =

+

10


Why is the intraclass correlation important? •  Later, we will see that we need the intraclass correlation to

obtain the values of our standardized mean difference and its variance

•  Also, if we know the intraclass correlation, we can obtain the value of one of the variances knowing the other two. For example:

2 2

2 2

(1 ) /

(1 )W B

W T

σ ρ σ ρ

σ ρ σ

= −

= −


So far, we have •  Discussed the structure of the data •  Seen how to compute three different sampling variances:

S2W, S2

B, and S2T

•  Discussed the underlying distributions of our clustered data •  Seen how the variance components in the distributions of our

data are related to one another •  Next we will see how to estimate the variance components

using our sampling variances •  (Yes, we are building up to the effect sizes…)

11


Estimator of σ2W

•  As we might expect, we estimate σ2W using our estimate of

the within-cluster sampling variance, or,

2 2ˆW WSσ =


Estimator of σ2B

•  Unfortunately, the estimate of σ2B is not as straightforward as

for σ2W

•  This is due to the fact that S2B includes some of the within-

cluster variance. •  The expected value for S2

B is

22 WB n

σσ +

12


Thus, an estimator for σ2B is

22 2ˆ WB B

SSn

σ = −


And, given prior information, an estimate of σ2T is

2 2 21ˆT B WnS Sn

σ−⎛ ⎞= + ⎜ ⎟

⎝ ⎠

13


Now, we are ready for computing effect sizes •  There are 3 possibilities for computing an effect size in a

clustered randomized trial •  These correspond to the 3 different components of variance

we have been discussing •  The choice among them depends on the research design

and the information reported in the other studies in our meta-analysis


The most common: δW •  This effect size is comparable to the standardized mean

difference computed from single site designs

T C

WW

µ µδ

σ−

= g g

14


We estimate δW using dW

T C

WW

Y YdS−

= gg ggTrt & Cntl means

Pooled within-cluster variance


Variance of dW •  Variance of dW depends on the intraclass correlation, sample

sizes, and dW

{ }21 ( 1)

1 2( )

T CW

W T C

dN N nV dN N N M

ρρ

⎛ ⎞⎛ ⎞+ + −= +⎜ ⎟⎜ ⎟− −⎝ ⎠⎝ ⎠

When SB is 0 (no between-group variation among the cluster means), the variance of dW is equal to the variance for the standardized mean difference in a single-site study

15


A second estimator: δT •  This estimator uses the total variation in the denominator •  We may compute this effect size when the other studies in

our meta-analysis are also multi-site studies •  The general form is:

T C

TT

µ µδ

σ−

= g g


There are two ways to compute δT •  Method 1: dT1

–  Used when the study reports S2B , the variance of the cluster

means, and S2W , the pooled within-cluster variance

•  Method 2: dT2 –  Used when we the study reports the intraclass correlation, and

S2T , the total variance

16


Estimator dT1 : When we have S2B and S2

W

1

2 2

, withˆ

1ˆ

T C

TT

T B W

Y Yd

nS Sn

σ

σ

−=

−⎛ ⎞= + ⎜ ⎟⎝ ⎠

gg gg


The variance of dT1 (a little messy)

{ }

[ ]

1

2 2 2212 2

(1 ( 1) )

1 ( 1) ( 1) (1 )2 ( 2) 2 ( )

T C

T T C

T

N NV d nN N

n n dn M n N M

ρ

ρ ρ

⎛ ⎞+= + − +⎜ ⎟⎝ ⎠

⎡ ⎤+ − − −+⎢ ⎥

− −⎢ ⎥⎣ ⎦

17


Estimator dT2 : When we have S2T and ρ

22( 1)1

2

T C

TT

Y Y ndS N

ρ⎛ ⎞− −= −⎜ ⎟ −⎝ ⎠

gg gg


Variance of dT2 (also messy)

{ }

[ ]

2

2 222

(1 ( 1) )

( 2)(1 ) ( 2 ) 2( 2 ) (1 )2( 2) ( 2) 2( 1)

T C

T T C

T

N NV d nN N

N n N n N ndN N n

ρ

ρ ρ ρ ρρ

⎛ ⎞+= + − +⎜ ⎟⎝ ⎠

⎛ ⎞− − + − + − −⎜ ⎟⎜ ⎟− − − −⎝ ⎠

18


A third estimator based on S2B

•  We use this when the treatment effect is defined at the level of the clusters, or when the other studies in our meta-analysis have been analyzed using cluster means as the unit of analysis

•  Though it might not be of general interest, we can use this effect size to obtain other effect sizes of interest

•  There are two estimates of this effect


Estimator dB1 : When we have S2B and S2

W

1

22

, withˆ

ˆ

T C

BB

WB B

Y Yd

SSn

σ

σ

−=

= −

gg gg

19


The variance of dB1 (a little messy)

{ }

[ ]

1

2 2212 2 2 2

1 ( 1)

1 ( 1) (1 )2( 2) 2( )

T C

B T C

B

m m nV dm m n

nd

M n N M n

ρρ

ρ ρρ ρ

⎛ ⎞+ + −= +⎜ ⎟⎝ ⎠

⎡ ⎤+ − −+⎢ ⎥

− −⎢ ⎥⎣ ⎦


Estimator dB2 : When we have SB and ρ

21 ( 1)T C

BB

Y Y ndS n

ρρ

− + −= gg gg

20


The variance of dB2 (a little messy)

{ }

( )

2

22

1 ( 1)

1 ( 1)2( 2)

T C

B T C

B

m m nV dm m n

n dM n

ρρ

ρ

ρ

⎛ ⎞+ + −= +⎜ ⎟⎝ ⎠

+ −

−


IMPORTANT: Can compute any δ from any other effect size and ρ

1 1

1

TW B

T B W

δρδ δ

ρ ρ

δ δ ρ δ ρ

= =− −

= = −

21


And, can compute the variance of the effect size using the same transformations

{ } { } { }

{ } { } { }1 1

(1 )

TW B

T B W

VarVar Var

Var Var Var

δρδ δ

ρ ρ

δ δ ρ δ ρ

= =− −

= = −


Where are we now? •  We have looked at the structure of the data •  We have examined the different components of variation •  We have defined estimators for the different components of

variation •  We have outlined the computation of three different effect

sizes, dW , dT , and dB and their associated variances

22


But we all know it is not that simple: Some common problems •  We have unequal cluster sample sizes

–  There are formulas for unequal cluster sample sizes that are much more complex

–  We can assume equal cluster sizes since most studies attempt to use equal cluster sizes in the design

•  We don’t know the intraclass correlation coefficient –  We can estimate it from a number of sources listed in the

References –  Several researchers have provided estimates from other studies

and from large-scale national samples


Let’s look at some examples

23


From the analysis section


Another clue that we have cluster means and sds reported

24


The ICCs are also reported


But I could not find sample sizes within clusters so looked for another report on the intervention

25



So, what do we have in this example? •  We have ρ for SAT Math of 0.72 •  There are 10 matched-pairs of schools, with NT = 976, and

NC = 738 (conservative common n = 73) •  We have the mean of the school (cluster) means and SDs for

the treatment and control group at the level of school

26


We have all the information needed to compute dB2

2 22

2

(7.26) (7.48) 54.33, 7.372

82.33 78.77 1 (73 1)0.727.37 73(0.72)

0.48 1.005 0.48

B B

B

S S

d

+= = =

− + −=

= =


To compute Var{dB2 }

{ }

( )

210 10 1 (73 1)(0.72)10*10 (73)(0.72)1 (73 1)0.72 (0.48)2(20 2) (73)(0.72)20 52.84 (52.84)(0.48) 0.201 0.013100 52.56 1892.160.214

BV d + + −⎛ ⎞= ⎜ ⎟⎝ ⎠

+ −+

−

⎛ ⎞⎛ ⎞= + = +⎜ ⎟⎜ ⎟⎝ ⎠⎝ ⎠

=

27


Say we want dW in our meta-analysis

0.720.481 1 0.72

0.48(1.60) 0.77

W Bd d ρρ

= =− −

= =


The variance of dW follows the same transformation

{ } { }

{ }

2

1

0.720.214(1 0.72)

0.214(2.57) 0.550.74

W B

W

Var d Var d

SD d

ρρ

⎛ ⎞= ⎜ ⎟⎜ ⎟−⎝ ⎠

⎛ ⎞= ⎜ ⎟−⎝ ⎠= =

=

28


If instead, we want dT

{ } { }

{ }

0.48 0.720.48(0.85) 0.41

(0.214)(0.72) 0.1540.392

T B

T W

T

d d

Var d Var d

SD d

ρ

ρ

= =

= =

=

= =

=


Some observations on this example •  The intraclass correlation is large for this measure so we

have a lot of between school variability •  Thus, it is not surprising that none of the effect sizes we

calculated are significantly different from zero •  In this example, we have the intraclass correlation reported,

but we may not be so lucky

29


Another example


We might have individual level info in Table 2

30



What information do we have? •  mT = 14 schools in the treatment group, mC = 10 schools in

the control group •  For treatment group, sample size average is nT = 3429/14 =

244.9. For control group, sample size average is nC = 1047/10 = 104.7

•  We have individual level means and standard deviations BUT •  We don’t have the intraclass correlation, or other estimates

of the variance

31


To compute effect sizes, we need either 2 estimates of the variance, or 1 estimate and ρ

•  We can only compute the estimate of S2T from our 2

estimates of the individual level sds •  We will need to find an estimate for ρ from another source •  From: Murray & Blitstein (2003). Methods to reduce the

impact of intraclass correlation in group-randomized trials. Evaluation Review, 27: 79-103

•  The upper bound of the ICC for youth cohort studies with outcomes focused on diet is 0.0310


Since we have individual level stats:

2 22

2

(3429 1)(0.52) (1047 1)(0.25) 0.223429 1047 2

3.40 2.16 2(104 1)0.03110.47 208 2

2.64 0.97 2.59

T

T

S

d

− + −= =

+ −

− −= −

−

= =

32


The variance of dT

{ }

[ ]

{ }

2

2 22

2

2

1456 1040 (1 (104 1)*0.031)1456*1040

(2496 2)(1 0.031) 104(2496 2*104)0.031 2(2496 2*104)0.031(1 0.031)2.592(2496 2) (2496 2) 2(104 1)0.031

0.007 2.59 (0.0002) 0.00830.091

T

T

V d

SD d

+⎛ ⎞= + − +⎜ ⎟⎝ ⎠

⎛ ⎞− − + − + − −⎜ ⎟⎜ ⎟− − − −⎝ ⎠

= + =

=


If we want dW

{ } { }

{ }

2.59 2.641 1 0.031

0.0083 0.00861 1 0.031

0.093

TW

TW

W

dd

Var dVar d

SD d

ρ

ρ

= = =− −

= = =− −

=

33


What about other types of effect sizes, like odds-ratios?

•  We do not yet have the same research for odds ratios •  What we can do is inflate the odds ratios from a clustered

trial using recommendations from the Cochrane Handbook


Recommendations from the Cochrane Handbook •  We first compute the design effect given by

1 + (nave – 1) ρ where nave is the average cluster size, and ρ is the intraclass

correlation coefficient •  We divide the total sample sizes and number of events by

the design effect to obtain the corrected numbers to compute the odds-ratio

•  We adjust the variance by computing the variance of the log-odds ratio using the original sample sizes and dividing by the square root of the design effect

34


From the Positive Action program

No smoking Smoked

Intervention 937 4.0% of 976 = 39

Control 682 7.6% of 738 = 56


And the intraclass correlation coefficient

35


Adjusted values •  Average cluster size: (738+976)/20 = 85.7 •  Design effect: 1 + (85.7 – 1) 0.05 = 5.235

No smoking Smoked

Intervention 937/5.235=178.99 4.0% of 976 = 39/5.235 = 7.45

Control 682/5.235=130.28 7.6% of 738 = 56/5.235 = 10.7


Variance for the log-odds ratio •  OR = (937*56)/(682*39) = 1.97 •  lnOR = ln(1.97) = 0.68 •  SE2(lnOR) = 1/937 + 1/56 + 1/682 + 1/39 = 0.046 •  SE(lnOR) = 0.214 •  Adjusted SE(lnOR) = 0.214/√5.235 = 0.094

36


References Hedges, L. V. (2007). Effect sizes in cluster-randomized

designs. Journal of Educational and Behavioral Statistics, 32, 341-370.

Hedges, L. V. & Hedberg, E. C. (2007). Intraclass correlation values for planning group randomized experiments in education. Educational Evaluation and Policy Analysis, 29, 60-87.

Cochrane Handbook: http://www.cochrane-handbook.org/

The Campbell Collaboration www.campbellcollaboration.org

P.O. Box 7004 St. Olavs plass 0130 Oslo, Norway

E-mail: [email protected]

http://www.campbellcollaboration.org

computing effect sizes for clustered randomized trials€¦ · in cluster randomized trials, smd...

Documents