1

Interim Analysis in Clinical Trials

Professor Bikas K Sinha [ ISI, KolkatA ]e-mail : sinhabikas@yahoo.comRU Workshop : April18, 2012

Books/Papers/Lecture Notes…..

Sample Size Calculations in Clinical Research By Shein-Chung Chow/Jun Shao/ Hansheng Wang

Interim Analysis deals with Early Stopping Rules in Testing of Hypotheses problems involving Comparison of Efficacies of Test & Standard Drugs

2

3

An Example of “Multiple Looks:” Consider planning a comparative trial in

which two treatments are being compared for efficacy (response rate).

H0: p2 = p1

H1: p2 > p1

A standard design says that for 80% power and with alpha of 0.05, you need about 100 patients per arm based on the assumption p2 = 0.50, p1= 0.30 which results in 0.20 for the difference.

So what happens if we find p < 0.05 before all patients are enrolled ?

Why can’t we look at the data a few times in the middle of the trial and conclude that one treatment is better if we see p < 0.05?

4

The plots to the right show simulated data where p1 = 0.40 and p2 = 0.50

In our trial, looking to find a difference between 0.30 to 0.50, we would not expect to conclude that there is evidence for a difference.

However, if we look after every 4 patients, we get the scenario where we would stop at 96 patients and conclude that there is a significant difference.

Number of Patients

Ris

k R

atio

0 50 100 150 200

0.0

0.5

1.0

1.5

Number of Patients

pval

ue

0 50 100 150 200

0.2

0.4

0.6

0.8

1.0

• H 1

5

If we look after every 10 patients, we get the scenario where we would not stop until all 200 patients were observed and would conclude that there is not a significant difference

(p =0.40)

Number of Patients

Ris

k R

atio

50 100 150 200

1.0

1.2

1.4

1.6

Number of Patients

pval

ue

50 100 150 200

0.2

0.4

0.6

0.8

1.0

• H 1

6

If we look after every 40 patients, we get the scenario where we would not stop either.

If we wait until the END of the trial (N = 200), then we estimate p1 to be 0.45 and p2 to be 0.52. The p-value for testing that there is a significant difference is 0.40.

Number of Patients

Risk

Rat

io

50 100 150 200

1.0

1.2

1.4

Number of Patients

pval

ue

50 100 150 200

0.2

0.4

0.6

0.8

1.0

• H 1

7

Would we have messed up if we looked early on? Every time we look at the data and consider

stopping, we introduce the chance of falsely rejecting the null hypothesis.

In other words, every time we look at the data, we have the chance of a type 1 error.

If we look at the data multiple times, and we use alpha of 0.05 as our criterion for significance, then we have a 5% chance of stopping each time.

Under the true null hypothesis and just 2 looks at the data, then we “approximate” the error rates as: Probability stop at first look: 0.05 Probability stop at second look: 0.95*0.05 =

0.0475 Total probability of stopping is 0.0975

Illustrative Examples :Interim AnalysisExample 1. It is desired to carry out an experiment

to examine the superiority, or otherwise, of a thera-

peutic drug over a standard drug with 5% level and

90% power for detection of 10% difference in the

proportions ‘cured’.

‘C’ : Standard Drug ‘T’ : Therapeutic Drug

H_0 : P_C - P_T = 0

H_1 : P_C # P_T

Size = 0.05, Power = 0.90 for ∆=P_T – P_C = 0.10.

IT IS A BOTH-SIDED TEST. 8

Sample Size Determination…. Reference : Pages 22/23/30 of Last

Reference Ref: Lachin, Controlled Clinical Trials

2:93-113, 1981.

9

Determination of Sample Size for Full Analysis

10

Two-sided Test

α = 0.05; Z_ α /2 = 1.96

Power = 0.90; β = 0.10, Z_β = 1.282, ∆ =0.10

N = 2(Z_ α /2 + Z_ β)^2 pbar(1-pbar)/∆^2

Assume pbar = 0.35 [suggestive cure rate]

N = 2(1.96 + 1.282)^2 (0.35)(0.65)/(0.10)^2

= 21.021128 x 22.75= 478.23……480

Conclusion: Each arm involves 480 subjects.

Full Experiment vs. Interim AnalysisFor Full Experiment : Needed 480 subjects in

each ‘arm’.

At the end of the entire experiment, suppose

we observe :

‘C’ : # cured = 156 out of 480 i.e., 32.5%

‘T’ : # cured = 190 out of 480 i.e., 39.6%

Therefore, p^_C = 0.325 and p^_T = 0.396.

Hence, pbar = [p^_C + p^_T]/2 = 0.3605.

Finally, we compute the value of z given by

11

Full Analysis….. Z_obs. = [p^_C – p^_T]/sqrt[pbar(1-pbar)2/N] =[.325-.396]/sqrt[.36x.64x2/480] = -[.071]/sqrt[0.00192] = -2.29 In absolute value, z_obs. is computed as 2.29

which is more than the ‘critical’ value of z given by 1.96 [for a both-sided test with size 5%].

Hence, we conclude that the Null Hypothesis is ‘not tenable’, given the experimental outputs.

12

Interim Analysis : 2 ‘Looks’ First Look : use 50% of data

2nd Look : At the end, if continued after 1st.

Q. What is the size of the test at 1st look ? Also, what is the size at the 2nd look so that on the whole the size is 5 % ?

Ans. If we use 5% for the size at each of 1st and

2nd looks, then the over-all size becomes 0.08 approx. [Proof ?]

Hence……both can NOT be taken at 5%.

Start with < 5 % and then take > 5%.....13

Interim Analysis : 2 Looks

Defining Equation :

α = P[│Z_I │> z*] + P[│Z_I│< z*,│Z_{I,II}│> z**]

where Z_I and Z_II are based on 50% data in

two identical and independent segments so

that their distributions are identical. Further,

Z_{I,II} = [z_I + z_II]/sqrt(2) is based on

combined evidence of I & II and hence Z_I and

Z_{I,II} are dependent.

Choices of z* and z** : intricate formulae.

14

Interim Analysis : 2 Looks Z-computation….

z_I obs. is to be based on 50% data upto the

1st look for each of ‘C’ and ‘T’.

Data : C (90/240) & T(120/240) & n = 240.

p^_C = 90/240 = 0.375; p^_T = 120/240=0.50

pbar = (0.375 + 0.50)/2 = 0.4375.

z_I obs. = [p^_C – p^_T]/sqrt[pbar(1-pbar)2/n] = - [ 0.125 ]/sqrt{.4375x.5625x2/240} = - (0.125)/sqrt{0.002050} = - 2.76 implies ???

15

Interim Analysis : 2 Looks

16

Suggested cut-off points for 2 Looks [approx.]

z_c Hebittle-Peto Pocock O’Brien-Fleming

z* 3.0 2.178 2.796

z** 2.0 2.178 1.977 z_I obs. in absolute value = 2.76 Conclusion ? Reject H_0 ….suggested by Pocock’s Rule Continue …suggested by other two. Finally, z = - 2.29 suggests rejection of H_0 by

all the three rules

Interim Analysis : 3 looks

Cut-off points : Suggested Rules [approx.]

z_c Hebittle-Peto Pocock O’Brien-Fleming

z* 3.0 2.289 3.471

z** 3.0 2.289 2.454

z*** 2.0 2.289 2.004

: 1st look; ** : 2nd look; *** : 3rd look •

17

Interim Analysis : 4 Looks Cut-off points : Suggested Rules

z_c Hebittle-Peto Pocock O’Brien-Fleming

z* 3.0 2.361 4.106

z** 3.0 2.361 2.903

z*** 3.0 2.361 2.370

z**** 2.0 2.361 2.053

• : 1st look; ** : 2nd look; *** : 3rd look and • **** : last [4th] look

18

Interim Analysis : 4 Looks Details of data sets :

C : 48/120; 42/120; 30/120; 36/120 …Total 156/480

T : 54/120; 66/120; 32/120; 38/120 …Total 190/480

Progressive proportions for ‘C’ :

48/120=0.40; (48+42)/240= 0.375;

(48+42+30)/360=0.333; 156/480=0.325

Progressive proportions for ‘T’ :

54/120=0.45; (54+66)/240= 0.50;

(54+ 66+32)/360=0.422; 190/480=0.39619

Interim Analysis : 4 Looks

Progressive computations of pbar……

1st Look : pbar = (0.40 + 0.45)/2 = 0.425

2nd Look : pbar = (0.375 + 0.50)/2 = 0.4375

3rd Look : pbar = ( 0.333 + 0.422)/2 = 0.3639

4th Look : pbar = (0.325 + 0.396)/2 = 0.3605

20

Interim Analysis : 4 Looks

Progressive Computations of z-statistic

Generic Formula :

z-obs. for ‘Look # i’ is the ratio of

(a) [p^_C(i)– p^_T(i)] for i-th Look

(b) sqrt[pbar(i)(1-pbar(i))2/n(i)]

where pbar(i) corresponds to Look # i and

also ‘n(i) ’ corresponds to size of each arm

of Look # i for each i = 1, 2, 3,4.

Note : n(1)=120; n(2)=240; n(3)=360, n(4)=480

21

Interim Analysis : 1st Look z_(Look I) obs.

= [p^_C – p^_T]/sqrt[pbar(1-pbar)2/n*] = [ 0.40-0.45 ]/sqrt{.425x.575x2/120} = - (0.05)/sqrt{0.004073} = -0.7835

Conclusion : All Rules are suggestive of Continuation to 2nd Look

22

Interim Analysis : 2nd Look

z_(Look II) obs.

= [p^_C – p^_T]/sqrt[pbar(1-pbar)2/n**] = [0.375-0.50 ]/sqrt{.4375x.5625x2/240} = - (0.125)/sqrt{0.002050} = - 2.76 Conclusion : Reject H_0 by Pocock’s Rule However, continue to 3rd Look according

to the other two rules.

23

Interim Analysis : 3rd Look

z_(Look III) obs.

= [p^_C – p^_T]/sqrt[pbar(1-pbar)2/n***] = [0.333-0.422 ]/sqrt{.3639x.6361x2/360} = - (0.089)/sqrt{0.001286} = - 2.48 Conclusion : Reject H_0 by Pocock & OBF

Rules but Continue by H-P Rule Last Look : z_obs. = -2.29

Accept H_0 by Pocock’s Rule only

24

Data Analysis….Interpretations Decision Rules :

Pocock’s Rule : Maintains uniformity in critical values ….so …apparently ‘conservative’ at the start…slowly turns into ‘liberal’ !

Other Rules : Liberal at the start and conservative at the end…..

All Rules have to maintain the ‘averaging principle’ to meet alpha at the end.

No Rule can be strict/liberal all through the Looks.

25

Interim Analysis : Example 2 Continuous data : Testing for equality of

mean effects of two treatments : ’C’ & ’T’. As before, we have Null and Alt. Hypotheses

and we have a specified value of DELTA = Mean of T – Mean of C and a specified power, say 90% to detect

this. Taking size equal to 5%, we solve for the sample size in each arm.

This is routine computation and we take sample size N = 525 in each arm.

Full Analysis : Sample Size Computation Assume normal distribution with sigma = 5.

Two-sided Test

α = 0.05; Z_ α /2 = 1.96

Power = 0.90; β = 0.10, Z_β = 1.282,

∆ = 0.20 times sigma = 20% of sigma = 1.0

N = 2(Z_ α /2 + Z_ β)^2 x sigma^2 /∆^2

= 2(1.96 + 1.282)^2 / 0.04

= 525 [approx.]

We can think of 5 Looks altogether…at equal

Steps…..each with approx. 105 observations.

Interim Analysis…Example contd.Details of data sets : (mean, sample size)

C : (30.5,105); (31.8, 105); (29.7, 105);

(30.2, 105); (31.3, 105)

T : (31.7,105); (32.0, 105); (30.8, 105);

(33.7, 105); (32.8, 105)

Progressive sample means for ‘C’ :

30.5, 31.15, 30.67, 30.55, 30.70

Progressive sample means for ‘T’ :

31.7, 31.85, 30.83, 32.55, 32.60

Interim Analysis : Example contd….Progressive Computations of z-statistic Generic Formula : z-obs. for ‘Look # i’ is the ratio of (a) [mean_C(i)– mean_T(i)] for i-th Look (b) sigma times Sqrt 2/n(i)]where mean refers to sample mean for and also ‘n(i) ’ corresponds to size of each arm of Look # i for each i = 1, 2, 3,4, 5.Note : n(1)=105; n(2)=210; n(3)=315,

n(4)=420 and n(5) = 525.

Interim Analysis : Example contd. Cut-off points : Suggested Rules

z_c Hebittle-Peto Pocock O’Brien-Fleming z* 3.0 2.413 4.562 z** 3.0 2.413 3.225 z*** 3.0 2.413 2.634 z**** 3.0 2.413 2.281 z***** 2.0 2.413 2.040 • : 1st look; ** : 2nd look; *** : 3rd look; • **** : 4th look & ***** : Last [5th] look

Interim Analysis…Example contd. z_(Look I) obs.

= [mean_C – mean_T]/sigma x sqrt[2/n*] = - [ 1.2] / 5 x sqrt{2/105} = - 1.74 Conclusion : Continue to 2nd Look

Interim Analysis : Example contd. z_(Look II) obs.

= [mean_C – mean_T]/sigma x sqrt[2/n**] = - [ 0.7 ] / 5 x sqrt{2/210} = - 1.43 Conclusion : Continue to 3rd Look

Interim Analysis : Example contd.z_(Look III) obs.

= [mean_C – mean_T]/sigma x sqrt[2/n***] = - [ 0.16 ] / 5 x sqrt{2/315} = - 0.40 Conclusion : Continue to 4th Look

Interim Analysis : Example contd. z_(Look IV) obs.

= [mean_C – mean_T]/sigma x sqrt[2/n****] = - [ 2.0 ] / 5 x sqrt{2/420} = - 5.80 Conclusion : Stop and Reject H_0. Strong evidence against H_0 and yet 105

observations per arm are left to be studied. What if the expt was continued till the end

anyway ?

Interim Analysis : Example contd. z_(Look V) obs.

= [mean_C – mean_T]/sigma x sqrt[2/n*****] = - [ 1.90 ] / 5 x sqrt{2/525} = - 6.16 Conclusion : Reject H_0. Quite a strong evidence against H_0

Open Problems….. 1. Computations of Boundaries for Interim

Analysis [IA] :Theory & Computational Algorithms for Different ‘Looks’

2. Comparison of Different IA Rules ? 3. New Simpler Rules ? 4. Power Performance Within/Between Rules ? 5. Sample Size Adjustment : Why & How ? 6. Miscellaneous Topics [Alpha Spending Functions etc]

36