august, 2002mcp a general multiplicity adjustment approach and its application to evaluating several...
TRANSCRIPT
August, 2002 MCP
A General Multiplicity Adjustment Approach and
Its Application to Evaluating Several Independently Conducted Studies
Qian Li and Mohammad HuqueCDER/FDA
August, 2002 MCP
Disclaimer
• The views expressed in this talk are those of the authors and do not necessarily represent those of the Food and Drug Administration
August, 2002 MCP
Outlines
• Motivation• Extending the union-intersection method• Issues in application
– relationship of the decision errors and decision rules– choosing a decision error– power characteristics
• An example and closing remarks
August, 2002 MCP
Motivation
• More than one independently conducted Phase III study in NDA submissions to support efficacy evaluation
• Current practice– count the number of studies that are significant– combine studies
• Interpretation of regulatory requirement– two successful studies – no consistent interpretation when more than two studies
are conducted
August, 2002 MCP
Union-Intersection method
• Roy (1953) first proposed a method of constructing a hypothesis test:H0: K
i=1H0i, for K hypothesis tests.
PH0(Ti>, i=1,2,…,K)=
where is a critical cut point.
August, 2002 MCP
Extending the UI Approach
• PH0(p(1)1 p(2)2... p(K)K) ’
– where 1 2 ... K 1 are p-value cut points
– p(1), p(2), …, p(K) are ordered p-values of p1, p2, …, pK
’ is an overall type I error
August, 2002 MCP
Definitions of overall hypotheses
• Possible choices of overall hypotheses:– at least one alternative is true
H01/K: i=1
KH0i vs. HA1/K: i=1
KHAi
– at least two alternatives are trueH0
2/K: j=1K (i=1 to K,ijH0i ) vs. HA
2/K : j=1K (i=1 to K,ijH0i )
…
– all the alternatives are trueH0
K/K: j=1K H0i vs. HA
K/K : j=1K HAi
August, 2002 MCP
Overall type I error and p-value cut points
• For H01/k, the extended approach can be rewritten
as follows when p-values are independent
K
KK ppK
pppdpdppdK
1
2
12
1
1
210
!
August, 2002 MCP
Overall type I error and p-value cut points
• For HAm+1/k (m>1) ,
– max overall type I error occur when m studies have power 1 to reject individual null
I’s satisfy the following
and 1= 2= … = m m+1.
K
KK
m
mm
m
mpp
Kpp mm
pdpdppdmK
1
2
12
1
1
210
)!(
August, 2002 MCP
A special case of two hypotheses H0
1/2: H01 H02
• H01: 10, H02: 2 0
• The null space is the third quadrant
• max decision error occur at (1=0, 2=0)
• 1 and 2 satisfy 21 2 -12’
• More than one set of 1 and 2
• For ’=0.05, • if 1 = 0.025, 2 =1
• if 1 = 0.05, 2 =0.525
2
1(0,0)
August, 2002 MCP
Rejection regions
0.525
0.525
1=0.050
2=0.525
1=0.025
2=1.000
1
p1 p1
p2p20.05
0.05
1
0.0250.025
August, 2002 MCP
Possible decision rules for two studieswhen ’= 0.0252
1
2
0.025 0.0250.020 0.0260.010 0.0360.004 0.075
August, 2002 MCP
Special case of two studiesH0
2/2: H01 H02
• The null space is all the area except first quadrant
• max decision error occurs at (1= , 2=0) & (1=0, 2=)
• max overall type I error is controlled when 1 =2’
1
2
(0,0)
August, 2002 MCP
Issues in application
• Choice of overall hypothesis
• Choice of decision error
• Power characteristics
August, 2002 MCP
Relationship of decision errors among different overall hypotheses
• For a set of K independent p-values, there exists a common rejection region p1 = p2 =…=pk .
• The corresponding decision errors for the overall hypotheses are: H0
1/k: ’= k
H02/k : ’= k-1
…
H0k-1/k: ’= 2
H0k/k : ’=
August, 2002 MCP
Relationship of decision rules
• It can be shown that the decision rules derived from a stringent hypothesis can also be derived from a less stringent hypothesis, given the relationship of the decision errors among different overall hypotheses.
August, 2002 MCP
Relationship of decision errors
• Exist a common rejection region in (p1, p2)
That is to require two significant studies,
p1 = p2
Decision errors : – H0
1/2: H01 H02 ’= 2
– H02/2 : H01 H02 ’= p1
p2
0
August, 2002 MCP
Strategies for choosing decision errors
• Considering 4 strategies– Use the same decision error for all HA
k/k
– Use the same decision error for all HA1/k
– Find the decision errors that keep a constant power for HA
k/k
– Control the power increase for HAk/k
August, 2002 MCP
Same decision error for HAk/k
• Require all the K studies significant at level ’• Similar to UI decision rules • Power is low
Each study has 90% power at level 0.025 K: 12 3 4 5 6
Error: 0.025 0.025 0.025 0.025 0.025 0.025
Power: 90.0 80.8 72.9 65.6 59.0 53.1
• Not fair to use the same decision error for HAk/k when
K is large
August, 2002 MCP
Same decision error for HA1/K
• Decision error and power for HAK/K
Each study has 90% power at level 0.025 K: 1 2 3 4 5 6
Error: 0.0006 0.025 0.085 0.158 0.2290.292
Power: 50.0 81.0 96.9 98.7 99.4 99.6
• When K increases, there is a large increase in decision error,
• therefore large increase in power
August, 2002 MCP
Keeping consistent power for HAK/K
• This strategy is in between the first two strategies
• Decision error and power for HAK/K
Each study has 90% power at level 0.025 K: 1 2 3 4 5 6
Error: 0.009 0.025 0.040 0.054 0.0660.077
Power: 81.0 81.0 81.0 81.0 81.0 81.0
• In case not satisfied ...
August, 2002 MCP
Increasing power for HAK/K
• Decide a reasonable power for HAK/K, then figure
out the error rate
• Decision error and power for HAK/K
Each study has 90% power at level 0.025 K: 1 2 3 4 5 6
Error: 0.007 0.025 0.046 0.068 0.0930.121
Power: 78.6 81.0 83.0 85.0 87.0 89.0
• Less conservative than the previous strategy
August, 2002 MCP
Power characteristics
• Power function
yK=PHA(p(1)1 p(2)2... p(K)K)
• Tedious to write when K>3
• Can be evaluated numerically
• Search the optimal power numerically
August, 2002 MCP
Two studies - H01/2: H01 H02 Power curve
for 1= 2=1.5, ’=0.05
August, 2002 MCP
An example
• Three studies are conducted
• Use HA1/3 , ’=0.0463
• p-value cut points are– 0.025, 0.025, 0.067
• Observed p-values were – 0.0065, 0.0125, 0.06
August, 2002 MCP
Remarks
• having the flexibility to choose p-value cut point • allows us to control decision error for multiple
studies• possible to balance variations among p-values