false discovery rates for discrete data joseph f. heyse merck research laboratories graybill...
TRANSCRIPT
False Discovery Rates for Discrete Data
Joseph F. Heyse Merck Research Laboratories
Graybill Conference
June 13, 2008
Graybill.ppt.
Introduction Almost all multiplicity considerations in clinical trial applications are
designed to control the Family Wise Error Rate (FWER).
Benjamini and Hochberg (1995) argued that in certain settings, requiring control of the FWER is often too conservative.
They suggested controlling the “False Discovery Rate” (FDR) is a more powerful alternative.
Accounting for the discrete endpoints can further improve the power of FDR (and FWER) methods.
Graybill.ppt.
Outline
1. Definition and properties of FDR
2. FDR for discrete data
3. Application: Genetic variants of HIV
4. Summary of simulation results
5. Application: Rodent carcinogenicity study
6. Concluding remarks
Graybill.ppt.
Familywise Error Rate (FWER)
Let F = {H1,H2 … HK} denote a family of K hypotheses.
FWER = Pr(any true Hi F is rejected).
The procedures currently used for clinical studies are intended to control the FWER .
Benjamini & Hochberg (1995) proposed controlling the “False Discovery Rate” (FDR) as a more powerful alternative to FWER.
Graybill.ppt.
Declared
Insignificant
Declared Significant
Total
# of true Hi U V K0
# of false Hi T S K K0
Total K R R K
False Discovery Rate (FDR)(Benjamini & Hochberg, 1995)
When R=0, FDR is defined to be 0.
rejected.y incorrectl are whichhypotheses
null rejected of proportion expected RV
EFDR
Graybill.ppt.
False Discovery Rate (FDR) (cont’d)
(Benjamini & Hochberg)
Kj
P if H,,H,H jectRe jj21
1Kj,PjK
,P~
minP~
PP~ :values-P Adjusted
j1jj
KK
Unadjusted p-values .0193 .0280 .2038 .4941
FDR-adjusted p-values .0560 .0560 .2718 .4941
Example (K=4)
Graybill.ppt.
Properties of FDR Control The B&H sequential procedure controls the FDR at
FDR < FWER and equality holds if K=K0.
The Hochberg (1988) stepwise procedure compares
while the FDR procedure compares
FDR is potentially more powerful than FWER controlling
procedures.
KK0
j-1K to P j
.Kj to P j
for independent hypotheses.
Graybill.ppt.
Comparing FDR and FWER
FDR adjusted P-values < FWER adjusted P-values
Example (K=4)
Unadjusted P-values
FDR-adjusted P-values
FWER-adjusted P-values
.0193
.0560
.0772
.0280
.0560
.0840
.2038
.2718
.4076
.4941
.4941
.4941
Graybill.ppt.
Modified FDR for Discrete Data
Adjusted P-values for FDR
For discrete data
Where is largest P-value achievable for hypothesis i that is less than or equal to P(j).
jj Pj
KP ][~
*iP
jPPK
iij
1
*][
~
Graybill.ppt.
FDR for Discrete Data
Gain in power for the discrete data FDR comes from the difference .
If endpoint i is not able to achieve a P-value ≤ P(j) then and the dimensionality is reduced.
If endpoint i is able to achieve a P-value ≤ P(j) then and a smaller quantity adds to .
*)( iPP j
0* iP
)(*
ji PP ][
~jP
Graybill.ppt.
Other Approaches for Discrete Data
Tarone (1990) proposed a modified Bonferroni procedure for discrete data by removing those endpoints unable to reach that level of statistical significance.
Gilbert (2005) proposed a 2 step FDR method for discrete data.
1. Apply Tarone’s method to identify endpoints suitable for adjustment.
2. Apply B-H FDR to those endpoints.
Calculating the FDR adjusted P-value is expected to improve upon these approaches by using the complete exact distribution.
][
~jP
Graybill.ppt.
Example: Genetic Variants of HIV
• Gilbert (2005) compared the mutation rates at 118 positions in HIV amino-acid sequences of 73 patients with subtype C to 73 patients with subtype B.
• The B-H FDR procedure identified 12 significant positions.
• The Tarone modified FDR procedure reduced the dimensionality to 25 and identified 15 significant positions.
• The fully discrete FDR identified 20 significant positions.
Graybill.ppt.
Simulation Study for Independent Hypotheses
A simulation study was conducted to evaluate the statistical properties of the FDR controlling methods for discrete data using Fisher’s Exact Test.
Simulation parameters– Number of Hypotheses: K = 5, 10, 15, 20
– Varying numbers of false hypotheses (K-K0)
– Background rates chosen randomly from U(.01, .5)– Odds Ratios for Effect Size: OR = 1.5, 2, 2.5, 3– Sample sizes: N = 10, 25, 50, 100 = 0.05 1-Tailed
Graybill.ppt.
Rate of Rejecting True Hypotheses When All Hypotheses are True
(K0=K)
5 10 15 20
0.01
0.02
0.03
0.04
0.05(N=25)
Discrete FDRT-G FDRB-H FDR
5 10 15 20
0.01
0.02
0.03
0.04
0.05(N=100)
No. of Hypotheses (K)
Graybill.ppt.
Rate of Rejecting True Hypotheses When Some Hypotheses are False
(K0<K)
1 2 3 4 5 6 7 8 9
0.01
0.02
0.03
0.04
0.05(K=10, N=25, OR=2.5)
Discrete FDRT-G FDRB-H FDR Reference Level
No. of True Hypotheses (K0)
1 2 3 4 5 6 7 8 9
0.01
0.02
0.03
0.04
0.05(K=10, N=100, OR=2.5)
Graybill.ppt.
Rate of Rejecting False Hypotheses
1.5 2.0 2.5 3.0
0.2
0.4
0.6
0.8
1.0(K=10, K-K0=4, N=25)
Discrete FDRT-G FDRB-H FDR
1.5 2.0 2.5 3.0
0.2
0.4
0.6
0.8
1.0(K=10, K-K0=4, N=100)
Odds Ratio
Graybill.ppt.
Other Applications
Analysis of Tier II clinical trial adverse experiences
Trend test analysis of rodent carcinogenicity data
Similar modification applied to Bonferonni adjustment for discrete endpoints
Graybill.ppt.
Rodent Carcinogenicity Studies
Long-term carcinogenicity studies typically test candidate drugs in several graded doses and use a vehicle control group.
50 male and 50 female rodents are randomly assigned to each drug treated group with 100 rodents of each sex assigned to control. Male and female studies are considered separately.
Treatment is administered daily and a terminal necropsy is performed at the end of the study.
Graybill.ppt.
Rodent Carcinogenicity Studies (cont’d)
Each individual tumor site encountered is described by a combination of organ or tissue with tumor type.
A statistical analysis of trend is performed for all tumor sites encountered.
An exact test uses the permutation distribution of the trend statistic.
Exact tests can account for age at necropsy and lethality of tumor.
For illustration purposes, this analysis only considered dose and presence of tumor.
Graybill.ppt.
Summary of Statistical Results from a Long-Term Carcinogenicity Study in Male Mice
Tumor Site
Liver, Hepatocellular CarcinomaP.S.U., HemangiosarcomaAdrenal Cortex, AdenomaP.S.U., SarcomaP.S.U., LymphomaLung, AdenomaLiver, Hepatocellular AdenomaLiver, HemangiosarcomaHarderian Gland, AdenomaSkin, FibromaThyroid, Follicular Cell CarcinomaP.S.U., LeukemiaLung, AdenocarcinomaTestes, Interstitial Cell TumorStomach, Papilloma
Number of Mice on Study
Control0
10007
1612
22115012
100
2
010048600002310
50
4
100016700112010
50
8
31116
11611002000
50
TrendP-value
.0342
.20
.20
.20
.24
.24
.49
.50
.50
.50
.50
.44N
.41N
.41N
.16N
Trend P-value is reported 1-tailed using exact permutational distribution. N indicates 1-tailed P-value for negative trend.
Test Agent Dose
Graybill.ppt.
Available Methods
Adjusting P-values to account for multiple tumor types. (Mantel 1980, Brown and Fears 1981, Mantel et al. 1982)
Adapting for interpreting unadjusted P-values (Haseman 1983 and 1990, Lin and Rahman 1998)
Resampling methods to adjust P-values (Heyse and Rom 1988, Westfall and Young 1989, Westfall and Soper 1998)
Bayesian methods using historical control priors (Westfall and Soper 2001)
Graybill.ppt.
Multiplicity of Statistical Tests
Liver, hepatocellular carcinoma was only 1 of K=15 tumor sites encountered.
P(1)=0.0342 was the most extreme individual trend P-
value.
Interest is in the likelihood of observing P(1)=0.0342 as
the most extreme P-value among the K=15 in this study.
Need to consider the discrete nature of the data since several tumor sites may not be able to achieve significance levels of P(1).
Graybill.ppt.
P-value Adjustment Methods
Mantel (1980) attributed to J.W. Tukey
Where k=number of tumor sites that could yield P-values as extreme as P(1).
Mantel et al. (1982)
Where is largest P-value achievable for tumor site i that is less than or equal to P(1).
Discrete FDR adjustment: = 0.264
268.0P11P k1]1[
*iP
235.0P11P *i
K
1i
*]1[
]1[P~
Graybill.ppt.
Conclusions
FDR control provides higher power than FWER control when some hypotheses are false.
Proposed procedure based on exact analysis of binomial data controls FDR at
Discrete nature of data results in slightly conservative FDR control. FDR is less conservative for increasing sample size and increasing numbers of hypotheses.
Accounting for discrete endpoints increases the power of FDR.