Discussion: Sun & Cai, JASA 2007
John PuraBIOS900
September 30, 2016
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Hypothesis Testing
Well developed in single test scenario (e.g. Neyman-PearsonLemma)Goal: maximize power (minimize Type II error) whilecontrolling Type I errorProcedure (frequentist): Compute test statistic (e.g. likelihoodratio) and compare to a single critical value, while maintainingType I error
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Multiple Hypothesis Testing
Controlling Type I error more complicated when we have manytestsCommon problem in many big-data applications - e.g. imaging,genomics, Netflix: number of tests on the order of 103-106
Goal: separate the null from the nonnulls while still balancingType I and Type II errorsPower is critical in these applications because the mostinteresting effects are usually at the edge of detection.Rich literature on the topic spanning frequentist and Bayesianapproaches
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Data Structures in Large-Scale Studies
m features (e.g. voxels, genes, movies) and k groups
Table 1: Common Data Matrix in Large Scale Studies
X1 · · · Xn1 X2 · · · Xn2 · · · Xk · · · Xnk
X11 · · · Xn11 X21 · · · Xn21 · · · Xk1 · · · Xnk1X12 · · · Xn12 X22 · · · Xn22 · · · Xk2 · · · Xnk2
......
......
X1m · · · Xn1m X2m · · · Xn2m · · · Xkm · · · Xnkm
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Multiple Testing Framework
Suppose we perform m simultaneous hypothesis tests of H0,i vs.H1,i , 1 ≤ i ≤ m under a given procedureSummarize the outcomes of such a procedure as follows:
Table 2: Classification of tested hypotheses
Claimed nonsignificant Claimed significant TotalNull N00 N10 m0
Nonnull N01 N11 m1Total S R m
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Type I Error Control
Different ways of addressing Type I errorPer comparison error rate (PCER) =E (N10/m) = Pr(N10,i > 0) ≤ α, 1 ≤ i ≤ mFamilywise error rate (FWER) = Pr(N10 > 0) = 1− (1− α)m
False-discovery rate (FDR) = E (N10/R|R > 0)Pr(R > 0) andits variants. . .
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Some Notation
α = prespecified level, 0 < α < 1Pi=p-value for testing H0,i vs. H1,i , 1 ≤ i ≤ m
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Classical Procedures to Type I Error Control
FWERBonferroni
single step procedureReject H0,i if Pi ≤ α/mguarantees FWER ≤ α
Holm (1979)stepwise procedureOrder p-values: 0 < P(1) ≤ P(2) ≤ ... ≤ P(m)Reject H0,i if P(i) ≤ α/(m − i + 1)more powerful than Bonferroni
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Classical Procedures to Type I Error Control
FDRBenjamini-Hochberg (1995)
For m independent tests, of which m0 are true, order p-valuesin decreasing order: P(m) ≥ P(m−1) ≥ ... ≥ P(1) > 0For a fixed α ∈ (0, 1), find k = maxi{P(i) ≤ m0
m α}Reject H0,(i) for all i ≤ kguarantees FDR ≤ αmore powerful than FWER control procedures (greater numberof rejections)equivalent to FWER-control when all m null hypotheses aretrue (m0 = m)works well for sparse situations (m0 ≈ m)
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Additional Notation
Data: Xi |θi ∼ (1− θi )F0 + θiF1
Target of inference: θ1, ...θmindep∼ Bern(p)
null: θi = 0, nonnull: θi = 1
Test statistic: T(x)=[Ti(x): 1 ≤ i ≤ m]Z-values, Zi = Φ−1(F (Ti )) and corresponding p-values, Pi ,1 ≤ i ≤ mDecision rule: δ = (δ1, ..., δm) ∈ I = {0, 1}mmFDR = E (N01)/E (R), mFNR = E (N01)/E (S)p = Pr(θi = 1) 1 ≤ i ≤ m
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Sun and Cai’s Approach
Compound decision view of multiple testing problem:Find δ that minimizes mFNR (efficient) while controlling mFDRat level α (valid) among all SMLR decision rules
Procedures based on thresholding ordered p-values are notgenerally efficient (e.g.δ(p(·), c) = {I[p(Xi ) < c] : 1 ≤ i ≤ m})Remark 1. Copas (1974): if a symmetric (akapermutation-invariant) classification rule for dichotomies isadmissible in a weighted classification problem, then it must beordered by the likelihood ratios
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Sun and Cai’s Approach
Spoiler Alert!Optimal solution to multiple testing problem equivalent tooptimal solution to weighted classification problem amongsymmetric/permutation-invariant decision rulesThe local FDR (Lfdr) is a fundamental quantity which can beused directly for optimal FDR control.An adaptive Lfdr procedure based on z-values unifies the goalsof global error control and individual case interpretation
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Oracle Setting - Example 1
Suppose we have m tests of H0,i : µ = 0 vs. H1,i : µ 6= 0,1 ≤ i ≤ m with corresponding z-values z1, ..., zm ∼ f (z),f (z) = p0φ(z) + p1φ(z − µ1) + p2φ(z − µ2)p0 = 0.8, p1 + p2 = 0.2, mFDR = 0.1Compare the oracle procedure (more on this in a bit) based onp-value (Genovese & Wasserman, 2002) to one based onz-value
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Oracle Setting - Example 1
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Remarks on Example 1
p-value oracle procedure (Disclaimer: for two-sidedalternatives) is dominated by z-value oracle procedure, withgreater difference in performance as p1 > p2.The Lfdr oracle procedure based on z-values may accept moreextreme (i.e. larger absolute z-value or equivalently lowerp-value) observations while rejecting less extreme ones =⇒rejection region is asymmetric
z = −2 (Lfdr = 0.227, p-value = 0.046) is rejected, but z = 3(Lfdr = 0.543, p-value = 0.003) is aceptedThis is due to the inherent properties of the Lfdr statistic
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Local FDR (Lfdr)
Inherently Bayesian ideaLfdr = Pr(null |z) = (1− p)f0(z)/f (z)Allows for local inference, in which outcomes such as z = 3 are“judged on their own terms and not with respect tohypothetical possibility of more extreme results” (Efron, 2010)
In other words, our interest in an outlying zi is judged relativeto the local feature f0/f about zi , rather than relative tofeatures of the theoretical null distribution. See Remark 4.
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Oracle Procedure
What is the optimal procedure and how does it behave if weknow p, f0, and f1?Assume SMLR property for a given test statistic. SeeProposition 1. Why is this useful?The optimal solution to the classification problem is alsooptimal in the multiple testing problem. See Theorem 1.Oracle rule:δλ(Λ, 1/λ) = [I{Λ(xi ) < 1/λ}, 1 ≤ i ≤ m] = Lfdr(·)
1−Lfdr(·) , whereΛ(·) = (1−p)f0(·)
pf1(·)
Bayes rule that minimizes the classification risk (alwayssymmetric, admissible, and ordered)
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Oracle Procedure based on z-values
Test statistic: TOR(Zi ) = (1− p)f0(Zi )/f (Zi ) ∼ GOR , whereGOR = (1− p)G0
OR + pG1OR
mFDR = (1− p)G0OR/GOR
Threshold: λOR = supt∈(0,1){QOR(t) ≤ α}Testing rule: δ(TOR , λOR) = [I{TOR(zi ) < λOR}, 1 ≤ i ≤ m]The Lfdr oracle procedure ranks the relative importance of thetest statistics according to their likelihood ratios
Rankings generally different from p-values, unless alternativedistribution is symmetric about null hypothesis
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Another Example
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Remarks - Example 2
When |µ1| = |µ2|, mFNR is constant for p-value oracleprocedure; equivalence only under symmetric alternative (a)Lfdr oracle procedure utilizes the distributional information ofthe nonnulls (i.e. adapts to asymmetry), but the p-value oracleprocedure does not (a)Under large degree of asymmetry in alternative distribution,oracle procedures differ the greatest when |µ1| < µ2 andp1 > p2 (b,c)mFNR decreases with mFDR for a given range (d)
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Adaptive Procedure
Suppose we have consistent estimators p̂0, f̂0, f̂ (e.g. Jin & Cai,2007)Define T̂OR(zi ) = min([(1− p̂)f̂0(zi )/f̂ (zi )], 1)Order the T̂OR(zi ) as ˆLfdr (1) ≤ ... ≤ ˆLfdr (m)
Let k = max1≤i≤m{1i
∑ij=1 ˆLfdr (j) ≤ α}
Reject H0,(i) for all i ≤ k ⇐⇒δ(T̂OR , λ̂OR) = [I{T̂OR(zi ) < λ̂OR}]Asymptotically efficient: asymptotically controls mFDR at levelα and attains the mFNR level achieved by the Lfdr oracleprocedure. See Theorems 4 & 5.
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Yet, More Examples
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007
Conclusion
The adaptive Lfdr procedure based on z-values is asymptoticallyoptimal, particularly under asymmetric alternative distributionThe Lfdr can be viewed as ranking the relative importance ofthe LRs, allowing for local inference and efficient separation ofnulls and nonnulls
John Pura BIOS900 Discussion: Sun & Cai, JASA 2007