controlling the familywise error rate in functional...

Controlling the familywise error rate in functionalneuroimaging a comparative reviewThomas Nichols and Satoru Hayasaka Department of Biostatistics University ofMichigan Ann Arbor MI USA

Functional neuroimaging data embodies a massive multiple testing problem where 100 000 correlated teststatistics must be assessed The familywise error rate the chance of any false positives is the standardmeasure of Type I errors in multiple testing In this paper we review and evaluate three approaches tothresholding images of test statistics Bonferroni random eld and the permutation test Owing to recentdevelopments improved Bonferroni procedures such as Hochbergrsquos methods are now applicable todependent data Continuous random eld methods use the smoothness of the image to adapt to the severityof the multiple testing problem Also increased computing power has made both permutation andbootstrap methods applicable to functional neuroimaging We evaluate these approaches on t imagesusing simulations and a collection of real datasets We nd that Bonferroni-related tests offer littleimprovement over Bonferroni while the permutation method offers substantial improvement over therandom eld method for low smoothness and low degrees of freedom We also show the limitations oftrying to nd an equivalent number of independent tests for an image of correlated test statistics

1 Introduction

Functional neuroimaging refers to an array of technologies used to measure neuronalactivity in the living brain Two widely used methods positron emission tomography(PET) and functional magnetic resonance imaging (fMRI) both use blood ow as anindirect measure of brain activity An experimenter images a subject repeatedly underdifferent cognitive states and typically ts a massively univariate model That is aunivariate model is independently t at each of hundreds of thousands of volumeelements or voxels Images of statistics are created that assess evidence for anexperimental effect Naive thresholding of 100 000 voxels at a ˆ 5 threshold isinappropriate since 5000 false positives would be expected in null data

False positives must be controlled over all tests but there is no single measure of TypeI error in multiple testing problems The standard measure is the chance of any Type Ierrors the familywise error rate (FWE) A relatively new development is the falsediscovery rate (FDR) error metric the expected proportion of rejected hypotheses thatare false positives FDR-controlling procedures are more powerful then FWE proce-dures yet still control false positives in a useful manner We predict that FDR may sooneclipse FWE as the most common multiple false positive measure In light of this webelieve that this is a choice moment to review FWE-controlling measures (We prefer the

Address for correspondence Thomas Nichols Department of Biostatistics University of Michigan AnnArbor MI 48109 USA E-mail nicholsumichedu

Statistical Methods in Medical Research 2003 12 419^446

Arnold 2003 1011910962280203sm341ra

term multiple testing problem over multiple comparisons problem lsquoMultiple compar-isonsrsquo can allude to pairwise comparisons on a single model whereas in imaging a largecollection of models is each subjected to a hypothesis test)

In this paper we attempt to describe and evaluate all FWE multiple testing proceduresuseful for functional neuroimaging Owing to the spatial dependence of functionalneuroimaging data there are actually quite a small number of applicable methods Theonly methods that are appropriate under these conditions are Bonferroni randomeld methods and resampling methods We limit our attention to ndingFWE-corrected thresholds and P-values for Gaussian Z and Student t images(An a0 FWE-corrected threshold is one that controls the FWE at a0 while an FWE-corrected P-value is the most signicant a0 corrected threshold such that a test can berejected) In particular we do not consider inference on size of contiguous suprathres-hold regions or clusters We focus particular attention on low degrees-of-freedom (DF) timages as these are unfortunately common in group analyses The typical images havetens to hundreds of thousands of tests where the tests have complicated though usuallypositive dependence structure

In 1987 Hochberg and Tamhane wrote lsquoThe question of which error rate tocontrol has generated much discussion in the literaturersquo1 While that could be truein the statistics literature in the decades before their book was published the samecannot be said of neuroimaging literature When multiple testing has been acknowl-edged at all the familywise error rate has usually been assumed implicitly For exampleof the two papers that introduced corrected inference methods to neuroimaging onlyone explicitly mentions familywise error2 3 It is hoped that the introduction of FDRwill enrich the discussion of multiple testing issues in the neuroimaging literature

The remainder of the paper is organized as follows We rst introduce the generalmultiple comparison problem and FWE We then review Bonferroni-related methodsfollowed by random eld methods and then resampling methods We then evaluatethese different methods with 11 real datasets and simulations

2 Multiple testing background in functional neuroimaging

In this section we formally dene strong and weak control of familywise error as wellas other measures of false positives in multiple testing We describe the relationshipbetween the maximum statistic and FWE and also introduce step-up and step-downtests and the notion of equivalent number of independent tests

21 NotationConsider image data on a two- (2D) or three-dimensional (3D) lattice Usually the

lattice will be regular but it may also be a irregular corresponding to a 2D surfaceThrough some modeling process we have an image of test statistics T ˆ Ti Let Ti bethe value of the statistic image at spatial location i i 2 V ˆ 1 V where V is thenumber of voxels in the brain Let the H ˆ Hi be a hypothesis image where Hi ˆ 0indicates that the null hypothesis holds at voxel i and Hi ˆ 1 indicates that thealternative hypothesis holds Let H0 indicate the complete null case where Hi ˆ 0 forall i A decision to reject the null for voxel i will be written HHi ˆ 1 not rejecting HHi ˆ 0

420 T Nichols et al

Write the null distribution of Ti as F0Ti and let the image of P-values be P ˆ Pi We

require nothing about the modeling process or statistic other than the test beingunbiased and for simplicity of exposition we will assume that all distributions arecontinuous

22 Measures of false positives in multiple testingA valid a-level test at location i corresponds to a rejection threshold u where

PTi para ujHi ˆ 0 micro a The challenge of the multiple testing problems to nd a thresholdu that controls some measure of false positives across the entire image While FWE is thefocus of this paper it is useful to compare its denition with other measures To this endconsider the cross-classication of all voxels in a threshold statistic image of Table 1

In Table 1 V is the total number of voxels tested Vcentj1 ˆP

i Hi is the number of voxelswith a false null hypothesis Vcentj0 ˆ V iexcl

Pi Hi the number true nulls and V1jcent ˆ

Pi HHi

is the number of voxels above threshold V0jcent ˆ V iexclP

i HHi the number below V1j1 isthe number of suprathreshold voxels correctly classied as signal V1j0 the numberincorrectly classied as signal V0j0 is the number of subthreshold voxels correctlyclassied as noise and V0j1 the number incorrectly classied as noise

With this notation a range of false positive measures can be dened as shown inTable 2 An observed familywise error (oFWE) occurs whenever V1j0 is greater thanzero and the familywise error rate (FWE) is dened as the probability of this event Theobserved FDR (oFDR) is the proportion of rejected voxels that have been falselyrejected and the expected false discovery rate (FDR) is dened as the expected value ofoFDR Note the contradictory notation an lsquoobserved familywise errorrsquo and thelsquoobserved false discovery ratersquo are actually unobservable since they require knowledgeof which tests are falsely rejected The measures actually controlled FWE and FDRare the probability and expectation of the unobservable oFWE and oFDR respectivelyOther variants on FDR have been proposed including the positive FDR (pFDR) theFDR conditional on at least one rejection4 and controlling the oFDR at some speciedlevel of condence (FDRc)5 The per-comparison error rate (PCE) is essentially thelsquouncorrectedrsquo measure of false positive which makes no accommodation for multi-plicity while the per-family error rate (PFE) measures the expected count of falsepositives Per-family errors can also be controlled with some level condence (PFEc)6

This taxonomy demonstrates that there are many potentially useful multiple falsepositive metrics but we are choosing to focus on but one

Table 1 Cross-classi cation of voxels in a threshold statistic image

Null not rejected(declared inactive)

Null rejected(declared active)

Null true (inactive) V0j0 V1j0 V 7 Vcentj1Null false (active) V0j1 V1j1 Vcentj1

V 7 V1jcent V1jcent V

Familywise error methods in neuroimaging 421

There are two senses of FWE control weak and strong Weak control of FWE onlyrequires that false positives are controlled under the complete null H0

P[

i2VTi para u

shyshyshyshyH0

Aacute

micro a0 (1)

where a0 is the nominal FWE Strong control requires that false positives are controlledfor any subset V0 raquo V where the null hypothesis holds

P[

i2V0

Ti para ushyshyshyshyHi ˆ 0 i 2 V0

Aacute

micro a0 (2)

Signicance determined by a method with weak control only implies that H0 is falseand does not allow the localization of individual signicant voxels Because of this testswith only weak control are sometimes called lsquoomnibusrsquo tests Signicance obtained by amethod with strong control allows rejection of individual His while controlling theFWE at all nonsignicant voxels This localization is essential to neuroimaging and inthe rest of this document we focus on strong control of FWE and we will omit thequalier unless needed

Note that variable thresholds ui could be used instead of a common threshold uAny collection of thresholds ui can be used as long as the overall FWE is controlledAlso note that FDR controls FWE weakly Under the complete null oFDR becomes anindicator for an oFWE and the expected oFDR exactly the probability of an oFWE

23 The maximum statistic and FWEThe maximum statistic MT ˆ maxi Ti plays a key role in FWE control The

connection is that one or more voxels will exceed a threshold if and only if themaximum exceeds the threshold

[

i

Ti para u ˆ MT para u (3)

Table 2 Different measures of false positives in the multiple testing problem IA is theindicator function for event A

Measure of false positives Abbreviation De nition

Observed familywise error oFWE V1j0 gt 0Familywise error rate FWE PoFWEObserved false discovery rate oFDR V1j0=V1jcent IV1jcentgt 0Expected false discovery rate FDR EoFDRPositive false discovery rate pFDR EoFDRjV1jcent gt 0False discovery rate con dence FDRc PoFDR micro aPer-comparison error rate PCE EV1j0=VPer-family error rate PFE EV1j0Per-family error rate con dence PFEc PV1j0 micro a

422 T Nichols et al

This relationship can be used to directly obtain a FWE-controlling threshold from thedistribution of the maximum statistic under H0 To control FWE at a0 letua0

ˆ Fiexcl1MT jH0

(1 iexcl a0 ) the (1 iexcl a0 )100th percentile of the maximum distribution underthe complete null hypothesis Then ua0

has weak control of FWE

P[

i

Ti para ua0shyshyshyshyH0

Aacute ˆ P(MT para ua0

jH0 )

ˆ 1 iexcl FMT jH0(ua0

)

ˆ a0

Further this ua0also has strong control of FWE although an assumption of subset

pivotality is required7 A family of tests has subset pivotality if the null distribution of asubset of voxels does not depend on the state of other null hypotheses Strong controlfollows

P[

i2V0

Ti para ua0shyshyshyshyHi ˆ 0 i 2 V0

Aacute

ˆ P[

i2V0


Aacute

(4)

micro P[

i2VTi para ua0

shyshyshyshyH0

Aacute

(5)

ˆ a0 (6)

where the rst equality uses the assumption of subset pivotality In imaging subsetpivotality is trivially satised as the image of hypotheses H satises the free combinationcondition That is there are no logical constraints between different voxelrsquos null hypoth-eses and all combinations of voxel level nulls (01V ) are possible Situations where subsetpivotality can fail include tests of elements of a correlation matrix (Ref 7 p 43)

Note that we could equivalently nd P-value thresholds using the distribution of theminimum P-value NP ˆ mini Pi Whether with MT or NP we stress that we are notsimply making inference on the extremal statistic but rather using its distribution tond a threshold that controls FWE strongly

24 Step-up and step-down testsA generalization of the single threshold test takes the form of multi-step tests Multi-

step tests consider the sequence of sorted P-values and compare each P-value to adifferent threshold Let the ordered P-values be P(1) micro P(2) micro cent cent cent micro P(V) and H(i) be thenull hypothesis corresponding to P(i) Each P-value is assessed according to

P(i) micro ui (7)

Usually u1 will correspond to a standard xed threshold test (eg Bonferroniu1 ˆ a0=V)


There are two types of multi-step tests step-up and step-down A step-up testproceeds from the least to most signicant P-value (P(V) P(Viexcl1) ) and successivelyapplies equation (7) The rst i0 that satises (7) implies that all smaller P-values aresignicant that is HH(i) ˆ 1 for all i micro i0 HH(i) ˆ 0 otherwise A step-down test proceedsfrom the most to least signicant P-value (P(1) P(2) ) The rst i0 that does not satisfy(7) implies that all strictly smaller P-values are signicant that is HH(i) ˆ 1 for all i lt i0HH(i) ˆ 0 otherwise For the same critical values ui a step-up test will be as or morepowerful than a step-down test

25 Equivalent number of independent testsThe main challenge in FWE control is dealing with dependence Under independence

the maximum null distribution FMT jH0is easily derived

FMT(t) ˆ

Y

i

FTi(t) ˆ FV

T1(t) (8)

where we have suppressed the H0 subscript and the last equality comes from assuminga common null distribution for all V voxels However in neuroimaging data inde-pendence is rarely a tenable assumption as there is usually some form of spatialdependence either due to acquisition physics or physiology In such cases the maximumdistribution will not be known or even have a closed form The methods describedin this paper can be seen as different approaches to bounding or approximating themaximum distribution

One approach that has an intuitive appeal but which has not been a fruitful avenueof research is to nd an equivalent number of independent observations That is to nda multiplier y such that

FMT(t) ˆ FyV

T1(t) (9)

or in terms of P-values

FNP(t) ˆ 1 iexcl (1 iexcl FP(1)

(t))yV

ˆ 1 iexcl (1 iexcl t)yV (10)

where the second equality comes from the uniform distribution of P-values under thenull hypothesis We are drawn to the second form for P-values because of its simplicityand the log-linearity of 1 iexcl FNP

(t)For the simulations considered below we will assess if minimum P-value distribu-

tions follow equation (10) for a small t If they do one can nd the effective number oftests yV This could be called the number of maximum-equivalent elements in the dataor maxels where a single maxel would consist of V=(yV) ˆ yiexcl1 voxels

424 T Nichols et al

3 FWE methods for functional neuroimaging

There are two broad classes of FWE methods those based on the Bonferroni inequalityand those based on the maximum statistic (or minimum P-value) distribution We rstdescribe Bonferroni-type methods and then two types of maximum distributionmethods random eld theory-based and resampling-based methods

31 Bonferroni and relatedThe most widely known multiple testing procedure is the lsquoBonferroni correctionrsquo It is

based on the Bonferroni inequality a truncation of Boolersquos formula1 We write theBonferroni inequality as

Praquo [

i

Ai

frac14micro

X

i

PAi (11)

where Ai corresponds to the event of test i rejecting the null hypothesis when true Theinequality makes no assumption on dependence between tests although it can be quiteconservative for strong dependence As an extreme consider that V tests with perfectdependence (Ti ˆ Ti0 for i 6ˆ i0) require no correction at all for multiple testing

However for many independent tests the Bonferroni inequality is quite tight fortypical a0 For example for V ˆ 323 independent voxels and a0 ˆ 005 the exactone-sided P-value threshold is 1 iexcl ((1 iexcl a0 )1=V ) ˆ 15653 pound 10iexcl6 while Bonferronigives a0=V ˆ 15259 pound 10iexcl6 For a t9 distribution this is the difference between101616 and 101928 Surprisingly for weakly-dependent data Bonferroni can alsobe fairly tight To preview the results below for 323-voxel t9 statistic image based onGaussian data with isotropic FWHM smoothness of three voxels we nd that thecorrect threshold is 100209 Hence Bonferroni thresholds and P-values can indeed beuseful in imaging

Considering another term of Boolersquos formula yields a second-order Bonferroni or theKounias inequality1

Praquo [

i

Ai

frac14micro

X

i

PAi iexcl maxkˆ1V

raquo X

i 6ˆk

PAi Akfrac14

(12)

The Slepian or DunnndashSNidak inequalities can be used to replace the bivariate probabil-ities with products The Slepian inequality also known as the positive orthantdependence property is used when Ai corresponds to a one-sided test ndash it requiressome form of positive dependence like Gaussian data with positive correlations8

DunnndashSNidak is used for two-sided tests and is a weaker condition for example onlyrequiring the data follow a multivariate Gaussian t or F distribution (Ref 7 p 45)

If all the null distributions are identical and the appropriate inequality holds (Slepianor DunnndashSNidak) the second-order Bonferroni P-value threshold is c such that

Vc iexcl (V iexcl 1)c2 ˆ a0 (13)


When V is large however c will have to be quite small making c2 negligible essentiallyreducing to the rst-order Bonferroni For the example considered above with V ˆ 323

and a0 ˆ 005 the Bonferroni and Kounias P-value thresholds agree to ve decimalplaces (005=V ˆ 1525881 pound 10iexcl6 versus c ˆ 1525879 pound 10iexcl6)

Other approaches to extending the Bonferroni method are step-up or step-down testsA Bonferroni step-down test can be motivated as follows If we compare the smallestP-value P(1) to a0=V and reject then our multiple testing problem has been reduced byone test and we should compare the next smallest P-value P(2) to a0=(V iexcl 1) In generalwe have

P(i) micro a01

V iexcl i Dagger 1(14)

This yields the Holm step-down test9 which starts at i ˆ 1 and stops as soon as theinequality is violated rejecting all tests with smaller P-values Using the very samecritical values the Hochberg step-up test1 0 starts at i ˆ V and stops as soon as theinequality is satised rejecting all tests with smaller P-values However the Hochbergtest depends on a result of Simes

Simes1 1 proposed a step-up method that only controlled FWE weakly and was onlyproven to be valid under independence In their seminal paper Benjamini andHochberg1 2 showed that Simesrsquo method controlled what they named the lsquoFalseDiscovery Ratersquo Both Simesrsquo test and Benjamini and Hochbergrsquos FDR have the form

P(i) micro a0iV

(15)

Both are step-up tests which work from i ˆ V The Holm method like Bonferroni makes no assumption on the dependence of the

tests But if the Slepian or DunnndashSNidak inequality holds the lsquoSNidak improvement onHolmrsquo can be used1 3 The SNidak method is also a step-down test but uses thresholdsui ˆ 1 iexcl (1 iexcl a0)1=(ViexcliDagger1) instead

Recently Benjamini and Yekutieli1 4 showed that the Simes=FDR method is validunder lsquopositive regression dependency on subsetsrsquo (PRDS) As with Slepian inequalityGaussian data with positive correlations will satisfy the PRDS condition but it is morelenient than other results in that it only requires positive dependency on the null that isonly between test statistics for which Hi ˆ 0 Interestingly since Hochbergrsquos methoddepended on Simesrsquo result so Ref 14 also implies that Hochbergrsquos step-up method isvalid under dependence

Table 3 summarizes the multi-step methods The Hochberg step-up method andSNidak step-down method appear to be the most powerful Bonferroni-related FWEprocedures available for functional neuroimaging Hochberg uses the same criticalvalues as Holm but Hochberg can only be more powerful since it is a step-up test TheSNidak has slightly more lenient critical values but may be more conservative thanHochberg because it is a step-down method If positive dependence cannot beassumed for one-sided tests Holmrsquos step-down method would be the best alternativeThe Simes=FDR procedure has the most lenient critical values but only controls FWE

426 T Nichols et al

weakly See works by Sarkar1 5 1 6 for a more detailed overview of recent developmentsin multiple testing and the interaction between FDR and FWE methods

The multi-step methods can adapt to the signal present in the data unlike BonferroniFor the characteristics of neuroimaging data with large images with sparse signalshowever we are not optimistic these methods will offer much improvement overBonferroni For example with a0 ˆ 005 and V ˆ 323 consider a case where 3276voxels (10) of the image have a very strong signal that is innitesimal P-values Forthe 3276th smallest P-value the relevant critical value would be 005=(V iexcl 3276 Dagger 1) ˆ16953 pound 10iexcl6 for Hochberg and 1 iexcl (1 iexcl 005)1=(Viexcl3276Dagger1) ˆ 17392 pound 10iexcl6 for SNidakeach only slightly more generous than the Bonferroni threshold of 15259 pound 10iexcl6 and adecrease of less than 016 in t9 units For more typical even more sparse signals therewill be less of a difference (Note that the Simes=FDR critical value would be005 pound 3276=V ˆ 0005 although with no strong FWE control)

The strength of Bonferroni and related methods are their lack of assumptions or onlyweak assumptions on dependence However none of the methods makes use of thespatial structure of the data or the particular form of dependence The following twomethods explicitly account for image smoothness and dependence

32 Random centeld theoryBoth the random eld theory (RFT) methods and the resampling-based methods

account for dependence in the data as captured by the maximum distribution Therandom eld methods approximate the upper tail of the maximum distribution the endneeded to nd small a0 FWE thresholds In this section we review the general approachof the random eld methods and specically use the Gaussian random eld results togive intuition to the methods Instead of detailed formulae for every type of statisticimage we motivate the general approach for thresholding a statistic image and thenbriey review important details of the theory and common misconceptions

For a detailed description of the random eld results we refer to a number of usefulpapers The original paper introducing RFT to brain imaging2 is a very readable andwell-illustra ted introduction to the Gaussian random eld method A later work1 7

unies Gaussian and t w2 and F eld results A very technical but comprehensivesummary1 8 also includes results on Hotellings T2 and correlation elds As part of abroad review of statistica l methods in neuroimaging Ref 19 describes RFT methodsand highlights their limitations and assumptions

Table 3 Summary of multi-step procedures

Procedure ui Type FWE control Assumptions

Holm a0(1=V 7 i Dagger 1) Step-down Strong NoneSN idak 1 7 (1 7 a0)1=(V 7 i Dagger 1) Step-down Strong Slepian=DunnndashSN idakHochberg a0(1=V 7 i Dagger 1) Step-up Strong PRDSSimes=FDR a0(i=V) Step-up Weak PRDS


321 RFT intuitionConsider a continuous Gaussian random eld Z(s) dened on s 2 O raquo D where D

is the dimension of the process typically 2 or 3 Let Z(s) have zero mean and unitvariance as would be required by the null hypothesis For a threshold u applied to theeld the regions above the threshold are known as the excursion set Au ˆO s Z(s) gt u The Euler characteris tic w(Au) sup2 wu is a topological measure of theexcursion set While the precise denition involves the curvature of the boundary of theexcursion set (Ref 20 cited in Ref 21) it essentially counts the number of connectedsuprathreshold regions or lsquoclustersrsquo minus the number of lsquoholesrsquo plus the number oflsquohollowsrsquo (Figure 1) For high thresholds the excursion set will have no holes or hollowsand wu will just count the number of clusters for yet higher thresholds the wu will beeither 0 or 1 an indicator of the presence of any clusters This seemingly esoterictopological measure is actually very relevant for FWE control If a null Gaussianstatistic image T approximates a continuous random eld then

FWE ˆ PoFWE (16)

ˆ P[

i

Ti para u

( )

(17)

ˆ P maxi

Ti para u (18)

ordm Pwu gt 0 (19)

ordm Ewu (20)

The rst approximation comes from assuming that the threshold u is high enough forthere to be no holes or hollows and hence the wu is just counting clusters The secondapproximation is obtained when u is high enough such that the probability of two ormore clusters is negligible

Figure 1 Illustration of Euler characteristic (wu) for different thresholds u The left gure shows the random eld and the remaining images show the excursion set for different thresholds The wu illustrated here onlycounts clusters minus holes (neglecting hollows) For high thresholds there are no holes and wu just countsthe number clusters

428 T Nichols et al

The expected value of the wu has a closed-form approximation2 1 for D ˆ 3

Ewu ordm l(O)jLj1=2 (u2 iexcl 1) exp( iexcl u2=2)=(2p)2 (21)

where l(O) is the Lesbegue measure of the search region the volume in three dimen-sions and L is the variancendashcovariance matrix of the gradient of the process

L ˆ Varq

qxZ

qqy

Zqqz

Zmicro paragt

Aacute (22)

Its determinant jLj is measure of roughness the more lsquowigglyrsquo a process the morevariable the partial derivatives the larger the determinant

Consider an observed value z of the process at some location To build intuitionconsider the impact of search region size and smoothness on corrected P-value Pc

z Thecorrected P-value is the upper tail area of the maximum distribution

Pcz ˆ P( max

sZ(s) para z) ordm Ewz (23)

For large z equation (21) gives

Pcz l(O)jLj1=2z2 exp iexcl z2

2

sup3 acute(24)

approximately First note that all other things constant increasing large z reduces thecorrected P-value Of course P-values must be nonincreasing in z but note thatequation (24) is not monotonic for all z and that Ewz can be greater than 1 or evennegative Next observe that increasing the search region l(O) increases the correctedP-value decreasing signicance This should be anticipated since an increased searchvolume results in a more severe multiple testing problem And next consider the impactof smoothness the inverse of roughness Increasing smoothness decreases jLj which inturn decreases the corrected P-value and increases signicance The intuition here is thatan increase in smoothness reduces the severity of the multiple testing problem in somesense there is less information with greater smoothness In particular consider that inthe limit of innite smoothness the entire processes has a common value and there is nomultiple testing problem at all

322 RFT strengths and weaknessesAs presented above the merit of the RFT results are that they adapt to the volume

searched (like Bonferroni) and to the smoothness of the image (unlike Bonferroni)When combined with the general linear model (GLM) the random eld methodscomprise a exible framework for neuroimaging inference For functional neuroima-ging data that is intrinsica lly smooth (PET SPECT MEG or EEG) or heavily smoothed(multisubject fMRI) these results provide a unied framework to nd FWE-correctedinferences for statistic images from a GLM While we only discuss results for peak


statistic height a family of available results includes P-values for the size of a cluster thenumber of clusters and joint inference on peak height and cluster size

Further the results only depend on volume searched and the smoothness (see belowfor more details on edge corrections) and are not computationally burdensomeFinally they have been incorporated into various neuroimaging software packagesand are widely used (if poorly understood) by thousands of users (The softwarepackages include SPM httpwwwlionuclacuk VoxBo wwwvoxboorg FSLwwwfmriboxacukfsl and Worsleyrsquos own fmristat httpwwwmathmcgillcakeithfmristat)

The principal weakness of the random eld methods are the assumptions Theassumptions are sometimes misstated so we carefully itemize them

1) The multivariate distribution of the image is Gaussian or derived from Gaussiandata (eg for t or F statistic image)

2) The discretely sampled statistic images are assumed to be sufciently smooth toapproximate the behaviour of a continuous random eld The recommended ruleof thumb is three voxels FWHM smoothness1 9 although we will critically assessthis with simulations and data (see below for denition of FWHM smoothness)

3) The spatial autocorrelation function (ACF) must have two derivatives at the originExcept for the joint cluster-peak height test2 2 the ACF is not assumed to have theform of a Gaussian density

4) The data must be stationary or there must exist a deformation of space such thatthe transformed data is stationary2 3 This assumption is most questionable forreconstructed MEG and EEG data which may have very convoluted covariancestructures Remarkably for the peak height results we discuss here nonstationarityneed not even be estimated (see below for more)

5) The results assume that roughness=smoothness is known with negligible errorPoline et al2 4 found that uncertainty in smoothness was in fact appreciablecausing corrected P-values to be accurate to only sect 20 if smoothness wasestimated from a single image (In a recent abstract Worsley proposedmethods for nonstationary cluster size inference which accounts for variabilityin smoothness estimation2 5 )

323 RFT essential detailsTo simplify the presentation the results above avoided several important details

which we now reviewRoughness and RESELS Because the roughness parameter jLj1=2 lacks interpret-

ability Worsley proposed a reparameterization in terms of the convolution of a whitenoise eld into a random eld with smoothness that matches the data

Consider a white noise Gaussian random eld convolved with a Gaussian kernel Ifthe kernel has variance matrix S then the roughness of the convolved random eld isL ˆ Siexcl1=22 6 If S has s2

x s2y s2

z on the diagonal and zero elsewhere then

jLj1=2 ˆ (jSjiexcl12iexcl3 )1=2 (25)

ˆ (sxsysz)iexcl12iexcl3=2 (26)

430 T Nichols et al

To parameterize in terms of smoothnessWorsley used the full width at half maximum(FWHM) a general measure of spread of a density For a symmetric density f centeredabout zero FWHM is the value x such that f ( iexcl x=2) ˆ f (x=2) ˆ f (0)=2 A Gaussiankernel has a FWHM of s

8 log 2

p If a white noise eld is convolved with a Gaussian

kernel with scale (FWHMx FWHMy FWHMz) (and zero correlations) the roughnessparameter is

jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)

Worsley dened a RESolution ELement or RESEL to be a spatial element withdimensions FWHMx pound FWHMy pound FWHMz The denominator of (27) is then thevolume of one RESEL

Noting that Ewu in equation (21) depends on the volume and roughnessthrough l(O)jLj1=2 it can be seen that search volume and RESEL size can be combinedand instead written as the search volume measured in RESELs

R3 ˆ l(O)FWHMxFWHMyFWHMz

(28)

The RFT results then depend only on this single quantity a resolution-adjustedsearch volume the RESEL volume The essential observation was that jLj1=2 can beinterpreted as a function of FWHM the scale of a Gaussian kernel required to convolvea white noise eld into one with the same smoothness as the data When jLj1=2 isunknown it is estimated from the data (see below) but not by assuming that the ACF isGaussian A Gaussian ACF is not assumed by random eld results rather GaussianACF is only used to reparameterize roughness into interpretable units of smoothness Ifthe true ACF is not Gaussian the accuracy of the resulting threshold is not impactedonly the precise interpretation of RESELs is disturbed

Component elds and smoothness estimation For the Gaussian case presentedabove the smoothness of the statistic image under the null hypothesis is the keyparameter of interest For results on other types of elds including t w2 and F thesmoothness parameter describes the smoothness of the component elds If each voxelrsquosdata are t with a general linear model Y ˆ Xb Dagger E the component elds are images ofEj=

VarEj

p where Ej is scan jrsquos error That is the component elds are the unobserv-

able mean zero unit variance Gaussian noise images that underlie the observed dataEstimation of component eld smoothness is performed on standardized residual

images2 7 not the statistic image itself The statistic image is not used because it willgenerally contain signal increasing roughness and decreasing estimated smoothnessAdditionally except for the Z statistic the smoothness of the null statistic image will bedifferent from that of the component elds For example see Ref 21 Equation (7) andRef 26 appendix G for the relationship between t and component eld smoothness

Edge corrections and unied RFT results The original Gaussian RFT results (21)assumed a negligible chance of the excursion set Au touching the boundary of the searchregion O2 1 If clusters did contact the surface of O they would have a contribution less


than unity to wu Worsley developed correction terms to (21) to account for thiseffect1 7 1 8 These lsquouniedrsquo results have the form

Pcu ˆ

XD

dˆ0

Rdrd(u) (29)

where D is the number of dimensions of the data Rd is the d-dimensional RESELmeasure and rd(u) is the Euler characteristic density These results are convenient asthey dissociate the terms that depend only on the topology of the search regions (Rd)from those that depend only on the type of statistical eld (rd(u))

Nonstationarity and cluster size tests For inferences on peak height with theappropriate estimator of average smoothness2 3 equation (21) will be accurate in thepresence of nonstationarity or variable smoothness However cluster size inference isgreatly effected by nonstationarity In a null statistic image large clusters will be morelikely in smooth regions and small clusters more likely in rough regions Hence anincorrect assumption of stationarity will likely lead to inated false positive rate insmooth regions and reduced power in rough regions

As alluded to above the solution is to deform space until stationarity holds(if possible2 9 ) Explicit application of this transformation is actually not required andlocal roughness can be used to determine cluster sizes in the transformed space2 3 2 5 3 0

RESEL Bonferroni A common misconception is that the random eld results applya Bonferroni correction based on the RESEL volume3 1 They are actually quite differentresults Using Millrsquos ratio the Bonferroni corrected P-value can be seen to beapproximately

PcBonf Veiexclu2 =2uiexcl1 (30)

While the RFT P-value for 3D data is approximately

PcRFT R3eiexclu2 =2u2 (31)

where R3 is the RESEL volume (28) Replacing V with R3 obviously does not align thesetwo results nor are they even proportional We will characterize the performance of anaive RESEL Bonferroni approach in Section 4

Gaussianized t images Early implementation of random eld methods (eg SPM96and previous versions) used Gaussian RFT results on t images While an image of tstatistics can be converted into Z statistics with the probability integral transformthe resulting processes is not a t random eld Worsley1 7 found that the degrees offreedom would have to be quite high as many as 120 for a t eld to behave like aGaussian eld We will examine the performance of the Gaussianized t approach withsimulations

324 RFT conclusionIn this subsection we have tried to motivate the RFT results as well as highlight

important details of their application They are a powerful set of tools for data that are

432 T Nichols et al

smooth enough to approximate continuous random elds When the data are insuf-ciently smooth or when other assumptions are dubious nonparametric resamplingtechniques may be preferred

33 Resampling methods for FWE controlThe central purpose of the random eld methods is to approximate the upper tail of

the maximal distribution FMT(t) Instead of making assumptions on the smoothness and

distribution of the data another approach is to use the data itself to obtain an empiricalestimate of the maximal distribution There are two general approaches permutation-based and bootstrap-based Excellent treatments of permutation tests3 2 3 3 and thebootstrap3 4 3 5 are available so here we only briey review the methods and focuson the differences between the two approaches and specic issues relevant toneuroimaging

To summarize briey both permutation and bootstrap methods proceed by resam-pling the data under the null hypothesis In the permutation test the data is resampledwithout replacement while in a bootstrap test the residuals are resampled withreplacement and a null dataset constituted To preserve the spatial dependence theresampling is not performed independently voxel by voxel but rather entire images areresampled as a whole For each resampled null dataset the model is t the statisticimage computed and the maximal statistic recorded By repeating this processmany times an empirical null distribution of maximum FFMT

is constructed and the100(1 iexcl a0 )th percentile provides an FWE-controlling threshold For a more detailedintroduction to the permutation test for functional neuroimaging see Ref 36

We now consider three issues that have an impact on the application of resamplingmethods to neuroimaging

331 Voxel-level statistic and homogeneity of specicity and sensitivityThe rst issue we consider is common to all resampling methods used to nd the

maximum distribution and an FWE threshold While a parametric method will assumea common null distribution for each voxel in the statistic image F0Ti

ˆ F0 FWEresampling methods are actually valid regardless of whether the null distributions F0Ti

are the same This is because the maximum distribution captures any heterogeneity asequation (4) shows the relationship between the maximum and FWE makes noassumption of homogeneous voxel null distributions Nonparametric methods willaccurately reect the maximum distribution of whatever statistic is chosen and producevalid estimates of FWE-controlling thresholds

However once an FWE-controlling threshold is chosen the false positive rate andpower at each voxel depends on each voxelrsquos distribution As shown in Figure 2 FWEcan be controlled overall but if the voxel null distributions are heterogeneous the TypeI error rate will be higher at some voxels and lower at others As a result even thoughnonparametric methods admit the use of virtually any statistic (eg raw percentchange or mean difference) we prefer a voxel-level statistic that has a common nulldistribution F0 (eg t or F) Hence the usual statistics motivated by parametric theoryare used to provide a more pivotal Ti than an un-normalized statistic Note that eventhough the statistic image may use a parametric statistic the FWE-corrected P-valuesare nonparametric


332 Randomization versus permutation versus bootstrapThe permutation and bootstrap tests are most readily distinguished by their sampling

schemes (without versus with replacement) However there are several other importantdifferences and subtle aspects to the justication of the permutation test These aresummarized in Table 4

A permutation test requires an assumption of exchangeability under the nullhypothesis This is typically justied by an assumption of independence and identicaldistribution However if a randomized design is used no assumptions on the data arerequired at all A randomization test uses the random selection of the experimentaldesign to justify the resampling of the data (or more precisely the relabeling of thedata) While the permutation test and randomization test are often referred to by thesame name we nd it useful to distinguish them As the randomization test supposes no

Figure 2 Impact of heterogeneous null distributions on FWE control Shown are the null distributions for veindependent voxels the null distribution of the maximum of the ve voxels and the 5 FWE thresholds (a)Use of the mean difference statistic allows variance to vary from voxel to voxel even under the nullhypothesis Voxel 2 has relatively large variance and shifts the maximum distribution to the right the riskof Type I error is largely due to voxel 2 and in contrast voxel 3 will almost never generate a false positive (b)If a t statistic is used the variance is standardized but the data may still exhibit variable skew This would occurif the data are not Gaussian and have heterogeneous skew Here voxels 2 and 4 bear most of the FWE risk (c) Ifthe voxel-level null distributions are homogeneous (eg if the t statistic is used and the data are Gaussian)there will be uniform risk of false positives In all three of these cases the FWE is controlled but the risk ofType I error may not be evenly distributed

434 T Nichols et al

population the resulting inferences are specic to the sample at hand The permutationtest in contrast uses a population distribution to justify resampling and hence makesinference on the population sampled3 7

A strength of randomization and permutation tests is that they exactly control thefalse positive rate Bootstrap tests are only asymptotically exact and each particulartype of model should be assessed for specicity of the FWE thresholds We are unawareof any studies of the accuracy of the bootstrap for the maximum distribution infunctional neuroimaging Further the permutation test allows the slightly more generalcondition of exchangeability in contrast to the bootstraprsquos independent and identicallydistributed assumption

The clear advantage of the bootstrap is that it is a general modeling method With thepermutation test each type of model must be studied to nd the nature of exchange-ability under the null hypothesis And some data such as positive one-sample data(ie not difference data) cannot be analysed with a permutation test as the usualstatistics are invariant to permutations of the data The bootstrap can be implementedgenerally for a broad array of models While we do not assert that bootstrap tests areautomatic and indeed general linear model design matrices can be found wherethe bootstrap performs horribly (see Ref 35 p 276) it is a more exible approachthan the permutation test

333 Exchangeability and fMRI time seriesFunctional magnetic resonance imaging (fMRI) data is currently the most widely used

functional neuroimaging modality However fMRI time series exhibit temporal auto-correlation that violates the exchangeability=independence assumption of the resam-pling methods Three strategies to deal with temporal dependence have been applied donothing resample ignoring autocorrelation3 8 use a randomized design and randomiza-tion test3 9 and decorrelate and then resample4 0 ndash4 2 Ignoring the autocorrelation inparametric settings tends to inate signicance due to biased variance estimates withnonparametric analyses there may be either inated or deated signicance dependingon the resampling schemes In a randomization test the data is considered xed andhence any autocorrelation is irrelevant to the validity of the test (power surely doesdepend on the autocorrelation and the resampling scheme but this has not been studiedto our knowledge) The preferred approach is the last one The process consists of ttinga model estimating the autocorrelation with the residuals decorrelating the residualsresampling and then recorrela ting the resampled residuals creating null hypothesisrealizations of the data The challenges of this approach are the estimation of theautocorrelation and the computational burden of the decorrelationndashrecorrelation

Table 4 Differences between randomization permutation and bootstrap tests

Randomization Permutation Bootstrap

P-values Exact Exact Asymptotic=approximateAssumption Randomized experiment Ho-exchangeability IIDInference Sample only Population PopulationModels Simple Simple General


process To have an exact permutation test the residuals must be exactly whitened butthis is impossible without the true autocorrelation However in simulations and withreal null data Brammer and colleagues found that the false positive rate was wellcontrolled4 0 To reduce the computational burden Fadili and Bullmore4 3 proposedperforming the entire analysis in the whitened (ie wavelet) domain

334 Resampling conclusionNonparametric permutation and bootstrap methods provide estimation of the

maximum distribution without strong assumptions and without inequalities thatloosen with increasing dependence Only their computational intensity and lack ofgenerality preclude their widespread use

4 Evaluation of FWE methods

We evaluated methods from the three classes of FWE-controlling procedures Ofparticular interest is a comparison of random eld and resampling methods permuta-tion in particular In earlier work3 6 comparing permutation and RFT methods onsmall group PET and fMRI data we found the permutation method to be much moresensitive and the RFT method comparable to Bonferroni The present goal is toexamine more datasets to see if those results generalize and to examine simulations todiscover if the RFT method is intrinsica lly conservative or if specic assumptions didnot hold in the datasets considered In particular we seek the minimum smoothnessrequired by the random eld theory methods to perform well We also investigate iftwo of the extended Bonferroni methods enhance the sensitivity of Bonferroni

41 Real data resultsWe applied Bonferroni-related random eld and permutation methods to nine fMRI

group datasets and two PET datasets All data were analysed with a mixed effect modelbased on summary statistics4 4 This approach consists of tting intrasubject generallinear models on each subject creating a contrast image of the effect of interest andassessing the population effect with a one-sample t test on the contrast images Thesmoothness parameter of the random eld results were estimated from the standardizedresidual images of the one-sample t2 7 Random eld results were obtained with SPM99(httpwwwlionuclacukspm) and nonparametric results were obtained withSnPM99 (httpwwwlionuclacukspmsnpm)

A detailed description of each dataset is omitted for reasons of space but wesummarize each briey Verbal Fluency is a ve-subject PET dataset comparing abaseline of passive listening versus word generation as cued by single letter (completedataset available at httpwwwlionuclacukspmdata) Location Switching andTask Switching are two different effects from a 10-subject fMRI study of attentionswitching (Tor Wager et al in preparation) Faces Main Effect and Faces Interactionare two effects (main effect data available at httpwwwlionuclacukspmdata)from a 12-subject fMRI study of repetition priming4 5 Item Recognition is one effectfrom a 12-subject fMRI study of working memory4 6 Visual Motion is a 12-subject PETstudy of visual motion perception comparing moving squares to xed ones4 7

436 T Nichols et al

Emotional Pictures is one effect from a 13-subject fMRI study of emotional processingas probed by photographs of varying emotional intensity4 8 Pain WarningPain Anticipation and Pain Pain are three effects from a 23-subject fMRI study ofpain and the placebo effect (Tor Wager et al in preparation)

Tables 5 and 6 shows the results for the 11 datasets Table 5 shows that for everydataset the permutation method has the lowest threshold often dramatically so Usingeither Bonferroni or permutation as a reference the RFT becomes more conservativewith decreasing degrees of freedom (DF) for example specifying a threshold of 470132for a 4 DF analysis The Bonferroni threshold is lower than the RFT threshold for allthe low-DF studies Only for the 22 DF study is the RFT threshold below Bonferronialthough the two approaches have comparable thresholds for one of the 11 DF studiesand the 2 DF study The smoothness is greater than three voxel FWHM for all studiesexcept for the z-smoothness in the visual motion study This suggests that a three voxelFWHM rule of thumb1 9 is insufcient for low-DF t statistic images

Degrees of freedom and not smoothness seems to be the biggest factor in theconvergence of the RFT and permutation results That is RFT comes closest topermutation not when smoothness is particularly large (eg Task switching) butwhen degrees of freedom exceed 12 (eg the Pain dataset) This suggest a conserva-tiveness in the low-DF RFT t results that is not explained by excessive roughness

Comparing the 11 DF studies Item recognition and Visual motion is informative asone has twice as many voxels and yet half as many RESELs This situation results inBonferroni being higher on Item recognition (980 versus 892) yet RFT being lower(987 versus 1107) Item recognition has the lower permutation threshold (767 versus840) suggesting that the resampling approach is adapting to greater smoothness despitethe larger number of voxels

Hochberg and SNidak are often innity indicating that no a0 ˆ 005 threshold exists[ie no P-value satised equation (7)] Also note that Hochberg and SNidak can be more

Table 5 Summary of FWE inferences for 11 PET and fMRI studies 5 FWE thresholds for ve differentmethods are presented RFT Bonferroni Hochberg step-up SN idak step-down and permutation Note how RFTonly outperforms other methods for studies with the largest degrees of freedom Hochberg and SN idakrsquosmethod rarely differs from Bonferroni by much Permutation always has the lowest threshold

Study DF Voxels Voxel FWHMsmoothness

RESELvolume

FEW-corrected t threshold

RFT Bonf Hoch SN idak Perm

Verbal uency 4 55027 56 63 39 3999 470132 4259 1 1 1014Location switching 9 36124 61 59 51 1968 1117 1031 1 1 583Task switching 9 37181 64 69 52 1619 1079 1035 1029 1029 510Faces main effect 11 51560 41 41 43 7133 1043 907 907 904 792Faces interaction 11 51560 38 39 40 8698 1070 907 1 1 826Item recognition 11 110776 51 68 69 4629 987 980 999 999 767Visual motion 11 43724 39 44 22 11582 1107 892 890 887 840Emotional pictures 12 44552 56 54 50 2947 848 841 1 1 715Pain warning 22 23263 47 49 35 2886 593 605 609 604 499Pain anticipation 22 23263 50 51 36 2534 587 605 607 607 505Pain pain 22 23263 46 48 34 3099 595 605 605 605 515


stringent than Bonferroni even though their critical values ui are never less than a0=V This occurs because the critical P-value falls below both ui and a0=V

Table 6 shows how even though the permutation thresholds are always lower it failsto detect any voxels in some studies (As noted in Ref 45 the Faces Interaction effectis signicant in an a priori region of interest) While truth is unknown here this isevidence of permutationrsquos specicity The last column of this table includes results usinga smoothed variance t statistic a means to boost degrees of freedom by lsquoborrowingstrengthrsquo from neighboring voxels4 9 3 6 In all of these studies it increased the number ofdetected voxels in some cases dramatically

42 Simulation methodsWe simulated 32 pound 32 pound 32 images since a voxel count of 323 ˆ 32 767 is typical for

moderate resolution (ordm3 mm3 ) data Smooth images were generated as Gaussian whitenoise convolved with a 3D isotropic Gaussian kernel of size 0 15 3 6 and 12 voxelsFWHM (s ˆ 0 064 127 255 51) We did not simulate in the Fourier domain toavoid wrap-around effects and to avoid edge effects of the kernel we padded all sides ofimage by a factor of three FWHM and then truncated after convolution In total 3000realizations of one-sample t statistic images were simulated for three different n n ˆ 1020 30 Each t statistic image was based on n realizations of smooth Gaussian randomelds to our knowledge there is no direct way to simulate smooth t elds For eachsimulated dataset a simple linear model was t and residuals computed and used toestimate the smoothness as in Ref 27 To stress for each realized dataset both theestimated and known smoothness was used for the random eld inferences allowingthe assessment of this important source of variability

For each simulated dataset we computed a permutation test with 100 resamplesWhile the exactness of the permutation test is given by exchangeability holding in theseexamples this serves to validate our code and support other work

Table 6 Summary of FWE inferences for 11 PET and fMRI studies (continued) Shown are the number ofsigni cant voxels detected with the ve methods discussed along with permutation method on the smoothedvariance t statistic

Study Number of signi cant voxels

t Sm Var t Perm


Verbal uency 0 0 0 0 0 0Location switching 0 0 0 0 158 354Task switching 4 6 7 7 2241 3447Faces main effect 127 371 372 379 917 4088Faces interaction 0 0 0 0 0 0Item recognition 5 5 4 4 58 378Visual motion 626 1260 1269 1281 1480 4064Emotional pictures 0 0 0 0 0 7Pain warning 127 116 116 118 221 347Pain anticipation 74 55 55 55 182 402Pain pain 387 349 350 353 732 1300

438 T Nichols et al

We also simulated Gaussian statistic images with the same set of smoothness but wedid not apply the permutation test nor estimate smoothness

For each realized statistic image we computed the Bonferroni random eld theoryand permutation thresholds (except Gaussian) and noted the proportion of realizationsfor which maximal statistic was above the threshold which is the Monte Carlo estimateof the familywise error in these null simulations

For each realization we also computed three other FWER procedures an FDRthreshold a threshold based on Gaussianizing the t images and a Bonferroni thresholdusing the estimated number of RESELS

To estimate the equivalent number of independent test (see Section 25) we estimatedy with regression through the origin based on a transformation of equation (10)

log (1 iexcl FP(1)(t)) ˆ log (1 iexcl t)Vy (32)

We replace FP(1)(t) with the empirical cumulative distribution function of the minimum

P-value found under simulation Because we are generally interested in a0 ˆ 005 weonly use values of t such that 003 micro FP(1)

(t) micro 007

43 Simulation resultsFigure 3 shows the accuracy in the smoothness estimate As in Ref 27 we found

positive bias for low smoothness although for higher smoothness we found slightnegative bias Positive bias or overestimation of smoothness under estimates the degreeof the multiple testing problem and can cause inferences to be anticonservative(However anticonservativeness was not a problem see below)

Figure 4 shows the results using the estimated smoothness Figure 4(a) shows thepermutation and true results tracking closely while the RFT results are very conserva-tive only approaching truth for very high smoothness Bonferroni is of course not

Figure 3 Smoothness estimation bias as function of smoothness Bias (estimated minus true) is smaller forlarger DF and larger FWHM smoothness Overestimated bias (for low smoothness) could result in anti-conservative inferences (though apparently does not see other results)


adaptive to smoothness but is very close to truth for low smoothness especially for lowDF The Gaussian results are much closer to truth than any of the t results (note they-axis range) Figure 4(b) shows the familywise error rates which magnify performancedifferences RFT is seen to be severely conservative for all but extremely smooth dataand Bonferroni is indistinguishable from truth for FWHM of three or less with DF of9 and 19 The permutation performance is consistent with its exactness For six FWHMand above the Gaussian result is close to nominal

Figure 4 Simulation results (a) FWE threshold found by three different methods compared to truthThe Bonferroni threshold is nonadaptive while permutation and random eld methods both use lowerthresholds with higher smoothness The low-smoothness conservativeness of the random eld thresholdsintensi es with decreasing degrees of freedom (b) Rejection rate of null simulations for a nominal a0 ˆ 005threshold with a pointwise Monte Carlo 95 con dence interval shown with ne dotted line The random eld theory results are valid but quite conservative for all but high smoothnesses Bonferroni results aresurprisingly satisfactory for up to three voxels FWHM smoothness but then become conservative

440 T Nichols et al

Using true smoothness instead of estimated smoothness had little impact on theresults The rejection rates never differed by more than 0003 except for the case of 9DF and 12 FWHM where it increased the rejection rate by 00084

Figure 5 plots the cumulative density functions (CDFs) of the minimum P-valuefound by simulation and compares it to other methods for approximating or boundingFWE The CDF approximation provided by Bonferroni is the same for all gures sincethe number of voxels is xed The RFT approximation (dashndashdot line) changes withsmoothness but is far from true CDF for low smoothness and low DF critically forany given FWHM and DF the RFT results do not improve with (decreasing P-value)

Figure 5 Approximating minimum P-value distributions with FWE methods The minimum P-value CDFobtained by simulation (lsquoTruthrsquo solid line with dots) is compared to three different approximations Bonferroniinequality (lsquoBonfrsquo solid line) random eld theory (lsquoRFTrsquo dotndashdashed line) and the equivalent independent n(EIN dashed line) a corrected P-value of 005 is indicated (horizontal dotted line) These plots re ect the ndings in Figure 4 Bonferroni is accurate for data as smooth as 15 FWHM data RFT is more conservativethan Bonferroni for data as smooth as three FWHM and for six FWHM for 6 DF While Figure 4 only depictedresults for a0 ˆ 005 note that for a given smoothness and DF the RFT results do not improve with morestringent thresholds (less than 005 corrected) For the 12 FWHM smoothness data the RFT results are quiteaccurate and provide a better approximation than the equivalent independent n approach Particularly for 9DF and 12 FWHM note that the EIN approach fails to have the correct slope (it intersects the true CDF aroundFNP

(005) by construction see Section 25)


threshold This indicates that the poor RFT performance is not due to use of aninsufciently high threshold Finally note that the CDF of an equivalent independentnumber (EIN) of observations (dashed line) follows the true CDF quite well formoderate smoothness but at high smoothness it has the wrong slope and cannotmatch the CDF in general [as predicted by equations (30) and (31)] That the EINapproach performs so well for moderate smoothness suggests that it may yet be atenable theoretical approach

For 9 DF point estimates for y were found to be 090 094 087 0043 and 006 for0 15 3 6 12 voxel FWHM smoothness respectively While Figure 5 indicates that theEIN approach is inappropriate for high smoothness for three voxel FWHM smooth-

Figure 6 Comparison of other FWE methods The RESELndashBonferroni approach fails to control FWE for anysmoothness considered The Gaussianized T approach does not reliably control FWE in particular beinganticonservative for smooth low DF images FDR does control FWE (weakly) but becomes somewhat conserva-tive for increasing smoothness Fine dotted line indicates pointwise Monte Carlo 95 con dence interval

442 T Nichols et al

ness a 323 voxel t9 image has the same FWE threshold as yV ˆ 087 pound 323 ˆ 28 623independent voxels

Figure 6 shows the performance of three alternative methods The RESEL Bonferroniapproach fails to control FWE and for moderate to high smoothness exceeded a FWEof 05 (off the plot not shown) The Gaussianized t method exhibits conservativenessfor low smoothness but for low DF it is anticonservative suggesting it would beinappropriate to use for all but the high DF In this complete null simulation Benjaminiand Hochbergrsquos FDR controls FWE although it becomes somewhat conservative forincreasing smoothness

44 Results discussionWhile some authors have observed RFT conservativeness3 6 5 0 5 1 other have not2 2 6

However our ndings are consistent with the literature because the authors that foundRFT results to be accurate used Gaussian data with high smoothness For exampleWorsley et al2 found the expected wu was quite accurate on Z images but thesmoothness of their data was approximately 10 voxels FWHM Our Gaussiansimulations are consistent with this and for all but the lowest DF our t simulationsalso suggest that 10 FWHM is sufcient

With our real data studies the permutation method was found to be more sensitive inall 11 datasets This is consistent with our simulations in particular that the RFTmethod was increasingly conservative for shrinking degrees of freedom By conven-tional standards in functional neuroimaging our real data would be considered quitesmooth (4ndash6 voxel FWHM) but our simulations indicate this is still insufcient foraccurate RFT thresholds

As a note on the selection of these datasets they represent a three-year process ofcollecting group-level fMRI and PET datasets The only data omitted were other effectsfrom the studies included usually other nonorthogonal contrasts with qualitativelysimilar results In ve years of applying these methods we have never seen a small DFdataset (lt 10) where the t random eld method outperforms the permutation test

5 Discussion

We have attempted to provide a comprehensive review and a representative comparisonof FWE methods for functional neuroimaging From Bonferroni and its extensions tocutting-edge random eld theory methods to permutation methods of Fisher we haveattempted to cull all available tools that are relevant for the massive dependent data offunctional neuroimaging With an assumption of positive dependence we can make useof slightly improved Bonferroni methods With an assumption of smoothness we canmake use of smoothness-adaptive RFT methods And with few assumptions at all andsome computational effort we have both an adaptive and powerful method

There are several limitations of these ndings First yet more datasets should bestudied over yet a wider range of smoothnesses and group sizes We have focused on verysmall group data to demonstrate a suspected conservativeness of RFT methodsHowever more moderate group sizes are needed to see exactly when RFT methodslose power Secondly more simulations are needed for larger volumes and for more


realistica lly shaped search regions Our 32-cubed volume is too small when 1 mm3 voxelsare used and does not reect the wrinkled-ellipsoida l topology of real brain data Andnally the computational burden of the permutation tests must be considered alongwith the exibility of a general linear modeling tool combined with RFT inference

AcknowledgementsThe authors would like to thank Keith Worsley for many valuable conversations on

random eld theory especially Tor Wager for help with the pain data The authorswish to thank all of the individuals who contributed data to our evaluations

References

1 Hochberg Y Tamhane AC Multiplecomparison procedures New York Wiley1987

2 Worsley K Evans A Marrett S Neelin PA three-dimensional statistical analysis for CBFactivation studies in human brain Journal ofCerebral Blood Flow amp Metabolism 1992 12900ndash18

3 Friston KJ Frith CD Liddle PF FrackowiakRSJ Comparing functional (PET) images theassessment of signicant change Journal ofCerebral Blood Flow amp Metabolism 1991 11690ndash99

4 Storey JD A direct approach to false discoveryrates Journal of the Royal Statistical SocietySeries B 2002 64 479ndash498

5 McShane LM Statistical issues in the analysisof microarray data In Proceedings of theInternational Biometrics Society FrieburgGermany July 2002

6 Korn DL Troendle JF McShane LM SimonR Controlling the number of falsediscoveries application to high-dimensionalgenomic data Technical report BiometricResearch Branch National Cancer InstituteBethesda Maryland 20892 USA August2001

7 Westfall PH Young SS Resampling-basedmultiple testing examples and methods forp-value adjustment New York Wiley 1993

8 Tong Y Probability inequalities inmultivariate distributions New YorkAcademic Press 1980

9 Holm S A simple sequentially rejectivemultiple test procedure Scandinavian Journalof Statistics 1979 6 65ndash70

10 Hochberg Y A sharper Bonferroni procedurefor multiple tests of signicance Biometrika1988 75 800ndash802

11 Simes RJ An improved Bonferroni procedurefor multiple tests of signicance Biometrika1986 73 751ndash54

12 Benjamini Y Hochberg Y Controlling thefalse discovery rate A practical and powerfulapproach to multiple testing Journal of theRoyal Statistical Society Series BMethodological 1995 57 289ndash300

13 Holland BS Copenhaver MD An improvedsequentially rejective Bonferroni testprocedure (Corr V43 p 737) Biometrics1987 43 417ndash23

14 Benjamini Y Yekutieli D The control of thefalse discovery rate in multiple testing underdependency Annals of Statistics 2001 29(4)1165ndash88

15 Sarkar SK Some results on false discovery ratein stepwise multiple testing proceduresAnnals of Statistics 2002 30(1) 239ndash57

16 Sarkar SK Recent advances in multiple testingTechnical report Philadelphia TempleUniversity 2002

17 Worsley KJ Marrett S Neelin P Vandal ACFriston KJ Evans AC A unied statisticalapproach for determining signicant signals inimages of cerebral activation Human BrainMapping 1995 4 58ndash73

18 Cao J Worsley KJ Spatial statistics metho-dological aspects and applications chapter 8Applications of random elds in humanbrain mapping Lecture Notes in Statisticsv 159 New York Springer 2001 pp169ndash82

19 Petersson KM Nichols TE Poline J-BHolmes AP Statistical limitations infunctional neuroimaging II Signal detectionand statistical inference PhilosophicalTransactions of the Royal Society Series B1999 354 1261ndash81

444 T Nichols et al

20 Adler RJ The geometry of random elds NewYork Wiley 1981

21 Worsley KJ Evans AC Marrett S Neelin P Athree-dimensional statistical analysis for CBFactivation studies in human brain Journal ofCerebral Blood Flow amp Metabolism 199212(6) 900ndash18

22 Poline JB Worsley KJ Evans AC Friston KJCombining spatial extent and peak intensityto test for activations in functional imagingNeuroImage 1997 5(2) 83ndash96

23 Worsley KJ Andermann M Koulis TMacDonald D Evans AC Detecting changesin nonisotropic images Human BrainMapping 1999 8 98ndash101

24 Poline J-B Worsley KJ Holmes APFrackowiak RSJ Friston KJ Estimatingsmoothness in statistical parametric mapsvariability of p values Journal of ComputerAssisted Tomography 1995 19 788ndash96

25 Worsley KJ Non-stationary FWHM and itseffect on statistical inference of fMRI dataPresented at the 8th International Conferenceon Functional Mapping of the Human Brain2ndash6 June 2002 Sendai Japan Available onCDRom NeuroImage 2002 16(2) 779ndash80

26 Holmes AP Statistical issues in functionalbrain mapping PhD thesis University ofGlasgow Glasgow 1994 Available fromhttp==wwwlionuclacuk=spm=papers=APH_thesis

27 Kiebel S Poline J Friston K Holmes AWorsley K Robust smoothness estimation instatistical parametric maps using standardizedresiduals from the general linear modelNeuroImage 1999 10 756ndash66

28 Worsley KJ Estimating the number of peaksin a random eld using the Hadwigercharacteristic of excursion sets withapplications to medical images Annals ofStatistics 1995 23 640ndash69

29 Sampson PD Guttorp P Nonparametricestimation of nonstationary spatial covariancestructure Journal of the American StatisticalAssociation 1992 87 108ndash19

30 Hayasaka S Nichols TE A resel-based clustersize permutation test for non-stationaryimages Presented at the 8th InternationalConference on Functional Mapping of theHuman Brain 2ndash6 June 2002 Sendai JapanAvailable on CDRom NeuroImage 200216(2) 1062ndash63

31 Dinov ID Mega MS Thompson PM et alAnalyzing functional brain images in aprobabilistic atlas a validation of sub-volume

thresholding Journal of Computer AidedTomography 2000 24(1) 128ndash38

32 Good P Permutation tests A practical guideto resampling methods for testing hypothesesNew York Springer Verlag 1994

33 Peasarin F Multivariate permutation testswith applications in biostatistics New YorkWiley 2002

34 Efron B Tibshirani R An introduction to thebootstrap Boca Raton Chapman amp Hall1993

35 Davison AC Hinkley DV Bootstrap methodsand their application Cambridge CambridgeUniversity Press 1997

36 Nichols TE Holmes AP Nonparametricpermutation tests for functionalneuroimaging a primer with examplesHuman Brain Mapping 2001 15 1ndash25

37 Scheffe H Statistical inference in the non-parametric case Annals of MathematicalStatistics 1947 14 304ndash32

38 Belmonte M Yurgelun-Todd D Permutationtesting made practical for functional magneticresonance image analysis IEEE Transactionson Medical Imaging 2001 20 243ndash48

39 Liu C Raz J Turetsky B An estimator andpermutation test for single-trial fMRI dataIn Abstracts of ENAR Meeting of theInternational Biometric Society PittsburghMarch 1998

40 Brammer MJ Bullmore ET Simmons A et alGeneric brain activation mapping in functionalmri a nonparametric approach MagneticResonance Imaging 1997 15 763ndash70

41 Locascio JJ Jennings PJ Moore CI Corkin STime series analysis in the time domain andresampling methods for studies of functionalmagnetic resonance brain imaging HumanBrain Mapping 1997 3 168ndash93

42 Bullmore E Long C Suckling J Colored noiseand computational inference inneurophysiological (fMRI) time seriesanalysis resampling methods in time andwavelet domains Human Brain Mapping2001 12 61ndash78

43 Fadili J Bullmore ET Wavelet-generalisedleast squares a new BLU estimator ofregression models with long-memory errorsNeuroImage 2001 15 217ndash32

44 Holmes AP Friston KJ Generalisabilityrandom effects amp population inferenceNeuroImage 1999 7(4) S754 Proceedingsof Fourth International Conference onFunctional Mapping of the Human Brain7ndash12 June 1998 Montreal Canada


45 Henson RNA Shallice T Gorno-TempiniML Dolan RJ Face repetition effects inimplicit and explicit memory tests as measuredby fMRI Cerebral Cortex 2002 12 178ndash86

46 Marshuetz C Smith EE Jonides J DeGutis JChenevert TL Order information inworking memory fMRI evidence for parietaland prefrontal mechanisms Journal ofCognitive Neuroscience 2000 12(S2)130ndash44

47 Watson JDG Myers R Frackowiak RSJ et alArea V5 of the human brain evidence from acombined study using positron emissiontomography and magnetic resonance imagingCerebral Cortex 1993 3 79ndash94

48 Phan KL Taylor SF Welsh RC et alActivation of the medial prefrontal cortex and

extended amygdala by individual ratings ofemotional arousal An fMRI study BiologicalPsychiatry 2003 53 211ndash15

49 Holmes AP Blair RC Watson JDG Ford INonparametric analysis of statistic imagesfrom functional mapping experiments Journalof Cerebral Blood Flow amp Metabolism 199616(1) 7ndash22

50 Stoeckl J Poline J-B Malandain G Ayache NDarcourt J Smoothness and degrees offreedom restrictions when using spm99NeuroImage 2001 13 S259

51 Singh KD Barnes GR Hillebrand A Groupimaging of task-related changes in corticalsynchronisation using non-parametricpermutation testing NeuroImage 2003 191589ndash1601

446 T Nichols et al

term multiple testing problem over multiple comparisons problem lsquoMultiple compar-isonsrsquo can allude to pairwise comparisons on a single model whereas in imaging a largecollection of models is each subjected to a hypothesis test)

In this paper we attempt to describe and evaluate all FWE multiple testing proceduresuseful for functional neuroimaging Owing to the spatial dependence of functionalneuroimaging data there are actually quite a small number of applicable methods Theonly methods that are appropriate under these conditions are Bonferroni randomeld methods and resampling methods We limit our attention to ndingFWE-corrected thresholds and P-values for Gaussian Z and Student t images(An a0 FWE-corrected threshold is one that controls the FWE at a0 while an FWE-corrected P-value is the most signicant a0 corrected threshold such that a test can berejected) In particular we do not consider inference on size of contiguous suprathres-hold regions or clusters We focus particular attention on low degrees-of-freedom (DF) timages as these are unfortunately common in group analyses The typical images havetens to hundreds of thousands of tests where the tests have complicated though usuallypositive dependence structure

In 1987 Hochberg and Tamhane wrote lsquoThe question of which error rate tocontrol has generated much discussion in the literaturersquo1 While that could be truein the statistics literature in the decades before their book was published the samecannot be said of neuroimaging literature When multiple testing has been acknowl-edged at all the familywise error rate has usually been assumed implicitly For exampleof the two papers that introduced corrected inference methods to neuroimaging onlyone explicitly mentions familywise error2 3 It is hoped that the introduction of FDRwill enrich the discussion of multiple testing issues in the neuroimaging literature

The remainder of the paper is organized as follows We rst introduce the generalmultiple comparison problem and FWE We then review Bonferroni-related methodsfollowed by random eld methods and then resampling methods We then evaluatethese different methods with 11 real datasets and simulations

2 Multiple testing background in functional neuroimaging

In this section we formally dene strong and weak control of familywise error as wellas other measures of false positives in multiple testing We describe the relationshipbetween the maximum statistic and FWE and also introduce step-up and step-downtests and the notion of equivalent number of independent tests

21 NotationConsider image data on a two- (2D) or three-dimensional (3D) lattice Usually the

lattice will be regular but it may also be a irregular corresponding to a 2D surfaceThrough some modeling process we have an image of test statistics T ˆ Ti Let Ti bethe value of the statistic image at spatial location i i 2 V ˆ 1 V where V is thenumber of voxels in the brain Let the H ˆ Hi be a hypothesis image where Hi ˆ 0indicates that the null hypothesis holds at voxel i and Hi ˆ 1 indicates that thealternative hypothesis holds Let H0 indicate the complete null case where Hi ˆ 0 forall i A decision to reject the null for voxel i will be written HHi ˆ 1 not rejecting HHi ˆ 0

420 T Nichols et al








Pi HHi












P[

i2VTi para u

shyshyshyshyH0

Aacute

micro a0 (1)


P[

i2V0


Aacute

micro a0 (2)





[

i





422 T Nichols et al


ˆ Fiexcl1MT jH0



P[

i



jH0 )


)

ˆ a0



P[

i2V0


Aacute

ˆ P[

i2V0


Aacute

(4)

micro P[

i2VTi para ua0

shyshyshyshyH0

Aacute

(5)

ˆ a0 (6)





P(i) micro ui (7)






FMT(t) ˆ

Y

i

FTi(t) ˆ FV

T1(t) (8)



FMT(t) ˆ FyV

T1(t) (9)



(t))yV





424 T Nichols et al





Praquo [

i

Ai

frac14micro

X

i

PAi (11)




Praquo [

i

Ai

frac14micro

X

i

PAi iexcl maxkˆ1V

raquo X

i 6ˆk

PAi Akfrac14

(12)









P(i) micro a01




P(i) micro a0iV

(15)





426 T Nichols et al














FWE ˆ PoFWE (16)

ˆ P[

i

Ti para u

( )

(17)

ˆ P maxi

Ti para u (18)

ordm Pwu gt 0 (19)

ordm Ewu (20)



428 T Nichols et al




L ˆ Varq

qxZ

qqy

Zqqz

Zmicro paragt

Aacute (22)




Pcz ˆ P( max




2

sup3 acute(24)

















x s2y s2




430 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al








Pi HHi












P[

i2VTi para u

shyshyshyshyH0

Aacute

micro a0 (1)


P[

i2V0


Aacute

micro a0 (2)





[

i





422 T Nichols et al


ˆ Fiexcl1MT jH0



P[

i



jH0 )


)

ˆ a0



P[

i2V0


Aacute

ˆ P[

i2V0


Aacute

(4)

micro P[

i2VTi para ua0

shyshyshyshyH0

Aacute

(5)

ˆ a0 (6)





P(i) micro ui (7)






FMT(t) ˆ

Y

i

FTi(t) ˆ FV

T1(t) (8)



FMT(t) ˆ FyV

T1(t) (9)



(t))yV





424 T Nichols et al





Praquo [

i

Ai

frac14micro

X

i

PAi (11)




Praquo [

i

Ai

frac14micro

X

i

PAi iexcl maxkˆ1V

raquo X

i 6ˆk

PAi Akfrac14

(12)









P(i) micro a01




P(i) micro a0iV

(15)





426 T Nichols et al














FWE ˆ PoFWE (16)

ˆ P[

i

Ti para u

( )

(17)

ˆ P maxi

Ti para u (18)

ordm Pwu gt 0 (19)

ordm Ewu (20)



428 T Nichols et al




L ˆ Varq

qxZ

qqy

Zqqz

Zmicro paragt

Aacute (22)




Pcz ˆ P( max




2

sup3 acute(24)

















x s2y s2




430 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al


P[

i2VTi para u

shyshyshyshyH0

Aacute

micro a0 (1)


P[

i2V0


Aacute

micro a0 (2)





[

i





422 T Nichols et al


ˆ Fiexcl1MT jH0



P[

i



jH0 )


)

ˆ a0



P[

i2V0


Aacute

ˆ P[

i2V0


Aacute

(4)

micro P[

i2VTi para ua0

shyshyshyshyH0

Aacute

(5)

ˆ a0 (6)





P(i) micro ui (7)






FMT(t) ˆ

Y

i

FTi(t) ˆ FV

T1(t) (8)



FMT(t) ˆ FyV

T1(t) (9)



(t))yV





424 T Nichols et al





Praquo [

i

Ai

frac14micro

X

i

PAi (11)




Praquo [

i

Ai

frac14micro

X

i

PAi iexcl maxkˆ1V

raquo X

i 6ˆk

PAi Akfrac14

(12)









P(i) micro a01




P(i) micro a0iV

(15)





426 T Nichols et al














FWE ˆ PoFWE (16)

ˆ P[

i

Ti para u

( )

(17)

ˆ P maxi

Ti para u (18)

ordm Pwu gt 0 (19)

ordm Ewu (20)



428 T Nichols et al




L ˆ Varq

qxZ

qqy

Zqqz

Zmicro paragt

Aacute (22)




Pcz ˆ P( max




2

sup3 acute(24)

















x s2y s2




430 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al


ˆ Fiexcl1MT jH0



P[

i



jH0 )


)

ˆ a0



P[

i2V0


Aacute

ˆ P[

i2V0


Aacute

(4)

micro P[

i2VTi para ua0

shyshyshyshyH0

Aacute

(5)

ˆ a0 (6)





P(i) micro ui (7)






FMT(t) ˆ

Y

i

FTi(t) ˆ FV

T1(t) (8)



FMT(t) ˆ FyV

T1(t) (9)



(t))yV





424 T Nichols et al





Praquo [

i

Ai

frac14micro

X

i

PAi (11)




Praquo [

i

Ai

frac14micro

X

i

PAi iexcl maxkˆ1V

raquo X

i 6ˆk

PAi Akfrac14

(12)









P(i) micro a01




P(i) micro a0iV

(15)





426 T Nichols et al














FWE ˆ PoFWE (16)

ˆ P[

i

Ti para u

( )

(17)

ˆ P maxi

Ti para u (18)

ordm Pwu gt 0 (19)

ordm Ewu (20)



428 T Nichols et al




L ˆ Varq

qxZ

qqy

Zqqz

Zmicro paragt

Aacute (22)




Pcz ˆ P( max




2

sup3 acute(24)

















x s2y s2




430 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al




FMT(t) ˆ

Y

i

FTi(t) ˆ FV

T1(t) (8)



FMT(t) ˆ FyV

T1(t) (9)



(t))yV





424 T Nichols et al





Praquo [

i

Ai

frac14micro

X

i

PAi (11)




Praquo [

i

Ai

frac14micro

X

i

PAi iexcl maxkˆ1V

raquo X

i 6ˆk

PAi Akfrac14

(12)









P(i) micro a01




P(i) micro a0iV

(15)





426 T Nichols et al














FWE ˆ PoFWE (16)

ˆ P[

i

Ti para u

( )

(17)

ˆ P maxi

Ti para u (18)

ordm Pwu gt 0 (19)

ordm Ewu (20)



428 T Nichols et al




L ˆ Varq

qxZ

qqy

Zqqz

Zmicro paragt

Aacute (22)




Pcz ˆ P( max




2

sup3 acute(24)

















x s2y s2




430 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al





Praquo [

i

Ai

frac14micro

X

i

PAi (11)




Praquo [

i

Ai

frac14micro

X

i

PAi iexcl maxkˆ1V

raquo X

i 6ˆk

PAi Akfrac14

(12)









P(i) micro a01




P(i) micro a0iV

(15)





426 T Nichols et al














FWE ˆ PoFWE (16)

ˆ P[

i

Ti para u

( )

(17)

ˆ P maxi

Ti para u (18)

ordm Pwu gt 0 (19)

ordm Ewu (20)



428 T Nichols et al




L ˆ Varq

qxZ

qqy

Zqqz

Zmicro paragt

Aacute (22)




Pcz ˆ P( max




2

sup3 acute(24)

















x s2y s2




430 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al




P(i) micro a01




P(i) micro a0iV

(15)





426 T Nichols et al














FWE ˆ PoFWE (16)

ˆ P[

i

Ti para u

( )

(17)

ˆ P maxi

Ti para u (18)

ordm Pwu gt 0 (19)

ordm Ewu (20)



428 T Nichols et al




L ˆ Varq

qxZ

qqy

Zqqz

Zmicro paragt

Aacute (22)




Pcz ˆ P( max




2

sup3 acute(24)

















x s2y s2




430 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al














FWE ˆ PoFWE (16)

ˆ P[

i

Ti para u

( )

(17)

ˆ P maxi

Ti para u (18)

ordm Pwu gt 0 (19)

ordm Ewu (20)



428 T Nichols et al




L ˆ Varq

qxZ

qqy

Zqqz

Zmicro paragt

Aacute (22)




Pcz ˆ P( max




2

sup3 acute(24)

















x s2y s2




430 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al




L ˆ Varq

qxZ

qqy

Zqqz

Zmicro paragt

Aacute (22)




Pcz ˆ P( max




2

sup3 acute(24)

















x s2y s2




430 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al













x s2y s2




430 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al


8 log 2



jLj1=2 ˆ (4 log 2)3=2

FWHMxFWHMyFWHMz(27)




(28)



VarEj







Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al


Pcu ˆ

XD

dˆ0

Rdrd(u) (29)












432 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al


















434 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al


















436 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al








RESELvolume












t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al








t Sm Var t Perm



438 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al








(t) micro 007








440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al



440 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al









442 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al







5 Discussion







References




















444 T Nichols et al




































446 T Nichols et al




References




















444 T Nichols et al




































446 T Nichols et al




































446 T Nichols et al

controlling the familywise error rate in functional...

Documents