scanlab meta-analysis of neuroimaging data what, why, and how tor d. wager columbia university

SCANLab

http://www.columbia.edu/cu/psychology/tor/

Meta-analysis of neuroimaging Meta-analysis of neuroimaging datadata

What, Why, and HowWhat, Why, and How

Tor D. WagerTor D. WagerColumbia UniversityColumbia University

SCANLab


Uses of meta-analysis in neuroimaging

• Meta-analysis is an essential tool for summarizing the vast and growing neuroimaging literature

Wager, Lindquist, & Hernandez, in press

SCANLab


Uses of meta-analysis in neuroimaging

Wager, Lindquist, & Kapan, 2007

• Assess consistency of activation across laboratories and task variants

• Compare across many types of tasks and evaluate the specificity of activated regions for particular psychological conditions

• Identify and define boundaries of functional regions

• Co-activation: Develop models of functional systems and pathways

SCANLab


Functional networks in meta-analysis

• Use regions or distributed networks in a priori tests in future studies

SCANLab


Meta-analyses of cognitive control

SCANLab


Meta-analyses of emotion & motivation

SCANLab


Meta-analyses of disorders

SCANLab


Meta-analyses of language

SCANLab


Meta-analyses of other stuff

SCANLab


Using meta-analysis to evaluate consistency:

Why?

SCANLab


Locating emotion-responsive regions

164 PET/fMRI studies, 437 activation maps, 2478 coordinates

SCANLab


Why identify consistent areas?

• Making statistic maps in neuroimaging studies involves many tests (~100,000 per brain map)

• Many studies use uncorrected or improperly corrected p-values

Long-term Memory

P-value thresholds used

Corr.

# of

Map

s

Uncorrected

How many false positives?A rough estimate: 663 peaks, 17% of reported activations

Wager, Lindquist, & Kaplan, 2007

SCANLab


Consistency

Reported peaks163 studies

ConsistentlyActivatedregions

Emotion: 163 studies

SCANLab


mTC

pOFC

vmPFC

BF

aINSsTC

latOFClFG TC

pgACCrdACC

dmPFCPCC

OCC

sgACC

vmPFC

CM, MD

Deep nuclei

Gyrus rectusCentral sulcus

dmPFC Pre SMA

Fig 4: MKDA Results

Ventral surfaceLateral surface (R)Medial surface (L)

Kober et al., in press, NI

SCANLab


Using meta-analysis to evaluate specificity:

Why?

SCANLab


Disgust responses: Specificity in insula?

Insula

SCANLab


Disgust responses: Specificity in insula?

Feldman-Barrett & Wager, 2005; Phan, Wager, Taylor, & Liberzon, 2002;Phan, Wager, Liberzon & Taylor, 2004

Search Area: Insula

SCANLab


Meta-analysis plays a unique role in answering…

• Is it reliable?– Would each activated region replicate in future studies?– Would activation be insensitive to minor variations in task

design?

• Is it task-specific? – Predictive of a particular psychological state or task type?– Diagnostic value?

The Neural Correlates of Task X

SCANLab


Using meta-analysis to evaluate consistency:

How?

SCANLab


Monte Carlo:Expected maximum proportionUnder the null hypothesis

Apply threshold

Weightedaverage

E

Damasio, 2000 Liberzon, 2000 Wicker, 2003

Peak coordinate locations (437 maps)

…

Kernel convolution

Comparison indicator maps

…

Proportion of activated Comparisons map

(from 437 comparisons)

Significant regions

Meta-analysis: Multilevel kernel density estimate (MKDE)

Wager, Lindquist, & Kaplan, 2007; Etkin & Wager, in press

Permute blobs within study maps

Permute blobs within study maps

SCANLab


MKDA: Key points

• Statistic reflects consistency across studies. Study comparison map is treated as a random effect. Peaks from one study cannot dominate.

• Studies are weighted by quality (see additional info on handouts for rationale)

• Spatial covariance is preserved in Monte Carlo. Less sensitive to arbitrary standards for how many peaks to report.

SCANLab


Whether and how to weight studies/peaks

MKDA analysis weights by sqrt(sample size) and study quality (including fixed/random effects)

€

P = CIMc

δ c N c

δ c N cc

∑

⎛

⎝

⎜ ⎜ ⎜

⎞

⎠

⎟ ⎟ ⎟c

∑

€

δc =1

€

δc = 0.75Fixed effectsRandom effects

Activation indicator (1 or 0) for map c

Study quality weightSample size for map c

Weighted proportion of activating studiesWeightedaverage

SCANLab


Monte Carlo Simulation

• Simulation vs. theory (e.g. Poisson process)

• Simulation allows:– Non-stationary spatial distribution of peaks

(clumps) under null hypothesis; randomize blob locations

– Family-wise error rate control with irregular (brain-shaped) search volume

– Cluster size inference, given primary threshold

Monte Carlo:E(max(P|H0))

SCANLab


Compare with Activation Likelihood Estimate (ALE), Kernel Density

Analysis (KDA)

Peak coordinatesCombined across studies

€

⊗

Kernel convolutionDensity kernel

ALE kernelOR

€

=

Peak density orALE map

Apply significance threshold

Significant results

Density kernel: Chein, 1998; Phan et al., 2002; Wager et al., 2003, 2004, 2007, in press

Gaussian density kernel + ALE: Turkeltaub et al., 2002; Laird et al., 2005; others

Ignores the fact that some studies report more peaks than others!

SCANLab


Comparison with other methods

• Statistic reflects consistency across studies. Study comparison map is treated as a random effect. Peaks from one study cannot dominate.

• Studies are weighted by quality

• Spatial covariance is preserved in Monte Carlo. Less sensitive to arbitrary standards for how many peaks to report.

• Peaks are lumped together, study is fixed effect. Peaks from one study can dominate, studies that report more peaks dominate.

• No weighting, or z-score weighting (problematic)

• Spatial covariance is not preserved in Monte Carlo. Effects of reporting standards large.

MKDA KDA/ALE

See handouts for more comparison points

SCANLab


ALE approach

• Treats points as if they were Gaussian probability distributions.

• Summarize the union of probabilities at each voxel: probability of any peak “truly” lying in that voxel

€

P(X1 ∪ X2...∪ Xn ) =1− P(∪X) =1− P(X1) * P(X2) * ...P(Xn )

€

P(X i ) is the probability that peak Xi lies in a given voxelThe bar indicates the complement operator

Null hypothesis: No peaks lie in voxel

€

P(∪X) = 0Alt hypothesis: At least one peak lies in voxel

€

P(∪X) ≠ 0

SCANLab


ALE meta-analysis

• Analyst chooses smoothing kernel• ALE analysis with zero smoothing:

– Every voxel reported in any study is significant in the meta-analysis

• Test case: 3-peak meta analysis, one peak activates in voxel:

€

P(X1) =1,P(X2) = 0,P(X3) = 0

€

1− Pr(∪X ) = 1− (0)* (1)* (1) = 1

ALE statistic:Highest possible value!

• In practice: 10 – 15 mm FWHM kernel

SCANLab


Comparison across methods: Inference

Property KDA ALE Multilevel KDA Kernel Spherical Gaussian SphericalInterpretation of statistic

Num nearby peaks Prob. that at least one peak nearby

Num. study maps activating nearby

Null hypothesis Peaks are not spatially consistent

No peaks truly activate

Study maps are not spatially consistent

Interpretation of significant result

More peaks lie near voxel than expected by chance

One or more peaks lies at this voxel

A higher proportion of studies activate near voxel than expected by chance

Assumptions 1. Study is fixed effect (homogenous sample of studies)2. Peaks are spatially independent under the null hypothesis

1. Study is fixed effect (homogenous sample of studies)2. Peaks are spatially independent under the null hypothesis

Activation ‘blobs’ are spatially independent under the null hypothesis

Generalize to New peaks from same studies

New peaks from same studies

New study maps

SCANLab


Comparison: Correction and Weighting

Property KDA ALE Multilevel KDA

Multiple comparisons

FWER FDR FWER (recommended) or FDR

Weighting None, or weight peaks by z-score

None Weight studies by sample size, fixed/random effects, quality

SCANLab


Density analysis: SummaryWorking memoryExecutive WMLong-term memory

Inhibition Task switching

Memory

Response selection

Wager et al., 2004; Nee, Wager, & Jonides, 2007; Wager et al., in press;

Van Snellenberg & Wager, in press

SCANLab


Using meta-analysis to evaluate specificity:

How?

SCANLab


Specificity

• Task-related differences in relative activation frequency across the brain: – MKDA difference maps (e.g., Wager et al.,

2008)

• Task-related differences in absolute activation frequency– Nonparametric chi-square maps (Wager,

Lindquist, & Kaplan, 2007)

• Classifier systems to predict task type from distributed patterns of peaks (e.g., Gilbert)

SCANLab


MKDA Difference maps: Emotion example

• Approach: – Calculate density maps for two conditions, subtract to get

difference maps– Monte Carlo: Randomize blob locations within each study,

re-calculate density difference maps and save max– Repeat for many (e.g., 10,000) iterations to get max

distribution– Threshold based on Monte Carlo simulation

Experienced

Perceived

SCANLab


Emotion example: Selective regions

AmyTP OFC

Amy

aIns

Experience > Perception

aIns

OFC

vaIns

dmPFC

Hy vaIns

TP

PAG

PAG

Midb

mOFC

TP

OFC

OFC

Midb

TP

Hy

Hy

Perception > Experience

pgACC

Amy

CB

IFG

IFGAmy

CB

Wager et al., in press, Handbook of Emotion

SCANLab


Task-brain activity associations in meta-analysis

Study contrast map

Region/Voxel 1

Task condition

Study 1 1 Disgust

Study 2 0 Fear

Study 3 1 Disgust

Study 4 1 Happiness

Study 5 0 Anger

… … …

Study N 0 Sadness

Measures of association:Chi-square• But requires high expected counts (> 5) in each cell. Not appropriate for map-wise testing over many voxelsFisher’s exact test (2 categories only)Multinomial exact test• Computationally impractical!Nonparametric chi-square• Approximation to exact test• OK for low expected counts

SCANLab


Nonparametric chi-square: Details

Study contrast map

Region/Voxel 1

Task condition

Study 1 1 Disgust

Study 2 0 Fear

Study 3 1 Disgust

Study 4 1 Happiness

Study 5 0 Anger

… … …

Study N 0 Sadness

Idea of exact test: • Conditionalize on marginal counts for activation and task conditions. • Null hypothesis: no systematic association between activation and task• P-value is proportion of null-hypothesis possible arrangements that can produce distribution across task conditions as large as observed or larger.

SCANLab


Nonparametric chi-square: Details

Study contrast map

Region/Voxel 1

Task condition

Study 1 0 Disgust

Study 2 1 Fear

Study 3 0 Disgust

Study 4 1 Happiness

Study 5 0 Anger

… … …

Study N 1 Sadness

Permutation test:• Permute activation indicator vector, creating null-hypothesis data (no systematic association)

• Marginal counts are preserved. • Test 5,000 or more samples and calculate P-value based on observed null-hypothesis distribution

SCANLab


Density difference vs. Chi-square

• Relative vs. absolute differences

Voxels (one-dimensional brain)

ExperiencePerception

Chi-square

Density

SCANLab


Can we predict the emotion from the pattern of brain activity?

• Approach: predict studies based on their pattern of reported peaks (e.g., Gilbert, 2006)

• Use naïve Bayesian classifier (see work by Laconte;

Tong; Norman; Haxby). Cross-validate: predict emotion type for new studies that are not part of training set.

Experienced

Perceived

SCANLab


Classifying experienced emotion vs. perceived emotion: 80% accurate

Exp

eri

en

ceP

erc

ep

tion

PAG vs. Ant. thalamus

Deep cerebellar nuc. vs. Lat. cerebellum

DMPFC vs. Pre-SMA

EXP vs. PER

DMPFC

EXP

PAG

Deep cerebellar nuc.

SCANLab


Outline: Why and How…

• Consistency: Replicability across studies– Consistency in single-region results: MKDA– Consistency in functional networks: MKDA + Co-

activation

• Specificity and “reverse inference”– Brain-activity – psychological category mappings

for individual brain regions: MKDA difference maps; Nonparametric Chi-square

– Brain-activity – psychological category mappings for distributed networksApplying classifier systems to meta-analytic data

SCANLab


Extending meta-analysis to connectivity

Study contrast map

Region/Voxel 1

Region/Voxel 2

Study 1 1 0

Study 2 0 0

Study 3 1 1

Study 4 1 1

Study 5 0 0

… … …

Study N 0 1

Co-activation: If a study (contrast map) activates within k mm of voxel 1, is it more likely to also activate within k mm of voxel 2?

Measures of association:Kendall’s Tau-bFisher’s exact testNonparametric chi-square

Others…

N = 45 Region 1No

Region 1 Yes

Region 2 Yes

6 23

Region 2No

12 4

SCANLab


Kendall’s Tau: Details• Ordinal “nonparametric” association between

two variables, x and y• Uses ranks; no assumption of linearity or

normal distribution (Kendall, 1938, Biometrika)• Values between [-1 to 1], like Pearson’s

correlation

€

τ =4 min(rank(x),rank(y)

i=1

N−1

∑ ) > i

N(N −1)

⎛

⎝

⎜ ⎜ ⎜ ⎜

⎞

⎠

⎟ ⎟ ⎟ ⎟

−1

Tau is proportion of concordant pairs of observations sign(x diff. between pairs)= sign(y diff. between pairs)Tau = (# concordant pairs - # discordant pairs) / total # pairs

SCANLab


Meta-analysis functional networks: Examples

• Emotion: Kober et al. (in press), 437 maps

SCANLab


Acknowledgements

Ed SmithEd Smith

Funding agencies: National Science FoundationNational Institute of Mental Health

Martin LindquistMartin Lindquist

Derek NeeDerek NeeJohn JonidesJohn JonidesEd SmithEd Smith

Tom NicholsTom Nichols

Lisa Feldman Lisa Feldman BarrettBarrett

Hedy KoberLauren KaplanJason BuhleJared Van Snellenberg

Luan PhanLuan PhanSteve TaylorSteve TaylorIsrael LiberzonIsrael Liberzon

Meta-analysis of emotion

StatisticsMeta-analysis of cognitive

function

Students

SCANLab


Weighting

SCANLab



• Studies (and peaks) differ in sample size, methodology, analysis type, smoothness, etc.

• Advantageous to give more weight to more reliable studies/peaks

• Z-score weighting– Advantages: Weights nominally more

reliable peaks more heavily– Disadvantages: Small studies can produce

variable results. Reporting bias: High z-score peaks are high partially due to error; “capitalizing on chance”• Must convert to common Z-score metric across

different analysis types in different studies

SCANLab



• Alternative: Sample-size weighting– Advantages:

• Weights studies by the quality of information their peaks are likely to reflect

• Avoids overweighting peaks reported due to “capitalizing on chance”

– Disadvantages: Ignores relative reliability of various peaks within studies

SCANLab


MKDA vs. KDA vs. ALE:Comparison chart

SCANLab


More details on reverse inference

SCANLab


Is brain activity diagnostic of a particular psychological state?

‘Forward’ and ‘reverse’ inference are not the same!Reverse inference requires comparing across many psychological states!

Pleasure?

Punishing wrongdoersBrain activity

Given a psychological state

We observe brain activity

P(Brain | Psy)Forward inference

Can we infer psychological pleasure?

P(Psy | Brain)Given brain activity

Reverse inference

SCANLab


The predictive value problem: Worked example

For a brain region to be used as a marker of pleasure

– The brain region must respond consistently to pleasure

– The brain region must respond specifically to pleasure (not activated by other things)

Ventral caudatePleasure

P(Brain|Pleasure) = .9Forward inference; Sensitivity

Non-pleasure

P(Brain|no pleasure) = .41-Specificity

P(pleasure) = .1

Prior

Caculate reverse inference:

P(Pleasure|Brain) = .2

SCANLab


More details on connectivity

SCANLab


More details on MKDA difference maps and nonparametric chi-square

maps

scanlab meta-analysis of neuroimaging data what, why, and how tor d. wager columbia university

Documents