![Page 1: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/1.jpg)
fMRI Techniques to Investigate Neural Coding:
fMRA and MVPA
http://www.fmri4newbies.com/
Last Update: January 18, 2012Last Course: Psychology 9223, W2010, University of Western OntarioLast Update: January 18, 2012
Last Course: Psychology 9223, W2010, University of Western Ontario
Jody CulhamBrain and Mind Institute
Department of PsychologyUniversity of Western Ontario
![Page 2: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/2.jpg)
Limitations of Subtraction Logic• Example: We know that neurons in the brain can be tuned
for individual faces
“Jennifer Aniston” neuron in human medial temporal lobeQuiroga et al., 2005, Nature
![Page 3: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/3.jpg)
Limitations of Subtraction LogicF
irin
g R
ate
Fir
ing
Rat
e
Fir
ing
Rat
e
Act
iva
tion
Neuron 1“likes”
Jennifer Aniston
Neuron 2“likes”
Julia Roberts
Neuron 3“likes”
Brad Pitt Even though there are neurons tuned to each object, the population as a whole shows no preference
• fMRI resolution is typically around 3 x 3 x 6 mm so each sample comes from millions of neurons. Let’s consider just three neurons.
![Page 4: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/4.jpg)
Two Techniques with “Subvoxel Resolution”
• “subvoxel resolution” = the ability to investigate coding in neuronal populations smaller than the voxel size being sampled
1. fMR Adaptation (or repetition suppression or priming)
2. Multivoxel Pattern Analysis (or decoding)
![Page 6: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/6.jpg)
fMR Adaptation
• If you show a stimulus twice in a row, you get a reduced response the second time
Repeated
FaceTrial
Unrepeated
FaceTrial
Time
Hypothetical Activity inFace-Selective Area (e.g., FFA)
Act
ivat
ion
![Page 7: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/7.jpg)
500-1000 msec
fMRI Adaptation
Slide modified from Russell Epstein
“different” trial:
“same” trial:
![Page 9: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/9.jpg)
Why is adaptation useful?
• Now we can ask what it takes for stimulus to be considered the “same” in an area
• For example, do face-selective areas care about viewpoint?
TimeA
ctiv
atio
n
Repeated Individual, Different Viewpoint
Viewpoint invariance:• area codes the face as the same despite the viewpoint change
Viewpoint selectivity:• area codes the face as different when viewpoint changes
![Page 11: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/11.jpg)
And more…
• We could use this technique to determine the selectivity of face-selective areas to many other dimensions
Repeated Individual, Different
Expression
Repeated Expression,
Different Individual
![Page 13: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/13.jpg)
Evidence for “Fatigue” Model
Data from: Li et al., 1993, J NeurophysiolFigure from: Grill-Spector, Henson & Martin, 2006, TICS
![Page 17: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/17.jpg)
fMRA Does Not Accurately Reflect Tuning
• MT+: most neurons are direction-selective (DS), high DS in fMRA
• V4: few (20%?) neurons are DS, very high DS in fMRA
• perhaps fMRA is more driven by inputs than outputs?
Tolias et al., 2001, J. Neurosci
![Page 19: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/19.jpg)
Basic Assumption/Hypothesis
• if a neuronal population responds equally to two stimuli, those stimuli should yield cross-adaptation
Ne
ura
l Re
spo
nse
Pre
dict
ed
fMR
I Re
spo
nse
A B C A-A B-B A-B C-A
![Page 20: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/20.jpg)
Experimental Question
• the human lateral occipital complex (LOC) is arguably analogous/homologous to macaque inferotemporal (IT) cortex
• both human LOC and macaque IT show fMRI adaptation to repeated objects
• Does neurophysiology in macaque IT show object adaptation at the single neuron level?
![Page 21: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/21.jpg)
Experiment 1Block Design Adaptation
Experiment 2Event-Related
Adaptation
Design
Sawamura et al., 2006, Neuron
![Page 23: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/23.jpg)
… but cross-adaptation is less clear
BLOCK
EVENT-RELATED
EXAMPLE A-A ADAPTA=B
B-A ADAPTA=B
WHOLEPOPULATION
A-AB-BC-AB-A
Sawamura et al., 2006, Neuron
![Page 24: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/24.jpg)
Sawamura et al. Conclusions
• Evidence for adaptation at the single neuron level is clear
• Cross-adaptation is not as strong as expected, particularly for event-related designs
• They don’t think it’s just attention• Something special about repeated stimuli
![Page 26: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/26.jpg)
Design
REP BLOCK (75% rep trials, 25% alt trials)AA BB CD EE FF GH II JJ…
ALT BLOCK (25% rep trials, 75% alt trials)AB CC DE FG HI JK LM NN…
Task: press
button for inverted
face
Summerfield et al., 2008, Nat Neurosci
![Page 27: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/27.jpg)
Results
IndividualFFAROIs
SIG INTERACTION:stronger fMRA in blocks with freq.
reps
22%p<.001
9%p<.05
Summerfield et al., 2008, Nat Neurosci
![Page 28: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/28.jpg)
Replication
• results were replicated with a different task
Task: press
button for small face
Summerfield et al., 2008, Nat Neurosci
![Page 29: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/29.jpg)
New Explanation of fMRA
• “repetition suppression reflects a reduction in perceptual ‘prediction error’”
• mismatch between expectations and stimulus increases fMRI activation
• mismatch is higher on novel trials than repetition trials
![Page 30: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/30.jpg)
Additional Caveats• Adaptation effects can be quite unreliable
– variability between labs and studies– even effects that are well-established in neurophysiology and
psychophysics don’t always replicate in fMRA• e.g., orientation selectivity in primary visual cortex
– David Heeger suggests that it may be critical to control attention
• The effect may also depend on other factors– e.g., time elapsed from first and second presentation
• days, hours, minutes, seconds, milliseconds?• number of intervening items
– attention (especially in block designs)– memory encoding
• Different areas may demonstrate fMRA for different reasons– reflected in variety of terms: repetition suppression, priming
![Page 31: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/31.jpg)
So is fMRA dead? No.Criticism: fMRA may reflect inputs rather than outputs• Response: This is a general caveat of all fMRI studies.
Inputs are interesting too, just harder to interpret. Focus on outputs oversimplifies neural processing when presumably feedback loops are an essential component.
Criticism: fMRA may not reveal cross-adaptation even in populations that do show cross-coding
• Response: This suggests that caution is especially warranted when there is a failure to find adaptation (or a finding of “recovery from adaptation”). However, cross-adaptation can occur and is meaningful when it does. Many past fMRA studies have found it.
![Page 32: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/32.jpg)
So is fMRA dead? No.Criticism: None of the basic models of fMRA seem to work.• Response: In some ways, it doesn’t matter. The essential
use of fMRA is to determine whether neural populations are sensitive to stimulus dimensions. The exact mechanism for such sensitivity may not be critical.
Criticism: fMRA, and maybe fMRI in general, is just responding to predictions.
• Response: Prediction is interesting too. Regarding fMRA, why do some brain areas make predictions about a stimulus while others don’t?
![Page 34: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/34.jpg)
3 mm
3 mm
Voxels
• Modern scanner can collect ~150,000 voxels in 2 s
3 mm
lowactivity
highactivity
![Page 35: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/35.jpg)
“Movement 1” or
“Movement 2” “Beep”
Next trialPreview Plan Execute ITI
Light
Difficulty with Standard fMRI analysisB
rain
Act
ivat
ion
(% B
SC
)
Time (seconds)
RLMovement 1Movement 2
![Page 36: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/36.jpg)
RL
3 mm
3 mm
Voxel Pattern Information
Movement 1 Movement 23 mm
![Page 37: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/37.jpg)
Standard Analysis
trial 1
trial 3
trial 2
trial 1
trial 2
trial 3
Movement 1 Movement 2
AverageSummedActivation
VoxelwiseActivityin ROI
![Page 38: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/38.jpg)
Spatial Smoothing
• most conventional fMRI studies spatially smooth (blur) the data– increases signal-to-noise– facilitates intersubject averaging
• loses information about the patterns across voxels
No smoothing 4 mm FWHM 7 mm FWHM 10 mm FWHM
![Page 39: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/39.jpg)
Effect of Spatial Smoothingand Intersubject Averaging
3 mm
3 mm
3 mm
![Page 40: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/40.jpg)
Perhaps voxels contain useful information
• In traditional fMRI analyses, we average across the voxels within an area, but these voxels may contain valuable information
• In traditional fMRI analyses, we assume that an area encodes a stimulus if it responds more, but perhaps encoding depends on pattern of high and low activation instead
• But perhaps there is information in the pattern of activation across voxels
![Page 41: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/41.jpg)
Multi-voxel pattern analysis (MVPA)
TrainingTrials
TestTrials
(not in training set)
trial 1
Can an algorithm correctly “guess” trial identity better than
chance (50%)?
trial 3
trial 2
trial 1
trial 2
trial 3
Movement 1 Movement 2
![Page 42: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/42.jpg)
Coding in Voxel Patterns
• Simple experiment: Show subjects pictures of different objects (e.g., shoes vs. bottles) on different trials of different runs
![Page 43: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/43.jpg)
Simple Correlation Analysis
• Measure within-category correlations– within bottles (B1:B2)– within shoes (S1:S2)
• Measure between-category correlations– between bottles: shoes (B1: S2; S1: B2)
• If within-category correlations > between-category correlations, conclude that area encodes different stimuli
![Page 48: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/48.jpg)
Decoding Algorithms
• Train algorithm to distinguish two object categories on a training set
• Test success of algorithm on distinguishing two object categories on a test set
• If algorithm succeeds better than chance, conclude that area encodes different stimuli
Norman et al., 2006, Trends Cogn. Sci.
![Page 49: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/49.jpg)
MVPA Methods
• block or event-related data• resolution
– works even with moderate resolution (e.g., 3 mm isovoxel)
– tradeoff between resolution and coverage, SNR– 2 mm isovoxel recommended at 3 T– preprocessing
• usually steps apply (slice scan time correction, motion correction, low pass temporal filter)– EXCEPT: No spatial smoothing!
• Model single subjects, not combined group data (at least initially)
![Page 50: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/50.jpg)
MVPA Methods
1. separate data into independent training and test sets– e.g., even and odd runs
e.g., iterate sequence of “leave one run out”
2. pick the area to analyze– ROI localizer– contrast in training set
3. train the classifier– input: beta weights from each voxel in area– variety of classifiers available– e.g., linear support vector machine
4. test the classifier– does classifier perform better than chance?
• e.g., chi-squared test
Summarized from Mur et al., 2009, Social Cognitive and Affective Neuroscience
![Page 51: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/51.jpg)
simple 2D example
Classifier can act on single voxels. Conventional fMRI analysis would detect the difference.
Classifier would require curved decision boundary
Classifier can not act on single voxels because distributions overlapClassifier can act on combination of voxels using a linear decision boundary
Each dot is one measurement (trial) from one trial type (red circles) or the other (blue squares)
decision boundary
White and black circles show examples of correct and erroneous classification in the test set
9 voxels 9 dimensions Haynes & Rees, 2006, Nat Rev Neurosci
![Page 52: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/52.jpg)
How can MVPA see patterns < 1 voxel?
Data from: Kamitami & Tong, 2005, Nat NeurosciFigure from: Norman et al., 2006, TICS
![Page 53: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/53.jpg)
MVPA Searchlight
• define a spherical searchlight– optimal searchlight has radius = 4 mm
• contains 33 2-mm-isovoxel voxels
• compute multivariate effect within all possible locations within brain volume
• calculate voxelwise p values and threshold them at false discovery rate q values
Kriegeskorte, Goebel & Bandettini, 2006, PNAS
![Page 56: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/56.jpg)
Does MVPA (decoding) make fMRA obsolete?
• MVPA allows us to address similar questions about what is coded in an area.
• MVPA may have some advantages (e.g., less susceptible to attentional confounds)
• MVPA utility depends on numerous factors (e.g., region size… are there enough voxels to get a meaningful pattern)
• MVPA requires clustering of neural populations and is sensitive to scanning parameters (voxel size); fMRA does not
• MVPA has the same problem as fMRA: it’s very hard to draw conclusions from a null result
![Page 57: FMRI Techniques to Investigate Neural Coding: fMRA and MVPA Last Update: January 18, 2012 Last Course: Psychology 9223, W2010,](https://reader036.vdocuments.net/reader036/viewer/2022062500/5697bfe21a28abf838cb4160/html5/thumbnails/57.jpg)
Activation vs. Patterns
Mur et al., 2009, Social Cognitive and Affective Neuroscience