reverse correlation methods for social cognition...reverse correlation methods for social cognition...
TRANSCRIPT
Reverse Correlation Methods for Social Cognition
Ron Dotsch
Daniel WigboldusAd van Knippenberg
Gent, March 9, 2010
Saturday, November 6, 2010
Interspecies differences in face categorization
Saturday, November 6, 2010
Reverse Correlation
• Classification image technique (Ahumada, 1996)
Saturday, November 6, 2010
Reverse Correlation
• Classification image technique (Ahumada, 1996)
• Responses to stimuli in experiment contain information about how task is performed
Saturday, November 6, 2010
Reverse Correlation
• Classification image technique (Ahumada, 1996)
• Responses to stimuli in experiment contain information about how task is performed
• Randomly (instead of systematically) vary stimuli on a lot of (instead of specific) dimensions
Saturday, November 6, 2010
Reverse Correlation
• Classification image technique (Ahumada, 1996)
• Responses to stimuli in experiment contain information about how task is performed
• Randomly (instead of systematically) vary stimuli on a lot of (instead of specific) dimensions
• Use correlation between stimulus features and response pattern to infer decision criteria (classification image)
Saturday, November 6, 2010
Reverse correlation image classification task(Dotsch, Wigboldus, Langner, & van Knippenberg, 2008; Mangini & Biederman, 2004)
Reverse correlation task
Saturday, November 6, 2010
Reverse correlation image classification task(Dotsch, Wigboldus, Langner, & van Knippenberg, 2008; Mangini & Biederman, 2004)
Reverse correlation task
Saturday, November 6, 2010
Reverse correlation image classification task(Dotsch, Wigboldus, Langner, & van Knippenberg, 2008; Mangini & Biederman, 2004)
Reverse correlation task
Saturday, November 6, 2010
Reverse correlation task
Base + noise Base - noise
A 6
Saturday, November 6, 2010
Classification images
MoroccanChinese
Saturday, November 6, 2010
Classification images
MoroccanChinese
Saturday, November 6, 2010
Classification images
MoroccanChinese
Saturday, November 6, 2010
y = sin x y = sin x at 90º
Noise generation
Saturday, November 6, 2010
y = sin x y = sin x at 90º
Noise generation
Saturday, November 6, 2010
Noise generation212 M.C. Mangini, I. Biederman / Cognitive Science 28 (2004) 209–226
Fig. 1. Creating the noise. (a) Six orientations and two phases (cosine upper row; sine lower row) of randomamplitude sinusoids are summed to create the 2 cycles/image noise. The same process is repeated four times foreach “tile” of the 4 cycles/image noise, 16 times for the 8 cycles/image noise, etc. (b) All five octaves are summed tocreate the noise pattern. To the right are shown the 2, 4, 8, 16, and 32 cycles/image noise patterns of all orientations.
visible in photographs. Image morphing software (Gryphon Morph 2.5) was used to morpheach model to, two arbitrarily chosen models of the same gender, resulting in a total of 200images. Taking the mathematical average of these 200 faces resulted in the base image.1 Forthe individuation task, a morph of images of John Travolta and Tom Cruise was used as thebase image. The base images are shown in the first column of Fig. 3.For each trial a random noise stimulus was generated. Although most experiments using the
response classification technique have used Gaussian white pixel noise, our noise was com-posed of truncated sinusoids that are nominally localized in space, frequency, orientation, andphase.2 Each sinusoid consisted of two cycles of a sine wave in a square envelope. Sinusoidalpatches at five octave scales (2, 4, 8, 16, and 32 cycles/image), six orientations (0, 30, 60, 90,120, and 150◦), and two phases (0 and π/2) were summed to create one noise pattern (Fig. 1).The resulting noise pattern was described by 4096 parameters that corresponded to the ampli-tudes of the sinusoids. The amplitude parameters were selected randomly from a zero meanuniform distribution. The separate scales were multiplied by constant factors that insured thatthe range of each scale did not exceed the dynamic range available for the noise. These factorsalso contributed to the frequency profile presented in Fig. 5. A justification of why this noisewas chosen is presented in Section 3.Image creation and presentation was performed on a Macintosh G4 computer using Matlab
with the Psychophysics and VideoToolbox extensions (Brainard, 1997; Pelli, 1997).
2.1.3. ProcedureSubjectswere assigned to one of three discrimination tasks: gender, expression (happyversus
unhappy), or identity (John Travolta versus Tom Cruise). At the beginning of the experimentthe subjects were told that they would be making a simple discrimination but that it wouldbe difficult to perform. For the Gender and Expression tasks subjects were told that a set of
4092 parameters (contrasts/amplitudes)Saturday, November 6, 2010
Mangini & Biederman (2004)M.C. Mangini, I. Biederman / Cognitive Science 28 (2004) 209–226 215
Fig. 3. The results from the three tasks based on the high confidence “probably” trials (a) Happy/unhappy, (b)male/female, (c) Tom Cruise/John Travolta. The base images were identical for the expression and gender tasks.The darkest and lightest areas of the classification images, which are from the data of all 36 subjects, indicate theareas that most influenced the subjects’ classifications. The addition of the classification image to the base faceresults in Class Image 1, which appears happy. The subtraction of the classification image results in Class Image2, which appears unhappy. The same addition and subtraction operations produce the class images for (b) femaleand male and (c) Cruise and Travolta, respectively. The rightmost two columns show the classification images forthe median subject calculated in terms of Euclidean pixel distance for a given subject’s classification image and theaverage classification image.
Gender in Columns 3 and 4, it becomes clear that while the eyes and mouth play a large role,the diffuse energy in the center of the face creates distinctive gender changes of the nose. Thisillustrates how this technique can discover the diffuse, subtle information employed by faceperceivers.A classification image can be constructed from only those sinusoidal components that dif-
fered significantly between the two categories on each task. Significance was tested withrepeated independent t tests for each of the 4092 components with adjustment for multiplecomparisons (Rom, 1990). For the Expression task, 187 components reached significance(p < .0005), for Gender, 85 components (p < .001), and for the celebrity identity task, 52components (p < .001). The images in Fig. 4 reveal that, indeed, relatively few components,in the order of 100, are adequate to recreate the class differences.Valentin et al. (1994) and Sergent (1989) have speculated, on the basis of statistical analysis
of sets of faces, that whereas gender and expression can be conveyed by low frequency infor-mation, individuation is carried in higher frequency channels. Fig. 5 shows the class imagesfor all sinusoids separately for each of the five scales. For all tasks, including individuation, alarge portion of the information distinguishing the classes appears to be at 4 and 8 cycles/image.This shows that human observers categorizing faces by identity do chose to make use of lowfrequency information in performing their categorizations. Furthermore, this low frequency in-
Saturday, November 6, 2010
Humans vs. Baboons
Martin-Malivel, Mangini, Fagot, and Biederman (2006)
Saturday, November 6, 2010
Humans vs. Baboons
Next, to determine the similarities between the baboons’ andhumans’ use of information, we compared 1,000 bootstrapsamples pulled from each baboon’s data with bootstrap data from
each of the six leave-one-out human populations. Averagingacross the 6,000 similarity measures gave the average similaritybetween a baboon’s information use and the humans’ informa-tion use. The percentile rank of the average similarity value wasless than 0.5% (for B06) and 0.8% (for B07). The across-speciesdifferences were significantly greater than the within-humansdifferences (B06: prep 5 .966; B07: prep 5 .956).
Principal Component Analysis (PCA)Although the bootstrap analysis provided a test for the plannedcomparison of baboons’ and humans’ information use, we alsosubjected the CIs (unlabeled as to which species they came from)to a PCA. The PCA extracted the dimensions accounting for thegreatest variance among the individuals’ CIs. The first principalcomponent answered the question: What is the greatest differ-ence among the eight CIs? The first step in the PCA was tosubtract themean of all the CIs from all the images. This was doneso that the first principal component would represent the di-mension that accounted for the most variance among the images(rather than the mean of the images themselves). The principalcomponents were computed as the eigenvectors of the innerproduct of thematrix containing all of themean-subtractedCIs asvectors. The eigenvalues of these components represent theamount of variance accounted for by each principal component.Results showed that the first principal component (after the
mean of all subjects had been subtracted) accounted for 20% ofthe variance (chance 5 14%). Because there is no standardmethod for testing the statistical significance of a principalcomponent, we performed a Monte Carlo simulation. The nullhypothesis tested was that the eight human and baboon CIs wereno more redundant than a set of eight ‘‘random observer’’ CIs.These random CIs were generated by taking 1,000 randomlygenerated noise patterns and randomly assigning them to the‘‘human’’ and ‘‘baboon’’ categories without any consideration oftheir visual structure. The eigen-decomposition of the eightrandom CIs was performed as described for the human andbaboon data. The eigenvalues for the random observers wererecorded, and the process was repeated 10,000 times to obtaina null distribution. The eigenvalue of the first principal compo-nent of our human-baboon PCAwas greater than the eigenvaluesof all 10,000Monte Carlo simulations (prep5 .996), showing thatthere was more redundancy in the eight human and baboon CIsthan could be expected by chance. In other words, the 20% ofvariance accounted for by the first principal component repre-sents a significantly reliable difference among the subjects.Figure 4a shows the correlation between each subject’s CI and
the first principal component. This component clearly splits thesubjects into two distinct groups: humans and baboons. Theimages to the left of the graph illustrate the information subjectsused to classify the images as human (left) and baboon (right).The top pictures (row 1) illustrate the information that humanstended to use, and the bottom pictures illustrate the information
Fig. 3. Individual results. The column on the left shows the classificationimage (CI) for each human (H1–H6) or baboon (B06, B07) subject. EachCI was obtained by subtracting the average of the noise patterns that in-duced the subject to respond ‘‘baboon’’ from the average of the noisepatterns that induced the subject to respond ‘‘human.’’ The CIs presentedhere show the information that each subject used to categorize the picturesas ‘‘human.’’ The middle column shows the result of adding the CI to theoriginal ambiguous morph, thus re-creating the typical image elicitinga ‘‘human’’ response. The column on the right shows the result of sub-tracting the CI from the original ambiguous morph, thus re-creating thetypical image eliciting a ‘‘baboon’’ response.
Volume 17—Number 7 603
J. Martin-Malivel et al.
Martin-Malivel, Mangini, Fagot, and Biederman (2006)Next, to determine the similarities between the baboons’ andhumans’ use of information, we compared 1,000 bootstrapsamples pulled from each baboon’s data with bootstrap data from
each of the six leave-one-out human populations. Averagingacross the 6,000 similarity measures gave the average similaritybetween a baboon’s information use and the humans’ informa-tion use. The percentile rank of the average similarity value wasless than 0.5% (for B06) and 0.8% (for B07). The across-speciesdifferences were significantly greater than the within-humansdifferences (B06: prep 5 .966; B07: prep 5 .956).
Principal Component Analysis (PCA)Although the bootstrap analysis provided a test for the plannedcomparison of baboons’ and humans’ information use, we alsosubjected the CIs (unlabeled as to which species they came from)to a PCA. The PCA extracted the dimensions accounting for thegreatest variance among the individuals’ CIs. The first principalcomponent answered the question: What is the greatest differ-ence among the eight CIs? The first step in the PCA was tosubtract themean of all the CIs from all the images. This was doneso that the first principal component would represent the di-mension that accounted for the most variance among the images(rather than the mean of the images themselves). The principalcomponents were computed as the eigenvectors of the innerproduct of thematrix containing all of themean-subtractedCIs asvectors. The eigenvalues of these components represent theamount of variance accounted for by each principal component.Results showed that the first principal component (after the
mean of all subjects had been subtracted) accounted for 20% ofthe variance (chance 5 14%). Because there is no standardmethod for testing the statistical significance of a principalcomponent, we performed a Monte Carlo simulation. The nullhypothesis tested was that the eight human and baboon CIs wereno more redundant than a set of eight ‘‘random observer’’ CIs.These random CIs were generated by taking 1,000 randomlygenerated noise patterns and randomly assigning them to the‘‘human’’ and ‘‘baboon’’ categories without any consideration oftheir visual structure. The eigen-decomposition of the eightrandom CIs was performed as described for the human andbaboon data. The eigenvalues for the random observers wererecorded, and the process was repeated 10,000 times to obtaina null distribution. The eigenvalue of the first principal compo-nent of our human-baboon PCAwas greater than the eigenvaluesof all 10,000Monte Carlo simulations (prep5 .996), showing thatthere was more redundancy in the eight human and baboon CIsthan could be expected by chance. In other words, the 20% ofvariance accounted for by the first principal component repre-sents a significantly reliable difference among the subjects.Figure 4a shows the correlation between each subject’s CI and
the first principal component. This component clearly splits thesubjects into two distinct groups: humans and baboons. Theimages to the left of the graph illustrate the information subjectsused to classify the images as human (left) and baboon (right).The top pictures (row 1) illustrate the information that humanstended to use, and the bottom pictures illustrate the information
Fig. 3. Individual results. The column on the left shows the classificationimage (CI) for each human (H1–H6) or baboon (B06, B07) subject. EachCI was obtained by subtracting the average of the noise patterns that in-duced the subject to respond ‘‘baboon’’ from the average of the noisepatterns that induced the subject to respond ‘‘human.’’ The CIs presentedhere show the information that each subject used to categorize the picturesas ‘‘human.’’ The middle column shows the result of adding the CI to theoriginal ambiguous morph, thus re-creating the typical image elicitinga ‘‘human’’ response. The column on the right shows the result of sub-tracting the CI from the original ambiguous morph, thus re-creating thetypical image eliciting a ‘‘baboon’’ response.
Volume 17—Number 7 603
J. Martin-Malivel et al.
Saturday, November 6, 2010
In more detail
that the baboons used (row 3). Inspection of these images revealsthat the baboons categorized the faces mainly on the basis ofcontrast between the eyes and the surrounding head region,responding ‘‘human’’ in the case of a lighter eye area and ‘‘ba-boon’’ in the case of a darker eye area, whereas the humans paidattention to a broader range of facial features, such as the shapeof the nose and mouth and facial contour.
DISCUSSION
The results demonstrate that different species given identicaltraining regimens can utilize substantially different information
to achieve comparable performance. This difference could notbe discovered from experiments in which only the performanceover all trials was computed, without considering the responsesto individual trials. By correlating observers’ responses to ran-dom noise patterns trial by trial, we were able to determine thatthe two species used different information to achieve successfulcategorization. The PCA shows that the largest difference amongthe CIs was the difference between the CIs of humans andbaboons. Both baboons relied heavily on the coarse contrastbetween the eyes and the surrounding face, unlike humans, whorelied on both coarse and detailed information across the entireface.
Fig. 4. Results of the principal component analyses (a) for human and baboon subjects and (b) for human andbaboon subjects and two runs of the theoretical observer (TO). The individual bars represent the correlationsbetween each subject’s classification image and the first eigenvector. The images along the ordinate of each graphillustrate the dimension accounting for the greatest variance among subjects. The three rows depict, respectively,the information used by subjects with high positive projections (11) on the first principal component, by subjectsat the midpoint (0 projection), and by subjects with high negative projections (!1) to categorize faces as human(left) or baboon (right). The subjects with high positive projections tended to use many of the finer details acrossthe entire face, including the eyes, nose, lips, and jawline, whereas those with high negative projections tended touse a coarse contrast between the eyes and the surrounding face to make their discriminations.
604 Volume 17—Number 7
Interspecies Differences in Categorizing Faces
Saturday, November 6, 2010
Reverse Correlation History
• Auditory work (Ahumada & Lovell, 1971)
• Low level visual perception: humans differ from ideal observers (Ahumada, 1996; Beard & Ahumada, 1998)
• Sparked host of studies on visual perception:
• Letter discrimination (Watson & Rosenholtz, 1997)
• Perceptual learning (Knoblauch, Thomas, & D’Zmura, 1999)
• Illusory contours (Gold, Murray, Bennett, & Sekuler, 2000)
• Many more! (see Eckstein & Ahumada, 2002)
Saturday, November 6, 2010
• Higher order visual perception (face processing):
Reverse Correlation History
Saturday, November 6, 2010
• Higher order visual perception (face processing):
• Bubbles (Gosselin & Schyns, 2001)
Reverse Correlation History
Saturday, November 6, 2010
• Higher order visual perception (face processing):
• Bubbles (Gosselin & Schyns, 2001)
• Mangini & Biederman (2004)
Reverse Correlation History
Saturday, November 6, 2010
• Higher order visual perception (face processing):
• Bubbles (Gosselin & Schyns, 2001)
• Mangini & Biederman (2004)
• And now: Social cognition?
Reverse Correlation History
Saturday, November 6, 2010
Visualizing bias
Dotsch, Wigboldus, Langner, & van Knippenberg (2008)
Saturday, November 6, 2010
Chinese, Japanese, or Korean?
Saturday, November 6, 2010
Japanese
Chinese, Japanese, or Korean?
Saturday, November 6, 2010
Chinese, Japanese, or Korean?
Saturday, November 6, 2010
Chinese, Japanese, or Korean?
Chinese
Saturday, November 6, 2010
Chinese, Japanese, or Korean?
Saturday, November 6, 2010
Chinese
Chinese, Japanese, or Korean?
Saturday, November 6, 2010
Image classification task
MoroccanChinese
Saturday, November 6, 2010
Image classification task
Low Moderate High
Saturday, November 6, 2010
Image classification task
Low Moderate High
Saturday, November 6, 2010
Part 2:Trait ratings
-2
-1
0
1
2
Low Moderate High
ST IAT
(N = 70)
-2
-1
0
1
2
Low Moderate High
ST IAT
F(2, 68) = 43.95, p < .001, partial η! = .56F(2, 68) = 39.00, p < .001, partial η! = .53
Criminal Trustworthy
Saturday, November 6, 2010
Criminal Classification Image
Saturday, November 6, 2010
Moroccan + Criminal
+ =
Saturday, November 6, 2010
Moroccan + Criminal
+ =
Saturday, November 6, 2010
Criminality added
Saturday, November 6, 2010
Original
Saturday, November 6, 2010
Results
15
20
25
30
35
Low Prejudice High Prejudice
% C
ateg
oriz
ed a
s M
oroc
can
Moroccan Faces Criminal Moroccan Faces
GLM with prejudice as continuous variable:Prejudice main effect: F(1, 76) = 7.45, p < .01, partial η! = .09
Face main effect: F(1, 76) = 24.35, p < .01, partial η! = .24Face x Prejudice interaction: F(1, 76) = 5.58, p = .02, partial η! = .07.
Saturday, November 6, 2010
Other Reverse Correlation Examples
Saturday, November 6, 2010
Memory bias for attractive alternatives
Karremans, Dotsch, & Corneille (in prep)
Saturday, November 6, 2010
Conditions
UnattractiveAttractive
(N = 82, all females)
Saturday, November 6, 2010
Base image (50% morph)
Saturday, November 6, 2010
Main effect
Attractive
Saturday, November 6, 2010
Main effect
Unattractive
Saturday, November 6, 2010
Interaction (within attractive condition)
Relation > no relationNo relation > relation
Saturday, November 6, 2010
Interaction (within unattractive condition)
Relation > no relationNo relation > relation
Saturday, November 6, 2010
Ingroup projection
Imhoff, Dotsch, Bianchi, Wigboldus, & Banse (in prep.)
Saturday, November 6, 2010
Portugal vs. Germany
Germany's striker Miroslav Klose and Portuguese defender Pepe argue during the Euro 2008 Championships quarter-final football match between Portugal and Germany at the St. Jakob Park Stadium on June 19, 2008 in Basel, Switzerland.
Saturday, November 6, 2010
Germany
Saturday, November 6, 2010
Portugal
Saturday, November 6, 2010
Base image (50% morph)
Saturday, November 6, 2010
Results
German European(N = 53)
Saturday, November 6, 2010
Results
Portuguese European(N = 50)
Saturday, November 6, 2010
General discussion
• Classification image is a index of decision strategy, not necessarily a picture in the head
• Can be spontaneous
• Not necessarily exploratory
• Holds lots of promise for social cognition research
Saturday, November 6, 2010
Thank you!
My advisors:
• Daniel Wigboldus
• Ad van Knippenberg
Contact me:
My collaborators:
• Oliver Langner
• Alexander Todorov
• Johan Karremans
• Olivier Corneille
• Roland Imhoff
• Mauro Bianchi
Saturday, November 6, 2010
Dimensions of face evaluation
Dotsch, Todorov, & Wigboldus (in prep)Saturday, November 6, 2010
Primary trait dimensions
Trustworthy and Untrustworthy
Dominant and Submissive
Saturday, November 6, 2010
Primary trait dimensions
Trustworthy and Untrustworthy
Dominant and Submissive
Saturday, November 6, 2010
Primary trait dimensions
Trustworthy and Untrustworthy
Dominant and Submissive
Saturday, November 6, 2010
+ =
Threat
Dominant Untrustworthy
Saturday, November 6, 2010
Hi-Speed Threat
100 ms 500 ms
Saturday, November 6, 2010
High Speed Threat
100 ms
Saturday, November 6, 2010
High Speed Threat
500 ms
Saturday, November 6, 2010
High Speed Threat
500 ms > 100 ms
Saturday, November 6, 2010
Thank you!
My advisors:
• Daniel Wigboldus
• Ad van Knippenberg
Contact me:
My collaborators:
• Oliver Langner
• Alexander Todorov
• Johan Karremans
• Olivier Corneille
• Roland Imhoff
• Mauro Bianchi
Saturday, November 6, 2010