Nonparametric Bayes and human cognition
Tom GriffithsDepartment of Psychology
Program in Cognitive Science
University of California, Berkeley
Analyzing psychological data• Dirichlet process mixture models for capturing
individual differences(Navarro, Griffiths, Steyvers, & Lee, 2006)
• Infinite latent feature models…– …for features influencing similarity
(Navarro & Griffiths, 2007; 2008)– …for features influencing decisions
()
Categorization
How do people represent categories?
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
catdog ?
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
cat
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
cat
Prototypes
(Posner & Keele, 1968; Reed, 1972)
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Prototype
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
cat
cat
cat
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
cat
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
cat
Exemplars
(Medin & Schaffer, 1978; Nosofsky, 1986)
Store everyinstance
(exemplar)in memory
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
cat
cat
cat
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
cat
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
cat
Something in between
(Love et al., 2004; Vanpaemel et al., 2005)
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
cat
cat
cat
A computational problem• Categorization is a classic inductive problem
– data: stimulus x– hypotheses: category c
• We can apply Bayes’ rule:
and choose c such that P(c|x) is maximized
€
P(c | x) =p(x | c)P(c)
p(x | c)P(c)c
∑
Density estimation• We need to estimate some probability
distributions– what is P(c)?– what is p(x|c)?
• Two approaches:– parametric– nonparametric
• These approaches correspond to prototype and exemplar models respectively
(Ashby & Alfonso-Reese, 1995)
Parametric density estimation
Assume that p(x|c) has a simple form, characterized by parameters (indicating the
prototype)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Pro
babi
lity
den
sity
x
Nonparametric density estimation
Approximate a probability distribution as a sum of many “kernels” (one per data point)
estimated functionindividual kernelstrue function
n = 10
Pro
babi
lity
den
sity
x
Something in between
mixture distributionmixture components
Pro
babi
lity
x
Use a “mixture” distribution, with more than one component per data point
(Rosseel, 2002)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Anderson’s rational model(Anderson, 1990, 1991)
• Treat category labels like any other feature• Define a joint distribution p(x,c) on features using
a mixture model, breaking objects into clusters• Allow the number of clusters to vary…
€
P(cluster j)∝n j j is old
α j is new
⎧ ⎨ ⎩
a Dirichlet process mixture model(Neal, 1998; Sanborn et al., 2006)
A unifying rational model• Density estimation is a unifying framework
– a way of viewing models of categorization
• We can go beyond this to define a unifying model– one model, of which all others are special cases
• Learners can adopt different representations by adaptively selecting between these cases
• Basic tool: two interacting levels of clusters– results from the hierarchical Dirichlet process
(Teh, Jordan, Beal, & Blei, 2004)
The hierarchical Dirichlet process
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
A unifying rational modelclusterexemplarcategory
€
γ∈ (0,∞)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
€
γ→ ∞
€
α → ∞€
α ∈ (0,∞)€
α → 0
exemplar
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
prototype
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Anderson
HDP+, and Smith & Minda (1998)
• HDP+, will automatically infer a representation using exemplars, prototypes, or something in between (with α being learned from the data)
• Test on Smith & Minda (1998, Experiment 2)111111011111101111110111111011111110000100
000000100000010000001000000010000001111101
Category A: Category B:
exceptions
HDP+, and Smith & Minda (1998)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Pro
babi
lity
of
A prototype
exemplar HDP
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Log
-lik
elih
ood
exemplar
prototype
HDP
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.The promise of HDP+,+
• In HDP+,+, clusters are shared between categories
– a property of hierarchical Bayesian models
• Learning one category has a direct effect on the prior on probability densities for the next category
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.
Learning the features of objects
• Most models of human cognition assume objects are represented in terms of abstract features
• What are the features of this object?
• What determines what features we identify?
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
(Austerweil & Griffiths, submitted)
The nonparametric approach
Use the Indian buffet process as a prior on Z
Assume that the total number of features is unbounded, but only a finite number will be
expressed in any finite dataset
(Griffiths & Ghahramani, 2006)
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
(Austerweil & Griffiths, submitted)
An experiment…
(Austerweil & Griffiths, submitted)
Training Testing
Cor
rela
ted
Fac
tori
al
See
nU
nsee
nS
huff
led
Conclusions
• Approaching cognitive problems as computational problems allows cognitive science and machine learning to be mutually informative
• Machine
Credits
CategorizationAdam SanbornKevin CaniniDan Navarro
Learning featuresJoe Austerweil
MCMC with peopleAdam Sanborn
QuickTime™ and aTIFF (Uncompressed) decompressorare needed to see this picture.
QuickTime™ and aTIFF (LZW) decompressor
are needed to see this picture.
Computational Cognitive Science Labhttp://cocosci.berkeley.edu/
QuickTime™ and aTIFF (Uncompressed) decompressor
are needed to see this picture.