nonparametric bayesian statistics: part...
TRANSCRIPT
Nonparametric Bayesian Statistics: Part II
Tamara BroderickITT Career Development Assistant Professor Electrical Engineering & Computer Science
MIT
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
…
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
Vkiid⇠ Beta(1,↵) ⇢k =
2
4k�1Y
j=1
(1� Vj)
3
5Vk
1 2 3 4 ……
• Part of: DP mixture model
1
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
…
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
Vkiid⇠ Beta(1,↵) ⇢k =
2
4k�1Y
j=1
(1� Vj)
3
5Vk
1 2 3 4 ……
• Part of: DP mixture model
1
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
…
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
Vkiid⇠ Beta(1,↵) ⇢k =
2
4k�1Y
j=1
(1� Vj)
3
5Vk
1 2 3 4 ……
• Part of: DP mixture model
1
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
…
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
Vkiid⇠ Beta(1,↵) ⇢k =
2
4k�1Y
j=1
(1� Vj)
3
5Vk
1 2 3 4 ……
• Part of: DP mixture model
1
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
…
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
Vkiid⇠ Beta(1,↵) ⇢k =
2
4k�1Y
j=1
(1� Vj)
3
5Vk
1 2 3 4 ……
• Part of: DP mixture model
1
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
1 2 3 4 ……
• Part of: DP mixture model
1
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
1 2 3 4 ……
• Part of: DP mixture model
1
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
1 2 3 4 ……
• Part of: DP mixture model
1
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
1 2 3 4 ……
• Part of: DP mixture model
1
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
…
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
1 2 3 4 ……
• Part of: DP mixture model
1
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
…
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
1 2 3 4 ……
• Part of: DP mixture model
1
Vkiid⇠ Beta(1,↵)
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
…
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
1 2 3 4 ……
• Part of: DP mixture model
1
Vkiid⇠ Beta(1,↵) ⇢k =
2
4k�1Y
j=1
(1� Vj)
3
5Vk
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
…
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]
1 2 3 4 ……
• Part of: DP mixture model
1
Vkiid⇠ Beta(1,↵) ⇢k =
2
4k�1Y
j=1
(1� Vj)
3
5Vk
Recall: Part I• Dirichlet process (DP) stick-breaking • Griffiths-Engen-McCloskey (GEM) distribution:
…
⇢ = (⇢1, ⇢2, . . .) ⇠ GEM(↵)
[McCloskey 1965; Engen 1975; Patil and Taillie 1977; Ewens 1987; Sethuraman 1994; Ishwaran, James 2001]1
Vkiid⇠ Beta(1,↵) ⇢k =
2
4k�1Y
j=1
(1� Vj)
3
5Vk
1 2 3 4 ……
• Part of: DP mixture model
DP or not DP, that is the question• GEM: • Compare to:
• Finite (small K) mixture model !
!
• Finite (large K) mixture model !
!
• Time series
…
…
…
2
DP or not DP, that is the question• GEM: • Compare to:
• Finite (small K) mixture model !
!
• Finite (large K) mixture model !
!
• Time series
…
…
…
2
DP or not DP, that is the question• GEM: • Compare to:
• Finite (small K) mixture model !
!
• Finite (large K) mixture model !
!
• Time series
…
…
…
2
DP or not DP, that is the question• GEM: • Compare to:
• Finite (small K) mixture model !
!
• Finite (large K) mixture model !
!
• Time series
…
…
…
2
DP or not DP, that is the question• GEM: • Compare to:
• Finite (small K) mixture model !
!
• Finite (large K) mixture model !
!
• Time series
…
…
…
2
DP or not DP, that is the question• GEM: • Compare to:
• Finite (small K) mixture model !
!
• Finite (large K) mixture model !
!
• Time series
…
…
…
2
Nonparametric Bayes: Part II• Last time:
• Understand what it means to have an infinite/growing number of parameters
• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html
• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process
3
Nonparametric Bayes: Part II• Last time:
• Understand what it means to have an infinite/growing number of parameters
• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html
• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process
3
Nonparametric Bayes: Part II• Last time:
• Understand what it means to have an infinite/growing number of parameters
• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html
• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process
3
Nonparametric Bayes: Part II• Last time:
• Understand what it means to have an infinite/growing number of parameters
• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html
• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process
3
Nonparametric Bayes: Part II• Last time:
• Understand what it means to have an infinite/growing number of parameters
• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html
• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process
3
Nonparametric Bayes: Part II• Last time:
• Understand what it means to have an infinite/growing number of parameters
• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html
• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process
3
Nonparametric Bayes: Part II• Last time:
• Understand what it means to have an infinite/growing number of parameters
• Finite representation allows use of infinite model • www.tamarabroderick.com/tutorials.html
• This time: • Avoid the infinity of parameters for inference • e.g. Chinese restaurant process
3
• Integrate out the frequenciesMarginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2), zniid⇠ Cat(⇢1, ⇢2)
4
• Integrate out the frequenciesMarginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
4
• Integrate out the frequenciesMarginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=
Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1
4
• Integrate out the frequenciesMarginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=
Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1
=
Z⇢1Beta(⇢1|a1,n, a2,n)d⇢1
4
• Integrate out the frequenciesMarginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=
Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1
=
Z⇢1Beta(⇢1|a1,n, a2,n)d⇢1
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
4
• Integrate out the frequenciesMarginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=
Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1
=
Z⇢1Beta(⇢1|a1,n, a2,n)d⇢1
=
Z⇢1
�(a1,n + a2,n)
�(a1,n)�(a2,n)⇢a1,n�11 (1� ⇢1)
a2,n�1d⇢1
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
4
• Integrate out the frequenciesMarginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=
Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1
=
Z⇢1Beta(⇢1|a1,n, a2,n)d⇢1
=
Z⇢1
�(a1,n + a2,n)
�(a1,n)�(a2,n)⇢a1,n�11 (1� ⇢1)
a2,n�1d⇢1
=�(a1,n + a2,n)
�(a1,n)�(a2,n)
�(a1,n + 1)�(a2,n)
�(a1,n + a2,n + 1)
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
4
• Integrate out the frequenciesMarginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=
Zp(zn = 1|⇢1)p(⇢1|z1, . . . , zn�1)d⇢1
=
Z⇢1Beta(⇢1|a1,n, a2,n)d⇢1
=
Z⇢1
�(a1,n + a2,n)
�(a1,n)�(a2,n)⇢a1,n�11 (1� ⇢1)
a2,n�1d⇢1
=�(a1,n + a2,n)
�(a1,n)�(a2,n)
�(a1,n + 1)�(a2,n)
�(a1,n + a2,n + 1)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
4
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with equal probability • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with equal probability • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with equal probability • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with equal probability • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with equal probability • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with equal probability • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with equal probability • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with equal probability • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with equal probability • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with equal probability • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• Pólya urn
Marginal cluster assignments
1 2
⇢1 ⇠ Beta(a1, a2)
p(zn = 1|z1, . . . , zn�1), zn
iid⇠ Cat(⇢1, ⇢2)
=a1,n
a1,n + a2,n
a1,n := a1 +n�1X
m=1
1{zm = 1}, a2,n = a2 +n�1X
m=1
1{zm = 2}
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
lim
n!1
# orange
# total
= ⇢orange
d= Beta(a
orange
, agreen
)
5
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
6
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)
6
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)
p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n
6
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)
p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n
ak,n := ak +n�1X
m=1
1{zm = k}
6
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)
p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n
ak,n := ak +n�1X
m=1
1{zm = k}
6
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)
p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n
ak,n := ak +n�1X
m=1
1{zm = k}
6
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)
p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n
ak,n := ak +n�1X
m=1
1{zm = k}
6
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)
p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n
ak,n := ak +n�1X
m=1
1{zm = k}
6
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)
p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n
ak,n := ak +n�1X
m=1
1{zm = k}
lim
n!1
(# orange, # green, # red, # yellow)
# total
6
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)
p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n
ak,n := ak +n�1X
m=1
1{zm = k}
lim
n!1
(# orange, # green, # red, # yellow)
# total
! (⇢orange
, ⇢green
, ⇢red
, ⇢yellow
)
6
• Integrate out the frequencies !
!
!
!
• multivariate Pólya urn
Marginal cluster assignments
• Choose any ball with prob proportional to its mass • Replace and add ball of same color
1 2 3 4
⇢1:K ⇠ Dirichlet(a1:K), zniid⇠ Cat(⇢1:K)
p(zn = k|z1, . . . , zn�1) =ak,nPKj=1 aj,n
ak,n := ak +n�1X
m=1
1{zm = k}
lim
n!1
(# orange, # green, # red, # yellow)
# total
! (⇢orange
, ⇢green
, ⇢red
, ⇢yellow
)d= Dirichlet(a
orange
, agreen
, ared
, ayellow
)6
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1 Step 2
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1 Step 2 Step 3
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1 Step 2 Step 3 Step 4
[Blackwell, MacQueen 1973; Hoppe 1984]7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1 Step 2 Step 3 Step 4
(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)
(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:
7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1 Step 2 Step 3 Step 4
(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)
(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:
7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1 Step 2 Step 3 Step 4
(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)
(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:
7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1 Step 2 Step 3 Step 4
(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)
(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:
Vkiid⇠ Beta(1,↵)
7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1 Step 2 Step 3 Step 4
(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)
(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:
Vkiid⇠ Beta(1,↵)
⇢1 = V1
7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1 Step 2 Step 3 Step 4
(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)
(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:
Vkiid⇠ Beta(1,↵)
⇢1 = V1
⇢2 = (1� V1)V2
7
Marginal cluster assignments• Hoppe urn / Blackwell-MacQueen urn
• Choose ball with prob proportional to its mass • If black, replace and add ball of new color • Else, replace and add ball of same color
Step 0 Step 1 Step 2 Step 3 Step 4
(#orange,#other) = PolyaUrn(1,↵)(#green,#other) = PolyaUrn(1,↵)
(#red,#other) = PolyaUrn(1,↵)• not orange:• not orange, green:
Vkiid⇠ Beta(1,↵)
⇢1 = V1
⇢2 = (1� V1)V2
⇢3 = [Y2
k=1(1� Vk)]V3
7
Chinese restaurant process
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1
�1
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1
�1
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1
2�1
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2�1
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2�1
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2�1 �2
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2�1 �2
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 4�1 �2
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 4�1 �2
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 4�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 4�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 45�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 45
6
�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 45
67
�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}[Aldous 1983]8
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Same thing we just did • Each customer walks into the restaurant
• Sits at existing table with prob proportional to # people there
• Forms new table with prob proportional to α • Marginal for the Categorical likelihood with GEM prior !
• Partition of [8]: set of mutually exclusive & exhaustive sets of
z1 = z2 = z7 = z8 = 1, z3 = z5 = z6 = 2, z4 = 3) ⇧8 = {{1, 2, 7, 8}, {3, 5, 6}, {4}}
[8] = {1, . . . , 8}8
• Probability of this seating: !
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
KN nk
p(v1, v2, v3)
9
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
KN nk
p(v1, v2, v3)
• Probability of this seating: !
9
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
KN nk
p(v1, v2, v3)
• Probability of this seating: !
9
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
KN nk
p(v1, v2, v3)
• Probability of this seating: !
9
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
KN nk
p(v1, v2, v3)
• Probability of this seating: !
9
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Probability of this seating: !
• Probability of N customers ( tables, at table k):
!
• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution
• Start: • t th step:
KN nk
p(v1, v2, v3)
9
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Probability of this seating: !
• Probability of N customers ( tables, at table k):
!
• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution
• Start: • t th step:
KN nk
p(v1, v2, v3)
9
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Probability of this seating: !
• Probability of N customers ( tables, at table k):
!
• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution
• Start: • t th step:
KN nk
p(v1, v2, v3)
9
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Probability of this seating: !
• Probability of N customers ( tables, at table k):
!
• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution
• Start: • t th step:
KN nk
p(v1, v2, v3)
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
9
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Probability of this seating: !
• Probability of N customers ( tables, at table k):
!
• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution
• Start: • t th step:
KN nk
p(v1, v2, v3)
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
9
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Probability of this seating: !
• Probability of N customers ( tables, at table k):
!
• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution
• Start: • t th step:
KN nk
p(v1, v2, v3)
↵KNQKN
k=1(nk � 1)!
↵ · · · (↵+N � 1)= P(⇧N = ⇡N )
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
9
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Probability of this seating: !
• Probability of N customers ( tables, at table k):
!
• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution
• Start: • t th step:
KN nk
p(v1, v2, v3)
↵KNQKN
k=1(nk � 1)!
↵ · · · (↵+N � 1)= P(⇧N = ⇡N )
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
9
Chinese restaurant process1 3
2 45
67
8
�1 �2 �3
• Probability of this seating: !
• Probability of N customers ( tables, at table k):
!
• Prob doesn’t depend on customer order: exchangeable • Gibbs sampling review: target distribution
• Start: • t th step:
KN nk
p(v1, v2, v3)
↵KNQKN
k=1(nk � 1)!
↵ · · · (↵+N � 1)= P(⇧N = ⇡N )
↵
↵· 1
↵+ 1· ↵
↵+ 2· ↵
↵+ 3· 1
↵+ 4· 2
↵+ 5· 2
↵+ 6· 3
↵+ 7
9
References (Part II)
10
DJ Aldous. Exchangeability and related topics. Springer, 1983.
D Blackwell and JB MacQueen. Ferguson distributions via Pólya urn schemes. The Annals of Statistics, 1973.
S Engen. A note on the geometric series as a species frequency model. Biometrika, 1975.
W Ewens. Population genetics theory -- the past and the future. Mathematical and Statistical Developments of Evolutionary Theory, 1987.
FM Hoppe. Pólya-like urns and the Ewens' sampling formula. Journal of Mathematical Biology, 1984.
H Ishwaran and LF James. Gibbs sampling methods for stick-breaking priors. Journal of the American Statistical Association, 2001.
JW McCloskey. A model for the distribution of individuals by species in an environment. Ph.D. thesis, Michigan State University, 1965.
GP Patil and C Taillie. Diversity as a concept and its implications for random communities. Bulletin of the International Statistical Institute, 1977.
J Sethuraman. A constructive definition of Dirichlet priors. Statistica Sinica, 1994.