論文紹介 semi-supervised learning with deep generative models
TRANSCRIPT
-
Semi-supervised Learning
with Deep Generative ModelsNIPS2014 @ , 2015/01/20
Preferred Networks,
@beam2d
-
l
l
(semi-supervised learning)
2
-
4
3
Transductive SVM
Manifold Tangent Classifier (MTC), AtlasRBF
-
MNIST 100 3.33%
SVHN NORB
4
-
5
x
z
p(x, z) = p(z)p(x|z)
-
M1
6
Neural Net
z N (z;0, I)
(,)
x N (x|, diag2)
-
M1
7
Neural Net
z N (z;0, I)
x Bernoulli(x|)
Gaussian Bernoulli
-
M2Gaussian
8
Neural Net
z N (z;0, I)
(,)
y Cat(y|)
x N (x|, diag2)
-
AutoEncoder
l
l NN
9
p(z)p(x|z) x z
p(z)p(x|z) q(x)q(z|x)
z
x
NN( ) NN( )
q(x)
-
NN
10
l M1
l M2
q(z|x) = N (z|(x), diag2(x)).NN
NN
q(z|y,x) = N (z|(y,x), diag2(y,x)),q(y|x) = Cat(y|(x)).
-
M1 AutoEncoder
11
log p(x) Eq(z|x)[log p(x|z)]KL[q(z|x)p(z)]
q(x, z) = p(x, z)
TSVM M2
z q(z|x)
AutoEncoder z
-
M2
12
log p(x, y) L(x, y) :=Eq(z|x,y)[log p(x|y, z) + log p(y) + log p(z) log q(z|x, y)]
log p(x) U(x) :=Eq(y,z|x)[log p(x|y, z) + log p(y) + log p(z) log q(y, z|x)]
(x,y):labeled
L(x, y) +
x:unlabaled
U(x)
(x,y):labeled
log q(y|x)
q(y|x)
-
SGVB (SBP)
l
l
l Gaussian
Stochastic Gradient Variational Bayes Stochastic BackProp ICLR14, ICML14
13
Eq(z|x,y)Eq(z|x,y)[f(x, y, z)]
Eq(z|x,y)[f(x, y, z)] = EN (|0,I)[f(x, y,(x) + (x) )]
Eq(z|x,y)[f(x, y, z)] = EN (|0,I)[f(x, y,(x) + (x) )]
-
SGVB(SBP) +
l OK
l AdaGrad RMSprop
3.2 4.4 4.4
14
-
2 2
l (MNIST, SVHN, NORB)
l 2
2 (MNIST)
(MNIST, SVHN)
15
z zyx|y, zy x|y, zx
z|x
-
%
16
-
17
2 (style)
zz
-
18
10
zx|y, z
-
l
l
l
l DBM NN
l DBM
l
19
-
Kingma, D. P., Mohamed, S., Jimenez Rezende, D., & Welling, M. (2014). Semi-supervised Learning with Deep Generative Models. In Advances in Neural Information Processing Systems 27 (pp. 35813589).
Stochastic Gradient VB AutoEncoder
Kingma, D. P., & Welling, M. (2014). Auto-Encoding Variational Bayes.International Conference on Learning Representations.
Stochastic BackProp
Rezende, D. J., Mohamed, S., & Wierstra, D. (2014). Stochastic Backpropagation and Approximate Inference in Deep Generative Models. In Proceedings of the 31st International Conference on Machine Learning (pp. 12781286).
20