Discussion

Ying Nian WuUCLA Department of Statistics

JSM 2011

Population value decomposition

Latent variable models

Hidden

Observed

Learning: Examples

Inference:

Latent variable models Mixture model

Factor analysis

Computational neural science

Z: Internal representation by neurons

Y: Sensory data from outside environment

Hidden

Observed

Connection weights

Hierarchical extension: modeling Z by another layer of hidden variables explaining Y instead of Z

Inference / explaining away

Source: Scientific American, 1999

Visual cortex: layered hierarchical architecture

V1: primary visual cortex simple cells complex cells

bottom-up/top-down

Independent Component Analysis Bell and Sejnowski, 1996

CBcBcI NN B ...11

Nicpci ,...,1tly independen )(~

)dim(IN

IIC AB 1

mNNmmm CBcBcI B ,11, ...

mmm IIC AB 1

Laplacian/Cauchy

Hyvarinen, 2000

Sparse coding Olshausen and Field, 1996

Laplacian/Cauchy/mixture Gaussians

Nicpci ,...,1tly independen )(~

NNBcBcI ...11

mNNmmm BcBcI ,11, ...)dim(IN

Inference: sparsification, non-linear lasso/basis pursuit/matching pursuit mode and uncertainty of p(C|I) explaining-away, lateral inhibition

Nicpci ,...,1tly independen )(~

Sparse coding / variable selection

Learning: mNNmmm BcBcI ,11, ...

)dim(IN

A dictionary of representational elements (regressors)

NNBcBcI ...11

Olshausen and Field, 1996

}exp{)(

1),(

,, j

jiiji vhW

WZVHp

Nihi ,...,1 ,

V

Restricted Boltzmann Machine Hinton, Osindero and Teh, 2006

P(V|H)P(H|V): factorized no-explaining away

hidden, binary

visible

Source: Scientific American, 1999

Visual cortex: layered hierarchical architecture

bottom-up/top-down

What is beyond V1?Hierarchical model?

P(V,H) = P(H)P(V|H) P(H) P(V’,H)

I

H

V

V’

Discriminative correction by back-propagation

Unfolding, untying, re-learning

Hierarchical RBM Hinton, Osindero and Teh, 2006

Hierarchical sparse coding

NNBcBcI ...11

,,sxB

Attributed sparse coding elements transformation group topological neighborhood system

UBcIii sx

n

ii

,,

1

Layer above : further coding of the attributes of selected sparse coding elements

Active basis modelWu, Si, Gong, Zhu, 10Zhu, Guo, Wang, Xu, 05

n-stroke templaten = 40 to 60, box= 100x100

Learning and Inference

Finding n strokes to sketch M images simultaneouslyn = 60, M = 9

Scan over multiple resolutions

Scan over multiple resolutions and orientations (rotating template)

Learning active basis models from non-aligned imageEM-type maximum likelihood learning, Initialized by single image learning

Learning active basis models from non-aligned image

Learning active basis models from non-aligned image

Hierarchical active basis

Lowlog-likelihood

Highlog-like

Model based clustering

MNIST500 total