Representation and Modeling of Natural Scenes

Ying Nian WuUCLA Department of Statistics

http://www.stat.ucla.edu/~ywu/research/

Song Chun Zhu Stefano Soatto

observed image synthesized image

Wu, Zhu, Liu, IJCV 2000; Zhu, Liu, Wu, PAMI 2000

Malik and Perona, late 80s

Image I ,,, lyxB ,,, lyxB

,,,,,, , lyxlyx BIf

Histogram matching (Heeger and Bergen, mid 90s)

Filter response

Filtered image ),,( ,,,, yxfF lyxl

Histogram ,lh

Julesz ensemble

Image universe

Draw random samples from the Julesz ensemble

),,(})(:{)(

, lhhhIhIh

l

DDD

2ZD

Global statistical property

Image lattice

Zhu, Liu, Wu, PAMI 2000

0D

2ZD})(:{)( hIhIh DDD

})|(,exp{)(

1);|(0000

DDDD IIHZ

hIIp

Local statistical property

Gibbs (1902): equivalence of ensemblesExponential family model

Julesz ensemble

Markov random field

Small patch

Large lattice

Wu, Zhu, Liu, IJCV 2000

Iobs from a unknown hc Isyn ~ h with h= Isyn with h= histogram

h= histograms h= histograms h= histograms

JmJmmm BcBcBcI ...2211

Data: a collection of natural image patches },...,1,{ MmIm

Learning: basis },...,{ 1 JBB

Linear representation:

),...,( 1 mJmm ccI

Sparseness of coefficients linear bases

Olshausen & Field: Sparse coding

Mallat and Zhang: matching pursuitCandes and Donoho: curvelets

mjJ

j mjm

mj

BcI

indepcpc

1

),(~

),0()1()( 20 Ncp

Two-Level Generative Model

Mixture prior for sparseness

Bell & Sejnowski (96)Lewiki & Olshausen (99)Olshausen & Millman (00)Pece (01) George & McCulloch (95)

i

lyxi iiiiBcI

S

,,,

~

Wu, Zhu, Guo, ECCV 2002.

Model fitting (EM-type iteration)Estimate S based on I and Sketch Model (MCMC)

Fit Sketch Model on S

Sketch Model

},...,1),,,,,({ nilcyxsS iiiiii

SimplificationEstimate S from I using matching pursuit (Mallat & Zhang)

Fit Sketch Model on S (ignoring c and e)

Math representations of sketch

},...,1),,,,({ nilyxsS iiiii List:

)},,({ ,,,, yxyxyxyx lsS Bit-map:

Causal model for sketch

)|()( ),(, yxNy

yxx

SspSp

)}|,(),(exp{)|(

1)|( ,,2,,1,0),(

),(,),(

iyxyxSsyxyxyxyxN

yxNyx sllSZ

SspyxNi

Pairwise interactions

Soatto,Doretto,Wu, ICCV 2001

Modeling dynamic scenesData:

Model: time series

Representation: principal components (Fourier bases)

Autoregressive model

},...,,{ 21 TIII

ttt

ttt

CWIqAWW

1

Fourier’s solution to heat equation

Soatto,Doretto,Wu, ICCV 2001

World W = (W_high, W_low)

Image I

Knowledge K

Why generative modeling?Representing knowledgeUnsupervised learning of causesModel selection as explaining awayModel checking by synthesis

Physics model and image-based rendering

P(W; K)P(I | W; K) P(W | I; K)