![Page 1: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/1.jpg)
Point Distribution Models Active Appearance Models
Compilation based on: Dhruv Batra ECE CMU Tim Cootes Machester
![Page 2: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/2.jpg)
Essence of the Idea (cont.)
Explain a new example in terms of the model parameters
![Page 3: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/3.jpg)
So what’s a model
Model
“Shape” “texture”
![Page 4: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/4.jpg)
Active Shape Models
training set
![Page 5: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/5.jpg)
Texture Models
warp to mean shape
![Page 6: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/6.jpg)
Intensity Normalisation
Allow for global lighting variations Common linear approach
Shift and scale so that Mean of elements is zero Variance of elements is 1
Alternative non-linear approach Histogram equalization
Transforms so similar numbers of each grey-scale value
st /)1'( gg
'1 ign
t22 )'(
1
1
tgn
s i
![Page 7: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/7.jpg)
Shape: Review of Construction
Mark face regionon training set
Sample region
Normalise
Statistical Analysis
'g
g
Pbgg
The Fun Step
![Page 8: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/8.jpg)
Multivariate Statistical Analysis
Need to model the distribution of normalised vectors Generate plausible new examples Test if new region similar to training set Classify region
![Page 9: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/9.jpg)
Fitting a gaussian
Mean and covariance matrix of data define a gaussian model
g
1g
2g
![Page 10: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/10.jpg)
Principal Component Analysis
Compute eigenvectors of covariance, S
Eigenvectors : main directions Eigenvalue : variance along
eigenvector
11p22p
g
1g
2g
![Page 11: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/11.jpg)
Eigenvector Decomposition
If A is a square matrix then an eigenvector of A is a vector, p, such that
Usually p is scaled to have unit length,|p|=1
pAp λp with associated eigenvalue theis λ
![Page 12: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/12.jpg)
Eigenvector Decomposition
If K is an n x n covariance matrix, there exist n linearly independent eigenvectors, and all the corresponding eigenvalues are non-negative.
We can decompose K as
TPDPK
)( 1 nppP
n
ndiag
00
00
00
)( 2
1
1D
![Page 13: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/13.jpg)
Eigenvector Decomposition
Recall that a normal pdf has
The inverse of the covariance matrix is
)5.0exp()( 1xKxx Tp
TPPDK 11
TPDPK IPPPP TT
1
12
11
1
00
00
00
n
D
![Page 14: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/14.jpg)
Fun with Eigenvectors
The normal distribution has form
exp(...)||)2()( 5.02/ Kx np
n
ii
TT
1
|||||||||||| DPDPPDPK
![Page 15: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/15.jpg)
Fun with Eigenvectors
Consider the transformation
)( xxPb T
1p
2p
1x
2x
x1b
2b
![Page 16: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/16.jpg)
Fun with Eigenvectors
The exponent of the distribution becomes
M
bn
i i
i
T
TT
T
5.0
5.0
5.0
)()(5.0
)()(5.0
1
2
1
1
1
bDb
xxPPDxx
xxKxx
mean thefrom distance' is`Mahalanob theis M
![Page 17: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/17.jpg)
Normal distribution
Thus by applying the transformation
The normal distribution is simplified to)( xxPb T
)5.0exp()()( Mkpp bx
n
i i
ibM1
2
5.0
1
5.0 )()2(
n
ii
nk
![Page 18: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/18.jpg)
Dimensionality Reduction
Co-ords often correllated Nearby points move together
11bpxx
1b
xx
1p
![Page 19: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/19.jpg)
Dimensionality Reduction
Data lies in subspace of reduced dim.
However, for some t,
i
i
nnbb ppxPbxx 11
tjb j if 0
t
) is of (Variance jjb
![Page 20: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/20.jpg)
Approximation
Each element of the data can be written
rbPxx t )( 1 tt ppP
n
tiir n 1
2 1 , of elements of Variance r
)( xxPb Tt
222 |||||| error,ion Approximat bxxr
bPxx t
![Page 21: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/21.jpg)
Normal PDF
)5.0exp()( ttt Mkp x
2
2
1
2 ||
r
t
i i
it
bM
r
5.0
1
)(25.0 )()2(
t
ii
tnr
ntk
:others all along and
, directions along varianceAssuming2
i
r
it
p
![Page 22: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/22.jpg)
Useful Trick
If x of high dimension, S huge If No. samples, N<dim(x) use
),,( 1 xxxxD N
NN x T
NDDS
1 DDT T
N
1
iiλ uT r eigenvecto with of eigenvaluean is If
iiλ DuS r eigenvecto with of eigenvaluean is then
![Page 23: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/23.jpg)
Building Eigen-Models
Given examples Compute mean and eigenvectors of
covar. Model is then
P – First t eigenvectors of covar. matrix
b – Shape model parameters
}{ ig
Pbgg
![Page 24: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/24.jpg)
Eigen-Face models
Model of variation in a region
1b 2b
4b3b
Pbgg
![Page 25: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/25.jpg)
Applications: Locating objects
Scan window over target region At each position:
Sample, normalise, evaluate p(g) Select position with largest p(g)
![Page 26: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/26.jpg)
Multi-Resolution Search
Train models at each level of pyramid Gaussian pyramid with step size 2 Use same points but different local
models Start search at coarse resolution
Refine at finer resolution
![Page 27: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/27.jpg)
Application: Object Detection
Scan image to find points with largest p(g)
If p(g)>pmin then object is present Strictly should use a background
model:
This only works if the PDFs are good approximations – often not the case
)background()()model()( backgroundmodel PpPp gg
![Page 28: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/28.jpg)
Back (sadly) to Texture Models
raster scan
Normalizations
![Page 29: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/29.jpg)
PCA Galore
Reduce Dimensions of shape vector
Reduce Dimension of “texture” vector
They are still correlated; repeat..
![Page 30: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/30.jpg)
Object/Image to Parameters
modeling
~80
![Page 31: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/31.jpg)
Playing with the Parameters
First two modes of shape variation First two modes of gray-level variation
First four modes of appearance variation
![Page 32: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/32.jpg)
Active Appearance Model Search
Given: Full training model set, new image to be interpreted, “reasonable” starting approximation
Goal: Find model with least approximation error
High Dimensional Search: Curse of the dimensions strikes again
![Page 33: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/33.jpg)
Active Appearance Model Search
Trick: Each optimization is a similar problem, can be learnt
Assumption: Linearity
Perturb model parameters with known amount
Generate perturbed image and sample error
Learn multivariate regression for many such perterbuations
![Page 34: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/34.jpg)
Active Appearance Model Search
Algorithm: current estimate of model parameters: normalized image sample at current estimate
![Page 35: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/35.jpg)
Active Appearance Model Search
Slightly different modeling:
Error term:
Taylor expansion (with linear assumption)
Min (RMS sense) error:
Systematically perturb and estimate by numerical differentiation
![Page 36: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/36.jpg)
Active Appearance Model Search (Results)
![Page 37: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/37.jpg)
Sub-cortical Structures
Initial Position Converged
![Page 38: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/38.jpg)
Random Aside
Shape Vector provides alignment
=
43
Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt
![Page 39: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/39.jpg)
Random Aside
Alignment is the key
1. Warp to mean shape
2. Average pixels
Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt
![Page 40: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/40.jpg)
Random Aside
Enhancing Gender
more same original androgynous more opposite
D. Rowland, D. Perrett. “Manipulating Facial Appearance through Shape and Color”, IEEE Computer Graphics and Applications, Vol. 15, No. 5: September 1995, pp. 70-76
![Page 41: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/41.jpg)
Random Aside (can’t escape structure!)
Alexei (Alyosha) Efros, 15-463 (15-862): Computational Photography, http://graphics.cs.cmu.edu/courses/15-463/2005_fall/www/Lectures/faces.ppt
Antonio Torralba & Aude Oliva (2002)
Averages: Hundreds of images containing a person are
averaged to reveal regularities in the intensity patterns across
all the images.
![Page 42: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/42.jpg)
Random Aside (can’t escape structure!)
“100 Special Moments” by Jason Salavon
Jason Salavon, http://salavon.com/PlayboyDecades/PlayboyDecades.shtml
![Page 43: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/43.jpg)
Random Aside (can’t escape structure!)
“Every Playboy Centerfold, The Decades (normalized)” by Jason Salavon
1960s 1970s 1980sJason Salavon, http://salavon.com/PlayboyDecades/PlayboyDecades.shtml
![Page 44: Point Distribution Models Active Appearance Models](https://reader036.vdocuments.net/reader036/viewer/2022062314/568146eb550346895db4252d/html5/thumbnails/44.jpg)