Computer VisionColorado School of Mines
Colorado School of Mines
Computer Vision
Professor William HoffDept of Electrical Engineering &Computer Science
http://inside.mines.edu/~whoff/ 1
Computer VisionColorado School of Mines
Statistical Models for Shape and Appearance
2
Note – some material for these slides came from• Computer Vision Algorithms and Applications, by Szeliski, Springer 2011• Computer vision: models, learning and inference, by Prince, Cambridge University,
2012
Computer VisionColorado School of Mines
Statistical Models for Shape and Appearance
• We often have knowledge about the possible ways that shape and appearance can vary
• When matching to a new image, we can use this to constrain the shape and possibly give us better results
3
Computer VisionColorado School of Mines
Principal Components
• One way to model the variation in your data, is to use principal components
• Principal components:– Assume we have a population of measurements vectors – The covariance matrix is – Find the eigenvectors and eigenvalues of – The are called “principal components”
• We can express any of the original vectors as a linear combination of PCs:
where , , … , , , , … ,
4
Computer VisionColorado School of Mines
Generative Model
• This is a “generative” model (as opposed to a “discriminative” model)
• Given coefficients h (which represent the parameters of the model in the world), we can generate examples of measurements x:
where is normally distributed noise with covariance
5
Computer VisionColorado School of Mines
Approaches
• We’ll look at 3 approaches that use PC‐based statistical models:1. Appearance‐based (use images)
• Also called “eigenimages”
2. Shape‐based (use contour points or surface points)• Also called “active shape models” or “point distribution models”
3. Shape and appearance‐based (use both)• Also called “active appearance models”
6
Computer VisionColorado School of Mines
Appearance‐Based (eigenimages)
• We have a collection of images, I1..IK– We subtract off the mean of the collection of images
• We transform each MxN image Ii into a column vectorxi = [Ii(1,1), Ii(1,2), … Ii(1,N), Ii(2,1), … Ii(M,N)]T
7
I1I2
I3
IK
• Find the covariance matrix Cx
• The eigenvectors (principal components) represent “basis” images (or “eigenimages”)
x1 x2 x3 xK
Computer VisionColorado School of Mines
Application ‐ Face images
from: http://www.cs.princeton.edu/~cdecoro/eigenfaces/
Input images (Yale database)
8
Computer VisionColorado School of Mines
Eigenfaces
• 10 PC’s capture 36% of the variance
• 25 PC’s capture 56% of the variance
9
Computer VisionColorado School of Mines
Reconstruction with a small number of Principal Components
Each new image from left to right corresponds to using 1 additional PC for reconstruction
Each new image from left to right corresponds to using 8 additional PC for reconstruction
10
Computer VisionColorado School of Mines
Recognition
• To recognize a new image x, project it onto the linear subspace spanned by the eigenimages (i.e., find the coefficients ):
11
DFFS: test if x is a face
DIFS: measure how close x is to a training image face
Computer VisionColorado School of Mines
Shape‐based (Active Shape Models)
• Represent a shape with a set of points– For example, evenly spaced points along a contour
• Let x be the vector consisting of the set of points• We compute the covariance of a set of training vectors, and
get the eigenvectors (principal components)
12
Computer VisionColorado School of Mines
Examples of contours
• Resistors
13
Computer VisionColorado School of Mines
• Hand
14
Computer VisionColorado School of Mines
Recognition
• We assume that the coefficients of the PCs (the shape parameters) are normally distributed
• To recognize a new shape x, we project it onto the linear subspace formed by the PCs, and compute the Mahalanobisdistances to each category center
15
Computer VisionColorado School of Mines
Alignment of shapes
• The shapes must be registered and points sampled in the same places
16
Computer VisionColorado School of Mines
Registration to image
• To register a point distribution model with an image, each control point searches in a direction normal to the contour to find the most likely corresponding image edge point
• Then search over (a) the shape parameters and (b) the parameters of a transform to align the shapes
17
Computer VisionColorado School of Mines
Examples
18
Computer VisionColorado School of Mines19
Computer VisionColorado School of Mines
3D Shape Models
• Example: Creating bone models from 2D silhouettes using a statistical atlas
20
Josh King (MS 2007), Colorado School of Mines. Thesis title: “Bi‐Planar Image Registration and Modeling of Bones”
A. Szymczak, W. Hoff and M. Mahfouz, “3D Shape from Silhouette Points in Registered 2D Images Using Conjugate Gradient Method,” Medical Imaging 2010: Image Processing. Proceedings of the SPIE, Volume 7623, pp. 762316‐762316‐8, 2010.
Computer VisionColorado School of Mines
Shape and Appearance‐Based
• You get better results if you model both shape andappearance– This is called an “Active Appearance Model (AAM)”
• We use principal components for shape (encoded by a set of control points) and appearance (ie, the image texture)– The texture is normalized to a canonical shape before being analyzed
21
Computer VisionColorado School of Mines
Active Appearance Model
• Both shape and texture are represented as deviations from a mean shape and mean texture
• Note – the same appearance parameters h simultaneously control both the shape and texture deformations – these are correlated!
22
Computer VisionColorado School of Mines
Example – face modeling
23
Computer VisionColorado School of Mines24
Computer VisionColorado School of Mines
Warping images
25Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
To align the image texture according to shape displacements, use a piecewise affine transformation
Triangulate image points using Delaunay triangulation.
Image in each triangle is warped by an affine transformation.
Computer VisionColorado School of Mines
Registering a new image
26Computer vision: models, learning and inference. ©2011 Simon J.D. Prince
To register the face model to a new image, search over (a) the appearance parameters and (b) the warping parameters of a transform to align the shapes
Computer VisionColorado School of Mines
Example
• AAMs have been used directly for recognition, but their main use is to align faces into a canonical pose so that other methods of face recognition can be used
27