Face Alignment with Part-Based Modeling

Vahid KazemiJosephine SullivanCVAPKTH Institute of Technology

Objective: Face Alignment

•Find the correspondences between landmarks of a template face model and the target face.

Annotated images (source: IMM dataset) Test image (source: YouTube)

Why: Possible Applications

•The outcome can be used for:- Motion Capture: by determining head pose and facial

expressions.- Face Recognition: by comparing registered facial features with

a database.- 3D Reconstruction: by determining camera parameters using

correspondences in an image sequence- Etc.

Global Methods

•Overview:- Create a constrained generative template model- Start with a rough estimate of face position. - Refine the template to match the target face.

•Properties:- Model deformations more precisely- Arbitrary number of landmarks

•Examples: - Active Shape Models [Cootes 95] - Active Appearance Model [Cootes 98]- 3D Morphable Models [Blanz 99]

Part-Based Methods

•Overview:- Train different classifiers for each part. - Learn constraints on relative positions of parts.

•Properties:- More robust to partial occlusion- Better generalization ability- Sparse results

•Examples: - Elastic Bunch Graph Matching [Wiskott 97]- Pictorial Structures [Felzenszwalb 2003]

Our approach to face alignment

•How can we avoid the draw backs of existing models?

Our approach to face alignment

•Find the mapping, q, from appearance to the landmark positions:

•But q is complex and non-linear…

€

q

Linearizing the model

•Use piece-wise linear functions

€

qi

Linearizing the model

•Use a part based model

€

qi

Linearizing the model

•Use a suitable feature descriptor

Feature Descriptor

Part Selection Criteria

•Detect the parts accurately and reliably- Contain strong features

•Ensure a simple (linear) model- Minimum variation

•Capture the global appearance- Cover the whole object

Part Selection for the face

We chose nose, eyes, and mouth as good candidates

Image from IMM dataset

Appearance descriptor

•Variation of PHOG descriptor- Divide the patch into 8 sub-regions- Recursively repeat for square regions

Part detection

•Build a tree-structured model of the face, with nose at the root, and eyes and mouth as the leafs of the tree.

Part detection

•Detect the parts by sliding a patch on image and calculating the Mahalanobis distance of the patch from the mean model

Part detection

•Find the optimal solution by minimizing the pictorial structure cost function:

•We can solve this efficiently by using generalized distance transform [Felzenszwalb 2003] by limiting the cost function

Regression

•Model the mapping between the patch’s appearance feature (f) and its landmark positions (x) as a linear function:

•Estimate weights from training set using Ridge regression

Regression

•Comparison of different regression methods

Robustify the regression function

•Why• Compensate for bad part detection • Deformable parts don’t exactly fit in a box

•How• Extend training set by adding noise to part positions

Experiments

•Use 240 face images from IMM dataset. •Dataset contains still images from 40 individual subjects

with various facial expressions under the same lighting settings

•58 landmarks are used to represent the shape of subjects

Results

•Comparison of localization accuracy of our algorithm comparing to some existing methods on IMM dataset.

* Mean error is the mean Euclidean distance between predicted and ground truth location of landmarks in pixels

Results

•The results of cross validation on IMM dataset

PredictedGround truth

Demo

More videos: http://www.csc.kth.se/~vahidk/face/

Conclusion and future work

•Part-Based models can be used to simplify complicated models

•The choice of parts is very important•HOG descriptors are not fully descriptive

•Questions?