face detection face alignment face parsingpages.cs.wisc.edu/~dyer/cs534-spring11/slides/... · face...
TRANSCRIPT
12/6/2013
1
Face detection, face alignment, and face image parsing
Brandon M. Smith
Guest Lecturer, CS 534
Monday, October 21, 2013
12/6/2013 1 CS 534: Computation Photography
Lecture overview
• Brief introduction to
local features
• Face detection
• Face alignment and
landmark localization
• Face image parsing
12/6/2013 CS 534: Computation Photography 2
http://www.ima.umn.edu/2008-2009/MM8.5-14.09/activities/Wohlberg-Brendt/sift.png
http://www.noio.nl/assets/2011-03-01-stitching-smiles/violajones.png
http://www.mathworks.com/matlabcentral/fx_files/32704/4/icaam.jpg
http://homes.cs.washington.edu/~neeraj/projects/face-parts/images/teaser.png
Local features: broad goal
• What are local features trying to capture?
The local appearance in a region of the image
12/6/2013 CS 534: Computation Photography 3
David G. Lowe, "Distinctive image features from scale-invariant keypoints," International Journal of Computer Vision, 60, 2 (2004)
Local features: motivation What are their uses?
o Matching
12/6/2013 CS 534: Computation Photography 4
http://docs.opencv.org/_images/Featur_FlannMatcher_Result.jpg
12/6/2013
2
Local features: motivation What are their uses?
o Matching
o Image indexing and retrieval
12/6/2013 CS 534: Computation Photography 5
Shen et al., CVPR 2012
Local features: motivation What are their uses?
o Matching
o Image indexing and retrieval
o Aligning images, e.g., for panorama stitching
12/6/2013 CS 534: Computation Photography 6
http://www.leet.it/home/lale/files/Garda-pano.jpg
Local features: motivation What are their uses?
o Matching
o Image indexing and retrieval
o Aligning images, e.g., for panorama stitching
o Video stabilization
12/6/2013 CS 534: Computation Photography 7
Local features: motivation What are their uses?
o Matching
o Image indexing and retrieval
o Aligning images, e.g., for panorama stitching
o Video stabilization
o 3D reconstruction
12/6/2013 CS 534: Computation Photography 8
http://www.nsf.gov/news/special_reports/science_nation/images/virtualrealitymaps/duomopisa500.jpg
12/6/2013
3
Local features: motivation What are their uses?
o Matching
o Image indexing and retrieval
o Aligning images, e.g., for panorama stitching
o Video stabilization
o 3D reconstruction
o Object recognition, including face recognition
12/6/2013 CS 534: Computation Photography 9
http://doi.ieeecomputersociety.org/cms/Computer.org/dl/trans/tp/2007/11/figures/i192714.gif
Local features: types
Types of features and feature descriptors o Image intensity or gradient patches
o Shift Invariance Feature Transform (SIFT) – very popular!
o DAISY
o SURF
o Many more…
12/6/2013 CS 534: Computation Photography 12
Face detection: goal
Automatically detect the presence and
location of faces in images.
12/6/2013 CS 534: Computation Photography 13
Shen et al., Detecting and Aligning Faces by Image Retrieval, CVPR 2013
Face detection: motivation
• Automatic camera focus
12/6/2013 CS 534: Computation Photography 14
http://cdn.conversations.nokia.com.s3.amazonaws.com/wp-content/uploads/2013/09/Nokia-Pro-Camera-auto-focus_half-press.jpg http://cdn.conversations.nokia.com.s3.amazonaws.com/wp-content/uploads/2013/09/Nokia-Pro-Camera-auto-focus_half-press.jpg
12/6/2013
4
Face detection: motivation
• Automatic camera focus
• Easier photo tagging
12/6/2013 CS 534: Computation Photography 15
http://sphotos-d.ak.fbcdn.net/hphotos-ak-ash3/163475_10150118904661729_7246884_n.jpg
Face detection: motivation
• Automatic camera focus
• Easier photo tagging
• First step in any face recognition algorithm
12/6/2013 CS 534: Computation Photography 16
http://images.fastcompany.com/upload/camo1.jpg
Face detection: challenges
• Large face shape and appearance variation
• Scale and rotation (yaw, roll, pitch) variation
• Background clutter
• Occlusions
• Image noise
• Efficiency
• False positives
12/6/2013 CS 534: Computation Photography 17
Face detection: Viola-Jones*
• Paul Viola and Michael Jones, Robust Real-time
Face Detection, International Journal of Computer
Vision (IJCV), 2004.
o Feature type?
o Which features are important?
o Decide: face or not a face
12/6/2013 CS 534: Computation Photography 18
* Next few slides are based on a presentation by Kostantina Pall & Alfredo Kalaitzis, available at http://www1.cs.columbia.edu/~belhumeur/courses/biometrics/2010/violajones.ppt
12/6/2013
5
Face detection: Viola-Jones
Feature type?
• Useful domain knowledge: o The eye region is darker than the forehead or the
upper cheeks
o The nose bridge region is brighter than the eyes
o The mouth is darker than the chin
• Encoding o Location and size: eyes, nose bridge, mouth, etc.
o Value: darker vs. brighter
12/6/2013 CS 534: Computation Photography 19
Face detection: Viola-Jones
Feature type?
• Rectangle features o Value = ∑(pixels in black)
- ∑(pixels in white)
o Three types: 2,3,4 rectangles
o Very fast: integral image
12/6/2013 CS 534: Computation Photography 20
Face detection: Viola-Jones
12/6/2013 CS 534: Computation Photography 21
* From http://www.cs.ubc.ca/~lowe/425/slides/13-ViolaJones.pdf
Which features are important? • Tens of thousands of features to choose from
• AdaBoost (Singer and Schapire, 1997)
o Given a set of weak classifiers: ℎ𝑡 𝑥 ∈ {−1,1}
o Iteratively combine classifiers to form a strong
classifier:
𝐻 𝑥 = 1 𝑖𝑓 𝛼𝑡ℎ𝑡(𝑥)
𝑡> 𝑡ℎ𝑟𝑒𝑠ℎ𝑜𝑙𝑑
0 𝑜𝑡ℎ𝑒𝑟𝑤𝑖𝑠𝑒
Face detection: Viola-Jones
Final decision: face or not a face
• Cascade of classifiers 1. Two-feature classifier: >99% recall, >60% precision
2. Five-feature classifier
3. 10-feature classifier
…
10. 200-feature classifier
12/6/2013 CS 534: Computation Photography 22
12/6/2013
6
Face detection: Viola-Jones
12/6/2013 CS 534: Computation Photography 23
http://vimeo.com/12774628#
Face detection: recent approaches Xiangxin Zhu and Deva Ramanan, Face Detection,
Pose Estimation, and Landmark Localization in the
Wild, CVPR 2012.
12/6/2013 CS 534: Computation Photography 24
Face detection: recent approaches
Shen et al., Detecting and Aligning Faces by Image
Retrieval, CVPR 2013.
12/6/2013 CS 534: Computation Photography 25
Face detection: recent approaches
Shen et al., Detecting and Aligning Faces by Image
Retrieval, CVPR 2013.
12/6/2013 CS 534: Computation Photography 26
12/6/2013
7
Face alignment and landmark
localization: goal
Goal of face alignment: automatically align a face
(usually non-rigidly) to a canonical reference
Goal of face landmark localization: automatically
locate face landmarks of interests
12/6/2013 CS 534: Computation Photography 27
http://www.mathworks.com/matlabcentral/fx_files/32704/4/icaam.jpg
http://homes.cs.washington.edu/~neeraj/projects/face-parts/images/teaser.png
Face alignment and landmark
localization: motivation
• Preprocess for:
o Face recognition
o Portrait editing wizards
o Face image retrieval
o …
• Face tracking
• Expression recognition
• Facial pose recognition
12/6/2013 CS 534: Computation Photography 28
http://static3.businessinsider.com/image/52127e2169bedd4d60000012-752-564/realeyes-facial-recognition.png
http://mission0ps.com/wp-content/uploads/2013/04/10-special-effects.jpg
Face alignment and landmark
localization: challenges
• Pose
• Expression
• Identity variation
• Occlusions
• Image noise
12/6/2013 CS 534: Computation Photography 29
Face alignment and landmark
localization: approaches
Parametric appearance models o Cootes, Edwards, and Taylor, Active Appearance Models, ECCV 1998
12/6/2013 CS 534: Computation Photography 30
12/6/2013
8
Face alignment and landmark
localization: approaches
Parametric appearance models o Cootes, Edwards, and Taylor, Active Appearance Models, ECCV 1998
12/6/2013 CS 534: Computation Photography 31
Face alignment and landmark
localization: approaches
Part-based deformable models o Saragih et al., Face Alignment through Subspace Constrained Mean-
Shifts, ICCV 2009
12/6/2013 CS 534: Computation Photography 32
Face alignment and landmark
localization: approaches
Part-based deformable models o Saragih et al., Face Alignment through Subspace Constrained Mean-
Shifts, ICCV 2009
12/6/2013 CS 534: Computation Photography 33
Face alignment and landmark
localization: approaches
Supervised descent o Xiong and De la Torre, Supervised Descent Method and its Applications to
Face Alignment, CVPR 2013
12/6/2013 CS 534: Computation Photography 34
12/6/2013
9
Face alignment and landmark
localization: approaches
Exemplar-based/non-parametric methods o Shen et al., Detecting and Aligning Faces by Image Retrieval, CVPR 2013.
12/6/2013 CS 534: Computation Photography 35
Face image parsing
Smith, Zhang, Brandt, Lin, and Yang, Exemplar-Based
Face Parsing, CVPR 2013.
12/6/2013 CS 534: Computation Photography 36
Face image parsing: goal
Given an input face image, automatically segment
the face into its constituent parts.
12/6/2013 CS 534: Computation Photography 37
Face image parsing: motivation
• Like face alignment, can be used as a preprocess
for face recognition, automated portrait editing,
etc.
• Encodes ambiguity
• Generalizes to hair, teeth, ears etc. across datasets
12/6/2013 CS 534: Computation Photography 38
12/6/2013
10
Face image parsing: our approach
12/6/2013 CS 534: Computation Photography 39
2K exemplar images 11 landmarks
~150 SIFT features
Exemplar labels
Database
Face image parsing: our approach
12/6/2013 CS 534: Computation Photography 40
2K exemplar images 11 landmarks
~150 SIFT features
Exemplar labels
Database Step 0: Rough alignment & Top exemplar selection
100 top exemplars
Input
Face image parsing: our approach
12/6/2013 CS 534: Computation Photography 41
2K exemplar images 11 landmarks
~150 SIFT features
Exemplar labels
Database Step 0: Rough alignment & Top exemplar selection
Step 1: Nonrigid alignment
Input
Face image parsing: our approach
12/6/2013 CS 534: Computation Photography 42
2K exemplar images 11 landmarks
~150 SIFT features
Exemplar labels
Database Step 0: Rough alignment & Top exemplar selection
Step 1: Nonrigid alignment
Input
Step 2: Exemplar label aggregation
*
+ *
+ …
=
12/6/2013
11
Face image parsing: our approach
12/6/2013 CS 534: Computation Photography 43
2K exemplar images 11 landmarks
~150 SIFT features
Exemplar labels
Database Step 0: Rough alignment & Top exemplar selection
Step 1: Nonrigid alignment
Input
Step 2: Exemplar label aggregation
*
+ *
+
= Step 3: Pixel-wise label selection
w1 * + w2 *
+ w9 *
Label 1 Label 2
… Label 9
Output
∝
Face image parsing: quantitative
results
12/6/2013 CS 534: Computation Photography 44
Face image parsing: qualitative
results
12/6/2013 CS 534: Computation Photography 45
+ Input Soft segments Hard segments Ground truth
Face image parsing: qualitative
results
12/6/2013 CS 534: Computation Photography 46
+ Input Soft segments Hard segments Ground truth