1. Facial Expression Editing in Video Using a Temporally-Smooth Factorization
2. Face Swapping: Automatically Replacing Faces in Photographs
Facial Expression Editing in Video Using a Temporally-Smooth Factorization
Fei Yang, Lubomir Bourdev, Eli Shechtman, Jue Wang, Dimitris Metaxas
CVPR 2012
Goal
The goal is to allow for semantic-level editing of
expressions in a video:
magnifying an expression
suppressing an expression
replacing by another expressions
3
Example
4
Challenges
Natural expression
Different parts changes accordingly
Unique identity
Temporal coherency
5
Related Work 2D based methods
[Theobald09], [Liu01], [Williams90], …
3D based methods [Blanz03], [Pighin98], …
Expression flow [Yang11]…
Frame reorder method [Bregler98], [Kemelmacher- Shlizerman11]
Tensor factorization methods [Vlasic05], [Dale11]…
6
Algorithm
7
Expression Information
Identity Information
3D Tensor Model - [Vlasic et al siggraph05]
Modify
Mode-n Product
8
Algorithm
goal to identify a and
method 2D v.s. 3D
frame t
Minimize: | – |
=
Weak Projective Matrix Rt
Algorithm
Fitting Error:
Shape Distribution Constraint:
Temporal coherence:
Algorithm
11
Levenberg-Marquardt (Siggraph98)
Algorithm
12
Adjust to achieve expression
modification Dynamic Time Warping (DTW)
[Sakoe78]
Residual Expression Flow
Correcting boundary compatibility
Results
13
Face Swapping: Automatically Replacing Faces in Photographs
Dmitri Bitouk Neeraj Kumar Samreen Dhillon Peter Belhumeur Shree K. Nayar
Siggraph 2008
Examples
15
Goals
16
For an input image:
Automatically find the best candidate
Automatically replace the face
Automatically color and lighting adjustmet
Library Building
17
OKAO face detector to detect face pose [Omron07]
Process
18
Alignment
19
Pose, Resolution, and Image Blur:
Yaw, pitch threshold between two images ( )
Eye distance as a measure of distance (80%)
Similarity of the blur degrees [Kundur and
Hatzinakos 1996; Fergus et al. 2006]
Color and Lighting
20
To ensure the similarity between the replaced and original face, a linear combination of 9 spherical harmonics [Ramamoorthi and Hanrahan 2001; Basri and Jacobs 2003] is used as measure metric:Each pixel I(x, y) can be approximated by:
Distance:
Seam Signature
21
256-by-256 patch from the face is used for replacement.
Unfold:
L2 Norm is used to compute the distance
Appearance Adjustment
Using simple scaling on the Harmonics coefficients
, are the original and replacement images
Scale the replaced image
Results
The End
24
Any Questions
?