![Page 1: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/1.jpg)
1. Facial Expression Editing in Video Using a Temporally-Smooth Factorization
2. Face Swapping: Automatically Replacing Faces in Photographs
![Page 2: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/2.jpg)
Facial Expression Editing in Video Using a Temporally-Smooth Factorization
Fei Yang, Lubomir Bourdev, Eli Shechtman, Jue Wang, Dimitris Metaxas
CVPR 2012
![Page 3: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/3.jpg)
Goal
The goal is to allow for semantic-level editing of
expressions in a video:
magnifying an expression
suppressing an expression
replacing by another expressions
3
![Page 4: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/4.jpg)
Example
4
![Page 5: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/5.jpg)
Challenges
Natural expression
Different parts changes accordingly
Unique identity
Temporal coherency
5
![Page 6: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/6.jpg)
Related Work 2D based methods
[Theobald09], [Liu01], [Williams90], …
3D based methods [Blanz03], [Pighin98], …
Expression flow [Yang11]…
Frame reorder method [Bregler98], [Kemelmacher- Shlizerman11]
Tensor factorization methods [Vlasic05], [Dale11]…
6
![Page 7: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/7.jpg)
Algorithm
7
Expression Information
Identity Information
3D Tensor Model - [Vlasic et al siggraph05]
Modify
![Page 8: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/8.jpg)
Mode-n Product
8
![Page 9: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/9.jpg)
Algorithm
goal to identify a and
method 2D v.s. 3D
frame t
Minimize: | – |
=
Weak Projective Matrix Rt
![Page 10: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/10.jpg)
Algorithm
Fitting Error:
Shape Distribution Constraint:
Temporal coherence:
![Page 11: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/11.jpg)
Algorithm
11
Levenberg-Marquardt (Siggraph98)
![Page 12: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/12.jpg)
Algorithm
12
Adjust to achieve expression
modification Dynamic Time Warping (DTW)
[Sakoe78]
Residual Expression Flow
Correcting boundary compatibility
![Page 13: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/13.jpg)
Results
13
![Page 14: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/14.jpg)
Face Swapping: Automatically Replacing Faces in Photographs
Dmitri Bitouk Neeraj Kumar Samreen Dhillon Peter Belhumeur Shree K. Nayar
Siggraph 2008
![Page 15: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/15.jpg)
Examples
15
![Page 16: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/16.jpg)
Goals
16
For an input image:
Automatically find the best candidate
Automatically replace the face
Automatically color and lighting adjustmet
![Page 17: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/17.jpg)
Library Building
17
OKAO face detector to detect face pose [Omron07]
![Page 18: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/18.jpg)
Process
18
![Page 19: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/19.jpg)
Alignment
19
Pose, Resolution, and Image Blur:
Yaw, pitch threshold between two images ( )
Eye distance as a measure of distance (80%)
Similarity of the blur degrees [Kundur and
Hatzinakos 1996; Fergus et al. 2006]
![Page 20: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/20.jpg)
Color and Lighting
20
To ensure the similarity between the replaced and original face, a linear combination of 9 spherical harmonics [Ramamoorthi and Hanrahan 2001; Basri and Jacobs 2003] is used as measure metric:Each pixel I(x, y) can be approximated by:
Distance:
![Page 21: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/21.jpg)
Seam Signature
21
256-by-256 patch from the face is used for replacement.
Unfold:
L2 Norm is used to compute the distance
![Page 22: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/22.jpg)
Appearance Adjustment
Using simple scaling on the Harmonics coefficients
, are the original and replacement images
Scale the replaced image
![Page 23: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/23.jpg)
Results
![Page 24: 1. Facial Expression Editing in Video Using a Temporally- Smooth Factorization 2. Face Swapping: Automatically Replacing Faces in Photographs](https://reader038.vdocuments.net/reader038/viewer/2022110205/56649cb95503460f949810da/html5/thumbnails/24.jpg)
The End
24
Any Questions
?