![Page 1: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/1.jpg)
Face Poser:Interactive Modeling of 3D Facial Expressions Using Model Priors
Manfred Lau1,3 Jinxiang Chai2 Ying-Qing Xu3 Heung-Yeung Shum3
1Carnegie Mellon University 2Texas A&M University 3Microsoft Research Asia
![Page 2: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/2.jpg)
Face Poser
Inputs
Generate new facial expressions with a simple and intuitive interface
![Page 3: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/3.jpg)
Face Poser
Inputs Output
Generate new facial expressions with a simple and intuitive interface
![Page 4: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/4.jpg)
Why Face Poser?
Pre-defined controlsDifficult to build and use
Complex facial expressions
![Page 5: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/5.jpg)
Applications
Films, Games Virtual Reality
Educational
![Page 6: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/6.jpg)
Related Work
Sketched-based interfacesZeleznik et al. 96
Igarashi et al. 99Nealen et al. 05Kho and Garland 05Chang and Jenkins 06
Nealen et al. 05
Igarashi et al. 99
![Page 7: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/7.jpg)
Related Work
Example-based modelingBlanz and Vetter 99
Pighin et al. 99Chai et al. 03Zhang et al. 04Grochow et al. 04Sumner et al. 05
Sumner et al. 05
Grochow et al. 04
![Page 8: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/8.jpg)
OverviewDatabase
PreprocessingModel Prior
![Page 9: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/9.jpg)
OverviewDatabase
PreprocessingModel Prior
Neutral Pose
User Constraints
Interface
![Page 10: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/10.jpg)
OverviewDatabase
PreprocessingModel Prior
RuntimeOptimization
Neutral Pose
New Pose
User Constraints
Interface
Textured Pose
![Page 11: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/11.jpg)
Motion capture data
Captured mesh animations of various facial expressions: anger, fear, surprise, sadness, joy, disgust, speaking, singing
All meshes translated and rotated to a standard view:
![Page 12: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/12.jpg)
Data: PCA representation
x =
v1x
v1y
v1z
v2x
. . .
p is low-dimensional representation of x
![Page 13: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/13.jpg)
Problem statement
Find best p satisfying user-constraints c:
Best p is:
Given a face model, how well does it match user-constraints
Likelihood of face model using knowledge of data
![Page 14: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/14.jpg)
Point Constraints
More detailed control
User inputs:blue – 3D source vertexgreen – 2D target pixel
Can select in any camera view
![Page 15: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/15.jpg)
Point Constraints
We optimize for best p
For each p: compute whole mesh x take selected 3D source vertex project it to 2D screen space compare to target pixel
![Page 16: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/16.jpg)
Point Constraints
Optimization term:
Jacobian term:
![Page 17: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/17.jpg)
Point Constraints
Inputs Solution
![Page 18: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/18.jpg)
Point Constraints – Results
![Page 19: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/19.jpg)
Point Constraints – Dragging interface
![Page 20: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/20.jpg)
Stroke Constraints
Large-scale changes with minimal input
User inputs:blue – 2D source stroke (selects
3D points on mesh)green – 2D target stroke
Any curve, line, or freeform region
![Page 21: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/21.jpg)
Stroke Constraints
2D source stroke raytrace each pixel to mesh surface to get dark gray points
These can be 3D points on mesh surface (not just original mesh vertices)
![Page 22: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/22.jpg)
Stroke Constraints
We optimize for best p
For each p: compute whole mesh x take selected 3D points project them to 2D screen space compare to target stroke
![Page 23: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/23.jpg)
Stroke Constraints
Optimization term:
Jacobian term:
![Page 24: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/24.jpg)
Stroke Constraints
Inputs Solution
![Page 25: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/25.jpg)
Stroke Constraints – Results
![Page 26: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/26.jpg)
Stroke Constraints – Tablet interface
![Page 27: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/27.jpg)
Stroke Constraints – Additional term
If strokes are far away from each other, energy term will reach local minimum
Need additional optimization term to minimize distance between “center” of source stroke and “center” of target stroke
Without additional term
![Page 28: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/28.jpg)
Stroke Constraints – Additional term
Without additional term
Optimization term:
Jacobian term:
![Page 29: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/29.jpg)
Stroke Constraints – Additional term
Without additional term With additional term
![Page 30: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/30.jpg)
Stroke Constraints – Results
![Page 31: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/31.jpg)
Problem statement
Find best p satisfying user-constraints c:
Best p is:
Given a face model, how well does it match user-constraints
Likelihood of face model using knowledge of data
![Page 32: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/32.jpg)
Model Priors
There can be many solutions satisfying user constraints. Some of them are not realistic.
We add another optimization term to constrain the solution to the space defined by the motion capture data.
Without model priors term
![Page 33: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/33.jpg)
Model Priors
Without model priors term
Learn a Mixtures of Factor Analyzers (MFA) model
Probability density function to measure naturalness of facial expression
MFA has been applied to high-dimensional nonlinear data
![Page 34: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/34.jpg)
Model Priors
Without model priors term
Optimization term:
Jacobian term:
![Page 35: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/35.jpg)
Model Priors – Result
![Page 36: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/36.jpg)
Model Priors – Result
increasing weight on Model Prior term
![Page 37: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/37.jpg)
Model Priors – Result
![Page 38: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/38.jpg)
Computation time
Standard PC hardware (Pentium IV 2 GHz)
Point constraintstakes 0.18 seconds for 10 pointstime increases linearly with number of points
Stroke constraintstakes 0.4 seconds for source stroke of ~900 pixels (about
size of eyebrow)time increases linearly with number of pixelsfaster if using intermediate spline representation
![Page 39: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/39.jpg)
Cross validation
New face expression samples for testingUse new samples to get target constraintsGenerate solution and compare with test sample
![Page 40: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/40.jpg)
Cross validation
Ground truth Interpolation Optimization
![Page 41: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/41.jpg)
Comparison with other techniques
Opt-blend: FaceIK [Zhang et al. 04]PCA: Morphable model [Blanz and Vetter 99; Blanz et al. 03]LWR: Locally weighted regression
3D errors
![Page 42: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/42.jpg)
Comparison with other techniques
Ground truth, Optimization with PCA, Optimization with MFA
![Page 43: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/43.jpg)
Application: Trajectory Keyframing
Green points – given 2D target pixelsBlue points and mesh – solution
![Page 44: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/44.jpg)
Application: Trajectory Keyframing
Ground truth Result
![Page 45: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/45.jpg)
Application: Trajectory Keyframing
Ground truth Result
![Page 46: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/46.jpg)
Application: Trajectory Keyframing
Ground truth Result
![Page 47: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/47.jpg)
Summary: Face Poser
Inputs Output
Users can learn to use our system within minutes and can create new facial expressions within seconds
![Page 48: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/48.jpg)
Limitation
Global control changing mouth
also changes eyes this is natural, but
difficult to control sometimes
Local control changing mouth
without changing eyes
but this might lead to “fake smiles”
![Page 49: Face Poser: Interactive Modeling of 3D Facial Expressions Using Model Priors Manfred Lau 1,3 Jinxiang Chai 2 Ying-Qing Xu 3 Heung-Yeung Shum 3 1 Carnegie](https://reader033.vdocuments.net/reader033/viewer/2022051401/56649d4a5503460f94a26ba3/html5/thumbnails/49.jpg)
Extensions / Future work
We have added different types of constraints within the same optimization framework
More general: model face as separate regions, generate each region separately, and blend them back together