face poser: interactive modeling of 3d facial expressions using model priors manfred lau 1,3...
Post on 15-Jan-2016
224 Views
Preview:
TRANSCRIPT
Face Poser:Interactive Modeling of 3D Facial Expressions Using Model Priors
Manfred Lau1,3 Jinxiang Chai2 Ying-Qing Xu3 Heung-Yeung Shum3
1Carnegie Mellon University 2Texas A&M University 3Microsoft Research Asia
Face Poser
Inputs
Generate new facial expressions with a simple and intuitive interface
Face Poser
Inputs Output
Generate new facial expressions with a simple and intuitive interface
Why Face Poser?
Pre-defined controlsDifficult to build and use
Complex facial expressions
Applications
Films, Games Virtual Reality
Educational
Related Work
Sketched-based interfacesZeleznik et al. 96
Igarashi et al. 99Nealen et al. 05Kho and Garland 05Chang and Jenkins 06
Nealen et al. 05
Igarashi et al. 99
Related Work
Example-based modelingBlanz and Vetter 99
Pighin et al. 99Chai et al. 03Zhang et al. 04Grochow et al. 04Sumner et al. 05
Sumner et al. 05
Grochow et al. 04
OverviewDatabase
PreprocessingModel Prior
OverviewDatabase
PreprocessingModel Prior
Neutral Pose
User Constraints
Interface
OverviewDatabase
PreprocessingModel Prior
RuntimeOptimization
Neutral Pose
New Pose
User Constraints
Interface
Textured Pose
Motion capture data
Captured mesh animations of various facial expressions: anger, fear, surprise, sadness, joy, disgust, speaking, singing
All meshes translated and rotated to a standard view:
Data: PCA representation
x =
v1x
v1y
v1z
v2x
. . .
p is low-dimensional representation of x
Problem statement
Find best p satisfying user-constraints c:
Best p is:
Given a face model, how well does it match user-constraints
Likelihood of face model using knowledge of data
Point Constraints
More detailed control
User inputs:blue – 3D source vertexgreen – 2D target pixel
Can select in any camera view
Point Constraints
We optimize for best p
For each p: compute whole mesh x take selected 3D source vertex project it to 2D screen space compare to target pixel
Point Constraints
Optimization term:
Jacobian term:
Point Constraints
Inputs Solution
Point Constraints – Results
Point Constraints – Dragging interface
Stroke Constraints
Large-scale changes with minimal input
User inputs:blue – 2D source stroke (selects
3D points on mesh)green – 2D target stroke
Any curve, line, or freeform region
Stroke Constraints
2D source stroke raytrace each pixel to mesh surface to get dark gray points
These can be 3D points on mesh surface (not just original mesh vertices)
Stroke Constraints
We optimize for best p
For each p: compute whole mesh x take selected 3D points project them to 2D screen space compare to target stroke
Stroke Constraints
Optimization term:
Jacobian term:
Stroke Constraints
Inputs Solution
Stroke Constraints – Results
Stroke Constraints – Tablet interface
Stroke Constraints – Additional term
If strokes are far away from each other, energy term will reach local minimum
Need additional optimization term to minimize distance between “center” of source stroke and “center” of target stroke
Without additional term
Stroke Constraints – Additional term
Without additional term
Optimization term:
Jacobian term:
Stroke Constraints – Additional term
Without additional term With additional term
Stroke Constraints – Results
Problem statement
Find best p satisfying user-constraints c:
Best p is:
Given a face model, how well does it match user-constraints
Likelihood of face model using knowledge of data
Model Priors
There can be many solutions satisfying user constraints. Some of them are not realistic.
We add another optimization term to constrain the solution to the space defined by the motion capture data.
Without model priors term
Model Priors
Without model priors term
Learn a Mixtures of Factor Analyzers (MFA) model
Probability density function to measure naturalness of facial expression
MFA has been applied to high-dimensional nonlinear data
Model Priors
Without model priors term
Optimization term:
Jacobian term:
Model Priors – Result
Model Priors – Result
increasing weight on Model Prior term
Model Priors – Result
Computation time
Standard PC hardware (Pentium IV 2 GHz)
Point constraintstakes 0.18 seconds for 10 pointstime increases linearly with number of points
Stroke constraintstakes 0.4 seconds for source stroke of ~900 pixels (about
size of eyebrow)time increases linearly with number of pixelsfaster if using intermediate spline representation
Cross validation
New face expression samples for testingUse new samples to get target constraintsGenerate solution and compare with test sample
Cross validation
Ground truth Interpolation Optimization
Comparison with other techniques
Opt-blend: FaceIK [Zhang et al. 04]PCA: Morphable model [Blanz and Vetter 99; Blanz et al. 03]LWR: Locally weighted regression
3D errors
Comparison with other techniques
Ground truth, Optimization with PCA, Optimization with MFA
Application: Trajectory Keyframing
Green points – given 2D target pixelsBlue points and mesh – solution
Application: Trajectory Keyframing
Ground truth Result
Application: Trajectory Keyframing
Ground truth Result
Application: Trajectory Keyframing
Ground truth Result
Summary: Face Poser
Inputs Output
Users can learn to use our system within minutes and can create new facial expressions within seconds
Limitation
Global control changing mouth
also changes eyes this is natural, but
difficult to control sometimes
Local control changing mouth
without changing eyes
but this might lead to “fake smiles”
Extensions / Future work
We have added different types of constraints within the same optimization framework
More general: model face as separate regions, generate each region separately, and blend them back together
top related