3D Face Modeling
Michaël De Smet
Topics to Discuss
3D Morphable Models 3D face reconstruction Face recognition Lip synchronization
Topics to Discuss
3D Morphable Models 3D face reconstruction Face recognition Lip synchronization
3D Morphable Models
Statistical model of shape and texture Derived from laser scans
USF DARPA HumanID 3D Face Database Processing
Hole filling Surface smoothing Albedo estimation Dense correspondence
3D Morphable Models
-2 +2
…
-2 +2
…
Topics to Discuss
3D Morphable Models 3D face reconstruction Face recognition Lip synchronization
3D Face Reconstruction
Fitting the 3DMM to one or more images of the same face
•Scale•Rotation•Translation•Illumination
•Shape•Texture
?
?
Optimization problem with > 100 parameters
3D Face Reconstruction
Feature points Feature alignment Fitting result
3D Face Reconstruction
Dealing with Occlusions
Dealing with Occlusions
Without occlusionhandling
With occlusionhandling
3D Face Reconstruction
3D Face Reconstruction
3D Face Reconstruction
3D Face Reconstruction
3D Face Reconstruction
3D Face Reconstruction
Topics to Discuss
3D Morphable Models 3D face reconstruction Face recognition Lip synchronization
Face RecognitionFit the 3DMM to an image of an unknown face
•Scale•Rotation•Translation•Illumination
•Shape•Texture
Compare to database Recognition
?
?
Face Recognition
In controlled settings, almost perfect recognition is possible
Pose 1100.0%
Pose 2100.0%
Pose 3N/A
Pose 4100.0%
Pose 595.7%
Training view
Face Recognition
Uncontrolled environments are challenging Face orientation unknown Difficult illumination Facial expressions Occlusions Low resolution Motion blur …
Face Recognition
European parliament video: 21 persons, 86% correct
Face Recognition
VRT news broadcasts: 12 persons
Face Recognition
VRT news broadcasts: 12 persons
82.3% correctrecognition
Topics to Discuss
3D Morphable Models 3D face reconstruction Face recognition Lip synchronization
Lip Synchronization Speech driven animation Texture based, i.e. shape is fixed Strategy:
Extract 3D model of speaker’s face Track rigid motion of the face in video Extract texture for each frame Compute PCA model of texture Train ANN to link phonemes and PCA
coefficients (visemes)
System Overview
AutomaticAutomaticPhonePhone
RecognitionRecognition
NeuralNeuralNetworkNetwork
FaceFaceSynthesisSynthesis
SpeechSpeechfeature vectorsfeature vectors
FacialFacialfeature vectorsfeature vectors
Training Setup
AutomaticAutomaticPhonePhone
RecognitionRecognition
NeuralNeuralNetworkNetworkTrainingTraining
FaceFaceAnalysisAnalysis
SpeechSpeechfeature vectorsfeature vectors
FacialFacialfeature vectorsfeature vectors
Video Processing
3D face model acquisition Rigid motion tracking Normalized texture extraction Texture feature extraction (PCA)
Conclusion
3DMMs are a very powerful tool for face modeling
Many applications in computer vision and computer graphics