3d landmark model discovery from a registered set of ...€¦ · 3d landmark model discovery from a...
TRANSCRIPT
3D Landmark Model Discoveryfrom a Registered Set of Organic Shapes
Clement Creusot, Nick Pears, Jim Austin
Department of Computer science
PCP, CVPRW, Providence (RI), June 2012
What/Why
How
Results
Conclusion
References
Plan
What/Why: Generalities and Problems
How: Proposed method
Results
Conclusion
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 2 / 20
What/Why
How
Results
Conclusion
References
Generalities
Where is Wally?Waldo?Charlie?Walter?
ウォーリー?威利?
. . .
Scene
Output
Detector
Model
Model Discoverer
?
s
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 3 / 20
What/Why
How
Results
Conclusion
References
Generalities
Where is Wally?Waldo?Charlie?Walter?
ウォーリー?威利?
. . .
Scene Output
Detector
Model
Model Discoverer
?
s
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 3 / 20
What/Why
How
Results
Conclusion
References
Generalities
Where is Wally?Waldo?Charlie?Walter?
ウォーリー?威利?
. . .
Scene Output
Detector
Model
Model Discoverer
?
s
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 3 / 20
What/Why
How
Results
Conclusion
References
Generalities
Where is Wally?Waldo?Charlie?Walter?
ウォーリー?威利?
. . .
Scene Output
Detector
Model
Model Discoverer
?
s
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 3 / 20
What/Why
How
Results
Conclusion
References
Generalities
Where is Wally?Waldo?Charlie?Walter?
ウォーリー?威利?
. . .
Scene Output
Detector
Model
Model Discoverer
?
s
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 3 / 20
What/Why
How
Results
Conclusion
References
Model Discovery for 3D Face Landmarking
s
Scene Output
Detector
Landmarker
Model
Model
Model Discoverer
?
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 4 / 20
What/Why
How
Results
Conclusion
References
Model Discovery for 3D Face Landmarking
s
Scene Output
Detector
Landmarker
Model
Model
Model Discoverer
?
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 4 / 20
What/Why
How
Results
Conclusion
References
Model Discovery for 3D Face Landmarking
s
Scene Output
Detector
Landmarker
Model
Model
Model Discoverer
?
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 4 / 20
What/Why
How
Results
Conclusion
References
Why? - Gap in Research
[Amberg et al., 2007] [Creusot et al., 2011] [Gupta et al., 2007]
[Romero-Huertas and Pears, 2008] [Szeptycki et al., 2009] [Zhao et al., 2011]
Easy to label or explain to an operator
Linked to 2D projections and plane symmetries
Overall arbitrary
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 5 / 20
What/Why
How
Results
Conclusion
References
Nature of a model for a 3D-object class
Sparse
“Descriptive”
Featural/Local information (nodes)
Structural/Global information (edges/hyperedges)
Possible Local Features:
Object Points Curves Surfaces Volumes
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 6 / 20
What/Why
How
Results
Conclusion
References
Nature of a model for a 3D-object class
Sparse
“Descriptive”
Featural/Local information (nodes)
Structural/Global information (edges/hyperedges)
Possible Local Features:
Object Points Curves Surfaces Volumes
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 6 / 20
What/Why
How
Results
Conclusion
References
Organicly-shape objects
More possible point-models than geometric shapes
Less intuition about what model is good
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 7 / 20
What/Why
How
Results
Conclusion
References
Example of 3D-objects point models
Articulated Models:
Articulations
Extremities
Non-Articulated Models:
???
???
[Shotton et al., 2011] [Bray et al., 2004] [Creusot et al., 2011]
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 8 / 20
What/Why
How
Results
Conclusion
References
Hypothesis
“Probabilistic”response mapavailable
One point permodel
Scene Output
Landmarker
Model
Model Discoverer
?
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 9 / 20
What/Why
How
Results
Conclusion
References
Hypothesis
“Probabilistic”response mapavailable
One point permodel
Scene Output
Landmarker
Model
Model Discoverer
?
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 9 / 20
What/Why
How
Results
Conclusion
References
Our Approach
Use Detector and Neighborhood definition from[Creusot et al., 2011]
8 Local DescriptorsGaussian DistributionsLinear Combination (LDA based)
Test as many models as there are vertices in thetemplate mesh (∼ 2000)
Define two cost functions for each model:
Saliency: Different from its neighborhood (good)Ubiquity: Ubiquitous over the face (bad)
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 10 / 20
What/Why
How
Results
Conclusion
References
Databases
FRGC (real) BFM (synthetic)(Coarse Correspondence) (Fine Correspondence)
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 11 / 20
What/Why
How
Results
Conclusion
References
Saliency Score per Vertex
0.962 0.962 0.981 0.981 1.00 1.00
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0 0.2 0.4 0.6 0.8 1
Densi
ty
Score
0
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
0 0.2 0.4 0.6 0.8 1
Densi
ty
Score
0
0.05
0.1
0.15
0.2
0.25
0 0.2 0.4 0.6 0.8 1
Densi
ty
Score
0
0.05
0.1
0.15
0.2
0.25
0 0.2 0.4 0.6 0.8 1
Densi
ty
Score
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.2 0.4 0.6 0.8 1
Densi
ty
Score
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
0 0.2 0.4 0.6 0.8 1
Densi
ty
Score
0
0.05
0.1
0.15
0.2
0.25
0.3
0 0.2 0.4 0.6 0.8 1
Densi
ty
Score
0
0.05
0.1
0.15
0.2
0.25
0.3
0 0.2 0.4 0.6 0.8 1
Densi
ty
Score
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 12 / 20
What/Why
How
Results
Conclusion
References
Ubiquity Score per Vertex
15.8 15.8 586. 586. 1.16e+031.16e+03
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 13 / 20
What/Why
How
Results
Conclusion
References
Results
Manual Automatic
Init
ial
Sym
met
ry
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 14 / 20
What/Why
How
Results
Conclusion
References
Saliency
Similar
ID 8 3 1 2 5 6 7U 658.37 215.52 287.70 67.452 252.50 393.68 324.40
Au
tom
atic
ID 0 1 2 3 4 5 6U 683.62 192.43 492.43 87.408 328.08 226.66 206.76
Man
ual
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 15 / 20
What/Why
How
Results
Conclusion
References
Saliency
Different
0 4 9U 325.51 561.31 416.18
Au
tom
atic
ID 7 8 9U 701.64 954.16 444.54
Man
ual
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 16 / 20
What/Why
How
Results
Conclusion
References
Problems
Different answers depending on the registrationmethod:
Fine registration on clean data (BFM)Coarse registration on unclean data (FRGC)Fine registration on unclean data (???) Needed
Optimization method → Depends on the detectorused (and its parameters)
How to include structural information in the modeldiscovery?
How to project a newly discovered model to unseentraining data? (again a registration problem)
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 17 / 20
What/Why
How
Results
Conclusion
References
Conclusion
Good:
Optimize model for a detectorValidate most human-chosen landmarksGive quantifiable measure of landmark quality
Bad:
Only non-articulated objects for nowRequires a large set of finely-registered objects (Doyou have one to share?)
Questions to you:
How do you learn a model structure in yourapplication domain?Are there applications where you think this mighthelp?Brain teaser: How do you extend the idea tomulti-dimensional features (curves, area, volumes)?
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 18 / 20
What/Why
How
Results
Conclusion
References
References I
Amberg, B., Romdhani, S., and Vetter, T. (2007).
Optimal step nonrigid icp algorithms for surface registration.In IEEE Int Conf. CVPR.
Bray, M., Koller-Meier, E., Mueller, P., Gool, L. V., and Schraudolph, N. N.
(2004).3d hand tracking by rapid stochastic gradient descent using a skinning model.In Chambers, A. and Hilton, A., editors, 1st European Conference on Visual MediaProduction (CVMP), pages 59–68. IEE.
Creusot, C., Pears, N., and Austin, J. (2011).
Automatic keypoint detection on 3d faces using a dictionary of local shapes.In 3DIMPVT, pages 204–211.
Gupta, S., Markey, M. K., Aggarwal, J., and Bovik, A. C. (2007).
Three dimensional face recognition based on geodesic and euclidean distances.In IS&T/SPIE Symp. on Electronic Imaging: Vision Geometry XV.
Romero-Huertas, M. and Pears, N. (2008).
3d facial landmark localisation by matching simple descriptors.In IEEE Int. Conf. BTAS, pages 1–6.
Shotton, J., Fitzgibbon, A., Cook, M., Sharp, T., Finocchio, M., Moore, R.,
Kipman, A., and Blake, A. (2011).Real-time human pose recognition in parts from single depth images.In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on, pages 1297–1304.
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 19 / 20
What/Why
How
Results
Conclusion
References
References II
Szeptycki, P., Ardabilian, M., and Chen, L. (2009).
A coarse-to-fine curvature analysis-based rotation invariant 3D facelandmarking.In IEEE Int. Conf. BTAS, pages 32–37.
Zhao, X., Dellandre anda, E., Chen, L., and Kakadiaris, I. A. (2011).
Accurate landmarking of three-dimensional facial data in the presence of facialexpressions and occlusions using a three-dimensional statistical facial featuremodel.IEEE Trans. Syst. Man, and Cybernetics, Part B: Cybernetics, 41(5):1417–1428.
Clement Creusot PCP, CVPRW, Providence (RI), June 2012, 20 / 20