model-based stereo with occlusions
DESCRIPTION
Model-Based Stereo with Occlusions. Fabiano Romeiro and Todd Zickler. TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: A A A A A A A A A. Introduction. Varying illumination. Varying pose. Occlusions. Varying expressions. Introduction. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/1.jpg)
Model-Based Stereo with Occlusions
Fabiano Romeiro and Todd Zickler
![Page 2: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/2.jpg)
Introduction
Varying illumination
Varying pose
Occlusions
Varying expressions
![Page 3: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/3.jpg)
Introduction
Eigenfaces [Turk and Pentland, 1991]
[Belhumeur et al, 1997]Fisherfaces
Past Work: Image-based
For example:
![Page 4: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/4.jpg)
3D Morphable Models (3DMMs)
[Blanz et al, 1999; Blanz et al, 2003; Blanz et al, 2005; Smet et al, 2006]
Introduction
2D AAMs
[Cootes et al, 1998; Baker et al, 2004; Mathews et al, 2004; Gross et al, 2006]
2D+3D AAMs
[Xiao et al, 2004]
Past Work: Model-based
![Page 5: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/5.jpg)
Introduction
Pros
- Self-occlusion handled by model itself-- Allows direct modeling of illumination
Cons
- Difficult and expensive fitting process
Past Work: 3DMMs3D Morphable Models (3DMMs)
[Blanz et al, 1999; Blanz et al, 2003; Blanz et al, 2005; Smet et al, 2006]
![Page 6: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/6.jpg)
Introduction
- Texture model not needed
Past Work: Stereo 3DMMs
Our work[Fransens et al, 2005]
- Stereo based cost
d(I 1; I 2)
-Stereo fitting with both shape and texture
→ Improved Accuracy
→ Robust to foreign-body occlusions
![Page 7: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/7.jpg)
Outline
• 3DMM Background• Joint Shape and Texture Stereo Fitting• Handling Occlusions• Conclusions
![Page 8: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/8.jpg)
Background
3DMMs
Vectorization of laser scans:
PCA performed:
F Si = [[X i1Y
i1Z i
1]T :::[X i
N Y iN Z i
N ]T ]T
- Basis for shape fSi gi=1;::;m
F Ti = [[R i1G
i1B
i1]
T :::[R iN Gi
N B iN ]T ]T
f Ti gi=1;::;m- Basis for texture[Blanz and Vetter, 1999]
![Page 9: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/9.jpg)
Representation of face shape and texture:
3DMMs
Background
S = S0 +mX
i=1
®i Si
Prior probabilities on the coefficients:
P (~®) / exp(¡12
mX
i=1
(®i
¾i)2)
P (~̄) / exp(¡12
mX
i=1
(¯ i
°i)2) [Blanz and Vetter, 1999]
T = T0 +mX
i=1
¯ i Ti
![Page 10: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/10.jpg)
Stereo Match
I 1(P1sk) ¼I 2(P2sk)
Shape (®)Pose (R;t)
sk
P1 P2
I 1 I 2
![Page 11: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/11.jpg)
Texture Match
P1 P2
I m(k) ¼ ¹I (sk) = I 1(P1sk )+I 2(P2sk )2
I m(k) Texture (¯)Shape (®)
Lighting (K o®set, K amb, K dir)K =fR,G,Bg
skShape (®)Pose (R;t)
I 1 I 2
![Page 12: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/12.jpg)
Joint Shape and Texture Stereo Fitting of 3DMMs
E =
P (Shape;Texture;P ose;L ightingjI 1; I 2) =
= P (Shape;P osejI 1; I 2)¢
X
kjvk 2V
jjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
| {z }Stereo Match
+mX
i=1
(®i
¾i)2
| {z }Shape Prior
+
+X
kjvk 2V
jjI m(k) ¡ ¹I (sk)jj2
¾2t
| {z }Texture Match
+mX
i=1
(¯ i
°i)2
| {z }TexturePrior
P (Texture;L ightingjI 1; I 2;Shape;P ose)
![Page 13: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/13.jpg)
X
kjvk 2V
ha
µjjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
+jjI m(k) ¡ ¹I (sk)jj2
¾2t
¶X
kjvk 2V
jjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
| {z }Stereo Match
Binary occlusion map O : f1;::;N g ! f0;1gN
Robust Stereo Fitting of 3DMMs
E = +mX
i=1
(®i
¾i)2
| {z }Shape Prior
+
+X
kjvk 2V
jjI m(k) ¡ ¹I (sk)jj2
¾2t
| {z }TextureMatch
+mX
i=1
(¯ i
°i)2
| {z }Texture Prior
+jjI m(k) ¡ ¹I (sk)jj2
¾2t
+
P (Shape;Texture;P ose;L ighting;OjI 1; I 2) =
= P (Shape;P ose;OjI 1; I 2)¢P (Texture;L ightingjI 1; I 2;Shape;P ose;O)
![Page 14: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/14.jpg)
Eo =X
kjvk 2V
ha
µjjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
+jjI m(k) ¡ ¹I (sk)jj2
¾2t
¶
+mX
i=1
(®i
¾i)2 +
mX
i=1
(¯ i
°i)2
Optimization Procedure
Initial fitE f- Fit Shape, Pose to minimize reprojection error
on selected feature points
- Rough initial estimates of Shape and Pose
Optimization procedure
Eo + ¸ ¢E f
Eo
4 experiments
Stereo and texture
Eo Stereo
Eo Robust Stereo and texture
Eo Robust Stereo
Eo =X
kjvk 2V
jjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
| {z }Stereo Match
+mX
i=1
(®i
¾i)2
| {z }ShapePrior
+
X
kjvk 2V
jjI m(k) ¡ ¹I (sk)jj2
¾2t
| {z }TextureMatch
+mX
i=1
(¯ i
°i)2
| {z }TexturePrior
Eo =X
kjvk 2V
ha
µjjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
¶
+mX
i=1
(®i
¾i)2
Eo =X
kjvk 2V
jjI 1(P1sk) ¡ I 2(P2sk)jj2
¾2s
| {z }Stereo Match
+mX
i=1
(®i
¾i)2
| {z }Shape Prior
![Page 15: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/15.jpg)
Results
480 recovered shape models (60 individuals, 8 poses)
K.U. Leuven Stereo face database
First 2 experiments: Stereo and Texture vs. Stereo
[Fransens et al, 2005]
![Page 16: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/16.jpg)
ResultsQualitative Results
Stereo
Stereo and Texture
Stereo Matching CostStereo 280:77Stereo+Texture 340:17
![Page 17: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/17.jpg)
Results
Stereo
Stereo+texture
Qualitative Results
![Page 18: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/18.jpg)
Results
Stereo Stereo+Texturetr(S¡ 1
w Sb) 69:4101 104:0478det(S¡ 1
w Sb) 1:3418e¡ 11 2:9640e¡ 5
Quantitative Results
![Page 19: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/19.jpg)
Results
Occluder classi¯cation Half Full near Full fartextureless (non-skincolor) Stereo Texture Texturetextured Stereo Texture Stereo+Texturetextureless (skincolor) X X X
Half-Occlusion Full Occlusion near Full-Occlusion far
Under Occlusions
![Page 20: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/20.jpg)
Results
Occluder classi¯cation Half Full near Full fartextureless (non-skincolor) Stereo Texture Texturetextured Stereo Texture Stereo+Texturetextureless (skincolor) X X X
Under Occlusions
Input
Shape Estimate
Occlusion Map
Robust Stereo Robust S+T Robust Stereo Robust S+T
![Page 21: Model-Based Stereo with Occlusions](https://reader035.vdocuments.net/reader035/viewer/2022062409/568150c7550346895dbee857/html5/thumbnails/21.jpg)
Conclusions
Robust stereo fitting of 3DMMs
- Uses both stereo constraint, texture information
- Increased accuracy of fit
- Ability to handle occlusions
Future Work
- More sophisticated stereo matching term
- Different feature spaces
- Break model into segments respecting occlusion boundaries