stereo matching with nonparametric smoothness priors in...
TRANSCRIPT
![Page 1: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/1.jpg)
Stereo Matching with Nonparametric Smoothness Priors in Feature Space
Brandon M. Smith1, Li Zhang1, and Hailin Jin2
1 University of Wisconsin – Madison
2 Adobe Systems Incorporated1
![Page 2: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/2.jpg)
Motivation
State-of-the-art two-view stereo methods9 out of top 10 employ image segmentation
UW-Madison Computer Graphics and Vision Group2
![Page 3: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/3.jpg)
UW-Madison Computer Graphics and Vision Group
Motivation
Image segmentation problems
3
![Page 4: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/4.jpg)
UW-Madison Computer Graphics and Vision Group
Motivation
Segmentation artifacts in video: temporal instability
1 of 5 input views 2nd-order smoothness method with segmentation[Woodford et al. CVPR ’08]
4
![Page 5: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/5.jpg)
UW-Madison Computer Graphics and Vision Group
Inspiration: Adaptive Support Weight
Yoon & Kweon, Locally adapt. support-weight approach for vis. corr. search, CVPR ‘05
Intensity-encoded weightsClose-up views of matching window
5
![Page 6: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/6.jpg)
UW-Madison Computer Graphics and Vision Group
Inspiration: Adaptive Support Weight
Can we incorporate this idea into a global inference algorithm?
Large, weighted smoothness nbrhoodClose-up views of matching window
6
![Page 7: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/7.jpg)
UW-Madison Computer Graphics and Vision Group
Inspiration: Sparse Neighborhoods
O. Veksler, Stereo Correspondence by Dynamic Programming on a Tree, CVPR ‘05
Image close-up views Common smoothness neighborhood structure
Sparse tree smoothness neighborhood structure
Minimum spanning tree
algorithm
7
![Page 8: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/8.jpg)
UW-Madison Computer Graphics and Vision Group
Our Approach
Minimum spanning tree
algorithm
Most important edgesLarge, weighted smoothness nbrhood
Global inference using large, sparse smoothness neighborhoods
8
![Page 9: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/9.jpg)
Problem Formulation
UW-Madison Computer Graphics and Vision Group
2sm1sm21ph21 ,, DDDDDD
by minimizing:
compute disparity maps, and1D 2D
Given a stereo image pair, and , 1I 2I
9
![Page 10: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/10.jpg)
Energy Minimization Function
2sm1sm21ph21 ,, DDDDDD
UW-Madison Computer Graphics and Vision Group
Photo consistency term[Kolmogorov & Zabih ECCV ‘02]
21ph , DD
Smoothness (regularization) terms
2sm1sm DD
10
![Page 11: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/11.jpg)
UW-Madison Computer Graphics and Vision Group
Spatial Smoothness Terms
21ph21 ,, DDDD 2sm1sm DD
• 2nd-order smoothness priors
Previous global methods:• 1st-order smoothness priors
Another approach:• Kernel density estimation
Large neighborhood 11
![Page 12: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/12.jpg)
Large neighborhood UW-Madison Computer Graphics and Vision Group
Kernel Density Estimation
dg
dd
d
qp
Nq p
p,qw
g(x)
x
p||
1 qqgcx
cc
c
xx
xNpp g
qp
Weight based on proximity, color between p,q [Yoon & Kweon CVPR`05]
21ph21 ,, DDDD 2sm1sm DD
12
Kernel function for disparity
![Page 13: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/13.jpg)
Large neighborhood UW-Madison Computer Graphics and Vision Group
Upperbound Approximation
Nq p
p,qw
qp
21ph21 ,, DDDD 2sm1sm DD
dg
dd
d
qp
p||
1 qqgcx
cc
c
xx
xNp
g(x)
x
p g
Weight based on proximity, color between p,q [Yoon & Kweon CVPR`05]
Ip
Dsm
13
Kernel function for disparity
min( λ |dp dq|, τ)
Truncated linear difference function
![Page 14: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/14.jpg)
UW-Madison Computer Graphics and Vision Group
Optimization Challenge
21ph21 ,, DDDD 2sm1sm DDsm is dense, expensive to minimize
p,qw
Nq pIp
Dsm
min( λ |dp dq|, τ)
Solution: use a sparse approximation
Large neighborhood 14
![Page 15: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/15.jpg)
UW-Madison Computer Graphics and Vision Group
Sparse Graph Approximation
p,qw
Nq pIp
Dsm
min( λ |dp dq|, τ)
sm is dense, expensive to minimize
Solution: use a sparse graph approximation
DenseGraph
SparseGraph
15
![Page 16: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/16.jpg)
UW-Madison Computer Graphics and Vision Group
Sparse Graph Approximation
DenseGraph
SparseGraph
1st Spanning Tree
ResidualGraph
2st Spanning Tree Nth Spanning Tree…
16
![Page 17: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/17.jpg)
UW-Madison Computer Graphics and Vision Group
Dense smoothness neighborhood
Minimum spanning tree
algorithm
Sparse smoothness neighborhood
Graph Edges On Real Images
17
![Page 18: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/18.jpg)
UW-Madison Computer Graphics and Vision Group
0% pixels connected 82% pixels connected 95% pixels connected 98% pixels connected 100% pixels connected
Connection to Image Segmentation
Kruskal’s minimum spanning tree algorithm
C. Zahn. Graph-theoretic methods for detecting and describinggestalt clusters. IEEE Trans. on Computing, 1971.
18
![Page 19: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/19.jpg)
Results on Stereo Images
UW-Madison Computer Graphics and Vision Group
Left input image
Ground truth
Multi-view graph cuts result[Kolmogorov & Zabih, ‘02]
4.82% bad pixels
Our result
3.41% bad pixels 25.06% bad pixels
Second-order smoothness[Woodford et al. ’08]
19
![Page 20: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/20.jpg)
UW-Madison Computer Graphics and Vision Group
2.48% bad pixels 4.21% bad pixels
Klaus et al. ‘06 results Multi-view graph cuts[Kolmogrov & Zabih ‘02]
(tailored parameters)
Left input image Ground truth depth
Results on Stereo Images
20
3.29% bad pixels
Our results(tailored parameters)
![Page 21: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/21.jpg)
Results on Videos
UW-Madison Computer Graphics and Vision Group21
![Page 22: Stereo Matching with Nonparametric Smoothness Priors in …pages.cs.wisc.edu/.../cvpr2009/SmithCVPR09_presentation.pdf · 2009-07-11 · Stereo Matching with Nonparametric Smoothness](https://reader033.vdocuments.net/reader033/viewer/2022050103/5f41fa4f32b0d31ee07cf9bb/html5/thumbnails/22.jpg)
Future Work
UW-Madison Computer Graphics and Vision Group22
• Automatic parameter estimation (scale in kernel function)
• generalizes to any feature vector (not just x,y,r,g,b) explore other feature vectors
p,qw
• Better handle View-dependent brightness inconsistencies