mulitiview photometric stereocs6320/cv_files/stereomatching.pdf · matching costs c(x,y,d) 9 find...
TRANSCRIPT
![Page 1: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/1.jpg)
1
Stereo MatchingSrikumar Ramalingam
School of Computing
University of Utah
![Page 2: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/2.jpg)
2Reference
• Daniel Scharstein and Richard Szeliski, A Taxonomy and Evaluation of Dense Two-Frame
Stereo Correspondence Algorithms, IJCV, 2002.
http://vision.middlebury.edu/stereo/taxonomy-IJCV.pdf
![Page 3: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/3.jpg)
3Dense Correspondence in Computer Vision
Multiview stereoBinocular Stereo
Slide Courtesy: Sudipta Sinha
![Page 4: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/4.jpg)
4Stereo Matching
Left (reference) Right
▪ Dense pixel correspondence in rectified pairs
▪ Disparity Map: 𝐷 𝑥, 𝑦 (treated as +ve)
𝑥′= 𝑥 − 𝐷 𝑥, 𝑦 , 𝑦′= 𝑦
▪ Depth Map: 𝑍 𝑥, 𝑦 =𝑏𝑎𝑠𝑒𝑙𝑖𝑛𝑒 ∗ 𝑓𝑜𝑐𝑎𝑙 𝑙𝑒𝑛𝑔𝑡ℎ
𝐷(𝑥,𝑦)
Depth Map
Left Disparity Map
Slide Courtesy: Sudipta Sinha
![Page 5: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/5.jpg)
5Binocular Stereo Matching
𝑥
𝑦𝑧
𝑝′
Left
Right
Slide Courtesy: Sudipta Sinha
![Page 6: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/6.jpg)
6Binocular Stereo Matching
𝑥
𝑦𝑧
Left
Right
Slide Courtesy: Sudipta Sinha
![Page 7: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/7.jpg)
7Binocular stereo matching (Top View)
Left camera center Right camera center
P(X,Y,Z)
f fB
(x,y) (x’,y’)D(x,y) = x-x’
Z=bf/DisparityX=Zx/fY=Zy/f
![Page 8: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/8.jpg)
8Binocular stereo matching (Top View)
Left camera center Right camera center
P(X,Y,Z)
f fB
(x,y) (x’,y’)D(x,y) = x-x’
Z=bf/DisparityX=Zx/fY=Zy/f
![Page 9: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/9.jpg)
9Matching costs C(x,y,d)
▪ Find pairs of pixels (or local patches) with similar appearance
▪ Minimize matching cost (maximize photo-consistency)
▪ Patch-based (parametric vs non-parametric)
- Sum of Absolute Difference (SAD),
- Sum of Squared Difference (SSD),
- Normalized Cross Correlation (ZNCC)
▪ Descriptor-based
- (hand-crafted features) SIFT, DAISY, …
- (learnt features) Deep learning (revisit later)
Slide Courtesy: Sudipta Sinha
![Page 10: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/10.jpg)
10Local Optimization
▪ Minimize matching cost at each pixel in the left image
independently
▪ Winner-take-all (WTA)
Slide Courtesy: Sudipta Sinha
![Page 11: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/11.jpg)
11Local evidence not enough …
▪ Photometric Variations
▪ Reflections
▪ Transparent surfaces
▪ Texture-less Areas
▪ Non-Lambertian Surfaces
▪ Repetitive patterns
▪ Complex Occlusions
(Image Source: Lectures on stereo matching, Christian Unger and Nassir Navab, TU Munchen) http://campar.in.tum.de/twiki/pub/Chair/TeachingWs09Cv2/3D_CV2_WS_2009_Stereo.pdf
![Page 12: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/12.jpg)
12Local evidence not enough …
▪ Photometric Variations
▪ Reflections
▪ Transparent surfaces
▪ Texture-less Areas
▪ Non-Lambertian Surfaces
▪ Repetitive patterns
▪ Complex Occlusions
(Image Source: Lectures on stereo matching, Christian Unger and Nassir Navab, TU Munchen) http://campar.in.tum.de/twiki/pub/Chair/TeachingWs09Cv2/3D_CV2_WS_2009_Stereo.pdf
![Page 13: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/13.jpg)
13Local evidence not enough …
▪ Photometric Variations
▪ Reflections
▪ Transparent surfaces
▪ Texture-less Areas
▪ Non-Lambertian Surfaces
▪ Repetitive patterns
▪ Complex Occlusions
(Image Source: Lectures on stereo matching, Christian Unger and Nassir Navab, TU Munchen) http://campar.in.tum.de/twiki/pub/Chair/TeachingWs09Cv2/3D_CV2_WS_2009_Stereo.pdf
![Page 14: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/14.jpg)
14Global Optimization
▪ Solve for all disparities simultaneously …
▪ Solve a pixel labeling problem
▪ Labels are discrete (ordered), 𝑑 ∈ 𝐿𝐷𝐿D = [𝑑𝑚𝑖𝑛, 𝑑𝑚𝑎𝑥]
▪ Incorporate regularization into objective
▪ Data term encodes matching costs
▪ Smoothness term encodes priors
▪ Encourage adjacent pixels to take similar disparities
Slide Courtesy: Sudipta Sinha
![Page 15: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/15.jpg)
15Disparity Computation
Let the disparity map be given by d, where d(x,y) denotes the disparity for a pixel in the left image at location (x,y).Let all the edges in the image graph be given by 𝐸 by treating the image as a 4-connected or 8-connected graph:
Data term: 𝜆 is the constant term that can be manually fixed or learned from data.
![Page 16: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/16.jpg)
16Different smoothness terms
𝐸𝑠𝑚𝑜𝑜𝑡ℎ(𝑑) =
𝑖,𝑗 ∈𝐸
𝑓(𝑑 𝑥𝑖 , 𝑦𝑖 , 𝑑(𝑥𝑗 , 𝑦𝑗))
𝑑𝑖𝑓𝑓 = 𝑑 𝑥𝑖 , 𝑦𝑖 − 𝑑 𝑥𝑗 , 𝑦𝑗
Potts: 𝑓 = 1 𝑖𝑓 𝑑𝑖𝑓𝑓 > 0 𝑒𝑙𝑠𝑒 𝑓 = 0 (Robust)
Linear: 𝑓 = 𝑑𝑖𝑓𝑓 (Not Robust)
Truncated model: 𝑓 = 𝑑𝑖𝑓𝑓 𝑖𝑓 𝑑𝑖𝑓𝑓 < 𝑇 𝑒𝑙𝑠𝑒 𝑓 = 𝑇(Robust)
𝑑 𝑥𝑖 , 𝑦𝑖 − 𝑑 𝑥𝑗 , 𝑦𝑗
𝑓
𝑇
𝑑 𝑥𝑖 , 𝑦𝑖 − 𝑑 𝑥𝑗 , 𝑦𝑗
𝑓
𝑑 𝑥𝑖 , 𝑦𝑖 − 𝑑 𝑥𝑗 , 𝑦𝑗
𝑓
![Page 17: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/17.jpg)
17Stereo Image with Groundtruth
Left image from orig. stereo pair Groundtruth disparity
Texture-less regions Depth discontinuities (white) and Occlusion (Black)
![Page 18: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/18.jpg)
18BP Results
![Page 19: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/19.jpg)
19Sample Problem (not the same as real setup)
![Page 20: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/20.jpg)
20Solution
Solution: Kyle Pierson
![Page 21: Mulitiview Photometric Stereocs6320/cv_files/StereoMatching.pdf · Matching costs C(x,y,d) 9 Find pairs of pixels (or local patches) with similar appearance Minimize matching cost](https://reader033.vdocuments.net/reader033/viewer/2022043010/5f9fd57c4f9d6b03592c160d/html5/thumbnails/21.jpg)
21Solution