![Page 1: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/1.jpg)
Multi-view stereo
CS5670: Computer VisionNoah Snavely / Zhengqi Li
![Page 2: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/2.jpg)
Recommended Reading
Szeliski Chapter 11.6
Multi-View Stereo: A Tutorial
Furukawa and Hernandez, 2015
http://www.cse.wustl.edu/~furukawa/papers/fnt_mvs.pdf
![Page 3: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/3.jpg)
Multi-view Stereo
Stereo Multi-view stereo
What is stereo vision?
Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape
![Page 4: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/4.jpg)
Multi-view Stereo
CMU’s 3D Room
Point Grey’s Bumblebee XB3
Point Grey’s ProFusion 25
![Page 5: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/5.jpg)
Multi-view Stereo
Figures by Carlos Hernandez
Input: calibrated images from several viewpoints
Output: 3D object model
![Page 6: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/6.jpg)
What is stereo vision?• Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape.
• “Images of the same object or scene”
• Arbitrary number of images (from two to thousands)
• Arbitrary camera positions (camera network or video sequence)
• Calibration may be initially unknown
• “Representation of 3D shape Representation of 3D shape ”
• Depth maps
• Meshes
• Point clouds
• Patch clouds
• Volumetric models
• Layered models
![Page 7: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/7.jpg)
Fua Narayanan, Rander, KanadeSeitz, Dyer
1995 1997 1998
Faugeras, Keriven
1998
Hernandez, Schmitt Pons, Keriven, Faugeras Furukawa, Ponce
2004 2005 2006
Goesele et al.
2007
Furukawa et al., 2010
![Page 8: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/8.jpg)
https://www.youtube.com/watch?v=ofHFOr2nRxU
https://www.youtube.com/watch?v=NdeD4cjLI0c&t=64s
![Page 9: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/9.jpg)
Applications
![Page 10: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/10.jpg)
![Page 11: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/11.jpg)
![Page 12: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/12.jpg)
![Page 13: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/13.jpg)
![Page 14: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/14.jpg)
![Page 15: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/15.jpg)
![Page 16: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/16.jpg)
https://code.facebook.com/posts/1755691291326688/introducing-facebook-surround-360-an-open-high-quality-3d-360-video-capture-system?hc_location=ufi
![Page 17: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/17.jpg)
Multi-view stereo: Basic idea
Source: Y. Furukawa
![Page 18: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/18.jpg)
Multi-view stereo: Basic idea
Source: Y. Furukawa
![Page 19: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/19.jpg)
Multi-view stereo: Basic idea
Source: Y. Furukawa
![Page 20: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/20.jpg)
Multi-view stereo: Basic idea
Source: Y. Furukawa
![Page 21: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/21.jpg)
![Page 22: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/22.jpg)
![Page 23: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/23.jpg)
There are many variations on this problem.
![Page 24: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/24.jpg)
Why MVS?
• Different points on the object’s surface will be more clearly visible in some subset of cameras
–Could have high-res closeups of some regions
–Some surfaces are foreshortened from certain views
![Page 25: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/25.jpg)
1 2
3
p
Cameras 2 and 3 can more clearly see point p.
![Page 26: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/26.jpg)
1 2
3
q
Cameras 1 and 2 can more clearly see point q.
![Page 27: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/27.jpg)
Why MVS?
• Different points on the object’s surface will be more clearly visible in some subset of cameras
– Could have high res close-ups of some regions
– Some surfaces are foreshortened from certain views
• Some points may be occluded entirely in certain views
![Page 28: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/28.jpg)
Camera 5 can’t see point r.
1
2 3 4
5
r
![Page 29: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/29.jpg)
Camera 1 can’t see point s.
1
2 3 4
5
s
![Page 30: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/30.jpg)
Why MVS?
• Different points on the object’s surface will be more clearly visible in some subset of cameras
– Could have high res closeups of some regions
– Some surfaces are foreshortened from certain views
• Some points may be occluded entirely in certain views
• More measurements per point can reduce error
![Page 31: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/31.jpg)
1 2
![Page 32: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/32.jpg)
1 2
![Page 33: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/33.jpg)
Estimated points contain some error.
1 2
![Page 34: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/34.jpg)
Estimated points contain some error.
1 2
![Page 35: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/35.jpg)
Estimated points contain some error.3
4
![Page 36: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/36.jpg)
Additional views reduce error.
1 2
3
4
![Page 37: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/37.jpg)
Depth maps reconstruction • Multiple-baseline stereo – Rectification of several cameras onto common plane – Problems with wide baselines and distortions after rectification
• Plane sweep stereo – Choose a reference view – Sweep family of planes at different depths with respect to the reference camera
http://www.uio.no/studier/emner/matnat/its/UNIK4690/v16/forelesninger/lecture_8_3_multiple_view_stereo.pdf
![Page 38: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/38.jpg)
width of
a pixel
Choosing the stereo baseline
What’s the optimal baseline?
– Too small: large depth error
– Too large: difficult search problem
Large Baseline Small Baseline
all of these
points project
to the same
pair of pixels
![Page 39: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/39.jpg)
The Effect of Baseline on Depth Estimation• Pick a reference image, and slide the corresponding
window along the corresponding epipolar lines of all
other images, using inverse depth relative to the first
image as the search parameter
![Page 40: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/40.jpg)
Multiple-baseline stereo
z
width of
a pixel
width of
a pixel
z
pixel matching score
M. Okutomi and T. Kanade, “A Multiple-Baseline Stereo System,” IEEE Trans. on
Pattern Analysis and Machine Intelligence, 15(4):353-363 (1993).
• For larger baselines, must search larger
area in second image
• For short baselines, estimated depth will be less
precise due to narrow triangulation
![Page 41: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/41.jpg)
Multiple-baseline stereo
![Page 42: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/42.jpg)
I1 I2 I10
Multiple-baseline stereo results
M. Okutomi and T. Kanade, “A Multiple-Baseline Stereo System,” IEEE Trans. on Pattern Analysis and Machine Intelligence, 15(4):353-363 (1993).
![Page 43: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/43.jpg)
Multibaseline Stereo
Basic Approach
– Choose a reference view
– Use your favorite stereo algorithm BUT• replace two-view SSD with SSSD over all baselines
• SSSD: the SSD values are computed first for each pair of stereo images, and then add all together from multiple stereo pairs.
Limitations
– Only gives a depth map (not an “object model”)
– Won’t work for widely distributed views.
![Page 44: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/44.jpg)
Some Solutions
• Match only nearby photos [Narayanan 98]
• Use NCC instead of SSD,
Ignore NCC values > threshold
[Hernandez & Schmitt 03]
Problem: visibility
![Page 45: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/45.jpg)
Photo-consistency measures (matching score)
http://carlos-hernandez.org/papers/fnt_mvs_2015.pdf
![Page 46: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/46.jpg)
Popular matching scores• SSD (Sum Squared Distance)
• SAD (Sum of Absolute Difference)
• ZNCC (Zero-mean Normalized Cross Correlation)
– where
– what advantages might NCC have?
![Page 47: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/47.jpg)
Summary
http://carlos-hernandez.org/papers/fnt_mvs_2015.pdf
![Page 48: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/48.jpg)
Plane sweep stereo
D. Gallup, J.-M. Frahm, P. Mordohai, Q. Yang and M. Pollefeys, Real-Time Plane-Sweeping Stereo with Multiple Sweeping
Directions, CVPR 2007
![Page 49: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/49.jpg)
Plane sweep stereo
![Page 50: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/50.jpg)
Plane sweep through oriented planes
![Page 51: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/51.jpg)
Plane Sweep: Enhanced Robustness through oriented planes
https://demuc.de/tutorials/cvpr2017/dense-modeling.pdf
![Page 52: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/52.jpg)
Streetside reconstructions by plane sweeping stereo
http://carlos-hernandez.org/papers/fnt_mvs_2015.pdf
![Page 53: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/53.jpg)
Single depth map often isn’t
enough
![Page 54: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/54.jpg)
Really want full coverage
![Page 55: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/55.jpg)
Idea: Combine many depth maps
Many depth maps, each with error. How can we fuse these?
![Page 56: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/56.jpg)
Merging depth maps
• Given a group of images, choose each
one as reference and compute a depth
map w.r.t. that view using a multi-baseline
approach
• Merge multiple depth maps to a
volume or a mesh (see, e.g., Curless
and Levoy 96)
Map 1 Map 2 Merged
![Page 57: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/57.jpg)
Volumetric fusion
A common world-space coordinate system.
![Page 58: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/58.jpg)
Volumetric fusion
A common world-space coordinate system.
![Page 59: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/59.jpg)
Volumetric stereo
• In plane sweep stereo, the sampling of the scene depends
on the reference view
• We can use a voxel volume to get a view independent
representation
Discretized
Scene Volume
Input Images
(Calibrated)
Goal:GOAL: Assign RGB values to voxels in V
photo-consistent with images
![Page 60: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/60.jpg)
![Page 61: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/61.jpg)
![Page 62: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/62.jpg)
Iso surface
![Page 63: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/63.jpg)
Newcombe, et. al. ISMAR 2011.
![Page 64: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/64.jpg)
https://www.youtube.com/watch?v=quGhaggn3cQ
![Page 65: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/65.jpg)
Questions?
![Page 66: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/66.jpg)
Questions?
![Page 67: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/67.jpg)
Multi-view stereo from Internet Collections
[Goesele, Snavely, Curless, Hoppe, Seitz, ICCV 2007]
![Page 68: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/68.jpg)
Stereo from community photo collections• Need structure from motion to recover unknown
camera parameters
• Need view selection to find good groups of images on which to run dense stereo
![Page 69: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/69.jpg)
• appearance variation
• resolution
• massive collections
Challenges
![Page 70: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/70.jpg)
206 Flickr images taken by 92 photographers
Law of Large Image Collections
![Page 71: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/71.jpg)
Local view selection• Automatically select neighboring views for each point in the image
• Desiderata: good matches AND good baselines
4 best neighboring views
reference view
![Page 72: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/72.jpg)
Local view selection• Automatically select neighboring views for each point in the image
• Desiderata: good matches AND good baselines
4 best neighboring views
reference view
![Page 73: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/73.jpg)
Local view selection• Automatically select neighboring views for each point in the image
• Desiderata: good matches AND good baselines
4 best neighboring views
reference view
![Page 74: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/74.jpg)
St. Peter
151 images
50 photographers
Trevi Fountain
106 images
51 photographers
Mt. Rushmore
160 images
60 photographers
Results
![Page 75: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/75.jpg)
Notre Dame de Paris
653 images
313 photographers
![Page 76: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/76.jpg)
![Page 77: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/77.jpg)
![Page 78: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/78.jpg)
129 Flickr images taken by 98 photographers
![Page 79: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/79.jpg)
merged model of Venus de Milo
![Page 80: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/80.jpg)
56 Flickr images taken by 8 photographers
![Page 81: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/81.jpg)
merged model of Pisa Cathedral
![Page 82: CS5670: Computer Vision · What is stereo vision? • Generic problem formulation: given several images of the same object or scene, compute a representation of its 3D shape. •](https://reader033.vdocuments.net/reader033/viewer/2022052814/609dc2e1211f271c7b61ea35/html5/thumbnails/82.jpg)
Accuracy compared to laser scanned model:
90% of points within 0.25% of ground truth