image stitching
DESCRIPTION
Image Stitching. Shangliang Jiang Kate Harrison. What is image stitching?. What is image stitching?. Introduction. Are you getting the whole picture? Compact Camera FOV = 50 x 35°. Introduction. Are you getting the whole picture? Compact Camera FOV = 50 x 35° - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/1.jpg)
Image Stitching
Shangliang JiangKate Harrison
![Page 2: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/2.jpg)
What is image stitching?
![Page 3: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/3.jpg)
What is image stitching?
![Page 4: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/4.jpg)
Introduction• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°
![Page 5: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/5.jpg)
Introduction• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°
![Page 6: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/6.jpg)
Introduction• Are you getting the whole picture?
– Compact Camera FOV = 50 x 35°– Human FOV = 200 x 135°– Panoramic Mosaic = 360 x 180°
![Page 7: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/7.jpg)
Recognizing Panoramas• 1D Rotations ()
– Ordering matching images
![Page 8: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/8.jpg)
Recognizing Panoramas• 1D Rotations ()
– Ordering matching images
![Page 9: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/9.jpg)
Recognizing Panoramas• 1D Rotations ()
– Ordering matching images
![Page 10: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/10.jpg)
Recognizing Panoramas
• 2D Rotations (, )– Ordering matching images
• 1D Rotations ()– Ordering matching images
![Page 11: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/11.jpg)
Recognizing Panoramas• 1D Rotations ()
– Ordering matching images
• 2D Rotations (, )– Ordering matching images
![Page 12: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/12.jpg)
Recognizing Panoramas• 1D Rotations ()
– Ordering matching images
• 2D Rotations (, )– Ordering matching images
![Page 13: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/13.jpg)
Recognizing Panoramas
![Page 14: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/14.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 15: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/15.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 16: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/16.jpg)
Overview• Feature Matching
– SIFT Features– Nearest Neighbor Matching
• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 17: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/17.jpg)
SIFT Features• SIFT features are…
– Geometrically invariant to similarity transforms,• some robustness to affine change
– Photometrically invariant to affine changes in intensity
![Page 18: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/18.jpg)
Overview• Feature Matching
– SIFT Features– Nearest Neighbor Matching
• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 19: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/19.jpg)
Nearest Neighbor Matching• Find k nearest neighbors for each feature
– k number of overlapping images (we use k = 4)• Use k-d tree
– k-d tree recursively bi-partitions data at mean in the dimension of maximum variance
– Approximate nearest neighbors found in O(nlogn)
![Page 20: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/20.jpg)
Overview• Feature Matching
– SIFT Features– Nearest Neighbor Matching
• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 21: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/21.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 22: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/22.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 23: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/23.jpg)
Overview• Feature Matching• Image Matching
– Random Sample Consensus (RANSAC) for Homography
– Probabilistic model for verification• Bundle Adjustment• Image Compositing• Conclusions
![Page 24: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/24.jpg)
Overview• Feature Matching• Image Matching
– Random Sample Consensus (RANSAC) for Homography
– Probabilistic model for verification• Bundle Adjustment• Image Compositing• Conclusions
![Page 25: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/25.jpg)
RANSAC for Homography
![Page 26: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/26.jpg)
RANSAC for Homography
![Page 27: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/27.jpg)
RANSAC for Homography
![Page 28: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/28.jpg)
RANSAC for Homography
![Page 29: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/29.jpg)
RANSAC for Homography
![Page 30: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/30.jpg)
Overview• Feature Matching• Image Matching
– Random Sample Consensus (RANSAC) for Homography
– Probabilistic model for verification• Bundle Adjustment• Image Compositing• Conclusions
![Page 31: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/31.jpg)
Probabilistic model for verification
![Page 32: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/32.jpg)
Finding the panoramas
![Page 33: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/33.jpg)
Finding the panoramas
![Page 34: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/34.jpg)
Finding the panoramas
![Page 35: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/35.jpg)
Overview• Feature Matching• Image Matching
– RANSAC for Homography– Probabilistic model for verification
• Bundle Adjustment• Image Compositing• Conclusions
![Page 36: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/36.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 37: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/37.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 38: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/38.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment
– Error function• Image Compositing• Conclusions
![Page 39: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/39.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment
– Error function• Image Compositing• Conclusions
![Page 40: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/40.jpg)
Bundle Adjustment• New images initialised with rotation, focal
length of best matching image
![Page 41: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/41.jpg)
Bundle Adjustment• New images initialised with rotation, focal
length of best matching image
![Page 42: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/42.jpg)
Error function• Sum of squared projection errors
– n = #images– I(i) = set of image matches to image i– F(i, j) = set of feature matches between images i,j– rij
k = residual of kth feature match between images i,j
• Robust error function
![Page 43: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/43.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment
– Error function• Image Compositing• Conclusions
![Page 44: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/44.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 45: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/45.jpg)
![Page 46: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/46.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 47: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/47.jpg)
Blending
![Page 48: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/48.jpg)
Gain compensation
![Page 49: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/49.jpg)
How do we blend?
Linear blending Multi-band blending
![Page 50: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/50.jpg)
Multi-band Blending• Burt & Adelson 1983
– Blend frequency bands over range
![Page 51: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/51.jpg)
Low frequency ( > 2 pixels)
High frequency ( < 2 pixels)
2-band Blending
![Page 52: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/52.jpg)
3-band blendingBand 1: high frequencies
![Page 53: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/53.jpg)
3-band blendingBand 2: mid-range frequencies
![Page 54: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/54.jpg)
3-band blendingBand 3: low frequencies
![Page 55: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/55.jpg)
Panorama straightening
Heuristic: people tend to shoot pictures in a certain way
![Page 56: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/56.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 57: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/57.jpg)
Overview• Feature Matching• Image Matching• Bundle Adjustment• Image Compositing• Conclusions
![Page 58: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/58.jpg)
Conclusion
![Page 59: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/59.jpg)
Algorithm
![Page 60: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/60.jpg)
AutoStitch program
AutoStitch.net
![Page 61: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/61.jpg)
Open questions• Advanced camera modeling
– radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash …
• Full 3D case – recognizing 3D objects/scenes in unordered datasets
![Page 62: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/62.jpg)
Credits• Automatic Panoramic Image Stitching Using
Invariant Features, 2007– Matthew Brown and David G. Lowe (Uni. of British
Columbia)• Recognising Panoramas, 2003
– Matthew Brown and David G. Lowe (Uni. of British Columbia)
– 2003– Thanks for the slides!
• Image Alignment and Stitching: A Tutorial, 2006– Richard Szeliski (Microsoft)
![Page 63: Image Stitching](https://reader035.vdocuments.net/reader035/viewer/2022062400/5681595d550346895dc69afd/html5/thumbnails/63.jpg)
Questions?