image stitching -...
TRANSCRIPT
![Page 1: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/1.jpg)
Image Stitching
Ali Farhadi CSE 576
Several slides from Rick Szeliski, Steve Seitz, Derek Hoiem, and Ira Kemelmacher
![Page 2: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/2.jpg)
• Combine two or more overlapping images to make one larger image
Add example
Slide credit: Vaibhav Vaish
![Page 3: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/3.jpg)
How to do it?
• Basic Procedure 1. Take a sequence of images from the same
position 1. Rotate the camera about its optical center
2. Compute transformation between second image and first
3. Shift the second image to overlap with the first
4. Blend the two together to create a mosaic 5. If there are more images, repeat
![Page 4: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/4.jpg)
1. Take a sequence of images from the same position
• Rotate the camera about its optical center
![Page 5: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/5.jpg)
2. Compute transformation between images
• Extract interest points • Find Matches • Compute transformation?
![Page 6: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/6.jpg)
3. Shift the images to overlap
![Page 7: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/7.jpg)
4. Blend the two together to create a mosaic
![Page 8: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/8.jpg)
5. Repeat for all images
![Page 9: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/9.jpg)
How to do it?
• Basic Procedure 1. Take a sequence of images from the same
position 1. Rotate the camera about its optical center
2. Compute transformation between second image and first
3. Shift the second image to overlap with the first
4. Blend the two together to create a mosaic 5. If there are more images, repeat
✓
![Page 10: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/10.jpg)
Compute Transformations
• Extract interest points • Find good matches • Compute transformation
✓
Let’s assume we are given a set of good matching interest points
✓
![Page 11: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/11.jpg)
mosaic PP
Image reprojection
• The mosaic has a natural interpretation in 3D – The images are reprojected onto a common plane – The mosaic is formed on this plane
![Page 12: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/12.jpg)
Example
Camera Center
![Page 13: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/13.jpg)
Image reprojection
• Observation – Rather than thinking of this as a 3D reprojection,
think of it as a 2D image warp from one image to another
![Page 14: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/14.jpg)
Motion models
• What happens when we take two images with a camera and try to align them?
• translation? • rotation? • scale? • affine? • Perspective?
![Page 15: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/15.jpg)
Recall: Projective transformations
• (aka homographies)
![Page 16: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/16.jpg)
Parametric (global) warping
• Examples of parametric warps:
translation rotation aspect
affineperspective
![Page 17: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/17.jpg)
2D coordinate transformations
• translation: x’ = x + t x = (x,y) • rotation: x’ = R x + t • similarity: x’ = s R x + t • affine: x’ = A x + t • perspective: x’ ≅ H x x = (x,y,1)
(x is a homogeneous coordinate)
![Page 18: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/18.jpg)
Image Warping
• Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?
f(x) g(x’)x x’
h(x)
![Page 19: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/19.jpg)
Forward Warping
• Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?
![Page 20: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/20.jpg)
Forward Warping
• Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?• Answer: add “contribution” to several pixels,
normalize later (splatting)
![Page 21: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/21.jpg)
Image StitchingRichard Szeliski 21
Inverse Warping
• Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)
f(x) g(x’)x x’
h-1(x)
• What if pixel comes from “between” two pixels?
![Page 22: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/22.jpg)
Image StitchingRichard Szeliski 22
Inverse Warping
• Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)
• What if pixel comes from “between” two pixels?• Answer: resample color value from interpolated source image
f(x) g(x’)x x’
h-1(x)
![Page 23: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/23.jpg)
Interpolation
• Possible interpolation filters: – nearest neighbor – bilinear – bicubic (interpolating)
![Page 24: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/24.jpg)
Motion models
Translation
2 unknowns
Affine
6 unknowns
Perspective
8 unknowns
![Page 25: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/25.jpg)
Finding the transformation
• Translation = 2 degrees of freedom • Similarity = 4 degrees of freedom • Affine = 6 degrees of freedom • Homography = 8 degrees of freedom
• How many corresponding points do we need to solve?
![Page 26: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/26.jpg)
Simple case: translations
How do we solve for ?
![Page 27: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/27.jpg)
Mean displacement =
Simple case: translations
Displacement of match i =
![Page 28: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/28.jpg)
Simple case: translations
• System of linear equations – What are the knowns? Unknowns? – How many unknowns? How many equations (per match)?
![Page 29: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/29.jpg)
Simple case: translations
• Problem: more equations than unknowns – “Overdetermined” system of equations – We will find the least squares solution
![Page 30: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/30.jpg)
Least squares formulation
• For each point
• we define the residuals as
![Page 31: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/31.jpg)
Least squares formulation
• Goal: minimize sum of squared residuals
• “Least squares” solution
• For translations, is equal to mean displacement
![Page 32: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/32.jpg)
Least squares
• Find t that minimizes
• To solve, form the normal equations
![Page 33: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/33.jpg)
Solving for translations
• Using least squares
2n x 2 2 x 1 2n x 1
![Page 34: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/34.jpg)
Affine transformations
• How many unknowns? • How many equations per match? • How many matches do we need?
![Page 35: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/35.jpg)
Affine transformations
• Residuals:
• Cost function:
![Page 36: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/36.jpg)
Affine transformations
• Matrix form
2n x 6 6 x 1 2n x 1
![Page 37: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/37.jpg)
Solving for homographies
![Page 38: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/38.jpg)
Solving for homographies
![Page 39: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/39.jpg)
Direct Linear Transforms
Defines a least squares problem:
• Since is only defined up to scale, solve for unit vector • Solution: = eigenvector of with smallest
eigenvalue • Works with 4 or more points
2n × 9 9 2n
![Page 40: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/40.jpg)
Image StitchingRichard Szeliski 40
Matching features
What do we do about the “bad” matches?
![Page 41: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/41.jpg)
Image StitchingRichard Szeliski 41
RAndom SAmple Consensus
Select one match, count inliers
![Page 42: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/42.jpg)
Image StitchingRichard Szeliski 42
RAndom SAmple Consensus
Select one match, count inliers
![Page 43: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/43.jpg)
Image StitchingRichard Szeliski 43
Least squares fit
Find “average” translation vector
![Page 44: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/44.jpg)
![Page 45: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/45.jpg)
Structure from Motion
RANSAC for estimating homography
• RANSAC loop: 1. Select four feature pairs (at random) 2. Compute homography H (exact) 3. Compute inliers where ||pi’, H pi|| < ε • Keep largest set of inliers • Re-compute least-squares H estimate
using all of the inliers
CSE 576, Spring 2008 45
![Page 46: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/46.jpg)
46
Simple example: fit a line
• Rather than homography H (8 numbers) fit y=ax+b (2 numbers a, b) to 2D pairs
47
![Page 47: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/47.jpg)
47
Simple example: fit a line
• Pick 2 points • Fit line • Count inliers
48
3 inliers
![Page 48: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/48.jpg)
48
Simple example: fit a line
• Pick 2 points • Fit line • Count inliers
49
4 inliers
![Page 49: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/49.jpg)
49
Simple example: fit a line
• Pick 2 points • Fit line • Count inliers
50
9 inliers
![Page 50: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/50.jpg)
50
Simple example: fit a line
• Pick 2 points • Fit line • Count inliers
51
8 inliers
![Page 51: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/51.jpg)
51
Simple example: fit a line
• Use biggest set of inliers • Do least-square fit
52
![Page 52: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/52.jpg)
RANSAC
Red: rejected by 2nd nearest neighbor criterion Blue: Ransac outliers Yellow: inliers
![Page 53: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/53.jpg)
Computing homography
• Assume we have four matched points: How do we compute homography H?
Normalized DLT 1. Normalize coordinates for each image
a) Translate for zero mean b) Scale so that average distance to origin is ~sqrt(2)
– This makes problem better behaved numerically
2. Compute using DLT in normalized coordinates 3. Unnormalize:
Txx =~ xTx !!=!~
THTH ~1−"=
ii Hxx =!
H~
![Page 54: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/54.jpg)
Computing homography• Assume we have matched points with outliers: How
do we compute homography H?
Automatic Homography Estimation with RANSAC 1. Choose number of samples N 2. Choose 4 random potential matches 3. Compute H using normalized DLT 4. Project points from x to x’ for each potentially
matching pair: 5. Count points with projected distance < t
– E.g., t = 3 pixels 6. Repeat steps 2-5 N times
– Choose H with most inliers
HZ Tutorial ‘99
ii Hxx =!
![Page 55: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/55.jpg)
Automatic Image Stitching
1. Compute interest points on each image
2. Find candidate matches
3. Estimate homography H using matched points and RANSAC with normalized DLT
4. Project each image onto the same surface and blend
![Page 56: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/56.jpg)
RANSAC for Homography
Initial Matched Points
![Page 57: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/57.jpg)
RANSAC for Homography
Final Matched Points
![Page 58: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/58.jpg)
RANSAC for Homography
![Page 59: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/59.jpg)
Image Blending
![Page 60: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/60.jpg)
Feathering
01
01
+
=
![Page 61: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/61.jpg)
Effect of window (ramp-width) size
0
1 left
right0
1
![Page 62: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/62.jpg)
Effect of window size
0
1
0
1
![Page 63: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/63.jpg)
Good window size
0
1
“Optimal” window: smooth but not ghosted • Doesn’t always work...
![Page 64: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/64.jpg)
Pyramid blending
Create a Laplacian pyramid, blend each level • Burt, P. J. and Adelson, E. H., A multiresolution spline with applications to image mosaics, ACM Transactions on
Graphics, 42(4), October 1983, 217-236.
![Page 65: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/65.jpg)
Gaussian Pyramid Laplacian Pyramid
The Laplacian Pyramid
0G
1G
2GnG
- =
0L
- = 1L
- = 2Lnn GL =
)expand( 1+−= iii GGL)expand( 1++= iii GLG
![Page 66: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/66.jpg)
Laplacian level
4
Laplacian level
2
Laplacian level
0
left pyramid right pyramid blended pyramid
![Page 67: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/67.jpg)
Laplacian image blend
1. Compute Laplacian pyramid 2. Compute Gaussian pyramid on weight
image 3. Blend Laplacians using Gaussian blurred
weights 4. Reconstruct the final image
![Page 68: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/68.jpg)
Multiband Blending with Laplacian Pyramid
0
1
0
1
0
1
Left pyramid Right pyramidblend
• At low frequencies, blend slowly • At high frequencies, blend quickly
![Page 69: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/69.jpg)
Multiband blending
1.Compute Laplacian pyramid of images and mask
2.Create blended image at each level of pyramid
3.Reconstruct complete image
Laplacian pyramids
![Page 70: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/70.jpg)
Blending comparison (IJCV 2007)
![Page 71: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/71.jpg)
Poisson Image Editing
• For more info: Perez et al, SIGGRAPH 2003 – http://research.microsoft.com/vision/cambridge/papers/perez_siggraph03.pdf
![Page 72: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/72.jpg)
Encoding blend weights: I(x,y) = (αR, αG, αB, α)
color at p =
Implement this in two steps: 1. accumulate: add up the (α premultiplied) RGBα values at each pixel
2. normalize: divide each pixel’s accumulated RGB by its α value
Alpha Blending
Optional: see Blinn (CGA, 1994) for details: http://ieeexplore.ieee.org/iel1/38/7531/00310740.pdf?isNumber=7531&prod=JNL&arnumber=310740&arSt=83&ared=87&arAuthor=Blinn%2C+J.F.
I1
I2
I3
p
![Page 73: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/73.jpg)
Choosing seams
Image 1
Image 2
x x
im1 im2
• Easy method – Assign each pixel to image with nearest
center
![Page 74: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/74.jpg)
Choosing seams
• Easy method – Assign each pixel to image with nearest center – Create a mask: – Smooth boundaries ( “feathering”): – Composite
Image 1
Image 2
x x
im1 im2
![Page 75: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/75.jpg)
Choosing seams
• Better method: dynamic program to find seam along well-matched regions
Illustration: http://en.wikipedia.org/wiki/File:Rochester_NY.jpg
![Page 76: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/76.jpg)
Gain compensation• Simple gain adjustment
– Compute average RGB intensity of each image in overlapping region
– Normalize intensities by ratio of averages
![Page 77: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/77.jpg)
Blending Comparison
![Page 78: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/78.jpg)
Recognizing Panoramas
Brown and Lowe 2003, 2007Some of following material from Brown and Lowe 2003 talk
![Page 79: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/79.jpg)
Recognizing Panoramas
Input: N images 1. Extract SIFT points, descriptors from all
images 2. Find K-nearest neighbors for each point
(K=4) 3. For each image
a) Select M candidate matching images by counting matched keypoints (m=6)
b) Solve homography Hij for each matched image
![Page 80: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/80.jpg)
Recognizing Panoramas
Input: N images 1. Extract SIFT points, descriptors from all
images 2. Find K-nearest neighbors for each point (K=4) 3. For each image
a) Select M candidate matching images by counting matched keypoints (m=6)
b) Solve homography Hij for each matched image c) Decide if match is valid (ni > 8 + 0.3 nf )
# inliers # keypoints in overlapping area
![Page 81: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/81.jpg)
Recognizing Panoramas (cont.)
(now we have matched pairs of images) 4. Find connected components
![Page 82: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/82.jpg)
Finding the panoramas
![Page 83: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/83.jpg)
Finding the panoramas
![Page 84: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/84.jpg)
Finding the panoramas
![Page 85: Image Stitching - courses.cs.washington.educourses.cs.washington.edu/courses/cse576/16sp/Slides/10_ImageStitching.pdfImage Stitching Ali Farhadi CSE 576 Several slides from Rick Szeliski,](https://reader034.vdocuments.net/reader034/viewer/2022050502/5f94646836427826c44b4c3e/html5/thumbnails/85.jpg)
Recognizing Panoramas (cont.)
(now we have matched pairs of images) 4. Find connected components 5. For each connected component
a) Solve for rotation and f b) Project to a surface (plane, cylinder, or
sphere) c) Render with multiband blending