prof. feng liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/lecture10.pdfcylindrical panorama 1....
TRANSCRIPT
![Page 1: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/1.jpg)
Prof. Feng Liu
Spring 2020
http://www.cs.pdx.edu/~fliu/courses/cs510/
04/30/2020
With slides by C. Dyer, Y. Chuang, R. Szeliski, S. Seitz, M. Brown and V. Hlavac
![Page 2: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/2.jpg)
Last Time
Panorama
◼ Feature detection
◼ Feature matching
◼ Homography estimation
2
![Page 3: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/3.jpg)
Today
Panorama
◼ Cylindrical panorama
◼ Blending
3
![Page 4: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/4.jpg)
Stitching Recipe
Align pairs of images
◼ Feature Detection
◼ Feature Matching
◼ Homography Estimation
Align all to a common surface
Adjust (Global) & Blend
4
![Page 5: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/5.jpg)
5
Invariant Local Features
Goal: Detect the same interest points
regardless of image changes due to
translation, rotation, scale, viewpoint
![Page 6: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/6.jpg)
After detecting points (and patches) in each image,
Next question: How to match them?
?
Feature Point Descriptors
![Page 7: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/7.jpg)
RANSAC for Homography
Credit: Y.Y. Chuang
![Page 8: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/8.jpg)
A case study: cylindrical panorama
What if you want a 360 field of view?
mosaic projection cylinder
Credit: Y.Y. Chuang
![Page 9: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/9.jpg)
Cylindrical panoramas
Steps◼ Reproject each image onto a cylinder
◼ Blend
◼ Output the resulting mosaic
Credit: Y.Y. Chuang
![Page 10: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/10.jpg)
Cylindrical panorama
1. Take pictures on a tripod (or handheld)
2. Warp to cylindrical coordinate
3. Compute pair-wise alignments
4. Fix up the end-to-end alignment
5. Blending
6. Crop the result and import into a viewer
It is required to do radial distortion
correction for better stitching results!
Credit: Y.Y. Chuang
![Page 11: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/11.jpg)
Taking pictures
Kaidan panoramic tripod head
Credit: Y.Y. Chuang
![Page 12: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/12.jpg)
Where should the synthetic camera be
The projection plan of some camera
Onto a cylinder
realcamera
syntheticcamera
Credit: Y.Y. Chuang
![Page 13: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/13.jpg)
Cylindrical projection
Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htmCredit: Y.Y. Chuang
![Page 14: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/14.jpg)
Cylindrical projection
Credit: Y.Y. Chuang
![Page 15: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/15.jpg)
Cylindrical projection
Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htmCredit: Y.Y. Chuang
![Page 16: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/16.jpg)
Cylindrical projection
unwrapped cylinder
xy
f
xθ
Credit: Y.Y. Chuang
![Page 17: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/17.jpg)
z
Cylindrical projection
unwrapped cylinder
xy
θ
x
y
f
Credit: Y.Y. Chuang
h
yx
![Page 18: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/18.jpg)
Cylindrical projection
unwrapped cylinder
xy
z
x
y
f
s defines size of the final image,
often convenient to set s = f
cylindrical image'x
'y],[ cc yx
Credit: Y.Y. Chuang
![Page 19: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/19.jpg)
f = 180 (pixels)
Cylindrical reprojection
f = 380f = 280Image 384x300
top-down view Focal length – the dirty secret…
Credit: Y.Y. Chuang
![Page 20: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/20.jpg)
A simple method for estimating f
Or, you can use other software, such as AutoStich,
to help.
d
f
w
p
Credit: Y.Y. Chuang
![Page 21: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/21.jpg)
Input images
Credit: Y.Y. Chuang
![Page 22: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/22.jpg)
Cylindrical warping
Credit: Y.Y. Chuang
![Page 23: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/23.jpg)
Alignment
a rotation of the camera is a translation of the
cylinder!
( )
( )
−
−
=
yx
y
yx
x
yx
y
yx
yx
yx
yx
yx
x
yxIyxJI
yxIyxJI
v
u
III
III
,
,
,
2
,
,,
2
),(),(
),(),(
Credit: Y.Y. Chuang
![Page 24: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/24.jpg)
Alignment
Credit: Y.Y. Chuang
![Page 25: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/25.jpg)
Blending
Why blending: parallax, lens distortion, scene
motion, exposure difference
Alpha-blending
Poisson blending
Adelson’s pyramid blending
![Page 26: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/26.jpg)
Blending
Credit: Y.Y. Chuang
![Page 27: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/27.jpg)
Linear Blending
Credit: Y.Y. Chuang
![Page 28: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/28.jpg)
Linear Blending
Credit: Y.Y. Chuang
![Page 29: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/29.jpg)
Linear Blending
29
Image from http://www.cs.ubc.ca/~lowe/425/slides/11-PanoramasAR.pdf
![Page 30: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/30.jpg)
Multi-band Blending [BURT and ADELSON 83]
30
Linear blending Multi-band blending
A multi-resolution spline with application to image mosaics.
Peter J. Burt and Edward Adelson. ACM Transactions on Graphics, 1983.
![Page 31: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/31.jpg)
Multi-band Blending
1. Laplacian pyramids LA and LB are
constructed for images A and B respectively.
2. A third Laplacian pyramid LS is constructed
by copying nodes from the left half of LA to
the corresponding nodes of LS, and nodes in
the right half of LB to the right half of LS
3. The final image S is obtained by expanding
and summing the levels of LS.
31
![Page 32: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/32.jpg)
Multi-band Blending
32
Source: https://docs.opencv.org/3.1.0/dc/dff/tutorial_py_pyramids.html
![Page 33: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/33.jpg)
Low frequency (l > 2 pixels)
High frequency (l < 2 pixels)
2-band Blending
Credit: Y.Y. Chuang
![Page 34: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/34.jpg)
Linear Blending
![Page 35: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/35.jpg)
2-band Blending
![Page 36: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/36.jpg)
Assembling the panorama
Stitch pairs together, blend, then crop
Credit: Y.Y. Chuang
![Page 37: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/37.jpg)
Problem: Drift
Error accumulation
◼ small errors accumulate over time
Credit: Y.Y. Chuang
![Page 38: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/38.jpg)
Problem: Drift
Solution◼ add another copy of first image at the end
◼ there are a bunch of ways to solve this problem add displacement of (y1 – yn)/(n -1) to each image after the first
compute a global warp: y’ = y + ax
run a big optimization problem, incorporating this constraint
◼ best solution, but more complicated
◼ known as “bundle adjustment”
(x1,y1)
• copy of first
image
(xn,yn)
Credit: Y.Y. Chuang
![Page 39: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/39.jpg)
End-to-end alignment and crop
Credit: Y.Y. Chuang
![Page 40: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/40.jpg)
Viewer: panorama
+
+
+
+
example: http://www.cs.washington.edu/education/courses/cse590ss/01wi/projects/project1/students/dougz/index.html
Credit: Y.Y. Chuang
![Page 41: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/41.jpg)
Demo
http://graphics.stanford.edu/courses/cs178/applets/projection.html 41
![Page 42: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/42.jpg)
Student paper presentations
◼ Everett, Riley
Photographic tone reproduction for digital images.
E. Reinhard, M. Stark, P. Shirley, and J. Ferwerda,
SIGGRAPH 2012
◼ Fan, Yubin
Burst photography for high dynamic range and low-
light imaging on mobile cameras.
Samuel W. Hasinoff, Dillon Sharlet, Ryan Geiss,
Andrew Adams, Jonathan T. Barron, Florian Kainz,
Jiawen Chen, and Marc Levoy.
SIGGRAPH Asia 2016
43
![Page 43: Prof. Feng Liuweb.cecs.pdx.edu/~fliu/courses/cs510/lectures/Lecture10.pdfCylindrical panorama 1. Take pictures on a tripod (or handheld) 2. Warp to cylindrical coordinate 3. Compute](https://reader036.vdocuments.net/reader036/viewer/2022062318/5fcfd280772a5845ef6b3d71/html5/thumbnails/43.jpg)
Next Time
Multi-perspective panorama
Image segmentation
Student paper presentations◼ 05/05: Gatehouse, Christopher
Deep High Dynamic Range Imaging of Dynamic Scenes. N. K. Kalantari and R. Ramamoorthi,
SIGGRAPH 2017
◼ 05/05: Ghildyal, Abhijay Night Sight: Seeing in the Dark on Pixel Phones
◼ https://ai.googleblog.com/2018/11/night-sight-seeing-in-dark-on-pixel.html
https://www.blog.google/products/pixel/see-light-night-sight/
◼ 05/07: Giampietro, James AutoCollage
C. Rother, L. Bordeaux, Y. Hamadi, A. Blake, SIGGRAPH 2006
◼ 05/07: Green, Jordan Picture Collage
J. Wang, J. Sun, L. Quan, X. Tang, H. Shum, CVPR 2006
44