panoramas. creating full view panoramic image mosaics and environment maps richard szeliski and...
TRANSCRIPT
![Page 1: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/1.jpg)
Panoramas
![Page 2: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/2.jpg)
Creating Full View Panoramic Image Mosaics and Environment Maps
Richard Szeliski and Heung-Yeung Shum
Microsoft Research
![Page 3: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/3.jpg)
Outline
• Main Contribution• Introduction • Details • Results
![Page 4: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/4.jpg)
Contributions• Novel approach to creating full view panoramic
mosaics from image sequence– Not necessarily pure horizontal camera panning– Do not require any controlled motions or constraints
• Represent image mosaics using a set of transforms
• Fast and robust • Method to recover camera focal length• Extract environment map from image mosaic
![Page 5: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/5.jpg)
Introduction
• Image-based rending– Realistic rendering without geometry models
• IBR without depth info – Only support user panning, rotation and zoom– QuickTime VR, Surround Video…– Cylindrical image, spherical maps..
![Page 6: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/6.jpg)
Introduction
• To capture panoramic images– Using panoramic camera to get cylindrical image– Using a lens with a large field of view (fisheye lens)– Mirrored pyramids and parabolic mirrors
• Hardware-intensive methods– Take a series regular picture or video and stitch them
together– Require carefully-controlled camera motion– Produce only cylindrical image
![Page 7: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/7.jpg)
Novel Algorithms
• They use 3-parameter rotational motion model: fewer unknowns, more robust – Instead of general 8-parameter planar perspective
motion model
• Estimate focal length from a set of 8-parameter perspective registration
• Gap closing
![Page 8: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/8.jpg)
Cylindrical Panoramas
• Cylindrical panorama is easy to construct• Coordinate transformation
XY
Z
cylindrical image
h
![Page 9: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/9.jpg)
Cylindrical Panorama
• With ideal pinhole camera and known f,
• Distortion: horizontal lines becomes curved
( , , ) ( , , )X Y Z w u v f
f
( , )u v
![Page 10: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/10.jpg)
Spherical Panorama
2 2 2
2 2
2 2
sin
sin
cos
Y
X Y ZX
X ZZ
X Z
X
YZ
(X,Y,Z)
(cos sin ,sin ,cos cos )
2 2tan
Y
X Z
![Page 11: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/11.jpg)
Motion Model
• Warp image to cylindrical panorama• Ideal horizontal panning sequence – Rotation->Translation in angle
• In practice– Vertical translation to compensate for vertical
jitter and optical twist( , )x yt t On WARPED Image!
![Page 12: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/12.jpg)
Motion Recovery
• Estimate incremental translation• by minimizing the intensity error
• Taylor series expansion
( , )x yt t t
21 0( ) ( ) ( )i ii
E I I t x t x
2( ) T
i ii
E e t g t
1 0( ) ( )i i ie I I x x Current intensity or color error
1( )Ti iI g x Image gradient of I_1 at position x’i
![Page 13: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/13.jpg)
Motion Recovery
• Minimization -> Least square solution
Ti i i i
i i
e g g t g
2( ) T
i ii
E e t g t
![Page 14: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/14.jpg)
Motion Recovery• Large initial displacement– Coarse to fine optimization scheme
J. R. Bergen, P. Anandan, K. J. Hanna, and R. Hingorani. Hierarchical model-based motion estimation
![Page 15: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/15.jpg)
Motion Recovery
• Large initial displacement– Coarse to fine optimization scheme
• Discontinuities in intensity or color between images being composed– Feathering algorithm: weighted by distance map
![Page 16: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/16.jpg)
Limitations of Cylindrical or Spherical Panorama
• Only handle pure panning motion• Ill-sampling at north pole and south pole
cause big registration errors• Require knowing the focal length f – Estimation of focal length of lens by registering
images is not very accurate
![Page 17: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/17.jpg)
Perspective Panoramas
• Planar perspective transform between images using 8 parameters
For example, if only translation, m2, m5 are the unknowns
![Page 18: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/18.jpg)
Perspective Panoramas
• Iteratively update transform matrix using
• Resampling image I_1 with x’’=(I+D)Mx to
M (I +D)M
1( '')I x
![Page 19: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/19.jpg)
Perspective Panoramas
• Minimize2 2
1 0( ) ( ) ( ) T Ti i i i i
i i
E I I x e d x g J d
( )i i dJ J x
![Page 20: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/20.jpg)
Perspective Panoramas
• Least square minimization 2 2
1 0( ) ( ) ( ) T Ti i i i i
i i
E I I x e d x g J d
Ad b
Hessian Matrix Accumulated gradient or residual
![Page 21: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/21.jpg)
Perspective Panoramas
• Works well if initial estimates of correct transformation are close enough
• Slow convergence • Get stuck in local minima
![Page 22: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/22.jpg)
Rotational Panoramas
• Cameras centered at the origin
~x TVRp1
x
y
x
X
Y
Z
p
Simplicity of rotation: set c_x=c_y=0, pixel start from image center
1 1 p R V x
![Page 23: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/23.jpg)
Rotational Panoramas
• Camera rotating around its center of projection
• Focal length is known and the same for all images V_k = V_l = V
• Angular velocity z( , , )x y
![Page 24: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/24.jpg)
Rotational Panoramas
• Incremental rotation matrix (Rodriguez’s formula)
![Page 25: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/25.jpg)
Rotational Panoramas
• is the deformation matrix as• Jacobian of
M (I +D)M
0 0 0
0 0 1
0 0
0 0 1
0 0 0
0 0
0 1/ 0
1/ 0 0
f
f
f
f
![Page 26: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/26.jpg)
Rotational Panoramas• 3 parameters Incremental
rotation vector• Update R_k in
• Much easier and more intuitive to interactively adjust
![Page 27: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/27.jpg)
Estimate Focal Length
1/
1/
![Page 28: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/28.jpg)
Estimate Focal Length
• If fixed focal length, take average of f_0 and f_1:
• If multiple focal length for every images, use median value for final estimate
• We can also update the focal length as part of the image registration process using least squares approach
0 1f f f
![Page 29: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/29.jpg)
Closing gap in a panorama
• Matching the first image and the last one• Compute the gap angle • Distribute the gap angle evenly across the
whole sequence– Modify rotations by – Update focal length
• Only works for 1D panorama where the camera is continuously turning in the same direction
g
/g imageN
' (1 / 360 )gf f
![Page 30: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/30.jpg)
Conclusion
• Does not place constraints on how the images to be taken with hand held cameras
• Accurate and robust – Estimate only 3 rotation parameters instead of 8
parameters in general perspective transforms
• Increases accuracy, flexibility and ease of use • Focal length estimation
![Page 31: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/31.jpg)
Results
![Page 32: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/32.jpg)
Photographing Long Scenes with Multi-Viewpoint Panoramas
http://grail.cs.washington.edu/projects/multipano/
![Page 33: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/33.jpg)
Abstract• Multi-viewpoint panoramas of long, roughly planar scenes
– Façades of buildings along a city street
• Panoramas are composed of relatively large regions of ordinary perspective
• User interactions – to identify the dominant plane– To draw strokes indicating various high-level goals
• Markov Random Field optimization
![Page 34: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/34.jpg)
Introduction
• Long scene – Hard to take photographs at one point– Wider field of view: large distortion– Single perspective photographs are not very
effective at conveying long scenes• Street side in a city• Bank of a river • Aisle of a grocery store
![Page 35: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/35.jpg)
Introduction
• Take photographs– Walk along the other side and take handheld
photographs at intervals of one large step (roughly one meter)
• Output – a single panorama that visualizes the entire extent
of the scene captured in the input photographs and resembles what a human would see when walking along the street
![Page 36: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/36.jpg)
Contributions
• A practical approach to creating high quality, high-resolution, multi-viewpoint panoramas with a simple and casual capture method.
• A number of novel techniques, including – An objective function that describes desirable
properties of a multi-viewpoint panorama, and– A novel technique for propagating user-drawn
strokes that annotate 3D objects in the scene
![Page 37: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/37.jpg)
Related Work
• Single-viewpoint panoramas– Rotating a camera around its optical center
• Strip Panoramas– Translating camera– Orthographic projection along the horizontal axis– Perspective along vertical – Varying strip width by depth estimation of
appearance optimization– High-speed video camera and special setups
![Page 38: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/38.jpg)
Strip Panoramas• Exhibit distortion for scene with varying depths,
especially if these depth variations occur across the vertical axis of the image
• Created from video sequences, and still images created from video rarely have the same quality as those captured by a still camera– Low resolution, compression artifacts
• Capturing a suitable video can be cumbersome
![Page 39: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/39.jpg)
Approach
• Inspired by the work of artist Michael Koller– Multi-viewpoint panoramas of San Francisco
streets – Large regions of ordinary perspective photographs– Artfully seamed together to hide the transitions– Attractive and informative
![Page 40: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/40.jpg)
multi-viewpoint panoramas • Each object in the scene is rendered from a viewpoint roughly in
front of it to avoid perspective distortion.
• The panoramas are composed of large regions of linear perspective seen from a viewpoint where a person would naturally stand (for example, a city block is viewed from across the street, rather than from some faraway viewpoint).
• Local perspective effects are evident; objects closer to the image plane are larger than objects further away, and multiple vanishing points can be seen.
• The seams between these perspective regions do not draw attention; that is, the image appears natural and continuous.
![Page 41: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/41.jpg)
Properties
• A dominant plane in the scene • Not attempt to – Create multi-viewpoint panoramas that turn around street
corners– show all four sides of a building
![Page 42: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/42.jpg)
System Overview
Pre-processing• Takes the source images• Removes radial distortion• Recovers the camera projection matrices• Compensates for exposure variation
Panorama Surface• Defines the picture surface• Source photographs are then projectedonto this surface
Composition• Selects a viewpoint for each pixel in the output panorama• Interactively refine by drawing strokes
![Page 43: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/43.jpg)
Capture Images
• Use a digital SLR camera with auto-focus and manually control the exposure to avoid large exposure shifts
• Use a fisheye lens to insure a wide field of view for some data
![Page 44: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/44.jpg)
Pre-Processing
• Recover projection matrices of each camera so that we can later project the source images onto a picture surface
• Use structure-from-motion system [Hartley and Zisserman 2004] built by Snavely et al. [2006]– Using sift for keypoint detection and matching– Bundle adjustment
, ,i i iR t f
http://phototour.cs.washington.edu/bundler/
![Page 45: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/45.jpg)
Exposure Compensation
• Adjust the exposure of the various photographs so that they match better in overlapping regions
• Recover the radiometric response function of each photograph [Mitsunaga and Nayar 1999]
• Simpler approach
i i j jk I k I
Least squares for all the pairs of SIFT match
![Page 46: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/46.jpg)
Picture Surface Selection
• The picture surface should be roughly aligned with the dominant plane of the scene
• Extrude in Y direction
![Page 47: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/47.jpg)
Picture Surface Selection• Define the coordinate system of the recovered 3D scene
– Automatic: fit a plane to the camera viewpoints using principal component analysis• The dimension of greatest variation (the first principal component) is
the new x-axis, and • The dimension of least variation the new y-axis
– Interactive: user selects a few of these projected points that lie along the desired axes
• Draw the curve in the xz plane that defines the picture surface
![Page 48: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/48.jpg)
Picture Surface Selection
• Easy to identify for street scenes• River bank: hard to specify by drawing strokes– The user selects clusters of scene points that
should lie along the picture surface– The system fits a third-degree polynomial z(x) to
the z-coordinates of these 3D scene points as a function of their x-coordinates
![Page 49: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/49.jpg)
Sample Picture Surface
• Project each S(I,j) on picture surface to source photograph
![Page 50: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/50.jpg)
Average Image
the average image of all the projected sources
the average image after unwarping to straighten the ground plane and cropping
![Page 51: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/51.jpg)
Interactive Refinement• Small drifts that can accumulate during structure-from-
motion estimation and lead to ground planes that slowly curve
• The user clicks a few points along the average image to indicate y values of the image that should be warped straight
• Resample and crop
the average image after unwarping to straighten the ground plane and cropping
![Page 52: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/52.jpg)
Viewpoint Selection
• How to choose color for each pixel on panorama from one of the source image I_i(p)
Determine L(p): L is the image no.
![Page 53: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/53.jpg)
Viewpoint Selection
• Optimization using Markov Random Field• Cost function:
Gives straight-on view
Seamless transition
Encourages the panorama to resemble the average image in areas where the scene geometry intersects the picture surface
![Page 54: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/54.jpg)
Viewpoint Selection
• Minimize the overall cost function for each pixel and each pair of neighboring pixel
• Solve using min-cut optimization– compute the panorama at a lower resolution so that the
MRF optimization can be computed in reasonable time– create higher-resolution versions using the hierarchical
approach described by Agarwala et al. [2005].– We thus composite the final Panorama in the gradient
domain to smooth errors across these seams
![Page 55: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/55.jpg)
Viewpoint Selection• Not vertical strips
![Page 56: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/56.jpg)
Interactive Refinement
• The user should be able to express desired changes to the panorama without tedious manual editing of the exact seam locations.
• Three types of strokes:– View selection: use certain viewpoint– Seam suppression: no seam should pass an object– Inpainting: eliminate undesirable features
![Page 57: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/57.jpg)
Interactive Refinement
![Page 58: Panoramas. Creating Full View Panoramic Image Mosaics and Environment Maps Richard Szeliski and Heung-Yeung Shum Microsoft Research](https://reader035.vdocuments.net/reader035/viewer/2022062314/56649dc75503460f94abb98b/html5/thumbnails/58.jpg)
Results
• 1 hour to capture images (100) and 20 mins for interactions
• Not for every scene– Suburban scenes with a range of different depths
• More results can be found at http://grail.cs.washington.edu/projects/multipano/