image processing and morphing
DESCRIPTION
Image Processing and Morphing. Vision for Graphics CSE 590SS, Winter 2001 Richard Szeliski. Image Pyramids. Image Pyramids. Pyramid Creation. “Laplacian” Pyramid Created from Gaussian pyramid by subtraction L l = G l – expand(G l+1 ). filter mask. “Gaussian” Pyramid. - PowerPoint PPT PresentationTRANSCRIPT
Image Processing and Morphing
Vision for GraphicsCSE 590SS, Winter 2001
Richard Szeliski
Image Pyramids
1/8/2001 Vision for Graphics 3
Image Pyramids
1/8/2001 Vision for Graphics 4
Pyramid Creation
“Laplacian” Pyramid• Created from Gaussian
pyramid by subtractionLl = Gl – expand(Gl+1)
filter mask
“Gaussian” Pyramid
1/8/2001 Vision for Graphics 5
Octaves in the Spatial Domain
Bandpass Images
Lowpass Images
1/8/2001 Vision for Graphics 6
PyramidsAdvantages of pyramids
• Faster than Fourier transform• Avoids “ringing” artifacts
Many applications• small images faster to process• good for multiresolution processing• compression• progressive transmission
Known as “mip-maps” in graphics communityPrecursor to wavelets
• Wavelets also have these advantages
1/8/2001 Vision for Graphics 7
laplacianlevel
4
laplacianlevel
2
laplacianlevel
0
left pyramid right pyramid blended pyramid
1/8/2001 Vision for Graphics 8
Pyramid Blending
Image Warping
1/8/2001 Vision for Graphics 10
Image Warpingimage filtering: change range of image
g(x) = h(f(x))
image warping: change domain of imageg(x) = f(h(x))
f
x
hf
x
f
x
hf
x
1/8/2001 Vision for Graphics 11
Image Warpingimage filtering: change range of image
g(x) = h(f(x))
image warping: change domain of imageg(x) = f(h(x))
h
h
f
f g
g
1/8/2001 Vision for Graphics 12
Parametric (global) warping
Examples of parametric warps:
translation rotation aspect
affineperspective
cylindrical
1/8/2001 Vision for Graphics 13
2D coordinate transformations
translation: x’ = x + t x = (x,y)rotation: x’ = R x + tsimilarity: x’ = s R x + taffine: x’ = A x + tperspective: x’ H x x = (x,y,1)
(x is a homogeneous coordinate)
These all form a nested group (closed w/ inv.)
1/8/2001 Vision for Graphics 14
Image Warping
Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?
f(x) g(x’)x x’
h(x)
1/8/2001 Vision for Graphics 15
Forward Warping
Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?
1/8/2001 Vision for Graphics 16
Forward Warping
Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
f(x) g(x’)x x’
h(x)
• What if pixel lands “between” two pixels?• Answer: add “contribution” to several pixels,
normalize later (splatting)
1/8/2001 Vision for Graphics 17
Inverse Warping
Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x)
f(x) g(x’)x x’
h-1(x’)
• What if pixel comes from “between” two pixels?
1/8/2001 Vision for Graphics 18
Inverse Warping
Get each pixel g(x’) from its corresponding location x = h-1(x’) in f(x)
• What if pixel comes from “between” two pixels?• Answer: resample color value from
interpolated (prefiltered) source image
f(x) g(x’)x x’
1/8/2001 Vision for Graphics 19
Interpolation
Possible interpolation filters:• nearest neighbor• bilinear• bicubic (interpolating)• sinc / FIR
Needed to prevent “jaggies” and “texture crawl” (see demo)
1/8/2001 Vision for Graphics 20
Prefiltering
Essential for downsampling (decimation) to prevent aliasing
MIP-mapping [Williams’83]:1. build pyramid (but what decimation filter?):
– block averaging– Burt & Adelson (5-tap binomial)– 7-tap wavelet-based filter (better)
2. trilinear interpolation– bilinear within each 2 adjacent levels– linear blend between levels (determined by pixel size)
1/8/2001 Vision for Graphics 21
Prefiltering
Essential for downsampling (decimation) to prevent aliasing
Other possibilities:• summed area tables• elliptically weighted Gaussians (EWA)
[Heckbert’86]
1/8/2001 Vision for Graphics 22
Image Warping – non-parametric
Specify more detailed warp function
Examples: • splines• triangles• optical flow (per-pixel motion)
1/8/2001 Vision for Graphics 23
Image Warping – non-parametric
Move control points to specify spline warp
Image Morphing
1/8/2001 Vision for Graphics 25
Image Morphing
How can we in-between two images?1. Cross-dissolve
(all examples from [Gomes et al.’99])
1/8/2001 Vision for Graphics 26
Image Morphing
How can we in-between two images?2. Warp then cross-dissolve = morph
1/8/2001 Vision for Graphics 27
Warp specification
How can we specify the warp?1. Specify corresponding points
• interpolate to a complete warping function
• Nielson, Scattered Data Modeling, IEEE CG&A’93]
1/8/2001 Vision for Graphics 28
Warp specification
How can we specify the warp?2. Specify corresponding vectors
• interpolate to a complete warping function
1/8/2001 Vision for Graphics 29
Warp specification
How can we specify the warp?2. Specify corresponding vectors
• interpolate [Beier & Neely, SIGGRAPH’92]
1/8/2001 Vision for Graphics 30
Warp specification
How can we specify the warp?3. Specify corresponding spline control points
• interpolate to a complete warping function
1/8/2001 Vision for Graphics 31
Final Morph Result
Image Mosaics
1/8/2001 Vision for Graphics 33
Image Mosaics (stitching)
Blend together several overlapping images into one seamless mosaic (composite)
+ + … +=
1/8/2001 Vision for Graphics
Mosaics for Video CodingConvert masked images into a background sprite
for content-based coding
+ + +
=
1/8/2001 Vision for Graphics 35
f = 180 (pixels)
Cylindrical Panoramas
Map image to cylindrical or spherical coordinates• need known focal length
Image 384x300 f = 380f = 280
1/8/2001 Vision for Graphics 36
Cylindrical warpingGiven focal length f and image center (xc,yc)
X
YZ
(X,Y,Z)
(sin,h,cos)
1/8/2001 Vision for Graphics 37
Radial distortion
Correct for “bending” in wide field of view lenses
1/8/2001 Vision for Graphics 38
Image Stitching
1. Align the images over each other2. Blend the images together (demo)
1/8/2001 Vision for Graphics 39
Image Stitching Assignment
1. Take pictures on a tripod (or handheld)2. Warp to cylindrical coordinates3. Automatically compute pairwise alignments4. Fix up the end-to-end alignment5. Blend the images together6. Crop the result and import into a viewer
1/8/2001 Vision for Graphics 40
Image registration (preview)
How do we determine alignment between images?
• One possible answer: block matching (correlation),
i.e., find minimum squared error
2),(
01 ),(),(),( yx
yxIvyuxIvuE
1/8/2001 Vision for Graphics 41
Image registration (preview)
How do we determine alignment between images?
• Another possible answer: Fourier-domain correlation
[Brown’92] or phase correlation [Kuglin & Hines’75]
• <…discuss Fourier-domain convolution formulas…>
• More on alignment/registration on Wednesday
[Anandan]
1/8/2001 Vision for Graphics 42
Full-view Panorama
++
++
++
++
1/8/2001 Vision for Graphics 43
Texture Mapped Model
Image Enhancement
1/8/2001 Vision for Graphics 45
Image Enhancement
Noise removal:• low-pass filtering
g(x) = h(x) f(x) = ih(-i) f(x+i)
• median filtering• anisotropic diffusion
– iteratively smooth with similar neighbors [Perona & Malik’90]
Sharpening:• “unsharp masking”
– remove a little bit of blurred image from original(darkroom trick)
1/8/2001 Vision for Graphics 46
Image Enhancement
Super-resolution:• hallucinate more detail from
single low-res image[Freeman & Pasztor,ICCV’99]
– “learn” mapping between levels
• combine several low-res images using known (linear) formation model [Irani’91;Mann’94]
1/8/2001 Vision for Graphics 47
Image Enhancement
Super-resolution:• [Freeman &
Pasztor,ICCV’99]– mapping between
levels depends on “training data”
1/8/2001 Vision for Graphics 48
Image Enhancement
Brightness and gamma correction• linearize input brightness
Histogram equalization• balance (“flatten”) distribution
of brightness values
1/8/2001 Vision for Graphics 49
Image Enhancement
High dynamic range photography[Debevec et al.’97; Mitsunaga & Nayar’99]• combine several different exposures together
1/8/2001 Vision for Graphics 52
BibliographyL. Williams. Pyramidal parametrics.Computer Graphics, 17(3):1--11, July 1983.
L. G. Brown. A survey of image registration techniques.Computing Surveys, 24(4):325--376, December 1992.
C. D. Kuglin and D. C. Hines. The phase correlation image alignment method.In IEEE 1975 Conference on Cybernetics and Society, pages 163--165, New York, September 1975.
J. Gomes, L. Darsa, B. Costa, and L. Velho. Warping and Morphing of Graphical Objects.
Morgan Kaufmann Publishers, San Francisco Altos, California, 1999.
G. M. Nielson. Scattered data modeling.IEEE Computer Graphics and Applications, 13(1):60--70, January 1993.
T. Beier and S. Neely. Feature-based image metamorphosis.Computer Graphics (SIGGRAPH'92), 26(2):35--42, July 1992.
1/8/2001 Vision for Graphics 53
BibliographyL. Williams. Performace driven facial animation.Computer Graphics, 24(4):235--242, 1990.
P. Perona and J. Malik. Scale space and edge detection using anisotropic diffusion.IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(7):629--639, July 1990.
W. T. Freeman and E. C. Pasztor. Learning low-level vision.In Seventh International Conference on Computer Vision (ICCV'99), pages 1182--1189, Kerkyra, Greece, September 1999.
M. Irani and S. Peleg. Improving resolution by image registration.Graphical Models and Image Processing, 53(3):231--239, May 1991.
S. Mann and R. W. Picard. Virtual bellows: Constructing high-quality images from video.
In First IEEE International Conference on Image Processing (ICIP-94), volume I, pages 363--367, Austin, Texas, November 1994.
1/8/2001 Vision for Graphics 54
BibliographyPaul E. Debevec and Jitendra Malik.Recovering high dynamic range radiance maps from photographs.Proceedings of SIGGRAPH 97, pages 369--378, August 1997.ISBN 0-89791-896-7. Held in Los Angeles, California.
T. Mitsunaga and S. K. Nayar.Radiometric self calibration.In IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'99), volume 1, pages 374--380, Fort Collins, June 1999.