image blending - computer...
TRANSCRIPT
![Page 1: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/1.jpg)
Image Blending
15-463: Computational PhotographyAlexei Efros, CMU, Fall 2008
© NASA
![Page 2: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/2.jpg)
Image Compositing
![Page 3: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/3.jpg)
Compositing Procedure1. Extract Sprites (e.g using Intelligent Scissors in Photoshop)
Composite by David Dewey
2. Blend them into the composite (in the right order)
![Page 4: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/4.jpg)
Need blending
![Page 5: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/5.jpg)
Alpha Blending / Feathering
01
01
+
=Iblend = αIleft + (1-α)Iright
![Page 6: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/6.jpg)
Setting alpha: simple averaging
Alpha = .5 in overlap region
![Page 7: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/7.jpg)
Setting alpha: center seam
Alpha = logical(dtrans1>dtrans2)
DistanceTransformbwdist
![Page 8: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/8.jpg)
Setting alpha: blurred seam
Alpha = blurred
Distancetransform
![Page 9: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/9.jpg)
Setting alpha: center weighting
Alpha = dtrans1 / (dtrans1+dtrans2)
Distancetransform
Ghost!
![Page 10: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/10.jpg)
Affect of Window Size
0
1 left
right0
1
![Page 11: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/11.jpg)
Affect of Window Size
0
1
0
1
![Page 12: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/12.jpg)
Good Window Size
0
1
“Optimal” Window: smooth but not ghosted
![Page 13: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/13.jpg)
What is the Optimal Window?To avoid seams
• window = size of largest prominent feature
To avoid ghosting• window <= 2*size of smallest prominent feature
Natural to cast this in the Fourier domain• largest frequency <= 2*size of smallest frequency• image frequency content should occupy one “octave” (power of two)
FFT
![Page 14: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/14.jpg)
What if the Frequency Spread is Wide
Idea (Burt and Adelson)• Compute Fleft = FFT(Ileft ), Fright = FFT(Iright )• Decompose Fourier image into octaves (bands)
– Fleft = Fleft1 + Fleft
2 + …• Feather corresponding octaves Fleft
i with Frighti
– Can compute inverse FFT and feather in spatial domain• Sum feathered octave images in frequency domain
Better implemented in spatial domain
FFT
![Page 15: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/15.jpg)
Octaves in the Spatial Domain
Bandpass Images
Lowpass Images
![Page 16: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/16.jpg)
Pyramid Blending
0
1
0
1
0
1
Left pyramid Right pyramidblend
![Page 17: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/17.jpg)
Pyramid Blending
![Page 18: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/18.jpg)
laplacianlevel
4
laplacianlevel
2
laplacianlevel
0
left pyramid right pyramid blended pyramid
![Page 19: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/19.jpg)
Laplacian Pyramid: BlendingGeneral Approach:
1. Build Laplacian pyramids LA and LB from images A and B2. Build a Gaussian pyramid GR from selected region R3. Form a combined pyramid LS from LA and LB using nodes
of GR as weights:• LS(i,j) = GR(I,j,)*LA(I,j) + (1-GR(I,j))*LB(I,j)
4. Collapse the LS pyramid to get the final blended image
![Page 20: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/20.jpg)
Blending Regions
![Page 21: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/21.jpg)
Horror Photo
© david dmartin (Boston College)
![Page 22: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/22.jpg)
Results from this class (fall 2005)
© Chris Cameron
![Page 23: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/23.jpg)
Season Blending (St. Petersburg)
![Page 24: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/24.jpg)
Season Blending (St. Petersburg)
![Page 25: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/25.jpg)
Simplification: Two-band BlendingBrown & Lowe, 2003
• Only use two bands: high freq. and low freq.• Blends low freq. smoothly• Blend high freq. with no smoothing: use binary alpha
![Page 26: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/26.jpg)
Low frequency (λ
> 2 pixels)
High frequency (λ
< 2 pixels)
2-band Blending
![Page 27: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/27.jpg)
Linear Blending
![Page 28: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/28.jpg)
2-band Blending
![Page 29: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/29.jpg)
Don’t blend, CUT!
So far we only tried to blend between two images. What about finding an optimal seam?
Moving objects become ghosts
![Page 30: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/30.jpg)
Davis, 1998Segment the mosaic
• Single source image per segment• Avoid artifacts along boundries
– Dijkstra’s algorithm
![Page 31: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/31.jpg)
min. error boundary
Minimal error boundary
overlapping blocks vertical boundary
__ ==22
overlap error
![Page 32: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/32.jpg)
GraphcutsWhat if we want similar “cut-where-things-
agree” idea, but for closed regions?• Dynamic programming can’t handle loops
![Page 33: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/33.jpg)
Graph cuts (simple example à la Boykov&Jolly, ICCV’01)
n-links
s
t a cuthard constraint
hard constraint
Minimum cost cut can be computed in polynomial time(max-flow/min-cut algorithms)
![Page 34: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/34.jpg)
Kwatra et al, 2003
Actually, for this example, DP will work just as well…
![Page 35: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/35.jpg)
Lazy Snapping
Interactive segmentation using graphcuts
![Page 36: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/36.jpg)
Gradient DomainIn Pyramid Blending, we decomposed our
image into 2nd derivatives (Laplacian) and a low-res image
Let us now look at 1st derivatives (gradients):• No need for low-res image
– captures everything (up to a constant)• Idea:
– Differentiate– Blend– Reintegrate
![Page 37: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/37.jpg)
Gradient Domain blending (1D)
Twosignals
Regularblending
Blendingderivatives
bright
dark
![Page 38: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/38.jpg)
Gradient Domain Blending (2D)
Trickier in 2D:• Take partial derivatives dx and dy (the gradient field)• Fidle around with them (smooth, blend, feather, etc)• Reintegrate
– But now integral(dx) might not equal integral(dy)• Find the most agreeable solution
– Equivalent to solving Poisson equation– Can use FFT, deconvolution, multigrid solvers, etc.
![Page 39: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/39.jpg)
Perez et al., 2003
![Page 40: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/40.jpg)
Perez et al, 2003
Limitations:• Can’t do contrast reversal (gray on black -> gray on white)• Colored backgrounds “bleed through”• Images need to be very well aligned
editing
![Page 41: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/41.jpg)
Painting in Gradient Domain! (McCann)
Code available!
See Jim’s talk this Friday:James McCann
Real-Time Gradient-Domain Painting, 12:00 p.m., 5409 Wean Hall
![Page 42: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/42.jpg)
Putting it all togetherCompositing images
• Have a clever blending function– Feathering– Center-weighted– blend different frequencies differently– Gradient based blending
• Choose the right pixels from each image– Dynamic programming – optimal seams– Graph-cuts
Now, let’s put it all together:• Interactive Digital Photomontage, 2004 (video)
![Page 43: Image Blending - Computer Graphicsgraphics.cs.cmu.edu/courses/15-463/2008_fall/Lectures/blending.pdf · • Decompose Fourier image into octaves (bands) – F left = F left 1 + F](https://reader035.vdocuments.net/reader035/viewer/2022062414/5f0613447e708231d4162a72/html5/thumbnails/43.jpg)