1

Image Stitching and Composition

Lots of slides from from Bill Freeman, Alyosha Efros, Steve Seitz, Rick Szeliski, A. Agrawal , R. Raskar, Hong Chen, and Shmuel Peleg.

© prof. dmartin

• Stitching:– Capturing different portions of a scene with an

overlap region viewed in both images.– Combine the two images together.– The seam between the images should be

invisible.

• Composition:– Cut a portion of a scene from one image and

paste it in another image.

Image Stitching and CompositionImage Stitching and Composition

Overlap Regions

Image StitchingImage Stitching Image StitchingImage Stitching

• Naïve stitching might seem unnatural due to parallax, lens distortion, scene motion, different exposure, different illuminations

2

Image CompositionImage Composition

Target ImageSource Images

Topics to be CoveredTopics to be Covered

• Alpha Blending• Multi-band blending• Optimal-cut composing• Blending in gradient domain

Image StitchingImage Stitching Alpha Blending (feathering)Alpha Blending (feathering)

01

01

+

=Encoding transparency

I(x,y) = (αR, αG, αB, α)

Iblend = Ileft + Iright

3

Effect of Window SizeEffect of Window Size

0

1 left

right0

1

Effect of Window SizeEffect of Window Size

0

1

0

1

Good Window SizeGood Window Size

0

1

“Optimal” Window: smooth but not ghosted

Left Image Right Image

Left + Right Narrow Transition Wide Transition Multibands blend

Alpha blending is problematic in the Alpha blending is problematic in the presence of high and low frequencies:presence of high and low frequencies:

4

What is the Optimal Window?What is the Optimal Window?

• To avoid seams– window >= size of largest prominent feature

• To avoid ghosting– window <= 2*size of smallest prominent feature

• Solution: Multi-band blending (Burt & Adelson 83):

– Decompose the image into multi-band freq.– Blend each band appropriately

Multiband PyramidMultiband Pyramid

expand

-

expand

expand

-

- =

=

=

Gaussian Pyramid

LaplacianPyramid

Pyramid BlendingPyramid Blending

0

1

0

1

0

1

Left pyramid Right pyramidblend

Multiband BlendingMultiband Blending

5

Example: blend apple and orangeExample: blend apple and orange laplacianlevel

4

laplacianlevel

2

laplacianlevel

0

left pyramid right pyramid blended pyramid

Multiband BlendingMultiband Blending Multiband Blending: exampleMultiband Blending: example

6

© prof. dmartin

Multiband Blending: exampleMultiband Blending: example Optimal CutOptimal Cut• Imperfect registration or presence of moving

objects generate ghost objects• Solution: Optimal Cut

Optimal Cut Optimal Cut (Davis, 1998)(Davis, 1998)• Segment the mosaic

– Single source image per segment– Avoid artifacts along boundries– Dijkstra’s algorithm

Input texture

B1 B2

Random placement of blocks

block

B1 B2

Neighboring blocksconstrained by overlap

B1 B2

Minimal errorboundary cut

EfrosEfros & Freeman, 2001& Freeman, 2001

7

min. error boundary

Minimal error boundaryMinimal error boundaryoverlapping blocks vertical boundary

__ ==22

overlap error

Optimal Cut Optimal Cut -- AlgorithmAlgorithm• e(k,j) – penalty for cutting the kth row at the jth column • Cut trajectory is monotonic and continues• Can be solved using Dynamic Programming:

e(k,j)= I1(k,j)-I2(k,j-1) j

k

I1I2

E(k,j)=e(k,j)+MINE(k-1,j-1), E(k-1,j), E(k-1,j+1)

Graph CutGraph Cut

• What if we want similar “cut-where-things-agree” idea, but for non monotonic cut or closed regions?

• Dynamic programming can’t handle loops• Solution: GraphCut

Graph Cut Graph Cut (simple example (simple example Boykov&JollyBoykov&Jolly, ICCV, ICCV’’01)01)

n-links

s

t a cuthard constraint

hard constraint

Minimum cost cut can be computed in polynomial time(max-flow/min-cut algorithms)

8

Graph Cut Composition Graph Cut Composition ((KwatraKwatra et al, 2003)et al, 2003) Graph Cut: Graph Cut: ss--tt GraphGraph

• G=(V,E) • a source node s, and a sink node t• Directed edge (i,j)∈E from node i to node j• Each edge has a non-negative capacity C(i,j)• C(i,j)=0 for non-exist edges.

s t2 3

1

6 3i j

C(i,j)

Flow in Flow in ss--tt GraphGraph• A Flow is a real value f(i,j) that assign a real

value to edge (i,j) under the constraints:– Capacity constraint : f(i,j) ≤ C(i,j)– Flow balance constraint:

for each i∉s,t flow_in = flow_outTotal flow:

s t2/2 1/3

1/1

1/62/3i j

f(i,j)/C(i,j)

( ) ( ), ,i j

f s i f j t f= − =∑ ∑

ss--tt CutCut

• A cut is a partition of V into two exclusive subsets S and T, s.t. s∈S, and t ∈T

s t2 3

1

6 3 s t2 3

1

6 3

s-t cutnot s-t cut

9

Capacity of Capacity of ss--tt CutCut

• A capacity of a s-t cut is the accumulated capacities of edges from S towards T:

s t2 3

1

6 3

( ),

([ , ]) ,i S j T

C S T c i j∈ ∈

= ∑

([ , ]) 2 3C S T = +

Max Flow = Min cutMax Flow = Min cut

• Maximum flow is the flow that has maximum value among all possible flow assignments

• Minimum cut is the s-t cut whose capacity is minimum among all possible s-t cut

• maximum flow = minimum cut

• Polynomial time solution

Back to Blending: Common ProblemsBack to Blending: Common Problems

Global Intensity Difference

Horizontal Misalignment

Vertical Misalignment

Previous ApproachesPrevious Approaches• Optimal Seam

– Search for a curve in the overlap region on which the differences between the images is minimal

– Poorly handles global intensity differences

10

Previous ApproachesPrevious Approaches

• Alpha Blending– Smooth the transition by weighting alpha mask

as a function of the distance from the seam– Poorly handles misalignments

Previous ApproachesPrevious Approaches

• Pyramid Blending– Combine different frequency bands with different

alpha masks– Poorly handles misalignments

Blending in Gradient DomainBlending in Gradient Domain• In Pyramid Blending, we decomposed our

image into 2nd derivatives (Laplacian) and a low-res image

• Let us now look at 1st derivatives (gradients):– No need for low-res image – captures everything (up to a constant)

• Algorithm: – Differentiate– Blend– Reintegrate

Gradient Domain Blending (1D)Gradient Domain Blending (1D)

Twosignals

Regularblending

Blendingderivatives

bright

dark

11

Comparisons: Comparisons: Levin et al, 2004Levin et al, 2004 BasicsBasics

• Images as scalar fieldsR2 -> R

Vector FieldVector Field

• A vector function G: R2 →R2

• Each point (x,y) is associated with a vector (u,v)

G(x,y)=[ u(x,y) , v(x,y) ]

Gradient FieldGradient Field• Partial derivatives of scalar field • Direction

– Maximum rate of change of scalar field

• Magnitude– Rate of change

,yI

xII

∂∂

∂∂

=∇

),( yxI

12

Continuous Continuous v.sv.s. Discrete. Discrete

Image I(x,y) Ix Iy

Continues case → Derivative

Discrete case → Finite differences

[ ]

[ ] IyI

IxI

T ∗−→∂∂

∗−→∂∂

110

110

Editing in Gradient DomainEditing in Gradient Domain

• Given vector field G=(u(x,y),v(x,y)) (pasted gradient) in a bounded region Ω. Find the values of I in Ω that optimize:

Ω∂∗

Ω∂Ω=−∇∫∫ IIwithGI

I

2min

G=(u,v)

I*I

I

I*

Ω

Intuition Intuition -- What if G is null?What if G is null?

• 1D:

• 2D:

x1 x2

Ω∂∗

Ω∂Ω=∇∫∫ IIwithI

I

2min

What if What if GG is not null?is not null?

• 1D case

Seamlessly paste onto

- Add a linear function so that the boundary condition is respected- Gradient error is equally distributed all over Ω in order to respect the boundary condition

13

2D case2D case

From Perez et al. 2003

2D case2D case

From Perez et al. 2003

2D case2D case How do we reconstruct from gradient field?How do we reconstruct from gradient field?

• I minimizes the integral:

( ) dydxGIE∫∫ ∇ ,

( )22

2, ⎟⎟⎠

⎞⎜⎜⎝

⎛−

∂∂

+⎟⎠⎞

⎜⎝⎛ −

∂∂

=−∇=∇ vyIu

xIGIGIE

14

Calculus of VariationsCalculus of Variations: Euler: Euler--Lagrange EquationLagrange Equation

• I must satisfy

• Substituting E we get:

0=∂∂

−∂∂

−∂∂

yx IE

dyd

IE

dxd

IE

022 2

2

2

2

=⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

−∂∂

+⎟⎟⎠

⎞⎜⎜⎝

⎛∂∂

−∂∂

yv

yI

xu

xI

Calculus of VariationsCalculus of Variations: Euler: Euler--Lagrange EquationLagrange Equation

2

2

2

22

yI

xII

∂∂

+∂∂

=∇

GI div2 =∇

yv

xuGdiv

∂∂

+∂∂

=

(laplacian)

Poisson EquationPoisson Equation

yv

xuGdivI

∂∂

+∂∂

==∇2

• Second order PDE• Iterative solution (Conjugate Gradient)

Alternative Derivation (discrete notation)Alternative Derivation (discrete notation)

⎟⎟⎠

⎞⎜⎜⎝

⎛=⎟⎟

⎠

⎞⎜⎜⎝

⎛vu

Iy

x

DD

⎥⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

−−

−−

−−

−

=

1111

1111

1111

11

xD[ ]*110* −=∂∂x

IxDIx

=∗∂∂

• Let Dx - Toeplitz matrix

15

( ) ( ) ⎟⎟⎠

⎞⎜⎜⎝

⎛=⎟⎟

⎠

⎞⎜⎜⎝

⎛vu

I Ty

Tx

y

xTy

Tx DD

DD

DD

• Normal equation:

( ) vuI Ty

Txy

Tyx

Tx DDDDDD +=+

[ ] [ ]*011*110, −=−⇒ flipDNote Tx

[ ]*121 −=xTx DD

GdivI =∇ 2 Numerical SolutionNumerical Solution

• Discretize Laplacian

=+≡∇ yTyx

Tx DDDD2 [ ] ∗

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡−=∗

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡−+−

010141010

12

1121

⎥⎥⎥⎥⎥⎥⎥⎥⎥

⎦

⎤

⎢⎢⎢⎢⎢⎢⎢⎢⎢

⎣

⎡

−−

−−

−−

−

14111411

1141111411

114111141

1141

Sparse Toeplitz Matrix

Comments:– A is sparse.– A is symmetric and can be inverted.– If Ω is rectangular, A is a Teoplitz matrix.– Size of A is ∼NxN.– Impractical to form or store A.– Impractical to invert A

( ) vuI Ty

Txy

Tyx

Tx DDDDDD +=+

bI =A

Iterative Solution: Conjugate GradientIterative Solution: Conjugate Gradient

• Solves a linear system Ax=b (in our case x=I)• A is square, symmetric, positive semi-definite.• Advantages:

– Fast!– No need to store A but calculating Ax – In our case Ax can be calculated using a single

convolution.– Can deal with constraints.

16

Conjugate gradientConjugate gradient

• For each step i: – Take the residual d(i)=b-Ax(i) (= -gradient)– Make it A-orthogonal to the previous ones – Find minimum along this direction

• Needs at most N iterations.• Matlab command:

x=cgs(A,b)A can be a function handle afun

such that afun(x) returns A*x

Solving Poisson equation with boundary conditionsSolving Poisson equation with boundary conditions

Ω

*ΩS T

( ) ( )ΩΩ

+=+ SDDDDDDDD yTyx

Txy

Tyx

Tx I s.t. ∗∗ ΩΩ

= TI

Ω∇S

• Define a circumscribing square Π=Ω∪Ω*– Let Ω⊂ Π denotes the edited image area.– Let Ω*= Π-Ω denotes the surrounding area.

yyxxk ∂∗∂+∂∗∂=

( ) ∗ΩΩ∗= IIkAI U

Ω

*ΩS T( ) *ΩΩ∗= TSkb U

x=cgs(<computeAI>,b)

• The above requirements can be express as a linear set of equations:

Ω∇S

[ ] [ ]bAI

TSDDDD

IIDDDD yyxxyyxx =⇒⎟

⎟⎠

⎞⎜⎜⎝

⎛ +=⎟

⎟⎠

⎞⎜⎜⎝

⎛ +⇒

∗∗ Ω

Ω

Ω

Ω

Image stitchingImage stitching

17

Gradient Domain CompositionGradient Domain Composition Cut & Paste Cut & Paste Paste in Gradient Domain Paste in Gradient Domain

Another exampleAnother example

18

Transparent CloningTransparent CloningTransparent Cloning

I SΩ Ω∇ = ∇2

S TI Ω ΩΩ

∇ + ∇∇ = ( )max ,I S TΩ Ω Ω∇ = ∇ ∇

Transparent CloningTransparent Cloning

Shadow Transferring Shadow Transferring

19

Other Gradient Domain ApplicationsOther Gradient Domain Applications

Changing local illuminationChanging local illuminationDefect concealmentDefect concealment

High Dynamic Range CompressionHigh Dynamic Range Compression

Small exposure: Dark inside

High Dynamic Range CompressionHigh Dynamic Range Compression

Large exposure: Outside Saturated

20

Software Tone Mapping

Short Exposure

Long Exposure

High Dynamic Range CompressionHigh Dynamic Range CompressionShadow RemovalShadow Removal

Shadow RemovalShadow Removal

Null gradient on boundary and integrateshadow removal

Shadow RemovalShadow Removal

21

CompositingCompositing

• Compositing images– Have a clever blending function

Alpha blendingblend different frequencies differentlyGradient based blending

– Choose the right pixels from each imageDynamic programming – optimal seamsGraph-cuts

• Put it all together:– Interactive Digital Photomontage

Interactive Digital PhotomontageInteractive Digital Photomontage

Aseem Agarwala, Mira Dontcheva, Maneesh Agrawala, Steven Drucker, Alex Colburn, Brian Curless, David Salesin, Michael Cohen, “Interactive Digital Photomontage”, SIGGRAPH 2004

• Combining multiple photos• Find seams using graph cuts• Combine gradients and integrate

22

23

photomontageset of originals

Source images Brush strokes Computed labeling

Composite

Brush strokes Computed labeling

SummarySummaryStitching & Compositing

• Alpha Blending• Multi-band blending• Optimal-cut composing• Blending in gradient domain • Applications in Gradient Domain• Photomontage

24

T H E E N DT H E E N D