Prof. Feng Liu

Winter 2020

http://www.cs.pdx.edu/~fliu/courses/cs410/

01/30/2020

Last Time

Hough transform

2

Today

Image alignment

3The following slides are from Prof. S. Lazebnik

Image alignment

Image from http://graphics.cs.cmu.edu/courses/15-463/2010_fall/

A look into the past

http://blog.flickr.net/en/2010/01/27/a-look-into-the-past/

A look into the past

Leningrad during the blockade

http://komen-dant.livejournal.com/345684.html

Bing streetside images

http://www.bing.com/community/blogs/maps/archive/2010/01/12/new-bing-maps-

application-streetside-photos.aspx

Image alignment: Applications

Panorama stitching

Recognition

of object

instances

Image alignment: Applications

Small degree of overlap, intensity changes

Occlusion,

clutter

Image alignment

◼ Direct (pixel-based) alignment

Search for alignment where most pixels agree

◼ Feature-based alignment

Search for alignment where extracted features agree

Can be verified using pixel-based alignment

Alignment as fitting

• Previous lectures: fitting a model to features in one

image

i

i Mx ),(residual

Find model M that minimizes

M

xi

Alignment as fitting

• Previous lectures: fitting a model to features in one

image

• Alignment: fitting a model to a transformation between

pairs of features (matches) in two images

i

i Mx ),(residual

i

ii xxT )),((residual

Find model M that minimizes

Find transformation T

that minimizes

M

xi

T

xixi'

2D transformation models

• Similarity

(translation,

scale, rotation)

• Affine

• Projective

(homography)

Let’s start with affine transformations

• Simple fitting procedure (linear least squares)

• Approximates viewpoint changes for roughly planar

objects and roughly orthographic cameras

• Can be used to initialize fitting for more complex

models

Fitting an affine transformation

• Assume we know the correspondences, how do we

get the transformation?

),( ii yx ),( ii yx

+

=

2

1

43

21

t

t

y

x

mm

mm

y

x

i

i

i

i

=

i

i

ii

ii

y

x

t

t

m

m

m

m

yx

yx

2

1

4

3

2

1

1000

0100

Fitting an affine transformation

• Assume we know the correspondences, how do we

get the transformation?

),( ii yx ),( ii yx

+

=

2

1

43

21

t

t

y

x

mm

mm

y

x

i

i

i

i

=

i

i

ii

ii

y

x

t

t

m

m

m

m

yx

yx

2

1

4

3

2

1

1000

0100

affine model parameters

Fitting an affine transformation

• Linear system with six unknowns

• Each match gives us two linearly

independent equations: need at least three

to solve for the transformation parameters

=

i

i

ii

ii

y

x

t

t

m

m

m

m

yx

yx

2

1

4

3

2

1

1000

0100

Fitting a plane projective transformation

• Homography: plane projective transformation

(transformation taking a quad to another

arbitrary quad)

Homography

• The transformation between two views of a planar surface

• The transformation between images from two cameras

that share the same center

Application: Panorama stitching

Source: Hartley & Zisserman

Fitting a homography

• Homogeneous coordinates

Converting to homogeneous

image coordinates

Converting from homogeneous

image coordinates

Fitting a homography

• Homogeneous coordinates

• Equation for homography:

Converting to homogeneous

image coordinates

Converting from homogeneous

image coordinates

=

11 333231

232221

131211

y

x

hhh

hhh

hhh

y

x

Fitting a homography

• Equation for homography:

ii xHx =

=

11 333231

232221

131211

i

i

i

i

y

x

hhh

hhh

hhh

y

x

0= ii xHx

−

−

−

=

i

T

ii

T

i

i

T

ii

T

i

T

i

T

i

i

T

i

T

i

T

i

i

yx

x

y

y

x

xhxh

xhxh

xhxh

xh

xh

xh

3

2

1

12

31

23

1

0

0

0

0

3

2

1

=

−

−

−

h

h

h

xx

xx

xx

TT

ii

T

ii

T

ii

TT

i

T

ii

T

i

T

xy

x

y3 equations,

only 2 linearly

independent

Direct linear transform

• H has 8 degrees of freedom (9 parameters, but scale is arbitrary)

• One match gives us two linearly independent equations

• Four matches needed for a minimal solution (null space of 8x9 matrix)

• More than four: homogeneous least squares

0

0

0

0

0

3

2

1111

111

=

−

−

−

−

h

h

h

xx

xx

xx

xx

T

nn

TT

n

T

nn

T

n

T

TTT

TTT

x

y

x

y

0=hA

Geometric approach

25

𝐻= arg min𝐻

σ𝑖=1𝑁 𝑑2 𝑥𝑖 , 𝐻𝑥𝑖

′ + 𝑑2 𝑥𝑖′, 𝐻−1𝑥𝑖

𝐻𝑥𝑖′: The location to which 𝑥𝑖

′ in Image I’ is mapped by H in Image I

Symmetric transfer error to minimize the

geometric errors

𝑑2 𝑥𝑖 , 𝐻𝑥𝑖′ : The error between the projected point and the target point

Robust and large-scale alignment

Image from http://graphics.cs.cmu.edu/courses/15-463/2010_fall/

RANSAC

❑ The set of putative matches contains a very high

percentage of outliers

❑ RANSAC loop:

✓ Randomly select a seed group of matches

✓ Compute transformation from seed group

✓ Find inliers to this transformation

✓ If the number of inliers is sufficiently large, re-compute least-

squares estimate of transformation on all of the inliers

❑ Keep the transformation with the largest number of

inliers

Homework 2: Image Stitching

Feature detection using SIFT

Feature matching

Homography estimation

◼ Use RANSAC to remove feature outliers

Image warping and blending

28

Image warping

29

Image 1 Image 2

Image warping

30

Image 1

Image warping

31

Image 1

Image warping

32

Image 1

I_pano(x,y) = blend(I1(x, y), I2(H-1(x, y))

Backward warping

Next Time

Warping

33