Computational Photography

Si Lu

Spring 2018

http://web.cecs.pdx.edu/~lusi/CS510/CS510_Computational_Photography.htm

04/26/2018

Last Time

o Panorama n Overviewn Feature detectionn Feature matching

2With slides by Prof. C. Dyer and K. Grauman

Today

o Panoraman Homography estimationn Blendingn Multi-perspective panoramas

3

Stitching Recipeo Align pairs of images

n Feature Detectionn Feature Matchingn Homography Estimation

o Align all to a common frameo Adjust (Global) & Blend

4

What can be globally aligned?

o In image stitching, we seek for a model to globally warp one image into another. Are any two images of the same scene can be aligned this way?n Images captured with the same center of

projectionn A planar scene or far-away scene

Credit: Y.Y. Chuang

A pencil of rays contains all views

realcamera

syntheticcamera

Can generate any synthetic camera viewas long as it has the same center of projection!

Credit: Y.Y. Chuang

Mosaic as an image reprojection

mosaic projection plane

o The images are reprojected onto a common planeo The mosaic is formed on this planeo Mosaic is a synthetic wide-angle camera

Credit: Y.Y. Chuang

Changing camera center

Does it still work?PP1

PP2

Credit: Y.Y. Chuang

Planar scene (or a faraway one)

o PP3 is a projection plane of both centers of projection, so we are OK!

o This is how big aerial photographs are made

PP1

PP3

PP2

Credit: Y.Y. Chuang

Motion models

o Parametric models as the assumptions on the relation between two images.

Credit: Y.Y. Chuang

2D Motion models

Credit: Y.Y. Chuang

Motion models

Translation

2 unknowns

Affine

6 unknowns

Perspective

8 unknowns

3D rotation

3 unknowns

Credit: Y.Y. Chuang

Determine pairwise alignment?

o Feature-based methods: only use feature points to estimate parameters

o We will study the “Recognising panorama” paper published in ICCV 2003

o Run SIFT (or other feature algorithms) for each image, find feature matches.

Credit: Y.Y. Chuang

Determine pairwise alignment

o p’=Mp, where M is a transformation matrix, p and p’ are feature matches

o It is possible to use more complicated models such as affine or perspective

o For example, assume M is a 2x2 matrix

o Find M with the least square error

n

ipMp

1

2'

yx

mmmm

yx

2221

1211

''

Credit: Y.Y. Chuang

Determine pairwise alignment

yx

mmmm

yx

2221

1211

''

'1221211

'1121111

ymymxxmymx

'

'

'2

'1

'1

22

21

12

11

22

11

11

0000

0000

00

n

n

nn

nn

yx

xyx

mmmm

yxyx

yxyx

yx

Over-determined systemCredit: Y.Y. Chuang

Normal equation

Given an over-determined system

bAx

bAAxA TT

the normal equation is that which minimizes the sum of the square differences between left and right sides

Why?

Credit: Y.Y. Chuang

Normal equation

bbxbAxbAAxAx

bbbAxAxbAxAxbAxbAxbAxbAx

bAxbAx

bAx

TTTTTTT

TTTTTT

TTT

TT

T

E 2

bAAxAx

TT 22 E

Credit: Y.Y. Chuang

Determine pairwise alignment

o p’=Mp, where M is a transformation matrix, p and p’ are feature matches

o For translation model, it is easier.

o What if the match is false? Avoid impact of outliers.

n

iiiii yymxxmE

1

2'2

2'1

1

0mE

Credit: Y.Y. Chuang

RANSAC [Fischler and Bolles 81]

o RANSAC = Random Sample Consensuso An algorithm for robust fitting of models in the

presence of many data outliersn Compare to robust statistics

o Given N data points xi, assume that majority of them are generated from a model with parameters , try to recover .

M. A. Fischler and R. C. Bolles. "Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography". Comm. of the ACM 24 (6): 381–395

Credit: Y.Y. Chuang

RANSAC algorithm

Run k times: (1) draw n samples randomly (2) fit parameters with these n samples (3) for each of other N-n points, calculate its distance to the fitted model, count the number of inlier points, cOutput with the largest c

How many times?How big? Smaller is better

How to define?Depends on the problem.

Credit: Y.Y. Chuang

How to determine kp: probability of real inliersP: probability of success after k trials

knpP )1(1 n samples are all inliers

a failure

failure after k trials

)1log()1log(npPk

n p k

3 0.5 35

6 0.6 97

6 0.5 293for P=0.99

Credit: Y.Y. Chuang

Example: line fitting

Credit: Y.Y. Chuang

Example: line fitting

n=2

Credit: Y.Y. Chuang

Model fitting

Credit: Y.Y. Chuang

Measure distances

Credit: Y.Y. Chuang

Count inliers

c=3

Credit: Y.Y. Chuang

Another trial

c=3

Credit: Y.Y. Chuang

The best model

c=15

Credit: Y.Y. Chuang

RANSAC for Homography

Credit: Y.Y. Chuang

RANSAC for Homography

Credit: Y.Y. Chuang

RANSAC for Homography

Credit: Y.Y. Chuang

A case study: cylindrical panorama

o What if you want a 360 field of view?

mosaic projection cylinder

Credit: Y.Y. Chuang

Cylindrical panoramas

o Stepsn Reproject each image onto a cylindern Blend n Output the resulting mosaic

Credit: Y.Y. Chuang

Cylindrical panorama

1. Take pictures on a tripod (or handheld)2. Warp to cylindrical coordinate3. Compute pair-wise alignments4. Fix up the end-to-end alignment5. Blending6. Crop the result and import into a viewer

It is required to do radial distortion correction for better stitching results!

Credit: Y.Y. Chuang

Taking pictures

Kaidan panoramic tripod head

Credit: Y.Y. Chuang

Where should the synthetic camera be

o The projection plan of some camerao Onto a cylinder

realcamera

syntheticcamera

Credit: Y.Y. Chuang

Cylindrical projection

Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htmCredit: Y.Y. Chuang

Cylindrical projection

Credit: Y.Y. Chuang

Cylindrical projection

Adopted from http://www.cambridgeincolour.com/tutorials/image-projections.htmCredit: Y.Y. Chuang

Input images

Credit: Y.Y. Chuang

Cylindrical warping

Credit: Y.Y. Chuang

Alignment

o a rotation of the camera is a translation of the cylinder!

yxy

yxx

yxy

yxyx

yxyx

yxx

yxIyxJI

yxIyxJI

vu

III

III

,

,

,

2

,

,,

2

),(),(

),(),(

Credit: Y.Y. Chuang

Alignment

Credit: Y.Y. Chuang

Blending

o Why blending: parallax, lens distortion, scene motion, exposure difference

o Alpha-blendingo Poisson blendingo Adelson’s pyramid blending

Blending

Credit: Y.Y. Chuang

Linear Blending

Credit: Y.Y. Chuang

Linear Blending

Credit: Y.Y. Chuang

Linear Blending

48

Image from http://www.cs.ubc.ca/~lowe/425/slides/11-PanoramasAR.pdf

Multi-band Blending [BURT and ADELSON 83]

49

Linear blending Multi-band blending

A multi-resolution spline with application to image mosaics. Peter J. Burt and Edward Adelson. ACM Transactions on Graphics, 1983.

Multi-band Blending

1. Laplacian pyramids LA and LB are constructed for images A and B respectively.

2. A third Laplacian pyramid LS is constructed by copying nodes from the left half of LA to the corresponding nodes of LS, and nodes in the right half of LB to the right half of LS

3. The final image S is obtained by expanding and summing the levels of LS.

50

Low frequency (l > 2 pixels)

High frequency (l < 2 pixels)

2-band Blending

Credit: Y.Y. Chuang

Linear Blending

2-band Blending

Assembling the panorama

o Stitch pairs together, blend, then crop

Credit: Y.Y. Chuang

Problem: Drift

o Error accumulationn small errors accumulate over time

Credit: Y.Y. Chuang

Problem: Drift

o Solutionn add another copy of first image at the endn there are a bunch of ways to solve this problemo add displacement of (y1 – yn)/(n -1) to each image after the firsto compute a global warp: y’ = y + axo run a big optimization problem, incorporating this constraintn best solution, but more complicatedn known as “bundle adjustment”

(x1,y1)

• copy of first image

(xn,yn)

Credit: Y.Y. Chuang

End-to-end alignment and crop

Credit: Y.Y. Chuang

Viewer: panorama

example: http://www.cs.washington.edu/education/courses/cse590ss/01wi/projects/project1/students/dougz/index.html

Credit: Y.Y. Chuang

Demo

http://graphics.stanford.edu/courses/cs178/applets/projection.html 59

Multi-perspective Panorama

60Reprint from Seitz and Kim 03

Multi-perspective Panorama

61

A. Agarwala, M. Agrawala, M. Cohen, D. Salesin, R. Szeliski. Photographing long scenes with multi-viewpoint panoramas. SIGGRAPH 2006

A Good Multiple Perspective Panorama

o Each object in the scene is rendered from a viewpoint roughly in front of it to avoid perspective distortion.

o The panoramas are composed of large regions of linear perspective seen from a viewpoint where a person would naturally stand (for example, a city block is viewed from across the street, rather than from some faraway viewpoint).

o Local perspective effects are evident; objects closer to the image plane are larger than objects further away, and multiple vanishing points can be seen.

o The seams between these perspective regions do not draw attention

62

Picture Surface Selection

63

A plan view (xz slice) of a scene. The extracted camera locations are shown in red, and the recovered 3D scene points in black. The blue polyline of the picture surface is drawn by the user to follow the building facades. The y-axis of the scene extends out of the page; the polyline is swept up and down the y-axis to form the picture surface.

z

x

Reprint from Agarwala et al. 2006

Project Images onto Picture Surface

64Reprint from Agarwala et al. 2006

View Point Selection

65

1.Each image is a view2.Pick a good view for each pixel

Reprint from Agarwala et al. 2006

Examples

66http://grail.cs.washington.edu/projects/multipano/

Next Week

o Mid-term project presentationn Speak clearly and slowlyn It is just a warm upn Relax and enjoy

67