IMAGE-BASED MODELING AND RENDERING

8. IMAGE AND VIEW MORPHING

I-Chen Lin, Dept. of CS, National Chiao Tung University

• Image Morphing

• View Morphing

Outline

Reference:

• Prof. S. Seitz and P. Heckbert (CMU), Image-based modeling and rendering course notes.

• T. Beier, S. Neely, "Feature Based Image Metamorphosis," Proc. SIGGRAPH'92, pp. 35-42, 1992.

• S. M. Seitz, C. R. Dyer, "View Morphing", Proc. SIGGRAPH’96, pp. 21-30.

• Rearranging pixels of a picture.

• It’s useful for both image processing and for computer graphics (namely, for texture mapping).

• Finding corresponding points in the source and destination images.

• This function is called the “mapping” or “transformation”.

Image Warping

• How to warp an image with a more complex function?

• If the source and destination is of structural similarity

• piecewise affine over a triangulation.

• piecewise projective over a quadrilaterization.

• piecewise bilinear over a rectangular grid.

More Complex Situations

• Arbitrary functions can be used.

• Store u[x,y] and v[x,y] in large arrays

• Learning techniques: e.g. neural network

• ………

• Popular methods in computer graphics:

• Beier-Neely warp (popular for image morphing)

• Radial-basis-function (RBF) based data scattering.

• ………

More Complex Situations …

• Short for “metamorphosis”.

• Early films used cross-dissolving only, but that looks artificial, non-physical.

• Morph = warp the shape + cross-dissolve the colors.

• If the mappings are defined, cross-dissolving can simply be performed by interpolation.

Morphing

Proposed by Thaddeus Beier, Shawn Neely

Prof. ACM SIGGRAPH’92, pp. 35-42.

FEATURE-BASED IMAGE METAMORPHOSIS

• To transform one image smoothly into another image

Motivation

Overview

Image warping

Image warping

Color Blending

• How to find a controllable mapping function with little manual assistance?

• What’s the problem of mesh-based warping tech.?

• How about the concept of “field”?

Concepts (Beier-Neely warp )

• Features

• Points

• Lines

• Curves

• ……

• Fields

• Linear

• quadratic

• ……

How to control the mapping?

Source destination

Using control lines

With only one pair of lines

• To map the pixel X from pixel X' by corresponding lines PQ and P'Q'.

With only one pair of lines

• Scan every pixels on the destination image and compute the corresponding pixels on the source image.

An example (only one pair of lines)

• Transforms each pixel coordinate by a rotation, translation ,and/or scale.

• The image is scaled along the direction of lines by the ratio of the length of the line.

• Pure rotation and translation.

• Affine transformations. (uniform scales and shear are not possible to specify.)

The properties of this approach

With multiple pairs of lines

• Specifying more complex transformations.

• The displacement Di = Xi’-X.

• A weighted average of these displacements is calculated.

With multiple pairs of lines (cont.)

For each pixel X in the destination

DSUM = (0,0)

weightsum = 0

For each line Pi Qi

calculate u,v based on Pi Qi

calculate X'i based on u,v and Pi'Qi'

calculate displacement Di = Xi' - Xi for this line

dist = shortest distance from X to Pi Qi

weight = (length^p / (a + dist))^b

DSUM += Di * weight

weightsum += weight

X' = X + DSUM / weightsum

destinationImage(X) = sourceImage(X')

With multiple pairs of lines (cont.)

An example with two line pairs

Morphing between two images

1. Assign the control lines.

2. CL(t) = (1-t)CL(0)+t CL(1)

3. Generate I0(t), I1(t) according to CL(t)

4. I(t) = (1-t)I0(t) + t I1(t)

Source

CL(0)

I(0)

Destination

CL(1)

I(1)

CL(t)

I0(t)

CL(t)

I1(t)

• Interpolating the endpoints of each line. • A rotating line would shrink.

• Interpolating the center position and orientation of each line. • This isn’t very obvious to the user, who might be surprised by how the lines

interpolate.

Two different ways of interpolating

source

destination

intermediate

• It’s much more expressive.

• The animator simply has to describe how lines in a source image are mapped into lines in a destination image.

• The mesh warping is much less intuitive.

Advantages

• This algorithm must compute the effect of every feature line on the movement of that pixel. slow!

• In contrast, mesh warping has only local control .

• Sometimes unexpected interpolations are generated.

• pixel may be far away from all lines.

Disadvantages

Disadvantage (cont.)

• The algorithm might make a mistake in computing how that pixel should move during morphing.

• Need a set of line segments at key frames for each sequence.

Animated sequences

A sequence from Michael Jackson’s MTV “Black or White”

Questions

• Is this warping/morphing tech. applicable to all kinds of image transitions?

2D image morphing

What we want :

• Finding out when image morphing tech. is “correct” (between perspective views)?

• How to extend the image morphing method?

Questions (cont.)

Proposed by S. M. Seitz, C. R. Dyer

Proc. SIGGRAPH’96, pp. 21-30.

VIEW MORPHING

View Morphing

Morphed View

Virtual Camera

View Morphing (cont.)

2D vs. View Morphing

• Distortions occur if only image morphing is applied.

Image morphing

View morphing

• 3D modeling with stereo vision techniques.

• What’re the advantages and disadvantages?

• Image morphing

• Shape-distorting.

• 3D shapes often results in shapes that are mathematically quite different.

• Unless …….

• Is there any special case that image morphing works “correct” for perspective views?

Synthesizing Novel Views

Synthesizing Novel Views (cont.)

• Image morphing for perspective views.

• Delicately designing the transition of control lines.

• Still requiring an approximate model.

• Parallel views

• Linear image interpolation should produce new perspective views when camera moves parallel to the image plane.

Parallel Views

• Linear image interpolation should produce new perspective views when camera moves parallel to the image plane.

0100

000

000

0

0

0 f

f

M

Projection Matrix M = [ H | -HC ] P^ = MP

Assume

C0 = (0,0,0) C1 = (CX, CY, 0)

0100

00

00

11

11

1 Y

X

Cff

Cff

M

Parallel Views (cont.)

PMZ

PMZ

sPMZ

sspps

s

1

11)1()1( 1010

10)1( sMMsM s

p0 I0 , p1 I1 , P = [X Y Z 1]T

p = (1-s) p0 + s p1 = ?

Cs = (sCX, sCY, 0)

fs = (1-s) f0+s f1

Parallel Views (cont.)

• Interpolating images produced from parallel cameras

• Producing the illusion of simultaneously moving the camera on the line C0C1.

• Moving the optic center and zooming continuously.

• How about non-parallel views?

Non-Parallel Views

1. Prewarp align views

Non-Parallel Views

1. Prewarp align views 2. Morph move camera

Non-Parallel Views

1. Prewarp align views 2. Morph move camera 3. Postwarp point camera

View Morphing

1. Pre-warp: • Warp the source and

destination image to form parallel views.

2. Morphing: • Interpolating the

prewarped images.

3. Post-warp • Warp the interpolated

image to form the Is

1

1 2

2

3

H0-1

H1-1

Hs

View Morphing (cont.)

• For simplicity, the pre-warped images are of f = 1.

0100

010

001

ˆ0

0

0 Y

X

C

C

M

0100

010

001

ˆ1

1

1 Y

X

C

C

M

• Prewarping + morphing + postwarping

• Multiple image resampling blurring

• Folds

• Occlusion, visibility.

• Holes

Problems

• Prewarping + morphing + postwarping • Multiple image resampling blurring • Solution1: supersampling. • Solution2: aggregate warp. • ……

• Folds • Occlusion, visibility. • Solution1: Comparing Z values (Z-buffer). • Solution2: Disparity • ……

• Holes • Solution1: interpolating (or extrapolating) neighboring pixels. • Solution2: additional images. • ……

Problems (cont.)

• Besides the interpolated Ms.

• Specifying a continuous transformation of the image plane.

• E.g. interpolating the orientations of image planes.

Producing the Morphing

• Provides • A mobile virtual camera • Better image morphs

• Requires no prior knowledge • No 3D shape information • No dense correspondence • No camera information • No training on other images

• Cons • Limited to interpolation • Problems with occlusions (leads to ghosting) • Prewarp can be unstable (need good F-estimator)

Features of View Morphing

• Visualization • User-controlled virtual camera

• Image postprocessing

• Image databases • Frontal face images for recognition ...

Applications

• Camera calibration

• E.g. Camera calibration tool box (for OpenCV / Matlab)

• Through point correspondences

• Estimating the fundamental matrix F (or essential matrix E)

• We will introduce in the later lecture notes…

How to estimate H?