3d imaging motion. extract visual information from image sequences assume illumination conditions...

3D IMAGING

Motion

Motion

Extract visual information from image sequences

Assume illumination conditions do not change

Differences in image Camera moves (Some) Objects in a scene move Both camera and objects move

Difference from stereo

Stereo from multiple views Images taken at the same time moment

Motion Images taken in small time intervals

Differences Camera position might not be known Changes are smaller Not necessary caused by a single 3D rigid transform

Rotation and Translation Multiple objects

Why analyze motion

Understand scene structure Objects, Distance, Velocities

Understand scene properties without a priori knowledge on the world

Random Dot

Time to Collision

f

L

v

L

Do

l(t)

An object of height L moves with constant velocity v:•At time t=0 the object is at:• D(0) = Do

•At time t it is at • D(t) = Do – vt

•It will crash with the camera at time:• D(t) = Do – vt = 0

• t = Do/v

t=0t

D(t)

Time to Collision

f

L

v

L

Do

l(t)

t=0t

D(t)

The image of the object has size l(t):

Taking derivative wrt time:

Time to Collision

f

L

v

L

Do

l(t)

t=0t

D(t)

And their ratio is:

Time to Collision

f

L

v

L

Do

l(t)

t=0t

D(t)

And time to collision:

Can be directly measured from image

Can be found, without knowing L or Do or v !!

Parallax

Parallax=apparent change in position when viewed from different lines of sights

Closer objects have larger parallax

Depth cues

Kinetic Depth Effect

Motion Field

3D velocity field Each point has a velocity vector

2D Motion field Projection of these 3D velocity vector onto

image plane

2D Motion Field

2D Optical Flow

Apparent motion of image brightness pattern

Assuming that illumination does not change:

Image changes are due to the RELATIVE MOTION between the scene and the camera.

There are 3 possibilities: Camera still, moving scene Moving camera, still scene Moving camera, moving scene

Motion Analysis Problems

Correspondence Problem Track corresponding elements across frames

Reconstruction Problem Given a number of corresponding elements, and

camera parameters, what can we say about the 3D motion and structure of the observed scene?

Segmentation Problem What are the regions of the image plane which

correspond to different moving objects?

Motion Field (MF)

The MF assigns a velocity vector to each pixel in the image.

These velocities are INDUCED by the RELATIVE MOTION btw the camera and the 3D scene

The MF can be thought as the projection of the 3D velocities on the image plane.

2D Motion Field and 2D Optical Flow Motion field: projection of 3D motion vectors on image plane

• Optical flow field: apparent motion of brightness patterns

• We equate motion field with optical flow field

0 0

00

Object point has velocity , induces in imagei

ii

P

d d

dt dt

v v

r rv v

2 Cases Where this Assumption Clearly is not Valid

(a) (b)

(a) A smooth sphere is rotating under constant illumination. Thus the optical flow field is zero, but the motion field is not.

(b) A fixed sphere is illuminated by a moving source—the shading of the image changes. Thus the motion field is zero, but the optical flow field is not.

What is Meant by Apparent Motion of Brightness Pattern?

The apparent motion of brightness patterns is an awkward concept. It is not easy to decide which point P' on a contour C' of constant brightness in the second image corresponds to a particular point P on the corresponding contour C in the first image.

Aperture Problem

(a) Line feature observed through a small aperture at time t.

(b) At time t+t the feature has moved to a new position. It is not possible to determine exactly where each point has moved. From local image measurements only the flow component perpendicular to the line feature can be computed.

Normal flow: Component of flow perpendicular to line feature.

(a) (b)

Brightness Constancy Equation Let P be a moving point in 3D:

At time t, P has coords (X(t),Y(t),Z(t)) Let p=(x(t),y(t)) be the coords. of its image

at time t. Let E(x(t),y(t),t) be the brightness at p at

time t. Brightness Constancy Assumption:

As P moves over time, E(x(t),y(t),t) remains constant.

Brightness Constraint Equation

flow. optical of components the, ,, and irradiance thebe ,,Let yxvyxutyxE

expansionTaylor

,,,, tyxEtttvytuxE

0limit takingand by dividing

,,,,

tt

tyxEet

Et

y

Ey

x

ExtyxE

0

derivative total theofexpansion theiswhich

0

dt

dE

t

E

dt

dy

y

E

dt

dx

x

E

short: 0 tyx EvEuE

Brightness Constancy Equation

Taking derivative wrt time:

Brightness Constancy Equation

Let(Frame spatial gradient)

(optical flow)

and(derivative across frames)

Brightness Constancy Equation

Becomes:

The OF is CONSTRAINED to be on a line !

Normal Motion/Aperture Problem

Barber Pole Illusion

Solving the aperture problem How to get more equations for a pixel?

Basic idea: impose additional constraints most common is to assume that the flow field is

smooth locally one method: pretend the pixel’s neighbors have the

same (u,v) If we use a 5x5 window, that gives us 25 equations per

pixel!

Constant flow Prob: we have more equations than unknowns

– The summations are over all pixels in the K x K window

• Solution: solve least squares problem

– minimum least squares solution given by solution (in d) of:

Taking a closer look at (ATA)

This is the same matrix we used for corner detection!

Taking a closer look at (ATA)

The matrix for corner detection:

is singular (not invertible) when det(ATA) = 0

One e.v. = 0 -> no corner, just an edgeTwo e.v. = 0 -> no corner, homogeneous region

Aperture Problem !

But det(ATA) = Õ li = 0 -> one or both e.v. are 0

A Pattern of Hajime Ouchi

Copyright A.Kitaoka 2003

Reconstruction

Rotation

P

rrr

rrr

rrr

3,32,31,3

3,22,21,2

3,12,11,1 Represents a 3D rotation of the points in P.

First, look at 2D rotation (easier)

n

n

yyy

xxx

21

21 ...

cossin

sincos

cossin

sincosR

Matrix R acts on points by rotating them.

• Also, RRT = Identity. RT is also a rotation matrix, in the opposite direction to R.

Simple 3D Rotation

n

n

n

zzz

yyy

xxx

21

21

21...

100

0cossin

0sincos

Rotation about z axis.

Rotates x,y coordinates. Leaves z coordinates fixed.

Full 3D Rotation

cossin0

sincos0

001

cos0sin

010

sin0cos

100

0cossin

0sincos

R

• Any rotation can be expressed as combination of three rotations about three axes.

100

010

001TRR

• Rows (and columns) of R are orthonormal vectors.

• R has determinant 1 (not -1).

Alper Yilmaz, Fall 2004 UCF

3D Motion

Displacement model Velocity model

Z

Y

X

TZYXZ

TZYXY

TZYXX

Z

Y

X

T

T

T

Z

Y

X

Z

Y

X

Z

Y

X

1

1

1

TR

Displacement Model and Its Projection onto Image Space

Z

Y

X

TZYXZ

TZYXY

TZYXX

Perspective projection

ZT

yx

ZT

yxy

ZT

yx

ZT

yxx

Z

Y

Z

X

1

1

x X Y Z TX

X Y Z TZ

y X Y Z TY

X Y Z TZ

Alper Yilmaz, Fall 2004 UCF

Velocity Model in 3DOptical Flow

Z

Y

X

T

T

T

Z

Y

X

Z

Y

X

1

1

1

Z

Y

X

T

T

T

Z

Y

X

ZZ

YY

XX

0

0

0

Z

Y

X

T

T

T

Z

Y

X

Z

Y

X

100

010

001

0

0

0

Alper Yilmaz, Fall 2004 UCF

Velocity Model in 3DOptical Flow

3

2

1

12

13

23

0

0

0

V

V

V

Z

Y

X

Z

Y

X

321

213

132

VXYZ

VZXY

VYZX

Z

Y

X

T

T

T

Z

Y

X

ZZ

YY

XX

0

0

0

rotational velocities

translational velocities

3D optical flow

VXX

3

2

1

Z

Y

X

X

cross product

Velocity Model in 2D

Perspective projection

321

213

132

VXYZ

VZXY

VYZX

Z

Yfy

Z

Xfx

yv

xu

212331

2

2213

32

1

)(

)(

yf

xyf

yZ

Vx

Z

Vfv

xf

xyf

yxZ

V

Z

Vfu