motion computing in image analysis - mani v thomas cisc 489/689

Motion Computing in Image Analysis

- Mani V ThomasCISC 489/689

Roadmap Optic Flow Constraint Optic Flow Computation

Gradient Based Approach Feature Based Approach

Estimation Criterion Block Matching algorithms Conclusion

Some slides and illustrations are from M. Pollefeys and M. Shah

Importance of Visual Motion Apparent motion of objects on the image

plane is a strong cue to understand structure and 3D motion

Biological visual systems infer properties of the 3D world via motion

Two sub-problems of motion Problem of correspondence estimation

Which elements of a frame correspond to which elements of the next frame

Problem of reconstruction Given the correspondence and the camera’s

intrinsic parameters can we infer 3D motion and/or structure

Courtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

Apparent Motion Apparent motion of objects on the

image plane Caution required!!

Consider a perfectly uniform sphere that is rotating but no change in the light direction Optic flow is zero

Perfectly uniform sphere that is stationary but the light is changing Optic flow exists

Hope – apparent motion is very close to the actual motion


Optic Flow Computation Two strategies for computing motion

Differential Methods Spatio temporal derivatives for estimation of

flow at every position Multi-scale analysis required if motion not

constrained within a small range Dense flow measurements

Matching Methods Feature extraction(Image edges, corners) Feature/Block Matching and error minimization

Sparse flow measurements


Optic Flow Computation Image Brightness Constancy

assumption Let E be the image intensity as captured

by the camera Using Taylor series to expand E

Apparent brightness of moving objects remains constant

tt

Ey

y

Ex

x

EtyxEttyyxxE

,,,,

0

dt

dE

t

E

dt

dy

y

E

dt

dx

x

E

t

E

t

y

y

E

t

x

x

ELt

t

tyxEttyyxxELt

tt

00

,,,,

Optic Flow Computation Image Brightness Constancy

assumption Apparent brightness of moving objects

remains constant

The are the image gradient while the are the components of the motion field


0

t

E

dt

dy

y

E

dt

dx

x

E

EyExE , vdtdydtdx ,

0 tT EE v

Aperture Problem We can measure

Terms that can be measured Terms to be computed Number of equations - 1

The component of the motion field that is orthogonal to the spatial image gradient is not constrained by the image brightness constancy assumption

Intuitively The component of the flow in the gradient direction is

determined The component of the flow parallel to an edge is

unknownCourtesy: E. Trucco and A. Verri, “Introductory techniques for 3D Computer Vision”

tEyExE ,,

dtdydtdx ,

Different physical motion but same measurable motion within a fixed window

Optic Flow Constraint How to get more equations for a pixel?

Basic idea: impose additional constraints Most common is to assume that the flow field is

smooth locally One method: pretend the pixel’s neighbors have

the same (u,v) If we use a 5x5 window, that gives us 25 equations

per pixel!

0. iti EvuE pp

12512225

25

2

1

2525

22

11

bdA

E

E

E

v

u

EE

EE

EE

t

t

t

yx

yx

yx

p

p

p

pp

pp

pp

Lucas-Kanade Optic Flow We now have more equations than

unknowns

Solve the least squares problem Minimum least squares solution (in d) is given by

First proposed by Lucas-Kanade in 1981 Summation performed over all the pixels in the

window

bAdbdA min12512225

ty

tx

yyxy

yxxx

TT

EE

EE

v

u

EEEE

EEEE

bAdAA 1252521222

Lucas-Kanade Optic Flow Lucas-Kanade Optic flow

When is the Lucas-Kanade equations solvable ATA should be invertible ATA should not be too small (effects of noise)

Eigenvalues of ATA, 1 and 2 should not be small

ATA should be well conditioned 1/2 should not be large (1 = larger eigenvalue)

ty

tx

yyxy

yxxx

EE

EE

v

u

EEEE

EEEE

Edge

Gradient is large in magnitude Large 1 but small 2

Low texture region

Gradients has small magnitude Small 1 and small 2

High texture region

Gradients are different with large magnitudes

Large 1 and large 2

Improving the Lucas-Kanade method

When our assumptions are violated Brightness constancy is not satisfied The motion is not small A point does not move like its neighbors

Iterative Lucas-Kanade Algorithm Estimate velocity at each pixel by solving

Lucas-Kanade equations Warp H towards I using the estimated flow

field use image warping techniques

Repeat until convergence

Iterative Lucas-Kanade method

image Iimage H

Gaussian pyramid of image H Gaussian pyramid of image I

image Iimage H u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

Iterative Lucas-Kanade method

image Iimage J

Gaussian pyramid of image H Gaussian pyramid of image I

image Iimage H

run iterative L-K

run iterative L-K

warp & upsample

.

.

.

Feature Based Method Feature Extraction

Maxima in first derivative of the Image Local peak in the first derivative

Numerical Approximation

Compute the motion parameters from the best bipartite graph Correspondence between the feature points in one

image with those in the other

For more information: Ramesh Jain, Rangachar Kasturi, Brian Schunck: Machine Vision 1995 (140 - 159)

T

y

f

x

fyxfG

,

jifjifGx ,1, jifjifGy ,,1

Estimation Criterion Pixel domain Criterion

MAE/MSE Lorentzian Correlation

Frequency Domain Criterion Cross Correlation Phase Correlation

Estimation Criterion(contd.) Pixel Domain Criterion

Estimation criterion aim at minimizing

prediction error is sensitive to noise if number of pixels is not large or if region is poorly textured

Common choice of estimation criterion

Quadratic function is not good since a single large error can bias the estimate of the field

Absolute value function is better than the quadratic since cost grows linearly with error Does not require multiplications and is better suited for real-

time video encoders

)()()( 1 dnInId kkk

dn

,

Rdn

kk dnInId

,1)(

Estimation Criterion(contd.) A more robust criterion is based on the Lorentzian

function

Grows slower than |x| for larger errors Similarity measure using Correlation

Computationally complex because of the multiplications This criterion requires maximization

Usually the normalized Cross correlation is computed

2

2

21log

)()()( 1,

dnInIdC kRdnk

11,

r

dnInIr

dnInI

dnIknIk

kk

kk

For more details: M. Black, “Robust Incremental Optical Flow”

Estimation Criterion(contd.)

Estimation Criterion

0

1

2

3

4

5

6

-6 -4 -2 0 2 4 6

absolute lorentzian w=0.1 lorentzian w=0.3

Estimation Criterion(contd.) Frequency Domain Criterion

Amplitudes of both the FT are independent of z Argument difference depends linearly on translation

Global motion is recovered by evaluating the phase difference over a number of frequencies and solving the resulting system of equations

In practice, this method will work only for a single object moving across a uniform background

zujkk

T

euIdnIF 211 )(ˆ)(

)(ˆ)( uInIF kk

zuznIFnIF Tkk 2)()( 1

Estimation Criterion(contd.) Phase Correlation

In the case of a single global translation, the correlation surface becomes a Kronecker delta function

In practice, there are numerous peaks which correspond to the dominant displacements between the two images

The locations are relatively independent to illumination changes

)(ˆ)(ˆ)(ˆ)(ˆ

)(1

11,1

uIuI

uIuIFn

kk

kkkk

)()( 2

)()(,11

zneFn zuj

znInIkkkk

01

00)(

x

xx

Block Matching Algorithms Sparse motion measurements

Motion is spatially constant and temporally linear over a rectangular region of support

The minimization problem is

is an M x N block of pixels with the top-left corner co-ordinate at

xdtxtxtxx tt ,

2

1

b

bx

mPdd

m

min

mBn

mkkm dnInId

1 m

PnPPnPnnnP 2121 ,:,

mB 21,mmm

Block Matching Algorithms(contd.)

M

N

M

N

pp

-p-p

Current Picture Reference Frame

M

N(x,y)

(x+u,y+v)

p

p

-p

-p u

v


Principle of Locality of Reference Block Matching algorithms

Exhaustive Search Always finds the “deepest” minimum Computationally very expensive

If I x J is the picture resolution and rate is F fps the overall operations in comparing MxN blocks would be

This corresponds to 29.89 GOPS for p=15 at 30fps for a 720x480 image (3 operations per pixel of one subtraction, one absolute value and one addition)

3**12 2 MNpMN

IJF

Block Matching Algorithms(contd.) Logarithmic Search

Sub-optimal and may get trapped in a local minima Computationally feasible for real-time video encoders Search Method

Divide the search space at [-p/2, -p/2] Search at (0,0) and at 8 major points at the perimeter of the

rectangle at [-p/2, -p/2] Using best match position as starting point, search in the

eight perimeter points at the half distance window If I x J is the picture resolution and rate is F fps the

overall operations in comparing MxN blocks would be

This corresponds to 1.03 GOPS for p=15 at 30fps for a 720x480 image

3**18 MNkMN

IJF pk 2log

For more information refer the work by Dr. Lai-Man Po and C. K. Cheung (http://www.ee.cityu.edu.hk/~lmpo/publications/index.html)


Hierarchical Search Sub-optimal for regions containing detail and increased storage

requirements Computationally feasible for real-time video encoders Search method

Form several low resolution images by low pass filtering At the lowest resolution perform a sub-optimal search like log

search Propagate search vectors to higher resolution images and perform

search If I x J is the picture resolution and rate is F fps the overall

operations in comparing MxN blocks would be

This corresponds to 507.38 MOPS for p=15 at 30fps for a 720x480 image

3*16

*18014

22

MNp

MN

IJF

Conclusion Motion estimation Aperture problem Different algorithms to perform motion

analysis Lucas-Kanade algorithm

Estimation criterion for motion field computation

Block Matching Algorithms Computational complexity of motion analysis

motion computing in image analysis - mani v thomas cisc 489/689

Documents

estimation of flow

flow field

motion computing

measurable motion

flow parallel

motion field courtesy

hope apparent motion

actual motion courtesy