Motion from normal flow

Optical flow difficulties

• The aperture problem • Depth discontinuities

Translational Normal Flow

• In the case of translation each normal flow vector constrains the location of the FOE to ahalf-plane.

• Intersection of half-planes provides FOE.

nu

u Zn

tr

Egoestimation from normal flow

• Idea: choose particular directions: patterns defined on the sign of normal flow along particular orientation fields

• positive depth constraint

• 2 classes of orientation fields: copoint vectors and coaxis vectors

Optical flow and normal flow

Optical flow and normal flow

Coaxis vectorswith respect to axis (A,B,C)

Coaxis vectors

Translational coaxis vectors

Translational coaxis vectors

h passes through FOE and (Af/C, Bf/C), defined by 2 parameters

Rotational coaxis vectors

Rotational coaxis vectors

Rotational coaxis vectors

g passes through AOR and (Af/C, Bf/C), defined by 1 parameter

Combine translation and rotation

Positive + positive positive

Negative + negative negative

Positive + negative don’t know (depends on structure)

Coaxis patterntranslational

rotational

combined

-vectors: Translation

-vectors: Rotation

cossin22

rot

f

r

f

ru

-vectors: Translation and Rotation

UW

VW

alpha beta gamma

Three coaxis vector fields

Copoint vectors

O

copoint vectors

Copoint vectors

defined by point (r,s)

Translational copoint vectors

FOE

AOR

FOE

AOR

FOE

AOR

: Negative

: Positive

: Don't know

FOE

AOR

Translational copoint vectors

k passes through FOE and (r,s) defined by 1 parameter

(r, s)

FOE

Rotational copoint vectors

Rotational copoint vectors

l passes through AOR and (r,s), is defined by 2 parameters

(r, s)

AOR

(r, s)

FOE

(r, s)

AOR

FOE

(r, s)

AOR

translational component rotational component

(a) (b) (c)

Three coaxis vector fields

a,b,c : positive and negative vectors

c,d,e: Fitting of patterns

g: Separation of (coaxis patternh: Separation of (x0, y0) copoint pattern

FOE

AOR

FOE

AOR

FOE

AOR

: Negative

: Positive

: Don't know

FOE

AOR

Opticalillusion

What is the Problem?

• Flow can be accurately estimated in an image patch corresponding to a smooth scene patch,

• But erroneous flow estimates are obtained for image patches corresponding to scene patches containing discontinuities

Image Flow

3D MotionScene structureDiscontinuities

Depth variability constraint

• Errors in motion estimates lead to distortion of the scene estimates.

• The distortion is such that the correct motion gives the “smoothest” (least varying) scene structure.

• Scene depth can be estimated from normal flow measurements:

ntu

nu

)(ˆ

ωˆ1

tr

rotnu

Z

Depth estimation

nununu rottrn

1

Zu

Visual Space Distortion

• Wrong 3D motion gives rise to a rugged (unsmooth) depth function (surface).

• The correct 3D motion leads to the “smoothest” estimated depth.

nutu

ntu

ωδ

ˆ ,ˆ

rottr

trDDZZ

Inverse depth estimates

correct motion

incorrect motion

The error function

• A normal flow measurement:

For an estimate

• The error function to be minimized:

nunu rottrn

1

Zu

2

))ˆ()(ˆ1

()ˆ(

ii

ii n

ZnuW tuωu trrotin

nωuntu )ˆ()ˆ(1

rottrn Zu

ωt ˆ,ˆ

The error function• Estimated normal flow

• The error function to be minimized:

• Global parameters:• Local parameter: locally planar patches:

nωuntu )ˆ()ˆ(ˆ1

ˆ rottrnZ

u

2

nnˆ R i

i uuW

ωt ˆ,ˆZ

cbyaxZ

ˆ1

Error function evaluation

• Given a translation candidate , each local depth can be computed as a linear function of the rotation .

• We obtain a second order function of the rotation; its minimization provides both the rotation and the value of the error function.

t

ω

Is derived from image gradients only

Brightness consistency:

Flow:

Planar patch:

Handling depth discontinuities

• Given a candidate motion, the scene depth can be estimated and further processed to find depth discontinuities.

• Split a region if it corresponds to two depth values separated in space.

The algorithm

• Compute spatio-temporal image derivatives and normal flow.

• Find the direction of translation that minimizes the depth-variability criterion.– Hierarchical search of the 2D space.– Iterative minimization.– Utilize continuity of the solution in time;

usually the motion changes slowly over time.

Divide image into small patches

Search in the 2D space of translations

For each candidate 3D motion, using normal flow

measurements in each patch, compute depth of

the scene.

For each image patch

NO

Depth variation small?

Distinguish between two cases

wrong 3D motionexistence of a

discontinuity at the patch

Use the error Split the patch and repeat the

process

Use the error

YES

The Algorithm

3D reconstruction

• Comparison of the original sequence and the re-projection of the 3D reconstruction.

3D model construction