epipolar geometry class 5. 3d photography course schedule (tentative) lectureexercise sept...

Epipolar geometryClass 5

3D photography course schedule(tentative)

Lecture Exercise

Sept 26 Introduction -

Oct. 3 Geometry & Camera model Camera calibration

Oct. 10 Single View Metrology Measuring in images

Oct. 17 Feature Tracking/matching(Friedrich Fraundorfer)

Correspondence computation

Oct. 24 Epipolar Geometry F-matrix computation

Oct. 31 Shape-from-Silhouettes(Li Guan)

Visual-hull computation

Nov. 7 Stereo matching Project proposals

Nov. 14 Structured light and active range sensing

Papers

Nov. 21 Structure from motion Papers

Nov. 28 Multi-view geometry and self-calibration

Papers

Dec. 5 Shape-from-X Papers

Dec. 12 3D modeling and registration Papers

Dec. 19 Appearance modeling and image-based rendering

Final project presentations

• Brightness constancy assumption

Optical flow

(small motion)

• 1D example

possibility for iterative refinement

• Brightness constancy assumption

Optical flow

(small motion)

• 2D example

(2 unknowns)

(1 constraint)?

isophote I(t)=Iisophote I(t+1)=I

the “aperture” problem

Optical flow• How to deal with aperture problem?

Assume neighbors have same displacement

(3 constraints if color gradients are different)

Lucas-Kanade

Assume neighbors have same displacement

least-squares:

Revisiting the small motion assumption

• Is this motion small enough?• Probably not—it’s much larger than one pixel

(2nd order terms dominate)• How might we solve this problem?

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

Reduce the resolution!

* From Khurram Hassan-Shafique CAP5415 Computer Vision 2003

image It-1 image I

Gaussian pyramid of image It-1 Gaussian pyramid of image I

image Iimage It-1u=10 pixels

u=5 pixels

u=2.5 pixels

u=1.25 pixels

Coarse-to-fine optical flow estimation

slides fromBradsky and Thrun

image Iimage J

Gaussian pyramid of image It-1 Gaussian pyramid of image I

image Iimage It-1

Coarse-to-fine optical flow estimation

run iterative L-K

run iterative L-K

warp & upsample

.

.

.

slides fromBradsky and Thrun

Feature tracking

• Identify features and track them over video• Small difference between frames• potential large difference overall

• Standard approach: KLT (Kanade-Lukas-Tomasi)

Good features to track

• Use same window in feature selection as for tracking itself

• Compute motion assuming it is small

Affine is also possible, but a bit harder (6x6 in stead of 2x2)

differentiate:

Example

Simple displacement is sufficient between consecutive frames, but not to compare to reference template

Example

Synthetic example

Good features to keep tracking

Perform affine alignment between first and last frameStop tracking features with too large errors

(i) Correspondence geometry: Given an image point x in the first image, how does this constrain the position of the

corresponding point x’ in the second image?

(ii) Camera geometry (motion): Given a set of corresponding image points {xi ↔x’i}, i=1,…,n, what are the cameras P and P’ for the two views?

(iii) Scene geometry (structure): Given corresponding image points xi ↔x’i and cameras P, P’, what is the position of (their pre-image) X in space?

Three questions:

Two-view geometry

The epipolar geometry

C,C’,x,x’ and X are coplanar


What if only C,C’,x are known?


All points on project on l and l’


Family of planes and lines l and l’ Intersection in e and e’


epipoles e,e’= intersection of baseline with image plane = projection of projection center in other image= vanishing point of camera motion direction

an epipolar plane = plane containing baseline (1-D family)

an epipolar line = intersection of epipolar plane with image(always come in corresponding pairs)

Example: converging cameras

Example: motion parallel with image plane

(simple for stereo rectification)

Example: forward motion

e

e’

The fundamental matrix F

algebraic representation of epipolar geometry

l'x

we will see that mapping is (singular) correlation (i.e. projective mapping from points to lines) represented by the fundamental matrix F


geometric derivation

xHx' π

x'e'l' FxxHe' π

mapping from 2-D to 1-D family (rank 2)


algebraic derivation

λCxPλX IPP

PP'e'F

xPP'CP'l'

(note: doesn’t work for C=C’ F=0)

xP

λX


correspondence condition

0Fxx'T

The fundamental matrix satisfies the condition that for any pair of corresponding points x↔x’ in the two images 0l'x'T


F is the unique 3x3 rank 2 matrix that satisfies x’TFx=0 for all x↔x’

(i) Transpose: if F is fundamental matrix for (P,P’), then FT is fundamental matrix for (P’,P)

(ii) Epipolar lines: l’=Fx & l=FTx’(iii) Epipoles: on all epipolar lines, thus e’TFx=0, x

e’TF=0, similarly Fe=0(iv) F has 7 d.o.f. , i.e. 3x3-1(homogeneous)-1(rank2)(v) F is a correlation, projective mapping from a point x to

a line l’=Fx (not a proper correlation, i.e. not invertible)

Fundamental matrix for pure translation

Fundamental matrix for pure translation

PP'e'F

0]|K[IP t]|K[IP'

0KP

-1

00

0e'F

xy

xz

yz

eeeeee

General motion

Pure translation

for pure translation F only has 2 degrees of freedom


relation to homographies

lHl' -T

π FHe'

π

valid for all plane homographies

eHe'π


relation to homographies

FxlxH'xππ

requires

πl

πx

x x

Fe'H e.g. 0e'e'T 0e'lT

π

Projective transformation and invariance

-1-T FHH'F̂ x'H''x̂ Hx,x̂

Derivation based purely on projective concepts

X̂P̂XHPHPXx -1

F invariant to transformations of projective 3-space

X̂'P̂XHHP'XP'x' -1

FP'P,

P'P,F

unique

not unique

canonical form

m]|[MP'0]|[IP

MmF

PP'e'F

Projective ambiguity of cameras given Fprevious slide: at least projective ambiguitythis slide: not more!

Show that if F is same for (P,P’) and (P,P’), there exists a projective transformation H so that P=HP and P’=HP’

~ ~

~ ~

]a~|A~

['P~

0]|[IP~

a]|[AP' 0]|[IP

A

~a~AaF

T1 avAA~

kaa~ kandlemma:

kaa~Fa~0AaaaF2rank

TavA-A~

k0A-A~

kaA~

a~Aa

kkIkT1

1

v0H

'P~

]a|av-A[

v0a]|[AHP'

T1

T1

1

kk

kkIk

(22-15=7, ok)

The projective reconstruction theorem

If a set of point correspondences in two views determine the fundamental matrix uniquely, then the scene and cameras may be reconstructed from these correspondences alone, and any two such reconstructions from these correspondences are projectively equivalent

allows reconstruction from pair of uncalibrated images!

C1

C2

l2

l1

e1

e20m m 1

T2 F

Fundamental matrix (3x3 rank 2

matrix)

1. Computable from corresponding points

2. Simplifies matching3. Allows to detect wrong

matches4. Related to calibration

Underlying structure in set of matches for rigid scenes

l2

C1m1

L1

m2

L2

M

C2

m1

m2

C1

C2

l2

l1

e1

e2

m1

L1

m2

L2

M

l2lT1

Epipolar geometry

Canonical representation:

]λe'|ve'F][[e'P' 0]|[IP T

Epipolar geometry?

courtesy Frank Dellaert

Other entities besides points?

Lines give no constraint for two view geometry(but will for three and more views)

Curves and surfaces yield some constraints related to tangency

(e.g. Sinha et al. CVPR’04)

Computation of F

• Linear (8-point)• Minimal (7-point)• Robust (RANSAC)• Non-linear refinement (MLE, …)

• Practical approach

Epipolar geometry: basic equation

0Fxx'T

separate known from unknown

0'''''' 333231232221131211 fyfxffyyfyxfyfxyfxxfx

0,,,,,,,,1,,,',',',',',' T333231232221131211 fffffffffyxyyyxyxyxxx

(data) (unknowns)(linear)

0Af

0f1''''''

1'''''' 111111111111

nnnnnnnnnnnn yxyyyxyxyxxx

yxyyyxyxyxxx

0

1´´´´´´

1´´´´´´

1´´´´´´

33

32

31

23

22

21

13

12

11

222222222222

111111111111

f

f

f

f

f

f

f

f

f

yxyyyyxxxyxx

yxyyyyxxxyxx

yxyyyyxxxyxx

nnnnnnnnnnnn

~10000 ~10000 ~10000 ~10000~100 ~100 1~100 ~100

!Orders of magnitude differencebetween column of data matrix least-squares yields poor results

the NOT normalized 8-point algorithm

Transform image to ~[-1,1]x[-1,1]

(0,0)

(700,500)

(700,0)

(0,500)

(1,-1)

(0,0)

(1,1)(-1,1)

(-1,-1)

1

1500

2

10700

2

normalized least squares yields good results (Hartley, PAMI´97)

the normalized 8-point algorithm

the singularity constraint

0Fe'T 0Fe 0detF 2Frank

T333

T222

T111

T

3

2

1

VσUVσUVσUVσ

σσ

UF

SVD from linearly computed F matrix (rank 3)

T222

T111

T2

1

VσUVσUV0

σσ

UF'

FF'-FminCompute closest rank-2 approximation

the minimum case – 7 point correspondences

0f1''''''

1''''''

777777777777

111111111111

yxyyyxyxyxxx

yxyyyxyxyxxx

T9x9717x7 V0,0,σ,...,σdiagUA

9x298 0]VA[V T8

T ] 000000010[Ve.g.V

1...70,)xλFF(x 21T iii

one parameter family of solutions

but F1+F2 not automatically rank 2

F1 F2

F

3

F7pts

0λλλ)λFFdet( 012

23

321 aaaa

(obtain 1 or 3 solutions)

(cubic equation)

0)λIFFdet(Fdet)λFFdet( 1-12221

the minimum case – impose rank 2

Compute possible as eigenvalues of (only real solutions are potential solutions)

1-12 FF

B.detAdetABdet

Step 1. Extract featuresStep 2. Compute a set of potential matchesStep 3. do

Step 3.1 select minimal sample (i.e. 7 matches)

Step 3.2 compute solution(s) for F

Step 3.3 determine inliers

until (#inliers,#samples)<95%

samples#7)1(1

matches#inliers#

#inliers 90%

80%

70% 60%

50%

#samples

5 13 35 106 382

Step 4. Compute F based on all inliersStep 5. Look for additional matchesStep 6. Refine F based on all correct matches

(generate hypothesis)

(verify hypothesis)

Automatic computation of F

RANSAC

restrict search range to neighborhood of epipolar line (e.g. 1.5 pixels)

relax disparity restriction (along epipolar line)

Finding more matches

• (Mostly) planar scene (see next slide)• Absence of sufficient features (no texture)• Repeated structure ambiguity

(Schaffalitzky and Zisserman, BMVC‘98)

• Robust matcher also finds Robust matcher also finds support for wrong hypothesissupport for wrong hypothesis• solution: detect repetition solution: detect repetition

Issues:

Computing F for quasi-planar scenes QDEGSAC

17% success for RANSAC

100% for QDEGSAC #i

nlie

rsdata rank

337 matches on plane, 11 off plane

%inclusion of out-of-plane inliers

geometric relations between two views is fully

described by recovered 3x3 matrix F

two-view geometry

epipolar geometry class 5. 3d photography course schedule (tentative) lectureexercise sept...

Documents

thrun slide

different slide

tomasi slide

view geometry slide

image point x

preimage x

aperture problem slide

iterative refinement