epipolar geometry - stanford...

Lecture 5 -Silvio Savarese 5-Feb-14

Professor Silvio Savarese

Computational Vision and Geometry Lab

Lecture 5Epipolar Geometry


• Why is stereo useful?• Epipolar constraints• Essential and fundamental matrix• Estimating F• Examples


Reading: [AZ] Chapter: 4 “Estimation – 2D perspective transformationsChapter: 9 “Epipolar Geometry and the Fundamental Matrix Transformation”Chapter: 11 “Computation of the Fundamental Matrix F”

[FP] Chapters: 10 “The geometry of multiple views”

Pinhole perspective projectionRecovering structure from a single view

C

Ow

Pp

Calibration rig

Scene

Camera K

From calibration rig

From points and lines at infinity

+ orthogonal lines and planes structure of the scene, K

location/pose of the rig, K

Knowledge about scene (point correspondences, geometry of lines & planes, etc…

Pinhole perspective projectionRecovering structure from a single view

C

Ow

Pp

Calibration rig

Scene

Camera K

Why is it so difficult?

Intrinsic ambiguity of the mapping from 3D to image (2D)

Recovering structure from a single view

Intrinsic ambiguity of the mapping from 3D to image (2D)

Courtesy slide S. Lazebnik

Two eyes help!

O2O1

x2

x1

?

Two eyes help!

This is called triangulation

K =knownK =known

R, T

llX X

l'l

• Find X that minimizes

),(),( 22

2

11

2XMxdXMxd

O1O2

x1

x2

X

Triangulation

Stereo-view geometry

• Correspondence: Given a point in one image, how can I find the corresponding point x’ in another one?

• Camera geometry: Given corresponding points in two images, find camera matrices, position and pose.

• Scene geometry: Find coordinates of 3D point from its projection into 2 or multiple images.

• Epipolar Plane • Epipoles e1, e2

• Epipolar Lines

• Baseline

Epipolar geometry

O1O2

x2

X

x1

e1 e2

= intersections of baseline with image planes

= projections of the other camera center

For details see CS131A

Lecture 9

Example: Converging image planes

e

e’

O1O2

X

e2

x1 x2

e1

Example: Parallel image planes

• Baseline intersects the image plane at infinity

• Epipoles are at infinity

• Epipolar lines are parallel to x axis

Example: Parallel Image Planes

e’ at

infinity

e at

infinity

Example: Forward translation

• The epipoles have same position in both images

• Epipole called FOE (focus of expansion)

O2

e1

e2

O2

- Two views of the same object

- Suppose I know the camera positions and camera matrices

- Given a point on left image, how can I find the corresponding point on right image?

Epipolar Constraint

Epipolar geometry

O1O2

X

- Two views of the same object

- Suppose I know the camera positions and camera matrices


Epipolar Constraint

1

v

u

PMP

0IKM

OO’

pp’

P

R, T

Epipolar Constraint

TRRKMTT

'

1

v

u

PMP

T

O

O’

T = O’ in the camera 1 reference systemR is such that a vector p’ in camera 2 is equal to R p’ in camera 1

The cameras are related by R, T

p’

Camera 1

Camera 2

0IKM

OO’

pp’

P

R, T

Epipolar Constraint

K is known

(canonical cameras)

0IM

100

010

001

K

TRRKMTT

'

TRRMTT

'

OO’

pp’

P

R, T

0)( pRTpT

Epipolar Constraint

is perpendicular to epipolar plane)( pRT

pR p’ in first camera reference system is

baba ][

0

0

0

z

y

x

xy

xz

yz

b

b

b

aa

aa

aa

Cross product as matrix multiplication

OO’

pp’

P

R, T

0)pR(TpT

0pRTpT

E = essential matrix(Longuet-Higgins, 1981)

Epipolar Constraint

• E p2 is the epipolar line associated with p2 (l1 = E p2)

• ET p1 is the epipolar line associated with p1 (l2 = ET p1)

• E e2 = 0 and ET e1 = 0

• E is 3x3 matrix; 5 DOF

• E is singular (rank two)

Epipolar Constraint

O1O2

p2

P

p1

e1e2

l1 l2

021 pEpT

Epipolar Constraint

OO’

pp’

P

0IKM K is unknown

R, T

TRRKMTT

'

OO’

pp’

P

Epipolar Constraint

0pFpT

F = Fundamental Matrix(Faugeras and Luong, 1992)

1

KRTKFT

Epipolar Constraint

O1O2

p2

P

p1

e1e2

• F p2 is the epipolar line associated with p2 (l1 = F p2)

• FT p1 is the epipolar line associated with x1 (l2 = FT p1)

• F e2 = 0 and FT e1 = 0

• F is 3x3 matrix; 7 DOF

• F is singular (rank two)

021 pFpT

Why F is useful?

- Suppose F is known

- No additional information about the scene and camera is given


l’ = FT xx

Why F is useful?

• F captures information about the epipolar geometry of 2 views + camera parameters

• MORE IMPORTANTLY: F gives constraints on how the scene changes under view point transformation (without reconstructing the scene!)

• Powerful tool in:• 3D reconstruction• Multi-view object/scene matching

OO’

pp’

P

Estimating F

The Eight-Point Algorithm

1

v

u

pP

1

v

u

pP 0pFpT

(Hartley, 1995)

(Longuet-Higgins, 1981)

0pFpT

Estimating F

Let’s take 8 corresponding points

11 12 13

21 22 23

31 32 33

'

, ,1 ' 0

1

F F F u

u v F F F v

F F F

11

12

13

21

22

23

31

32

33

', ', , ', ', , ', ',1 0

F

F

F

F

uu uv u vu vv v u v F

F

F

F

F

Estimating F

Estimating F

Lsq. solution by SVD!

• Rank 8 A non-zero solution exists (unique)

• Homogeneous system

• If N>8 F̂

f

1f

W 0f

W

1 1 1 1 1 1 1 1 1 1 1 1

2 2 2 2 2 2 2 2 2 2 2 2

3 3 3 3 3 3 3 3 3 3 3 3

4 4 4 4 4 4 4 4 4 4 4 4

5 5 5 5 5 5 5 5 5 5 5 5

6 6 6 6 6 6 6 6 6 6 6 6

7 7 7 7 7 7

' ' ' ' ' ' 1

' ' ' ' ' ' 1

' ' ' ' ' ' 1

' ' ' ' ' ' 1

' ' ' ' ' ' 1

' ' ' ' ' ' 1

' '

u u u v u v u v v v u v






u u u v u v

11

12

13

21

22

23

31

7 7 7 7 7 7

32

8 8 8 8 8 8 8 8 8 8 8 8

33

0

' ' ' ' 1

' ' ' ' ' ' 1

F

F

F

F

F

F

Fu v v v u v

Fu u u v u v u v v v u v

F

0pF̂pT

and estimated F may have full rank (det(F) ≠0)

0F̂F Find F that minimizes

Subject to det(F)=0

But remember: fundamental matrix is Rank2

Frobenius norm (*)

SVD (again!) can be used to solve this problem

^ ^

(*) Sqrt root of the sum of squares of all entries

satisfies:F̂

[HZ] pag 281, chapter 11, “Computation of F”

Data courtesy of R. Mohr and B. Boufama.

Mean errors:

10.0pixel

9.1pixel

Problems with the 8-Point Algorithm

- Recall the structure of W:- do we see any potential

(numerical) issue?

F1f

,0fWLsq solution

by SVD

Problems with the 8-Point Algorithm

• Highly un-balanced (not well conditioned)

• Values of W must have similar magnitude

• This creates problems during the SVD decomposition

f

W

1 1 1 1 1 1 1 1 1 1 1 1

2 2 2 2 2 2 2 2 2 2 2 2

3 3 3 3 3 3 3 3 3 3 3 3

4 4 4 4 4 4 4 4 4 4 4 4

5 5 5 5 5 5 5 5 5 5 5 5

6 6 6 6 6 6 6 6 6 6 6 6

7 7 7 7 7 7

' ' ' ' ' ' 1

' ' ' ' ' ' 1

' ' ' ' ' ' 1

' ' ' ' ' ' 1

' ' ' ' ' ' 1

' ' ' ' ' ' 1

' '







u u u v u v

11

12

13

21

22

23

31

7 7 7 7 7 7

32

8 8 8 8 8 8 8 8 8 8 8 8

33

0

' ' ' ' 1

' ' ' ' ' ' 1

F

F

F

F

F

F

Fu v v v u v

Fu u u v u v u v v v u v

F

0Wf

HZ

pag

10

8

Normalization

IDEA: Transform image coordinate

such that the matrix W become better

conditioned (pre-conditioning)

Apply following transformation T:

(translation and scaling)

• Origin = centroid of image points

• Mean square distance of the data

points from origin is 2 pixels

ii pTq ii pTq (normalization)

1. Normalize coordinates:

2. Use the eight-point algorithm to compute F’q from the

points q and q’ .

1. Enforce the rank-2 constraint.

2. De-normalize Fq:

i i

ii pTq ii pTq

0qFq q

T

The Normalized Eight-Point Algorithm

qF

'TFTF q

T

0)Fdet( q

0. Compute T and T’

With

tra

nsfo

rma

tio

nW

ith

ou

t tr

an

sfo

rmatio

n Mean errors:

10.0pixel

9.1pixel

Mean errors:

1.0pixel

0.9pixel

Same issue for the DLT algorithm

ixix

H

ii xHx [Section 4.4 in AZ]


• Why is stereo useful?• Epipolar constraints• Essential and fundamental matrix• Estimating F• Examples


Reading: [AZ] Chapter: 4 “Estimation – 2D perspective transformationsChapter: 9 “Epipolar Geometry and the Fundamental Matrix Transformation”Chapter: 11 “Computation of the Fundamental Matrix F”

[FP] Chapters: 10 “The geometry of multiple views”

O1O2

X

e2

x1 x2

e1


E=?

K K

K1=K2 = known

Hint :

x

y

z

x parallel to O1O2

R = I t = (T, 0, 0)

00

00

000

0

0

0

T

T

TT

TT

TT

xy

xz

yz

IE

Essential matrix for parallel images

T = [ T 0 0 ]

R = I

RTE

O1

X

e2

x1 x2

e1


K K

x

y

z

horizontal!What are the

directions of

epipolar lines?

2

3221

0

00

00

000

E

xT

xT

T

Tl xx

O1

X

e2

x1 x2

e1


K K

x

y

z

How are

x1 and x2

related?021 xEx

T

O1

X

e2

x1 x2

e1


K K

x

y

z

T

0 0 0 0

p E p 0 1 0 0 0 1 0

0 0 1

u

u v T v u v T Tv Tv

T Tv

How are

x1 and x2

related?

O1O2

X

e2

x1 x2

e1


K K

x

y

z

Rectification: making two images “parallel”

Why it is useful? • Epipolar constraint y = y’• New views can be synthesized by linear interpolation

For details on how to rectify

two views see CS131A

Lecture 9

Next lecture:

Stereo systems and affine SFM

epipolar geometry - stanford...

Documents