geometry for computer vision - linköping university · may 20, 2014 computer vision lecture 4a...

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Geometry for Computer VisionLecture 4a

Calibration and OrientedEpipolar Geometry

1

Per-Erik Forssén

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Overview1. Lens effects (distortion, vignetting)2. Extrinsic and intrinsic camera parameters3. Zhang’s camera calibration4. Calibrated epipolar geometry (intro)5. Oriented epipolar geometry

Break

2

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

The pin-hole camera

A brightly illuminated scene will be projected onto a wall opposite of the pin-hole.

The image is rotated .

3

180�

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

The pin-hole camera

• From similar triangles we get:

4

x = fX

Zy = f

Y

Z

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

The pin-hole camera

• From similar triangles we get:

5

�

�

⇤xy1

⇥

⌅ =

�

⇤f 0 00 f 00 0 1

⇥

⌅

�

⇤XYZ

⇥

⌅

• More generally, we write:

• f-focal length, s-skew, a-aspect ratio,(cx,cy)-projection of optical centre

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

The pin-hole camera

6

�

�

⇤xy1

⇥

⌅ =

�

⇤f 0 00 f 00 0 1

⇥

⌅

�

⇤XYZ

⇥

⌅

�

�

⇤xy1

⇥

⌅ =

�

⇤f s cx

0 fa cy

0 0 1

⇥

⌅

�

⇤XYZ

⇥

⌅

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

�

�

⇤xy1

⇥

⌅

⌥⌃⇧�x

=

�

⇤f s cx

0 fa cy

0 0 1

⇥

⌅

⌥ ⌃⇧ �K

�

⇤XYZ

⇥

⌅

⌥ ⌃⇧ �X̃

The pin-hole camera

x � KX̃

7

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

�

�

⇤xy1

⇥

⌅

⌥⌃⇧�x

=

�

⇤f s cx

0 fa cy

0 0 1

⇥

⌅

⌥ ⌃⇧ �K

�

⇤XYZ

⇥

⌅

⌥ ⌃⇧ �X̃

The pin-hole camera

Motivation:

f-focal length, s-skew, a-aspect ratio,(cx,cy)-projection of optical centre

x � KX̃

8

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Thin Lens CameraReal cameras use lenses, not pin-holes!

9

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Thin Lens CameraA thin lens is a (positive) lens with

Parallel rays converge at the focal pointsRays through the optical centre are not

refracted

d << f

10

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Thin Lens Camera

Thin lens relation (from similar triangles):1f

=1Z

+1l

11

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Thin Lens Camera

- Focus at one depth only. - Objects at other depths are blurred.

12

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Thin Lens Camera

An aperture increases the depth-of-field, the range which is sharp in the image.

A compromise between pinhole and thin lens.13

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Lens effects

• Radial distortion• For zoom lenses: Barrel at wide FoV

pin-cushion at narrow FoV

Barrel distortionCorrect Pin-cushion distortion

14

)

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Lens effects

• Modelling• Used in optimisation such as BA

DistortedCorrect image

15

x ⇠ Kf(u,⇥0)

)

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Lens effects

• Rectification• Used in dense stereo

Distorted image Correct

16

x

0 ⇠ f�1(x,⇥)

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Distorsion polynomials• Different models for different classes

of cameras• Radial model for normal and telecentric

lenses with moderate distortion

• Also model centre of distortion17

r =qx

21 + x

22 ' = atan2(x2, x1)

x

01 = r

0cos'

0x

02 = r

0sin'

0r0 = ✓1r + ✓2r3 . . . '0 = '

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Distorsion polynomials• For better accuracy:• Tangential distorsion• Rational model [Claus & Fitzgibbon 05]

• Specialised models:• wide-angle cameras [Kannala&Brandt 06]• catadioptric cameras [Micusik&Pajdla 03]• most simple is the FoV model [Devernay &

Faugeras 2001]:

18

r0 = atan(r✓1)/✓1

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Lens effects

• Vignetting and cos4-law• Stronger effects in wide FoV

Darkened peripheryCorrect

19

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Lens effectsVignetting

20

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Lens effectsVignetting

cos4-lawdampening withcos4(w)

http://software.canon-europe.com/files/documents/EF_Lens_Work_Book_10_EN.pdf

21

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

For a general position of the world coordinate system (WCS) we have:

x � K

�

⇤r11 r12 r13 t1r21 r22 r23 t2r31 r32 r33 t3

⇥

⌅

�⌥ ⌦[R|t]

�

⇧⇧⇤

XYZ1

⇥

⌃⌃⌅

�⌥ ⌦X

22

Camera parameters

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

For a general position of the world coordinate system (WCS) we have:

K contains the intrinsic parameters[R | t] contain the extrinsic parameters

x � K

�

⇤r11 r12 r13 t1r21 r22 r23 t2r31 r32 r33 t3

⇥

⌅

�⌥ ⌦[R|t]

�

⇧⇧⇤

XYZ1

⇥

⌃⌃⌅

�⌥ ⌦X

23

Camera parameters

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera parametersMetric points transformed to the camera’s

coordinate system are called normalised image coordinates

In contrast to regular image coordinates

K contains the intrinsic parameters[R | t] contain the extrinsic parameters

24

x ⇠ K [R|t]X

x̂ ⇠ [R|t]X

x = Kx̂

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibrationZhang’s camera calibration (A flexible new

technique for camera calibration, PAMI 2000)

In OpenCV, and in Matlab toolboxFinds K from 3 or more photos of a planar

calibration targetModerate lens

distorsion can alsobe estimated.

25

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibrationWe now imagine a world coordinate

system fixed to the planar target

�

�

⇤xy1

⇥

⌅ = K

�

⇤r11 r12 r13 t1r21 r22 r23 t2r31 r32 r33 t3

⇥

⌅

�

⇧⇧⇤

XYZ1

⇥

⌃⌃⌅

26

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibrationWe now imagine a world coordinate

system fixed to the planar target

�

�

⇤xy1

⇥

⌅ = K

�

⇤r11 r12 t1r21 r22 t2r31 r32 t3

⇥

⌅

�

⇤XY1

⇥

⌅ =

�

⇤h11 h12 h13

h21 h22 h23

h31 h32 h33

⇥

⌅

⌥ ⌃⇧ �H

�

⇤XY1

⇥

⌅

27

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

If we estimate a homography between the image and the model plane (lecture 3) we know H

We also know that and

28

rT1 r2 = 0 rT

1 r1 = rT2 r2

H = [h1 h2 h3] = K [r1 r2 t]

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

If we estimate a homography between the image and the model plane (lecture 3) we know H

We also know that and

29

rT1 r2 = 0 rT

1 r1 = rT2 r2

H = [h1 h2 h3] = K [r1 r2 t]

hT1 K�T K�1h2 = 0

hT1 K�T K�1h1 = hT

2 K�T K�1h2

)

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

For a K of the formIt can be shown that (use e.g. Maple)

30

K�T K�1 = B =

2

64

1↵2 � �

↵2�v0��u0�

↵2�

� �↵2�

�2

↵2�2 + 1�2 ��(v0��u0�)

↵2�2 � v0�2

v0��u0�↵2� ��(v0��u0�)

↵2�2 � v0�2

(v0��u0�)2

↵2�2 + v20

�2 + 1

3

75

K =

2

4↵ � u0

0 � v0

0 0 1

3

5

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

For a K of the formIt can be shown that (use e.g. Maple)

Remember our constraints and

31

K�T K�1 = B =

2

64

1↵2 � �

↵2�v0��u0�

↵2�

� �↵2�

�2

↵2�2 + 1�2 ��(v0��u0�)

↵2�2 � v0�2

v0��u0�↵2� ��(v0��u0�)

↵2�2 � v0�2

(v0��u0�)2

↵2�2 + v20

�2 + 1

3

75

K =

2

4↵ � u0

0 � v0

0 0 1

3

5

hT1 Bh2 = 0 hT

1 Bh1 � hT2 Bh2 = 0

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

As B is symmetric

32

B =

2

4b1 b2 b4

b2 b3 b5

b4 b5 b6

3

5

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

As B is symmetric

If we now defineThe constraints can be written as

33

B =

2

4b1 b2 b4

b2 b3 b5

b4 b5 b6

3

5

b =⇥b1 b2 b3 b4 b5 b6

⇤T

vT

12

(v11 � v22)T

�b = 0

vij = [hi1hj1, hi1hj2 + hi2hj1, hi2hj2, hi3hj1 + hi1hj3, hi3hj2 + hi2hj3, hi3hj3]T

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

Each view of the plane gives us two rows in the system:

As b has 6 unknowns, we need 3 views of the plane.

Two views can also work if we require

34

Vb = 0

� = 0

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibrationOnce b has been estimated, we can extract the

parameters in K according to

35

v0 = (b2b4 � b1b5)/(b1b3 � b22)

� = b6 � (b23 + v0(b2b4 � b1b5)/b1

� = �b2↵2�/�

u0 = �v0↵� b4↵2/�

↵ =p

�/b1

� =q

�b1/(b1b3 � b22)

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibrationOnce b has been estimated, we can extract the

parameters in K according to

The H&Z book instead suggests Cholesky factorisation36

v0 = (b2b4 � b1b5)/(b1b3 � b22)

� = b6 � (b23 + v0(b2b4 � b1b5)/b1

� = �b2↵2�/�

u0 = �v0↵� b4↵2/�

↵ =p

�/b1

� =q

�b1/(b1b3 � b22)

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibrationCholesky factorisation of B(b)

Gives us which is invertible.

37

B(b) = K�1TK�1

K�1

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

Once K is computed we can also find the extrinsic camera parameters R,t for each image:

( )

38

r1 = �K�1h1 r2 = �K�1h2 r3 = r1 ⇥ r2

R = [r1 r2 r3] t = �K�1h3

� = 1/||K�1h1|| = 1/||K�1h2||

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

Once K is computed we can also find the extrinsic camera parameters R,t for each image:

Finally, K, Ri, ti are refined using ML (minimising the cost function)

39

r1 = �K�1h1 r2 = �K�1h2 r3 = r1 ⇥ r2

R = [r1 r2 r3] t = �K�1h3

arg minnX

i=1

mX

j=1

||xij � x̂(K,Ri, ti,Xj)||2

𝚯

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

Once K is computed we can also find the extrinsic camera parameters R,t for each image:

Optionally, all of K, ,Ri, ti are refined using ML:

40

r1 = �K�1h1 r2 = �K�1h2 r3 = r1 ⇥ r2

R = [r1 r2 r3] t = �K�1h3

argminnX

i=1

mX

j=1

kxij � x̂(K,⇥,Ri, ti,Xj)k2

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

So what about the initial homographies?

Assign each point a WCS value

41

H = K [r1 r2 t]X = [x y 0]T

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibration

So what about the initial homographies?

Assign each point a WCS value Do we need to know which point is the upper left one on the checker-board? Why not?

42

H = K [r1 r2 t]X = [x y 0]T

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibrationCan we use any combination images of the calibration plane?

43

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Camera calibrationCan we use any combination images of the calibration plane?

The constraints used: andhave to be linearly independent.

Planes must not be parallel!

44

rT1 r2 = 0 rT

1 r1 = rT2 r2

)

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Calibrated epipolar geometry

Recall the epipolar constraint

45

x

T1 Fx2 = 0

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Calibrated epipolar geometry

Recall the epipolar constraint...and the normalised image coordinates

We can instead express the epipolar constraint in normalised coordinates

The matrix E is called the essential matrix.It has some interesting properties...

46

x

T1 Fx2 = 0

x = Kx̂

ˆ

x

T1 K

T1 FK2ˆ

x2 = 0 or

ˆ

x

T1 E

ˆ

x2 = 0

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Calibrated epipolar geometryIn lecture 2 we saw that for cameras P1 and P2:

Now, if

We get and

47

P+2 =

K�1

2

0T

�

F = [e12]⇥P1P+2

P2 = K2 [I|0] and P1 = K1 [R|t]

F = [K1t]⇥K1RK�12

e12 = P1O2

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Calibrated epipolar geometryUsing the cross-product-commutator rule:

(A4.3)

on

...we may express F as either of

48

F = K�T1 [t]⇥RK�1

2 F = K�T1 R

⇥RT t

⇤⇥K�1

2

F = K�T1 R [t2]⇥K�1

2

[b]⇥A = det(A)A�T⇥A�1b

⇤⇥

F = [K1t]⇥K1RK�12

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Calibrated epipolar geometryThis gives us the essential matrix

expressions:

E has only 5 dof (3 from R, 2 from t)recall that F has 7

A necessary and sufficient condition on E is that it has the singular values [a,a,0](see 9.6.1 in the H&Z book for proof)

49

E = [t]⇥R = R⇥RT t

⇤⇥

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Calibrated epipolar geometryThis gives us the essential matrix

expressions:

We can extract R and t (up to scale) from Eif we also make use of one point correspondence (a 3D point known to be in front of both cameras). See 9.6.2 in the H&Z book.

50

E = [t]⇥R = R⇥RT t

⇤⇥

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Calibrated epipolar geometry4 cases for R and t, just one has point in front

of both cameras.

51

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Oriented epipolar geometryThe regular epipolar constraint

ignores the knowledge that points are in frontof the camera.

52

x

T1 Fx2 = 0

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Oriented epipolar geometryIn oriented projective geometry a (visible)

point in front of the camera is defined as having a projection

and a (hidden) point behind the camera has a projection

53

x = �

⇥x1 x2 1

⇤Twith � > 0

x = �

⇥x1 x2 1

⇤Twith � < 0

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Oriented epipolar geometryThe oriented epipolar constraint properly

distinguishes points in front of and behind the camera

54

�e1 ⇥ x1 = Fx2 , � 2 R+

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Oriented epipolar geometryThe oriented epipolar constraint can be

interpreted as comparing oriented lines and

(NB! image planes drawn in front of cameras)55

�e1 ⇥ x1 Fx2

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Oriented epipolar geometryLine normalisation is not unique

The extra information in the sign can be used to encode the line orientation.

56

normD(l) =⇥cos↵ sin↵ �⇢

⇤T

normD(�l) =⇥� cos↵ � sin↵ ⇢

⇤T

May 20, 2014 Computer Vision lecture 4a

Computer Vision Laboratory

Oriented epipolar geometryUsage:The oriented epipolar constraint can be used

to quickly reject a hypothesized F inside a RANSAC loop.

See today’s paper: Chum, Werner and Matas, Epipolar Geometry Estimation via RANSAC benefits from the Oriented Epipolar Constraint, ICPR04

57