camera models and calibration - cvg @ ethz...# date topic 1 sep.19 introduction and geometry 2...

Camera models and calibration

Read tutorial chapter 2 and 3.1

http://www.cs.unc.edu/~marc/tutorial/

Szeliski’s book pp.29-73

Schedule (tentative)

2

# date topic 1 Sep.19 Introduction and geometry 2 Sep.26 Camera models and calibration 3 Oct.3 Multiple-view geometry (Richard Hartley) 4 Oct.10 Invariant features 5 Oct.7 Model fitting (RANSAC, EM, …) 6 Oct.24 Segmentation (Jianbo Shi)

7 Oct.31 Stereo Matching 8 Nov.7 Shape from Silhouettes 9 Nov.14 Optical Flow 10 Nov.21 Structure from motion 11 Nov.28 Tracking (Kalman, particle filter) 12 Dec.5 Object recognition 13 Dec.12 Object category recognition 14 Dec.19 Applications and demos

Brief geometry reminder

3

0=xlTl'×l=x x'×x=l

2D line-point coincidence relation: Point from lines:

2D Ideal points ( )T0,, 21 xx2D line at infinity ( )T1,0,0=l∞

0=XπT

0πXXX

3

2

1

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

T

T

T

0Xπππ

3

2

1

=⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

T

T

T3D plane-point coincidence relation: Point from planes: Plane from points:

Line from points:

3D line representation: (as two planes or two points) [ ] 2x2

PA B 0

Q⎡ ⎤

=⎢ ⎥⎣ ⎦

T

T

3D Ideal points ( )1 2 3, , ,0X X X T

3D plane at infinity ( )0,0,0,1∞Π =T

Conics and quadrics

4

lTC* l = 0xTCx = 0

l=Cx

l

x C

1-* CC =

0=QXXT 0=πQπ *T

Conics

Quadrics

Homographies

5

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

3

2

1

333231

232221

131211

3

2

1

'''

xxx

hhhhhhhhh

xxx

Transformation for lines

l=l' -THTransformation for conics

-1-TCHHC ='Transformation for dual conics

THHCC ** ='

x=x' HFor a point transformation

Fixed points and lines

e=e λH

(eigenvectors H =fixed points)

l lλ=H-T

(eigenvectors H-T =fixed lines)

(λ1=λ2 ⇒ pointwise fixed line)

Hierarchy of 2D transformations

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

1002221

1211

y

x

taataa

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

1002221

1211

y

x

tsrsrtsrsr

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

333231

232221

131211

hhhhhhhhh

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

1002221

1211

y

x

trrtrr

Projective 8dof

Affine 6dof

Similarity 4dof

Euclidean 3dof

Concurrency, collinearity, order of contact (intersection, tangency, inflection, etc.), cross ratio

Parallellism, ratio of areas, ratio of lengths on parallel lines (e.g midpoints), linear combinations of vectors (centroids). The line at infinity l∞

Ratios of lengths, angles. The circular points I,J

lengths, areas.

invariants transformed squares

The line at infinity

00

l l 0 lt 1

1A

−−

∞ ∞ ∞−

⎛ ⎞⎡ ⎤ ⎜ ⎟ʹ′ = = =⎢ ⎥ ⎜ ⎟−⎣ ⎦ ⎜ ⎟

⎝ ⎠

AH

A

TT

T T

The line at infinity l∞ is a fixed line under a projective transformation H if and only if H is an affinity

Note: not fixed pointwise

Affine properties from images

projection rectification

APA

lllHH⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

321

010001

[ ] 0≠,l 3321∞ llll T=

Affine rectification v1 v2

l1

l2 l4

l3

l∞

21∞ vvl ×=

211 llv ×=

432 l×l=v

The circular points

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

=

0

1I i

⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

−=

0

1J i

I0

1

0

1

100cossinsincos

II =⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

−==ʹ′ iseitsstss

iy

x

Sθθθ

θθ

H

The circular points I, J are fixed points under the projective transformation H iff H is a similarity

The circular points

“circular points”

0=++++ 233231

22

21 fxxexxdxxx 0=+ 2

221 xx

l∞

( )( )T

T

0,-,1J

0,,1I

i

i

=

=

( ) ( )TT 0,1,0+0,0,1=I i

Algebraically, encodes orthogonal directions

03 =x

Conic dual to the circular points

TT JIIJ*∞ +=C

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

000010001

*∞C

TSS HCHC *

∞*∞ =

The dual conic is fixed conic under the projective transformation H iff H is a similarity

*∞C

Note: has 4DOF l∞ is the nullvector

*∞C

l∞

I

J

Angles

( )( )2221

22

21

2211

+++

=cosmmll

mlmlθ

( )T321 ,,=l lll ( )T321 ,,=m mmm

Euclidean:

Projective: ( )( )mmll

mlcos**

*

∞∞

∞=CC

CTT

T

θ

0=ml *∞C

T

(orthogonal)

Transformation of 3D points, planes and quadrics

Transformation for planes

( )l=l' -TH

Transformation for quadrics ( )-1-TCHHC ='

Transformation for dual quadrics

( )THHCC ** ='

( )x=x' HFor a point transformation

X=X' H

π=π' -TH

-1-TQHH=Q'

THHQ=Q' **

(cfr. 2D equivalent)

Hierarchy of 3D transformations

⎥⎦

⎤⎢⎣

⎡

vTvtAProjective

15dof

Affine 12dof

Similarity 7dof

Euclidean 6dof

Intersection and tangency

Parallellism of planes, Volume ratios, centroids, The plane at infinity π∞

Angles, ratios of length The absolute conic Ω∞

Volume

⎥⎦

⎤⎢⎣

⎡

10tA

T

⎥⎦

⎤⎢⎣

⎡

10tR

T

s

⎥⎦

⎤⎢⎣

⎡

10tR

T

The plane at infinity

000

π π π0- t 11

A

−−

∞ ∞ ∞−

⎛ ⎞⎜ ⎟⎡ ⎤ ⎜ ⎟ʹ′ = = =⎢ ⎥ ⎜ ⎟⎣ ⎦⎜ ⎟⎝ ⎠

AH

A

TT

T T

The plane at infinity π∞ is a fixed plane under a projective transformation H iff H is an affinity

1.  canonical position 2.  contains directions 3.  two planes are parallel ⇔ line of intersection in π∞ 4.  line // line (or plane) ⇔ point of intersection in π∞

( )T1,0,0,0=π∞ ( )T0,,,=D 321 XXX

The absolute conic

The absolute conic Ω∞ is a fixed conic under the projective transformation H iff H is a similarity

04

23

22

21 =

⎭⎬⎫++

XXXX

The absolute conic Ω∞ is a (point) conic on π∞. In a metric frame:

( ) ( )T321321 ,,I,, XXXXXXor conic for directions: (with no real points)

1.  Ω∞ is only fixed as a set 2.  Circle intersect Ω∞ in two circular points 3.  Spheres intersect π∞ in Ω∞

The absolute dual quadric

⎥⎦

⎤⎢⎣

⎡=Ω

000I*

∞ T

The absolute dual quadric Ω*∞ is a fixed conic under

the projective transformation H iff H is a similarity

1.  8 dof (to describe location in projective space) 2.  plane at infinity π∞ is the nullvector of Ω∞ 3.  Angles:

( )( )2*∞21

*∞1

2*∞1

πΩππΩππΩπ

=cosTT

T

θ

Camera model

Relation between pixels and rays in space

?

Pinhole camera

Gemma Frisius, 1544

22

Distant objects appear smaller

23

Parallel lines meet

•  vanishing point

24

Vanishing points

VPL VPR H

VP1 VP2

VP3

To different directions correspond different vanishing points

25

Geometric properties of projection

•  Points go to points •  Lines go to lines •  Planes go to whole image

or half-plane •  Polygons go to polygons

•  Degenerate cases: –  line through focal point yields point –  plane through focal point yields line

Pinhole camera model

TT ZfYZfXZYX )/,/(),,(

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

10100

1ZYX

ff

ZfYfX

ZYX

linear projection in homogeneous coordinates!

Pinhole camera model

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

1010101

1ZYX

ff

ZfYfX

PXx = [ ]0|I)1,,(diagP ff=

Principal point offset

Tyx

T pZfYpZfXZYX )+/,+/(),,(

principal point Tyx pp ),(

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

+

+

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

10100

1ZYX

pfpf

ZZpfYZpfX

ZYX

y

x

x

x

Principal point offset

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=⎟⎟⎟

⎠

⎞

⎜⎜⎜

⎝

⎛

+

+

10100

ZYX

pfpf

ZZpfYZpfX

y

x

x

x

[ ] camX0|IKx =

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

1y

x

pfpf

K calibration matrix

Camera rotation and translation

( )C~-X~R=X~cam

X10RC-R

110C~R-RXcam ⎥

⎦

⎤⎢⎣

⎡=

⎟⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜⎜

⎝

⎛

⎥⎦

⎤⎢⎣

⎡=

ZYX

[ ] camX0|IKx = [ ]XC~-|IKRx =

[ ]t|RKP = C~R-t =PX=x

~

CCD camera

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

1yy

xx

pp

K α

α

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

11y

x

y

x

pfpf

mm

K

General projective camera

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

1yx

xx

pps

K α

α

⎥⎥⎥

⎦

⎤

⎢⎢⎢

⎣

⎡

=

1yx

xx

pp

K α

α

[ ]C~|IKRP =

non-singular

11 dof (5+3+3)

[ ]t|RKP =

intrinsic camera parameters extrinsic camera parameters

Radial distortion

•  Due to spherical lenses (cheap) •  Model:

R

2 2 2 2 21 2( , ) (1 ( ) ( ) ...)

xx y K x y K x y

y⎡ ⎤

= + + + + + ⎢ ⎥⎣ ⎦

R

http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpg straight lines are not straight anymore

Camera model

Relation between pixels and rays in space

?

Projector model

Relation between pixels and rays in space

(dual of camera) (main geometric difference is vertical principal point

offset to reduce keystone effect)

?

Meydenbauer camera

vertical lens shift to allow direct ortho-photographs

Affine cameras

Action of projective camera on points and lines

forward projection of line

( ) µbaµPBPAµB)P(AµX +=+=+=

back-projection of line

lPT=Π

PXlX TT =Π ( )PX x0;xlT ==

PXx =projection of point

Action of projective camera on conics and quadrics back-projection to cone

CPPQ Tco = 0CPXPXCxx TTT ==

( )PXx =

projection of quadric

TPPQC ** = 0lPPQlQ T*T*T ==ΠΠ

( )lPT=Π

ii xX ↔ ? P

Resectioning

Direct Linear Transform (DLT)

ii PXx = [ ] ii PXx ×

rank-2 matrix

Direct Linear Transform (DLT)

Minimal solution

Over-determined solution

⇒  5½ correspondences needed (say 6) P has 11 dof, 2 independent eq./points

n ≥ 6 points

Apminimize subject to constraint

1p =

use SVD

Degenerate configurations

(i)   Points lie on plane or single line passing

through projection center

(ii)   Camera and points on a twisted cubic

Scale data to values of order 1 1.  move center of mass to origin 2.  scale to yield order 1 values

Data normalization

D3σD2σ

Line correspondences

Extend DLT to lines

ilPT=Π

ii 1TPXl

(back-project line)

ii 2TPXl (2 independent eq.)

Geometric error

Gold Standard algorithm

Objective Given n≥6 2D to 3D point correspondences {Xi↔xi’}, determine the Maximum Likelyhood Estimation of P

Algorithm (i)   Linear solution:

(a)  Normalization: (b)  DLT

(ii)   Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error:

(iii)   Denormalization:

ii UXX~ = ii Txx~ =

UP~TP -1=

~ ~ ~

Calibration example

(i)  Canny edge detection (ii)  Straight line fitting to the detected edges (iii)  Intersecting the lines to obtain the images corners

typically precision <1/10 (H&Z rule of thumb: 5n constraints for n unknowns)

Errors in the world

Errors in the image and in the world

ii XPx

=

iX

Errors in the image

iPXx̂ =i

(standard case)

Restricted camera estimation

Minimize geometric error → impose constraint through parametrization

Find best fit that satisfies •  skew s is zero •  pixels are square •  principal point is known •  complete camera matrix K is known

Minimize algebraic error → assume map from param q → P=K[R|-RC], i.e. p=g(q) → minimize ||Ag(q)||

Restricted camera estimation

Initialization •  Use general DLT •  Clamp values to desired values, e.g. s=0, αx= αy Note: can sometimes cause big jump in error Alternative initialization •  Use general DLT •  Impose soft constraints

•  gradually increase weights

Image of absolute conic

A simple calibration device

(i)   compute H for each square (corners → (0,0),(1,0),(0,1),(1,1))

(ii)   compute the imaged circular points H(1,±i,0)T

(iii)   fit a conic to 6 circular points (iv)  compute K from ω through cholesky factorization

(≈ Zhang’s calibration method)

Some typical calibration algorithms Tsai calibration

Zhangs calibration

http://research.microsoft.com/~zhang/calib/

Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.

Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999.

http://www.vision.caltech.edu/bouguetj/calib_doc/

80 from Szeliski’s book

81

Next week: Multiple-view geometry

Computational 3D Ph

otography

82