institut für elektrische meßtechnik und meßsignalverarbeitung professor horst cerjak, 19.12.2005...

Institut für Elektrische Meßtechnik und Meßsignalverarbeitung

Professor Horst Cerjak, 19.12.20051

8.4.2008Augmented Reality VU 2 Calibration Axel Pinz

Calibration Issues• Linear Models

– Homography estimation H– Epipolar geometry F, E– Interior camera parameters K– Exterior camera parameters R,t– Camera pose R,t

• Interest Point Detection + Description• Algorithms

– Overdetermined systems of linear equations Error Minimization– Direct Linear Transform – DLT– Normalization– Nonlinearities iterative error minimization, Levenberg-Marquardt– Outliers Robustness, RANSAC

3

2

1

333231

232221

131211

3

2

1

'

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

x

x

H

PtPZ

Y

X

pppp

pppp

pppp

y

x

p

|

1

~

1 34333231

24232221

14131211

RKP




Pinhole Camera

• “real” camera

image plane πi (x,y): Zcam = -f

xy

Zcam

Xcam

Ycam

“principal”point (x0,y0) “optical axis”

p(x,y)

P(Xcam,Ycam,Zcam)

f

“focal length” f

• 2D projection 3D scene

• p(x,y) ↔ line of sight = viewing direction

P’(Xcam’,Ycam’,Zcam’)

• “Pinhole” C … “center of projection”

Ccam

XX YY

ZZ

P(X,Y,Z,1) P’(X’,Y’,Z’,1)

RR,,ttp(x,y,1)

• “ “interior” camera parametersinterior” camera parameters– xx00, y, y00, f, …, f, …

• ““exterior” parametersexterior” parameters– camera posecamera pose– RR, , tt




Pinhole Camera

image plane πi (x,y): Zcam = -f

xy

Zcam

Xcam

Ycam“principal”point (x0,y0) “optical axis”

f Ccam

XX YY

ZZ

P(X,Y,Z,1)P’(X’,Y’,Z’,1)

RR,,ttp(x,y,1)

PZ

Y

X

pppp

pppp

pppp

y

x

p

P

1

~

1 34333231

24232221

14131211PP : 3 x 4 matrix : 3 x 4 matrix““camera projection matrix”camera projection matrix”[Pollefeys p.24, eq. (3.8)][Pollefeys p.24, eq. (3.8)]

Mm P~




The Basic Pinhole Model

Note: Figures taken from, notation following [Hartley,Zisserman]

xZfYZfXZYXX TT ~)/,/(),,(~ ~ … inhomog. ~ … inhomog.

coord.coord.

XZ

Y

X

f

f

Z

fY

fX

xZ

Y

X

P

101

0

0

1

… … homog. coord.homog. coord.




The Basic Pinhole Model

0~

|)1,,diag(

01

01

01

101

0

0

IP fff

f

f

f




Principal Point Offset

camXZ

Y

X

yf

xf

Z

ZyfY

ZxfX

xZ

Y

X

P

101

0

0

1

0

0

0

0




Principal Point Offset

camcam XXyf

xf

x

0~

|

01

01

01

10

0

IK

10

0

yf

xf

Kcamera calibration matrixcamera calibration matrixinterior/internal parametersinterior/internal parametersinterior/internal orientationinterior/internal orientation




Camera Rotation and Translation

CXX cam

~~~ R

XC

X cam

10

~RR XCx

~| IKR

4 x 4

P3 P3

3 x 4

P2 P3





C~| IKRP 3 x 4 projection matrix P9 degrees of freedom

3 “internal parameters” in K3 rotation angles in R3 translations in C~





C~| IKRPSimplified notation: avoid explicit modeling of C

t|CttXX cam

RKPRR ~

,~~




• Pinhole– 3 parameters in K

• CCD– 4 parameters

• Finite projective camera– 5 parameters– “skew” s

From Pinhole Real Cameras: K

10

0

yf

xf

K

yyxxy

x

fmfmy

x

, ,

10

0

K

mx

my

10

0

y

xs

y

x

K

3 x R, 3 x t

9

10

11




Projective Camera

• Finite projective camera:– K is an upper triangular matrix– KR is non-singular

• General projective camera:– P is an arbitrary 3 x 4 matrix of rank 3– P has also 11 degrees of freedom

41|

~| pC

MIMIKRP

34333231

24232221

14131211

pppp

pppp

pppp

P

But: We model real camerasBut: We model real camerasas finite projective camerasas finite projective cameras(+ lens distortion)(+ lens distortion)




Camera Calibration in Practice (1)

• Take – 1 picture of a 3D calibration target, – or several pictures of a planar calibration target

(take care so that all parameters can be recovered !)

• Establish point correspondences

• Calculate P– set of linear equations

• Decompose P

niXx ii 1 ,~~

ii

i

i

i

i

i XZ

Y

X

pppp

pppp

pppp

y

x

x

P

1

~

1 34333231

24232221

14131211

t|RKP




3D Targets

[Hartley + Zisserman][Heikkilä]

Photogrammetry [Godding / Jähne]

• Many ways to build …• Corners vs. circles (center of gravity) …• Precision of building, attaching, …• CNC measured points …• EMT: coordinate measurement machine




2D vs. 3D Targets

f = 28mm, z ~ 300mmf = 28mm, z ~ 300mm f = 50mm, z ~ 470mmf = 50mm, z ~ 470mm f = 84mm, z ~ 720mmf = 84mm, z ~ 720mm




2D Targets

f = 28mm, z ~ 280mmf = 28mm, z ~ 280mm f = 50mm, z ~ 470mmf = 50mm, z ~ 470mm f = 84mm, z ~ 720mmf = 84mm, z ~ 720mm

• arbitrary scaling ! arbitrary scaling ! – z/f ~ const.z/f ~ const.– closeup of toy car vs. real car at a distance …closeup of toy car vs. real car at a distance …

• but: subtle differences in image quality !but: subtle differences in image quality !




Image Quality (1)

f = 28mm, z ~ 280mmf = 28mm, z ~ 280mm f = 50mm, z ~ 470mmf = 50mm, z ~ 470mm

lens distortion !lens distortion !




Image Quality (2)

f = 28mm, z ~ 280mmf = 28mm, z ~ 280mm f = 50mm, z ~ 470mmf = 50mm, z ~ 470mm

“ “chromatic aberration”chromatic aberration”




Lens Distortion Model

• Several ways to model

• Most common:– Radial lens distortion ki

– Tangential lens distortion tj

– Radial >> tangential– Polynomial approximation up to varying order

(x0,y0)

,0xxx ,

0yyy 22 yxr

)1)(2)2((' 232

221

642

231

rtyxtrxtrxkrxkrxkxx

)1)(2)2((' 232

221

642

231

rtyxtrytrykrykrykyy

r

x

y




Camera Calibration in Practice (2)

• Take – 1 picture of a 3D calibration target, – or several pictures of a planar calibration target

• Establish point correspondences• Calculate P

– set of linear equations

• Decompose P

A f

irst

estim

ate

for

A f

irst

estim

ate

for

linea

rlin

ear

Inte

rior

para

met

ers

(In

terio

r pa

ram

eter

s ( KK

))

• Add nonlinear relationships (model ki, tj)• Perform iterative optimization (w.r.t. some error)• Enforce constraints (such as structure of K and R)




More Matrices …

• Homography H

• Projection P (K, R, t )

• Multiple views:– Epipolar geometry– Uncalibrated stereo: “Fundamental” matrix F– Calibrated stereo: “Essential” matrix E– Stereo rig– Camera motion many views AR tracking




Epipolar Geometry (1)

• Figures from [Hartley + Zisserman]• C, C’, x, x’, X are co-planar (lie in the “epipolar plane” π)





• Assume that only C, C’, and x are known





• π projects on “epipolar lines” l and l’• “baseline”: connects C, C’• “epipoles”: e, e’

C C’





• When 3D position of X varies, π “rotates” about the baseline• Family of planes – “epipolar pencil” – “Ebenenbüschel”

C C’




Epipolar Geometry – Example 1:Converging Cameras [Hartley+Zisserman]




Epipolar Geometry – Example 2:Forward Translation [Hartley+Zisserman], [Pollefeys]

e

e’




The Fundamental Matrix F (1)

We had an example:Homography H





• Transfer xi via Xi in π to xi’• 2D homography Hπ maps each xi to xi’

''' xel

'''' xexe

0

0

0

' ,''1

'2

'1

'3

'2

'3

'3

'2

'1

ee

ee

ee

e

e

e

e

e

xxelxxFHH '' :'

“skew-symmetric”matrix

xlF'





• F relates x in one image with its corresponding epipolar line l’ in the other image (all X in R3 !):

• The corresponding point x’ must lie on l’:

• This relates to:

• How to estimate F?

xlF'

0'' ,0'' lxlx T

0' xx TF

Point correspondences




Calibration Issues• Linear Models

– Homography estimation H– Epipolar geometry F, E– Interior camera parameters K– Exterior camera parameters R,t– Camera pose R,t

• Interest Point Detection + Description• Algorithms

– Overdetermined systems of linear equations Error Minimization– Direct Linear Transform – DLT– Normalization– Nonlinearities iterative error minimization, Levenberg-Marquardt– Outliers Robustness, RANSAC

3

2

1

333231

232221

131211

3

2

1

'

'

'

'

x

x

x

hhh

hhh

hhh

x

x

x

x

x

H

PtPZ

Y

X

pppp

pppp

pppp

y

x

p

|

1

~

1 34333231

24232221

14131211

RKP




A final word on E

• “Essential matrix” E – Similar to F– Relates calibrated stereo rig– Internal matrices K and K’ are known

tt TRRRE R, t

xxxx T1ˆ ,0ˆ'ˆ KE “normalized coordinates” x̂

FKKEEKK TTT xx ' 0'' 1

institut für elektrische meßtechnik und meßsignalverarbeitung professor horst cerjak, 19.12.2005...

Documents

y px cam

augmented reality vu

optical axis f c cam

exterior camera parameters

y line of sight

z cam f focal length

viewing direction px

e interior camera parameters