ComputerVision

Calibration

Marc PollefeysCOMP 256

Read F&P Chapter 2Some slides/illustrations from Ponce, Hartley & Zisserman

ComputerVision

2

Jan 16/18 - Introduction

Jan 23/25 Cameras Radiometry

Jan 30/Feb1 Sources & Shadows Color

Feb 6/8 Linear filters & edges Texture

Feb 13/15 Multi-View Geometry Stereo

Feb 20/22 Optical flow Project proposals

Feb27/Mar1 Affine SfM Projective SfM

Mar 6/8 Camera Calibration Segmentation

Mar 13/15 Springbreak Springbreak

Mar 20/22 Fitting Prob. Segmentation

Mar 27/29 Silhouettes and Photoconsistency

Linear tracking

Apr 3/5 Project Update Non-linear Tracking

Apr 10/12 Object Recognition Object Recognition

Apr 17/19 Range data Range data

Apr 24/26 Final project Final project

Tentative class schedule

ComputerVision

3

PreviouslyHierarchy of 3D transformations

vTv

tAProjective15dof

Affine12dof

Similarity7dof

Euclidean6dof

Intersection and tangency

Parallellism of planes,Volume ratios, centroids,The plane at infinity π∞

Angles, ratios of length

The absolute conic Ω∞

Volume

10

tAT

10

tRT

s

10

tRT

ComputerVision

4

Camera calibration

Compute relation between pixels and rays in space

?

ComputerVision

5

Pinhole camera

ComputerVision

6

Pinhole camera model

TT ZfYZfXZYX )/,/(),,(

101

0

0

1

Z

Y

X

f

f

Z

fY

fX

Z

Y

X

linear projection in homogeneous coordinates!

homogeneous coordinates

non-homogeneous coordinates

ComputerVision

7

Pinhole camera model

101

0

0

Z

Y

X

f

f

Z

fY

fX

101

01

01

1Z

Y

X

f

f

Z

fY

fX

PXx

0|I)1,,(diagP ff

ComputerVision

8

Principal point offset

Tyx

T pZfYpZfXZYX )/,/(),,(

principal pointT

yx pp ),(

101

0

0

1

Z

Y

X

pf

pf

Z

ZpfY

ZpfX

Z

Y

X

y

x

x

x

ComputerVision

9

Principal point offset

101

0

0

Z

Y

X

pf

pf

Z

ZpfY

ZpfX

y

x

x

x

camX0|IKx

1y

x

pf

pf

K calibration matrix

ComputerVision

11

Object motion

ComputerVision

12

Camera motion

ComputerVision

13

CCD camera

1yy

xx

pp

K

11y

x

y

x

pfpf

mm

K

ComputerVision

14

General projective camera

1yx

xx

p

ps

K

1yx

xx

p

p

K

t|IKRP

non-singular

11 dof (5+3+3)

t|RKP

intrinsic camera parametersextrinsic camera parameters

ComputerVision

15

Camera matrix decomposition

Finding the camera center

0PC (use SVD to find null-space)

Finding the camera orientation and internal parameters

KR (use RQ decomposition ~QR)

Q R=( )-1= -1 -1QR

(if only QR, invert)

PXλC)P(Xx (for all X and λ C must be camera center)

t|RKP

ComputerVision

16

Affine cameras

ComputerVision

17

Radial distortion

• Due to spherical lenses (cheap)• Model:

R

yxyxKyxKyx ...))()(1(),( 222

2

22

1R

http://foto.hut.fi/opetus/260/luennot/11/atkinson_6-11_radial_distortion_zoom_lenses.jpgstraight lines are not straight anymore

pincushion dist.

barrel dist.

ComputerVision

18

Radial distortion example

ComputerVision

19

Camera model

Relation between pixels and rays in space

?

ComputerVision

20

Projector model

Relation between pixels and rays in space(dual of camera)

(main geometric difference is vertical principal point offset to reduce keystone effect)

?

ComputerVision

21

Meydenbauer camera

vertical lens shiftto allow direct ortho-photographs

ComputerVision

22

Action of projective camera on points and lines

forward projection of line

μbaμPBPAμB)P(AμX

back-projection of line

lPT

PXlX TT PX x0;xlT

PXx projection of point

ComputerVision

23

Action of projective camera on conics and quadricsback-projection to cone

CPPQ Tco 0CPXPXCxx TTT

PXx

projection of quadric

TPPQC ** 0lPPQlQ T*T*T

lPT

ComputerVision

24

ii xX ? P

Resectioning

ComputerVision

25

Direct Linear Transform (DLT)

ii PXx ii PXx

rank-2 matrix

ComputerVision

26

Direct Linear Transform (DLT)

Minimal solution

Over-determined solution

5½ correspondences needed (say 6)

P has 11 dof, 2 independent eq./points

n 6 points

Apminimize subject to constraint

1p

use SVD

ComputerVision

27

Singular Value Decomposition

XXVT XVΣ T XVUΣ T

Homogeneous least-squares

TVUΣA

1X AXmin subject to nVX solution

TVΣU

X

ComputerVision

28

Degenerate configurations

(i) Points lie on plane or single line passing through projection center

(ii) Camera and points on a twisted cubic

ComputerVision

29

Scale data to values of order 1

1. move center of mass to origin2. scale to yield order 1 values

Data normalization

D3

D2

ComputerVision

30

Line correspondences

Extend DLT to lines

ilPT

ii 1TPXl

(back-project line)

ii 2TPXl (2 independent eq.)

ComputerVision

31

Geometric error

ComputerVision

32

Gold Standard algorithm

ObjectiveGiven n≥6 2D to 2D point correspondences {Xi↔xi’}, determine the Maximum Likelyhood Estimation of P

Algorithm

(i) Linear solution:

(a) Normalization:

(b) DLT

(ii) Minimization of geometric error: using the linear estimate as a starting point minimize the geometric error:

(iii) Denormalization:

ii UXX~ ii Txx~

UP~

TP -1

~ ~~

ComputerVision

33

Calibration example

(i) Canny edge detection(ii) Straight line fitting to the detected edges(iii) Intersecting the lines to obtain the images corners

typically precision <1/10

(H&Z rule of thumb: 5n constraints for n unknowns)

ComputerVision

34

Errors in the world

Errors in the image and in the world

ii XPx

iX

Errors in the image

iPXx̂

i

(standard case)

ComputerVision

35

Restricted camera estimation

Minimize geometric error impose constraint through parametrization

Find best fit that satisfies• skew s is zero• pixels are square • principal point is known• complete camera matrix K is known

Minimize algebraic error assume map from param q P=K[R|-RC], i.e. p=g(q)minimize ||Ag(q)||

ComputerVision

36

Restricted camera estimation

Initialization • Use general DLT• Clamp values to desired values, e.g. s=0, x= y

Note: can sometimes cause big jump in error

Alternative initialization• Use general DLT• Impose soft constraints

• gradually increase weights

ComputerVision

37

ComputerVision

38

Image of absolute conic

ComputerVision

39

A simple calibration device

(i) compute H for each square (corners (0,0),(1,0),(0,1),(1,1))

(ii) compute the imaged circular points H(1,±i,0)T

(iii) fit a conic to 6 circular points(iv) compute K from through cholesky factorization

(≈ Zhang’s calibration method)

ComputerVision

40

Some typical calibration algorithmsTsai calibration

Zhangs calibration

http://research.microsoft.com/~zhang/calib/

Z. Zhang. A flexible new technique for camera calibration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(11):1330-1334, 2000.

Z. Zhang. Flexible Camera Calibration By Viewing a Plane From Unknown Orientations. International Conference on Computer Vision (ICCV'99), Corfu, Greece, pages 666-673, September 1999.

http://www.vision.caltech.edu/bouguetj/calib_doc/

ComputerVision

41

Sequential SfM

• Initialize motion from two images• Initialize structure• For each additional view

– Determine pose of camera– Refine and extend structure

• Refine structure and motion

ComputerVision

42

Initial projective camera motion

• Choose P and P´compatible with F

Reconstruction up to projective ambiguity

(reference plane;arbitrary)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Same for more views?

different projective basis

ComputerVision

43

Initializing projective structure

• Reconstruct matches in projective frame by minimizing the reprojection error

Non-iterative optimal solution •Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

ComputerVision

44

Projective pose estimation

• Infere 2D-3D matches from 2D-2D matches

• Compute pose from (RANSAC,6pts)

F

X

x

Inliers: inx,X x X DD iii P

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

ComputerVision

45

• Refining structure

• Extending structure2-view triangulation

X~

P

1

3

(Iterative linear)

•Initialize motion•Initialize structure•For each additional view

•Determine pose of camera•Refine and extend structure

•Refine structure and motion

Refining and extending structure

ComputerVision

46

Refining structure and motion

• use bundle adjustment

Also model radial distortion to avoid bias!

                       

ComputerVision

47

Metric structure and motion

Note that a fundamental problem of the uncalibrated approach is that it fails if a purely planar scene is observed (in one or more views)

(solution possible based on model selection)

use self-calibration (see next class)

ComputerVision

48

Dealing with dominant planes

ComputerVision

49

PPPgric

HHgric

ComputerVision

50

Farmhouse 3D models

(note: reconstruction much larger than camera field-of-view)

ComputerVision

51

Application: video augmentation

ComputerVision

52

Next class: Segmentation

Reading: Chapter 14