robotics - projective geometry and camera modelhome.deib.polimi.it/gini/robot/docs/vision1.pdf ·...

Robotics - Projective Geometry and Camera model

Marcello Restelli

[email protected]

Dipartimento di Elettronica, Informazione e BioingegneriaPolitecnico di Milano

May 2013

Inspired from Matteo Pirotta’s slides (Robotics @ Como 2013)

Camera Geometry Pin Hole Model

Outline

1 Camera Geometry

2 Pin Hole Model

2/26


Outline

1 Camera Geometry

2 Pin Hole Model

3/26


What is an image

Image

Two-dimensional brightness array: I

3× two-dimensional array: IR , IG , IB

RGB: Red, Green, Blue

others: YUV, HSV, HSL, · · ·

Ideal: I : Ω ⊂ R2 → R+

Discrete: I : Ω ⊂ N2 → R∗+e.g., Ω = [0, 639]× [0, 479] ⊂ N2

e.g., Ω = [1, 1024]× [1, 768] ⊂ N2

e.g., R∗+ = [0, 255] ⊂ N

e.g., R∗+ = [0, 1] ⊂ R

I(x , y) is the intensity

I result of 3D→ 2D projection: flat

we lose: angles, distances (lengths)

4/26


Camera

Optical system

Set of lenses to direct light

change in the direction of propagation

CCD sensor

integrate energy both

in time (exposure time)

in space (pixel size)

5/26


Thin lenses model

Thin lenses

Mathematical model

Optical axis (z)

Focal plane πf (⊥ z)

Optical center o

Parameters

f distance o, πf

Property

Parallel rays converge πf

Rays through o undeflected

z

y

o

πf

f

6/26


Rays from scene

Image from a scene point P

P = (Z ,Y )

Ray through o undeflected

Ray parallel to z cross in (−f , 0)

Similarities

Blue triangles:h

Y=

r − f

f

Green triangles:h

Y=

r

Z

Fresnel’s Law

1

Z+

1

r=

1

f

Note: Z →∞⇒ r → f

z

y

P

p

o

f

r

h

Z

Y

7/26


The image plane

Image plane πI

Plane ⊥ z at distance d

Blur Circle

If d 6= r

image of P is a circle C

Diameter of C:

φ(C) =a(d − r)

ra is the aperture

z

y

P

p

o

C

a

πI πf

f

r

d

Z

Focused image

φ(C) < pixel size

Depth of field : range [Z1,Z2] : φ(C) < pixel size

set of values that satisfies Fresnel’s law

8/26


Depth of field - Example 1

9/26


Depth of field - Example 2

The same scene - different aperture

10/26


Aperture controls depth of field

Why not make the aperture as small as possible?

Less light gets through;

Diffraction effects . . .

11/26


Aperture controls depth of field

Changing the aperture size affects depth of field

A smaller aperture increases the range in which the object is approximately in

focus;

But small aperture reduces amount of light need to increase exposure.

12/26


Outline

1 Camera Geometry

2 Pin Hole Model

13/26


Pinhole model - Definition

Hypothesis

Z a

Z f → r ∼ f

Image of P

lPo : line that join P and o

p = πI ∩ lPo

Notes

p is the image of ∀Pi ∈ lPo

lPo : interpretation line of p

Pin-Hole Model

Y

X

camera

centre

Z

principal axis

P

O

-f

p

image plane

O(I)

x

y

z

Frontal Pin-Hole Model

xp

Y

image plane

camera

centre

Z

principal axis

P

O

O(I)

f

y

X

14/26


Pinhole model - Geometry

camera centre: centre of projection or optical centre (O);

principal axis: line from the camera centre perpendicular to the image plane (z);

principal point: intersection between image plane an principal axis (O(I ));

principal plane: plane through the camera centre, parallel to the image plane.

xp

Y

image plane

camera

centre

Z

principal axis

P

O

O(I)

f

y

X

15/26


Pinhole model - Geometry (Cont’d)

Given

P(O) =[X ,Y ,Z , 1

]TP′

(O)=[X ′,Y ′,Z ′, 1

]Tp(I) =

[x , y , 1

]TProjection

y = fY

Z

x = fX

Zlook at the triangles

Note

λP(O) projects on p(I)[sX , sY , sZ , 1

]Tprojects on p(I),

∀s 6= 0

Z ′ = −f , X ′ = −fX

Z, Y ′ = −f

Y

Z

x = −X ′, y = −Y ′

z (O)

y (O)

P(O)

p(I)

O

y (I)

I

−f

y

Z

Y

16/26


Pinhole model - Frontal Camera Model

The pinhole camera model is inconvenience that the focal length f is negative. Thecoordinate system:

We will use the pinhole model as an approximation;

Put the camera centre (O) at the origin;

Put the image plane in front of the camera centre;

The camera looks up the positive z axis.

Projection equations

Compute intersection with IP of ray from P to O;

Derived using similar triangles

x = fX

Z, y = f

Y

Z

Ignoring the final image coordinate: (X ,Y ,Z)T → (f XZ, f Y

Z)T

17/26


Pinhole model - MatrixProjection equations

y = fY

Z

x = fX

ZIn matrix formx ′

y ′

w ′

=

f 0 00 f 00 0 1

1 0 0 00 1 0 00 0 1 0

XYZW

=

f 0 0 00 f 0 00 0 1 0

XYZW

p(I) = π P(O)

Define

K =

f 0 00 f 00 0 1

:

intrinsic parameters

π =[K 0

]: projection matrix

18/26


Distortions from Physical Lenses

Pinhole camera model assumes:

Perfect pinhole camera lenses;

Image x = I (p) ∈ R2 be measured in infinite accuracy;

Principal point is at the center of the image.

Physical camera lenses give us:

Distorted imaging projections;

Finite resolutions dened by the sensing devices in digital cameras;

Offset between image center and optical center.

x ′

y ′I′

x

y

I

p′

p

Ox

y

z

19/26


Pinhole model - Image coordinates - 1

Reference system on image

I: origin centered on z (O) ∩ πI

I′: origin centered top-left image

c(I′) =[cx , cy

]T: position of I in I′

Metric

I metric

I′ in pixel

c(I′) in pixel

Definition[0, 0]T (I)

≡[cx , cy

]T (I′)

: principal point

Image of the optical center (o) or z (O) ∩ πI

x ′

y ′

I′

x

y

I

20/26



The origin of coordinates in image planes is not

at the principal point[cx , cy

]T (I′)

:

(X ,Y ,Z)T → (fX

Z+ cx , f

Y

Z+ cy )T

x ′

y ′

w ′

=

f 0 cx 0

0 f cy 0

0 0 1 0

X

Y

Z

W

p(I′) = π P(O)

x ′

y ′

I′

x

y

c(I ′)

I

21/26


Pinhole model - Image coordinates - 3 (CCD Camera)

The pinhole camera model assumes that image coordinate Euclidean (equal scales

in both axial direction);

In CCD there is the possibility of non-square pixels;

Image coordinates measured in pixel introduces unequal scale factors in each

direction.

If the number of pixels per unit distance coordinates are mx and my in the x and

y direction:

π =

mx 0 0

0 my 0

0 0 1

f 0 cx 0

0 f cy 0

0 0 1 0

where (cx , cy ) is the principal point in terms of pixel dimensions.

22/26



Meters to pixels

Consider I′′: origin on I, in pixel

Scale meters to pixels

p(I′′)x = sxp

(I)x

p(I′′)y = syp

(I)y

sx = 1dx

, dx : width of a pixel [m]

sy = 1dy

, dy : height of a pixel [m]

sx = sy : square pixel

p(I′′) =

sx 0 0

0 sy 0

0 0 1

p(I)

Translation

p(I′) =

1 0 cx

0 1 cy

0 0 1

p(I′′)

x ′

y ′

I′

x

y

x

y

c(I ′)

I ≡ I′′

23/26


Pinhole model - Intrinsic camera matrix

Consider

p(I) =

f 0 0 0

0 f 0 0

0 0 1 0

P(O)

p(I′′) =

sx 0 0

0 sy 0

0 0 1

p(I)

p(I′) =

1 0 cx

0 1 cy

0 0 1

p(I′′)

In one step

p(I′) =

sx f 0 cx 0

0 sy f cy 0

0 0 1 0

P(O)

p(I′) = πP(O) = K[I∣∣0]P(O)

The intrinsic camera matrix

or calibration matrix

K =

fx s cx

0 fy cy

0 0 1

fx , fy : focal lenght (in pixels)

fy/fx = sy/sx = a: aspect ratio

s: skew factor

pixel not orthogonal

usually 0 in modern cameras

cx , cy : principal point (in pixel)

usually 6= half image size due to

misalignment of CCD

24/26


Camera rotation and translation

Consider

p(I′) = πP(O)

point is expressed in terms of world

coordinate frame: P(W ). W and O are

related via a rotation and a translation.

P(O) = T(O)OWP(W ) = R(O)

OW

(P(W ) −O(W )

)extrinsic camera matrix

One step

p(I′) =[K 0

] [R −RC0 1

]P(W )

π =[KR −KRC

]complete projection matrix

Note

R is R(O)OW , C is O(O)

OW (camera centre)

i.e., the position and orientation of W in O

X

Z

Y

R, t

Y

X

O Z

W

W

O

T(O)OW

p(W ) p(O)

25/26


Camera rotation and translation - 2

General pinhole camera

p(I′) =[KR −KRC

]P(W )

π = KR[I −C

]C is the camera centre in world reference

9 (11) dof: 3 (5) for K, 3 for R and 3 for C

parameters in K are called internal

parameters in R and C are called external

Other formulation

do not make the camera center explicit

p(I′) =[K 0

] [R t

0 1

]P(W )

π =[KR Kt

]t = −RC

The intrinsic parameters

K =

fx s cx

0 fy cy

0 0 1

The extrinsic parameters

R = R (φ, θ, ρ)

C = [c1, c2, c3]T

26/26

robotics - projective geometry and camera modelhome.deib.polimi.it/gini/robot/docs/vision1.pdf ·...

Documents