geometric models & camera calibrationcs4243/lecture/geometric.pdf · camera calibration...
TRANSCRIPT
![Page 1: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/1.jpg)
1
Geometric Models & Camera Calibration
Reading: Chap. 2 & 3
![Page 2: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/2.jpg)
2CS4243: Geometric Models
Introduction
We have seen that a camera captures both geometric and photometric information of a 3D scene.
Geometric: shape, e.g. lines, angles, curvesPhotometric: color, intensity
What is the geometric and photometric relationship between a 3D scene and its 2D image?We will understand these in terms of models.
![Page 3: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/3.jpg)
3CS4243: Geometric Models
Models
Models are approximations of reality.“Reality” is often too complex, or computationally intractable, to handle.Examples:
Newton’s Laws of Motion vs. Einstein’s Theory of RelativityLight: waves or particles?
No model is perfect.Need to understand its strengths & limitations.
![Page 4: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/4.jpg)
4CS4243: Geometric Models
Camera Projection Models
3D scenes project to 2D imagesMost common model: pinhole camera model
From this we may derive several types of projections:
orthographicweak-perspective para-perspectiveperspective
least accurate
most accurate
![Page 5: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/5.jpg)
5CS4243: Geometric Models
Pinhole Camera Model
real image plane virtual image plane
f f
YZ
optical centre
object
image is inverted image is not inverted
![Page 6: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/6.jpg)
6CS4243: Geometric Models
Real vs. Virtual Image PlaneThe image is physically formed on the real image plane (retina).The image is vertically and laterally inverted. We can imagine a virtual image plane at a distance of fin front of the camera optical center, where f is the focal length.The image on this virtual image plane is not inverted.
This is actually more convenient.Henceforth, when we say “image plane” we will mean the virtual image plane.
![Page 7: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/7.jpg)
7CS4243: Geometric Models
Perspective Projection
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
1ZYX
⎥⎦
⎤⎢⎣
⎡''
yx
f
u
![Page 8: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/8.jpg)
8CS4243: Geometric Models
Perspective ProjectionThe imaging process is a many-to-one mapping
all points on the 3D ray map to a single image point.
Therefore, depth information is lost
From similar triangles, can write down the perspective equation:
,','ZY
fy
ZX
fx
==
![Page 9: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/9.jpg)
9CS4243: Geometric Models
Perspective Projection
MPu
ZYX
ff
wvu
yx
=
⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡⇔⎥
⎦
⎤⎢⎣
⎡
or1
000
1000000
''
This assumes coordinate axes is at the pinhole.
![Page 10: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/10.jpg)
10CS4243: Geometric Models
t
X
Y
Z
fi
fj
fk
World coordinatesystem
Camera coordinatesystem
3D Scene point P
t : World origin w.r.t. cameracoordinate axes
Rotation R
P : 3D position of scene pointw.r.t world coordinate axes
R : Rotation matrix to align world coordinate axes to camera axes
![Page 11: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/11.jpg)
11CS4243: Geometric Models
Perspective Projection
[ ]
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
×=
×=
100sin0cot-
matrixparameterintrinsic33:
matrixprojection43:
0
0
vu
KtRK
MMPu
θβ
θαα
α, β : scaling in image u, v axes, respectivelyΘ : skew angle, that is, angle between u, v axesu0, v0 : origin offsetNote: α=kf, β=lf where k, l are the magnification factors
![Page 12: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/12.jpg)
12CS4243: Geometric Models
Intrinsic vs. Extrinsic parametersIntrinsic: internal camera parameters.
6 of them (5 if you don’t care about focal length)Focal length, horiz. & vert. magnification, horiz. & vert. offset, skew angle
Extrinsic: external parameters6 of them3 rotation angles, 3 translation param
Imposing assumptions will reduce # params.Estimating params is called camera calibration.
![Page 13: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/13.jpg)
13CS4243: Geometric Models
Representing Rotations
Euler angles
pitch: rotation about x axis :
yaw: rotation about y axis:
roll: rotation about z axis:X
Y
Z
ωφ
κ
![Page 14: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/14.jpg)
14CS4243: Geometric Models
Rotation Matrix
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
−−++−
=
=
)cos()cos()sin()cos()sin()sin()cos()cos()sin()sin()cos()cos()sin()sin()sin()cos()sin()sin()sin()cos()sin()cos()cos()sin()sin()sin()cos(cos)cos(
ωφωφφωκωφκωκωφκφκωκωφκωκωφκφκ
xyz RRRR
Two properties of rotation matrix:
R is orthogonal: RTR = Idet(R) = 1
![Page 15: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/15.jpg)
15CS4243: Geometric Models
Orthographic Projection
Projection rays are parallel
Camera centre
opticalaxis
Image plane
Parallel lines
![Page 16: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/16.jpg)
16CS4243: Geometric Models
Orthographic Projection Equations
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=
⎥⎦
⎤⎢⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
10
10100
000010001
2
1
y
x
T
T
T
T
tt
rr
tRM
TT rr 21 ,
yx tt ,: first two rows of R
: first two elements of t
This projection has 5 degrees of freedom.
![Page 17: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/17.jpg)
17CS4243: Geometric Models
Image plane Parallel lines
Weak-perspective Projection
![Page 18: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/18.jpg)
18CS4243: Geometric Models
Weak-perspective Projection
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
101000000
2
1
y
x
T
T
T
tt
rr
M βα
This projection has 7 degrees of freedom.
![Page 19: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/19.jpg)
19CS4243: Geometric Models
Parallel linesImage plane
Para-perspective Projection
![Page 20: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/20.jpg)
20CS4243: Geometric Models
Para-perspective Projection
Camera matrix given in textbook Table 2.1, page 36.Note: Written a bit differently.This has 9 d.o.f.
![Page 21: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/21.jpg)
21CS4243: Geometric Models
Real cameras
Real cameras use lensesLens distortion causes distortion in image
Lines may project into curvesSee Chap 1.2, 3.3 for details
Change of focal length (zooming) scales the image
not true if assuming orthographic projection)
Color distortions too
![Page 22: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/22.jpg)
22CS4243: Geometric Models
Color Aberration
Bad White Balance
![Page 23: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/23.jpg)
23CS4243: Geometric Models
Color Aberration
Purple Fringing
![Page 24: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/24.jpg)
24CS4243: Geometric Models
Color Aberration
Vignetting
![Page 25: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/25.jpg)
25CS4243: Geometric Models
Geometric Camera Calibration
![Page 26: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/26.jpg)
26CS4243: Geometric Models
Recap
A general projective matrix:
This has 11 d.o.f. , i.e. 11 parameters5 intrinsic, 6 extrinsic
How to estimate all of these?Geometric camera calibration
[ ]PtRKuMPu
==
![Page 27: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/27.jpg)
27CS4243: Geometric Models
Key Idea
Capture image of a known 3D object (calibration object).Establish correspondences between 3D points and their 2D image projections.Estimate MEstimate K, R, and t from MEstimation can be done using linear or non-linear methods.
We will study linear methods first.
![Page 28: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/28.jpg)
28CS4243: Geometric Models
Setting things up
Calibration object:3D object, or2D planar object captured at different locations
![Page 29: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/29.jpg)
29CS4243: Geometric Models
Setting it up
Suppose we have N point correspondences:P1, P2, … PN are 3D scene points u1, u2, … uN are corresponding image points
Let mT1, mT
2, mT3 be the 3 rows of M
Let (ui, vi) be the (non-homogeneous) coords of the i th image point.Let Pi be the homogeneous coords of i th scene point.
![Page 30: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/30.jpg)
30CS4243: Geometric Models
Math
For the i th point
Each point gives 2 equations.
Using all N points gives 2N equations:
0)(
0)(
32
31
=−
=−
iT
i
iT
i
Pmvm
Pmum
![Page 31: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/31.jpg)
31CS4243: Geometric Models
Math
00
0
00
3
2
1111
111
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
−−
−−
mmm
PvPu
PP
PvPPuP
TNN
TNN
TN
T
T
TN
TTT
TTT
A m = 0
A : 2N x 12 matrix, rank is 11m : 12 x 1 column vectorNote that m has only 11 d.o.f.
![Page 32: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/32.jpg)
32CS4243: Geometric Models
Math
Solution?m is in the Nullspace of A !
Use SVD: A = U ∑ VT
m = last column of V m is only up to an unknown scaleIn this case, SVD solution makes || m || = 1
![Page 33: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/33.jpg)
33CS4243: Geometric Models
Estimating K, R, t
Now that we have M (3x4 matrix)
c is the 3D coords of camera center wrt World axesLet ĉ be homogeneous coords of cIt can be shown that M ĉ = 0
So ĉ is in the Nullspace of MAnd t is computed from –Rc, once R is known.
[ ] [ ]Rct
pBtRKM−=
==
![Page 34: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/34.jpg)
34CS4243: Geometric Models
Estimating K, R, t
B, the left 3x3 submatrix of M is KRPerform “RQ” decomposition on B.The “Q” is the required rotation matrix RAnd the upper triangular “R” is our K matrix !
Impose the condition that diagonals of K are positive
We are done!
![Page 35: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/35.jpg)
35CS4243: Geometric Models
RQ factorization
Any n x n matrix A can be factored into A = RQWhere both R, Q are n x nR is upper triangular, Q is orthogonal
Not the same as QR factorizationTrick: post multiply A by Givens rotation matrices: Qx ,Qy, Qz
θθ sin,coswhere10000
,0
0100
,00
001
==
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡ −=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡−=
sc
cssc
Qcs
scQ
csscQ zyx
![Page 36: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/36.jpg)
36CS4243: Geometric Models
RQ factorization
c, s chosen to make a particular entry of A zeroFor example, to make a21 = 0, we solve
Choose Qx ,Qy, Qz such that
222
221
21222
221
22
2221
,
0
aaas
aaac
saca
+=
+
−=⇒
=+
RQQQRQA
RQQAQTx
Ty
Tz
zyx
==⇒
=
ngularupper tria
![Page 37: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/37.jpg)
37CS4243: Geometric Models
Degeneracy
Caution: The 3D scene points should not all lie on a plane.
Otherwise no solution
In practice, choose N >= 6 points in “general position”
Note: the above assumes no lens distortion.See Chap 3.3 for how to deal with lens distortion.
![Page 38: Geometric Models & Camera Calibrationcs4243/lecture/geometric.pdf · Camera Calibration Reading: Chap. 2 & 3. 2 CS4243: Geometric Models Introduction zWe have seen that a camera captures](https://reader034.vdocuments.net/reader034/viewer/2022042712/5fa0f74e43e4774ad3165837/html5/thumbnails/38.jpg)
38CS4243: Geometric Models
Summary
Presented pinhole camera modelFrom this we get hierarchy of projection typesPerspective, Para-perspective, Weak-perspective, Orthographic
Showed how to calibrate camera to estimate intrinsic + extrinsic parameters.