camera calibration and epipolar geometry - computer...
TRANSCRIPT
![Page 1: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/1.jpg)
Camera calibration and epipolar geometry
Odilon Redon, Cyclops, 1914
![Page 2: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/2.jpg)
Review: Alignment• What is the geometric relationship between
pictures taken by cameras that share the same center?
• How many points do we need to estimate a homography?
• How do we estimate a homography?
![Page 3: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/3.jpg)
Geometric vision• Goal: Recovery of 3D structure
• What cues in the image allow us to do this?
![Page 4: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/4.jpg)
Shading
Visual cues
Merle Norman Cosmetics, Los Angeles
Slide credit: S. Seitz
![Page 5: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/5.jpg)
Visual cuesShading
Texture
The Visual Cliff, by William Vandivert, 1960
Slide credit: S. Seitz
![Page 6: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/6.jpg)
Visual cues
From The Art of Photography, Canon
Shading
Texture
Focus
Slide credit: S. Seitz
![Page 7: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/7.jpg)
Visual cuesShading
Texture
Focus
Perspective
Slide credit: S. Seitz
![Page 8: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/8.jpg)
Visual cuesShading
Texture
Focus
Perspective
MotionSlide credit: S. Seitz
![Page 9: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/9.jpg)
Our goal: Recovery of 3D structure• We will focus on perspective and motion• We need multi-view geometry because
recovery of structure from one image is inherently ambiguous
x
X?X?
X?
![Page 10: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/10.jpg)
Our goal: Recovery of 3D structure• We will focus on perspective and motion• We need multi-view geometry because
recovery of structure from one image is inherently ambiguous
![Page 11: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/11.jpg)
Our goal: Recovery of 3D structure• We will focus on perspective and motion• We need multi-view geometry because
recovery of structure from one image is inherently ambiguous
![Page 12: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/12.jpg)
)/,/(),,( ZYfZXfZYX a
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
10100
1ZYX
ff
ZYfXf
ZYX
a
Recall: Pinhole camera model
PXx =
![Page 13: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/13.jpg)
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
1010101
1ZYX
ff
ZYfXf
PXx = [ ]0|I)1,,(diagP ff=
Pinhole camera model
![Page 14: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/14.jpg)
Camera coordinate system
• Principal axis: line from the camera center perpendicular to the image plane
• Normalized (camera) coordinate system: camera center is at the origin and the principal axis is the z-axis
• Principal point (p): point where principal axis intersects the image plane (origin of normalized coordinate system)
![Page 15: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/15.jpg)
Principal point offset
• Camera coordinate system: origin is at the prinicipal point
• Image coordinate system: origin is in the corner
principal point: ),( yx pp
![Page 16: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/16.jpg)
)/,/(),,( yx pZYfpZXfZYX ++a
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛++
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
10100
1ZYX
pfpf
ZpZYfpZXf
ZYX
y
x
x
x
a
Principal point offset
principal point: ),( yx pp
![Page 17: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/17.jpg)
⎟⎟⎟⎟⎟
⎠
⎞
⎜⎜⎜⎜⎜
⎝
⎛
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛++
1010101
1ZYX
pfpf
ZZpYfZpXf
y
x
x
x
Principal point offset
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
1y
x
pfpf
K calibration matrix [ ]0|IKP =
principal point: ),( yx pp
![Page 18: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/18.jpg)
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
111yy
xx
y
x
y
x
pfpf
mm
K βαβα
Pixel coordinates
mx pixels per meter in horizontal direction, my pixels per meter in vertical direction
Pixel size: yx mm
11×
pixels/m m pixels
![Page 19: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/19.jpg)
( )C~-X~RX~ cam =
Camera rotation and translation
• In general, the camera coordinate frame will be related to the world coordinate frame by a rotation and a translation
coords. of point in camera frame
coords. of camera center in world frame
coords. of a pointin world frame (nonhomogeneous)
![Page 20: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/20.jpg)
( )C~-X~RX~ cam =
X10
C~RR1X~
10C~RRXcam ⎥
⎦
⎤⎢⎣
⎡ −=⎟⎟
⎠
⎞⎜⎜⎝
⎛⎥⎦
⎤⎢⎣
⎡ −=
[ ] [ ]XC~R|RKX0|IKx cam −== [ ],t|RKP = C~Rt −=
Camera rotation and translation
In non-homogeneouscoordinates:
Note: C is the null space of the camera projection matrix (PC=0)
![Page 21: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/21.jpg)
Camera parameters• Intrinsic parameters
• Principal point coordinates• Focal length• Pixel magnification factors• Skew (non-rectangular pixels)• Radial distortion
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡=
111yy
xx
y
x
y
x
pfpf
mm
K βαβα
![Page 22: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/22.jpg)
Camera parameters• Intrinsic parameters
• Principal point coordinates• Focal length• Pixel magnification factors• Skew (non-rectangular pixels)• Radial distortion
• Extrinsic parameters• Rotation and translation relative to world coordinate system
![Page 23: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/23.jpg)
Camera calibration• Given n points with known 3D coordinates Xi
and known image projections xi, estimate the camera parameters
? P
Xi
xi
![Page 24: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/24.jpg)
ii PXx =λ
Camera calibration
0PXx =× iiiT
T
T
i
i
yx
XPPP
1 3
2
1
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡λ
0PPP
0XXX0X
XX0
3
2
1
=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−
−
Tii
Tii
Tii
Ti
Tii
Ti
xyx
y
Two linearly independent equations
![Page 25: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/25.jpg)
Camera calibration
• P has 11 degrees of freedom (12 parameters, but scale is arbitrary)
• One 2D/3D correspondence gives us two linearly independent equations
• Homogeneous least squares• 6 correspondences needed for a minimal solution
0pA =0PPP
X0XXX0
X0XXX0
3
2
1111
111
=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
−−
−−
Tnn
TTn
Tnn
Tn
T
TTT
TTT
xy
xy
LLL
![Page 26: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/26.jpg)
Camera calibration
• Note: for coplanar points that satisfy ΠTX=0,we will get degenerate solutions (Π,0,0), (0,Π,0), or (0,0,Π)
0Ap =0PPP
X0XXX0
X0XXX0
3
2
1111
111
=⎟⎟⎟
⎠
⎞
⎜⎜⎜
⎝
⎛
⎥⎥⎥⎥⎥⎥
⎦
⎤
⎢⎢⎢⎢⎢⎢
⎣
⎡
−−
−−
Tnn
TTn
Tnn
Tn
T
TTT
TTT
xy
xy
LLL
![Page 27: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/27.jpg)
Camera calibration• Once we’ve recovered the numerical form of
the camera matrix, we still have to figure out the intrinsic and extrinsic parameters
• This is a matrix decomposition problem, not an estimation problem (see F&P sec. 3.2, 3.3)
![Page 28: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/28.jpg)
Two-view geometry• Scene geometry (structure): Given
corresponding points in two or more images, where is the pre-image of these points in 3D?
• Correspondence (stereo matching): Given a point in just one image, how does it constrain the position of the corresponding point x’ in another image?
• Camera geometry (motion): Given a set of corresponding points in two images, what are the cameras for the two views?
![Page 29: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/29.jpg)
Triangulation• Given projections of a 3D point in two or more
images (with known camera matrices), find the coordinates of the point
O1 O2
x1x2
X?
![Page 30: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/30.jpg)
Triangulation• We want to intersect the two visual rays
corresponding to x1 and x2, but because of noise and numerical errors, they don’t meet exactly
O1 O2
x1x2
X?R1R2
![Page 31: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/31.jpg)
Triangulation: Geometric approach• Find shortest segment connecting the two
viewing rays and let X be the midpoint of that segment
O1 O2
x1x2
X
![Page 32: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/32.jpg)
Triangulation: Linear approach
baba ][0
00
×=⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
⎥⎥⎥
⎦
⎤
⎢⎢⎢
⎣
⎡
−−
−=×
z
y
x
xy
xz
yz
bbb
aaaa
aa
XPxXPx
222
111
==
λλ
0XPx0XPx
22
11
=×=×
0XP][x0XP][x
22
11
==
×
×
Cross product as matrix multiplication:
![Page 33: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/33.jpg)
Triangulation: Linear approach
XPxXPx
222
111
==
λλ
0XPx0XPx
22
11
=×=×
0XP][x0XP][x
22
11
==
×
×
Two independent equations each in terms of three unknown entries of X
![Page 34: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/34.jpg)
Triangulation: Nonlinear approachFind X that minimizes
O1 O2
x1x2
X?
x’1
x’2
),(),( 222
112 XPxdXPxd +
![Page 35: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/35.jpg)
• Epipolar Plane – plane containing baseline (1D family)• Epipoles= intersections of baseline with image planes = projections of the other camera center= vanishing points of camera motion direction• Epipolar Lines - intersections of epipolar plane with imageplanes (always come in corresponding pairs)
• Baseline – line connecting the two camera centers
Epipolar geometryX
x x’
![Page 36: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/36.jpg)
Example: Converging cameras
![Page 37: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/37.jpg)
Example: Motion parallel to image plane
![Page 38: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/38.jpg)
e
e’
Example: Forward motion
Epipole has same coordinates in both images.Points move along lines radiating from e: “Focus of expansion”
![Page 39: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/39.jpg)
Epipolar constraint
• If we observe a point x in one image, where can the corresponding point x’ be in the other image?
x x’
X
![Page 40: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/40.jpg)
• Potential matches for x have to lie on the corresponding epipolar line l’.
• Potential matches for x’ have to lie on the corresponding epipolar line l.
Epipolar constraint
x x’
X
x’
X
x’
X
![Page 41: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/41.jpg)
Epipolar constraint example
Source: K. Grauman
![Page 42: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/42.jpg)
X
x x’
Epipolar constraint: Calibrated case
• Assume that the intrinsic and extrinsic parameters of the cameras are known
• We can multiply the projection matrix of each camera (and the image points) by the inverse of the calibration matrix to get normalized image coordinates
• We can also set the global coordinate system to the coordinate system of the first camera
![Page 43: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/43.jpg)
X
x x’
Epipolar constraint: Calibrated case
Camera matrix: [I|0]X = (u, v, w, 1)T
x = (u, v, w)T
Camera matrix: [RT | –RTt]Vector x’ in second coord. system has coordinates Rx’ in the first one
Rt
The vectors x, t, and Rx’ are coplanar
![Page 44: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/44.jpg)
Essential Matrix(Longuet-Higgins, 1981)
Epipolar constraint: Calibrated case
0)]([ =′×⋅ xRtx RtExExT ][with0 ×==′
X
x x’
The vectors x, t, and Rx’ are coplanar
![Page 45: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/45.jpg)
X
x x’
Epipolar constraint: Calibrated case
• E x’ is the epipolar line associated with x’ (l = E x’)• ETx is the epipolar line associated with x (l’ = ETx)• E e’ = 0 and ETe = 0• E is singular (rank two)• E has five degrees of freedom (up to scale)
0)]([ =′×⋅ xRtx RtExExT ][with0 ×==′
![Page 46: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/46.jpg)
Epipolar constraint: Uncalibrated case
• The calibration matrices K and K’ of the two cameras are unknown
• We can write the epipolar constraint in terms of unknown normalized coordinates:
X
x x’
0ˆˆ =′xExT xKxxKx ′′=′= ˆ,ˆ
![Page 47: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/47.jpg)
Epipolar constraint: Uncalibrated caseX
x x’
Fundamental Matrix(Faugeras and Luong, 1992)
0ˆˆ =′xExT
xKxxKx′′=′
=ˆˆ
1with0 −− ′==′ KEKFxFx TT
![Page 48: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/48.jpg)
Epipolar constraint: Uncalibrated case
0ˆˆ =′xExT 1with0 −− ′==′ KEKFxFx TT
• F x’ is the epipolar line associated with x’ (l = F x’)• FTx is the epipolar line associated with x (l’ = FTx)• F e’ = 0 and FTe = 0• F is singular (rank two)• F has seven degrees of freedom
X
x x’
![Page 49: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/49.jpg)
The eight-point algorithm
x = (u, v, 1)T, x’ = (u’, v’, 1)T
Minimize:
under the constraint|F|2 = 1
2
1)( i
N
i
Ti xFx ′∑
=
![Page 50: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/50.jpg)
The eight-point algorithm
• Meaning of error
sum of Euclidean distances between points xi and epipolar lines Fx’i (or points x’i and epipolar lines FTxi) multiplied by a scale factor
• Nonlinear approach: minimize
:)( 2
1i
N
i
Ti xFx ′∑
=
[ ]∑=
′+′N
ii
Tiii xFxxFx
1
22 ),(d),(d
![Page 51: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/51.jpg)
Problem with eight-point algorithm
![Page 52: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/52.jpg)
Problem with eight-point algorithm
Poor numerical conditioningCan be fixed by rescaling the data
![Page 53: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/53.jpg)
The normalized eight-point algorithm
• Center the image data at the origin, and scale it so the mean squared distance between the origin and the data points is 2 pixels
• Use the eight-point algorithm to compute F from the normalized points
• Enforce the rank-2 constraint (for example, take SVD of F and throw out the smallest singular value)
• Transform fundamental matrix back to original units: if T and T’ are the normalizing transformations in the two images, than the fundamental matrix in original coordinates is TT F T’
(Hartley, 1995)
![Page 54: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/54.jpg)
Comparison of estimation algorithms
0.80 pixel0.85 pixel2.18 pixelsAv. Dist. 2
0.86 pixel0.92 pixel2.33 pixelsAv. Dist. 1
Nonlinear least squaresNormalized 8-point8-point
![Page 55: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/55.jpg)
Epipolar transfer• Assume the epipolar geometry is known• Given projections of the same point in two
images, how can we compute the projection of that point in a third image?
x1 x2? x3
![Page 56: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/56.jpg)
Epipolar transfer• Assume the epipolar geometry is known• Given projections of the same point in two
images, how can we compute the projection of that point in a third image?
x1 x2 x3 l32l31
l31 = FT13 x1
l32 = FT23 x2
When does epipolar transfer fail?
![Page 57: Camera calibration and epipolar geometry - Computer …cs.unc.edu/~lazebnik/research/spring08/lec11_epipolar.pdfEpipolar constraint: Calibrated case • Assume that the intrinsic and](https://reader031.vdocuments.net/reader031/viewer/2022030505/5ab2b34f7f8b9a284c8da300/html5/thumbnails/57.jpg)
Next time: Stereo