lecture6 affine sfm-2 - cvgl.stanford.edu · 1, e 2 • epipolar lines • baseline epipolar...
TRANSCRIPT
![Page 1: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/1.jpg)
Lecture 6 - Silvio Savarese 24-Jan-15
Professor Silvio Savarese Computational Vision and Geometry Lab
Lecture 6Stereo Systems Multi-‐view geometry
![Page 2: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/2.jpg)
Lecture 5 - Silvio Savarese 26-Jan-15
• Stereo systems • Rectification • Correspondence problem
• Multi-‐view geometry • The SFM problem • Affine SFM
Reading: [HZ] Chapter: 9 “Epip. Geom. and the Fundam. Matrix Transf.” [HZ] Chapter: 18 “N view computational methods” [FP] Chapters: 7 “Stereopsis” [FP] Chapters: 8 “Structure from Motion”
Lecture 6Stereo Systems Multi-‐view geometry
![Page 3: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/3.jpg)
• Epipolar Plane • Epipoles e1, e2
• Epipolar Lines• Baseline
Epipolar geometry
O O’
p’
P
p
e e’
= intersections of baseline with image planes = projections of the other camera center
![Page 4: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/4.jpg)
O O’
pp’
P
Epipolar Constraint
0=!pEpT
E = Essential Matrix(Longuet-Higgins, 1981)
[ ] RTE ⋅= ×
![Page 5: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/5.jpg)
O O’
pp’
P
Epipolar Constraint
0pFpT =!
F = Fundamental Matrix(Faugeras and Luong, 1992)
[ ] 1−×
− #⋅⋅= KRTKF T
![Page 6: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/6.jpg)
O O’
P
e’p p’e
Example: Parallel image planes
x
y
z
• Epipolar lines are horizontal• Epipoles go to infinity• y-coordinates are equal
p=uv1
!
"
###
$
%
&&&
p' =u'v1
!
"
###
$
%
&&&
![Page 7: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/7.jpg)
Rectification: making two images “parallel”
H
Courtesy figure S. Lazebnik
![Page 8: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/8.jpg)
Why are parallel images useful?
• Makes the correspondence problem easier • Makes triangulation easy • Enables schemes for image interpolation
![Page 9: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/9.jpg)
Application: view morphingS. M. Seitz and C. R. Dyer, Proc. SIGGRAPH 96, 1996, 21-30
![Page 10: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/10.jpg)
Why are parallel images useful?
• Makes the correspondence problem easier • Makes triangulation easy • Enables schemes for image interpolation
![Page 11: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/11.jpg)
Point triangulation
O O’
P
e2p p’e2
x
y
z
u
v
u
v
![Page 12: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/12.jpg)
u’uf
B
z
O O’
P
zfBuu ⋅
=− '
Note: Disparity is inversely proportional to depth
Computing depth
= disparity [Eq. 1]
![Page 13: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/13.jpg)
Disparity maps
http://vision.middlebury.edu/stereo/
Stereo pair
Disparity map / depth map Disparity map with occlusions
u u’
zfBuu ⋅
=− '
![Page 14: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/14.jpg)
Why are parallel images useful?
• Makes the correspondence problem easier • Makes triangulation easy • Enables schemes for image interpolation
![Page 15: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/15.jpg)
Correspondence problem
p’
P
p
O1 O2
Given a point in 3D, discover corresponding observations in left and right images [also called binocular fusion problem]
![Page 16: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/16.jpg)
Correspondence problem
p’
P
p
O1 O2
When images are rectified, this problem is much easier!
![Page 17: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/17.jpg)
•A Cooperative Model (Marr and Poggio, 1976)
•Correlation Methods (1970--)
•Multi-Scale Edge Matching (Marr, Poggio and Grimson, 1979-81)
Correspondence problem
[FP] Chapters: 7
![Page 18: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/18.jpg)
p
Correlation Methods (1970--)
210 195 150 209
Image 2image 1
p '
148
What’s the problem with this?
![Page 19: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/19.jpg)
Correlation Methods (1970--)
• Pick up a window W around
image 1
p
p = (u,v)
![Page 20: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/20.jpg)
Correlation Methods (1970--)
W
• Pick up a window W around • Build vector w• Slide the window W’ along v = in image 2 and compute w’• Compute the dot product w w’ for each u and retain the maximum value
image 1
v
u
v
v
u+d
p = (u,v)
![Page 21: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/21.jpg)
Correlation Methods (1970--)
W
• Pick up a window W around• Build vector w• Slide the window W’ along v = in image 2 and compute w’• Compute the dot product w w’ for each u and retain the maximum value
image 1
v
u
v
v
u+d
W’
w’
Image 2
p = (u,v)
![Page 22: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/22.jpg)
Correlation Methods (1970--)
W
image 1
v
u
v
u+d
W’
w’
Image 2
What’s the problem with this?
![Page 23: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/23.jpg)
Changes of brightness/exposure
Changes in the mean and the variance of intensity values in corresponding windows!
![Page 24: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/24.jpg)
Correlation Methods (1970--)
Maximize: (w−w)( "w − "w )(w−w) ( "w − "w ) [Eq. 2]
image 1 Image 2
v
u
v
u+d
w’
Normalized cross- correlation:
![Page 25: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/25.jpg)
Image 1 Image 2
scanline
Example
NCC
Credit slide S. Lazebnik
u
v
![Page 26: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/26.jpg)
– Smaller window- More detail- More noise
– Larger window- Smoother disparity maps- Less prone to noise
Window size = 3 Window size = 20
Effect of the window’s size
Credit slide S. Lazebnik
![Page 27: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/27.jpg)
Issues•Fore shortening effect
•Occlusions
O O’
O O’
![Page 28: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/28.jpg)
O O’
Issues
• Small error in measurements implies large error in estimating depth
• It is desirable to have small B / z ratio!
O O’
u u’ ue’ u u’ ue’
![Page 29: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/29.jpg)
•Homogeneous regions
Issues
mismatch
![Page 30: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/30.jpg)
•Repetitive patterns
Issues
![Page 31: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/31.jpg)
Correspondence problem is difficult!
- Occlusions- Fore shortening- Baseline trade-off- Homogeneous regions- Repetitive patterns
Apply non-local constraints to help enforce the correspondences
![Page 32: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/32.jpg)
• Uniqueness – For any point in one image, there should be at most
one matching point in the other image
• Ordering– Corresponding points should be in the same order in
both views
• Smoothness– Disparity is typically a smooth function of x (except in
occluding boundaries)
Non-local constraints
![Page 33: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/33.jpg)
Lecture 5 - Silvio Savarese 24-Jan-15
• Stereo systems • Rectification • Correspondence problem
• Multi-‐view geometry • The SFM problem • Affine SFM
Lecture 6Stereo Systems Multi-‐view geometry
![Page 34: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/34.jpg)
Courtesy of Oxford Visual Geometry Group
Structure from motion problem
![Page 35: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/35.jpg)
Structure from motion problem
x1j
x2j
xmj
Xj
M1
M2
Mm
Given m images of n fixed 3D points
• xij = Mi Xj , i = 1, … , m, j = 1, … , n
![Page 36: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/36.jpg)
From the mxn correspondences xij, estimate: •m projection matrices Mi
•n 3D points Xj
x1j
x2j
xmj
Xj
motionstructure
M1
M2
Mm
Structure from motion problem
![Page 37: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/37.jpg)
Affine structure from motion(simpler problem)
ImageWorld
Image
From the mxn correspondences xij, estimate: •m projection matrices Mi (affine cameras) •n 3D points Xj
![Page 38: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/38.jpg)
M =
m1
m2
m3
!
"
####
$
%
&&&&
=
m1
m2
0 0 0 1
!
"
####
$
%
&&&&
=
m1
m2
m3
!
"
####
$
%
&&&&
X =
m1Xm2 X1
!
"
####
$
%
&&&&
xE = (m1X,m2 X)
x=M X
=A2x3 b2x101x3 1
!
"##
$
%&&
magnification
!!!
"
#
$$$
%
&
=
3
2
1
mmm
x=M X !"
#$%
&=
1vbA
M=
m1
m2
m3
!
"
####
$
%
&&&&
X =
m1Xm2Xm3X
!
"
####
$
%
&&&&
xE = (m1Xm3X
, m2 Xm3X
)Perspective
Affine
= A b!"
#$X = A b!
"#$
xyz1
!
"
%%%%
#
$
&&&&
=AXE+b [Eq. 3]
![Page 39: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/39.jpg)
Xj
xij =AiX j +bi =M iX j
1
!
"##
$
%&&; M = A b!
"#$
Affine cameras
Camera matrix M for the affine case
xi j
x1j
Image i
Image 1
….
[Eq. 4]
![Page 40: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/40.jpg)
The Affine Structure-from-Motion Problem
Given m images of n fixed points Xj we can write
Problem: estimate the m 2×4 matrices Mi andthe n positions Xj from the m×n correspondences xij .
2m × n equations in 8m+3n unknowns
Two approaches:- Algebraic approach (affine epipolar geometry; estimate F; cameras; points)- Factorization method
How many equations and how many unknown?
N. of cameras N. of pointsxij =AiX j +bi =M i
X j
1
!
"##
$
%&&;
![Page 41: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/41.jpg)
A factorization method – Tomasi & Kanade algorithm
C. Tomasi and T. KanadeShape and motion from image streams under orthography: A factorization method. IJCV, 9(2):137-154, November 1992.
• Data centering • Factorization
![Page 42: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/42.jpg)
Centering: subtract the centroid of the image points
A factorization method - Centering the data
xik
∑=
−=n
kikijij n 1
1ˆ xxx
X k
Image i
xi-
xi-
[Eq. 5]
![Page 43: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/43.jpg)
Centering: subtract the centroid of the image points
A factorization method - Centering the data
ikiik bXAx +=
( )∑=
+−+=n
kikiiji n 1
1 bXAbXA∑=
−=n
kikijij n 1
1ˆ xxx
xikxi-
[Eq. 6]
[Eq. 4]X k
![Page 44: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/44.jpg)
Centering: subtract the centroid of the image points
A factorization method - Centering the data
( )∑=
+−+=n
kikiiji n 1
1 bXAbXA∑=
−=n
kikijij n 1
1ˆ xxx
!"
#$%
&−= ∑
=
n
kkji n 1
1 XXA
xikxi-
X
jiXA ˆ= [Eq. 7]
X k
X = 1n
X kk=1
n
∑Centroid of 3D points
ikiik bXAx +=[Eq. 4]
![Page 45: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/45.jpg)
Centering: subtract the centroid of the image points
A factorization method - Centering the data
( )∑=
+−+=n
kikiiji n 1
1 bXAbXA∑=
−=n
kikijij n 1
1ˆ xxx
!"
#$%
&−= ∑
=
n
kkji n 1
1 XXA jiXA ˆ= [Eq. 7]
jiij XAx ˆˆ =
After centering, each normalized observed point is related to the 3D point by
[Eq. 8]
![Page 46: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/46.jpg)
A factorization method - Centering the data
jijiij XAXAx == ˆˆIf the centroid of points in 3D = center of the world reference system
xijxi-
X j
X
[Eq. 9]
![Page 47: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/47.jpg)
!!!!
"
#
$$$$
%
&
=
mnmm
n
n
xxx
xxxxxx
D
ˆˆˆ
ˆˆˆˆˆˆ
21
22221
11211
!"!!
cameras(2 m )
points (n )
A factorization method - factorization
Let’s create a 2m × n data (measurement) matrix:
![Page 48: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/48.jpg)
Let’s create a 2m × n data (measurement) matrix:
[ ]n
mmnmm
n
n
XXX
A
AA
xxx
xxxxxx
D !"
!#!!
212
1
21
22221
11211
ˆˆˆ
ˆˆˆˆˆˆ
!!!!
"
#
$$$$
%
&
=
!!!!
"
#
$$$$
%
&
=
cameras(2 m × 3)
points (3 × n )
The measurement matrix D = M S has rank 3(it’s a product of a 2mx3 matrix and 3xn matrix)
A factorization method - factorization
(2 m × n) MS
![Page 49: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/49.jpg)
Factorizing the Measurement Matrix
= ×
2m
n 3
n
3Measurements Motion Structure
D =MS
![Page 50: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/50.jpg)
• Singular value decomposition of D:
=2m
n n
n n
× ×
n
D U W VT
Factorizing the Measurement Matrix
![Page 51: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/51.jpg)
Since rank (D)=3, there are only 3 non-zero singular values
Factorizing the Measurement Matrix
![Page 52: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/52.jpg)
M = Motion (cameras)
S = structure
Factorizing the Measurement Matrix
![Page 53: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/53.jpg)
Theorem: When has a rank greater than is the best possible rank-‐ approximation of D in the sense of the Frobenius norm.
D 3, U3W3V3T
3
3 3 3T=D U WV
M ≈ U3
S≈W3V3T
"#$
%$
What is the issue here?
Factorizing the Measurement Matrix
• measurement noise • affine approximation
D has rank>3 because of:
M
S
![Page 54: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/54.jpg)
Affine Ambiguity
• The decomposition is not unique. We get the same D by using any 3×3 matrix C and applying the transformations M → MC, S →C-‐1S
• We can enforce some Euclidean constraints to resolve this ambiguity (more on next lecture!)
= ×D M S
![Page 55: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/55.jpg)
Reconstruction results
C. Tomasi and T. Kanade. Shape and motion from image streams under orthography: A factorization method. IJCV, 9(2):137-154, November 1992.
![Page 56: lecture6 affine SFM-2 - cvgl.stanford.edu · 1, e 2 • Epipolar Lines • Baseline Epipolar geometry O O’ ... lecture6_affine_SFM-2 Author: Christopher B. Choy Created Date: 1/27/2015](https://reader035.vdocuments.net/reader035/viewer/2022062607/604dc8dace425f39f60d3763/html5/thumbnails/56.jpg)
Next lecture Multiple view geometryPerspective structure from Motion