multi-linear systems and invariant theory in the context of computer vision and graphics
DESCRIPTION
Multi-linear Systems and Invariant Theory in the Context of Computer Vision and Graphics Class 3: Infinitesimal Motion CS329 Stanford University. Amnon Shashua. Material We Will Cover Today. Infinitesimal Motion Model. Infinitesimal Planar Homography (8-parameter flow). - PowerPoint PPT PresentationTRANSCRIPT
Class 3 1
Multi-linear Systems and Invariant Theory
in the Context of Computer Vision and Graphics
Class 3: Infinitesimal Motion
CS329Stanford University
Amnon Shashua
Class 3 2
Material We Will Cover Today
• Infinitesimal Motion Model
• Infinitesimal Planar Homography (8-parameter flow)
• Factorization Principle for Motion/Structure Recovery
• Direct Estimation
Class 3 3
]ˆ)[(sinˆˆ)cos1()(cos wwwIR T w
Infinitesimal Motion Model
0 d
1)cos( d
0)sin( d
][]ˆ)[( wIwdIR
Rodriguez Formula:
Class 3 4
Z
Y
X
PtRPP '
Infinitesimal Motion Model
tPwI x )][(
PPdt
dPP '
tPwP x ][
Class 3 5
X
Y
Z
x
y
Z
Y
X
P
),( 00 yx
),0,0( f
0xZ
Xfx
0yZ
Yfy
Reminder:
Assume: 0,1 00 yxf
PZ
p1
Class 3 6
Infinitesimal Motion Model
tPwP x ][
ZxXZZ
XZZX
Z
X
dt
d
dt
dxu
12
Txs ),0,1( Let PsZ
u T 1
pwstsZ
u xTT ][
1
Class 3 7
Infinitesimal Motion Model
Txs ),0,1(
pwstsZdt
dxu x
TT ][1
Tyr ),1,0(
Tyxp )1,,(
pwrtrZdt
dyv x
TT ][1
Class 3 8
Infinitesimal Planar Motion(the 8-parameter flow)
1PnT 1 cZbYaX
Zcbyax
1 0Z
pwstscbyaxu xTT ][)(
pwrtrcbyaxv xTT ][)(
Class 3 9
Infinitesimal Planar Motion(the 8-parameter flow)
pwstscbyaxu xTT ][)(
pwxxttcbyax x])[,0,1())(( 31
yctbtxwatctwv )()()( 323221
23213 )()( xatwxywbt
ywbtxctatctwu )()()( 313112
xyatwywbt )()( 322
13
Class 3 10
Infinitesimal Planar Motion(the 8-parameter flow)
287321 xxyyxu
278654 yxyyxv
Note: unlike the discrete case, there is no scale factor
Class 3 11
Reconstruction of Structure/Motion(factorization principle)
pwstsZ
u xTT ][
1
pwrtrZ
v xTT ][
1
)()(][ spwpwspws TTx
T
Note:
][][][)( cabbcaabccbaT
][][][ acbcbabac
2 interchanges
1 interchanges
Class 3 12
Reconstruction of Structure/Motion(factorization principle)
pwstsZ
u xTT ][
1
pwrtrZ
v xTT ][
1
)(][ spwpws Tx
T
t
w
rZ
rp
sZ
sp
v
u
T
T
62
1
1
Class 3 13
Reconstruction of Structure/Motion(factorization principle)
Let ),( ijij vu be the “flow” of point i at image j (image 0 is ref frame)
mm
m
n
Tnnn
T
Tnnn
T
nmnn
m
nmnn
m
tt
ww
rZrp
rZrp
sZsp
sZsp
vvv
vvvuuu
uuu
61
1
62
111
111
21
11211
21
11211
...
...
)/1(
..
..
)/1(
)/1(
..
..
)/1(
.
.
...
.
.
.
.
.
....
.
.
...
.
.
.
.
.
....
SMMS
S
V
UW
y
x
Class 3 14
Reconstruction of Structure/Motion(factorization principle)
SMMS
S
V
UW
y
x
Given W, find S,M
Let KLW (using SVD)
SMLAKA ))(( 1 for some 66A
Goal: find 66A such that SKA
using the “structural” constraints on S
Class 3 15
Reconstruction of Structure/Motion(factorization principle)
Goal: find 66A such that SKA
using the “structural” constraints on S
Columns 1-3 of S are known, thus columns 1-3 of A can be determined.
Columns 4-6 of A contain 18 unknowns:
TT rZsZ )/1(,)/1( eliminate Z and one obtains 5 constraints
Class 3 16
Reconstruction of Structure/Motion(factorization principle)
Goal: find 66A such that SKA
using the “structural” constraints on S
62
ny
x
K
KKLet ],...,[ 61 AAA
0,0 45 AKAK yx because
),1,0)(/1(),,0,1)(/1( yzxZ
Class 3 17
Reconstruction of Structure/Motion(factorization principle)
0,0 45 AKAK yx because
),1,0)(/1(),,0,1)(/1( yzxZ
54 AKAK yx
iix
ix xAK
AK
)(
)(
4
6
iiy
iy yAK
AK
)(
)(
5
6
Each point provides 5 constraints,thus we need 4 points and 7 views
Class 3 18
Direct Estimation
),(2 yxgI
),(1 yxfI The grey values of images 1,2
vy
ux
y
x
Goal: find u,v per pixel
Ryx
yxfvyuxgvuS),(
2)],()ˆ,ˆ([)ˆ,ˆ(
Class 3 19
Direct Estimation
vy
ux
y
x
Ryx
yxfvyuxgvuS),(
2)],()ˆ,ˆ([)ˆ,ˆ(
Assume: )ˆ,ˆ(minarg),( ˆ,ˆ vuSvu vu
u
v
)ˆ,ˆ( vuS
v
u
We are assuming that (u,v) can befound by correlation principle (minimizingthe sum of square differences).
Class 3 20
Direct Estimation
Ryx
yxfvyuxgvuS),(
2)],()ˆ,ˆ([)ˆ,ˆ(
Taylor expansion:
)(),(),(),(),( 2 Oyxvgyxugyxgvyuxg yx
Ryx
tyx IgvguvuS),(
2]ˆˆ[)ˆ,ˆ(
),(),(),( yxgyxfyxI t
Class 3 21
Direct Estimation
Ryx
tyx IIvIuvuS),(
2]ˆˆ[)ˆ,ˆ(
tI
yx II ,
image 1 minus image 2
gradient of image 2
Ryx
tyxvu IvIuI),(
2, ][min
yx
yx
II
II
A
.
.
.
.
.
.
v
ux
t
t
I
I
b
.
.
.
Class 3 22
Direct Estimation
Ryx
tyxvu IvIuI),(
2, ][min
2||min bAxx
bAAxA TT
ty
tx
yyx
yxx
II
II
v
u
III
III2
2
“aperture problem” 1)( AArank T
Class 3 23
Direct Estimation
Estimating parametric flow:
287321 xxyyxu
278654 yxyyxv
0(....))( 287321 tyx IIIxxyyx
Every pixel contributes one linear equation for the 8 unknowns
Class 3 24
Direct Estimation
Estimating 3-frame Motion:
pwstsZ
u xTT ][
1
pwrtrZ
v xTT ][
1
Combine with: 0 tyx IvIuI
0][)()(1
tT
yxT
yx IpwrIsItrIsIZ
Class 3 25
Direct Estimation
0][)()(1
tT
yxT
yx IpwrIsItrIsIZ
Let
xy
yxx
yyx
yIxI
xyIIxI
IIyxyI
hpq 2
2
yx
y
x
yx
yIxI
I
I
rIsIh
01
tTT Iqwth
Z
Class 3 26
Direct Estimation
01
tTT Iqwth
Z
0'''1
tTT Iqwth
Z
image 1 to image 2
image 1 to image 3
0]''['' qwttwhthIthI TTTTt
Tt
Each pixel contributes a linear equation to the 15 unknown parameters
Class 3 27
Direct Estimation: Factorization
Let ),( ijij vu
mm
m
n
Tnnn
T
Tnnn
T
nmnn
m
nmnn
m
tt
ww
rZrp
rZrp
sZsp
sZsp
vvv
vvvuuu
uuu
61
1
62
111
111
21
11211
21
11211
...
...
)/1(
..
..
)/1(
)/1(
..
..
)/1(
.
.
...
.
.
.
.
.
....
.
.
...
.
.
.
.
.
....
SMMS
S
V
UW
y
x
be the “flow” of point i at image j (image 0 is ref frame)
Class 3 28
Direct Estimation: Factorization
02
2
t
mnnnyx IV
UII
),...,(1 nxxx IIdiagI
),...,(1 nyyy IIdiagI
mn
mtt
mtt
t
nnII
II
I
....
.
.
.
.
.
.
.
....
1
1
11
Class 3 29
ty
tx
yyx
yxx
II
II
v
u
III
III2
2
Direct Estimation: Factorization
Recall:
ij
ij
ij
ij
ii
ii
h
g
v
u
cb
ba
Class 3 30
mnmnnnH
G
V
U
CB
BA
2222
Direct Estimation: Factorization
),...,( 1 naadiagA
),...,( 1 nbbdiagB
),...,( 1 nccdiagC mnmnn
m
gg
gg
G
....
.
.
.
.
.
.
.
....
1
111
mnmnn
m
hh
hh
H
....
.
.
.
.
.
.
.
....
1
111
Class 3 31
mnmnnnH
G
V
U
CB
BA
2222
Direct Estimation: Factorization
Rank=6 Rank=6
Enforcing rank=6 constraint on the measurement matrix
H
G
removes errors in a least-squares sense.
Class 3 32
mnmnnnH
G
V
U
CB
BA
2222
Direct Estimation: Factorization
H
G
CB
BA
V
U#
Once U,V are recovered, one can solve for S,M as before.