cs 395/495-26: spring 2004
DESCRIPTION
CS 395/495-26: Spring 2004. IBMR: Projective 2D DLT Jack Tumblin [email protected]. Recall: Projective Transform H. x. y. Apply the 3x3 matrix H x’ = Hx. 2D image (x,y) Homog. coords [x,y, w ] T = x w=1. Homog. coords x’ = [x’,y’, w’ ] T w’=1 2D image (x’,y’). - PowerPoint PPT PresentationTRANSCRIPT
CS 395/495-26: Spring 2004CS 395/495-26: Spring 2004
IBMR:IBMR:Projective 2D DLTProjective 2D DLT
Jack TumblinJack Tumblin
[email protected]@cs.northwestern.edu
x’x’
Apply the Apply the 3x3 matrix 3x3 matrix HH
x’ = Hx x’ = Hx
Recall: Projective Transform Recall: Projective Transform HH
xx33
xx22
xx11
xx
yy
x’x’
y’y’
(x(x/w/w,y,y/w/w)) xx
(x’(x’/w’/w’,y’,y’/w’/w’))
Homog. coordsHomog. coords x’ =x’ = [x’,y’,[x’,y’,w’w’]]TT
w’=1w’=1
2D image 2D image (x’,y’)(x’,y’)
2D image 2D image (x,y)(x,y)
Homog. coords Homog. coords
[x,y,[x,y,ww]]TT = x = x w=1w=1
Recall: Projective Transform: Recall: Projective Transform: Hx = x’Hx = x’
xx11
xx22
xx33
hh11 11 h h1212 h h1313
hh2121 h h2222 h h2121
hh3131 h h32 32 h h3333
= = x’x’11
x’x’22
x’x’33
H defines point pairs (x,x’) H defines point pairs (x,x’) (correspondences)(correspondences)
x’ = =x’ = =
y’ = =y’ = =
= = xx
yy
ww
Input (or output) Input (or output) image is on image is on central planecentral plane
xx11
xx22
xx33
hh21 21 x + hx + h22 22 y + hy + h2121
hh31 31 x + hx + h32 32 y + hy + h3333
x’x’11
x’x’33
x’x’22
x’x’33
Write RWrite R22 expressions; expressions;(for each x,x’ pair)(for each x,x’ pair)
hh11 11 x + hx + h12 12 y + hy + h1313
hh31 31 x + hx + h32 32 y + hy + h3333
x’ (hx’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))
y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))Rearrange, solve as Rearrange, solve as a matrix problem…a matrix problem…
x = [x = [xx11, x, x22, x, x33] notation ] notation
is confusing for multiple points!is confusing for multiple points!
Instead, let’s use [x,y,w]:Instead, let’s use [x,y,w]:(so we can use subscripts(so we can use subscripts to number the point pairs x,x’…)to number the point pairs x,x’…)
Recall: Projective Transform: Recall: Projective Transform: Hx = x’Hx = x’
xx11
xx22
xx33
hh11 11 h h1212 h h1313
hh2121 h h2222 h h2121
hh3131 h h32 32 h h3333
= = x’x’11
x’x’22
x’x’33
H defines point pairs (x,x’) H defines point pairs (x,x’) (correspondences)(correspondences)
x’ = =x’ = =
y’ = =y’ = =
= = xx
yy
ww
Input (or output) Input (or output) image is on image is on central planecentral plane
xx11
xx22
xx33
hh21 21 x + hx + h22 22 y + hy + h2121
hh31 31 x + hx + h32 32 y + hy + h3333
x’x’11
x’x’33
x’x’22
x’x’33
Write RWrite R22 expressions; expressions;(for each x,x’ pair)(for each x,x’ pair)
hh11 11 x + hx + h12 12 y + hy + h1313
hh31 31 x + hx + h32 32 y + hy + h3333
x’ (hx’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))
y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))Rearrange, solve as Rearrange, solve as a matrix problem…a matrix problem…
let w=1let w=1
Recall: Projective Transform: Recall: Projective Transform: Hx = x’Hx = x’
xx11
xx22
xx33
hh11 11 h h1212 h h1313
hh2121 h h2222 h h2121
hh3131 h h32 32 h h3333
= = x’x’11
x’x’22
x’x’33
Finding H from point pairs Finding H from point pairs (correspondences)(correspondences)
x’ = =x’ = =
y’ = =y’ = =
= = xx
yy
1 1
Input (or output) Input (or output) image is on image is on central planecentral plane
xx11
xx22
xx33
hh21 21 x + hx + h22 22 y + hy + h2121
hh31 31 x + hx + h32 32 y + hy + h3333
x’x’11
x’x’33
x’x’22
x’x’33
Write RWrite R22 expressions; expressions;(for each x,x’ pair)(for each x,x’ pair)
hh11 11 x + hx + h12 12 y + hy + h1313
hh31 31 x + hx + h32 32 y + hy + h3333
x’ (hx’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))
y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))Rearrange, solve as Rearrange, solve as a matrix problem…a matrix problem…
Naïve Method:Naïve Method:
4 point correspondence:4 point correspondence:Book shows:Book shows:
Rearrange: Rearrange: knownknown vector (dot) vector (dot) unknownunknown vector vector
stack, solve for null space…stack, solve for null space…
But this assumes But this assumes xx11=x, x=x, x22=y, =y, xx33=w=1=w=1,,
There is a better way (DLT)…There is a better way (DLT)…
x’ (hx’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))
y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))
hh1111
hh1212
hh13 13
hh2121 hh2222
hh2323
hh3131 hh3232
hh3333
[x y 1 0 0 0 -x’x -x’y -x’] = 0[x y 1 0 0 0 -x’x -x’y -x’] = 0[0 0 0 x y 1 -y’x -y’y -y’][0 0 0 x y 1 -y’x -y’y -y’]
Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)
• Naive method: Solve Hx – x’ = 0Naive method: Solve Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 always)=1 always)
• compare:compare:
• A BETTER WAY:A BETTER WAY:DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– Accepts any scale (xAccepts any scale (x33=w=w1 OK…)1 OK…)
[ 0 0 0 x y 1 -y’x -y’y -y’ ] [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ]
hh1111
hh1212
hh1313
hh2121
hh2222
hh2323
hh3131
hh3232
hh3333
[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w] [w’x w’y w’w 0 0 0 -x’x -x’y -x’w]
Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)
• Naive method: Solve Hx – x’ = 0Naive method: Solve Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 always)=1 always)
• Compare to DLT:Compare to DLT:
• DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– Accepts Accepts anyany scale (x scale (x33=w=w1, x1, x33’=w’’=w’1)1)
[ 0 0 0 x y 1 -y’x -y’y -y’ ] [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ][ x y 1 0 0 0 -x’x -x’y -x’ ]
hh11
hh22
hh33
[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w][w’x w’y w’w 0 0 0 -x’x -x’y -x’w]
[[ 00TT --ww’’i i xxiiTT yy’’I I xxii
TT ]][[ ww’’i i xxii
TT 00TT - -xx’’i i xxiiTT ]]
Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)
• DLT method: Solve H DLT method: Solve H x = 0x = 0– Accepts any scale, any point (xAccepts any scale, any point (x33=w=w1 is OK)1 is OK)
– ‘‘Pure’, Compatible –uses only Pure’, Compatible –uses only PP22 terms.terms.– Much better suited to error measurements.Much better suited to error measurements.
• Subtlety:Subtlety:– Won’t divide-by-zero if Won’t divide-by-zero if w=0w=0 or or hh3333=0=0 (it happens!)(it happens!)
– has a 3has a 3rdrd row; it is not degenerate if/when w’=0 row; it is not degenerate if/when w’=0– OK to use it…OK to use it…
(Solve 8x12)(Solve 8x12) hh11
hh22
hh33
00TT - -ww’’i i xxiiTT yy’’I I xxii
TT ww’’i i xxii
TT 00TT - -xx’’i i xxiiTT =0 =0
yy’’i i xxiiTT - -xx’’i i xxii
TT 00TT
STOP:STOP:Learn about SVDs …Learn about SVDs …
Matrix Math & SVDsMatrix Math & SVDs
• Matrix multiply is a change of coordinatesMatrix multiply is a change of coordinates– Rows of A = coordinate axis vectorsRows of A = coordinate axis vectors– Ax = a dot product for each axisAx = a dot product for each axis
• input spaceinput spaceoutput spaceoutput space• SVD ‘factors’ matrix A into 3 parts:SVD ‘factors’ matrix A into 3 parts:
– U columns: orthonormal axes for OUTPUTU columns: orthonormal axes for OUTPUT– V columns: orthonormal axes for INPUTV columns: orthonormal axes for INPUT– S: scale factors VS: scale factors VUU
svd(A) = U S Vsvd(A) = U S VTT
Actual ‘Robustness’Actual ‘Robustness’
• Vectorizing Vectorizing (‘Flatten, Stack, Null Space’ method) (‘Flatten, Stack, Null Space’ method)
works for almost ANY input! works for almost ANY input! (Points, lines, planes, ||, (Points, lines, planes, ||, , conics, quadrics, , conics, quadrics, cross-ratios, vanishing points, twisted cubics…cross-ratios, vanishing points, twisted cubics…
• Key to DLT formulation: (Hx Key to DLT formulation: (Hx x’) = 0 x’) = 0
• Rearrange as dot product: Rearrange as dot product: (known)(known)··(unknown) = 0(unknown) = 0
• Be careful to have ENOUGH constraintsBe careful to have ENOUGH constraints tricky when you mix types: points, lines,… tricky when you mix types: points, lines,… (pg 75)(pg 75)
measured inputmeasured inputtransformedtransformed
measured outputmeasured output
= 0= 0
Deceptions: false ‘Robustness’…Deceptions: false ‘Robustness’…
• Suppose we have 4 pt-correspondences:Suppose we have 4 pt-correspondences:– Use DLT to write 8x9 (or 12x9) matrix Use DLT to write 8x9 (or 12x9) matrix AA: : Ah=0Ah=0– Solve for Solve for h h null space. null space. ALWAYSALWAYS gives gives HH matrix matrix
• But what if points are bad / fictional?But what if points are bad / fictional?– 3 collinear input pts, crooked out: 3 collinear input pts, crooked out: IMPOSSIBLE!IMPOSSIBLE!
– Yet we get an Yet we get an HH solution! Why? solution! Why?
• AA matrix rank is 7 or 6 matrix rank is 7 or 6 rank 1 rank 1 HH result(s) result(s)– ‘‘Null Space’ of Null Space’ of AA may contain >1 may contain >1 HH solution! solution!
– ‘‘Degenerate’ Degenerate’ HH solution of form solution of form aaHH11 + + bbHH22……
• Answer: SVD ranks Answer: SVD ranks AA; ; rejectreject bad point sets. bad point sets.
Most Important SVD IdeasMost Important SVD Ideas
SVD solves Null-Space problems SVD solves Null-Space problems Ax = 0Ax = 0 (‘flatten, (‘flatten,
stack, find null space’ to solve) stack, find null space’ to solve) – MANY problems can be written this wayMANY problems can be written this way– Expandable! Expandable!
--more data points--more data points tall, ‘overconstrained’ A matrix tall, ‘overconstrained’ A matrix --SVD finds optimal ‘least squares’ solution.--SVD finds optimal ‘least squares’ solution.
Robustness: bane of all inverse methodsRobustness: bane of all inverse methodsAccuracy vs. # of measurements Accuracy vs. # of measurements (more isn’t always better! Outliers...)(more isn’t always better! Outliers...)
Data-rich images: Data-rich images: quantityquantity easier than easier than qualityquality moremore is easier than is easier than betterbetter) )
?What links measurement errors?What links measurement errorsHH errors? errors??How can more measurements reduce error??How can more measurements reduce error?
HowHow else else can we use DLT? can we use DLT?
• I know this building I know this building consists of MANY parallel consists of MANY parallel and perpendicular lines, and perpendicular lines, corners, regularly spaced corners, regularly spaced points.points.
• How many How many differentdifferent ways ways can we find to ‘rectify’ this can we find to ‘rectify’ this image to get this one?image to get this one?
Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)
• Naïve method: Solves Hx – x’ = 0Naïve method: Solves Hx – x’ = 0– Uses constant scale (xUses constant scale (x33=w=1 in measurement)=w=1 in measurement)
• Compare to DLT:Compare to DLT:
• DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– (a different identity: x’ (a different identity: x’ x’ = 0 x’ = 0))
– Accepts any scale (xAccepts any scale (x33=w=w1 OK, xx1 OK, xx33))
[ 0 0 0 x y 1 -y’x -y’y -y’ ] [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ]
hh1111
hh1212
hh1313
hh2121
hh2222
hh2323
hh3131
hh3232
hh3333
[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w] [w’x w’y w’w 0 0 0 -x’x -x’y -x’w]
Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)
• Naïve method: Solve Hx – x’ = 0Naïve method: Solve Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 always)=1 always)
• compare:compare:
• DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– Accepts Accepts anyany scale (x scale (x33=w=w1, x1, x33’=w’’=w’1)1)
[ 0 0 0 x y 1 -y’x -y’y -y’ ] [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ]
hh11
hh22
hh33
[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w][w’x w’y w’w 0 0 0 -x’x -x’y -x’w]
[[ 00TT --ww’’i i xxiiTT yy’’I I xxii
TT ]][[ ww’’i i xxii
TT 00TT - -xx’’i i xxiiTT ]]
Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)
• DLT method: Solve H DLT method: Solve H x = 0x = 0– Accepts any scale, any point (xAccepts any scale, any point (x33=w=w1 is OK)1 is OK)
– ‘‘Pure’ and Compatible -- Pure’ and Compatible -- PP22 terms only, no Rterms only, no R22
– MuchMuch better suited to error measurements. better suited to error measurements.
• Subtlety:Subtlety:– Won’t divide-by-zero if Won’t divide-by-zero if w=0w=0 or or hh3333=0=0 (it happens!)(it happens!)
– has a 3has a 3rdrd row; it is row; it is notnot degenerate if w’=0 degenerate if w’=0– OK to use it…OK to use it…
(Solve 8x12)(Solve 8x12) hh11
hh22
hh33
00TT - -ww’’i i xxiiTT yy’’I I xxii
TT ww’’i i xxii
TT 00TT - -xx’’i i xxiiTT =0 =0
--yy’’i i xxiiTT - -xx’’i i xxii
TT 00TT
Adding More MeasurementsAdding More Measurements
How can we use How can we use >4>4 point correspondences? point correspondences?• Easy:Easy:
– Add more rows to our 8x9 matrix A: Add more rows to our 8x9 matrix A: AA··h = 0h = 0– Use SVD to find Null space Use SVD to find Null space (Always gives an answer!)(Always gives an answer!)
– Result:Result: ‘Least squares’ solution; error= ‘Least squares’ solution; error=AA··h = h = • minimizes || minimizes || A hA h || ||22 = = ii 22
i i
wherewhere,,ii is error for is error for ii-th pt. -th pt. correspondence: correspondence:
i i = || Hx= || Hxii x xii’ ||’ ||2 2 = = || (2 rows of A)*h |||| (2 rows of A)*h ||2 2 = ‘algebraic distance’= ‘algebraic distance’
– ‘‘Algebraic Distance’ ?Algebraic Distance’ ? No geometric meaning! No geometric meaning!
Adding More MeasurementsAdding More Measurements
• 2D‘Algebraic Distance’ ?2D‘Algebraic Distance’ ? No geometric meaning! No geometric meaning!
• 2D ‘Geometric Distance’ d2D ‘Geometric Distance’ d(a,b)(a,b)22 is Better: is Better: measurable length in input or output spacemeasurable length in input or output space
if a = (aif a = (a11 a a22 a a33) and b = (b) and b = (b11 b b2 2 b b33), then define), then define
Turns out that:Turns out that:
( - )( - )22 + + ( - )( - )22 dd(a,b)(a,b)22 = = aa11
aa33
bb22
bb33
bb11
bb33
aa22
aa33
dd(a,b)(a,b)22 = d = dalgebraicalgebraic(a,b)(a,b)aa33··bb33 (Not very surprising)(Not very surprising)
Adding More MeasurementsAdding More Measurements
Iterative Strategy:Iterative Strategy:• Overconstrain the answer Overconstrain the answer HH
– Collect extra measurements (>4 point pairs, etc. …)Collect extra measurements (>4 point pairs, etc. …)– expect errors; call them ‘estimates’ xexpect errors; call them ‘estimates’ x
• Compute a 1Compute a 1stst solution (probably by SVD) solution (probably by SVD)
• Compute error Compute error d(Hx, x’)d(Hx, x’)22, and use this to…, and use this to…
• ‘‘Tweak’ answer Tweak’ answer HH andand estimates estimates xx
• Compute new answerCompute new answer
• Stop when error < useful thresholdStop when error < useful threshold
^̂
^̂
^̂
^̂
Using EstimatesUsing Estimates
• Simplest: ‘one image’ transfer method:Simplest: ‘one image’ transfer method:– Assume inputs are a perfect test pattern:Assume inputs are a perfect test pattern:
onlyonly output output pts are in error pts are in error– Adjust output estimates x’ until d(Adjust output estimates x’ until d(HHx, x’)x, x’)22 00
(note we re-compute H as x’ changes)(note we re-compute H as x’ changes)
• Better: ‘Symmetric’ transfer method:Better: ‘Symmetric’ transfer method:– Assume BOTH input Assume BOTH input xx and output and output x’x’ has error. has error.– Adjust BOTH input and output estimates Adjust BOTH input and output estimates x x’ x x’
(note we also re-compute (note we also re-compute HH as as x, x’x, x’ change) change)– Stop when d(Hx,x’)Stop when d(Hx,x’)22 + d(H + d(H-1-1x’, x)x’, x)22 is nearly 0 is nearly 0
^̂ ^̂
^̂ ^̂
^ ^ ^ ^ ^ ^ ^ ^
ENDEND