cs 395/495-26: spring 2004 ibmr: projective 2d dlt jack tumblin [email protected]

23
CS 395/495-26: Spring CS 395/495-26: Spring 2004 2004 IBMR: IBMR: Projective 2D DLT Projective 2D DLT Jack Tumblin Jack Tumblin [email protected] [email protected]

Post on 21-Dec-2015

220 views

Category:

Documents


1 download

TRANSCRIPT

CS 395/495-26: Spring 2004CS 395/495-26: Spring 2004

IBMR:IBMR:Projective 2D DLTProjective 2D DLT

Jack TumblinJack Tumblin

[email protected]@cs.northwestern.edu

x’x’

Apply the Apply the 3x3 matrix 3x3 matrix HH

x’ = Hx x’ = Hx

Recall: Projective Transform Recall: Projective Transform HH

xx33

xx22

xx11

xx

yy

x’x’

y’y’

(x(x/w/w,y,y/w/w)) xx

(x’(x’/w’/w’,y’,y’/w’/w’))

Homog. coordsHomog. coords x’ =x’ = [x’,y’,[x’,y’,w’w’]]TT

w’=1w’=1

2D image 2D image (x’,y’)(x’,y’)

2D image 2D image (x,y)(x,y)

Homog. coords Homog. coords

[x,y,[x,y,ww]]TT = x = x w=1w=1

Recall: Projective Transform: Recall: Projective Transform: Hx = x’Hx = x’

xx11

xx22

xx33

hh11 11 h h1212 h h1313

hh2121 h h2222 h h2121

hh3131 h h32 32 h h3333

= = x’x’11

x’x’22

x’x’33

H defines point pairs (x,x’) H defines point pairs (x,x’) (correspondences)(correspondences)

x’ = =x’ = =

y’ = =y’ = =

= = xx

yy

ww

Input (or output) Input (or output) image is on image is on central planecentral plane

xx11

xx22

xx33

hh21 21 x + hx + h22 22 y + hy + h2121

hh31 31 x + hx + h32 32 y + hy + h3333

x’x’11

x’x’33

x’x’22

x’x’33

Write RWrite R22 expressions; expressions;(for each x,x’ pair)(for each x,x’ pair)

hh11 11 x + hx + h12 12 y + hy + h1313

hh31 31 x + hx + h32 32 y + hy + h3333

x’ (hx’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))

y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))Rearrange, solve as Rearrange, solve as a matrix problem…a matrix problem…

x = [x = [xx11, x, x22, x, x33] notation ] notation

is confusing for multiple points!is confusing for multiple points!

Instead, let’s use [x,y,w]:Instead, let’s use [x,y,w]:(so we can use subscripts(so we can use subscripts to number the point pairs x,x’…)to number the point pairs x,x’…)

Recall: Projective Transform: Recall: Projective Transform: Hx = x’Hx = x’

xx11

xx22

xx33

hh11 11 h h1212 h h1313

hh2121 h h2222 h h2121

hh3131 h h32 32 h h3333

= = x’x’11

x’x’22

x’x’33

H defines point pairs (x,x’) H defines point pairs (x,x’) (correspondences)(correspondences)

x’ = =x’ = =

y’ = =y’ = =

= = xx

yy

ww

Input (or output) Input (or output) image is on image is on central planecentral plane

xx11

xx22

xx33

hh21 21 x + hx + h22 22 y + hy + h2121

hh31 31 x + hx + h32 32 y + hy + h3333

x’x’11

x’x’33

x’x’22

x’x’33

Write RWrite R22 expressions; expressions;(for each x,x’ pair)(for each x,x’ pair)

hh11 11 x + hx + h12 12 y + hy + h1313

hh31 31 x + hx + h32 32 y + hy + h3333

x’ (hx’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))

y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))Rearrange, solve as Rearrange, solve as a matrix problem…a matrix problem…

let w=1let w=1

Recall: Projective Transform: Recall: Projective Transform: Hx = x’Hx = x’

xx11

xx22

xx33

hh11 11 h h1212 h h1313

hh2121 h h2222 h h2121

hh3131 h h32 32 h h3333

= = x’x’11

x’x’22

x’x’33

Finding H from point pairs Finding H from point pairs (correspondences)(correspondences)

x’ = =x’ = =

y’ = =y’ = =

= = xx

yy

1 1

Input (or output) Input (or output) image is on image is on central planecentral plane

xx11

xx22

xx33

hh21 21 x + hx + h22 22 y + hy + h2121

hh31 31 x + hx + h32 32 y + hy + h3333

x’x’11

x’x’33

x’x’22

x’x’33

Write RWrite R22 expressions; expressions;(for each x,x’ pair)(for each x,x’ pair)

hh11 11 x + hx + h12 12 y + hy + h1313

hh31 31 x + hx + h32 32 y + hy + h3333

x’ (hx’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))

y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))Rearrange, solve as Rearrange, solve as a matrix problem…a matrix problem…

Naïve Method:Naïve Method:

4 point correspondence:4 point correspondence:Book shows:Book shows:

Rearrange: Rearrange: knownknown vector (dot) vector (dot) unknownunknown vector vector

stack, solve for null space…stack, solve for null space…

But this assumes But this assumes xx11=x, x=x, x22=y, =y, xx33=w=1=w=1,,

There is a better way (DLT)…There is a better way (DLT)…

x’ (hx’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))

y’ (hy’ (h31 31 x + hx + h32 32 y + hy + h3333) = (h) = (h11 11 x + hx + h12 12 y + hy + h1313))

hh1111

hh1212

hh13 13

hh2121 hh2222

hh2323

hh3131 hh3232

hh3333

[x y 1 0 0 0 -x’x -x’y -x’] = 0[x y 1 0 0 0 -x’x -x’y -x’] = 0[0 0 0 x y 1 -y’x -y’y -y’][0 0 0 x y 1 -y’x -y’y -y’]

Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)

• Naive method: Solve Hx – x’ = 0Naive method: Solve Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 always)=1 always)

• compare:compare:

• A BETTER WAY:A BETTER WAY:DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– Accepts any scale (xAccepts any scale (x33=w=w1 OK…)1 OK…)

[ 0 0 0 x y 1 -y’x -y’y -y’ ] [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ]

hh1111

hh1212

hh1313

hh2121

hh2222

hh2323

hh3131

hh3232

hh3333

[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w] [w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)

• Naive method: Solve Hx – x’ = 0Naive method: Solve Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 always)=1 always)

• Compare to DLT:Compare to DLT:

• DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– Accepts Accepts anyany scale (x scale (x33=w=w1, x1, x33’=w’’=w’1)1)

[ 0 0 0 x y 1 -y’x -y’y -y’ ] [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ][ x y 1 0 0 0 -x’x -x’y -x’ ]

hh11

hh22

hh33

[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w][w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

[[ 00TT --ww’’i i xxiiTT yy’’I I xxii

TT ]][[ ww’’i i xxii

TT 00TT - -xx’’i i xxiiTT ]]

Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)

• DLT method: Solve H DLT method: Solve H x = 0x = 0– Accepts any scale, any point (xAccepts any scale, any point (x33=w=w1 is OK)1 is OK)

– ‘‘Pure’, Compatible –uses only Pure’, Compatible –uses only PP22 terms.terms.– Much better suited to error measurements.Much better suited to error measurements.

• Subtlety:Subtlety:– Won’t divide-by-zero if Won’t divide-by-zero if w=0w=0 or or hh3333=0=0 (it happens!)(it happens!)

– has a 3has a 3rdrd row; it is not degenerate if/when w’=0 row; it is not degenerate if/when w’=0– OK to use it…OK to use it…

(Solve 8x12)(Solve 8x12) hh11

hh22

hh33

00TT - -ww’’i i xxiiTT yy’’I I xxii

TT ww’’i i xxii

TT 00TT - -xx’’i i xxiiTT =0 =0

yy’’i i xxiiTT - -xx’’i i xxii

TT 00TT

STOP:STOP:Learn about SVDs …Learn about SVDs …

Matrix Math & SVDsMatrix Math & SVDs

• Matrix multiply is a change of coordinatesMatrix multiply is a change of coordinates– Rows of A = coordinate axis vectorsRows of A = coordinate axis vectors– Ax = a dot product for each axisAx = a dot product for each axis

• input spaceinput spaceoutput spaceoutput space• SVD ‘factors’ matrix A into 3 parts:SVD ‘factors’ matrix A into 3 parts:

– U columns: orthonormal axes for OUTPUTU columns: orthonormal axes for OUTPUT– V columns: orthonormal axes for INPUTV columns: orthonormal axes for INPUT– S: scale factors VS: scale factors VUU

svd(A) = U S Vsvd(A) = U S VTT

Actual ‘Robustness’Actual ‘Robustness’

• Vectorizing Vectorizing (‘Flatten, Stack, Null Space’ method) (‘Flatten, Stack, Null Space’ method)

works for almost ANY input! works for almost ANY input! (Points, lines, planes, ||, (Points, lines, planes, ||, , conics, quadrics, , conics, quadrics, cross-ratios, vanishing points, twisted cubics…cross-ratios, vanishing points, twisted cubics…

• Key to DLT formulation: (Hx Key to DLT formulation: (Hx x’) = 0 x’) = 0

• Rearrange as dot product: Rearrange as dot product: (known)(known)··(unknown) = 0(unknown) = 0

• Be careful to have ENOUGH constraintsBe careful to have ENOUGH constraints tricky when you mix types: points, lines,… tricky when you mix types: points, lines,… (pg 75)(pg 75)

measured inputmeasured inputtransformedtransformed

measured outputmeasured output

= 0= 0

Deceptions: false ‘Robustness’…Deceptions: false ‘Robustness’…

• Suppose we have 4 pt-correspondences:Suppose we have 4 pt-correspondences:– Use DLT to write 8x9 (or 12x9) matrix Use DLT to write 8x9 (or 12x9) matrix AA: : Ah=0Ah=0– Solve for Solve for h h null space. null space. ALWAYSALWAYS gives gives HH matrix matrix

• But what if points are bad / fictional?But what if points are bad / fictional?– 3 collinear input pts, crooked out: 3 collinear input pts, crooked out: IMPOSSIBLE!IMPOSSIBLE!

– Yet we get an Yet we get an HH solution! Why? solution! Why?

• AA matrix rank is 7 or 6 matrix rank is 7 or 6 rank 1 rank 1 HH result(s) result(s)– ‘‘Null Space’ of Null Space’ of AA may contain >1 may contain >1 HH solution! solution!

– ‘‘Degenerate’ Degenerate’ HH solution of form solution of form aaHH11 + + bbHH22……

• Answer: SVD ranks Answer: SVD ranks AA; ; rejectreject bad point sets. bad point sets.

Most Important SVD IdeasMost Important SVD Ideas

SVD solves Null-Space problems SVD solves Null-Space problems Ax = 0Ax = 0 (‘flatten, (‘flatten,

stack, find null space’ to solve) stack, find null space’ to solve) – MANY problems can be written this wayMANY problems can be written this way– Expandable! Expandable!

--more data points--more data points tall, ‘overconstrained’ A matrix tall, ‘overconstrained’ A matrix --SVD finds optimal ‘least squares’ solution.--SVD finds optimal ‘least squares’ solution.

Robustness: bane of all inverse methodsRobustness: bane of all inverse methodsAccuracy vs. # of measurements Accuracy vs. # of measurements (more isn’t always better! Outliers...)(more isn’t always better! Outliers...)

Data-rich images: Data-rich images: quantityquantity easier than easier than qualityquality moremore is easier than is easier than betterbetter) )

?What links measurement errors?What links measurement errorsHH errors? errors??How can more measurements reduce error??How can more measurements reduce error?

HowHow else else can we use DLT? can we use DLT?

• I know this building I know this building consists of MANY parallel consists of MANY parallel and perpendicular lines, and perpendicular lines, corners, regularly spaced corners, regularly spaced points.points.

• How many How many differentdifferent ways ways can we find to ‘rectify’ this can we find to ‘rectify’ this image to get this one?image to get this one?

Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)

• Naïve method: Solves Hx – x’ = 0Naïve method: Solves Hx – x’ = 0– Uses constant scale (xUses constant scale (x33=w=1 in measurement)=w=1 in measurement)

• Compare to DLT:Compare to DLT:

• DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– (a different identity: x’ (a different identity: x’ x’ = 0 x’ = 0))

– Accepts any scale (xAccepts any scale (x33=w=w1 OK, xx1 OK, xx33))

[ 0 0 0 x y 1 -y’x -y’y -y’ ] [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ]

hh1111

hh1212

hh1313

hh2121

hh2222

hh2323

hh3131

hh3232

hh3333

[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w] [w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)

• Naïve method: Solve Hx – x’ = 0Naïve method: Solve Hx – x’ = 0– Requires constant scale (xRequires constant scale (x33=1 always)=1 always)

• compare:compare:

• DLT method: Solve Hx DLT method: Solve Hx x’ = 0x’ = 0– Accepts Accepts anyany scale (x scale (x33=w=w1, x1, x33’=w’’=w’1)1)

[ 0 0 0 x y 1 -y’x -y’y -y’ ] [ 0 0 0 x y 1 -y’x -y’y -y’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ] [ x y 1 0 0 0 -x’x -x’y -x’ ]

hh11

hh22

hh33

[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[ 0 0 0 -w’x -w’y -w’w y’x y’y y’w] = 0[w’x w’y w’w 0 0 0 -x’x -x’y -x’w][w’x w’y w’w 0 0 0 -x’x -x’y -x’w]

[[ 00TT --ww’’i i xxiiTT yy’’I I xxii

TT ]][[ ww’’i i xxii

TT 00TT - -xx’’i i xxiiTT ]]

Direct Linear Tranform (DLT)Direct Linear Tranform (DLT)

• DLT method: Solve H DLT method: Solve H x = 0x = 0– Accepts any scale, any point (xAccepts any scale, any point (x33=w=w1 is OK)1 is OK)

– ‘‘Pure’ and Compatible -- Pure’ and Compatible -- PP22 terms only, no Rterms only, no R22

– MuchMuch better suited to error measurements. better suited to error measurements.

• Subtlety:Subtlety:– Won’t divide-by-zero if Won’t divide-by-zero if w=0w=0 or or hh3333=0=0 (it happens!)(it happens!)

– has a 3has a 3rdrd row; it is row; it is notnot degenerate if w’=0 degenerate if w’=0– OK to use it…OK to use it…

(Solve 8x12)(Solve 8x12) hh11

hh22

hh33

00TT - -ww’’i i xxiiTT yy’’I I xxii

TT ww’’i i xxii

TT 00TT - -xx’’i i xxiiTT =0 =0

--yy’’i i xxiiTT - -xx’’i i xxii

TT 00TT

Adding More MeasurementsAdding More Measurements

How can we use How can we use >4>4 point correspondences? point correspondences?• Easy:Easy:

– Add more rows to our 8x9 matrix A: Add more rows to our 8x9 matrix A: AA··h = 0h = 0– Use SVD to find Null space Use SVD to find Null space (Always gives an answer!)(Always gives an answer!)

– Result:Result: ‘Least squares’ solution; error= ‘Least squares’ solution; error=AA··h = h = • minimizes || minimizes || A hA h || ||22 = = ii 22

i i

wherewhere,,ii is error for is error for ii-th pt. -th pt. correspondence: correspondence:

i i = || Hx= || Hxii x xii’ ||’ ||2 2 = = || (2 rows of A)*h |||| (2 rows of A)*h ||2 2 = ‘algebraic distance’= ‘algebraic distance’

– ‘‘Algebraic Distance’ ?Algebraic Distance’ ? No geometric meaning! No geometric meaning!

Adding More MeasurementsAdding More Measurements

• 2D‘Algebraic Distance’ ?2D‘Algebraic Distance’ ? No geometric meaning! No geometric meaning!

• 2D ‘Geometric Distance’ d2D ‘Geometric Distance’ d(a,b)(a,b)22 is Better: is Better: measurable length in input or output spacemeasurable length in input or output space

if a = (aif a = (a11 a a22 a a33) and b = (b) and b = (b11 b b2 2 b b33), then define), then define

Turns out that:Turns out that:

( - )( - )22 + + ( - )( - )22 dd(a,b)(a,b)22 = = aa11

aa33

bb22

bb33

bb11

bb33

aa22

aa33

dd(a,b)(a,b)22 = d = dalgebraicalgebraic(a,b)(a,b)aa33··bb33 (Not very surprising)(Not very surprising)

Adding More MeasurementsAdding More Measurements

Iterative Strategy:Iterative Strategy:• Overconstrain the answer Overconstrain the answer HH

– Collect extra measurements (>4 point pairs, etc. …)Collect extra measurements (>4 point pairs, etc. …)– expect errors; call them ‘estimates’ xexpect errors; call them ‘estimates’ x

• Compute a 1Compute a 1stst solution (probably by SVD) solution (probably by SVD)

• Compute error Compute error d(Hx, x’)d(Hx, x’)22, and use this to…, and use this to…

• ‘‘Tweak’ answer Tweak’ answer HH andand estimates estimates xx

• Compute new answerCompute new answer

• Stop when error < useful thresholdStop when error < useful threshold

Using EstimatesUsing Estimates

• Simplest: ‘one image’ transfer method:Simplest: ‘one image’ transfer method:– Assume inputs are a perfect test pattern:Assume inputs are a perfect test pattern:

onlyonly output output pts are in error pts are in error– Adjust output estimates x’ until d(Adjust output estimates x’ until d(HHx, x’)x, x’)22 00

(note we re-compute H as x’ changes)(note we re-compute H as x’ changes)

• Better: ‘Symmetric’ transfer method:Better: ‘Symmetric’ transfer method:– Assume BOTH input Assume BOTH input xx and output and output x’x’ has error. has error.– Adjust BOTH input and output estimates Adjust BOTH input and output estimates x x’ x x’

(note we also re-compute (note we also re-compute HH as as x, x’x, x’ change) change)– Stop when d(Hx,x’)Stop when d(Hx,x’)22 + d(H + d(H-1-1x’, x)x’, x)22 is nearly 0 is nearly 0

^̂ ^̂

^̂ ^̂

^ ^ ^ ^ ^ ^ ^ ^

ENDEND