,37&1 (8* & (truzyjw;nxnts 1&899.2*...

$: ,*3*7&1 (8* & (TRUZYJW;NXNTS 1&899.2* 74':89+.99.3,ayan/courses/cse559a/PDFs/lec...2JYJWXY T5 N]JQX 1THFYNTST SX JSXTWU QFSJN SR JYJWX 1JYIXX F^J FHMX JSXTWU N]JQN X R JYJWX\ NIJ 1THFYNTSN$
/

CSE 559A: Computer Vision

Fall 2020: T-R: 11:30-12:50pm @ Zoom

Instru‘tor: Ayan Chakrabarti ([email protected]’u).Course Staff: A’ith Boloor, Patri‘k Williams

O‘t 27, 2020

http://www.‘se.wustl.e’u/~ayan/‘ourses/‘se559a/

1 /

GENERALGENERALProblem Set 2 Due 11:59pm toni”ht.Problem Set 3 out an’ rea’y to ‘lone.Proposal repository rea’y to ‘lone.New offi‘e hours a er le‘ture (startin” at 1pm Central time).

Separate Zoom link available throu”h Zoom tab in Canvas.

2

/

GENERALGENERALNotes on Problem Set Submissions

Don t “or”et to in‘lu’e your solution writeup PDF !Make sure its ‘alle’ solution.p’“.As you solve questions, ’on t just rea’ the minimum require’ to solve the problem.

Make sure you un’erstan’ the un’erlyin” math.Go ba‘k an’ rea’ previous sli’es to see how thin”s were ’erive’.Re“resh your memories o“ linear / matrix al”ebra i’entities.

Take a‘a’emi‘ inte”rity seriously

3 /

LAST TIME: ROBUST FITTINGLAST TIME: ROBUST FITTINGIterative Version:

1. Fit the best to all samples in “ull set .

2. Given the ‘urrent estimate o“ , ‘ompute the inlier set

3. Up’ate estimate o“ by minimizin” error over only the inlier set

4. Goto step 2 Conver”es, be‘ause ‘ost never in‘reases at any iteration.But to a lo‘al minimum. It is possible that i“ you ‘hose an entirely ’ifferent inlier set, this woul’ ”ive you alower ‘ost.

Fun’amentally a ‘ombinatorial problem. Only way to solve exa‘tly is to ‘onsi’er all possible sub-sets o“ asoutlier sets.

h = arg min(ϵ, (h))min

h

∑

i∈C

E

i

h C

h = i : (h) ≤ ϵC

′

E

i

h C

′

C

4

/

RANSACRANSACRandom Sampling and Consensus

Lots o“ ’ifferent variants.

1. Ran’omly sele‘t points (‘orrespon’en‘es) as my inlier set. Choi‘e o“ ‘an vary: has to be at least 4 “or ‘omputin” homo”raphies.

2. Fit to these points.

3. Store an’ a measure o“ how ”oo’ a “it is to all points. This measure ‘an either be the threshol’e’ robust‘ost, or the number o“ outliers.

Repeat this times to ”et ’ifferent estimates o“ an’ asso‘iate’ ‘osts.

Choose the with the lowest ‘ost, an’ then use this as initialization “or the iterative al”orithm.

k

k

h k

h h

N N h

h

5 /

3D HOMOGENEOUS CO-ORDINATES3D HOMOGENEOUS CO-ORDINATESFour ’imensional ve‘tor ’e“ine’ upto s‘ale:

I“ is a “our-’imensional ve‘tor, what ’oes represent ?

p = [αx, αy, αz, α]

T

l p = 0l

T

6

/


I“ is a “our-’imensional ve‘tor, what ’oes represent ? A plane.

How ’o we represent a line ?


T

l p = 0l

T

7 /


I“ is a “our-’imensional ve‘tor, what ’oes represent ? A plane.

How ’o we represent a line ? where is a matrix.

Interpret line as interse‘tion o“ two planes.


T

l p = 0l

T

p = 0L

T

L 4 × 2

8

/

3D TRANSFORMATIONS3D TRANSFORMATIONSRepresente’ by matri‘es.

Translation

4 × 4

= pp

′

⎡

⎣

⎢

⎢

⎢

⎢

1

0

0

0

0

1

0

0

0

0

1

0

−c

x

−c

y

−c

z

1

⎤

⎦

⎥

⎥

⎥

⎥

9 /

3D TRANSFORMATIONS3D TRANSFORMATIONSRepresente’ by matri‘es.

Rotation

Where is now a matrix with .

Also ‘overs re“le‘tion. For rotation only,

Correspon’s to rotation aroun’ ea‘h axis. Not ‘ommutative.

4 × 4

=

[ ]

pp

′

R

0

T

0

1

R 3 × 3 R = IR

T

R = ( ) ( ) ( )R

x

θ

1

R

y

θ

2

R

z

θ

3

(θ) = , (θ) = , (θ) =R

x

⎡

⎣

⎢

⎢

1

0

0

0

cos θ

sin θ

0

− sin θ

cos θ

⎤

⎦

⎥

⎥

R

y

⎡

⎣

⎢

⎢

cos θ

0

sin θ

0

1

0

− sin θ

0

cos θ

⎤

⎦

⎥

⎥

R

z

⎡

⎣

⎢

⎢

cos θ

sin θ

0

− sin θ

cos θ

0

0

0

1

⎤

⎦

⎥

⎥

10

/

3D TRANSFORMATIONS3D TRANSFORMATIONSGeneral Eu‘li’ean Trans“ormation

Now is a rotation matrix, an’ is a translation ve‘tor.

=

[ ]

pp

′

R

0

T

t

1

R 3 × 3 t 3 × 1

11 /

CAMERA PROJECTIONCAMERA PROJECTION

12

/


13 /


14

/


15 /


16

/


17 /


18

/


19 /


20

/


21 /


22

/


23 /

CAMERA PROJECTIONCAMERA PROJECTIONSensor to Image Locations

This still assumes that an’ share the same ‘o-or’inate system.

What units is in ? What units is in ?

Meters to Pixels

Lo‘ation on sensor plane in meters:

Let s say ea‘h sensor pixel is meters wi’e.

Lo‘ation in pixels is

Or ‘an just assume is “o‘al len”th in pixels.

p =

⎡

⎣

⎢

⎢

f

0

0

0

f

0

0

0

1

0

0

0

⎤

⎦

⎥

⎥

p

′

p

′

p

p f

= fx

′

m

x

z

q

= /q =x

p

x

′

m

f

q

x

z

f

24

/


More General:

han’les the ‘ase where pixels aren t square (so you have in meters ’ivi’e’ by sensor wi’th an’sensor hei”ht separately)

implies the pixels are skewe’ (almost never happens).

an’ just pi‘ks the lo‘ation o“ ori”in on the ima”e plane.

O en, ok to assume , , .

p =

⎡

⎣

⎢

⎢

⎢

f

x

0

0

s

f

y

0

c

x

c

y

1

0

0

0

⎤

⎦

⎥

⎥

⎥

p

′

≠f

x

f

y

f

s ≠ 0

c

x

c

y

s = 0, = = ff

x

f

y

= W/2c

x

= H/2c

y

25 /


Still assumes that is with respe‘t to an ali”ne’ ‘o-or’inate system:

Camera ‘enter (pinhole) is at ori”in

an’ axes ali”ne’ with sensor plane

axis is viewin” ’ire‘tion

p =

⎡

⎣

⎢

⎢

⎢

f

x

0

0

s

f

y

0

c

x

c

y

1

0

0

0

⎤

⎦

⎥

⎥

⎥

p

′

p

′

x y

z

26

/


Still assumes that is with respe‘t to an ali”ne’ ‘o-or’inate system:

Camera ‘enter (pinhole) is at ori”in

an’ axes ali”ne’ with sensor plane

axis is viewin” ’ire‘tion

The matrix is ‘alle’ the intrinsi‘ ‘amera matrix.

p = = [K 0]

⎡

⎣

⎢

⎢

⎢

f

x

0

0

s

f

y

0

c

x

c

y

1

0

0

0

⎤

⎦

⎥

⎥

⎥

p

′

p

′

p

′

x y

z

K =

⎡

⎣

⎢

⎢

⎢

f

x

0

0

s

f

y

0

c

x

c

y

1

⎤

⎦

⎥

⎥

⎥

3 × 3 K

27 /

CAMERA PROJECTIONCAMERA PROJECTIONBut what i“ is in some other ‘o-or’inate system ?

Calibration tar”et (tryin” to estimate ‘amera parameters)Multi-view S‘enario

De“ine an’ are 3D homo”eneous ‘o-or’inates:

is in ‘amera ali”ne’ axes, is in worl’ axes

Both are relate’ by a eu‘li’ean / ri”i’ trans“ormation (rotation + translation)

Where is 3-D rotation matrix, an’ is translation ve‘tor.

p

′

p

″

p

′

p

″

p

′

=

[ ]

p

″

R

0

t

1

p

′

R 3 × 3 t 3 × 1

The proje‘tion matrix ‘an be “a‘torize’ into the upper trian”ular matrix intrinsi‘ matrix , an’ the

extrinsic matrix that represents ‘amera pose .

p = [K 0]

[ ]

= K [R|t] = P

R

0

t

1

p

′

p

′

p

′

P 3 × 3 K

3 × 4 [R|t]

28

/

CAMERA CALIBRATIONCAMERA CALIBRATION

’e“ine’ upto s‘ale.

Get a bun‘h o“ 3D-2D ‘orrespon’en‘es

Solve “or like “or Homo”raphies

Ex‘ept that now is a matrix instea’ o“

Nee’ six linearly in’epen’ent points (three i“ is known).

On‘e you have , ‘an ’e‘ompose into an’ usin” QR “a‘torization

Restri‘te’ versions possible i“ you assume no skew, square pixels, et‘.

p = [K 0]

[ ]

= K [R|t] = P

R

0

t

1

p

′

p

′

p

′

P

( , )p

′

i

p

i

× (P ) = 0p

i

p

′

i

P 3 × 4 3 × 3

K

P K [R|t]

29 /

CAMERA CALIBRATIONCAMERA CALIBRATION ’es‘ribes proje‘tion “rom ‘alibration obje‘t s ‘o-or’inate system

Many times we just want to estimate (or estimate it separately be“ore estimatin” pose).

Assume square pixels, no skew, opti‘al ‘enter at ‘enter o“ ima”e.

Is there a simpler way to ”et ?

P = K[R|t]

K

K =

⎡

⎣

⎢

⎢

f

0

0

0

f

0

W/2

H/2

1

⎤

⎦

⎥

⎥

f

30

/


31 /


32

/


33 /


34

/


35 /


36

/


37 /


38

/


39 /


40

/


41 /


42

/

CAMERA CALIBRATIONCAMERA CALIBRATIONVanishing Point

Alternate equation o“ line in 3D:

In 3-D ‘artesian, all points that satis“y “or some s‘alar : , where

is a 3-ve‘tor representin” the ‘artesian ‘o-or’inate o“ a point,

is a 3-ve‘tor representin” ’ire‘tion o“ line,

Same an’ ’ifferent s‘ale’ versions o“ represent same line.

Two lines with ’ifferent but same (upto s‘ale) are parallel to ea‘h other.

represents parallel line passin” throu”h ori”in.

In homo”eneous ‘o-or’inates,

Proje‘tion o“ (assumin” ‘amera-ali”ne’ ‘o-or’inate system):

r λ r = + λdr

0

r

0

d

r

0

d

r

0

d

r = λd

p = [( + λd , 1] = [( + d , ]r

0

)

T 1

λ

r

0

)

T 1

λ

p p

∼ [K 0]p = K + λKd ∼ K + Kdp

r

0

1

λ

r

0

43 /


Proje‘tion o“ (assumin” ‘amera-ali”ne’ ‘o-or’inate system):

As ,

is the 2D homo”eneous ‘o-or’inate o“ the proje‘tion o“ the point on the ”iven line at in“inity. It is theproje‘tion o“ all points on the line parallel to the ”iven line an’ passin” throu”h ori”in / ‘amera ‘enter (same

).

represents a ray in . All points in parallel line throu”h ori”in have ‘o-or’inate “or some , an’ allproje‘t to .

Note that vanishin” point will be at in“inity i“ -‘omponent o“ is 0, i.e., i“ line in 3D spa‘e is perpen’i‘ular to‘amera axis.

p p

∼ [K 0]p = K + λKd ∼ K + Kdp

r

0

1

λ

r

0

λ → ∞ ∼ Kdp

Kd

d, = 0r

0

d ℝ

3

[ , 1/λ]d

T

λ

Kd

z d

44

/


I“ s ‘artesian ‘o-or’inate is , “or simple :

So I ‘an write an equation relatin” to the ‘o-or’inate o“ it s vanishin” point an’ unknown “o‘al len”th .

But I ’on t know .

But what i“ I knew that an’ were perpen’i‘ular (in the real worl’) ?

p ∼ Kd ⇒ p ∼ dK

−1

p (x, y) K

d ∼ p ∼K

−1

⎡

⎣

⎢

⎢

(x −W/2)

(y −H/2)

f

⎤

⎦

⎥

⎥

d f

d

d

1

d

2

45 /


46

/


47 /


48

/


49 /


50

/


51 /


52

/


53 /


54

/


55 /


56

/

TWO-VIEW GEOMETRYTWO-VIEW GEOMETRY

What ‘an we say about the relationship between an’ , an’ what ’oes it say about ?

∼ [ | ]pp

1

K

1

R

1

t

1

∼ [ | ]pp

2

K

2

R

2

t

2

p

1

p

2

p

57 /


Let s just assume ,

an’ the ‘o-or’inate system is ali”ne’ with the “irst ‘amera: .

∼ [ | ]p, ∼ [ | ]pp

1

K

1

R

1

t

1

p

2

K

2

R

2

t

2

= = KK

1

K

2

= I, = 0R

1

t

1

58

/


What i“ ? Se‘on’ ima”e is “rom just rotatin” the ‘amera, but not movin” it s ‘enter.

Let . We re ”oin” to ’eal with by sayin” equal to some s‘alar “a‘tor

So i“ there s only rotation, points in two ima”es ‘an be relate’ by a Homo”raphy .

∼ K[I|0]p, ∼ K[R|t]pp

1

p

2

t = 0

p = [x, y, z, 1] ∼ , ,…λ

1

λ

2

= K[I|0]p = K[x, y, z , for some p

1

λ

1

λ

1

]

T

λ

1

= K[R|0]p = KR[x, y, z , for some p

2

λ

2

λ

2

]

T

λ

2

= KR ∼ KRp

2

λ

2

λ

1

K

−1

p

1

K

−1

p

1

= KRK

−1

59 /


60

/


61 /


62

/


63 /


64

/


65 /


66

/


67 /


68

/


69 /


70

/


71 /


72

/


73 /


74

/


75 /


76

/


77 /


78

/


79 /


80

/


81

,*3*7&1 (8* & (truzyjw;nxnts 1&899.2*...

Documents

,37&1 (8* & (truzyjw;nxnts 1&899.2*...