machine perception cis 580 - penn engineeringderpanis/steerable... · agenda • steerable filters...

Machine Perception CIS 580

Kostas Derpanis

February 8, 2012

Thursday, February 9, 2012

Agenda

• Steerable filters (recap)

• Harris corner detector

• Binomial filters


Steerable filters

Definition: Steerable filters are a class of filters where a filter of arbitrary orientation is synthesized as a linear combination of a set of basis filters [Freeman and Adelson, 1991].

Thursday, February 9, 2012White - positive filter lobesBlack - negative filter lobes

Steerable filters

Definition: Steerable filters are a class of filters where a filter of arbitrary orientation is synthesized as a linear combination of a set of basis filters [Freeman and Adelson, 1991].

second derivative of Gaussian

Thursday, February 9, 2012White - positive filter lobesBlack - negative filter lobes

Steerable derivatives of Gaussian


Steerable filter architecture

Inputimage

Basisfilterbank

Summingjunction

Adaptivelyfiltered image

Gainmaps

Steerable Filter Architecture

ki(!)

��

i

ki(θ)Bi

�∗ I ≡

�

i

ki(θ)IBi


0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 20

0.2

0.4

0.6

0.8

1

Derivatives of Gaussian frequency spectrum

normalized magnitude spectrum

ω

n

Gn(x) ←→ (jω)nG(ω)


Agenda





Motivation

?

“What stuff in one image matches with stuff in another?”


Motivation

“What stuff in one image matches with stuff in another?”


What makes a good feature?

• Repeatability• Same feature can be found in other

images despite geometric and photometric transformations.

• Saliency• Each feature is distinctive.

• Compactness and efficiency• Many fewer features than image

pixels.

• Locality• Feature occupies a relatively small area

of the image; robust to clutter and occlusion.


Detectors

Moravec [Moravec ’77]Hessian [Beaudet ’78]Harris/Forstner [Harris ’88, Forstner ‘87]Laplacian [Lindeberg ‘98]DoG [Lowe 1999]Harris-/Hessian-Laplace [Mikolajczyk & Schmid ‘01]Harris-/Hessian-Affine [Mikolajczyk & Schmid ‘04]EBR and IBR [Tuytelaars & Van Gool ‘04] MSER [Matas ‘02]Salient Regions [Kadir & Brady ‘01] FAST [Rosten et al. ‘2010]

Many Others…


Harris corner?


Harris detector: Intuition

• Analyze local variation of signal.

• “Corner”: Shifting window in any direction yields a large change in appearance.





“flat” region:no change in all directions





“edge”:no change along the edge direction






“edge”:no change along the edge direction

“corner”:significant change in all directions



Harris detector: Derivation

(∆x,∆y)Variation of intensity for the shift :

E(∆x,∆y) =�

x,y

w(x, y)[I(x, y)− I(x+∆x, y +∆y)]2



For nearly constant patches:

E(∆x,∆y) ≈ 0

Variation of intensity for the shift :(∆x,∆y)

E(∆x,∆y) =�

x,y

w(x, y)[I(x, y)− I(x+∆x, y +∆y)]2



For “edge” patches: E(∆x,∆y)

variation large along one orientation

(∆x,∆y)Variation of intensity for the shift :

E(∆x,∆y) =�

x,y

w(x, y)[I(x, y)− I(x+∆x, y +∆y)]2



(∆x,∆y)

For “corner” patches: E(∆x,∆y)

variation large along two orientations

Variation of intensity for the shift :

E(∆x,∆y) =�

x,y

w(x, y)[I(x, y)− I(x+∆x, y +∆y)]2


Review: Taylor series

f(x+ u, y + v) = f(x, y) + fx(x, y)u+ fy(x, y)v

+1

2[fxx(x, y)u

2 + fxy(x, y)uv + fyy(x, y)v2]

+ h.o.t.


Review: Taylor series

f(x+ u, y + v) = f(x, y) + fx(x, y)u+ fy(x, y)v

+1

2[fxx(x, y)u

2 + fxy(x, y)uv + fyy(x, y)v2]

+ h.o.t.

≈



E(∆x,∆y) =�

x,y

[I(x, y)− I(x+∆x, y +∆y)]2

Thursday, February 9, 2012- Understand local signal structure for small shifts- Notice the window, w(x,y), dropped for presentation convenience

Taylor series approximation --> simplify --> rewrite


E(∆x,∆y) =�

x,y

[I(x, y)− I(x+∆x, y +∆y)]2

≈�

x,y

[I(x, y)− I(x, y)− Ix(x, y)∆x− Iy(x, y)∆y]2

Taylor series expansion




E(∆x,∆y) =�

x,y

[I(x, y)− I(x+∆x, y +∆y)]2

≈�

x,y


=�

x,y

[Ix(x, y)2∆x2 + 2Ix(x, y)Iy(x, y)∆x∆y + Iy(x, y)

2∆y ]


expand




E(∆x,∆y) =�

x,y

[I(x, y)− I(x+∆x, y +∆y)]2

rewrite as matrix

≈�

x,y


=�

x,y

[Ix(x, y)2∆x2 + 2Ix(x, y)Iy(x, y)∆x∆y + Iy(x, y)

2∆y ]


expand




E(∆x,∆y) =�∆x ∆y

�M

�∆x∆y

�.

M =�

x,y

w(x, y)

�I2x IxIyIxIy I2y

�.

where

captures the variation of the gradients within the local patchM

Thursday, February 9, 2012M is generally a positive definite matrix

Note: Windowing function reintroduced and products of derivatives NOT second derivativesM: second-order moment, covariance matrix or scatter matrix

Back to intuition

1. Treat gradient vectors as a set of 2D points, (Ix, Iy).

2. Compute scatter matrix/ellipse.

3. Analyze scatter matrix/ellipse shape.


Intuition

−5 −4 −3 −2 −1 0 1 2 3 4 5−5

−4

−3

−2

−1

0

1

2

3

4

5

−5 −4 −3 −2 −1 0 1 2 3 4 5−5

−4

−3

−2

−1

0

1

2

3

4

5

−5 −4 −3 −2 −1 0 1 2 3 4 5−5

−4

−3

−2

−1

0

1

2

3

4

5

“Flat” “Corner”“Edge”

Ix

Iy

notice distribution of gradients vary for different types of patches


Intuition

−5 −4 −3 −2 −1 0 1 2 3 4 5−5

−4

−3

−2

−1

0

1

2

3

4

5

−5 −4 −3 −2 −1 0 1 2 3 4 5−5

−4

−3

−2

−1

0

1

2

3

4

5

−5 −4 −3 −2 −1 0 1 2 3 4 5−5

−4

−3

−2

−1

0

1

2

3

4

5

“Flat” “Corner”“Edge”

λ1 >> λ2λ1 λ2≈ λ1 λ2≈small large

Ix

Iy

eigenvalues, λ1 and λ1, of scatter matrix, M, quantify variation in the principle directions (i.e., eigenvectors of M) of the data


Corner response function

r = detM− k(trace M)2

detM = λ1λ2 trace M = λ1 + λ2

where

? ?

k is empirically set constant: 0.04 - 0.06





where

?






where



Classification of image patches

λ1

λ2

“Corner”

“Edge”

“Edge”

“Flat” region

Classification of image points using eigenvalues of (or r)M



λ1

λ2

“Corner”

“Edge”

“Edge”

“Flat” region


E is almost constant in all directions

λ1 and λ2 are small|r| small



λ1

λ2

“Corner”

“Edge”

“Edge”

“Flat” region


E increases about the vertical orientation

λ1 >> λ2

r < 0

E increases about the horizontal orientation

λ2 >> λ1

r < 0





λ1

λ2

“Corner”

“Edge”

“Edge”

“Flat” region

Classification of image points using eigenvalues of (or r)

E increases in all directionsλ1 λ2

λ1 and λ2 are large, r > 0 and |r| >> 0

≈

M

E increases about the vertical orientation

λ1 >> λ2

r < 0

E increases about the horizontal orientation

λ2 >> λ1

r < 0




Harris corner example


Extension to video

Thursday, February 9, 2012Video exampleFeature detectors operating at multiple scales

Agenda





Binomial filter

• Class of smoothing filter ( Gaussian).

• Attenuates high-frequencies and retains low frequencies.

• Filter set generated by a single filter combined with itself.

≈


Starting point

• Generating filter:

• Binomial filter set:

• By the Central-Limit Theorem, the shape of the Binomial filters tend to a Gaussian as R increases.

B1 =1

2

�1 1

�.

.

�BR =

1

2

�1 1

�∗ 1

2

�1 1

�∗ . . . ∗ 1

2

�1 1

��.

Thursday, February 9, 2012Notice that the binomial filters are closed under convolution.

Sum of iid random variables tend to a Gaussian (or normal distribution) as the number of random variables gets larger.

Sum of two or more independent variables is equivalent to the convolution of their densities.

Binomial filter: Spatial domain

n

0


Equivalence to ...

1 1 B1B2B3B4B5

1 2 1

1 3 3 1

1 4 6 4 1

1 5 10 10 5 1

B6 1 6 15 20 15 6 1 ...


Equivalence to ...

1 1 B1B2B3B4B5

1 2 1

1 3 3 1

1 4 6 4 1

1 5 10 10 5 1

B6 1 6 15 20 15 6 1 ...

Pascal’s Triangle

.

�nr

�

.

nth rowrth element


Binomial derivatives

1 1

1 2 1

1 3 3 1

1 4 6 4 1

1 5 10 10 5 1...

1


-1 -2 0 2 1

-1 -3 -2 2 3 1

-1 -4 -5 0 5 4 1

-1 -1 1 1

-1 0 1

-1 1

Binomial derivatives

...


B2 =�1 2 1

�

..

Frequency domain

DTFT:

(not normalized)

Thursday, February 9, 2012Note: using i instead of j for complex number representation

B2 =�1 2 1

�

..

B(ω) =∞�

n=−∞B2[n]e

−iωn

..

Frequency domain

DTFT:

(not normalized)


B2 =�1 2 1

�

..

B(ω) =∞�

n=−∞B2[n]e

−iωn

..=

∞�

n=−∞

�δ[n+ 1] 2δ[n] δ[n+ 1]

�e−iωn

..

+ +

Frequency domain

DTFT:

(not normalized)


B2 =�1 2 1

�

..

B(ω) =∞�

n=−∞B2[n]e

−iωn

..=

∞�

n=−∞

�δ[n+ 1] 2δ[n] δ[n+ 1]

�e−iωn

..

+ +

Frequency domain

=∞�

n=−∞δ[n+ 1]e−iωn +

∞�

n=−∞2δ[n]e−iωn +

∞�

n=−∞δ[n+ 1]e−iωn

..

DTFT:

(not normalized)


B2 =�1 2 1

�

..

B(ω) =∞�

n=−∞B2[n]e

−iωn

..=

∞�

n=−∞

�δ[n+ 1] 2δ[n] δ[n+ 1]

�e−iωn

..

+ +

Frequency domain

=∞�

n=−∞δ[n+ 1]e−iωn +

∞�


∞�


..

DTFT:

= 2 + e−iω + eiω

(not normalized)


B2 =�1 2 1

�

..

B(ω) =∞�

n=−∞B2[n]e

−iωn

..=

∞�

n=−∞

�δ[n+ 1] 2δ[n] δ[n+ 1]

�e−iωn

..

+ +

Frequency domain

=∞�

n=−∞δ[n+ 1]e−iωn +

∞�


∞�


..

DTFT:

= 2 + 2 cos(ω) Euler’s formula

(not normalized)


Binomial filter frequency domain

ω

0


2D Binomial filter

• 2D Binomial filters are constructed by 1D horizontal and vertical filters.

Example:

B2D2 = Bx

R ∗ByR

.

B2D2 =

1

4

�1 2 1

�∗ 1

4

121

=1

16

1 2 12 4 21 2 1

.

separable convolution


2D Binomial filter smoothing example


machine perception cis 580 - penn engineeringderpanis/steerable... · agenda • steerable filters...

Documents