local features: detection and...

5/10/2018

1

Local features:detection and description

May 10th, 2018

Yong Jae Lee

UC Davis

Last time

• Detecting corner‐like points in an image

2

Today

• Local invariant features– Detection of interest points

• (Harris corner detection)

• Scale invariant blob detection: LoG

– Description of local patches

• SIFT: Histograms of oriented gradients

3

5/10/2018

2

Local features: main components1) Detection: Identify the

interest points

2) Description: Extract vector feature descriptor surrounding each interest point.

3) Matching: Determine correspondence between descriptors in two views

],,[ )1()1(11 dxx x

],,[ )2()2(12 dxx x

Kristen Grauman 4

Goal: interest operator repeatability

• We want to detect (at least some of) the same points in both images.

• Yet we have to be able to run the detection procedure independently per image.

No chance to find true matches!

5

Kristen Grauman

Goal: descriptor distinctiveness

• We want to be able to reliably determine which point goes with which.

• Must provide some invariance to geometric and photometric differences between the two views.

?

6

Kristen Grauman

5/10/2018

3

Local features: main components1) Detection: Identify the

interest points

2) Description:Extract vector feature descriptor surrounding each interest point.


7

Kristen Grauman

M IxIx IxIy

IxIy IyIy

x

II x

y

II y

y

I

x

III yx

Recall: Corners as distinctive interest points

2 x 2 matrix of image derivatives (averaged in neighborhood of a point).

Notation:

8Kristen Grauman


TXXM

2

1

0

0

iii xMx

The eigenvalues of M reveal the amount of intensity change in the two principal orthogonal gradient directions in the window.

Since M is symmetric, we have

(Eigenvalue decomposition)

9

5/10/2018

4

“flat” region1 and 2 are small;

“edge”:1 >> 2

2 >> 1

“corner”:1 and 2 are large,1 ~ 2;

One way to score the cornerness:


10

Harris corner detector

1) Compute M matrix for image window surrounding each pixel to get its cornerness score.

2) Find points with large corner response (f > threshold)

3) Take the points of local maxima, i.e., perform non-maximum suppression

11

Kristen Grauman

Harris Detector: Steps

12

5/10/2018

5

Harris Detector: StepsCompute corner response f

13

Harris Detector: StepsFind points with large corner response: f > threshold

14

Harris Detector: StepsTake only the points of local maxima of f

15

5/10/2018

6

Harris Detector: Steps

16

Properties of the Harris corner detector

Rotation invariant? Yes

17


Rotation invariant?

Translation invariant?

Yes

Yes

18

5/10/2018

7


Rotation invariant?

Translation invariant?

Scale invariant?

All points will be classified as edges

Corner !

Yes

No

Yes

19

Scale invariant interest points

How can we independently select interest points in each image, such that the detections are repeatable across different scales?

20

Kristen Grauman

Automatic scale selection

Intuition: • Find scale that gives local maxima of some function

f in both position and scale.

f

region size

Image 1f

region size

Image 2

s1 s2Slide credit: Kristen Grauman 21

5/10/2018

8

What can be the “signature” function?

22

Kristen Grauman

gdx

df

2

2

f

gdx

d2

2

Edge

Second derivativeof Gaussian (Laplacian)

Edge = zero crossingof second derivative

Source: S. Seitz

Recall: Edge detection

23

From edges to blobs

• Edge = ripple

• Blob = superposition of two ripples

Spatial selection: the magnitude of the Laplacianresponse will achieve a maximum at the center ofthe blob, provided the scale of the Laplacian is“matched” to the scale of the blob

maximum

Slide credit: Lana Lazebnik

24

5/10/2018

9

Blob detection in 2D

Laplacian of Gaussian: Circularly symmetric operator for blob detection in 2D

2

2

2

22

y

g

x

gg

25

Blob detection in 2D: scale selection

Laplacian-of-Gaussian = “blob” detector2

2

2

22

y

g

x

gg

filte

r sc

ales

img1 img2 img3Bastian Leibe

26

Blob detection in 2D

We define the characteristic scale as the scale that produces peak of Laplacian response

characteristic scale

Slide credit: Lana Lazebnik

27

5/10/2018

10

Example

Kristen Grauman 28

Kristen Grauman

29

Kristen Grauman

30

5/10/2018

11

Kristen Grauman

31

Kristen Grauman

32

Kristen Grauman

33

5/10/2018

12

Kristen Grauman

34

)()( yyxx LL

List of(x, y, σ)

scale

Scale invariant interest points

Interest points are local maxima in both position and scale.

Squared filter response maps

Kristen Grauman

35

Scale-space blob detector: Example

Image credit: Lana Lazebnik

36

5/10/2018

13

We can approximate the Laplacian with a difference of Gaussians; more efficient to implement.

2 ( , , ) ( , , )xx yyL G x y G x y

( , , ) ( , , )DoG G x y k G x y

(Laplacian)

(Difference of Gaussians)

Technical detail

Slide credit: Kristen Grauman

37

Local features: main components

1) Detection: Identify the interest points

2) Description: Extract vector feature descriptor surrounding each interest point.


],,[ )1()1(11 dxx x

],,[ )2()2(12 dxx x


38

Geometric transformations

e.g. scale, translation, rotation


39

5/10/2018

14

Photometric transformations

Figure from T. Tuytelaars ECCV 2006 tutorial

40

Raw patches as local descriptors

The simplest way to describe the neighborhood around an interest point is to write down the list of intensities to form a feature vector.

But this is very sensitive to even small shifts, rotations.


41

SIFT descriptor [Lowe 2004]

Use histograms to bin pixels within sub-patches according to their orientation.

0 2

Why subpatches?

Why does SIFT have some illumination invariance?


42

5/10/2018

15

CSE 576: Computer Vision

Making descriptor rotation invariant

Image from Matthew Brown

• Rotate patch according to its dominant gradient orientation

• This puts the patches into a canonical orientation.43

• Extraordinarily robust matching technique• Can handle changes in viewpoint

• Up to about 60 degree out of plane rotation

• Can handle significant changes in illumination• Sometimes even day vs. night (below)

• Fast and efficient—can run in real time

• Lots of code available• http://people.csail.mit.edu/albert/ladypack/wiki/index.php/Known_implementations_of_SIFT

Steve Seitz

SIFT descriptor [Lowe 2004]

44

Example

NASA Mars Rover images

45

5/10/2018

16

NASA Mars Rover imageswith SIFT feature matchesFigure by Noah Snavely

Example

46

SIFT descriptor properties

Invariant to

• Scale

• Rotation

Partially invariant to

• Illumination changes

• Camera viewpoint

• Occlusion, clutter

47

Local features: main components

1) Detection: Identify the interest points

2) Description:Extract vector feature descriptor surrounding each interest point.



48

5/10/2018

17

Matching local features

Kristen Grauman

49

Matching local features

?

To generate candidate matches, find patches that have the most similar appearance (e.g., lowest SSD)

Simplest approach: compare them all, take the closest (or closest k, or within a thresholded distance)

Image 1 Image 2

Kristen Grauman

50

Ambiguous matches

To add robustness to matching, can consider ratio: distance to best match / distance to second best match

If low, first match looks good.

If high, could be ambiguous match.

Image 1 Image 2

? ? ? ?

Kristen Grauman

51

5/10/2018

18

Matching SIFT Descriptors

Nearest neighbor (Euclidean distance)

Threshold ratio of nearest to 2nd nearest descriptor

Lowe IJCV 200452

Recap: robust feature-based alignment

Source: L. Lazebnik

53


• Extract features

Source: L. Lazebnik

54

5/10/2018

19



• Compute putative matches

Source: L. Lazebnik

55




• Loop:• Hypothesize transformation T (small group of putative

matches that are related by T)

Source: L. Lazebnik

56






• Verify transformation (search for other matches consistent with T)

Source: L. Lazebnik

57

5/10/2018

20






• Verify transformation (search for other matches consistent with T)

Source: L. Lazebnik

58

Applications of local invariant features

Wide baseline stereoMotion trackingPanoramas3D reconstructionRecognition (better for instance matching)…

59

Automatic mosaicing

http://www.cs.ubc.ca/~mbrown/autostitch/autostitch.html


60

5/10/2018

21

Wide baseline stereo

[Image from T. Tuytelaars ECCV 2006 tutorial]

61

Recognition of specific objects, scenes

Rothganger et al. 2003 Lowe 2002

Schmid and Mohr 1997 Sivic and Zisserman, 2003

Kristen Grauman

62

Summary

Interest point detection• Harris corner detector

• Laplacian of Gaussian, automatic scale selection

Invariant descriptors• Rotation according to dominant gradient direction

• Histograms for robustness to small shifts and translations (SIFT descriptor)

63

5/10/2018

22

Questions?

64

local features: detection and...

Documents