Local Features

Correspondence Problem & Keypoint

• Correspondence: matching points, patches, edges, or regions across images

≈

Keypoint Matching

K. Grauman, B. Leibe

Af Bf

A1

A2 A3

Tffd BA ),(

DoG – Efficient Computation

• Computation in Gaussian scale pyramid

K. Grauman, B. Leibe

s

Original image 4

1

2s

Sampling with

step s4 =2

s

s

s

Find local maxima in position-scale space of Difference-of-Gaussian

K. Grauman, B. Leibe

)()( ss yyxx LL

s

s2

s3

s4

s5

List of (x, y, s)

Results: Difference-of-Gaussian

K. Grauman, B. Leibe

Challenges in Keypoints

• Scale change

• Rotation

• Occlusion

• Illumination

……

SIFT

• Scale-space extrema detection

• Keypoint localization

• Orientation assignment

• Keypoint descriptor

Scale-space extrema detection

• Find the points, whose surrounding patches (with some scale) are distinctive

• An approximation to the scale-normalized Laplacian of Gaussian

Maxima and minima in a

3*3*3 neighborhood

Eliminating edge points

Orientation assignment

• Assign an orientation to each keypoint, the keypoint descriptor can be represented relative to this orientation and therefore achieve invariance to image rotation

• Compute magnitude and orientation on the Gaussian smoothed images

Orientation assignment

• A histogram is formed by quantizing the orientations into 36 bins;

• Peaks in the histogram correspond to the orientations of the patch;

• For the same scale and location, there could be multiple keypoints with different orientations;

T. Tuytelaars, B. Leibe

Orientation normalization

• Compute orientation histogram

• Select dominant orientation

• Normalize: rotate to fixed orientation

0 2 p

SIFT vector formation • Computed on rotated and scaled version of window

according to computed orientation & scale

– resample the window

• Based on gradients weighted by a Gaussian of variance half the window (for smooth falloff)

SIFT vector formation • 4x4 array of gradient orientation histogram weighted by

magnitude

• 8 orientations x 4x4 array = 128 dimensions

• Motivation: some sensitivity to spatial layout, but not too much.

• Gaussian weight

showing only 2x2 here but is 4x4

A result

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Maximally Stable Extremal Regions [Matas ‘02]

• Select regions that stay stable over a large parameter range

K. Grauman, B. Leibe

Example Results: MSER

39 K. Grauman, B. Leibe

Self-similarity Descriptor

Matching Local Self-Similarities across Images and Videos, Shechtman and Irani, 2007

Self-similarity Descriptor

Matching Local Self-Similarities across Images and Videos, Shechtman and Irani, 2007

Self-similarity Descriptor

Matching Local Self-Similarities across Images and Videos, Shechtman and Irani, 2007

Local Descriptors

• Most features can be thought of as templates, histograms (counts), or combinations

• The ideal descriptor should be – Robust

– Distinctive

– Compact

– Efficient

• Most available descriptors focus on edge/gradient information – Capture texture information

– Color rarely used

K. Grauman, B. Leibe

SIFT Repeatability

Lowe IJCV 2004

SIFT Repeatability

Object Recognition

Computer Vision

3D Object Recognition

3D Shape Reconstruction

RGBD Camera based object recognition

3D Model based object recognition

3D Model based object recognition

3D Model based object recognition

Paper list

“Frustum PointNets for 3D Object Detection from RGB-D Data”

Charles R. Qi, Wei Liu, Chenxia Wu, Hao Su, Leonidas J. Guibas

“You Only Look Once: Unified, Real-Time Object Detection”

Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi

“Squeeze-and-Excitation Networks”

Jie Hu, Li Shen, Gang Sun

“Adversarial Complementary Learning forWeakly Supervised Object

Localization”

Xiaolin Zhang, Yunchao Wei, Jiashi Feng, Yi Yang, Thomas Huang