A feature-based kernel for object classification

P. Moreels - J-Y BouguetIntel

One scenario for image query : Relevance Feedback

Relevance Relevance FeedbackFeedback

ImageImageDatabaseDatabase

QueryQuery

+ “Baby with house”

SimilaritySimilarityMetricMetric

CandidateCandidateResultsResults Final ResultFinal Result

SVM classifier

???

Outline

• Motivation

• Features and distances

• Starting point: the pyramid match kernel

• Extension : the max-kernel.

• Experiments

• Conclusions

Similarity Metric between pairs of images

• Need to derive a measure of similarity between images

How similar are those two images?

sscore Similarity

Similarity Metric between pairs of images

• Need to derive a measure of similarity between images

How similar are those two images?

sscore Similarity

ss It is desired that:

Image descriptors: features

• Smaller volume of information

• Robustness to deformations (lighting, rotation, affine transformations)

• Detector: difference-of-gaussians

• Descriptor: SIFT = 128D collection of local gradients

Match-based Similarity

IDEA: First establish correspondence between the two sets of points and then compute a “distance” metric based on the matching result

STEP 1: Establish CorrespondenceSTEP 1: Establish Correspondence

2

2

2

Matches),(

2d

d

jiji

es

pp

STEP 2: Compute similaritySTEP 2: Compute similarity

PROBLEM: Until now, no matching-based image similarity metric has been shown to satisfy the MERCER conditions

Our main contribution

• Derive a new image similarity metric that is based on point correspondence and satisfies the Mercer condition

• Methodoly: generalize another metric developed by Grauman at MIT (pyramid kernel) while preserving its Mercer quality

Pyramid match – K.Grauman (ICCV05)

• Only appearance is considered

• Image represented in terms of multi-scale histograms

Appearance

space

Matching process

• Soft matches by histogram intersection• Fine resolution to coarse resolution• More weight at fine resolution: 2-level =1/size(bin)

Level 0

Level 1

Level 2

Final score (Kernel)

count intersection at current level

discards matches alreadycounted at previous levels

- Fine resolution first- Coarse resolution last

• This kernel verifies Mercer condition ! (Odone et all, TIP, 2005)

more weight givento best matches

Issues

should be matchedat this level

level 0

level 1

level 2

level 3

counted only here

• Boundary problems

x 2

x 2

x 2

Issues

level n

• 2level=size(bin) approximates poorly the distance between 2 points

• Weight function f(d)= 1/d over-emphasizes small distances

w = weight = 1/size(bin)

c = correct weight = 1/d

000

...

00

00

(w-c)/c

From discrete to continous

2k is a poor approximation increase the number of

resolution steps

Boundary problems

use translation of bins

level 0

level 1

level 2

level 3

• Verifies Mercer condition

(1+)

1

0

2

3

STEP 1: Establish CorrespondenceSTEP 1: Establish CorrespondenceOur kernel

• This kernel is easy to compute• Uses exact distances• No over-emphasis of low distances• still verifies Mercer condition

matches SORTED),(

21 )],([

1)I,K(I

ji jidf

best matches first

6

18.01][

d

dfwhere

Experiments – distance accuracy

• Random sets of 2D points• Compares distances based on: our kernel, pyramid match,

Earth Mover’s Distance (EMD = optimal solution to the matching problem, based on simplex)

Distances measured between simulated imagesfor pyramid match, max-kernel and EMD

Corresponding probability density function

Caltech database – 101 categories

Some data

• ~50 to ~300 images per class

• Performance of the competitors:– Chance : 1%– Fei-Fei & Perona : 16%– Berg & Malik : 48%– Holub & Perona : 40%– Grauman & Darrell : 43%

• Classification using a SVM

Classification results

• category vs. bg:

performance = 89%

• 10 random categories

performance: 61%

Classification results

• 7 good, 7 bad categories• Performance: 45%

Conclusions

• The MaxKernel is more accurate than pyramid match, more practical than EMD

• Good approximation of the optimal distance

• Verifies Mercer SVM classification OK

• Initial classification performance in same ballpark as the competition

• TODO: add some geometry – e.g. Hough transform to filter out wrong correspondences