keypoints in image retrieval

Bram Platel

Evgenya Balmashnova

Luc Florack

Bart ter Haar Romeny

Question: can top-points be used for object-retrieval tasks?

The locations of particularly characteristic points are called the interest points or key points.

These interest points have to be as invariant as possible, but at the same time they have to carry a lot of distinctive information.

Information in interest points is defined by their neighborhood. But how big should we choose this neighborhood?

• Let’s take the corners of the mouth as interest points.

• The red circles are the areas in which the information is gathered.

• If we make the picture bigger, the size of the neighborhood is too small.

• The neighborhood should scale with the image

When the interest points are detected in scale space they do not only have spatial coordinates x and y, but also a scale .

This scale tells us how big the neighborhood should be.

Our interest points have to be detected in scale space.

They also have to…◦ …contain a lot of information◦ …be reproducible◦ …be stable◦ …be well understood

Critical Points, Paths and Top-Points

Maxima

Minimum

SaddlesL=0Critical Points

Critical Points, Paths and Top-Points

Maxima

Minimum

SaddlesL=0Critical Points Det(H)=0

Top-Points

Original Gradient Magnitude

Laplacian Det(H)

Since for a critical path L=0

Intersection of Level Surfaces Lx=0 with Ly=0

Will give the critical paths.

Since for a top-point both L=0 and Det[H]=Lxx Lyy-Lxy2=0

We can find them by intersecting the paths with the level surface Det[H]=0

Top-points are invariant to certain transformations.

By invariant we mean that they move according to the transformation.

Allo

wed

Tra

ns.

It is possible to make a reconstruction of the original image from its top-points.

We can generate reconstructed images which give the same (plus more) top-points as the original image.

This reconstruction resembles the original image.

Original Image

Top-Points and Features

Reconstruction

OriginalBy adjusting boundary and smoothness constraints we can improve the visual performance.

For this 300x300 picture 1000 top-points with 6 features were used.

For points close to top-points it is possible to calculate a vector pointing towards the position of the top-point.

x

y

Approximated Top-Points

Displacement Vectors

Real Locations

For points close to top-points it is possible to calculate a vector pointing towards the position of the top-point.

This enables us to use fast top-point detection algorithms which do not have to be very accurate.

The locations of top-points change when noise is added to the image.

We can calculate the variance of the displacement of top- points under noise.

We need 4th order derivatives in the top-points for that.

Stable Paths Unstable Paths

1 2 3 4 5 87 96

Data

ba

seQuery Image

A simple image retrieval task. Using a small version of the Olivetti Faces

Database. Consisting of 200 images of 20 different

people (10 p.p.)

Look at all scales simultaneously

Scale

x

y

Critical Points, Paths and Top Points

Maxima

Minimum

Saddlesu=0Critical Points Det(H)=0

Top Points

CompareEMD

m

i

n

j ijij fc1 1:Minimize

wi fij cij

A Buj

miwf

njuf

njmif

i

n

j ij

j

m

i ij

ij

,,1

,,1

,,1,,1 ,0

:sConstraint

1

1

n

j

m

i ij uw1 1

:DemandSupply

n

j j

m

i

n

j ijij

u

fc

1

1 1u)EMD(w,

[*]Rubner, Tomasi, Guibas, 1998, IEEE Conf. on Computer Vision

Piles Holes

1 2 3 4 5 6 7 8 9

a 93% 78% 68% 62% 58% 54% 49% 45% 43%

b 93% 82% 90% 73% 70% 68% 63% 59% 56%

c 95% 88% 83% 76% 73% 66% 62% 59% 58%

d 100%

97% 96% 95% 92% 88% 86% 84% 81%

a. Using Euclidean Distance

b. Using Eberly Distance

c. As b. including stability norm

d. As c. including 2nd order derivatives.

To distinguish top-points from each other a set of distinctive features are needed in every top-point.

These local features describe the neighborhood of the top-point.

We use the complete set of irreducible 3rd order differential invariants.

These features are rotation and scaling invariant.

The top-points and differential invariants are calculated for the query object and the scene.

We now compare the differential invariant features.

compare

distance = 0.5distance = 0.2distance = 0.3

The vectors with the smallest distance are paired.

smallest distancedistance = 0.2

A set of coordinates is formed from the differences in scale (Log(o1)- Log(s2)) and in angles (o1- s2).

(, )

Important Clusters

For these clusters we calculate the mean and

If these coordinates are plotted in a scatter plot clusters can be identified.

• In this scatter plot we find two dense clusters

The stability criterion removes much of the scatter

Rotate and scale according to the cluster means.

The translations we find correspond to the location of the objects in the scene.

In this example we have two clusters of correctly matched points.

C1

C2

We can transform the outline of the query object and project it on the scene image.

Top-points have proved to be invariant interest points which are useful for matching.

The differential invariants have shown to be very distinctive.

Experiments show good results.