Efficiently searching for similar images (Kristen Grauman)

Universidad Católica San Pablo

Cristina Patricia Cáceres Jáuregui

cristina.caceres.jauregui@ucsp.edu.pe

Motivation

Fast image search is a useful component for a number of vision problems.

Plenty of nuisance parameters (lighting, pose, background clutter, etc.)

Nuisance parameters

OutlineScalable image search

• Fast correspondence-based search with local features

• Fast similarity search for learned metrics

Local image features

How to handle sets of features?Want to compare, index, cluster, etc. local representations, but:

• Each instance is unordered set of vectors• Varying number of vectors per instance

Comparing sets of local features Previous strategies:

• Match features individually, vote on small sets to verify

• Explicit search for one-to-one correspondences

• Bag-of-words: Compare frequencies of prototype features

Pyramid match kernel

optimal partial matching

Optimal match: O(m3)Pyramid match: O(mL)

m = # featuresL = # levels in pyramid

Pyramid match: main idea

descriptor space

Feature space partitions serve to “match” the local descriptors within successively wider regions.

Pyramid match: main idea

Histogram intersection counts number of possible matches at a given partitioning.

Image search with matching-sensitive hash functions

• Main idea:– Map point sets to a vector space in such a

way that a dot product reflects partial match similarity (normalized PMK value).

– Exploit random hyperplane properties to construct matching-sensitive hash functions.

– Perform approximate similarity search on hashed examples.

Locality Sensitive Hashing (LSH)

Q111101

110111

110101

h r1…rkXi

N

h r1…rk

<< N

Q

Guarantee “approximate”-nearest neighbors in sub-linear time, given appropriate hash functions.

Randomized LSHfunctions

LSH functions for dot productsThe probability that a random hyperplane separates two unit vectors depends on the angle between them:

A)High dot product: unlikely to split

B)Lower dot product: likely to split

Corresponding hash function:

Metric learningThere are various ways to judge appearance/shape similarity…

but often we know more about (some) data than just their appearance.

Metric learning• Exploit partially labeled

data and/or (dis)similarity constraints to construct more useful distance function

• Can dramatically boost performance on clustering, indexing, classification tasks.

• Various existing techniques

Fast similarity search for learned metrics

• Goal: – Maintain query time guarantees while performing approximate search with a learned metric

• Main idea:– Learn Mahalanobis distance parameterization– Use it to affect distribution from which random hash functions are selected

• LSH functions that preserve the learned metric

• Approximate NN search with existing methods

Fast Image Search for Learned Metrics

It should be unlikely that a hash function will split examples like those having similarity constraints…

…but likely that it splits those having dissimilarity constraints.

h( ) = h( ) h( ) ≠ h( )

Learn a Malhanobis metric for LSH

• Local image features useful, important to handle efficiently

• Introduced scalable methods to allow fast similarity search methods with

– Local feature matching– Learned Mahalanobis metrics

• Key idea: design hash functions that encode matching process, or the constraints provided

Summary