Learning Image Similarities via Probabilistic Feature Matching

Ziming Zhang*, Ze-Nian Li, Mark DrewSchool of Computing Science, Simon Fraser University, Vancouver, B.C., Canada{zza27, li, mark}@cs.sfu.caLearning Image Similarities via Probabilistic Feature Matching1*This work was done when the author was in SFU.Outline2IntroductionProbabilistic-matching based similarity learningProbabilistic feature matching functionProbabilistic feature matching learningExperimentsConclusionIntroduction3Object-based image similarityIdeally two images are supposed to have a higher similarity if they contain similar objects.Feature matchingA natural way to measure the image similaritiesMany different criteriaIn this talk, we simply match features only based on their appearance information

4

>Introduction5Several relevant feature matching approachesSummation kernel

Max-selection kernel

Optimal assignment kernel

Introduction6

Introduction7Our approach is a generalization of a family of SIMILARITY learning approaches, including the three above.Similarity matrix KernelA kernel matrix can be considered as a special similarity matrix (i.e. symmetric positive semi-definite)Classification with Support Vector Machines (SVM)

Probabilistic-matching based similarity learning8How to learn these feature matching probabilities?

Feature Matching Function9Given two images X={x1,,x|X|} and Y={y1,,y|Y|}, a feature matching function can be defined as

Explanations of matching processes in the Summation Kernel, Max-selection Kernel, and Optimal Assignment Kernel using feature matching functionsKsum:Kmax :KOA :

Probabilistic Feature Matching Function10Our probabilistic feature matching function is defined in the vector space covered by the following convex set:

Total matching probability of one featureTotal matching probability of all featuresEach matching probabilityProbabilistic Feature Matching Learning11Data-dependent optimization

Image similarityDistribution sparseness(or Regulizer)

Probabilistic Feature Matching Learning12TheoremsConsider max f(x) over x X, where f(x) is convex, and X is a closed convex set. If the optimum exists, a boundary point of X is the optimum.If a convex function f(x) attains its maximum on a convex polyhedron X with some extreme points, then this maximum is attained at an extreme point of X.Relation to Ksum, Kmax, and KOAKsum: C=+ and H={i,j}Kmax: C=0 and H={i}, and C=0 and H={j}KOA: C=0 and H={i,j}

Probabilistic Feature Matching Learning13PropositionFor two images X and Y , both the sparseness of and their similarity will decrease monotonically with increasing the parameter C .

Probabilistic Feature Matching Learning14

Experiments15Datasets: GrazDescriptor SIFT + dense samplingImage Representation3*3 spatial Bag-of-Word histograms with 200 codewordsFeature similarity: RBF-kernel with 2 distance50 runs

Graz-01Graz-02

Experiments16Graz-01

(a) PFM1 with H={i,j}(b) PFM2 with H={i} or H={j}(c) PFM3 with H=Experiments17BikePersonAverageSPK [1]86.32.582.33.184.3PDK [2]90.22.687.23.888.7PFM1 (C=0)90.65.388.24.689.4PFM2 (C=5)89.64.988.54.689.0PFM3 (C=+)89.64.887.95.188.8Table 1. Comparison results between different approaches on Graz-01 (%)[1] Lazebnik et. al., Beyond bags of features: Spatial pyramid matching for recognizing natural scene categories, in CVPR06.[2] Ling and Soatto, Proximity distribution kernels for geometric context in category recognition, in ICCV07.Experiments18Graz-02

(a) PFM1 with H={i,j}(b) PFM2 with H={i} or H={j}(c) PFM3 with H=Experiments19BikePersonCarAverageBoost.+SIFT [3]76.070.068.971.6Boost.+comb.[3]77.881.270.576.5PDK+SIFT [2]86.786.774.782.7PDK+hybrid [2]86.087.374.782.7PFM1+SIFT (C=5)88.988.185.287.4PFM1+SIFT (C=10)88.087.983.686.5PFM1+SIFT (C=+)87.787.882.686.0Table 2. Comparison results between different approaches on Graz-02 (%)Opelt et. al., Generic object recognition with boosting, PAMI, 2006.19Conclusion20Probabilistic feature matching schemeA generalization of a rich family of probabilistic feature matching approachesEasy to control the sparseness of the matching probability distributions and their corresponding image similarities21Thank you !!!