Slide 1

Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification Computer Vision, 2007. ICCV 2007. IEEE 11th International Conference on Andrea Frome, EECS, UC Berkeley Yoram Singer, Google, Inc Fei Sha, EECS, UC Berkeley Jitendra Malik, EECS, UC Berkeley

Slide 2

Outline Introduction Training step Testing step Experiment & Result Conclusion

Slide 3

Outline Introduction Training step Testing step Experiment & Result Conclusion

Slide 4

What we do? Goal classify an image to a more appropriate category Machine learning Two steps Training step Testing step

Slide 5

Outline Introduction Training step Testing step Experiment & Result Conclusion

Slide 6

Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Input distances to SVM for training, evaluate W Compute distance dji, dki

Slide 7

Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Input distances to SVM for training, evaluate W Compute distance dji, dki

Slide 8

Choosing features Dataset: Caltech101 Patch-based Features SIFT Old school Geometric Blur Its a notion of blurring The measure of similarity between image patches The extension of Gaussian blur

Slide 9

Geometric blur

Slide 10

Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Input distances to SVM for training, evaluate W Compute distance dji, dki

Slide 11

Triplet dji is the distance from image j to i Its not symmetric, ex: dji dij dki > dji djidki

Slide 12

How to compute distance L2 norm 1 2 3 dji, 1 m features dji, 1 distance vector dji Image j Image i

Slide 13

Example Given 101 category, 15 images each category 101*15 Feature j 101*15 distance vector Image j vs training data

Slide 14

Flow chart: training Generate features each image from dataset, ex: SIFT or geometric blur Input distances to SVM for training, evaluate W Compute distance dji, dki

Slide 15

Machine learning: SVM Support Vector Machine Function: Classify prediction Supervised learning Training data are n dimension vector

Slide 16

Example Male investigate Annual income Free time Have girlfriend?

Slide 17

Ex: Training data

Slide 18

space free income vector

Slide 19

Slide 20

Mathematical expression(1/2)

Slide 21

Mathematical expression(2/2)

Slide 22

Support vector Model free income

Slide 23

But the world is not so ideal.

Slide 24

Real world data

Slide 25

Hyper-dimension

Slide 26

Error cut

Slide 27

SVM standard mathematical expression Trade-off

Slide 28

In this paper Goal: to get the weight vector W 101*15 feature Image weight wj of W wj, 1 wj

Slide 29

Visualization of the weights

Slide 30

How to choose Triplets? Reference Image Good friend - In the same class Bad friend - In the different class Ex: 101category, 15 images per category 14 good friends & 15*100(1500) bad friends 15*101(1515) reference images total of about 31.8 million triplets

Slide 31

Mathematical expression(1/2) Idealistic: Scaling: Different: The length of Weight i 00 triplet

Slide 32

Mathematical expression(2/2) Empirical loss: Vector machine:

Slide 33

Dual problem

Slide 34

Dual variable Iterate the dual variables:

Slide 35

Early stopping Satisfy KTT condition In mathematics, a solution in nonlinear programming to be optimal.mathematicsnonlinear programming Threshold Dual variable update falls below a value

Slide 36

Outline Introduction Training step Testing step Experiment & Result Conclusion

Slide 37

Flow chart: testing Query an image i Output the most appropriate category Calculate Dxi, x is all training data, except itself.

Slide 38

Flow chart: testing Query an image i Output the most appropriate category Calculate Dxi, x is all training data, except itself.

Slide 39

Query image? Goal: classify the query image to an appropriate class Using the remaining images in the dataset as the query image

Slide 40

Flow chart: testing Query an image i Output the most appropriate category Calculate Dxi, x is all training data, except itself.

Slide 41

Distance function(1/2) Query image i Image i feature 101*15 distance vector Image i vs all training data dxi, 1

Slide 42

Distance function(2/2) 101*15 Image I vs all the training data Dji

Slide 43

Flow chart: testing Query an image i Output the most appropriate category Calculate Dxi, x is all training data, except itself.

Slide 44

How to choose the best image? Modified 3-NN classifier no two images agree on the class within the top 10 Take the class of the top-ranked image of the 10

Slide 45

Outline Introduction Training step Testing step Experiment & Result Conclusion

Slide 46

Experiment & Result Caltech 101 Feature Geometric blur (shape feature) HSV histograms (color feature) 5, 10, 15, 20 training images per category

Slide 47

Slide 48

Confusion matrix for 15

Slide 49

Outline Introduction Training step Testing step Experiment & Result Conclusion

Slide 50

Learning Globally-Consistent Local Distance Functions for Shape-Based Image Retrieval and Classification