![Page 1: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/1.jpg)
1
A Graph-Matching Kernel for Object Categorization
OlivierDuchenne
ArmandJoulin
JeanPonce
WillowLab
![Page 2: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/2.jpg)
2
Goal: Object Categorization
CAT
DINOSAUR
PANDA
![Page 3: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/3.jpg)
3
Comparing images
≈?
We need to design a good image similarity measure:
![Page 4: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/4.jpg)
4
Unaligned Images are hard to compare
![Page 5: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/5.jpg)
5
Unaligned Images are hard to compare
![Page 6: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/6.jpg)
6
Graph Matching
[1] A. C. Berg, T. L. Berg, and J. Malik. CVPR05
[2] M. Leordeanu and M. Hebert. ICCV05
![Page 7: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/7.jpg)
Feature Matching
Caputo (2004)Boiman, Shechtman & Irani (2008)Caputo & Jie (2009)
![Page 8: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/8.jpg)
Graph Matching
Nevatia & Binford’72Fischler & Elschlager’73Berg, Berg, & Malik’05Leordeanu & Hebert’05Cour & Shi’08Kim & Grauman’10Etc.
![Page 9: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/9.jpg)
• Lazebnik, Schmid and Ponce (2006)• Indyk and Thaper (2003) • Grauman and Darrell (2005)
Spatial pyramid kernels
(Courtesy of L. Lazebnik)
9
![Page 10: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/10.jpg)
10
![Page 11: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/11.jpg)
11
9 matches x 1
= 9
![Page 12: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/12.jpg)
12
9 matches x 1
= 9
![Page 13: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/13.jpg)
13
9 matches x 1 4 matches x ½
= 9 = 2
![Page 14: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/14.jpg)
14
9 matches x 1 4 matches x ½
= 9 = 2
![Page 15: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/15.jpg)
15
9 matches x 1 4 matches x ½ 2 matches x ¼
= 9 = 2 = 1/2
Total matching weight (value of spatial pyramid kernel ): 9 + 2 + 0.5 = 11.5
![Page 16: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/16.jpg)
16
Graph MatchingVSSpatial Pyramids
•Approximate Feature Matching•Does not take into account pair-wise relationship•Require quantization•Low Computational Coast
•Graph Matching•Take into account pair-wise relationship•Does not require quantization•High Computational Coast
![Page 17: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/17.jpg)
17
So, how does it perform in real life?
• Graph-Matching keeps more information, but is much slower…
• In addition, it appears to perform much worse than SPM kernel.
Caltech 101 (%)Graph Matching Based
Methods15 examples
Berg Matching [CVPR 2005] 48GBVote [Berg phD thesis] 52
Kim et Graumann [CVPR 2010] 61Spatial Pyramid MethodsBoureau et al. [CVPR10] 69.0
Yang et al.[ICCV09] 73.3
![Page 18: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/18.jpg)
18
Graph MatchingVSSpatial Pyramids
•Dense Features•SVM classifier•Fast•No pair-wise Information•State-of the-Artperformance
•Sparse Features•NN Classifier•Slow•Use pair-wiseInformation•Lower performance
![Page 19: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/19.jpg)
19
Our ApproachSpatial Pyramids
•Dense Features•SVM classifier•Fast•No pair-wise Information•State-of the-Artperformance
•As Dense•SVM Classifier•Fast enough•Use pair-wiseInformation•State-of-the-art performance
VS
![Page 20: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/20.jpg)
20
Outline of our method
K=
Image i
Image j
K(Imi,Imj) •As Dense•SVM Classifier•Fast enough•Use pair-wiseInformation•State-of-the-art performance
![Page 21: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/21.jpg)
21
≈?
![Page 22: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/22.jpg)
22
Similarity?≈
Sum of Local
Similarities
Deformation Smoothness
Maximized over all possible
deformations
![Page 23: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/23.jpg)
24
In our case: we allow each point to match to a 11x11 grid in the other image.
![Page 24: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/24.jpg)
25
Unary term
)(),()( nnn dnfnfdU
![Page 25: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/25.jpg)
26
GOOD
Binary term
![Page 26: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/26.jpg)
27
BAD
Binary term
![Page 27: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/27.jpg)
28
VERY BAD
Binary term II
otherwise 01 if ][1 if ][
),(, mnnm
mnnm
nmnm yydydyxxdxdx
ddv
![Page 28: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/28.jpg)
29
Optimization problem
),(),()()(),(
,),(
, nmEnm
nmnmEnm
nmVn
nn ddvddudUdE
Minimize with respect to the integer disparity vector d:
• This is (in general) an NP-hard problem…
…because there is no natural ordering on the vectors d
• We give an efficient approximation algorithm
![Page 29: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/29.jpg)
30
Vertical MoveThe general problem is NP Hard, but if we just allow vertical moves, it can be solved optimally.
![Page 30: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/30.jpg)
31
Horizontal MoveHorizontal moves can also be solved optimally.
![Page 31: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/31.jpg)
32
Diagonal MoveDiagonal moves can also be solved optimally.
![Page 32: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/32.jpg)
33
General MoveIf all points move in the same direction (e.g. dx increasing dy decreasing), it can also be solved optimally.
The allowed moves can be different for each point, as long as they are in the same direction.
![Page 33: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/33.jpg)
34
Optimization
• Even if the whole optimization is NP hard, for each of the sub-problems (horizontal move,…) we can find the global optimum.
• To optimize our objective function, we iterate over the different types of move, until convergence.
• We can do it using Ishikawa’s method.
![Page 34: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/34.jpg)
35
Optimization methodIshikawa’s method can solve:
with g convex, and the labels mapped on integer.
with
![Page 35: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/35.jpg)
36
Comparison with alpha-expansion
• Alpha Expansion: At each iteration, each node can choose between keeping its label, or taking the label alpha.
• Ours: At each iteration, each label can chose between K (=11) labels.
At convergence, the local minimum is better than its n neighbors:
• Alpha Expansion: • Ours:• Total number of configurations:
132102 nNlN
45010nN
lNp89910nN
lN
![Page 36: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/36.jpg)
37
SummaryExtract Features
Global Normalization
Match pairs of images
Make a kernel from
matching scores
Train 1vsAll SVM Test with SVM
![Page 37: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/37.jpg)
38
![Page 38: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/38.jpg)
39
Results for Graph Matching methods
Caltech 101 (%)Graph Matching Based
Methods15 examples
Berg Matching [CVPR 2005] 48GBVote [Berg PhD thesis] 52Kim and Graumann [CVPR
2010]61
Ours 75.3
+14%
![Page 39: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/39.jpg)
40
Results: Caltech 101Caltech 101 (%)
Feature Method 15 examples
30 examples
SingleNBNN (1 Desc.) [CVPR08] 65.0 -Boureau et al. [CVPR10] 69.0 75.7
Ours 75.3 80.3
MultipleGu et al.[CVPR09] - 77.5
Gehler et al.[ICCV09] - 77.7NBNN (5 Desc.)[CVPR08] 72.8 -
Yang et al.[ICCV09] 73.3 84.3
+5%+6%
![Page 40: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/40.jpg)
41
Results: Caltech 256
Caltech 256 (%)Feature Method 30 examples
Single
SPM+SVM[07] 34.1Kim et al. [CVPR10] 36.3
NBNN (1 desc.) 37.0Ours 38.1
Multiple NBNN (5 Desc.)[CVPR08]
42.0
![Page 41: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/41.jpg)
42
Matching Time
Method Number of Nodes TimeBerg el al.[CVPR 05] 50 5s
Leordaenu et al.[ICCV 05]
130 9s
Kim , Grauman[CVPR10]
4800 10s
Kim, Grauman[CVPR 10]
500 1s
Alpha-Expansion 500 1sTRW-S 500 10sOurs 500 0.04s
![Page 42: A Graph-Matching Kernel for Object Categorization](https://reader036.vdocuments.net/reader036/viewer/2022062301/568161bc550346895dd1979d/html5/thumbnails/42.jpg)
43
Conclusion• Graph-matching can make good
performance• It is more interpretable, and it makes it
easy to visualize what is happening.• Our main contributions are: the
combination of SVM and graph matching; and the design of a fast
matching algorithm.
Thank youQuestions?
完