From Partial Shape Matching through Local Deformation

to Robust Global Shape Similarity for Object Detection

Tianyang Ma and Longin Jan Latecki Temple University

Outline Contour

illumination texture

Contour is robust to changes in

Contour-based object detection

[V. Ferrari, F. Jurie, and C. Schmid. IJCV 2009][Q. Zhu, L. Wang, Y. Wu and J. Shi. ECCV 2008] [X. Bai, X. Wang, L. J. Latecki, W. Liu and Z. Tu. ICCV 2009][C. Lu, L. J. Latecki, N. Adluru, X. Yang, and H. Ling. ICCV 2009] [S. Riemenschneider, M. Donoser, and H. Bishof. ECCV 2010][P. Srinivasan, Q. Zhu, and J. Shi. CVPR 2010][A. Toshev, B. Taskar, and K. Daniilidis. CVPR 2010]

Challenges

2 . Part of the true contour of the target object may be wrongly connected to part of a background contour resulting in a single edge fragment part to part matching needed

1 . The contour of the desired object is typically fragmented over several pieces.

This has been addressed by other approaches.

Partial Shape Matching

[S. Riemenschneider, M. Donoser, and H. Bishof. ECCV 2010]

The key ideas:• each contour part is represented as a submatrix• efficient matching with integral image

We utilize these ideas but use different geometric features.Our object detection framework is very different.

Outline

1. Shape Representation2. Partial Matching in Images3. Contour Selection as Maximal Clique Computation4. Experiments

Outline

1. Shape Representation2. Partial Matching in Image3. Contour Selection as Maximal Clique Computation4. Experiments

Shape Representation: Histogramless Shape Context

Shape Context[Belongie et al. 2002]

j

i

P Q

P

Q

j

i

i

j

Distance Matrix:

Angle Matrix:

( , )2( , ) log(1 )P P

i jD i j p p

( , ) ( , ) ,P Pi ji j p p

Histogram

Contour part = dist. submatrix + angle submatrix

( , ) ( , )P PD i j

two matrices are used to represent the entire contour two block diagonal matrices represent the green contour part

MM

M

M

M

M

Single partial contour P

( , ) ( , )P P i j

Relation of any two contour parts is submatrices

( , ) ( , )P QD i j

( , ) ( , )P Q i j

two submatrices are used to represent spatial configuration of a part composed of two contour segments.

M

M

M

M

Partial Contours P and Q

2( , ) ( , )

( , ) 2

( , ) ( , )( , ) exp( )

( ( , ) )

P P T T

D P P

D i j D i jA P T

D i j

2( , ) ( , )

2

( , ) ( , )( , ) exp( )

P P T Ti j i jA P T

( , ) ( , ) ( , )DA P T A P T A P T

21 1

1( , ) ( , )m m

i i

P T A P Tm

( , )P PD ( , )P P

Shape Similarity

P

T

T

( , )T TD ( , )T T

( , )T TD ( , )T T

Shape Similarity

P Q

TU

T

U

( , )P QD ( , )P Q

( , )T UD ( , )T U

( , )T UD ( , )T U

Outline

1. Shape Representation2. Partial Matching in Image3. Contour Selection as Maximal Clique Computation4. Experiments

Partial matching between edge fragments and model contour

1( , ) { ,..., }k j j le j l q q

1( , ) { ,..., }i j lM i l p p i

1i l j

1j l

( , )M i l

( , )ke j l

Partial image edge fragment

Partial Model fragment

Partial matching determines:

( ( , ), ( , ))kM i l e j l

Key advantages: • no model decomposition into parts is needed• no breaking or connecting edge fragments is needed• tolerates missing contour fragments in edge image.

Partial matching between edge fragments and model contour

• 1) Construct a 4D tensor• 2) Take the maximum of the 4D matrix along different length ,

and suppress it tol

j

i

( , , )S i j k

Partial matching: ( ( , ), ( , ))kM i l e j lj - start point on edge contour k

k –

inde

x of

edg

e co

ntou

r

i - start point on model contour

j

i

( , , )S i j k

k

1i

1i

All relevant edge fragments are mapped to their corresponding model fragments.

Key benefits:1. Top down selection of relevant

edge subfragments.2. All selected edge subfragments

are mapped to 1D curve of the model contour.

j

i

( , , )S i j k

k

1i

1i

...

...

How to find the true contours of the target shape in the edge image?

Problem formulation

?

Key idea:Given a minimal required coverage of the model contour,we want to select non overlapping model fragments that maximizethe configuration similarity to the corresponding image fragments.

Outline

1. Shape Representation2. Partial Matching in Image3. Contour Selection as Maximal Clique Computation4. Experiments

Construction of Affinity Matrix

)),(),,(( 1111 ljeliMv mi

)),(),,(),,(),,((),( 22112211 ljeljeliMliMjiA nm

Each vertex of the graph corresponds to a partial match

The affinity between and is

1 1 1 1( ( , ), ( , ))mM i l e j l

2 2 2 2( ( , ), ( , ))nM i l e j l

jv

iv

iv jv

( , )A i j

The weighted affinity graph is denoted as G = (V, A).

Construction of Affinity Matrix

High affinity:

Low affinity:

Problem with Affinity Matrix

Wrong matches may also have high affinity:

Detected Contours are Maximal CliquesA maximal clique is a subset of V with maximal average affinity between all pairs of its vertices.

In this example, the maximal clique has 4 nodes selected from over 500 nodes. Therefore, most clustering based approach may not succeed.

[ M. Pavan and M. Pelillo. PAMI 2007]

In order to solve this combinational problem, we relax it to

A vertex is selected as belonging to a maximal clique iff v V 0vx

Each maximal clique corresponds to a local maximum of:

Each local solution is not a final solution but a detection hypothesis.

Indicator = selected maximal clique of vertices of V.

Computing Maximal Cliques

{0,1}NX

1: 0 and 1NX R X X

( ) Tf X X AX

We employ the new algorithm to find all significant local maxima:[H. Liu , L. J. Latecki, and S. Yan. NIPS 2010]

Furthermore, we locally deform the model, then evaluate and score each detection hypothesis.

Outline

1. Shape Representation2. Partial Matching in Image3. Contour Selection as Maximal Clique Computation4. Experiments

Experiments on ETHZ dataset

Quantitative Evaluation

Mean APOur method 0.877P. Srinivasan, Q. Zhu, and J. Shi. CVPR 2010 0.872S. Maji and J. MalikCVPR 2009 0.771P. Felzenszwalb, D. McAllester, and D. Ramanan CVPR 2008 0.712C. Lu, L. J. Latecki, N. Adluru, X. Yang, and H. Ling. ICCV 2009 0.709

Interpolated Average Precision : AP

Accuracy of boundary localization

Each entry is the coverage/precision for correct detections at 0.4 FPPI.

Thanks!