Shape similarity and Visual PartsLongin Jan Latecki

Temple Univ., latecki@temple.edu

In cooperation withRolf Lakamper (Temple Univ.),

Dietrich Wolter (Univ. of Bremen)

2

Object Recognition Process:

Source:2D image of a 3D object

Matching: Correspondence of Visual Parts

Contour Segmentation

Contour Extraction

Object Segmentation

Evolution

3

What is this talk about ?Comparison of object shape based on object contours

Object contours are naturally obtained in Computer Vision, Robot Navigation, and other applications as polylines (polygonal curves).Shape similarity reduces to similarity of polylines.

Shape similarity of polylines is not so simple:•simple 1-1 vertex correspondence does not work•a scale problem

4

• only part of a polyline may be visible

5

Since polylines are obtained as boundary parts of objects in usually noisy sensor data (e.g., digital images):

1. two similar polylines do not need to have the same number of vertices, i.e., do not have to be of comparable level of detail,

2. do not have to be of comparable size,3. may have only a subpart that is similar and that has a

significant contribution to their shape = visual part

Cognitive Similarity Requirements

6

Comparable level of detail (1)

It can be achieved in the context of•a single object or•a group of objects (e.g., a query and a goal shape)

How can we achieve a comparable level of detail forall objects if we treat each object separately?

By placing each object on the same level of theScale Space hierarchy.

7

Comparable level of detail Scale Space

8

Discrete Curve Evolution (DCE) We achieve a comparable level of detail with DCE.

Before a similarity measure is applied, the shape of objects is simplified by DCE in order to

• reduce influence of noise,

• simplify the shape by removing irrelevant shape features without changing relevant shape features.

Discrete Curve Evolution (DCE)

u

v

w u

v

w

It yields a sequence: P=P0, ..., Pm

Pi+1 is obtained from Pi by deleting the vertices of Pi that have minimal relevance measure

K(v, Pi) = |d(u,v)+d(v,w)-d(u,w)|

>

10

Discrete Curve Evolution: Preservation of position, no blurring

11

Discrete Curve Evolution: robustness with respect to noise

12

Discrete Curve Evolution: extraction of linear segments

13

Discrete Curve Evolution: mathematical properties

Convexity Theorem (trivial)Discrete curve evolution (when applied to a polygon)

converges to a convex polygon. Continuity Theorem (nontrivial)Discrete curve evolution is continuous.

L. J. Latecki, R.-R. Ghadially, R. Lakämper, and U. Eckhardt: Continuity of the discrete curve evolution. Journal of Electronic Imaging 9, pp. 317-326, 2000.

Polygon Recovery (nontrivial)DCE allows to recover polygons from their digital images.L.J. Latecki and A. Rosenfeld: Recovering a Polygon form Noisy Data. Computer

Vision and Image Understanding (CVIU) 86, 1-20, 2002.

14

Comparable level of detail for DCE (=stop condition) is based on a threshold on the relevance measure

15

Comparable level of detail for DCE is based on a threshold on the relevance measure

16

Scale Space Approaches to Curve Evolution

1. reaction-diffusion PDEs

2. polygonal analogs of the PDE-evolution (Bruckstein et al. 1995)

3. approximation (e.g., Bengtsson and Eklundh 1991)

Main differences:

[to 1, 2:] Each vertex of the polygon is moved at a single evolution step, whereas in our approach the remaining vertices do not change their positions.

[to 1, 3:] Our approach is parameter-free(we only need a stop condition)

17

18

(3) Local similarity

We need local similarity measure,i.e., we need to compare polylines but not

polygons.

Global similarity measures fail at:• - partial visibility (occlusion)• - not uniformly distributed noise• - actually everything that occurs under• real conditions

19

Local similarity

A simple solution to local similarity measure:Given a target polylineT, find the most similar part P of polyline Q.

Consider all subpolylines of Q combinatorial explosion

We consider only all connected subsets P of Q, O(n²) in the number of vertices n of Q.

However, a connected subset P may be a distorted version of T.

20

Optimal Shape Similarity The optimal similarity of P to T is the similarity of

modified P to T, where modified P is P with all features that make P distinct from T removed.

},...,{ 1 nppP

PSSPTsS TTT :),(minarg*

PSSPTsPTos TT :),(min),(

Consider all subpolylines of P combinatorial explosion

s is a global similarity measure

21

T PT*

Q P

22

• All shape similarity measures presented in the literature are global measures.

• Although they can be applied to local parts, they are not optimal.

23

Presented approachA two-step solution:

1. We consider only all connected subsets P of Q, O(n²), n vertices of Q.

1. We use a greedy algorithm to compute optimal similarity, O(n²), n vertices of P.

We need a global similarity measure s.

24

Shape Similarity in Tangent Space

A polyline P is mapped to its turn angle function T(P)in the tangent space

the height of each step shows the turn-angle,

monotonically increasing intervals represent convex arcs,

height-shifting corresponds to rotation,

the resulting curve can be interpreted as 1D signal

25

Shape Similarity Measure: Arkin at al. PAMI 1991.

26

Shape Comparison: Measure

Drawback: not adaptive to unequally distributed noise

27

Shape Comparison: Contour Segmentation

Solution: use this measure only locally,i.e., apply only to corresponding parts:

28

Shape Similarity

29

30

Correspondence of visual parts: Results

31

Correspondence of visual parts: non-rigid deformation

L. J. Latecki and R. Lakämper: Shape Similarity Measure Based on Correspondence of Visual Parts. IEEE Trans. Pattern Analysis and Machine Intelligence 22, 2000.

L. J. Latecki and R. Lakämper: Application of Planar Shape Comparison to Object Retrieval in Image Databases. Pattern Recognition 35, 2002.

32

Comparison to deformation energy

33

Partial Shape Matching

This shape similarity measure works fine if the whole contouris given:• Great performance in the MPEG-7 competition• Life web-based shape search engine

Can it be applied when only contour parts are given?

Yes, but only in the context of optimal shape similarity.

34

Subpart SelectionWe can use sliding window on the contour of the object Qto find a given target part T.However, we cannot expect to cut part P of Q that exactlycorresponds to T, due to • imperfect size of the window • noise and change of view point• scale selection.

Therefore, a similarity measure must be able to overlook parts of Q whose shape is irrelevant w.r.t. Q.

T

Q

35

T PT*

Q P

36

Matching and SimplificationThe main idea of the proposed computation ofthe optimal similarity is shape matching and simplificationsimilar to DCE in one process.

The main idea:We recursively remove a vertices of P to obtain P’ such that P’is the most similar to T. os(T,P)= global minimum of s(T,P’).

We achieve a comparable level of detail (1)for polyline P in the context of target T.Observe that this approach also solves the stop conditionproblem (DCE is applied to each object separately).

37

T PT*

Q P

0 100 200 300400

450

500

550

600

650

0 100 200 300400

450

500

550

600

650

0 100 200 300400

450

500

550

600

650

0 100 200 300400

450

500

550

600

650

0 100 200 300400

450

500

550

600

650

0 100 200 300400

450

500

550

600

650

A B C

D E

optimal subset

F

PT*=E

38

0 1 2 5 7 8 10 15 20 22 25 300

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

A

B

C D

E

F

optimal subset

39

Matching and Simplification IIGiven a target T and a polyline P.

os(T,P) = global minimum S(T,P’) + S(P’,P),

where and are weights. The simplest weight assignment is = 1 and =0.

40

41

Target T

Most similar 21 shapes

optimal subsets

*

*

42

Applications•Shape-based object recognition•Retrieval in image and video databases•Object tracking

•Shape-based tracking of objects in laser scans•Robot mapping•Robot localization in top view maps

43

Shape-based tracking of objects in laser scans

An example top view image of LRF scan data

44

Shape-based tracking of objects in laser scans

Row scan data (left) is segmented into polyline (right)

45

Scan polylines simplification with DCE

46

Matching scans by shape

Since shape may be very simple,we also use cyclic order and proximity information

47

Shape-based tracking of objects in laser scansBuilding a global map

Demo movie

Why go beyond the simple proximity of points computedby minimizing the least squared distance of all points?

Robot may slip or turn (unconsciously):Shape and order of polylines remain similar even if the displacement is significant

48

Robot mapping and localization

Shape similarity useful for: • globally consistent mapping • localization when odometry is not given or unreliable

49

Future WorkLearning of visual parts:How to select the most distinctive parts of objects?Our approach is based on statistics and our shape similaritymeasures:We use statistical methods to find the smallest possible setof most different parts within a given class of objectsand the smallest possible set of most separating partsamong different object classes.

Previous approaches to find visual parts are based on differential geometry. They are static in that the parts will bealways the same for different classes of objects.Our approach is dynamic: selected parts depend on the objects seen.

50

Automatically Selected Parts