1Michael Bronstein Heat diffusion descriptors

deformable

Michael Bronstein

Weizmann Institute of Science, 4 November 2010

Institute of Computational ScienceUniversita della Svizzera Italiana

Lugano, Switzerland

shapesforHeat diffusion descriptors

2Michael Bronstein Heat diffusion descriptors

Dan RavivTechnion

Ron KimmelTechnion

Maks OvsjanikovStanford

Leo GuibasStanford

Iasonas KokkinosECP Paris

Alex BronsteinTAU

3Michael Bronstein Heat diffusion descriptors

The next challenge

Text Visual data Geometric data

4Michael Bronstein Heat diffusion descriptors

Shape retrieval today

5Michael Bronstein Heat diffusion descriptors

Bags of words

Notre Dame de Paris is a Gothic cathedral in the fourth quarter of Paris, France. It was the first Gothic architecture cathedral, and its construction spanned the Gothic period.

cons

truc

tion

arch

itect

ure

Italy

Fran

ceca

thed

ral

chur

chba

silic

aPa

risR

ome

Got

hic

Rom

anSt. Peter’s basilica is the largest church in world, located in Rome, Italy. As a work of architecture, it is regarded as the best building of its age in Italy.

Notre Dame de Paris is a Gothic cathedral in the fourth quarter of Paris, France. It was the first Gothic architecture cathedral, and its construction spanned the Gothic period.

St. Peter’s basilica is the largest church in world, located in Rome, Italy. As a work of architecture, it is regarded as the best building of its age in Italy.

6Michael Bronstein Heat diffusion descriptors

Outline

Feature descriptor

Geometric words

Bag of geometric words

Geometric expressions

Spatially-sensitive bag of words

“ ”“ ”

Volumetric descriptors

Scale invariance

7Michael Bronstein Heat diffusion descriptors

Curvature

Rigid Bending Topology

Integralvolume1

Scale

Spin image2

Shape context3

Representation

HKS4

SI-HKS5

vHKS6

1 Gelfand et al. 2005; 2 Johnson, Hebert 1999; 3 Belongie et al. 2002; 4 Sun et al. 2009

Shape descriptors

Any

Volume/Mesh

Any

Any

Any

Any

Volume/Mesh

5 B, Kokkinos 2010; 6 Raviv, BBK 2010

8Michael Bronstein Heat diffusion descriptors

Diffusion geometry

Amount of heat transferred from point x to point y in time t

Heat equation

where

- positive semidefinite Laplace-Beltrami

operator

- heat distribution

Fundamental solution (heat kernel, ) – heat equation solution

for initial conditions

Spectral expression

9Michael Bronstein Heat diffusion descriptors

Sun, Ovsjanikov, Guibas, 2009

Heat kernel interpretation

Geometric interpretation: “multiscale Gaussian curvature”

Probabilistic interpretation: the probability of a random walk to remain

at point x after time t.

10Michael Bronstein Heat diffusion descriptors

Sun, Ovsjanikov, Guibas, 2009

Heat kernel signature

Multiscale descriptor

Time (scale)

■ Intrinsic, hence deformation-invariant

■ Provably informative

■ Efficiently computable on different shape representations

■Multiscale

11Michael Bronstein Heat diffusion descriptors

Ovsjanikov, BB, Guibas, 2009BB. Ovsjanikov, Guibas 2010

Shape

Geometric vocabulary

Bag of geometric words

12Michael Bronstein Heat diffusion descriptors

Index in geometric vocabulary1 64

Ovsjanikov, BB, Guibas, 2009BB. Ovsjanikov, Guibas 2010

Bags of geometric words

13Michael Bronstein Heat diffusion descriptors

B et al. 2010

SHREC 2010: Robust shape retrieval benchmark

Transformation

Query set

Database (>1K shapes)

14Michael Bronstein Heat diffusion descriptors

B et al. 2010

Query Toldo et al. 2009 Shape

15Michael Bronstein Heat diffusion descriptors

Bags of words using HKS descriptor,vocabulary of size 48

Shape

Bags of words using spin imagedescriptor

Performance results

Toldo et al. 2009

Toldo et al. 2009B et al. 2010

Performance criterion: mean average precision (mAP) in %

16Michael Bronstein Heat diffusion descriptors

Scale invariance

Original shape Scaled by

Not scale invariant!

17Michael Bronstein Heat diffusion descriptors

Scale-invariant HKS

B, Kokkinos CVPR 2010

Log scale-space

Scaling = shift and multiplicative constant

log + d/d

Undo scaling

Fourier transformmagnitude

Undo shift

0 100 200 300-15

-10

-5

0

0 100 200 300

-0.04

-0.03

-0.02

-0.01

0

0 2 4 6 8 10 12 14 16 18 200

1

2

3

4

=2k/T

18Michael Bronstein Heat diffusion descriptors

Scale invariant HKS

B, Kokkinos 2010

HKS SI-HKS

19Michael Bronstein Heat diffusion descriptors

B, Kokkinos 2010

Query HKS SI-HKS

20Michael Bronstein Heat diffusion descriptors

B, Kokkinos 2010

HKS, vocabulary of size 48 SI-HKS, vocabulary of size 48

HKS vs SI-HKS

Performance criterion: mean average precision (mAP) in %

21Michael Bronstein Heat diffusion descriptors

Expressions

In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem.

In biological science, decomposition is the process of organisms to break down into simpler form of matter. Usually, decomposition occurs after death.

Matrix is a science fiction movie released in 1999. Matrix refers to a simulated reality created by machines in order to subdue the human population.

mat

rix d

ecom

posi

tion

mat

rix fa

ctor

izat

ion

scie

nce

fictio

nca

noni

cal f

orm

In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem.

In biological science, decomposition is the process of organisms to break down into simpler form of matter. Usually, decomposition occurs after death.

Matrix is a science fiction movie released in 1999. Matrix refers to a simulated reality created by machines in order to subdue the human population.

mat

rixde

com

posi

tion is a

the of in to by

scie

nce

form

In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem.

Matrix is a science fiction movie released in 1999. Matrix refers to a simulated reality created by machines in order to subdue the human population.

Ovsjanikov, BB & Guibas 2009

22Michael Bronstein Heat diffusion descriptors

Expressions

In math science, matrix decomposition is a factorization of a matrix into some canonical form. Each type of decomposition is used in a particular problem.

mat

rixde

com

posi

tion is a

the of in to by

scie

nce

form

In particular matrix used type a some science, decomposition form a factorization of is canonical. matrix math decomposition is in a Each problem. into of

mat

rix d

ecom

posi

tion

mat

rix fa

ctor

izat

ion

scie

nce

fictio

nca

noni

cal f

orm

Ovsjanikov, BB & Guibas 2009

23Michael Bronstein Heat diffusion descriptors

Geometric expressions

Ovsjanikov, BB & Guibas 2009

“Yellow Yellow”Yellow

No total order between points (only “far” and “near”)

Geometric expression = a pair of spatially close geometric words

24Michael Bronstein Heat diffusion descriptors

Ovsjanikov, BB & Guibas 2009

Spatially-sensitive bags of words

25Michael Bronstein Heat diffusion descriptors

B et al. 2010

HKS, vocabulary of size 48 Spatially-sensitive HKS, vocabulary of size 8x8

HKS vs SI-HKS

Performance criterion: mean average precision (mAP) in %

26Michael Bronstein Heat diffusion descriptors

Is our shape model good?

Raviv, BBK 2010

Boundary ∂X

Interior

X

27Michael Bronstein Heat diffusion descriptors

Is our shape model good?

Camel illustration from Sumner et al.Raviv, BBK 2010

Volume isometryBoundary isometry

Preserves geodesic distances on the boundary surface

Preserves geodesic distances inside the volume

28Michael Bronstein Heat diffusion descriptors

where

Diffusion equation

Raviv, BBK 2010

Volumetric diffusionBoundary diffusion

- Laplace-Beltrami operator - Euclidean Laplacian

- normal to boundary surface

29Michael Bronstein Heat diffusion descriptors

where

Heat kernels

Raviv, BBK 2010

Volumetric heat kernelBoundary heat kernel

Geometric interpretation

“Multiscale Gaussian curvature”

30Michael Bronstein Heat diffusion descriptors

Heat kernel signatures

Raviv, BBK 2010

vHKSHKS

Boundary+volume isometry Boundary+volume isometry

Boundary isometry Boundary isometry

31Michael Bronstein Heat diffusion descriptors

Raviv, BBK 2010

vHKSHKS

32Michael Bronstein Heat diffusion descriptors

HKS, vocabulary of size 48 vHKS, vocabulary of size 48

HKS vs vHKS

Raviv, BBK 2010

Performance criterion: mean average precision (mAP) in %

33Michael Bronstein Heat diffusion descriptors

Summary

Feature descriptor

Geometric words

Bag of geometric words

Geometric expressions

Spatially-sensitive bag of words

“ ”“ ”

Volumetric descriptors

Scale invariance

34Michael Bronstein Heat diffusion descriptors

Thank you