methods for 3d shape matching and retrieval marcin novotni & reinhard klein university of bonn...

Methods for 3D Shape Matching and Retrieval

Marcin Novotni & Reinhard KleinMarcin Novotni & Reinhard KleinUniversity of BonnUniversity of Bonn

Computer Graphics GroupComputer Graphics Group

Marcin Novotni Marcin Novotni Reinhard Klein Reinhard KleinUniversity of Bonn University of Bonn Computer Graphics Group Computer Graphics Group

Our Aim #1Our Aim #1Our Aim #1Our Aim #1

Given an example:Given an example:

,, ,…

Find the most Find the most

similar object(s)similar object(s)

in a databasein a database


MotivationMotivationMotivationMotivation

Lots of 3D archives:Lots of 3D archives: WWW WWW Proprietary databasesProprietary databases ......

Search engines for data:Search engines for data: Text, 2D images, music (MIDI), …Text, 2D images, music (MIDI), … Emerging since 1998 for 3DEmerging since 1998 for 3D



Direct matching Direct matching AlignmentAlignment Establishing correspondencesEstablishing correspondences



Partial matching/retrievalPartial matching/retrieval



Partial matching/retrievalPartial matching/retrievalStatistical shape analysisStatistical shape analysisMorphingMorphingTexture transferTexture transferRegistrationRegistration


General ProblemGeneral ProblemGeneral ProblemGeneral Problem

Abstract representationAbstract representation facilitating: facilitating:

identification of salient features of 3D identification of salient features of 3D objectsobjects

description of featuresdescription of featurescomparison (matching)comparison (matching)


OverviewOverviewOverviewOverview

Matching for 3D Shape RetrievalMatching for 3D Shape Retrieval

Correspondence MatchingCorrespondence Matching


Matching for Matching for 3D Shape 3D Shape RetrievalRetrieval



We need a We need a DescriptorDescriptor

D : → D( )


d( , )

d( , )


We need a We need a Distance MeasureDistance Measure

:=

D( )

D( )


d( , ) d( , )


We need a We need a Distance MeasureDistance Measure : : Close to (application driven) notion of resemblanceClose to (application driven) notion of resemblance Computationally cheap and robustComputationally cheap and robust

d( , )≤ ≤


3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors3D Zernike Descriptors

Feature vectorsFeature vectors

XXii : 3D Zernike Descriptors : 3D Zernike Descriptors [Canterakis ’99, Novotni & Klein ’03, ’04][Canterakis ’99, Novotni & Klein ’03, ’04]Distance Measure: Euclidean DistanceDistance Measure: Euclidean Distance

D( ) ≡x1

xn



Retrieval performance Retrieval performance [Novotni & Klein ‘03 ’04][Novotni & Klein ‘03 ’04]

Slightly better than [Funkhouser et al. ’02]Slightly better than [Funkhouser et al. ’02]

Object class dependent performance!Object class dependent performance! Class dependent coefficient importance!Class dependent coefficient importance!


3D 3D Zernike DescriptorsZernike Descriptors3D 3D Zernike DescriptorsZernike Descriptors

Importance

Coeff No. (Frequency)

ChairsFaces Airplanes



Relevance feedback:Relevance feedback: User selects relevant / irrelevant itemsUser selects relevant / irrelevant items Distance measure is tunedDistance measure is tuned

Learning Machines:Learning Machines: SVM (Support vector machines) [Vapnik ‘95]SVM (Support vector machines) [Vapnik ‘95] One class SVM [SchOne class SVM [Schöölkopf et al. lkopf et al. ’’9999]] (K)BDA ((Kernel) Biased Discriminant (K)BDA ((Kernel) Biased Discriminant

Analysis) [Zhou et al. ‘01]Analysis) [Zhou et al. ‘01]


CorrespondenCorrespondence Matchingce Matching


Geometric Similarity EstimationGeometric Similarity EstimationGeometric Similarity EstimationGeometric Similarity Estimation

Idea Idea [Novotni & Klein 2001][Novotni & Klein 2001]:: Definition of „geometric“ similarity in terms of Definition of „geometric“ similarity in terms of

a geometric distancea geometric distance

Intuitive, simple, robust.Intuitive, simple, robust.



6.786.78 8.858.85 30.2930.29 38.0938.09 67.5367.53Normalized Normalized volumetric volumetric

errorerror0.000.00

Database objects

example



Classification by user set thresholdClassification by user set threshold



Measures deformation magnitudeMeasures deformation magnitude


Correspondence MatchingCorrespondence MatchingCorrespondence MatchingCorrespondence Matching

?



Ideally: dense mappingIdeally: dense mapping

?

22: f




Deformation by mapping semanticsDeformation by mapping semantics

[D’Arcy Thompson 1917: On Growth and Form ]

22: f

http://www.amazon.com/exec/obidos/tg/detail/-/0486671356?v=glance




Easier: mapping Easier: mapping salient pointssalient points Curvature extremesCurvature extremes Corners (Harris points in 2D)Corners (Harris points in 2D) Etc…Etc… Scale space extremesScale space extremes

22: f



Scale Space extremes Scale Space extremes [Lindeberg ‘94][Lindeberg ‘94]



We have:We have:Salient pointsSalient points

Spatial positionSpatial position Size of local blobsSize of local blobs

How to match???How to match???



Criteria for correspondences:Criteria for correspondences:

SimilarSimilarLocal geometriesLocal geometriesConstellations of pointsConstellations of points



Local descriptionLocal description



Assumption:Assumption:

Similar local descriptors Similar local descriptors

Similar local geometriesSimilar local geometries



Criteria for correspondences:Criteria for correspondences:

SimilarSimilarLocal geometriesLocal geometriesConstellations of pointsConstellations of points



Similar constellations of pointsSimilar constellations of points Smooth mappings leave constellations Smooth mappings leave constellations

consistentconsistent

IdeaIdea Constellations are consistent if mapping is Constellations are consistent if mapping is

smoothsmooth



Similar constellations of pointsSimilar constellations of pointsIdea: Idea:

Constellations are consistent if mapping is Constellations are consistent if mapping is smoothsmooth

Thin Plate Spline interpolation [Brookstein ’89]Thin Plate Spline interpolation [Brookstein ’89]

minimize:minimize:2( ) ( )

d

I f f d x x

Total curvature


minimize:minimize:

Minimizer (Minimizer (Thin Plate SplineThin Plate Spline interpolator): interpolator):


2( ) ( )d

I f f d x x

1

( )( ) ( )

( )

xx lkyy

k k

wf

wf

k

xf x Ax t x x

x

Affine part Nonlinear deformation


minimize:minimize:



2( ) ( )d

I f f d x x

1

( )( ) ( )

( )

xx lkyy

k k

wf

wf

k

xf x Ax t x x

x

2( ) logr r r 2D Thin Plate Spline


1

( )( ) ( )

( )

xx Nkyy

k k

wf

wf

k

xf x Ax t x x

x

minimize:minimize:



2( ) ( )d

I f f d x x

Can be computed by a (N+4)x(N+4) matrix

inversion



Find (sub)sets of correspondences:Find (sub)sets of correspondences: Small local descriptor distancesSmall local descriptor distances Small deformation energySmall deformation energy

Hierarchical pruning and clusteringHierarchical pruning and clusteringUsing:Using:

Local descriptorsLocal descriptors Geometrical constellation consistencyGeometrical constellation consistency



New avenuesNew avenues::Local Descriptions for retrievalLocal Descriptions for retrievalOnline Learning for local Online Learning for local

descriptionsdescriptionsDense matching from salient pointsDense matching from salient pointsEtc.Etc.


Danke,Danke,DFG!DFG!


Basis functions in the unit sphere:Basis functions in the unit sphere:


SH on the sphere

( , , ) ( ) ( , )m mnl nl lZ r R r Y




SH on the sphere

Function of the radius

( , , ) ( ) ( , )m mnl nl lZ r R r Y

Rotation invariant!



3D Zernike Moments 3D Zernike Moments [Canterakis ‘99]:[Canterakis ‘99]:

: ,m mnl nlf Z


( , , ) ( ) ( , )m mnl nl lZ r R r Y

Object function, e.g. voxel grid



3D Zernike Descriptors:3D Zernike Descriptors: Amplitudes of the Zernike decompositionAmplitudes of the Zernike decomposition

Rotation invariantRotation invariant

1

: ,

lnllnl

nl

lnl

F

0

0

. . ( ) is even

n N

l n

s t n l




SH on the sphere

( , , ) ( ) ( , )m mnl nl lr R r Y




SH on the sphere

Function of the radius

( , , ) ( ) ( , )m mnl nl lr R r Y

Rotation invariant!



3D Zernike Moments 3D Zernike Moments [Canterakis ‘99][Canterakis ‘99]::: ,m m

nl nlf Z


( , , ) ( ) ( , )m mnl nl lr R r Y

Object function, e.g. voxel grid





1

: ,

lnllnl

nl

lnl

F

0

0

. . ( ) is even

n N

l n

s t n l



For N=22 : 155 floats as search keyFor N=22 : 155 floats as search key

Timings (1.8 GHz Pentium):Timings (1.8 GHz Pentium):Voxelization: 0.3 – 10.0 sec / objectVoxelization: 0.3 – 10.0 sec / objectComputation: 0.2 sec / objectComputation: 0.2 sec / objectRetrieval (1814 objects): 0.3 secRetrieval (1814 objects): 0.3 sec



Retrieval performance Retrieval performance [Novotni & Klein ’04][Novotni & Klein ’04]




3D 3D Zernike DescriptorsZernike Descriptors3D 3D Zernike DescriptorsZernike Descriptors

Importance

Coeff No.

ChairsFaces Airplanes



3D Zernike functions [Canterakis ‘99]3D Zernike functions [Canterakis ‘99]

are polynomials such that are are polynomials such that are orthonormal within the unit ballorthonormal within the unit ball

( , , ) : ( ) ( , )m mnl nl lZ r R r Y

mnlZnlR



3D Zernike functions [Canterakis ‘99]3D Zernike functions [Canterakis ‘99]

are polynomials such that are orthonormal are polynomials such that are orthonormal within the unit ballwithin the unit ball

3D Zernike Moments:3D Zernike Moments:

( , , ) : ( ) ( , )m mnl nl lZ r R r Y

mnlZnlR

: ,m mnl nlf Z





1

: ,

lnllnl

nl

lnl

F

0

0

. . ( ) is even

n N

l n

s t n l



For N=20 : 121 floats as search keyFor N=20 : 121 floats as search key

Timings (1.8 GHz Pentium):Timings (1.8 GHz Pentium):Voxelization: 0.3 – 10.0 sec / objectVoxelization: 0.3 – 10.0 sec / objectComputation: 0.2 sec / objectComputation: 0.2 sec / objectRetrieval (1814 objects): 0.3 secRetrieval (1814 objects): 0.3 sec



Retrieval performance Retrieval performance [Novotni & Klein ’04][Novotni & Klein ’04]





Matching should be:Matching should be:

Independent of topologyIndependent of topologyRobustRobustSuitable for partial matchingSuitable for partial matching



Local descriptionLocal description Local shape histograms Local shape histograms

Not rotation invariantNot rotation invariant

Rotation invarianceRotation invariance

Amplitudes of the Fourier Amplitudes of the Fourier TransformTransform

( , )r

1ˆ ( , ) ( , ), ip

Sr p r e



155 floats as search key155 floats as search key

Retrieval (1814 objects): 0.3 secRetrieval (1814 objects): 0.3 sec



Stuff to rememberStuff to remember::

Salient points simplify the problemSalient points simplify the problemSmooth mapping iff consistent Smooth mapping iff consistent

constellationsconstellations




Salient points simplify the problemSalient points simplify the problem VolumetricVolumetric On the surfaceOn the surface




Salient points simplify the problemSalient points simplify the problemSmooth mapping iff consistent Smooth mapping iff consistent

constellationsconstellations



New avenuesNew avenues::Local Descriptions for retrievalLocal Descriptions for retrieval

Retrieval by part selection & recognitionRetrieval by part selection & recognitionRetrieval from large scenesRetrieval from large scenes




descriptionsdescriptionsAdopting pattern recognition methodsAdopting pattern recognition methods




descriptionsdescriptionsDense matching from salient pointsDense matching from salient points

Morphing, registration, object statisticsMorphing, registration, object statistics



Direct matching Direct matching AlignmentAlignment



Scale space extremes Scale space extremes [Lindeberg ‘94][Lindeberg ‘94] Blob detection by Blob detection by localizing extremes of Laplacian … localizing extremes of Laplacian … … … in scale and spacein scale and space

Size of the blob

Position of the blob



Maxima of Laplacian over scalesMaxima of Laplacian over scales



Spatial maximaSpatial maxima

methods for 3d shape matching and retrieval marcin novotni & reinhard klein university of bonn...

Documents

d shape matching

d archives

d images

correspondence matchingwe

dense mappingeasier

dense mappingdeformation

class svm schlkopf

similar objectsin