![Page 1: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/1.jpg)
Learning Decompositional Learning Decompositional Shape Models from Shape Models from
ExamplesExamples
Alex LevinshteinAlex Levinshtein
Cristian SminchisescuCristian Sminchisescu
Sven DickinsonSven Dickinson
University of TorontoUniversity of Toronto
![Page 2: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/2.jpg)
Hierarchical ModelsHierarchical ModelsManually built hierarchical model proposed by Marr And Nishihara(“Representation and recognition of the spatial organization of three dimensional shapes”, Proc. of Royal Soc. of London, 1978)
![Page 3: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/3.jpg)
Our goalOur goal
Automatically construct a generic hierarchical shape model from exemplars
Challenges:
Cannot assume similar appearance among different exemplars
Generic features are highly ambiguous
Generic features may not be in one-to-one correspondence
![Page 4: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/4.jpg)
Automatically constructed Automatically constructed Hierarchical ModelsHierarchical Models
Input:
Question: What is it?
Output:
![Page 5: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/5.jpg)
Stages of the systemStages of the systemExemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
![Page 6: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/6.jpg)
Blob Graph ConstructionBlob Graph ConstructionExemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
![Page 7: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/7.jpg)
Blob Graph ConstructionBlob Graph Construction
Edges are invariant to articulation
Choose the largest connected component.
On the Representation and Matching of Qualitative Shape at Multiple Scales
A. Shokoufandeh, S. Dickinson, C. Jonsson, L. Bretzner, and T. Lindeberg,ECCV 2002
![Page 8: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/8.jpg)
Feature matchingFeature matchingExemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
![Page 9: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/9.jpg)
Feature matchingFeature matchingOne-to-one matching. Rely on shape and context, not appearance!
?
Many-to-many matching
![Page 10: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/10.jpg)
Feature embedding and Feature embedding and EMDEMD
Spectral embedding
![Page 11: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/11.jpg)
Returning to our set of Returning to our set of inputsinputs
Many-to-many matching of every pair of exemplars.
![Page 12: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/12.jpg)
Part ExtractionPart ExtractionExemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
![Page 13: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/13.jpg)
![Page 14: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/14.jpg)
Results of the part extraction Results of the part extraction stagestage
![Page 15: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/15.jpg)
What is next?What is next?
![Page 16: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/16.jpg)
Extracting attachment Extracting attachment relationsrelations
Exemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
![Page 17: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/17.jpg)
Extracting attachment Extracting attachment relationsrelations
Right arm is typically connected to torso in exemplar images !
![Page 18: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/18.jpg)
Extracting decomposition Extracting decomposition relationsrelations
Exemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
![Page 19: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/19.jpg)
Extracting decomposition Extracting decomposition relationsrelations
![Page 20: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/20.jpg)
Model construction stage Model construction stage summarysummary
Model Construction:
Clustering blobs into parts based on one-to-one matching results.
Recovering relations between parts based on individual matching and attachment results.
![Page 21: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/21.jpg)
Assemble Final ModelAssemble Final ModelExemplar images
Extract Blob Graphs
Match Blob Graphs (many-
to-many)
Extract PartsExtract
Decomposition Relations
Extract Attachment Relations
Assemble Final Model
Blob graphs
Many-to-many correspondences
Model parts
Model decomposition relations
Model attachment relations
![Page 22: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/22.jpg)
![Page 23: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/23.jpg)
![Page 24: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/24.jpg)
ConclusionsConclusions General framework for constructing a General framework for constructing a
generic decompositional model from generic decompositional model from different exemplars with dissimilar different exemplars with dissimilar appearance.appearance.
Recovering decompositional relations Recovering decompositional relations requires solving the difficult many-to-requires solving the difficult many-to-many graph matching problem.many graph matching problem.
Preliminary results indicate good model Preliminary results indicate good model recovery from noisy features.recovery from noisy features.
![Page 25: Learning Decompositional Shape Models from Examples Alex Levinshtein Cristian Sminchisescu Sven Dickinson Sven Dickinson University of Toronto](https://reader030.vdocuments.net/reader030/viewer/2022032805/56649ee15503460f94bf21f6/html5/thumbnails/25.jpg)
Future workFuture work
Construct models for objects other than Construct models for objects other than humans.humans.
Provide scale invariance during matching.Provide scale invariance during matching.
Automatically learn perceptual grouping Automatically learn perceptual grouping relations from labeled examples.relations from labeled examples.
Develop indexing and matching framework Develop indexing and matching framework for decompositional models.for decompositional models.