shape matching - university of...

57
Shape Matching Brandon Smith and Shengnan Wang Computer Vision CS766 Fall 2007

Upload: others

Post on 10-May-2020

8 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Shape MatchingBrandon Smith and Shengnan Wang

Computer Vision – CS766 – Fall 2007

Page 2: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Outline

• Introduction and Background

• Uses of shape matching

• Kinds of shape matching

• Support Vector Machine (SVM)

• Matching with Shape Contexts

• Shape Context

• Bipartite Graph Matching

• Modeling Transformations

• Invariance and Robustness

• Results

• Questions

• Shengnan’s part…

Page 3: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Introduction and Background

Shape matching examples

Fruit InspectionFingerprint Matching

Hieroglyph Lookup Trademark Lookup

Page 4: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Introduction and Background

Page 5: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Introduction and Background

• Feature-Based Methods

• Brightness-Based Methods

Page 6: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Introduction and Background

Feature-Based Methods

Page 7: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Introduction and Background

Brightness-Based Methods

Two different frameworks:

• Explicitly find correspondences

• Build classifiers without explicitly finding

correspondences.

Page 8: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Introduction and Background

Support Vector Machine (SVM)

Page 9: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Introduction and Background

Approach:

1.Find correspondences between shapes

2.Estimate an aligning transform

3.Measure similarity

Page 10: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Shape Context

Page 11: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Shape Context

Page 12: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Shape Context

3

9

Page 13: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Shape Context

Belongie, et al, PAMI 2002, Shape matching and object recognition using shape contexts

Page 14: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Shape Context

Belongie, et al, PAMI 2002, Shape matching and object recognition using shape contexts

K

k ji

ji

jiijkhkh

khkhppCC

1

2

)()(

)()(

2

1),(

pi pj

Page 15: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Bipartite Graph Matching

i

ii qpCH ),()( )(

)( 3NOSolved in about

Belongie, et al, PAMI 2002, Shape matching and object recognition using shape contexts

Page 16: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Modeling Transformations

Belongie, et al, PAMI 2002, Shape matching and object recognition using shape contexts

Page 17: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Belongie, et al, PAMI 2002, Shape matching and object recognition using shape contexts

Thin Plane Spline (TPS) Model

(2D Generalization of Cubic Spline)

Page 18: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape ContextsThin Plane Spline (TPS) Model

(2D Generalization of Cubic Spline)

Page 19: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Belongie, et al, PAMI 2002, Shape matching and object recognition using shape contexts

Page 20: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Invariance and Robustness

Belongie, et al, PAMI 2002, Shape matching and object recognition using shape contexts

• Invariant under translation and scaling

• Insensitive to small affine distortion

• Can be made invariant to rotation

Page 21: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Evaluation and Results

Model Point Sets

Deformation Noise Outliers

Target Point Sets

Page 22: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Evaluation and Results

Deformation Noise Outliers

Belongie et al. Chui and Rangarajan Iterated closed point*

Page 23: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching with Shape Contexts

Evaluation and Results

Page 24: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Conclusion

Page 25: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Questions

Page 26: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Shape and Image Matching

Shengnan Wang

[email protected]

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 27: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Today

• The Pyramid Match Kernel: Discriminative Classification with Sets of Image Features

- Kristen Grauman & Trevor Darrell

- MIT

• Matching Local Self-Similarities across Images and Videos

- Eli Shechtman & Michal Irani

- @ CVPR07

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 28: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Set Representation

invariant region

descriptors

local shape features examples under varying

conditions

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 29: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Motivation

• How to build a discriminative classifier using the set representation?

• Kernel-based methods (e.g. SVM) are appealing for efficiency and generalization power…

• What determines the appropriates of a kernel?

- Each instance is unordered set of vectors

- Varying number of vectors per instance

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 30: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Pyramid Match Kernel

optimal partial

matching

Page 31: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 32: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Example pyramid match

Level 0

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 33: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Example pyramid match

Level 1

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 34: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Example pyramid match

Level 2

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 35: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Pyramid match kernel

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 36: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Example pyramid match

pyramid match

optimal match

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 37: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Summary: Pyramid match kernel

• linear time complexity: m features of dimension d, L-level pyramid

• model-free

• insensitive to clutter

• positive-definite function

• no independence assumption

• fast, effective object recognition

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 38: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Object recognition results

• ETH-80 database :8 object classes

• Features:

– Harris detector

– PCA-SIFT descriptor, d=10

Kernel Complexity Recognition rate

Match [Wallraven et al.]

84%

Bhattacharyya affinity [Kondor &

Jebara]

85%

Pyramid match 84%11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 39: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Object recognition results

• Caltech objects database 101 object classes

• Features:– SIFT detector– PCA-SIFT descriptor, d=10

• 30 training images / class• 43% recognition rate

(1% chance performance)• 0.002 seconds per match

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 40: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Localization

• Inspect intersections to obtain correspondences between features

• Higher confidence correspondences at finer resolution levels

observationtarget

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 41: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Future work

• Geometric constraints

• Fast search of large databases with the pyramid match for image retrieval

• Use as a filter for a slower, explicit correspondence method

• Alternative feature types and classification domains

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 42: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Next

• Matching Local Self-Similarities across Images and Videos

- Eli Shechtman & Michal Irani

- @ CVPR07

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 43: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

What do they do?

How to measure similarity between visual entities (images or videos)

Page 44: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

What’s new?

.VS.

• Traditional idea: they share some common visual properties (colors, intensity, gradients, edges or other filter response)

• New idea: they share the same local geometry layout.

Local self repeat

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 45: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Self-similarity descriptor

Patch Local Similarity Map Self-similarity descriptor

SSD LPC+Max

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 46: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Corresponding Self-similarity

descriptor

G

F

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 47: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Self-similarity descriptor

• Benefits

– self-similarity descriptor: local descriptors, wider applicability

– log-polar: accounts for local affine deformations

– maximal correlation value: insensitive to the exact best match position*

– use of patches: more meaningful image patterns

*T. Wolf, L. Bileschi, S. Riesenhuber, M. Poggio, T., Robust Object Recognition with Cortex-Like Mechanisms Serre, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2007

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 48: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching ensemble of patches*

*O. Boiman and M. Irani. Detecting irregularities in images and in video. In

IEEE International Conference on Computer Vision, Beijing, October 2005.

Center

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 49: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Matching ensemble of patches

• Two Implementation Details

– Filter out non-informative descriptors

• Descriptors that do not capture any local self-similarity (salient points)

• Descriptors that contain high self-similarity everywhere (homogeneous region)

– Scale space representation

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 50: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Overview

Matching

1. Self-similarity

descriptor

2. Matching ensemble of patches

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 51: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Self-similarity descriptor

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 52: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Experiments

• Sketch Detection

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 53: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Experiments (1)

• Object Detection

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 54: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Experiments (2)• Image

Retrieval

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 55: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Experiments (3)

• Action detection

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 56: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Experiments (4)

• Action detection

11/4/07 Shengnan Wang University of Wisconsin-Madison

Page 57: Shape Matching - University of Wisconsin–Madisonpages.cs.wisc.edu/~lizhang/courses/cs766-2007f/...–self-similarity descriptor: local descriptors, wider applicability –log-polar:

Questions?

Thank you!

11/4/07 Shengnan Wang University of Wisconsin-Madison