probabilistic object recognition and localization bernt schiele, alex pentland, iccv 99 presenter:...

72
Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV ‘99 Presenter: Matt Grimes

Upload: francesca-gelling

Post on 31-Mar-2015

220 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Probabilistic Object Recognition and Localization

Bernt Schiele, Alex Pentland, ICCV ‘99

Presenter: Matt Grimes

Page 2: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

What they did

1. Chose a set of local image descriptors whose outputs are robust to object orientation and lighting.

– Examples:

Laplacian

22),(),(),( yxGyxGyxMag yx

First-derivative magnitude:

Page 3: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

What they did

2. Learn a PDF for the outputs of these descriptors given an image of the object:

otherobjectMP ,|

Vector of descriptor

outputsA particular

object

Object orientation,

lighting, etc.

Page 4: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

What they did

2. Learn a PDF for the outputs of these descriptors given an image of the object:

objectMP |

Vector of descriptor

outputsA particular

object

Page 5: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

• Use Bayes’ rule to obtain the posterior…

• …which is the probability of an image containing an object, given local image measurements M.

• (Not quite this clean)

What they did

MobjectP |

Page 6: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

History of image-based object recognition

Two major genres: 1. Histogram-based approaches.

2. Comparison of local image features.

Page 7: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Histogramming approaches

• Object recognition by color histograms (Swain & Ballard, IJCV 1991)– Robust to changes in orientation, scale.– Brittle against lighting changes (dependency on

color).– Many classes of objects not distinguishable by

color distribution alone.

Page 8: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Histogramming approaches

• Combat color-brittleness using (quasi-) invariants of color histograms:– Eigenvalues of matrices of moments of color

histograms – Derivatives of logs of color channels– “Comprehensive color normalization”

Page 9: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Histogramming approaches

• Comprehensive color normalization examples:

Page 10: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Histogramming approaches

• Comprehensive color normalization examples:

Page 11: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Localized feature approaches

• Approaches include:– Using image “interest-points” to index into a

hashtable of known objects.– Comparing large vectors of local filter

responses.

Page 12: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Geometric Hashing

1. An interest point detector finds the same points on an object in different images.

Types of “interest points” include corners, T-junctions, sudden texture changes.

Page 13: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Geometric Hashing

From Schmid, Mohr, Bauckhage, “Comparing and Evaluating Interest Points,” ICCV ‘98

Page 14: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Geometric Hashing

From Schmid, Mohr, Bauckhage, “Comparing and Evaluating Interest Points,” ICCV ‘98

Page 15: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Geometric Hashing

2. Store points in an affine-transform-invariant representation.

3. Store all possible triplets of points as keys in a hashtable.

Page 16: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Geometric Hashing

4. For object recognition, find all triplets of interest points in an image, look for matches in the hashtable, accumulate votes for the correct object.

Hashtable approaches support multiple object recognition within the same image.

Page 17: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Geometric hashing weaknesses

• Dependent on the consistency of the interest point detector used.

From Schmid, Mohr, Bauckhage, “Comparing and Evaluating Interest Points,” ICCV ‘98

Page 18: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Geometric hashing weaknesses

• Shoddy repeatibility necessitates lots of points.

• Lots of points, combined with noise, leads to lots of false positives.

Page 19: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Vectors of filter responses

• Typically use vectors of oriented filters at fixed grid points, or at interest points.

• Pros: – Very robust to noise.

• Cons: – Fixed grid needs large representation, large grid is

sensitive to occlusion.

– If using an interest point detector instead, the detector must be consistent over a variety of scenes.

Page 20: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Also: eigenpictures

• Calculate the eigenpictures of a set of images of objects to be recognized.

• Pros: – Efficient representation of images by their eigenpicture

coefficients. (Fast searches)

• Cons: – Images must be pre-segmented. – Eigenpictures are not local (sensitive to occlusion).– Translation, image-plane rotation must be represented

in the eigenpictures.

Page 21: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

This paper:

• Uses vectors of filter responses, with probabilistic object recognition.

otherobjectMP ,|

MobjectP |Bayes’ rule

objectMP |

Learned from training images

Using scene-invariant M

Page 22: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Wins of this paper

• Uses hashtables for multiple object recognition.

• Unlike geometric hashing, doesn’t depend on point correspondence betw. images.– Uses location-unspecific filter responses, not

points.– Inherits robustness to noise of filter response

methods.

Page 23: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Wins of this paper

• Uses local filter responses.– Robust to occlusion compared to global

methods (e.g. eigenpictures or filter grids.)

• Probabilistic matching – Theoretically cleaner than voting.– Combined with local filter responses, allows for

localization of detected objects.

Page 24: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Details of the PDF

• What degrees of freedom are there in the “other” parameters?

otherobjectMP ,|

ILSTRoMP n ,,,,,|on: Object

R: Rotation (3 DOF)

T: Translation(3 DOF)

S: Scene (occlusions, background)

L: Lighting

I: Imaging (noise, pixelation/blur)

Page 25: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

P(M|on,R,T,S,L,I)

• Way too many params to get a reliable estimate from even a large image library.

• # of examples needed is exponential in the number of dimensions of the PDF.

• Solution: choose measurements (M) that are invariant with respect to as many params as possible (except on).

Page 26: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Techniques for invariance

• Imaging (noise:) see Schiele’s thesis.

• Lighting: apply a “energy normalization technique” to the filter outputs.

• Scene: probabilistic object recognition + local image measurements.– Gives best estimate using the visible portion of

the object.

Page 27: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Techniques for invariance

• Translation: – Tx, Ty (image-plane translation) are ignored for

non-localizing recognition.– Tz is equivalent to scale. For known scales,

compensate by scaling the filters’ regions of support.

Page 28: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Techniques for invariance

• Fairly robust to unknown scale:

Page 29: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Techniques for invariance

• Rotation:– Rz: rotation in the image plane. Filters invariant

to image-plane rotation may be used.– Rx, Ry must remain in the PDF. Impossible to

have viewpoint- invariant descriptors in the general case.

Page 30: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

• 4 parameters.• Still a large amount of training examples

needed, but feasible.• Example: algorithm has been successful

after training with 108 images per object.(108 = 16 orientations * 6 scales)

New PDF

zzyxn trrroMP ,,,,|

Page 31: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Learning & representation of the PDF

• Since the goal is discrimination, overgeneralization is scarier than overfitting.

• They chose multidimensional histograms over parametric representations.

• They mention that they could’ve used kernel function estimates.

Page 32: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multidimensional Histograms

zzyxn trrrommP ,,,,|, 21

Page 33: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multidimensional Histograms

• In their experiments, they use a 6-dimensional histogram.– X and Y derivative, at 3 different scales

• …with 24 buckets per axis.– Theoretical max for # of cells: 246=1.9 x 108

• Way too many cells to be meaningfully filled by even 512 x 512 (=262144 ) pixel images.

Page 34: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multidimensional Histograms

• Somehow, by exploiting dependencies betw. histogram axes, and applying a uniform prior bias, they get the number of calculated cells below 105.

• Factor of 1000 reduction.

• Anybody know how they do this?

Page 35: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

(Single) object recognition

k

nnkkn mP

oPomPmoP

||

Page 36: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

(Single) object recognition

iiik

nnk

oPomP

oPomP

|

|

k

nnkkn mP

oPomPmoP

||

Page 37: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

(Single) object recognition

iiijk

nnjkjkn opommp

opommpmmop

|

||

• A single measurement vector mk is insufficient for recognition.

Page 38: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

(Single) object recognition

iiijk

nnjkjkn opommp

opommpmmop

|

||

• A single measurement vector mk is insufficient for recognition.

iiijik

nnjnk

opompomp

opompomp

||

||

Page 39: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

(Single) object recognition

iiik

k

nnkk

kk

n opomp

opompmop

)()|(

)()|()|(

ik iki

k nkn

ompop

ompop

)|()(

)|()(

• For k measurement vectors:

Page 40: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

(Single) object recognition

Page 41: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

(Single) object recognition

Page 42: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

(Single) object recognition

Page 43: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

(Single) object recognition

• Measurement regions covering 10~20% of an object are usually sufficient for discrimination.

Page 44: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

(Single) object recognition

Page 45: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

• We can apply the single-object detector to many small regions in the image.

Page 46: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

• The algorithm is now O(NKJ)– N = # of known objects– K = # of measurement vectors in each region– J = # of regions

Page 47: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 48: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 49: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 50: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 51: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 52: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 53: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 54: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 55: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 56: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 57: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 58: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Multiple object recognition

Page 59: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

One drawback:

• For a given image, the algorithm calculates a probability for each object it knows of.

• The algorithm lists the objects in its library in decreasing order of probability.

• Need to know beforehand the number of objects in a test image, to know where to stop reading the list.

Page 60: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Failure example

Page 61: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Failure example

Page 62: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Failure example

Page 63: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Failure example

Page 64: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Failure example

Page 65: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Unfamiliar clutter

Page 66: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Unfamiliar clutter

Page 67: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Unfamiliar clutter

Page 68: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Unfamiliar clutter

Page 69: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

• Bite the dimensionality bullet and add an object position variable to the PDF:

Object localization

k

jnjnkkjn mp

xopxompmxop

,,||,

Page 70: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

• Stop assuming independence of mks, to account for structural dependencies:

Object localization

llk

jnjnljnlklkjn mpmmp

xopxompxommpmmxop

|

,,|,,|,|,

Page 71: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Object localization

• Tradeoff between recognition and localization, depending on region size.

Page 72: Probabilistic Object Recognition and Localization Bernt Schiele, Alex Pentland, ICCV 99 Presenter: Matt Grimes

Object localization

• Heirarchical discrimination with coarsefine region size refinement: