“Secret” of Object DetectionZheng Wu

(Summer intern in MSRNE)Sep. 3, 2010

Joint work with Ce Liu (MSRNE)

William T. Freeman (MIT)Adam Kalai (MSRNE)Jianxiong Xiao (MIT)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 2

motorbike

person

Intro Sliding Windows Features Cascade Classifier PASCAL Con 3

Outline IntroductionSliding-Window-based Object Detection

Window GenerationFeature ExtractionCascade Classifier

PASCAL ChallengeConclusion

Intro Sliding Windows Features Cascade Classifier PASCAL Con 4

Related Topic Image Matching

Image Classification

Object Detection

Intro Sliding Windows Features Cascade Classifier PASCAL Con

Object Detection

• Single Pattern • Multiple Patterns

5

Viola & Jones Face Detector

Dalal & Triggs Pedestrian Detector

Felzenszwalb’s Part-based Detector

Intro Sliding Windows Features Cascade Classifier PASCAL Con 6

Outline IntroductionSliding-Window-based Object Detection

Window GenerationFeature ExtractionCascade Classifier

PASCAL ChallengeConclusion

Intro Sliding Windows Features Cascade Classifier PASCAL Con 7

Sliding Windows

Search over:• Location• Scale• Aspect Ratio

Millions of windows!

Intro Sliding Windows Features Cascade Classifier PASCAL Con 8

Sliding Windows Subsample on grid

- have to set “optimal” step size manually Fix Aspect Ratio

- assume single pattern detection Fix Scale

- assume object’s resolution does not change much between training and test sets.

Search with branch-and-bound method- have to use special scoring function

Intro Sliding Windows Features Cascade Classifier PASCAL Con 9

Sliding Windows We propose sliding windows from segmentation

Superpixel Segmentation[Levinshtein et al, PAMI09]

Region Segmentation[Felzenszwalb & Huttenlocher, IJCV03]

< 100,000 sliding windows / image on PASCAL Dataset

Intro Sliding Windows Features Cascade Classifier PASCAL Con 10

Outline IntroductionSliding-Window-based Object Detection

Window GenerationFeature ExtractionCascade Classifier

PASCAL ChallengeConclusion

Intro Sliding Windows Features Cascade Classifier PASCAL Con 11

Generic Feature “Objectness” features [Alexe et al, CVPR10]

Intro Sliding Windows Features Cascade Classifier PASCAL Con 12

Generic Feature Each type of generic feature is weak, but

combination is stronger Low dimensional feature (=8) Not suitable for objects with “concave” shape, i.e.

table, chair

Intro Sliding Windows Features Cascade Classifier PASCAL Con 13

Generic Feature

aerop

lane

bicyc

le bird

boat

bottle bu

s car

cat

chair co

w

dining

table do

gho

rse

motorbi

ke

perso

n

potte

dplan

t

shee

pso

fatra

in

tvmon

itor

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1re

call

Intro Sliding Windows Features Cascade Classifier PASCAL Con 14

Class-specific Feature Histogram of Orientated Gradients

Intro Sliding Windows Features Cascade Classifier PASCAL Con 15

Class-specific Feature Dense grid (>10*10) (secret 1) Overlapping cells Histogram bin size High dimensional feature (>1000)

- redundant or overfitting? Normalization No spatial relationship maintained

Intro Sliding Windows Features Cascade Classifier PASCAL Con 16

Outline IntroductionSliding-Window-based Object Detection

Window GenerationFeature ExtractionCascade Classifier

PASCAL ChallengeConclusion

Intro Sliding Windows Features Cascade Classifier PASCAL Con 17

Cascade Classifier Same type of classifier with different features

Viola & Jones Face Detector, IJCV01 Different types of classifier with same features

Harzallah et al, ICCV09 (INRIA) Vedaldi et al, ICCV09 (Oxford)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 18

Cascade Classifier Training SVM is slow…

to train 20,000 examples with 4000 dimensions:>15min for Linear SVM

>3 hours for Nonlinear SVM Training SVM requires a lot of memory…

design matrix: 20,000*20,000 matrix Training with Imbalance data a few hundreds of positive examples billions of negative examples

Intro Sliding Windows Features Cascade Classifier PASCAL Con 19

Boosted SVM

Intro Sliding Windows Features Cascade Classifier PASCAL Con 20

42327 examples, half for training, half for testing Training error is 0.05 for all boosted classifiers

Boosted SVM

Intro Sliding Windows Features Cascade Classifier PASCAL Con 21

Positive Training Set

Intro Sliding Windows Features Cascade Classifier PASCAL Con 22

Negative Training Set

Random Samples

SVM ver. 1

Training Sample Pool

False Positives

SVM ver. 2

… … Secret 2

Intro Sliding Windows Features Cascade Classifier PASCAL Con 23

Outline IntroductionSliding-Window-based Object Detection

Window GenerationFeature ExtractionCascade Classifier

PASCAL ChallengeConclusion

Intro Sliding Windows Features Cascade Classifier PASCAL Con 24

PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 25

PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 26

PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 27

PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 28

PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 29

PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 30

PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 31

PASCAL Dataset 2009

Intro Sliding Windows Features Cascade Classifier PASCAL Con 32

PASCAL 2009 (trainval + test)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 33

PASCAL 2009 (train+val)

aeroplane bicycle bus car horse motorbike train tvmonitor0

5

10

15

20

25

Aver

age

Prec

isio

n

Intro Sliding Windows Features Cascade Classifier PASCAL Con 34

PASCAL 2009 (train+val,1/image)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 35

PASCAL 2009 (train+val,5/image)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 36

PASCAL 2009 (train+val,10/image)

Intro Sliding Windows Features Cascade Classifier PASCAL Con 37

True Positives - aeroplane

Intro Sliding Windows Features Cascade Classifier PASCAL Con 38

False Positives - aeroplane

Intro Sliding Windows Features Cascade Classifier PASCAL Con 39

True Positives - bicycle

Intro Sliding Windows Features Cascade Classifier PASCAL Con 40

False Positives - bicycle

Intro Sliding Windows Features Cascade Classifier PASCAL Con 41

True Positives - horse

Intro Sliding Windows Features Cascade Classifier PASCAL Con 42

False Positives - horse

Intro Sliding Windows Features Cascade Classifier PASCAL Con 43

Conclusion Proposing sliding windows without fixing scale or

aspect ratio is possible. Simple feature (saliency, contrast, etc) is only

useful for certain object classes. Histogram-based feature is not sufficient to

represent real world object, no matter what learning algorithm is used.

Boosting is helpful to speed up SVM-training and reduce the memory usage.

Digging out “hard” negative examples. Throwing away “hard” positive examples.

Intro Sliding Windows Features Cascade Classifier PASCAL Con 44

Future Work It is time to go beyond the histogram-of-X feature

- not every pixel within bounding box is informative - the appearance of object’s part is more robust

Evolve the classifier- even PASCAL dataset is too small

- the right decision boundary is still far away… - Active learning? Online learning?

Thank You !

45