object segmentation

Object Segmentation

Presented by

Sherin Aly

1

What is a ‘Good Segmentation’?

http://www.eecs.berkeley.edu/Research/Projects/CS/vision/grouping/resources.html

Learning a classification model for segmentation

Xiaofeng Ren and Jitendra Malik

4

methodology

• Two-class classification model

• Over segmentation as preprocessing

• They use classical Gestalt cues– Contour, texture, brightness and

continuation

• A linear classifier is used for training

5

Good Vs Bad segmentation

6

a) Image from Corel Imagebase

b) superimposed with a human markedsegmentation

c) Same image with Bad segmentation

How do we distinguish good segmentations from bad

segmentations?

7

How?

• Use “Classical Gestalt cues”– proximity, similarity and good continuation

• Instead of Ad-hoc decision about features combination

8

Gestalt Principles of Grouping

9

http://allpsych.com/psychology101/perception.html

In order to interpret what we receive through our senses,we attempt to organize this information into certain groups.

Methodology

• Preprocessing

• Feature extraction

• Feature evaluation

• Training

• Optimization

• Find good segmentaion

10

Preprocessing

11

Superpixel mapK=200

Reconstruction of human segmentation from Superpixels

a contour-based measure is used to quantify this approximation

•Local•Coherent•Preserve structure

•Contour •texture

12 The percentage of human marked boundaries covered by the superpixel maps

Tolerance 1,2,and 3

Feature Extraction

1. inter-region texture similarity

2. intra-region texture similarity

3. inter-region brightness similarity

4. intra-region brightness similarity

5. inter-region contour energy

6. intra-region contour energy

7. curvilinear continuity 13

Feature Extraction

1. inter-region texture similarity

2. intra-region texture similarity

3. inter-region brightness similarity

4. intra-region brightness similarity

5. inter-region contour energy

6. intra-region contour energy

7. curvilinear continuity 14

Feature Extraction

1. inter-region texture similarity

2. intra-region texture similarity

3. inter-region brightness similarity

4. intra-region brightness similarity

5. inter-region contour energy

6. intra-region contour energy

7. curvilinear continuity 15

Power of Gestalt cues

16

=

Training the classifier

• simple logistic regression classifier,

17Empirical distribution of pairs of features

18

Precision is the fraction of detections which are true positives. Recall is the fraction of true positives which are detected

Conclusion

• There simple linear classifier had promising results on a variety of natural images.

• boundary contour is the most informative grouping cue, and it is in essence discriminative.

19

Pros & Cons

• Cons– The larger spatial support that superpixels

provide, allowing more global features to be computed than on pixels alone.

– The use of superpixels improves the computational efficiency

– SuperPixels technique is very applicable

• Pros– Might fall in Local Minima

20

Combining Top-down and Bottom-up Segmentation

Eran Borenstein

Eitan Sharon

Shimon Ullman

21

Motivation

• Bottom-Up segmentation– Rely on continuity principle– Capture image properties “texture, grey level uniformity

and contour continuity”– Segmentation based on similarities between image

regions

• How can we capture prior knowledge of a specific object (class)?– Answer: Top-Down Segmentation– use prior knowledge about an object

Credit: Joseph Djugash

Bottom-Up Segmentation

Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.

Credit: Joseph Djugash

Normalized-Cut Measure

Slides from Eitan Sharon, “Segmentation and Boundary Detection Using Multiscale Intensity Measurements”.Credit: Joseph Djugash

Top-Down approachInput Fragments

Matching CoverCredit: Joseph Djugash

Another step towards the middle

Bottom-Up

Top-Down

Credit: Joseph Djugash

Some Definitions & Constraints

• Measure of saliency h(Γi), hi є [0,1)

• A configuration vector s contains labels si (1/-1) of all the segments (Si) in the tree

• The label si can be different from its parent’s label s i

–

• Cost function for a given s

Top-down term Bottom-up termDefines the weighted edge between Si & Si

–

Classification Costs

• The terminal segments of the tree determine the final classification

• The top-down term is defined as:

• The saliency of a segment should restrict its label (based on its parent’s label)

• The bottom-up term is defined as:

Confidence Map

• Evaluating the confidence of a region:

• Causes of Uncertainty of Classification– Bottom-up uncertainty – regions where there is no

salient bottom-up segment matching the top-down classification

– Top-down uncertainty – regions where the top-down classification is ambiguous (highly variable shape regions)

• The type of uncertainty and the confidence values can be used to select appropriate additional processing to improve segmentation

Results

• Calculate average distance between a given segmentation contour and a benchmark contour.

• Removing from the average all contour points having a confidence measure less than 0.1.

• The resulting confidence map efficiently separated regions of high and low consistency.

• The combined scheme improved the top-down contour by over 67% on average.

• This improvement was even larger in object parts with highly variable shape.

31

Results (cont.)

•top-down process may produce a figure-ground approximation that does not follow the image discontinuities.•Salient bottom-up segments can correct these errors and delineate precise region boundaries

Buttom up

The initial classificationmap T(x, y)

Results III (cont.)

Results III (cont.)

the top-down completely misses a part of the object . The confidence map may be helpful in identifying such cases,

Results III (cont.)

bottom-up segmentation may be insufficient in detecting the figure-ground contour, and the top-down process completes the missing information

Results III (cont.)

Results III (cont.)

Salient bottom-up segments can correct these errors and delineateprecise region boundaries

Conclusion

• Buttom-up and top-down merits• Provide reliable confidence map• It take into account all discontinuities at all

scales

But:• If the object is assigned a given category, the

specific features cannot be adopted for other categories

38

Constrained Parametric Min-Cuts for Automatic Object

SegmentationJoao Carreira

Cristian Sminchisescu

39

Traditional Segmentation: Finding Homogeneous Regions

40

gPb-owt-ucm: P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik. PAMI 2010.

Conventional Bottom-up Segmentation

Proposed approach

1. Split multiple times

2. Retain object-like segmentations

Bottom-up Object Segmentation

Credit: J. Carreira

High redundancy

Bottom-up Object Segmentation

42

Credit: J. Carreira

A single multi-region segmentation or a hierarchy

Proposed Bottom-up Object Segmentation

43

Credit: J. Carreira

single-shot multi-region segmentation

robust set of overlapping figure-ground segmentations

Segments with object-like regularitiessuperpixels

44

Constrained Parametric Min-Cuts for Automatic Object Segmentation

Credit: J. Carreira

parametric max-flow solver

Figure ground segmentation by growing regions around seeds

Ranking

45

Constrained Parametric Min-Cuts for Automatic Object Segmentation

Credit: J. Carreira

Initialization

• Foreground– Regular 5x5 grid geometry– Centroids of large N-Cuts regions– Centroids of superpixels closest to grid positions

• Background– Full image boundary– Horizontal boundaries– Vertical boundaries– All boundaries excluding the bottom one

Performance broadly invariant to different initializations

Generating a segment pool:constrained min-cut

min cuthard constraint

background

object

hard constraint

47

Credit: J. Carreira

Generating a Segment Pool:Constrained Parametric Min-Cuts

background

object

hard constraint

48

Credit: J. Carreira

background

object

hard constraint

49

Generating a Segment Pool:Constrained Parametric Min-Cuts

Credit: J. Carreira

background

object

hard constraint

50

Generating a Segment Pool:Constrained Parametric Min-Cuts

Credit: J. Carreira

background

object

hard constraint

background

object

hard constraint

Can solve for all values of object bias in the same time complexity of solving a single min-cut using a parametric max-flow solver

background

object

hard constraint

51

Generating a Segment Pool:Constrained Parametric Min-Cuts

Credit: J. Carreira

Fast Rejection

Large set of initial segmentations (~5500)

High Energy Low Energy

~2000 segments with the lowest energy

Cluster segments based on spatial overlap (at least 0.95)

Lowest energy member of each cluster (~154 in PASCAL VOC)

Credit:SasiKanth BendapudiYogeshwar Nagaraj

53

Constrained Parametric Min-Cuts for Automatic Object Segmentation

Credit: J. Carreira

•ranks all the sampled object segmentations•discard all but a small subset of confident ones.

Ranking object hypotheses

mid-level, category independent features Boundary – normalized boundary energy Region – location, perimeter, area, Euler

number, orientation, contrast with background Gestalt – convexity, smoothness

GoodLow boundary energy

Non smooth.

High Euler number

High boundary energy

Smooth.

Euler number = 0

Bad

54

Credit: J. Carreira

Segment Ranking

• Model data using a host of features– Graph partition properties– Region properties– Gestalt properties

• Apply Features Normalization• Train regressor with the largest overlap

ground-truth segment using Random Forests

• Diversify similar rankings using Maximal Marginal Relevance (MMR)

Graph Partition Properties

• Cut – Sum of affinities along segment boundary

• Ratio Cut – Sum along boundary divided by the number

• Normalized Cut – Sum of cut and affinity in foreground and background

• Unbalanced N-cut – N-cut divided by foreground affinity

• Thresholded boundary fraction of a cut

Region Properties

• Area• Perimeter• Relative Centroid• Bounding Box

properties• Fitting Ellipse

properties• Eccentricity• Orientation

• Convex Area• Euler Number• Diameter of Circle

with the same area of the segment

• Percentage of bounding box covered

• Absolute distance to the center of the image

Gestalt Properties

• Inter-region texton similarity

• Intra-region texton similarity

• Inter-region brightness similarity

• Intra-region brightness similarity

• Inter-region contour energy

• Intra-region contour energy

• Curvilinear continuity

• Convexity – Ratio of foreground area to convex hull area

Feature Importance for the Random Forest regressor

Feature Importance

How to Model Segment Quality ?

Best overlap with a ground truth object computed by intersection-over-union.

64

Credit: J. Carreira

Diversifying the Ranking

Diversified

Original

Best two hypotheses

Middle two hypotheses

Worst two hypotheses

Segment Ranking using Maximum Marginal Relevance

66

Performance

Credit:SasiKanth BendapudiYogeshwar Nagaraj

Ranking

79Credit: J. Carreira

Running Demos

• Methodologies employed– Kmeans using:

• Texture• RGB• Texture + RGB• RGB + HSV• Texture + Lab + HSV

80

Running Demos

• Data set used– Microsoft Research Cambridge Object Recognition

Image Database, version 1.0.– Used: 7 classes with 23 per class

• Animal-grass

• Trees-sky-grass

• Buildings-sky-grass

• Airplanes-sky-grass

• Animal-grass• Faces-BG

• Car-wall-ground81

Experiment ResultsFeatures Texture Texture +

RGBRGB RGB +HSV Texture+Lab+

HSV

Animal-grass 72.7% 74.1% 72.3% 72.6% 74.1%

Trees-sky-grass

37.1% 37.1% 40.7% 38.2% 37.1%

Buildings-sky-grass

44.6% 42.8% 51.9% 45.4% 44.7%

Airplanes-sky-grass

58.8% 58.8% 54.6% 59.7% 58.7%

Animal-grass 64.8% 64.8% 69.3% 71% 64.9%

Faces-BG 100% 100% 100% 100% 100%

Car-wall-ground 67.2% 67.2% 68.4% 64.9% 67.2%

Mean 63.6% 63.5% 65.3% 64.6% 63.8% 82

Experiment ResultsFeatures Textur

eTextur

e + RGB

RGB RGB +HSV Texture+Lab+ HSV

One iteration Elapsed time is

7.42 secs

12.26 secs.

1.62 secs

1.5 secs 7.84 sec

Overall Elabsed time for experiment

19.9 mins

32.9 mins

4.4 mins 4 min 21 mins

83

Microsoft Research Cambridge Object Recognition Image Database, version 1.0.

84

85

86

87

Acknowledgment

• Dr. Devi Parikh

• Dr. Joao Carreira

88