From Contours to Regions: An Empirical Evaluation

presented by: Yingjie Tang

Outline:• Motivation • Contours VS Regions • Contour Detection • Oriented Watershed Transform • Ultrametric Contour map • Evaluation:

Benchmark & Results • Conclusions & Contributions

Motivation of Segmentations: 1. Provide natural domains for computing features used in

recognition 2. Visual tasks can benefit from the complexity reduction

Contours VS Segmentation:

Contours: Not closed Segments: Closed and can provide partition

gPb Detector: Oriented Gradient of Histograms

gPb Detector: Multiscale/ Multi- channel Pb

gPb Detector: weighted combination of multi scale local cues

s: indexes scales i: indexes feature channels(brightness, color a, color b,

texture) measures the histogram difference in channel i between two halves of disc centered at (x,y) at angle θ.

gPb Detector: Globalization

gPb Detector: Globalization

gPb Detector: Globalization

Benefit of Globalization

gPb Detector: Globalization

Benefit of Globalization

Segmentation:

Contour Detector

Oriented Watershed

Transformation

Ultrametric Contour

Map

Hierarchical Segmentation▪ From contours to segmentation ▪ Watershed Transform ▫ Concept

13—Hsin-Min Cheng

Hierarchical Segmentation▪ From contours to segmentation ▪ Watershed Transform ▫ Example

14—Hsin-Min Cheng

Hierarchical Segmentation▪ From contours to segmentation ▪ Watershed Transform

15

Boundary strength ArtifactsWeight each arc

—Hsin-Min Cheng

Hierarchical Segmentation▪ From contours to segmentation ▪ Oriented Watershed Transform

16

WTOWT

—Hsin-Min Cheng

Segmentation:Watershed Transform.

Image Boundary Strength E(x,y)

Partition image Reweight

Disadvantages: Since the contour detector produces a spatially extended response around strong boundaries, simply weighting each arc by mean value of E(x,y) can introduce artifacts.

Segmentation:Watershed Transform

ImproveOriented Watershed Transform

Input boundary signal E(x, y) = maxθ E(x, y, θ).

Watershed arcs computed from E(x,y).

Watershed arcs with an approximating straight line segment subdivision overlaid.

Oriented boundary strength E(x, y, θ) for four orientations θ.

Oriented boundary strength E(x, y, θ) for four orientations θ.

Segmentation:Contour Subdivision

Scale Invariant

Segmentation:Ultrametric Contour Map Original

image

Base level: Over-segmentation

Upper level: Under-segmentation

Moving between levels offers a continuous trade-off between these extremes

Segmentation:Ultrametric Contour Map

1. Construct graph G = (P0, K0, W (K0)) given by OWT 2. Iteratively merge regions by removing min weight boundary. 3. Produces region tree where:

• Root is entire image • Leaves are P0 • Height(R) is boundary threshold at which R first appears • Distance(R1, R2) = min{Height(R) : R1, R2 ⊆ R}

Segmentation:Ultrametric Contour Map

Merging Algorithm Description:

Evaluation: On Berkeley Segmentation Dataset(BSDS)

• Precision-Recall on Boundaries • Variation of Information • Rand Index • Segmentation Covering

https://www.eecs.berkeley.edu/Research/Projects/CS/vision/bsds/BSDS300/html/dataset/humans/1105-0171-0180.html

Evaluation: Precision-Recall on Boundaries

A B

C D

prediction outcome

actual valueP F

P’

F’

R(recall) = A/(A+C)P(precision) = A/(A+B)

F-measure = 2*PR/(P+R)

• Optimal Dataset Scale(ODS) • Optimal Image Scale(OIS) • Average Precision(AP)

Evaluation: Precision-Recall on Boundaries

Evaluating boundaries on the Berkeley dataset

Contours Hierarchical Regions Segmentations

The proposed approach outperformed the state of art

Evaluation: Precision-Recall on Boundaries

Evaluating boundaries on the Berkeley dataset

Boundary benchmarks on the BSDS

Evaluation: Variation of Information & Rand Index & Segmentation Covering

Rand Index between test and ground- truth segmentations S and G is given by the sum of the number of pairs of pixels that have the same label in S and G and those that have different labels in both segmentations, divided by the total number of pairs of pixels.

V I(C, C′) = H(C) + H(C′) − 2I(C, C′)Measures the distance between two segmentations in terms of their average conditional entropy

Evaluation: Variation of Information & Rand Index & Segmentation Covering

Region benchmarks on the BSDS

Evaluating regions on the BSDS

Conclusions & Contributions:

• Algorithm gPb-owt-ucm, offers the best performance on every dataset and for every benchmark criteria tested

• This algorithm is straight- forward, fast, has no parameters to tune, and supports interactive user refinement

• The generic grouping machinery has found use in optical flow and object recognition applications.

Q & A

Constrained Parametric Min-Cuts for Automatic Object

Segmentation

Outline • Constrained Parametric Min-Cuts(CPMC) • The Grid Geometry • Fast Rejection • Ranking Segments • Experiments

Constrained Parametric Min-Cuts(CPMC)

Min-Cuts:a minimum cut of a graph is a cut (a partition of the vertices of a graph into two disjoint subsets that are joined by at least one edge) whose cut set has the smallest number of edges (unweighted case) or smallest sum of weights possible. —Wikipedia

Maximum flow: maximum flow problems involve finding a feasible flow through a single-source, single-sink flow network that is maximum. —Wikipedia

Constrained Parametric Min-Cuts(CPMC)

object segmentation framework

Constrained Parametric Min-Cuts(CPMC)

Energy function in pixels:

Fast RejectionLarge set of initial segmentations (~5500)

High Energy Low Energy

~2000 segments with the lowest energy

Cluster segments based on spatial overlap (at least 0.95)

Lowest energy member of each cluster (~154 in PASCAL VOC)Credit:

SasiKanth Bendapudi Yogeshwar Nagaraj

Ranking Segments:

3 sets of features: • Graph partition properties (8 features) • Region properties (18 features) • Gestalt properties (8 features)

Experiments:

Three Datasets: • Weizmann’s Segmentation Evaluation Database

(with only one prominent foreground object in each and test F-measure criteria)

• MSRC dataset(nearly 11 objects in each image and is used to evaluate the quality of the pool of segments

generated) • VOC 2009 dataset(most complicated and used

segmentation covering as an accuracy measure)

Experiments: Segment Pool Quality

Average of best segment F-measure scores over the entire dataset.

Average best image covering scores on MSRC and VOC2009 train+validation datasets

Experiments: Segment Pool Quality

Quality of the segments in VOC2009 joint train and validation sets for the segmentation problem

Evaluating the Ranking Object Hypothesis

Average best segment F-measure as the number of retained segments from the ranking is varied.

Evaluating the Ranking Object Hypothesis

Complementing the basic descriptor set with appearance and shape features improves the ranking slightly, but with a more expressive

regressor, random forests, the basic set is still superior.

Conclusion & Contributions

• Presented an algorithm that casts the automatic image segmentation problem as one of finding a set of plausible figure-ground object hypotheses.

• It is shown that the proposed framework is able to generate small sets of segments that represent the objects in an image more accurately than existing state of the art segmentation methods.

Q&A