Discriminative Segment Annotation in Weakly Labeled Video

Kevin Tang, Rahul Sukthankar

Appeared in CVPR 2013 (Oral)

Research Problem• Input: a weakly labeled video (eg., “dog”)• Output: identify segments that correspond to the label to generate the semantic

segmentation, i.e., classify each segment either as coming from concept “dog” (called concept segments), or not (called background segments).

• Pipeline– Perform unsupervised spatiotemporal segmentation.– Propose an algorithm to identify the meaningful segment.

Contributions

• Present a interpretation framework to analyze a broad class of existing weakly supervised learning algorithms about segment annotation problem.

• Propose a discriminative algorithm CRANE (Concept Ranking According to Negative Exemplars) for segment annotation.

Interpretation framework

• Pairwise distance matrix between segments

Segment: spatiotemporal volume (3D), represented as a point in feature space(such as RGB histogram, local binary pattern histogram, or dense optical histogram).

• Positive segment Concept segment Background segment

• Negative segment

Goal: classify the from in .

Rank the elements in in decreasing order of a score, such that top-rankedElements correspond to .

Interpretation framework• Baseline algorithms about segment annotation.– Kernel density estimation for Negative segments.

• Intuition: the distribution of is similar to distribution of .• Construct a probability density operated on block C.• Rank the elements according to .

– Negative Mining (MIN)• Intuition: distance from to the nearest > distance from to

nearest . • Operated on block D.

CRANE• Each negative segment in penalizes nearby segments in .• Segments in should be those far from negatives.

Penalty function

CRANE

• Advantages

1. Robust to noise.2. Parallelizable.

Experimental Results

YouTube Objects datasets

Experimental Results

• (a): Sucesses (b): Failures