Interactive Segmentation with Super-Labels

Andrew Delong

Western

Yuri BoykovOlga VekslerLena Gorelick Frank Schmidt

2

Natural Images: GMM or MRF?

MRFare pixels in this image i.i.d.? NO!

3

Natural Images: GMM or MRF?

GMM

4

Natural Images: GMM or MRF?

MRF

5

Natural Images: GMM or MRF?

MRF?

6

Boykov-Jolly / Grab-Cut

[Boykov & Jolly, ICCV 2001] [Rother, Kolmogorov, Blake, SIGGRAPH 2004]

7

Boykov-Jolly / Grab-Cut

[Boykov & Jolly, ICCV 2001] [Rother, Kolmogorov, Blake, SIGGRAPH 2004]

8

Boykov-Jolly / Grab-Cut

GMM

GMM

[Boykov & Jolly, ICCV 2001] [Rother, Kolmogorov, Blake, SIGGRAPH 2004]

9

• Objects within image can be as complex as image itself

• Where do we draw the line?

A Spectrum of Complexity

MRF?GMM?Gaussian? object recognition??

10

Single Model Per Class Label

GMM

GMM

11

Multiple Models Per Class Label

GMM

GMMGMM

GMMGMM

GMM

12

Multiple Models Per Class Label

MRFMRF

13

Our Energy ¼ Supervised Zhu & Yuille!

• Zhu & Yuille. PAMI’96; Tu & Zhu. PAMI’02• Unsupervised clustering of pixelsboundary

lengthMDL

regularizer+color

similarity+

14

Our Energy ¼ Supervised Zhu & Yuille!

• Zhu & Yuille. PAMI’96; Tu & Zhu. PAMI’02boundary

lengthMDL

regularizer+color

similarity+

15

Interactive Segmentation Example

16

Boykov-Jolly / Grab Cutsegmentation colour models

17

Ours

segmentation colour models“sub-labeling”

18

Main Idea

• Standard MRF:

• Two-level MRF:

object MRF

GMMs GMMs

background MRF

image-level MRF

object GMM background GMM

image-level MRF

unknown number of labels in each group!

19

The “Super-Pixel” View

• Complex object ¼ group of super-pixels• Interactive segmentation ¼

a“user-constrained super-pixel grouping”

20

The “Super-Pixel” View

• Why not just pre-compute super-pixels?– boundaries may contradict user constraints– user is helpful for making fine distinctions

• Combine automatic (unsupervised) and interactive (supervised) into single energy

Spatially coherent clustering+ MDL/complexity penalty

+ user constraints= 2-level MRF

Like Zabih & Kolmogorov, CVPR 2004

Label Costs, CVPR 2010

Like Boykov & Jolly, ICCV 2001

21

Process Overview

user constraintspropose models from current super-labeling1 solve 2-level MRF

via α-expansion2

refine all sub-models3

converged

E=503005E=452288

Boykov-Jolly + unsupervised clustering (random sampling) + iterated multi-label graph cuts (like grab-cut)

22

Our Problem Statement

• Input: set S of super-labels (e.g. ffg,bgg) constraints g : P ! S [ fanyg

fg

bg

any

23

Our Problem Statement

• Output: set L of sub-labels sub-labeling f : P ! L model params µ` for each `2L label grouping ¼ : L ! S

¼ ±ff`2

`1`3

GMM `1

white

GMM `2

dark gree

n

light green

24

Our Energy Functional

• Minimize single energy w.r.t. L, µ, f, ¼

E (L ;µ;¼;f ) =X

p2P

Dp(f p) +X

pq2N

wpqV(f p; f q) +X

`2L

h`±̀(f )

data costs smooth costs label costs

`4

`3 `1

`2

Dp( )̀ =½

¡ lnPr(I pjµ̀ ) if gp = any _ gp = ¼( )̀1 otherwise

forc

es tr

ansiti

on

25

Our Energy Functional

• Minimize single energy w.r.t. L, µ, f, ¼

E (L ;µ;¼;f ) =X

p2P

Dp(f p) +X

pq2N

wpqV(f p; f q) +X

`2L

h`±̀(f )

data costs smooth costs label costs

pay c2 `between group’

pay c1 `within group’V(¢;¢) 2 f0;c1;c2g

26

Our Energy Functional

• Minimize single energy w.r.t. L, µ, f, ¼

• Penalize number of GMMs used– prefer fewer, simpler models– MDL / information criterion

regularize “unsupervised” aspect

E (L ;µ;¼;f ) =X

p2P

Dp(f p) +X

pq2N

wpqV(f p; f q) +X

`2L

h`±̀(f )

data costs smooth costs label costs

GMMs GMMs

27

More Examples

Boykov-Jolly 2-level MRF

28

More Examples

Boykov-Jolly 2-level MRF

29

More Examples

Boykov-Jolly

2-level MRF

30

More Examples

Boykov-Jolly

grad studentsbaby panda

2-level MRF

GMM density for blue model

31

Interactive Co-segmentation

image collection 2-level MRFBoykov-Jolly

(like “iCoseg”, Batra et al., CVPR 2010)

32

More ExamplesBoykov-Jolly

2-level MRF

33

More ExamplesBoykov-Jolly

2-level MRF

34

Beyond GMMs

GMMs plane

GMMs only GMMs + planes

35

Synthetic Example

GMM

Boykov-Jolly(1 GMM each label)

GMM

GMMGMM

GMM

2-level MRF (GMMs only)

plane

plane

GMM

2-level MRF (GMM + planes)

• object = two planes in (x,y,grey) space• noise = one bi-modal GMM (black;white)

36

Synthetic Example

plane

plane

GMM

bla

ckw

hit

e

x

2 planes detected

1 GMM

detected

y

black

white

37

As Semi-Supervised Learning

• Interactive segmentation ¼ a semi-supervised learning– Duchenne , Audibert, Keriven, Ponce, Segonne.

Segmentation by Transduction. CVPR 2008.

– s-t min cut [Blum & Chawla, ICML’01]– random walker [Szummer & Jaakkola, NIPS’01]

38

Conclusions

• GMM not good enough for image ) GMM not good enough for complex objects

• Energy-based on 2-level MRF– data costs + smooth costs + label costs

• Algorithm: iterative random sampling, re-fitting, and ®-expansion.

• Semi-supervised learning of complex subspaces with ®-expansion