Ursina Caluori, ucaluori@student.ethz.ch 2.2.2006

Interactive Foreground Extraction using Iterated Graph Cuts

Carsten Rother - Vladimir Kolmogorov - Andrew Blake

Siggraph '04

Outline

Introduction & MotivationPrevious Approaches• Magic Wand• Intelligent Scissors• Graph CutThe Algorithm• Novelties• Behind the ScenesResultsConclusions

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 2

MotivationForeground Extraction: Say what?

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 3

We have We want

MotivationThe GrabCut Approach

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 4

User input Output

GrabCut

IntroductionOverview

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 5

Iterative Minimization

1. User marks object 2. Intermediate Result

3. User adjusts selection 4. Final Result

"Grab" "Cut"

The Goals...• interactive foreground extraction• high performance & quality - minimal user input• usability in non-trivial images

...can only be achieved with• good user interface ( rectangle / lasso)• accurate segmentation ( iterative graph cuts)• convincing alpha values ( border matting)

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 6

IntroductionAspirations

Previous ApproachesMagic Wand

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 7

Input: Seed Pixel(s) + Tolerance Output: Pixels within tolerance of seed pixels' colors statistics

Thresholding

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 8

Input: Points on segmentation boundary

Output: Pixels within minimum cost contour

Energy Minimization&

Interpolation

Previous ApproachesIntelligent Scissors

Previous Approaches

Graph Cut• foundation of GrabCut pay attention • uses boundary and region information• segmentation: min-cut by energy minimization

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 9

Input: Clue-mark inside and outside region

Output: Pixels within "best" inside region

Min-Cut

Graph Cut

Previous Approaches

A little bit more detailed1. Represent image as graph (weights ~ 1/energy)2. Energy minimization3. Find minimum cost path & cut it

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 10

Graph Cut

Source (Foreground)

Sink (Background)

∑= )( weightscutCost

The Algorithm

GrabCut: extension of Graph Cut• iterative optimization• incomplete labelling• simplified user interaction• border matting

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 11

Novelties

Behind the Scenes

Given

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 12

Model

GMM array

GMM componentsinitial alpha matte

initial trimapcolor array

N pixels),...,( 1 Nzzz =

K}{1,...,),,...,,...,( n1 ∈= kkkkk Nn

},,{ FUB TTTT =

}1,0{),,...,( 1 ∈= αααα N

Kff ,...,1 Kbb ,...,1

fromstart

afterinit

Behind the Scenes

Wanted

Energy functionU: fit of to given GMM (color/region information) V: smoothness term (boundary/edge information)

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 13

Model

final alpha matte

GMM parameters KkkkkΘ ,...,1;1,0)},,(),,(),,({ ==Σ= αααµαπ

]1,0[),,...,( 1 ∈= αααα N

αVUE +=

z

Behind the Scenes

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 14

Outline

• Trimap• Alpha matte• GMMs

Initialization

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization

until convergence

User: fg / bg brushGrabCut : • update trimap• step 3+4 once

User editing

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 15

Trimap & Alpha Matte

• Trimap• Alpha matte• GMMs

Initialization

Trimap(background, unknown and foreground region)

},,{ FUB TTTT =

?BT

UT

Behind the Scenes

Alpha Matte(alpha value per pixel)

α

U

Bn Tn

Tnif α

∈∈

=10

1. user indicates background region(hard constraints!)

2.

BT

BUF TTT == ,ø

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 16

Trimap & Alpha Matte

• Trimap• Alpha matte• GMMs

Initialization

Trimap(background, unknown and foreground region)

},,{ FUB TTTT =

Behind the Scenes

Alpha Matte(alpha value per pixel)

α

U

Bn Tn

Tnif α

∈∈

=10

1. user indicates background region(hard constraints!)

2.

BT

BUF TTT == ,ø

Introduction | Previous Approaches | The Algorithm | Results | Conclusions

Gaussian Mixture Models

• Trimap• Alpha matte• GMMs

Initialization

Behind the Scenes

Gaussian Mixture Model (GMM)• approximation of color distribution• weighted sum of K gaussians• parameters: weights , means , covariances• in color space

µ Σπ1D

1=α1. fit bg GMM to colors of pixels with 2. fit fg GMM to colors of pixels with

0=α

In GrabCut• K = 5• 1 foreground GMM, 1 background GMM

17

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 18

Energy function revisitedBehind the Scenes

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization Energy function

U: fit of to given GMM (region information) V: smoothness term (boundary/edge information)

α z

00 1

GMM

U

),(),,,(),,,( zVzΘkUzΘkE ααα +=

∑∈

−−≠=

Cnm

zz

mn

nm

eV),( differencecolor low penalize

)2

||||( 2

2

][ σααγ

)log(GMMU −=

the better GMMs, the lower U neighboring pixels at segmentation boundary with similar color get punished

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 19

ComputationBehind the Scenes

Iterative Minimizationwhile (!converged) {

1. Fill : for each pixel, find best GMM component2. Find best GMM parameters3. Perform Graph Cut

1. Minimize energy

2. Cut adjust }

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization

k

α),,,(minmin

}:{zΘkE

kTn Un

αα ∈

),( kΘ α

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 20

DemoBehind the Scenes

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization

1 2 3 4

Result Energy

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 21

DemoBehind the Scenes

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization

1 2 3 4

Result Energy

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 22

DemoBehind the Scenes

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization

1 2 3 4

Result Energy

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 23

DemoBehind the Scenes

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization

1 2 3 4

Result Energy

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 24

DemoBehind the Scenes

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization

1 2 3 4

Result Energy

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 25

Border Matting

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization

Behind the Scenesα

1

bg fg0

-w w

hard segmentation soft segmentation

Border Matting

α1

0bg fg-w w

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 26

Border Matting

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization

:pixel UT∈∀

∆ σ

α1

bg fg0

∆

σ

-w w

Behind the Scenes

Border Matting1. obtain C from hard segmentation, define w (here w=4)2. assign t(n)3. go along C, find best transition for every t

(i.e. find and by energy minimization)

Avoid color bleeding1. estimate foreground color2. get closest "matching" color from foreground neighborhood

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 27

Border Matting

1. Assign GMMs to pixels2. Adapt GMM parameters3. Estimate Segmentation4. Apply border matting

Iterative Minimization

:pixel UT∈∀

∆ σ

Behind the Scenes

Border Matting1. obtain C from hard segmentation, define w (here w=4)2. assign t(n)3. go along C, find best transition for every t

(i.e. find and by energy minimization)

Avoid color bleeding1. estimate foreground color2. get closest "matching" color from foreground neighborhood

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 28

User EditingBehind the Scenes

User (fore- / background brush)adjust hard constraints GrabCut• update trimap• estimate new segmentation (& apply border matting)

User: fg / bg brushGrabCut : • update trimap• step 3+4 once

User editing

Results

GrabCut• Target rectangle: 450 x 300 pixels• 2.5 GHz CPU, 512 MB RAM• Initial segmentation: 0.9 sec • after each brush stroke: 0.12 sec

GrabCut vs. Graph Cut• quality: perform comparably• time: Graph Cut probably faster• GrabCut fewer user interactions

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 29

Performance

Results

Problems• camouflage & low contrast• thin structures• inadequate bg representation• several objects

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 30

Problems

Results

Problems• camouflage & low contrast• thin structures• inadequate bg representation• several objects

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 31

Problems

Results

Problems• camouflage & low contrast• thin structures• inadequate bg representation• several objects

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 32

Problems

Results

Problems• camouflage & low contrast• thin structures• inadequate bg representation• several objects

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 33

Problems

Conclusions

The Algorithm• fast• good results in many cases• nice ideas

The Paper• claims proven (Gimp-plugin by Matthew Marsh, MS Expression)• rather minimalist explanations

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 34

My 2 Cents

Future Work• 3D support• video support • combination of GrabCut with

image completion

35

Future WorkConclusions

GCGC

IC

Introduction | Previous Approaches | The Algorithm | Results | Conclusions

Discussion

Introduction | Previous Approaches | The Algorithm | Results | Conclusions 36

Questions? Ideas?