Segmentation from Examples

By: A’laa KryeemLecturer: Hagit Hel-Or

What is Segmentation from Examples? Segment an image based on

one (or more) correctly segmented image(s) assumed to be from the same domain

Effective when making a semantic segmentation

Why to use Examples The example defines the

granularity of the desired output

Give us the ability to characterize meaningful parts in the image

Using example allow us to use non-parametric model

The example defines the granularity of the desired output:

Training image

Test Image

Desired Segmentation

Induced Segmentation

Why to use Examples The example defines the

granularity of the desired output

Give us the ability to characterize meaningful parts in the image

Give us the ability to characterize meaningful parts in the image

Semantic Segmentation from an example

We want to segment an image into semantically meaningful parts

Required in various applications

Semantic Segmentation from an example

We want to segment an image into semantically meaningful parts

Required in various applications Problems:

Meaningful parts are often too complex

Semantic interpretation is highly subjective, depending on both the application, and the user

Meaningful parts are often too complex

Semantic Segmentation from an example

We want to segment an image into semantically meaningful parts

Required in various applications Problems:

Meaningful parts are often too complex

Semantic interpretation is highly subjective, depending on both the application, and the user

Example of image different segmentation

Semantic Segmentation from an example

So, How to achieve semantic segmentation Getting segmented training

image(s) as input

Training set

Semantic Segmentation from an example

So, How to achieve semantic segmentation Getting segmented training

image(s) as input Using non-parametric

representation

non-parametric model

Each semantic part is represented by a set of square patches

Semantic Segmentation from an example

So, How to achieve semantic segmentation Getting segmented training

image(s) as input Using non-parametric representation Over-segmenting the Test image

into small fragments

Over segmented image

Semantic Segmentation from an example

So, How to achieve semantic segmentation Getting segmented training

image(s) as input Using non-parametric representation Over-segmenting the Test image into

small fragments Compute costs for fragment-

label pairs

(fragment,label )cost example

?

Semantic Segmentation from an example

So, How to achieve semantic segmentation Getting segmented training

image(s) as input Using non-parametric representation Over-segmenting the Test image into

small fragments Compute costs for fragment-label

pairs Graph-cuts multi-label

optimization

Why do we need graph-cuts Graph-cuts optimization is used to label

each fragment in a globally optimal manner

Training set

Test image

Fragmentation

Fragments

Patch sets

Clas

sifica

tion

Classification scores

Graph-Cuts optimization

Result

Over segmenting Fragment: small arbitrarily-shaped

and simply-connected pixel clusters

We assume that small homogeneous regions always belong to the same semantic part

Over segmenting Fragment: small arbitrarily-shaped

and simply connected pixel clusters We assume that small

homogeneous regions always belong to the same semantic part

Advantages: Enforces a locally coherent labeling Reduces the computational complexity

Graph-cuts multi-label optimization

For each fragment we have k cost values, we need to determine which one is the optimal

Using expanded version of the graph-cuts we saw at Jad’s lecture, where we may have more than two labels (background , object)

Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization

Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization

Pixel labeling costs Given Itrain and Ltrain

Representing each label(segment) in Ltrain by a set of square patches

We get k sets {Sl} l=1,…,k one for each label

Pixel labeling costs (cont.) Next, we define φ (p, l) for each

(pixel,label) pair The cost of assigning label l to pixel p Itest

= minP’Sl

p:pixel at Itestl : label in Ltrain

ssd(P,P’) is the sum of squared distances between P,P’M:mxmx3P:mXm neighborhood centered at pP’:mxm patch

Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization

Fragmentation We partition Itest into small,color-

homogeneous regions using mean shift segmentation

Fragmentation We partition Itest into small,color-

homogeneous regions using mean shift segmentation

Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)

Fragmentation We partition Itest into small,color-

homogeneous regions using mean shift segmentation

Fragment size is adjusted according to Itest.(fragments are smaller in more detailed areas of Itest, and larger in more homogeneous regions)

Fragment boundaries align with edges in the image

Fragmentation (cont.)

Random colorization Detailed close-up

Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization

Fragment labeling costs

Voting scheme in order to compute labeling costs of each fragment

For each fragment fItest we pick a few representative pixels:

Rep(f)={pi f } i=I,…,Rf

Rf is proportional to |f|

Fragment labeling costs (cont.)

We talked about the cost of assigning a pixel into a label

Rep(f)}

Fragment labeling vs. pixel labeling

Fragment labeling reducing complexity We have n=Assume |Rep(f)|

= ,then we need to compute costs only for pixels

Fragment labeling vs. pixel labeling Enforces a locally coherent labeling

Training image Training seg. Input image

Fragment labelingPixel

labeling

Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization

Graph-cuts optimizationAfter calculating labeling cost for all image fragments we get k images. Image i describes the cost assigning each pixel at the test image to label i

fragment labeling costs. Costs range in the interval [0,1]

Graph-cuts optimization Now for each pixel pItest we have a labeling

cost We need to find Ltest the globally optimal

labeling Requirements:

Minimizes the total labeling cost Consistent with presence (or absence) of

edges

Graph-cuts optimization (cont.)

For each pair of neighboring pixels we define:

Ψ(p,q,L(p),L(q))=

L(p),L(q) : labels assigned to p,q : difference between pixels p,q (RGB euclidian distance)

Graph-cuts optimization (cont.)

In order to force pixels within each fragment to be labeled the same, and reduce complexity we specify the energy term E(L) (E(L) the value of a labeling scheme) in terms of fragments instead of pixels :

: cost of assigning fragment f to label L(f), weighted by the size of each fragment.: neighboring fragments in Itest.

= :controls trade-off between regions and boundaries

Intuition:

Big value:For pixels p,q even with different colors the graph-cuts step prefers to connect them to the same label to have and reduce the energy. Instead of 1- althout can be big too. This mean we prefer continues regions and not edges.

𝛼∑⟨𝑝 ,𝑞 ⟩

Ψ (𝑝 ,𝑞 ,𝐿 (𝑝 ) ,𝐿 (𝑞))

Intuition:

s value:For pixels p,q even with similar colors the graph-cuts step won’t care about connecting them to different labels because of small value. Even with big value at1-multiplying still give us small E value. Favors boundaries, holding out non-continues regions .

𝛼 ∑⟨ 𝑓 1, 𝑓 2⟩

Ψ( 𝑓 1 , 𝑓 2 ,𝐿 ( 𝑓 1 ) ,𝐿 ( 𝑓 2 ))

Graph-cuts optimization (cont.)

Finally Ltest is determined by solving Ltest=minL E(L)

Fragment labeling Labeling after Graph-Cuts Optimization

Multi-label graph-cut

Colored nodes:labels Squares : fragmentsFor each (fragment,label) pair we have an edge.Edge weigh according to φ.Edges between two squares weighed according to Ψ .

Multi-label graph-cut

Induced graphEach fragment connected to a single label .

Multi-label graph-cut is NP-complete problem! Using Isolation Heuristic we can get an

approximation of E(L)

1. For 1i k construct a minimum weight isolating cut Ei for label Li .

2. Determine h such that Eh has max weight.3. E=.4. Return E.

effect

Training image Training segmentation

Input image

=0.1 =5 =1

Algorithm for semantic segmentation1.Pixel labeling costs2.Fragmentation3.Fragment labeling costs4.Graph-cuts optimization

Algorithm results

Training set

a b c

Bear results

invariant to the number of instances of each semantic part within the image, and insensitive to the shape of each part.

We can’t separate multiple objects belonging to the same label (c).

Algorithm results

Training set

a b c d

Summary We saw that giving segmented example from

the same domain of an image can effectively perform a semantic segmentation

Using example also defines the granularity of the desired output

determining whether an entity belongs to a particular semantic part is more easily done at the fragment level, than on a pixel-by-pixel basis

Using graph-cuts with multi-label support can help making global optimization step for finding optimal labeling

Only one parameter needed , controlling the trade-off between regions and boundaries

Thank You For Listening

References

Inducing Semantic Segmentation from an Example,

Yaar Schnitman, Yaron Caspi, Daniel Cohen-Or, and Dani Lischinski. "Segmentation by Example“,

Sameer Agarwal and Serge Belongie. Christoudias, C.M., Georgescu, B.: Edge detection

and image segmentation (edison) system. Boykov, Y., Veksler, O., Zabih, R.: Fast

approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239