some applications of the power watershed framework to ...bsdsagar/sravan-final-phd...2019/10/11 ·...

PhD Viva-Voce

Some Applications of the Power WatershedFramework to Image Segmentation and Image

Filtering

Sravan Danda∗

Supervisor: B.S.Daya Sagar∗Joint Supervisor: Laurent Najman†

∗SSIU, Indian Statistical Institute, Bangalore Centre†LIGM, Universite Paris Est, ESIEE, France

October 11, 2019

PhD Viva-Voce

Overview

1 Watershed Segmentation on Graphs

2 Power Watershed (PW) Framework

3 PW for Fast Isoperimetric Image Segmentation

4 Mutex Watershed : PW Limit of Multi-Cut Graph Partitioning

5 PW for Image Filtering

6 Perspectives

PhD Viva-VoceWatershed Segmentation on Graphs

A synthetic gray-scale image

27 29

29

26

28

28

27

29

30 27

26 28 29 28 26

73 69 68 70 68

72 70 71 71 73

67 74 71 72 70


Gradient Image2

2

3

1

2

1

1

2 2

1

3

1

0

3

2

3

2 1

2

47

1

41

1

39

2

42 42

4

1

1

1

2

3

2

1 5

2

5

1

4

0

0

2

1 3

7 3 1 2


Flooding Simulation1

0 1

1

0

0


Flooding Simulation

1 1

1

1 1

0 1

1 1

1

1

1 1

1 0

0 1



1 1

1

1 1

0 2 1

2 1 1

1

1

1

2

1

1 0

0

2

1

2



1 1

1

1 1

0 2 1

2 1 1

1

1

1

2

1

1 0

0

2

1

3 2



1 1

1

1 1

0 2 1

2 1 1

1

1

1

2

1

5

1 0

0

2

1

3 2


Watershed as a Graph-cut

Watershed Cut:Minimum Spanning Forest Cut w.r.t. Minima 1

1Watershed cuts: Minimum spanning forests and the drop of waterprinciple, J Cousty, G Bertrand, L Najman, M Couprie, IEEE Transactions onPattern Analysis and Machine Intelligence 31 (8), 1362-1374


Watershed as a Limit of Total Variation Minimizers

Power Watershed 2

limp→∞ x(p) where

x(p) = arg minx

(Q(p)(x))

Q(p)(x) =∑eij∈E

wpij |xi − xj |2 +

∑i∈Seed

wpi |xi − fi |2

2Power watershed: A unifying graph-based optimization framework, CCouprie, L Grady, L Najman, H Talbot, IEEE transactions on pattern analysisand machine intelligence 33 (7), 1384-1399


Power Watershed: Fast Watershed Cut

Power Watershed: Seeded Image Segmentation 3

3Power watershed: A unifying graph-based optimization framework, CCouprie, L Grady, L Najman, H Talbot, IEEE transactions on pattern analysisand machine intelligence 33 (7), 1384-1399

PhD Viva-VocePower Watershed (PW) Framework

Power Watershed Framework

Let 0 < λ1 < λ2 < · · · < λk

Q(x) =∑

iλi Qi (x)



Let 0 < λ1 < λ2 < · · · < λk

Q(p)(x) =∑

iλp

i Qi (x)



Let 0 < λ1 < λ2 < · · · < λk

Q(p)(x) =∑

iλp

i Qi (x)

x(p) = arg minx

Q(p)(x)



Let 0 < λ1 < λ2 < · · · < λk

Q(p)(x) =∑

iλp

i Qi (x)

x(p) = arg minx

Q(p)(x)

x(p) → x∗ (?)


Power Watershed - Generic Algorithm

Algorithm 1 Generic Algorithm to compute limit of minimizers 4

Input: Function Q(p)(x) =∑k

i=1 λpi Qi (x), where λk > λk−1 >

· · · > λ1 > 0.Output: x∗ :

1: Mk = arg min Qk(x) where x ∈ C2: for i from k − 1 to 1 do3: Compute Mi = arg min Qi (x) where x ∈ Mi+14: end for

4Extending the Power Watershed Framework Thanks to Γ-Convergence,LNajman, SIAM Journal on Imaging Sciences 10 (4), 2275-2292


Power Watershed Framework - Why?

Relates Watershed-Cuts with Random Walker and ShortestPath Segmentation

Results in a faster Watershed-Cut algorithm


Power Watershed Framework - Why?

Relates Watershed-Cuts with Random Walker and ShortestPath SegmentationResults in a faster Watershed-Cut algorithm


Laplacian Matrix

a

bc

d

e f

3

3

3

3

3

32

1 1

a b c d e fa 7 −3 0 −1 0 −3b −3 6 −3 0 0 0c 0 −3 7 −3 0 −1d −1 0 −3 9 −3 −2e 0 0 0 −3 6 −3f −3 0 −1 −2 −3 9


Fast Spectral Clustering using PW Framework

minimizeH∈Rn×m

Tr(HtLH)

subject to HtH = I(1)


Laplacian Matrix: Decomposition

a

bc

d

e f

3

3

3

3

3

3

a b c d e fa 6 −3 0 0 0 −3b −3 6 −3 0 0 0c 0 −3 6 −3 0 0d 0 0 −3 6 −3 0e 0 0 0 −3 6 −3f −3 0 0 0 −3 6



a

bc

d

e f

2

a b c d e fa 0 0 0 0 0 0b 0 0 0 0 0 0c 0 0 0 0 0 0d 0 0 0 2 0 −2e 0 0 0 0 0 0f 0 0 0 −2 0 2



a

bc

d

e f

1 1

a b c d e fa 1 0 0 −1 0 0b 0 0 0 0 0 0c 0 0 1 0 0 −1d −1 0 0 1 0 0e 0 0 0 0 0 0f 0 0 −1 0 0 1


Fast Spectral Clustering using PW Framework

minimizeH∈Rn×m

∑i

wpi Tr(HtLi H)

subject to HtH = I(2)


Scalability of Spectral Clustering Algorithms

Traditional Spectral Clustering: O(n 32 )

Power Spectral Clustering: O(nlogn)

where n are non-zero entries in L


PW Framework for other Image Segmentation Algorithms?

Can we obtain faster algorithms for other image segmentationmethods?

Yes!


PW Framework for other Image Segmentation Algorithms?

Can we obtain faster algorithms for other image segmentationmethods?Yes!

PhD Viva-VocePW for Fast Isoperimetric Image Segmentation

Image: Similarity Graph

2

1

1

3 3

2

2

3

2 1

13 10

wij = 100 exp(− ||i−j||σ )


Isoperimetric Partitioning

2

1

1

3 3

2

2

3

2 1

13 10



Isoperimetric Cost(A) = W (A,A)min{|A|, n − |A|}



xi ={

0 if vi ∈ A1 if vi ∈ A



xi ={


xtLx = xtDx− xtW x=

∑i ,j

xi dijxj −∑i ,j

xi wijxj

=∑i ,j

wijx2i −

∑i ,j

xi wijxj

=∑i ,j

wij(x2i − 2xi xj + x2

j )

=∑i ,j

wij(xi − xj)2



xi ={


xtLx = W (A,A)



xi ={


|A| = xt1

n − |A| = (1− x)t1



Discrete Formulation:

minimizex

xtLxmin{xt1, (1− x)t1}

subject to xi ∈ {0, 1} ∀i(3)



Continuous Relaxation:

minimizex

xtLxmin{xt1, (1− x)t1}

subject to xi ∈ [0, 1] ∀i(3)

Select best threshold!



Seed Constraint xj = 0

minimizex

xt−jL−jx−j

min{xt−j1, (1− x−j)t1}

subject to xi ∈ [0, 1] for i 6= j(3)

Select best threshold!



Lagrange Multipliers 5

L−jx−j = 1 (3)

5Isoperimetric graph partitioning for image segmentation, L Grady, ELSchwartz, IEEE Transactions on Pattern Analysis and Machine Intelligence,469-475


Fast Isoperimetric Partitioning

Solve 6

LMaxST−j x−j = 1 (4)

6Fast, quality, segmentation of large volumes - isoperimetric distance trees,L Grady, European Conference on Computer Vision, 449-462


Fast Isoperimetric Partitioning

2

3 3

2 3

2

13 10


Fast Isoperimetric Partitioning: Computational Cost

O(n) : where n are non-zero entries in L


Why does the MST heuristic work?

Power Watershed Framework ⇒ Enough to solve on UMaxST! 7

7Revisiting the Isoperimetric Graph Partitioning Problem, S Danda, AChalla, BD Sagar, L Najman, available athttps://hal.archives-ouvertes.fr/hal-01810249


Fast Isoperimetric Partitioning Using PW

2

3 3

2

2

3

2

13 10


Are Solutions on MST and UMST the same?

a c

e

db

g

1

1

2

1

3

3

0.9

Original Graph



a c

e

db

g

1

1

2

1

3

3

UMST Graph



a c

e

db

g

1

2

1

3

3

a c

e

db

g

1

1

2

3

3

a c

e

db

g

1

2

1

3

3

All Possible MSTs


Solving Linear System on Graph, UMST and an arbitraryMST are different!

Node Original UMST MST1 MST2 MST3g 0.00 0.00 0.00 0.00 0.00a 1.69 2.68 5.00 4.00 11.16b 1.54 2.32 9.66 1.00 5.00c 2.04 3.52 7.00 5.50 10.66d 2.09 3.64 8.66 6.50 9.00e 1.94 3.74 8.00 6.16 10.00


How different are Solutions on MST and UMST?

LemmaLet Tumst and Tmst denote the operators on UMST and MSTrespectively, as defined above. Then there exists two positiveconstants K1 and K2 such that

K1

k∑i=1

(ui −mi )2w2i ≤ ‖Tumst − Tmst‖ ≤ K2

k∑i=1

(ui −mi )2w2i . (5)


Results in Practice

F-Score

Comparison of MaxST and UMaxST as a sufficient statistic!


Results in Practice

Adjusted Rand Index



Results in Practice

Scatter plot of Normalized values of solutions

Strictly increasing plot implies perfectly consistent solutions!


Results in Practice

Inversions



Results in PracticeData Reduction

0 20 40 60 80 100Reduction (%)

0

20

40

60

80

100

120

Coun

t (To

tal o

f 500

)


Contributions

1 Detailed Analysis of the relaxed Cheeger Cut problem2

3

4


Contributions

1 Detailed Analysis of the relaxed Cheeger Cut problem2 Establish using PW framework that considering UMST graph

acts as a sufficient statistic3

4


Contributions


acts as a sufficient statistic3 Establish bounds between UMST and MST based

implementations4


Contributions


acts as a sufficient statistic3 Establish bounds between UMST and MST based

implementations4 Empirically establish that UMST based reduction is robust

compared to MST based implementation

PhD Viva-VoceMutex Watershed : PW Limit of Multi-Cut Graph Partitioning

Mutex Watershed: 8 The Setup

G = (V ,E ,W )

8Steffen Wolf, Constantin Pape, Alberto Bailoni, Nasim Rahaman, AnnaKreshuk, Ullrich Kothe, and Fred A. Hamprecht. The mutex watershed:Efficient, parameter-free image partitioning. In Vittorio Ferrari, Martial Hebert,Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 -15th European Conference, Munich, Germany, September 8-14, 2018,Proceedings, Part 4, volume 11208 of Lecture Notes in Computer Science,pages 571–587. Springer, 2018



G = (V ,E ,W )f : E → {−1,+1}




G = (V ,E ,W )f : E → {−1,+1}W : E → R+



Mutex Watershed: State-of-the-art on ISBI 2012


Mutex Watershed: Algorithm

Algorithm 2 Mutex Watershed

Initialize A = ∅.for each edge e in descending order of W (e) do

if A ∪ e does not violate the mutex condition thenA← A ∪ e

end ifend forreturn Subgraph induced by {e ∈ A|f (e) = +1}


Mutex Watershed: An Example

1 21

6

5

3-2

7

2

43

8

-4

54

9

2

10

9

2

11

9

1

12

-7

-3

13

5

8

14

-6

15

-7

-8

16

-5

9

17

2

5

18

8

4

19

5

20

7

2

21

3

9

22

1

6

23

9

2

24

8

25

2

1 3 4 7


Mutex Watershed: Walk-Through

1 2 3 4 5

10

9

6

11

9

7 8 9

12 139

14 15

16 17 189

23

9

19 20

21 22 24 25



1 2 3 4 5

10

9

6

11

9

7 8 98

12-8

139

18

8

14 15

16 179

23

9

19

24

8

20

21 22 25



1 2 3 4 5

10

9

6

11

9

7

12

-7

8 98

15

-7

-813

9

18

8

14

20

7

16 179

23

9

19

24

8

21 22 257



1 2 3 4 5

10

9

6

11

9

7

12

-7

8 98

14

-6

15

-7

-813

9

18

8

20

7

16 179

196

23

9

24

8

21 22 257



1

6

5

2 3 4 5

10

9

11

9

7

12

-7

8

13

5

98

14

-6

15

-7

-8

16

-5

9 5

18

8

20

7

179

196

23

9

24

8

21 22 257



1

6

5

2 3

8

-4

4 54

10

9

11

9

7

12

-7

13

5

98

14

-6

15

-7

-8

16

-5

9 5

18

8

4

20

7

179

196

23

9

24

8

21 22 257



1

6

5

2 3 43

8

-4

54

10

9

11

9

7

12

-7

9-3

13

5

8

14

-6

15

-7

-8

16

-5

9 5

18

8

4

20

7

21

3

179

196

23

9

24

8

223

257



1

6

5

2 3-2

7

2

43

8

-4

54

10

9

2

11

9

12

-7

9-3

13

5

8

14

-6

15

-7

-8

16

-5

9 5

18

8

4

20

7

172

21

3

919

6

23

9

24

8

223

257


Mutex Watershed: Segments

1

6

5

2

7

2

3 43

54

10

9

2

11

9

8

13

5

98

129

145

18

8

154

20

7

16 172

21

3

919

6

23

9

24

8

223

257


Multi-Cut Graph Partitioning

Q(a) = mina∈{0,1}|E |

−∑e∈E

aewe

s.t C1(A) = ∅ with A = {e ∈ E |ae = 1}(6)

NP-Hard!


Mutex Watershed: PW Limit of Multi-Cut

Q(p)(a) = mina∈{0,1}|E |

−∑e∈E

aewpe

s.t C1(A) = ∅ with A = {e ∈ E |ae = 1}(7)



Gk = (V ,Ek , W |Ek)

mina∈{0,1}|Ek |

−∑

e∈Ek

ae

s.t C1(A) = ∅ with A = {e ∈ Ek |ae = 1}(8)

denote the solution space by Ak .



Gk−1 = (V ,Ek−1, W |Ek−1)

mina∈{0,1}|Ek−1|

−∑

e∈Ek−1

ae

s.t C1(A) = ∅ with A = Ak ∪ {e ∈ Ek−1|ae = 1}(9)

denote the solution space by Ak−1.



Repeat until all edges are processed.



Sub-problems can be handled with a ‘Union-Find’ datastructure.

Sub-problems are tractable!



Sub-problems can be handled with a ‘Union-Find’ datastructure.Sub-problems are tractable!


Contributions

Mutex Watershed is PW limit of Multi-cut partitioning

PhD Viva-VocePW for Image Filtering

PW Framework for Image Filtering?

Can we relate image filtering tools?

Yes!


PW Framework for Image Filtering?

Can we relate image filtering tools?Yes!


Shortest Path Filter

SPF i =∑j∈V

gi (j)Ij ,



SPF i =∑j∈V

gi (j)Ij ,

where

gi (j) =exp

(−Θ(i ,j)

σ

)∑

k∈V exp(−Θ(i ,k)

σ

)



2

2

2

2

1

i

2

1

1 30

2

1

1

10

2

j

14

27

19

2

11

10

1

2

1

2

2

1

2 1

2 1 2

Θi (j) = 3


Tree Filter 9

TFi =∑

jti (j)Ij

9Linchao Bao, Yibing Song, Qingxiong Yang, Hao Yuan, and Gang Wang.Tree filtering: Efficient structure-preserving smoothing with a minimumspanning tree. IEEE TIP, 23(2): 555-569, 2014.


Tree Filter 9

TFi =∑

jti (j)Ij

where

ti (j) =exp(−D(i ,j)

σ )∑q exp(−D(i ,q)

σ )

9Linchao Bao, Yibing Song, Qingxiong Yang, Hao Yuan, and Gang Wang.Tree filtering: Efficient structure-preserving smoothing with a minimumspanning tree. IEEE TIP, 23(2): 555-569, 2014.


Tree Filter on a Synthetic Graph

1

2

2

3

1

2

2

2 2

2

2

1

3

3

2

1

2 2

2

11

3

11

2

10

1

10 11

2

1

2

3

2

1

2

2 2

1

1

1

2

2

100

2

2 2

3 100 101 3

Gradient image


Tree Filter on a Synthetic Graph

1

2

2 1

2

2

2

2

1

i

2

1

2

2 1

10

1

j2

1

2

2

1

1

1

2

2

100

2

2 2

ti (j) = 9


Tree Filter on a Synthetic Image

L to R: Noisy Image, TF, TF + BF


Can the Tree Filter be explained?

Power Watershed Framework ⇒ Tree Filter is an approximate limitof Shortest Path Filters 10

10Some Theoretical Links between Shortest Path Filters and MinimumSpanning Tree Filters, S Danda, A Challa, BD Sagar, L Najman, Journal ofMathematical Imaging and Vision, January 2019


Limit of Shortest Path Filters

j2

1

1

i

4 4

2

1

3

2 1

13 10

j4

1

1

i

16 16

4

1

9

4 1

169 100

L to R: Image Graph, Image Graph with edge weights raised topower 2


Limit of Shortest Path Filters: Characterization

LemmaLet G = (V ,E ,W ). For every pair of pixels i and j in V , thereexists p0 ≥ 1 such that, a path P(i , j) is a shortest path between iand j in G(p) for all p ≥ p0 if and only if P(i , j) is a smallest pathw.r.t. reverse dictionary order between i and j in G. Further, p0 isindependent of i and j.


Reverse Dictionary Order: Illustration

1

S

1

1

1

T

1

1 2

1

S

1

1

1

T

1

1 2


Limit of Shortest Path Filters: Characterization

LemmaEvery smallest path w.r.t. reverse dictionary order between any twoarbitrary nodes in G = (V ,E ,W ) lies on an MST of G and henceon the UMST of G.


Limit of Shortest Path Filters: UMST Filter

1 21

6

2

32

41

8

2

52

9

2

10

2

72

11

2

1

13

2

1

14

2

15

2

122 2

18

10

1

19

10

16 172

21

1

2 2

23

1

202

24

2

25

2

221

26

1

1

27

2

2

28

100

2

29

2

30

2

100


UMST Filter: Characterization

LemmaFor every pixel i in the image I, there exists a spanning tree Ti(termed as adaptive spanning tree), such that Ti contains asmallest path with respect to reverse dictionary ordering betweenpixels i and any other pixel j in I.


UMST Filter: Adaptive Spanning Trees

1

1

1

1 1

1

10

2

14 19

2 2

1

1

1 1

1

10

2 2

1

1

1

1

1

10

2 2


UMST Filter: Depth-Based Approximation

1 21

32

41

52

9

2

6 72

11

2

81

13

2

101

14

2

122 2

151

19

10

16

21

1

17 182 2

23

1

202

24

2

221

26

1

1

27

2

28

100

252

29

2

30

2


UMST Filter: Order-Based Approximation

1 21

32

41

52

9

2

6 72

11

2

81

13

2

101

14

2

122 2

151

19

10

16

21

1

17 182 2

23

1

202

24

2

221

26

1

1

27

2

28

100

252

29

2

30

2


Results in Practice

Salt and Pepper Noise

L to R: House Image, Bilateral Filter, Tree Filter, OurApproximation to Limit of SPFs


Results in Practice

Structural Similarity Indices

Mean SSIM on Salt Pepper NoiseBF TF UMSTF

House 0.69 0.80 0.83Barbara 0.72 0.66 0.72Lena 0.69 0.75 0.79Pepper 0.62 0.74 0.74Mean 0.68 0.74 0.77


Results in Practice

SSIM: Tree Filter vs UMST Filter 11

0.50 0.55 0.60 0.65 0.70 0.75 0.80 0.85 0.90Tree Filter

0.50

0.55

0.60

0.65

0.70

0.75

0.80

0.85

0.90Ou

r App

roximation

11Some theoretical links between shortest path filters and minimum spanningtree filters, S Danda, A Challa, BSD Sagar, L Najman, available athttps://hal.archives-ouvertes.fr/hal-01617799v5


Contributions

1 Establish UMST filter as a limit of shortest path filters2

3


Contributions

1 Establish UMST filter as a limit of shortest path filters2 Tree filter as an approximation to UMST filter3


Contributions

1 Establish UMST filter as a limit of shortest path filters2 Tree filter as an approximation to UMST filter3 Implement Depth-based and Order-based approximations of

UMST filter


Perspectives

1 Adaptive Spanning Trees can be processed in parallel!2


Perspectives

1 Adaptive Spanning Trees can be processed in parallel!2 Can we learn edge-aware features using the adaptive spanning

trees?

PhD Viva-VocePerspectives

Perspectives

1 Can we speed-up other tree-based algorithms such as scale-setanalysis?

2

3

4


Perspectives


2 Total Variation ↔ Cheeger Cut. Application to TVminimization!

3

4


Perspectives



3 Understanding working principle behind PW framework?4


Perspectives



3 Understanding working principle behind PW framework?4 PW implies UMST is a sufficient statistic for image

segmentation and filtering. Can we obtain sufficient statisticsfor graph-modelled data in general?


I would like to thank my advisors, Prof. B S Daya Sagar and Prof.Laurent Najman all their support, anonymous reviewers of myarticles and anonymous examiners and CCSD faculty for theirsuggestions, and Indian Statistical Institute for providing me

fellowship to pursue this research.

some applications of the power watershed framework to ...bsdsagar/sravan-final-phd...2019/10/11 ·...

Documents