Multistage Image Clustering and Segmentation with Normalised Cuts

M.Y. Choong C.F. Liau J. Mountstephens M.S. Arifianto K.T.K Teo Modelling, Simulation and Computing Laboratory, School of Engineering and Information Technology,

Universiti Malaysia Sabah, Kota Kinabalu, Malaysia. msclab@ums.edu.my ktkteo@ieee.org

Abstract—Normalised cuts algorithm requires massive similarity measurement computation for image segmentation. Since a digital camera at present has the capability to produce high resolution image, it will be inevitably that resizing image into suitable resolution at which the algorithm can perform image segmentation with minimal burden. While retaining the important features in the images, natural images are likely to be restricted for resizing them into a particular smaller resolution. Dividing an image into equal size of regions (named as image cells) for the segmentation is proposed here to solve the problem of missing important features when the image resolution is overly reduced. Gradually, the locally segmented clusters from the image cells are taken for second stage segmentation to merge them up globally. In this paper, experimental results using the mentioned method are shown. Experiment shows that it is capable to produce reasonable segmented clusters based on the proposed approach.

Keywords-image segmentation; normalised cuts; clustering

I. INTRODUCTION Image segmentation has been widely applied in various

fields. Many of the computer vision applications require image segmentation for information extraction. An example of information extraction in medical computer vision is detecting abnormities in certain body organ. Often, the goal of image segmentation is to decompose an image into meaningful segments that correlates with the real world and these meaningful segments are used for further important analysis. Extensive research has been done to create and develop various kind of methods to produce segmentation results that near human perceptions. Local features based segmentation method such as Canny edge detector, that makes use of local information for particular sets of pixels. Edge detection method has certain drawbacks. It cannot guarantee in generating closed edge contours of segments [1]. Thus, global properties extraction based image segmentation should be performed to solve the limitation. A method using graph theoretic formulation is introduced here and it improves the segmentation method. Wu and Leahy pointed out that this approach guarantees the formation of closed edge contour [2]. The graph theoretic method is then combined with a clustering algorithm that constructs meaningful segments by clustering out the eigenvector generated from the Laplacian matrix. K-means clustering is one of the simple clustering algorithm can be used by providing the appropriate number of clusters for it to construct.

The details of the segmentation process based on the proposed approach in this paper are organized as follows. In section II, implementation of the image segmentation in normalized cuts and image resizing are described. Explanation of the segmentation based on proposed approach is presented in section III. In section IV, the experimental results and discussions are provided. Lastly, this paper is concluded in section V.

II. IMAGE SEGMENTATION AND IMAGE RESIZING It is not likely that an image to be segmented in

accordance to human perspective, when the segmentation method is based on primitive information, such as image brightness [3]. Thus, the aim of the segmentation should be focused on locating objects and boundary so that the image can be meaningful and easier to analyse.

A. Image Segmentation in Normalised Cuts In this paper, the image segmentation method used here

is based on graph theoretic approach which generally performs pixels grouping into regions and it is treated as graph partitioning problem.

A set of image pixels can be represented as weighted graph G = (V, E) where V represents the vertices which are image pixels (a vertex is made up one node) and E represents the edges in the form of weights, w. Each of the w gives a measurement of the similarity between node i and node j. A graph is bi-partitioned into two distinct sub-graphs A and B with the condition that it minimizes the value of

,

( , ) ( , )i A j B

cut A B w i j∈ ∈

= ∑ , (1)

with constraint A∪B = V and A∩B = Ø [4, 5]. The degree of the dissimilarity between two sub-graphs A and sub-graphs B, which is the sum of weights of the pairs of nodes are to be removed is shown in (1). The bi-partitioning process recursively finds the minimum cuts until a number of k sub-graphs are formed, with the condition that the maximum possible cut across the sub-graphs is minimized. In other words, the formed distinct sub-graphs have high similarity within the sub-graphs and low similarity across different sub-graphs [6]. However, the minimum cut criterion falls short that its cut algorithm favors in cutting isolated nodes to form sub-graphs. This lead to proposing another cut algorithm based on normalised cut (Ncut) criterion to

2012 Third International Conference on Intelligent Systems Modelling and Simulation

978-0-7695-4668-1/12 $26.00 © 2012 IEEE

DOI 10.1109/ISMS.2012.112

359

2012 Third International Conference on Intelligent Systems Modelling and Simulation

978-0-7695-4668-1/12 $26.00 © 2012 IEEE

DOI 10.1109/ISMS.2012.112

362

alleviate the problem [4, 5]. Fig. 1 illustrates Ncut gives a better cut that the ordinary cut.

The normalised cut criterion is derived as in

( , ) ( , )( , )( , ) ( , )

cut A B cut A BNcut A Bassoc A V assoc B V

= − , (2)

whereby assoc(A,V) = ∑i∈A,j∈V w(i, j). The Ncut(A,B) is then transformed to

( )D W y Dyλ− = , (3) for solving the eigenvectors y and eigenvalues λ. D − W is called the Laplacian matrix whereby W is a symmetrical matrix with W(i, j) = w(i, j). Each of the weight elements in matrix W is defined as

( ) ( ) ( ) ( )

( ) ( )2222 2

2,

0 , otherwise

XI

i ji j

e if i j rw(i, j ) e σσ

− −− − ⎧⎪ − − <= ∗⎨⎪⎩

X XF F

X X , (4)

where F(i) is feature vector based on intensity value in colour of node i and X(i) is the spatial location of the node. When a pair of node i and node j is more than r number of pixels apart the weight w(i, j) is considered 0. D is a diagonal matrix with d(i)=∑j=1 w(i, j) on its diagonal [4,5,6].

Image segmentation is then achieved by clustering on the eigenvector. Each of the element in the computed eigenvector contains description value corresponds to each of the pixel in terms of spatial and brightness (pixel intensity) measurement in the image. Ideally, eigenvector with the smallest nonzero eigenvalue is chosen to perform clustering on it. In this paper, eigenvector corresponds to the second smallest eigenvalue is chosen for a clustering algorithm to partition the eigenvector. After the clustering process on the eigenvector, segmentation result is finally obtained based on the clusters formed among the eigenvector. Fig. 2 illustrates the clustering process on the eigenvector that produces the segmentation result.

B. Image Resizing Image resizing is normally conducted before performing

Figure 1. The Ncut and the ordinary cut.

Figure 2. The clustering process and the eigenvector graph.

image segmentation. There is a significant issue that lower resolution images suffered information lost as fine details of the image turned out to be coarse. Thus, producing a good segmentation result of an image shall not be aimed when the image is resized to the extent that important features in the image are missed out.

For example in the application of locating vehicle license plate, image resizing affects the characters in a particular region which are regarded as particular pattern are coarsened [6]. This particular region will fail the pattern recognition miserably since characters are not recognisable. However, performing image segmentation on high resolution image is not feasible for online image transfer. Fig. 3 demonstrates how an image might appear at different resolution. Notice that the head of the bird is not visible in the image with resolution of 18×24. In the image with 90×120, it is then only understood that the bird is being kept in a cage.

III. IMAGE CELLS GENERATION FOR IMAGE SEGMENTATION Normalised cuts implementation in image segmentation

has a drawback in its computation. An image that has a size of m × n will require the normalized cuts algorithm to form the W matrix with a size of (m × n) × (m × n). For example, an image with a size of 120 × 90 will end up computing a

Original resolution: 240×320

18×24 30×40 90×120

Figure 3. Image at different resolution.

360363

10800 × 10800 adjacency matrix, which is considered a very large matrix for a computer to solve for eigenvalues.

A. Image Cells Generation and Segmentation in Image Cells To avoid processing such large matrix, it is suggested

that a high resolution image be divided into equal size of box-shaped image cells [5, 8]. Since natural scene image has irregular pixels distribution such that the frequency of image pixels for particular range occurred unevenly throughout the whole area of the image, a preliminary check is run on every image cell to determine whether the particular cell requires normalised cuts algorithm to perform on it. The first stage of segmentation is begun by providing the k1 number of clusters, normalized cuts algorithm is then performed to segment out k1 number of clusters for the particular cell. In this segmentation stage, over-segmentation would likely to be occurred as discrimination power in an image cell is reduced compared with segmentation on a whole image directly. Nevertheless, it helps to reduce the tendency of object boundaries to be missed out. The implementation of simple k-means clustering plays an important part on clustering out each of the image cells based on the given number of clusters.

Fig. 4 shows an image is divided into designated number of cells and local segmentation on each of the cells. The segmentation on each of the cells is done independently. In other words, the segmentation done in a cell is not related with the segmentation done in other cell. Segmented clusters from the image cells are then used for second stage segmentation.

B. Clusters Merging and Second Stage Segmentation To proceed to the second stage segmentation, simple

representations of the clusters are computed. Each segmented cluster has a node containing the median value of the cluster and centroid location of the cluster. The computed nodes are then used for second stage segmentation using normalised cuts algorithm. The value of threshold, r, which is the maximum distance for a pair of pixels can be apart isincreased because the computed nodes are scattered in sparse manner. In addition, the standard deviation for the spatiallocation of the node, σX has to be increased due to the matter also.

Figure 4. 3 × 3 image cells and its segmentation on the image cells

individually.

The number of nodes is corresponded with the total number of clusters segmented from the first stage segmentation. For example, 18 segmented clusters from the first stage segmentation produces 18 nodes. The computed nodes are then taken for similarity measurement using normalised cuts algorithm. Clusters will be merged together when their nodes are grouped together and share common similarity based on color and spatial location. Table I presents pseudocode of normalised cuts with the implementation of image cells division. Fig. 5 illustrates the grouping of nodes produces the final segmentation result (based on input image in Fig. 4).

IV. EXPERIMENTAL RESULTS AND DISCUSSIONS To illustrate image segmentation based on the proposed

approach, a simple image shown in Fig. 6 contains the flowers as foreground and the area surrounding the flowers as the background is selected. It can be seen that the flower petal can be segmented out from the background. The boundaries of the segmented cluster follow nicely with the edge of the flower petal in the image. The segmentation for the image whereby the image is divided into numerous image cells takes about 8.31 minutes which is considered long.

In the attempt to reduce computation time, generation of 8×8 image cells is performed for the segmentation. Eventually, the computation takes about 3.6 minutes which is less than the previous one about half of the time. This indicates increasing number of cells helps to reduce the computation time as the area of an image cell has become smaller despite more cells is added for first stage segmentation. Small image cell is not suitable for image with large objects in it. This leads to the segmentation on the image cells is performed unnecessary. Fig. 7 shows the segmentation result when the number of clusters per cell is increased.

TABLE I. PSEUDOCODE OF NORMALISED CUTS WITH IMAGE CELLS DIVISION

Input: Colour/ grayscale image 1 Read input image 2 Construct M×N = n image cells 3 Perform statistical analysis of the image cells 4 Determine whether ith image cell required

segmentation if > threshold value

proceed to local normalised cuts segmentation else

compute background node in ith image cell 5 First stage segmentation

Normalised cuts local segmentation into k1 clusters on ith image cell until nth image cell.

6 Nodes computationEach of the clusters segmented from every image cell

is represented by a node. 7 Second stage segmentation

Perform normalised cuts on the nodes to segment k2 clusters.

Output:Produce segmented clusters and display result.

361364

1

1

1

1

1

1

11 1

1

1

1

1

1

1

1

1

1

2

2

2

2

22

Node computation Segmentated clusters after second stage segmentation

Segmentation result:

Cluster 1 Cluster 2

Figure 5. Second stage segmentation and its result.

Input image (300 × 400) 1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

30.

31.

32.

33.

34.

35.

36.

2

3

4

56

7

8

9

10

1112

13

14

15

16

17

18

19

20

21

2223

24

25

26

27

28

29

30

31

323334

35

36

37

38

3940

41

42

Image cells division First stage segmentation Segmentation result:

Figure 6. 6 × 6 image cells and segmentation result.

Often image captured in real world contains objects that

are constructed with tiny lines. An image from the Berkeley Segmentation Dataset (BSDS500) [9] contains the steel roller coaster tracks is tested for the segmentation and its result is shown in Fig. 8. As it can be seen that for the 6×6 image cells segmentation, the part marked with red circle

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

30.

31.

32.

33.

34.

35.

36.

37.

38.

39.

40.

41.

42.

43.

44.

45.

46.

47.

48.

49.

50.

51.

52.

53.

54.

55.

56.

57.

58.

59.

60.

61.

62.

63.

64.

1

2

3

4

5

67 8

9 10

11

121314

1516

17

18

1920

21

22

2324

25 26

27

28

29

303132

33

34

3536

37

38

39 40

41

42

43 44

4546

47

48

4950

51 52

53

5455

56

57

58

59

60

61 62

Image cells division First stage segmentation Segmentation result:

Figure 7. 8 × 8 image cells and segmentation result

ring on the top left of the cluster 2 a has sharp right angle edge. This is due to normalised cuts algorithm is not run on the corresponded cell. However, for the 5×5 image cells segmentation, the sharp corner edge does not occur. The edges of the cluster follow closely with the object boundaries. When the number of image cells increased on the same image, the area covered by each of the image cells has a zoom in effect on it. This decreases the discrimination power for the area. In the first cell (top left) of 5×5 image cells, there is a critical curve is supposed to be segmented. However, the curve does not take up much of the area in the cell. The normalised cuts algorithm omits the cell that required segmentation in it and eventually ends up having the final segmented clusters appeared blockish. This happens to other image cells when this similar situation occurred.

Notice that the roller coaster track is not segmented out

Input image 120×160

1

234

56

78

9

10

11

12

13

14

1516

1718

1920

2122

23

24

2526

2728

2930

3132

3334

3536

37

38

39

404142

4344

45 46

Cluster 1 Cluster 2

5×5 image cells

1

23

45

678

910

11

1213

1415 16

1718

1920

21

22

2324

25

26

27

282930

31

323334

3536

37

383940

41

42

4344

4546

4748

49

5051

52

53

545556

5758

5960

61 62

Cluster 1

Cluster 2

6×6 image cells

Figure 8. Comparison between 5×5 and 6×6 image cells and their segmentation result.

362365

together with blue background. This shows the normalised cuts algorithm is able to tackle fine details of the objects in the image and also across the image cells individually.

Fine tuning the number of clusters per cell gives an indicator such that to what extent that the segmentation has to be accurately performed. Increasing the number of clusters per cell does not significantly increase the computation time if the number of cells to be divided for the image is constant. Fig. 9 shows the effect of changing the number of clusters per cell on the segmentation process while retaining the number of image cells in the image.

As the number of clusters per cell increased, more detailed contents of the cell are segmented out. For the segmentation done with 10 clusters per cell, the letter at the tail section of the aircraft is segmented out. However, when the number of clusters per cell is increased to the extent where some of the edges in the image such as the shading and shadow of the same object are segmented out separately. Thus, selecting an optimal number of clusters per cell should be based on the image complexity to reduce the tendency of the image being segmented undesirably. Otherwise, image cells can be selectively chosen when there are certain objects in the image need more detailed segmentation on it.

Basically, image segmentation is performed to segment out objects in foreground from the background of an image. However, when the background of an image is rich with multiple colour components, it gives challenges to segment out the objects from background. Especially the object’s edges seemingly appear to be diffused with background. Unclear or blurry boundaries even give more challenges to for primitive image segmentation method to segment image into clusters.

Fig. 10 shows an image with multiple colour components being segmented out. Although the Fig. 10 shows that the segmentation does not produce desirable results, but in the least the running humans in the image are segmented out from the background which appears to be the crowd and grass field. The two segmented humans (foreground objects) have fewer colour components as compared to the background with more colour components because of the humans’ bodies are largely covered by the skin colour. This helps the normalised cuts algorithm to segment out the humans without much effort. The image cell size covering the human body in the image is adequate small enough to perform the first stage segmentation since the discrimination degree in each of the cells is sufficiently enough.

V. CONCLUSIONS This paper presents an alternative approach to normalised

cuts algorithm based image segmentation. With this approach, it enables normalised cuts algorithms to perform segmentation on image part by part individually instead of performing segmentation on whole image in one stage. This helps to speed up the normalised cuts algorithm.

To further improve the segmentation performance to be balanced up with the trade-off between efficiency and effectiveness, adaptive image cells division can be implemented. The number of image cells to be divided into can be adaptively set according to the image content. When foreground area takes up more than background area, the number of image cells should be cautiously not too many to reduce the tendency of waste in computation for the segmentation. The image cells can have multiple sizes since the frequency content of an image is not evenly distributed. For example, smaller image cells can cover on region with fine details, while bigger image cells cover on background which contains fewer details. Nevertheless, the optimal trade-off still has to be implied without affecting either the efficiency or accuracy significantly.

Although a higher level knowledge can be used extra in the segmentation, there is still a potential much to explore using different primitive domains to do segmentation in the absence of high level knowledge while the computation is not expensive.

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

Cluster 1

Cluster 2

5 clusters per cell

Cluster 1

Cluster 2

10 clusters per cell

Cluster 1

Cluster 2

20 clusters per cell

Figure 9. Comparison of 5 × 5 image cells segmentations with different number of clusters per cell.

363366

1.

2.

3.

4.

5.

6.

7.

8.

9.

10.

11.

12.

13.

14.

15.

16.

17.

18.

19.

20.

21.

22.

23.

24.

25.

26.

27.

28.

29.

30.

31.

32.

33.

34.

35.

36.

37.

38.

39.

40.

41.

42.

43.

44.

45.

46.

47.

48.

49.

50.

51.

52.

53.

54.

55.

56.

57.

58.

59.

60.

61.

62.

63.

64.

65.

66.

67.

68.

69.

70.

71.

72.

73.

74.

75.

76.

77.

78.

79.

80.

Cluster 1 Cluster 2

Cluster 3

Figure 10. 8×10 image cells segmentation and its 3 segmented clusters.

ACKNOWLEDGMENT The authors would like to acknowledge the funding

assistance of Postgraduate Centre of Universiti Malaysia Sabah (UMS) under Postgraduate Assistance Scheme.

REFERENCES [1] S. C. Zhu, and A. Yuille, “Region Competition: Unifying Snakes,

Region Growing, and Bayes/MDL for Multiband Image Segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 18, Sep. 1996, pp. 884-900, doi:10.1109/34.537343.

[2] Z. Wu and R. Leahy, “An optimal graph theoretic approach to data clustering: theory and its application to image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 15, Nov 1993, pp. 1101 -1113, doi: 10.1109/34.244673.S. Maji, N. Vishnoi, and J. Malik, “Biased normalized cuts,” 2011 IEEE Conference onComputer Vision and Pattern Recognition (CVPR), June 2011 ,pp. 2057 -2064.

[3] J. Shi and J. Malik, “Normalized cuts and image segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, Aug 2000, pp. 888 -905, doi: 10.1109/34.86868.

[4] A. M. Martı ́nez, P. Mittrapiyanuruk, and A. C. Kak, “On combining graph-partitioning with non-parametric clustering for image segmentation,” Computer Vision and Image Understanding, vol. 95, July 2004, pp. 72-85, doi: 10.1016/j.cviu.2004.01.003.

[5] L. G. Shapiro and G. C. Stockman, Computer Vision, Prentice Hall, 2001.

[6] L. Angeline, M. Choong, F. Wong, and K. Teo, “Tracking and localisation of moving vehicle license plate via signature analysis,” 4th International Conference On Mechatronics, May 2011, pp. 1 -7.

[7] J. Malik, S. Belongie, T. Leung, and J. Shi, “Contour and Texture Analysis for Image Segmentation,” International Journal of Computer Vision, vol. 43, June 2001, pp. 7-27, doi: 10.1023/A:1011174803800.

[8] P. Arbelaez, M. Maire, C. Fowlkes, and J. Malik, “Contour Detection and Hierarchical Image Segmentation,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 33, no. 5, May 2011, pp. 898-916, doi: 10.1109/TPAMI.2010.161.

364367