Extended Gaussian-filtered Local Binary Patterns forcolonoscopy image classification

Siyamalan Manivannan Ruixuan Wang Emanuele TruccoCVIP, School of Computing, University of Dundee, UK.

{msiyamalan, ruixuanwang, manueltrucco}@computing.dundee.ac.uk

Figure 1: Some colonoscopy images: normal (left) and abnormal (right).

Abstract

Local Binary Patterns (LBP) and its variants are widelyused for texture classification. In this paper we propose anew variant of LBP descriptor called the extended Gaus-sian filtered Local Binary Patterns (xGF-LBP) which is ro-bust to illumination changes, noise and captures more in-formative edge-like features for classification. Experimentson a colonoscopy image dataset with 2100 images for bi-nary (‘normal’ or ‘abnormal’) classification show that theproposed xGF-LBP descriptor significantly outperforms thestandard LBP descriptor and its considered variants.

1. Introduction

Colorectal cancer is the second leading cause of cancerdeath in the world [1]. Adenoma detection rate (ADR), interms of lesion detection, is a surrogate marker for qualityof colonoscopy [15]. A reliable image processing systemdetecting abnormalities (including polyps, cancer, ulcers,etc.) in colonoscopy videos would be a useful screeningtool to improve ADR by providing a consistent, repeatableand quantitative second opinion [13]. Here, we concentrateon normal-abnormal frame classification, a challenging taskas abnormalities in colon vary in size, type, color, and shape(Figure 1).

Methods proposed for colonoscopy image classificationare mainly focusing on feature design. Texture, color, shapeand their combinations have been used in colonoscopy im-age analysis as features. Gray level co-occurrence ma-trices (GLCM) (e.g., detection of precancerous polyps in[14]), Local binary patterns (LBP) (e.g. normal vs ab-

normal colonoscopy image classification in [20]), TextureSpectrum (e.g. normal-cancer classification in [7]), etc. areused as texture features. Statistics of color values in dif-ferent color channels (e.g. color histograms for bleedingdetection in [8]), shape features (e.g., ellipses approximat-ing contours for polyps detection in [6]) and combinationof color, texture and/or shape features (e.g. Crohn diseaseclassification in [9]) are also applied for colonoscopy imageclassification.

Local Binary Patterns (LBP)[17] is an efficient texturedescriptor. Several variants of LBP have been proposed inthe literature to improve its performance in classification.Due to its discriminative power and computational simplic-ity, LBP is proven to be a good descriptor for colonoscopyimage classification and outperforms other considered fea-tures proposed in the literature for colonoscopy image anal-ysis [20]. However, the standard LBP and its variants maynot be robust to large illumination changes which oftenhappens during colonoscopy with lighting source changedor adjusted. In this paper we propose a new variant ofLBP called the extended Gaussian filtered Local Binary Pat-terns (xGF-LBP) which is robust to noise and illuminationchanges. In our formulation the standard LBP can be con-sidered as a special case of xGF-LBP. We experimentallyshow that the proposed representation not only is robustto noise and illumination changes but also outperforms theLBP and its other variants on a colonoscopy dataset with2100 images.

In the following, LBP and its relevant variants will befirst introduced (Section 2) in order to explicitly comparewith the proposed new feature (Section 3), with quantitativecomparison shown later (Section 4).

2013 IEEE International Conference on Computer Vision Workshops

978-0-7695-5161-6/13 $31.00 © 2013 IEEE

DOI 10.1109/ICCVW.2013.31

184

2013 IEEE International Conference on Computer Vision Workshops

978-1-4799-3022-7/13 $31.00 © 2013 IEEE

DOI 10.1109/ICCVW.2013.31

184

2. Local Binary PatternsLBP[17] as a powerful texture descriptor has been

widely applied to texture classification, face recognition[5], and medical image classification [20], etc. Whilemany variations of LBP have been proposed, e.g. uniformLBP[16] and BlockLBP[12], here we only review LBP andthe most relevant variants.

LBP with N uniform sampling points on a circle of ra-dius R around a 2D point pc in a gray image I can be de-fined as:

LBPN,R(pc) =

N∑n=1

qn×2n−1 where, qn =

{1 In ≥ Ic0 In < Ic

(1)Ic and In respectively represents the intensity values at thecenter point pc and the n-th sampled image point. In isbilinearly interpolated when the sampling point is not coin-cided with a pixel coordinate. Since this operator gives 2N

different labels an image can be represented as a histogramwith 2N bins.

To make LBP robust to noise, a three level thresholdingis applied in Local Ternary Patterns (LTP) [22] by the in-troduction of a user specified threshold τ (Equation 2). Thehistogram representation of an image by LTP is obtained bysplitting each LTP into two LBP and then concatenating thetwo LBP-based histograms.

qn(τ) =

1 In ≥ Ic + τ0 |In − Ic| < τ−1 In ≤ Ic − τ

(2)

However, LTP may be sensitive to illumination changes.When a scene illuminated by a single distant light source,the observed luminance image I(x, y) at point (x, y) canbe approximated as the product of the reflectance imageR(x, y) and the illuminance image s(x, y) [2], i.e.,

I(x, y) = s(x, y)R(x, y) +G(x, y) (3)

An ideal LBP should be robust to the change in illumina-tions s and the Gaussian noise G. While LTP is robust tonoise with the introduction of the threshold τ , it is sensitiveto the changes in illumination s, where the same threshold τ(Equation 2) may result in different LTP for an image underdifferent illumination conditions (see Figure 2(c)).

To handle the variation in illumination, a variant of LTPcalled the Scale Invariant Local Ternary Pattern (SILTP)has been proposed in [11], i.e.,

SILTPN,R(pc, w) = ⊕Nn=1bn(w)

where, bn(w) =

01 In > (1 + w)Ic10 In < (1− w)Ic00 otherwise

(4)

where w is the scale factor and⊕ denotes concatenation op-erator of the 2-bit binary strings bn. Note that SILTP is notdesigned particularly for image classification. In fact, the‘2-bit’ codes can be converted to ‘ternary’ patterns to gen-erate a histogram representation for an image. In addition,SILTP does not consider the possible effect of noise, i.e.,the threshold value is fixed to 0 as in LBP, which may leadto a representation sensitive to noise (Figure 2(d)).

Different from LBP and its variants, the proposed newLBP variant, called extended Gaussian filtered local binarypatterns (xGF-LBP), is robust to both noise and illuminationchanges by capturing edge-like features.

3. Extended Gaussian-Filtered LBP

We first extend the standard LBP by adding a thresholdparameter τ and a scale factor w, i.e.,

x-LBPN,R(pc, w, τ) =

N∑n=1

qn(w, τ)× 2n−1 (5)

where,

qn(w, τ) =

{1 w × In − Ic ≥ τ0 otherwise (6)

In the extended LBP (i.e., x-LBP), w can account for illu-mination changes and τ can account for noise (e.g., Fig-ure 2(e)). The avoidance of ‘ternary pattern’ in x-LBPmakes it easy and efficient to generate a histogram repre-sentation for an image as by the standard LBP. In fact, thestandard LBP can be seen as a special case of x-LBP withw = 1 and τ = 0.

3.1. Effect of parameters

Here we qualitatively show that the proposed x-LBP cancapture edge-like features in an image by approximate pa-rameter setting. Figure 3 shows a noisy CT brain image andthe x-LBP codes obtained with different parameter settings.While the standard LBP (Figure 3(f)) codes are very noisy,changing the threshold values produces a better representa-tion (Figures 3(e) and 3(g)). Changing both the thresholdand the weights leads to an even better noise-reduced LBP(Figure 3(b)), with meaningful edge-like features preservedand noises reduced.

Figure 4 shows a colonoscopy image under two differentillumination conditions (Figures 4(a) and 4(b)) as well asand its LBP and x-LBP representations. It is clear that theoriginal LBP codes (second column) are very noisy com-pared to the x-LBP (last two columns). With appropriateparameter setting, x-LBP codes are also robust to illumina-tion variations (e.g., the two very similar x-LBP representa-tions in the last column).

185185

(a) Image patches (b) LBP (c) LTP (d) SILTP (e) x-LBP

Figure 2: A demonstrative example for the effect of illumination change and noise on (b) LBP, (c) LTP (τ = 5), (d) SILTP (w = 0.1), andthe proposed (e) generalized LBP (w = 0.9, τ = 5). (a) Top:original image patch with 3× 3 pixels (i.e., s = 1 and G = 0 in Equation 3);Bottom: the transformed image patch with a different illumination (s = 2) with noise added to the pixels in red. LTP is robust to noise butnot to illumination changes. LBP and SILTP are robust to illumination but not to noise. The proposed x-LBP is robust to both noise andillumination.

Figure 3: x-LBP of a CT image with different parameter settings.One LBP code (with range [0, 255]) is generated for each pixeland represented using a heat map, where color red indicates highervalues and color blue indicates lower values.

Figure 4: x-LBP of two colonoscopy images under different il-luminations with different parameter settings. (the illuminationchanged image (e) is generated by applying a constant illumina-tion field (s = 2) to the original image (a) according to Equation3 ).

3.2. Extended Gaussian-Filtered LBP (xGF-LBP)

Gaussian filtered LBP (GF-LBP) has been proposed [19]to capture a larger neighborhood information and reduce

noise effect, where at each sampling point a Gaussian fil-ter is applied to collect intensity information from an arealarger than the original single pixel. Figure 5 shows an ex-ample for the generation of GF-LBP, where the local neigh-borhood is quantized radially into three resolutions (radii),and at each resolution a set of (N = 8) sampling regions(indicated as circles) are considered. A LBP code is con-structed at each resolution by sampling the neighborhood atthe centers of the solid circles, after a Gaussian filter withstandard deviation proportional to the radius of the circle isapplied to each circle’s center [19]. Such a sampling pat-

Figure 5: The Gaussian filtered sampling points.

tern of GF-LBP is similar to the spatial structure of recep-tive field of human retina, and has been widely adopted inrecently developed visual features in computer vision, likeFREAK [3], BRISK [10], and DAISY [23]. The combi-nation of the sampling pattern with the proposed x-LBP iscalled xGF-LBP.

3.3. Image representation based on xGF-LBP

Denote by c a particular color channel (e.g., c ∈ {1, 2, 3}for a color image), and by m a particular resolution (e.g.,m ∈ {1, 2, 3} in Figure 5), and by h(c,m) the x-LBP his-

186186

togram of the an image I at the particular c and m (whichcan be generated as the standard LBP). Then the imagecan be easily represented by concatenating the histogramsh(c,m) of x-LBP over all the resolutions and the colorchannels, resulting in the larger-dimensional histogram Hof xGF-LBP. The final image representation is then obtainedas a normalized version of H to make the features from dif-ferent images to the same scale. Here we apply the L2 andpower normalizations [18]:

H←√H

‖H‖2, (7)

where√H represent the square root operation on each com-

ponent of H. Unlike LTP where the final histogram repre-sentation of an image doubles the size of the GF-LBP his-togram, in our case the dimensionality of the final featurevector is same as in the GF-LBP.

4. Experiments

This section describe the experimental settings and mul-tiple experimental comparisons between the proposed xGF-LBP and relevant features in classification of colonoscopyimages as normal or abnormal.

4.1. Experimental setup

The dataset contains 1050 normal and 1050 abnormalcolonoscopy images, where the abnormal images containvarious types of lesions or bleedings (Figure 1). Eachimage is rescaled by preserving its aspect ratio such thatthe maximum height is 300. For each image, xGF-LBPis extracted at every pixel in each color channel withthe sampling pattern shown in Figure 5, where a three-resolution version was used with radii = {1, 2.43, 5.44}and 8 sampling points were considered at each resolu-tion, and 2D Gaussian filters of {(window size = 3 ×3,standard devision = 0.38), (window size =7× 7,standard devision = 0.85)} were applied forthe 2nd and 3rd resolutions respectively [19]. Gaussian fil-ter is not applied for the first resolution level. Once thexGF-LBP is extracted, the histogram of xGF-LBP is usedas the representation for the image (see Section 3.3). Forall other LBP variants (including standard LBP and LTP), asimilar process is applied to generate the corresponding his-tograms for each image. In all the cases the uniform patternhistogram representation [16] is considered as it capturessome meaningful patterns such as edges, corners, etc. andreduces the dimensionality of the final histogram represen-tation.

With the extracted histograms as image features, SVMclassifier is trained (using the LibSVM toolbox [4]) with

the exponential chi-square kernel:

K(H1,H2) = exp

(−γ

2

d∑i=1

(H1i −H2i)2

H1i +H2i

)(8)

where H1 and H2 are d-dimensional histograms to repre-sent any two images, and H1i and H2i are the i-th compo-nent for the two histograms. The kernel parameter (γ) andthe regularization parameter (C) of the soft margin SVM arelearned based on a 5-fold cross validation on the trainingdata. Classification performances are measured based thepercentage of correctly classified images on the test dataset.Every classification result is based on the average over 10experimental runs.

4.2. Classification using xGF-LBP

This section evaluates the classification performance ofthe poposed xGF-LBP, in comparison with the followingbaselines:(1) LBP: the standard LBP.(2) LTP: different threshold values τ ∈ {5, 10, 20, 30} weretried and the best classification performance was reported.(3) DoG-LBP: Since the proposed xGF-LBP captures edge-like features, we also applied DoG-LBP [21], where adifference-of-Gaussian (DoG) is applied to an image to em-phasizes the edges and LBP is then computed from the DoGfiltered image. For the two (‘inner’ and ‘outer’) Gaussiansin the DoG filter, both the scale σ1 of the inner Gaussianand the scale σ2 of the outer Gaussian are chosen from theset Σ = {0.25, 0.5, 1, 2}, such that σ1 < σ1. Any of thetwo different window sizes {5× 5, 7× 7} is tried for bothGaussians in DoG. The best classification performance wasreported based on the different combinations of scales andwindow sizes.

For xGF-LBP, the best performance was reportedbased on the different combinations of the thresh-old τ ∈ {−10,−5, 0, 5, 10} and the weight w ∈{1.4, 1.2, 1, 0.8, 0.6}. The size of the histogram featuresfor LBP, DoG-LBP, and the proposed xGF-LBP is 531(3 colors × 3 resolutions × 59 histogram bins), the his-togram size for LTP doubles (i.e., 1062). For the classifi-cation, for each experimental run, a set of N training im-ages were randomly selected from each of the two classes,and the remaining images were used for testing, whereN ∈ {50, 100, 200, 300, 400, 500}

Figure 6 shows that the histogram based on the proposedxGF-LBP gives the best performance (with w = 0.8 andτ = 0) and significantly outperforms other LBP versions.LTP gives a better performance (with τ = 10) than LBPand DoG-LBP, and DoG-LBP gives the worse result. Table1 shows the effect of the parameters (w and τ ) when 500 im-ages from each class are considered for training. It showsthat varying the threshold value (τ ∈ −10, 0, 10) when

187187

Figure 6: Experiments with the proposed xGF-LBP and otherLBP variants.

Figure 7: Classification performance of different methods un-der different illumination conditions.

w = 1 does not affect the classification performance. How-ever, varying w results in different performance, with max-imumly about 3% improvement compared to the standardLBP. Such improvement may result from the illuminationrobustness of the proposed xGF-LBP. During colonoscopy,the movement of the camera and lighting source inside thecolon may lead to the imaging of the colon mucosa underdifferent lighting conditions. The next experiment will fur-ther support the illumination robustness of xGF-LBP.

Table 1: The effect of w and τ in xGF-LBP (accuracy ± std)

HHHHτw

1.4 1.2 1 0.8 0.6

-10 92.1± 0.7 92.0± 0.5 91.3± 0.5 90.6± 0.7 91.7± 1.20 93.9± 1.3 93.9± 0.8 91.0± 0.5 94.4± 0.9 93.1± 0.6

10 93.0± 0.9 91.5± 0.7 91.0± 0.7 92.1± 0.6 91.1± 0.6

4.3. xGF-LBP under different illuminations

This experiment investigates whether the proposed xGF-LBP is robust to illumination changes, where a set of im-ages (500 normal and 500 abnormal) were used for classifiertraining, but the remaining images were artificially changedthe illumination as the test images. For any original test im-age I , the illumination-changed image Il is created by mul-tiplying the intensity values of I by an illumination field s (aconstant illumination field s = λ was applied for the wholeimage) as in Equation 3. The intensity values of Il whichare greater than 255 are cropped to 255. Figure 8 showssome of the examples with different illumination factors λ.

With the new illumination-changed test images, the sameexperiment as in Section 4.2 were repeated and the re-sults are shown in Figure 7. One can see that classifica-tion performance based xGF-LBP is robust (approximatelyconstant) to different illuminations, while the performancebased on LBP and LTP significantly degrades when the il-lumination decreases or increases. It is a bit surprise that

(a) λ = 0.5 (b) λ = 0.75 (c) λ = 1 (d) λ = 1.25 (e) λ = 1.5

Figure 8: An image under different illumination conditions (orig-inal image is given by λ = 1).

LBP performs poorly with illumination changes because ingeneral LBP is believed to be robust to illumination. Figure9 uses an exemplar patch to illustrate the possible reasonfor the sensitivity of LBP to illumination changes. Figure9(b) shows the patch 9(a) under a different illumination (thevalues are multiplied by 0.5). The pixel values are roundedto its integer representations to make sure they are in in-teger set [0, 255]. As a result of round-off errors, the thepatches shown in Figure 9(a) and Figure 9(b) give differentLBP representations lead to decrease in classification per-formance.

(a) a patch and its LBP repre-sentation

(b) illumination changedversion of (a) and its LBPrepresentation

Figure 9: LBP codes of a patch under different illuminations. Theshaded codes shows the codes which are affected by illumination.

5. ConclusionsIn this paper we proposed a new variant of LBP,

called xGF-LBP, which is robust to noise and illuminationchanges. Based on the experiments on a colonoscopy imagedataset, we showed that the proposed method outperformsother LBP variants and robust to illumination changes in the

188188

binary classification task. Future experiments will be focus-ing on testing the proposed feature on different datasets andon a dataset which is affected by both noise and illumina-tion.

Acknowledgement This work is funded by 2011-2016EU FP7 ERC project “CODIR: colonic disease investi-gation by robotic hydrocolonoscopy", collaborative be-tween the Universities of Dundee (PI Prof Sir A Cuschieri)and Leeds (PI Prof A Neville). Thanks to Dr.AdrianHood (Leeds Institute of Molecular Medicine, Universityof Leeds, UK) for helping with the data annotations.

References[1] Cancer research UK. info.cancerresearchuk.org/

cancerstats.[2] E. H. Adelson and A. P. Pentland. Perception as Bayesian inference.

Cambridge University Press, 1996.[3] A. Alahi, R. Ortiz, and P. Vandergheynst. FREAK: Fast Retina Key-

point. In IEEE Conf. on CVPR, 2012.[4] C.-C. Chang and C.-J. Lin. LIBSVM: A library for support vector

machines. ACM Trans. on IST, 2, 2011. Software available at http://www.csie.ntu.edu.tw/~cjlin/libsvm.

[5] D. Huang, C. Shan, M. Ardabilian, Y. Wang, and L. Chen. Localbinary patterns and its application to facial image analysis: A survey.IEEE Trans. on SMC, 41(6), 2011.

[6] S. Hwang, J. H. Oh, W. Tavanpong, J. wong, and P. C. de Groen.Polyp detection in colonoscopy video using elliptical shape feature.IEEE ICIP, 2007.

[7] S. Karkanis, K. Galousi, and D. Maroulis. Classification of endo-scopic images based on texture spectrum. In In Proceedings of Work-shop on MLMA, Advance Course in AI, 1999.

[8] P. C. Khun, Z. Zhuo, L. Z. Yang, L. Liyuan, and L. Jiang. Featureselection and classification for wireless capsule endoscopic frames.ICBPE, 2009.

[9] R. Kumar, Q. Zhao, S. Seshamani, G. Mullin, G. Hanger, and T. Das-sopoulos. Assessment of Crohn’s disease lesions in wireless capsuleendoscopy images. Trans. on BEO, 59, 2012.

[10] S. Leutenegger, M. Chli, and R. Siegwart. Brisk: Binary robust in-variant scalable keypoints. In IEEE ICCV, 2011.

[11] S. Liao, G. Zhao, V. Kellokumpu, M. Pietikainen, and S. Li. Mod-eling pixel process with scale invariant local patterns for backgroundsubtraction in complex scenes. In IEEE Conf. on CVPR, 2010.

[12] S. Liao, X. Zhu, Z. Lei, L. Zhang, and S. Li. Learning multi-scaleblock local binary patterns for face recognition. In Advances in Bio-metrics, volume 4642 of Lecture Notes in CS. Springer Berlin Hei-delberg, 2007.

[13] M. Madhoun and W. Tierney. The impact of video recordingcolonoscopy on adenoma detection rates. Gastrointestinal En-doscopy, 75, 2012.

[14] D. E. Maroulis, D. K. Iakovidis, S. A. Karkanis, and D. A. Karras.Cold: a versatile detection system for colorectal lesions in endoscopyvideo-frames. Trans. on CMPB, 70, 2003.

[15] W. MB. Improving colorectal adenoma detection: technology ortechnique? Gastroenterology, 132, 2007.

[16] T. Ojala, M. Pietikainen, and T. Maenpaa. Multiresolution gray-scaleand rotation invariant texture classification with local binary patterns.IEEE Trans. on PAMI, 24(7), 2002.

[17] T. Ojala, M. Pietikäinen, and D. Harwood. Performance evaluation oftexture measures with classification based on kullback discriminationof distributions. Proceedings of the 12th IAPR ICPR, 1994.

[18] F. Perronnin, J. Sánchez, and T. Mensink. Improving the Fisher ker-nel for large-scale image classification. In ECCV, 2010.

[19] M. Pietikäinen, A. Hadid, G. Zhao, and T. Ahonen. Computer VisionUsing Local Binary Patterns. Springer, 2011.

[20] S.Manivannan, R.Wang, E.Trucco, and A.Hood. Automatic normal-abnormal video frame classification for colonoscopy. In IEEE ISBI,2013.

[21] X. Tan and B. Triggs. Preprocessing and feature sets for robust facerecognition. In IEEE CVPR, 2007.

[22] X. Tan and B. Triggs. Enhanced local texture feature sets for facerecognition under difficult lighting conditions. IEEE TIP, 19, 2010.

[23] E. Tola, V. Lepetit, and P. Fua. Daisy: An efficient dense descriptorapplied to wide-baseline stereo. IEEE Trans. on PAMI, 32(5), 2010.

189189