a ga-based fuzzy recommender system for region … ga-based fuzzy recommender system for...

International Journal of Fuzzy Systems, Vol. 16, No. 3, September 2014

© 2014 TFSA

290

A GA-Based Fuzzy Recommender System for Region-Based Image Retrieval

Tsun-Wei Chang, Yo-Ping Huang, and Frode Eika Sandnes

Abstract1

The emergence of digital cameras in recent years has led to an exponential increase in the amount of digital photographic images and the need for auto-matic image indexing and retrieval systems. In this paper, an efficient genetic algorithm-based image re-trieval strategy is proposed where user relevance feedback based on regions of interest is employed to improve the retrieval efficiency. The combination of low-level features from the selected regions forms the chromosomes of the genetic algorithm used for re-trieving the target images. The user relevance feed-back is used to direct the advanced search. Further-more, the retrieval performance is improved by min-ing association rules from the recorded feedback. Experimental results verify the effectiveness and scalability of the approach both in terms of retrieval precision and recall rates.

Keywords: Image retrieval, fuzzy model, relevance feedback, genetic algorithm, image data mining.

1. Introduction

In recent years, the volume of photographic images, videos and audio has grown rapidly. Advances in storage technology have facilitated the archival of huge quanti-ties of media data. The issue concerning how to effi-ciently search for digital images in such archives has long been the focus of research. Numerous retrieval tools, algorithms and strategies have been proposed to alleviate the image retrieval tasks. Conventional text based image

Corresponding Author: Yo-Ping Huang is with the Department of Electrical Engineering, National Taipei University of Technology, Taipei 10608, Taiwan. Email: [email protected] Tsun-Wei Chang is with the Department of Computer Science and Information Engineering, De Lin Institute of Technology, No. 1, Ln. 380, Qingyun Rd., Tucheng Dist, New Taipei City 236, Taiwan. E-mail: [email protected] Frode Eika Sandnes is with the Faculty of Technology, Art and De-sign, Oslo and Akershus University College of Applied Sciences, N-0130 Oslo, Norway. Email: [email protected] Manuscript received 25 April 2014; revised 10 June 2014; accepted 24 June 2014.

searching methods are subjective as they rely on textual descriptions. Manually annotating images with textual descriptions is time-consuming, laborious and impracti-cal. In addition, the semantics of text descriptions are subjectively captioned and are not universal. Different individuals perceive images differently. Even a lengthy textual description may be inadequate for expressing the rich contents of an image. Consequently, the limitations of text-based image retrieval will be subjective to the texture descriptions. The Content-Based Image Retrieval (CBIR) technique was proposed in the early nineties to tackle the barriers of text-based methods and consider the image visual information. The visual information including color [1-2], texture [3], shape [4] and location (spatial relationships) [5] is used to analyze the relevance between the queries and the database images. Since then, a wide range of applications has been developed in vari-ous academic and commercial domains.

Chen and Wang proposed a region-based algorithm for detecting skin color [6]. This algorithm utilized a special region, a key skin region, as the basis for classi-fying skin color. The proposed technique performed well, especially for facial regions. Moreover, the strategies were easy to implement. Huang et al. proposed a method for automatically locating tennis highlight by using computational intelligence strategies and information from both the audio and video domains [7]. Experimen-tal results from detecting tennis highlights show that both the mean precision and recall rates are higher than 89%. Huang et al. [8] employed principal component analysis (PCA) to reduce the number of candidate vari-ables in a system without sacrificing the principal com-ponents in the original data. Then, an APP on smart phone was presented to recognize facial quality through facial pictures taken by smart phone camera. Their ex-perimental results show high recognition accuracy. A two-stage shape-based image retrieval method first seg-mented an image into a fixed number of rectangular re-gions and then each region was represented by its low-frequency discrete cosine transform (DCT) coeffi-cients in the YUV color space [9]. The DCT is used to separate an image into parts of differing importance with respect to the image's visual quality. The DCT can also transform the image from the spatial domain to the fre-quency domain. A linear conversion between RGB and YUV color systems exists where the Y component cor-

T.-W. Chang, Y.-P. Huang, and F. E. Sandnes: A GA-Based Fuzzy Recommender System for Region-Based Image Retrieval

291

responds to luminance of color, U and V are in phase and quadrature, respectively. Both the U and V compo-nents can carry the color information. Besides, the Hue-Saturation-Intensity (HSI) color model is generally used in image processing. The hue (H) attribute, the perceived dominant color for users, is the main compo-nent in describing a pure color such as red, yellow, or-ange, green, etc. The saturation (S) attribute indicates the relative purity of a pure color. Intensity (I) is defined as the color brightness relative to white. It is a key factor in describing the quantity of color sensation.

The shape based image retrieval resulted in high ac-curacy. Tan and Isa proposed an approach that applies the histogram thresholding technique to obtain all possi-ble uniform regions in the color image [10]. Their method can obtain better cluster quality and segmenta-tion results. A relevance feedback mechanism in which positive and negative data are integrated to optimally retrieve images according to the available information was proposed by Arevalillo-Herráeza et al. [11]. Their results were compared to previous methods with other representative strategies to verify that a significant im-provement in performance was obtained.

Li and Wang proposed an optimization and estimation technique to serve as the basis for an automatic linguistic indexing of pictures-real time (ALIPR) system [12]. This system provided fully automatic and high speed annota-tion for online picture retrieval. Particularly, they devel-oped a D2-clustering method to group objects repre-sented by bags of weighted vectors. Together with a generalized mixture modeling technique, annotating pictures by computers can be achieved. The proposed approach can be applied in wide range of applications such as Web image search or online picture-sharing communities. A new cluster-based image retrieval scheme, called as CLUE, for improving user interaction with systems had been proposed in [13]. Unlike most CBIR systems, CLUE retrieved image clusters rather than single images. The proposed scheme first employs a graphic representation of images. All the images are viewed as nodes and the similarities between images would be denoted by weights of edges connecting the nodes. Clustering is then formulated as a graphic prob-lem, and Ncut technique is used to solve the similarity measures. This approach can not only efficiently retrieve images, but also can be embedded in many current CBIR systems. Chen, et al., proposed an approach for image categorization by learning and reasoning with regions [14]. In their work, images are viewed as bags, each of which contains a number of instances corresponding to regions obtained from image segmentation. Multi-ple-Instance Learning and Support Vector Machine (SVM) classifier are then applied to image categoriza-tion. Experimental results demonstrate higher categori-

zation accuracy is achieved. In most active CBIR systems, the query will be for-

mulated according to the example images provided by users. Based on the similarity of feature information between the query and the database images, the systems are capable of recommending target images. However, low-level image features are not suitable for expressing high-level semantics. Retrieval problems may occur when two different semantic objects share similar low-level features (false match), or when two semanti-cally similar objects have different low-level features (false mismatch). Recent strategies have attempted to narrow the semantic gap by establishing statistical rela-tionships between high-level and low-level features [15-16]. Another direction of research employs rele-vance feedback mechanism which incorporates user in-tervention in the retrieval model [17-20].

However, most of the conventional relevance feed-back mechanisms lack the ability to fully utilize the hu-man-system interaction history in the retrieval process. Only the current query session is considered and infor-mation about users’ historical retrieval behaviors is dis-carded. In order to solve this problem, in the proposed strategy data mining is used to discover common user image retrieval behavior in history data.

Research shows that humans do not place equal em-phasis on each object in an image. Therefore, re-gion-based image retrieval (RBIR) was proposed as a means of bridging the gap at the object-level (significant portion in an image), which is closer to human percep-tion. RBIR systems such as Netra [21] retrieve images based on the similarity between regions. The users can issue their interest query based on the specific regions associated with low-level features. As a result, different regions selected from multiple images comprise the complete image retrieval query. Nevertheless, few exist-ing models have considered the possible combination of regions selected from multiple images. In this paper, a genetic algorithm-based image retrieval model is pro-posed to support a more effective search in an image database. Fuzzy c-means coupling with multi-features algorithm are introduced for segmenting an image and dealing with the feature extraction. All database images are first segmented into several regions. A set of higher order statistical descriptors automatically extracted from color histograms, co-occurrence matrices and shape moments is used to represent a region. The regions can be chosen from single or multiple images to generate the chromosomes used for genetic algorithm. After a re-trieval task had finished, data mining is employed to analyze the history data for improving the retrieval effi-ciency. The proposed system architecture is shown in Fig. 1.


292

Figure 1. The proposed system architecture.

2. Feature Descriptors and Image

Representation

Image segmentation is still an active research area in computer vision [22]. The features used for retrieval can be extracted from segmented regions in images. The pro-posed image segmentation method is as follows.

The original images have different ratios of red, green, and blue. Since human vision is more sensitive to lumi-nance, we transform the Red-Green-Blue (RGB) space of a color image into HSI coordinates. After the trans-formation, the intensity component is used to represent the luminance of a pixel, which reflects the pixel inten-sity degree. The remaining two components H and S represent the hue and saturation of a pixel, respectively. Three steps are taken to fuzzify an image.

Step 1) Fuzzification: The fuzzification comprises the process of transforming crisp values into membership grades of fuzzy sets. The membership functions for H, S, and I are defined in Fig. 2.

Step 2) Fuzzy Reasoning: Fuzzy model characterizes the relationship between input and output of a system by some fuzzy rules. The form of a fuzzy rule is represented as follows. Ri: If I is Ai and H is Bi and S is Ci, then Z is Oi.

Some of these rules used in our system are as fol-lows.

R1: If Luminance is Black and Hue is Violet and Satu-ration is Low, then the Fuzzy Gray Level is Low.

R2: If Luminance is Black and Hue is Violet and Satu-ration is Medium, then the Fuzzy Gray Level is Low.

… R54: If Luminance is White and Hue is Red and Satura-

tion is High, then the Fuzzy Gray Level is High. Step 3) Defuzzification: The defuzzifier maps fuzzy

information into a crisp value. There are numerous de-fuzzification approaches, while the most important one is the center of area (COA) that is used in our model.

,)(

)(Z*

r

1i

r

1i

iC

iiC

COA

z

zz

μ

μ

(1)

where r is the number of quantization levels of the out-put, Zi is the amount of control output at the quantization level i and c(Zi) represents its membership value in c.

(a) Luminance membership function.

(b) Hue membership function.

(c) Saturation membership function.

(d) Aggregation.

(e) Defuzzification.

Figure 2. The H, S, and I variables and their corresponding membership functions used in the fuzzy model in generating fuzzy gray levels.

After preprocessing, a color image is transformed into a fuzzy gray image. This image segmentation procedure starts with a single pixel as the first region. According to the distance between a pixel and each region, the system can decide which region that pixel belongs to. The seg-mentation repeats the clustering process until all pixels of an image are properly classified into expected classes.


293

During the segmentation procedure, we have to calcu-late each cluster center. Fuzzy c-means algorithm repeats the Euclidean distances to decide the cluster centers. When the clustering has been done, each pixel iP will be properly classified into one region. The system de-fines c clusters such that the partition set can be denoted by },...,2,1{ cAAAA . The fuzzy pseudo partition set A

must satisfy the following conditions.

c

jnNiixjA

1. allfor , 1)( (2)

.1

allfor )(0

n

cNjnxAk

kj (3)

where n is total number of images, Nn is the set of n im-ages and NC is the set of C clusters.

The fuzzy clustering finds the partitions and the asso-ciated clusters by which the structure of the data is well represented. After all the feature vectors are classified, the correlation will be strong within clusters and weak between clusters. Further, the set of cluster center vec-tors },...,2,1{ cvvvV associated with each image in

the database can be calculated as follows.

,

1)]([

1)]([

n

k kxjAg

n

k kxkxjAg

jv (4)

where g is a real number and g > 1. Our goal is to retrieve similar images to the queries

based on region contents. A region is represented in terms of its pixel-related internal properties of color, texture and shape. Since most regions consist of complex patterns, a set of statistical descriptors, such as mean, variance, and higher order moments, is used to charac-terize a region. This section describes how the region content features are automatically extracted and how the statistical descriptors are computed. A. Color histogram moments

Color information represents the content of a color image. The hardware-centric RGB color model is not perceptually uniform as there are perceptual ambiguities between colors such as yellow and green. Therefore, the Hue-Luminance-Saturation (HLS) color components are utilized to construct the color histograms. A color histo-gram represents the color distribution of an image using an array of histogram bins. Each histogram bin reflects a color range in the color space.

Even though the color histogram approach is very practical in many applications it has certain drawbacks from an image retrieval perspective. The major challenge is that color histograms are sensitive to noise. Two simi-lar colors are categorized as identical provided they are placed in the same bin. On the other hand, a small illu-

mination change can cause two otherwise similar colors to be placed in different bins. Another problem is the high dimensionality. The number of features is propor-tional to the number of histogram bins. For example, the number of dimensions is 256 if 256 histogram bins (gray levels) are used. The computational effort associated with computing image retrieval similarity measures is directly related to the feature dimensionality. Finally, color histograms do not capture spatial information. To retain spatial relations, color histograms are statistically analyzed and the image color tendency is captured by the moments.

Given an image AI , let iZ denote a random gray

level and A

i

IZH , 1,...,1,0 Li be the corresponding i-th

gray histogram bin of an image IA, where L is the maxi-mum gray level. Each bin records the number of pixels with the corresponding gray value. To speed up the computation, the gray levels are reduced to 32 levels. The mean value m of gray levels (average gray level) is:

.1

0

L

i

AI

iZHiZm (5)

To provide an overview of analyzing data rather than only comparing the exact value of each pixel, 5 statisti-cal color descriptors are further utilized to evaluate the color histogram. This statistical quantum approach is helpful for distinguishing color information of an image described as follows.

The 2nd moment, also known as the variance 2 (also denoted as 2(Zi)), is used to describe the gray contrast and relative smoothness of an image as follows.

.)()(1

0

22

L

i

AI

iZi HmZz (6)

To avoid large value of the variance, the variance is normalized in the range of [0, 1] by dividing 2 by (L-1)2 and the normalized 2 equals to 2/(L-1)2.

Due to the decrease of backlight resolution, many im-age details will be lost. Some enhanced methods are ap-plied to brighten the image details such that the images are much clearer than that of unenhanced images. The contrast ratio (CR) is a property measuring the ratio of the luminance of the brightest color (white) to that of the darkest color (black). A high contrast ratio represents a better color representation. We utilize the inverse con-trast CR to measure the tendency of the smoothness of the relative intensity. The value of CR will be 0 for a constant area of a region, otherwise 1 for a high contrast area. The inverse contrast is as follows.

.1

11

2CR (7)

The histogram symmetry (skewness) is revealed by the 3rd moment (3). The positive and negative values indicating the histogram being respectively biased to the right and to the left are obtained as follows.


294

.)()(1

0

33

L

i

AI

iZHmiZz (8)

Since the summation of AI

iZH is equal to 1, the uni-

formity (U) will be maximum while all gray levels are equal.

.(1

0

2)

L

i

AI

iZHU (9)

The entropy that captures the information degree of an image is derived as follows.

.(log1

0)2

L

i

AI

iZAI

iZ HHe (10)

B. Co-Occurrence matrix for texture

A texture is a set of primitive descriptors to demon-strate the spatial relationship between pixels, especially the spatial relation of a pixel pair. The spatial relation of a pixel and its neighbors is mainly addressed in the pro-posed model. The statistical information can be derived from a co-occurrence matrix ],[ jiCd [10]. Moreover, the normalized co-occurrence matrix Nd is further used to define 5 statistical features.

.],[2i j

djiNEnergy (11)

An irregular co-occurrence matrix has a lower entropy than that of a regular co-occurrence matrix, namely:

].,[log],[ 22 jiNjiNEntropy d

i jd (12)

The contrast can be used to test the difference moment of a co-occurrence matrix. An image with large local variations has high contrast.

.],[)( 2 i j

d jiNjiContrast (13)

The co-occurrence matrix has a high homogeneity if large values occur in the matrix diagonal or if neighbor-ing elements have similar gray levels.

.||1

],[

i j

d

ji

jiNyHomogeneit (14)

Correlation is used to measure the linear dependency of the gray levels that appear in the matrix:

,

],[))((

ji

i jdji jiNji

nCorrelatio

(15)

where i, j, i, and j are the mean and the standard deviations of row i and column j, respectively. C. Shape moment features

The shape of a region within an object can describe the properties of the individual pixels or the groups of pixels for a given image. Both boundary-based and re-gion-based approaches are capable of describing the shape features of an image. In the proposed image re-trieval model, the spatial moments are employed to rep-

resent a region. A color image is first converted into gray. For the 2-D

digital image ),( yxf , the moment ( pqm ) of order (p+q) is

uniquely determined from ),( yxf [23]. The central moments are defined as follows.

),,()_

()_

( yxfyyxx qp

x ypq (16)

where 00

10_

mmx and .

_

00

01

mmy

The central moments of orders up to 3 will be derived and the normalized central moments are defined as fol-lows.

,00

pq

pq (17)

where 12

qp , for ,...3,2 qp .

Finally, a set of 7 spatial moments including 1 to

7 derived from the second and third moments is used to describe the shape of a region.

,02201

,4)( 2202202 11

,)3()3( 20321

212303

,)()( 20321

212304

])(3))[()(3( 20321

21230123012305

))(3( 03210321 ],)()(3[ 2

03212

1230 ])())[(( 2

03212

123002206 ),)((4 0321123011 ),)((4 0321123011

])(3))[()(3( 20321

21230123003217

))(3( 03213021 ].)()(3[ 2

03212

1230 (18)

3. A GA-Based Image Retrieval Model A. Genetic algorithm for image search

The genetic algorithm (GA) originated by John Hol-land is a model for machine learning through the process of evolution [24]. This learning purpose is achieved by generating a population of individuals to represent the chromosomes. A chromosome is a string organized by a set of elements which can be thought as human gene. The chromosomes in the population compete with others through evolution for survival. After the evolution is over, better chromosomes are used to represent possible solutions for the to-be-solved problems.

The GA starts with an initial population in which the


295

chromosomes are randomly generated. A fitness value of each individual is evaluated to determine how good this chromosome is to solve the given problem. Both cross-over and mutation operations are also used to help search better solutions for the given problem [24].

An important characteristic of the previous image re-trieval schemes is that a single signature for the entire image is required. Consequently, most conventional models fail to retrieve images with similar objects lo-cated in different locations or objects having different sizes. A GA-based image retrieval model is proposed to tackle this problem. The main idea is to treat a feature vector derived from a region as a bulk of genes. The chromosome is constructed as the possible combinations of the selected regions. Each chromosome is a possible solution in the search space. In other words, the possible target images will be found from the evolved chromo-somes. How to apply the GA to the proposed image re-trieval model is described next. B. The implementation of image retrieval model

1) Chromosome representation: The image features are used in computer vision to summarize the content of an image. A set of finite number of pixels is the basic elements of a digital image. Each pixel with corre-sponding property can describe the image content. The content, so called image features, is used with the pur-pose to make the digital image more practical in applica-tions. Usually, the visual content features can be classi-fied into two categories: low-level content features and high-level semantic features. The one-dimensional dis-tribution of image intensities and the three-dimensional distributions of image colors are two kinds of image color features. Others like texture, shape, and location are commonly used features to characterize the digital images. The high-level semantic features require training from the low-level content features so that the tasks of recognition images or objects in specific domain can be achieved. The proposed work emphasizes on the low-level content features such that an efficient image retrieval system can be constituted. Some commonly used low-level features in currently CBIR systems are introduced in the proposed work.

Table 1. The content descriptors used in the proposed image retrieval model.

Feature Content Descriptor Color 5 features (Variance, Contrast, Skewness, Uniformity, Entropy).

Texture 5 features (Energy, Entropy, Contrast, Homogeneity, Correla-tion).

Shape 7 features (1st moment, 2nd moment, 3rd moment, 4th Moment, 5th moment, 6th moment, 7th moment).

Total 17 features

The 17 image content descriptors used are listed in Table 1. In the retrieval task, users can select interesting regions as the queries. Assume that the feature descrip-tors in a region have a magnitude of M (M=1,2,…,17). Then, a chromosome Ci is constructed as follows.

},,...,,...,,...,,,...,{ 122

111

1rM

rMMi hhhhhhC (19)

where kgh is the magnitude of the gth feature descriptor

in the kth region. The chromosomes form a population as follows.

,21for },,...,,{ 21 ize,...,pop_s, i CCCP i (20) where pop_size is the population size. Fig. 3 illustrates how the population is constructed. Given r regions se-lected by users, there will be r! possible combinations. To maintain a medium sized population for the sake of retrieval efficiency, a population size of 50 is used in our model even when r is greater than 5. Otherwise, a popu-lation size of r! is used if the number of selected regions is less than 5.

(a)

(b)

(c)

Figure 3. Chromosome and population construction procedure: (a) a single image as the query; (b) multiple images as the query and (c) the chromosome generated from the selected regions.


296

2) Fitness Values: The fitness value is calculated from the distance summation between the feature descriptors of the query region q

bR in a chromosome and the can-

didate region tbR of a database image. The distance

which represents the dissimilarity between a query and a database image is used as the fitness value. The well-known Euclidean distance is used to compute the dissimilarity of ith chromosome to a database image as follows.

.])[(1

2/12

r

b

tb

qbi RRdist (21)

The current system segments each image into 5 re-gions. The number of selected regions r is therefore lim-ited up to 5. The distances for a chromosome Ci to any database image (Im) are ranked in descending order as follows. The lowest distance is taken as the fitness value for that chromosome. At the same time, the most poten-tial candidate image is kept in the system.

)}.(),...,(),(min{ 21 miiii IdistIdistIdistFitness (22) 3) Selection process: The selection strategy expects

“better chromosomes” to contribute more genes to off-spring, and the roulette wheel mechanism is employed as the selection strategy. A chromosome with a high fitness value represents a poor match, while a low fitness value represents a matching target image. The individual se-lection rate (Ind_Ratei) of a chromosome is determined by:

sizepop

jj

ii

fv

ftRateInd

_

1

1_ , (23)

where ift is the fitness value of the ith chromosome. Furthermore, the average selection rate of all indi-

viduals is calculated as the sum of all individual selec-tion rates divided by the size of chromosome population:

._

__

_

1

sizepop

RateIndRateAverage

sizepop

ii

(24)

The individual reproduction rate Ind_Pi is thus ob-tained as follows.

RateAverage

RateIndPInd i

i _

__ . (25)

The chromosomes with high individual reproduction rate (Ind_Pi) representing good solutions will have higher probability to survive next generation.

4) Crossover and mutation: Both crossover and muta-tion are the basic operations in search of better chromo-somes. Each chromosome represents the candidate target images in the search space. The single-point crossover operation is used in our model. Mutation operation fol-lows the crossover to extend the search space.

4. A Two Stage Relevance Feedback

Image retrieval based on content features is subjective and personal. Meanwhile, it is impractical to change the clustering structure of the database all the time. Besides, a significant portion of the retrieved images is irrelevant due to the gap between the image content features and the user’s needs. A two-stage relevance feedback (RF) mechanism is proposed to overcome this challenge. The proposed strategy includes three phases: (1) The initial retrieval images are from region-of-interest search. (2) Users view the displayed images and decide their de-grees of satisfaction. (3) The historical retrieval log files are used for data mining.

Stage 1: Weigh refinement

Users can select one or more of the presented images as feedbacks and specify the weights of the selected im-ages in accordance with their preference. Numeric values attached to the sliders (scroll-bar) show the user confi-dence in each of the selected regions. Then, the image search continues. The dissimilarity between query re-gions and the regions associated with an image in the database can be measured based on the overall weighted distances. These dissimilarities are used to establish a ranking list of the best candidate images. At this stage, we focus on adaptively refining the importance of the similarity from the search results to users’ desired targets. This information of relevant regions can provide a useful assistance for searching images. Based on the weight iw specified by users, the weighted distance is as follows.

.])[(_1

2/12

r

b

tb

qbi RRwdistweight (26)

Stage 2: Using data mining to improve image retrieval

accuracy Low-level content similarity relies on perceptual or-

ganization. In this paper, the knowledge of users’ inter-action with the system during the retrieval process is adopted to improve the retrieval accuracy. The server’s retrieval log files are used for data mining analysis. Us-ing data mining inference principles one can extract the semantic tendencies for average users. In particular, as-sociation rule mining can be used to uncover associa-tions or correlations from a set of data items. Given a set of items },...,,{ 21 niiiI , an itemset X is a non-empty subset of I. A transaction database DB contains a set of transactions; and each transaction T is a set of items in the transaction database. There exist two itemsets X and Y, where IX and IY . The itemset X is contained in itemset Y, if the relationship YX holds. An asso-ciation rule YX implies that a strong relation exists between X and Y. The support of YX represented as


297

)sup( YX indicates the confidences of items in a transaction X.

The following scenario outlines how the data mining technique is employed in the image retrieval model. Im-agine that the following data is logged in a transaction database. Each selected region is analogous to the pur-chased items in the transaction database. Mary, labeled user1, selected regions of mountain (labeled 1R ), cloud

(labeled 3R ) and snow (labeled 4R ) from two images as query in the first retrieval round. In the second re-trieval round, she selected regions of mountain (labeled

1R ), beach (labeled 2R ) and cloud (labeled 3R ) as que-ry. In the third retrieval round, she selected regions of mountain (labeled 1R ), beach (labeled 2R ) and grass

(labeled 6R ) from three images as query. Her target is

image 20ID . John, labeled user2, selected regions of

mountain (labeled 1R ), beach (labeled 2R ) and grass

(labeled 6R ) from three images as query in the first re-trieval round. In the second retrieval round, he only se-lected regions of flower (labeled 5R ) as query. His tar-

get is also image 20ID . The target image of Jennifer,

labeled user3, is also the image 20ID . She selected re-

gions of mountain (labeled 1R ), beach (labeled 2R ) and

grass (labeled 6R ) from three images as query. Peter,

labeled user4, selected regions of snow (labeled 4R ),

tree (labeled 7R ) and sea (labeled 8R ) as query in the first retrieval round. In the second retrieval round, he selected regions of beach (labeled 2R ) and grass (la-

beled 6R ) as query. In the third retrieval round, he se-

lected regions of mountain (labeled 1R ), tree (labeled

7R ) and sea (labeled 8R ) from three images as query.

His target is the image 20ID . The retrieval information is logged in a transaction database. The association rules can be mined from the transaction database (see Table 2). The mined association rules associated with the image ID are used as the matching templates for future images retrieval. For example, image 20ID is recommended

when a user selects regions { 1R , 2R , 6R } or { 4R }. Table 2. The regions selected for each retrieval round by dif-

ferent users are used as the query examples to retrieve images.

Round Selected

Regions User ID

First Round

Retrieval

Second Round

Retrieval

Third Round

Retrieval

Target Image

user1 R1, R3, R4 R1, R2, R3 R1, R2, R6 ID20 user2 R1, R2, R6 R5 N/A ID20 user3 R1, R2, R6 N/A N/A ID20 user4 R4, R7, R8 R2, R6 R1, R7, R8 ID20

association rule {R1, R2, R4},{R6}min_sup=2=>image ID20

5. Experimental Results and Discussion A. Experiments setup

Three datasets are used to test the efficiency of the proposed system. The first dataset, 1DS , consists of 1,000 general-purpose images with resolutions not ex-ceeding 200200 pixels from various categories in-cluding conveyances, people, animals, architecture, nat-ural scenes and flowers.

A rich image dataset can prevent the bias on searching a particular category of images. This dataset allows the claims of robustness to both rotation and transposition. The second dataset is a photograph collection from MIT Media Lab. 200 images from different categories were chosen and added to the dataset DS1. In total, DS2 com-prises 1,200 images. Besides, to demonstrate the validity of the proposed approach, the third dataset DS3 is ex-panded to comprise 2,000 images including light sensi-tive images such as sunrise and sunset ones. The pro-posed method can be shown to retrieve the relevant im-ages while illumination influences on images.

Crossover rate was 0.8, while mutation was set to 0.1 initially. These parameters were selected based on ex-periments that used various combinations of these pa-rameters. Finally, the GA crossover rate and mutation rates were selected as 0.6 and 0.1, respectively, when this particular combination worked best. The population size was 50 and the GA was run for 100 generations when retrieving images. The system initially displays some sample images with segmented regions. Each re-gion is displayed in one color to aid the user in selecting their interest regions. After receiving queries, the system starts the GA and displays the retrieval results in the user interface that allows users to provide feedbacks. The search continues until satisfactory retrieval results are obtained. Three types of experiments are carried out to demonstrate the robustness and effectiveness of the pro-posed GA-based model. The first experiment evaluates the ability of accessing images with rotated or trans-posed objects. The second experiment compares the re-trieval efficiency of the proposed system to that of two other approaches, namely strategies based on 256 gray histograms and the Haar wavelet transform. The purpose of the third experiment is to assess the improved re-trieval accuracy achieved with the proposed relevance feedback mechanism. B. Effectiveness evaluation

In the first experiment, an image which is not in the given dataset is added into the dataset DS1 at a random position. Then, a shifted image with transposed objects (the black high boots and the blue doll) is used as a que-ry image. The retrieval results demonstrate the ability of finding the transposed objects in images as shown in Fig.


298

4. Next, the original query image is rotated 90 clock-wise. The searching results comply with the previous test as shown in Fig. 5. According to the search results, the system ranks the images with similar objects at different locations or transposed objects at the top of the ranking list.

In the second experiment, two queries are conducted to demonstrate the effectiveness of the proposed model compared to the two other retrieval schemes. First, a natural scenery image not present in dataset DS1 is se-lected as the query image. The retrieval results obtained using 256 gray histograms and the proposed approach is compared. The proposed strategy achieves comparable, and sometimes better, results compared to the approach relying on 256 gray level histograms (see Fig. 6). The 256 gray level histogram method retrieves images with globally similar color distribution, but is unable to dis-tinguish local differences. Consequently, this approach gives the users limited control of the queries and thus returns irrelevant images (see Fig. 6(b)). The proposed strategy yields better search results than what is obtained using 256 gray level histogram-based method (see Fig. 6(c)), as it considers the differences within regions.

A flower image in dataset DS1 is used as the second query image. The retrieval results obtained using the Haar wavelet transform and the proposed approach are compared. The results of the proposed model are better than the ones obtained using the Haar wavelet transform (see Fig. 7). Although, the wavelet transform captures location, color, and texture information, the number of coefficients affects the retrieval accuracy. There is a trade-off between the number of coefficients and com-putational complexity. When the query image comprises a flower on a background of green leaves the proposed model retrieved most of the similar images except for two images of a bird with a background of green leaves (see Fig. 7(c)). The proposed model yields better re-trieval quality especially for the top 5 results.

In addition, the effectiveness is evaluated according to the precision and recall rates. A high precision indicates that a large portion of the returned images are relevant to a query. A high recall represents that most of the rele-vant images in the repository to the query can be found. Precision P is defined as the ratio of relevant images re-turned over all retrieved images. Recall R is defined as the ratio of relevant images returned over the number of relevant images in the database.

Four kinds of features selected to queries: col-or-feature (Color), texture-feature (Texture), shape-fea-ture (Shape), and the proposed method (Combination) are compared over the dataset DS1. Four images from different categories, i.e., flower, beach, car, and bird, are used for the queries. For each query, the precision and recall pairs corresponding to the top 1,2, , up to 50

Figure 4. The query image is transposed and inserted in the dataset DS1 randomly. (a) the query image; (b)~(f) The search results are arranged according to retrieval ranking list.

Figure 5. (a) The rotated query image; (b)~(f) The search re-sults from dataset DS1 are arranged according to retrieval ranking list.

Figure 6. (a) A natural scenery image not in the dataset DS1 is used as the query image at the top-left. The retrieved results between (b) color histogram, and (c) the proposed method.

Figure 7. (a) A flower image in the dataset DS1 is issued as the query image at the top-left. The retrieved results between, (b) the wavelet transform, and (c) the proposed method. retrieved images are calculated. The average precision and recall pairs are plotted for recall rates of 0.05, 0.1,…, 0.4. Fig. 8(a) shows that when the proposed multi-features scheme is used with the flower query im-age almost 70% of all of the relevant images are among the top 50 returned images. Less than 70% of the images


299

(a)

(b)

(c)

(d)

Figure 8. Precision versus recall rates using different query examples from dataset DS1 for (a) a flower image as the query, (b) a beach image as the query, (c) a mountain image as the query, and (d) a car image as a query. are returned on average when using a single feature. Fig. 8(a), Fig. 8(c) and Fig. 8(d) show that the precision rates are 1.0 for recall levels of 0.05 using the proposed multi-features (Combination). It indicates that the rele-vant images can be retained among the top 3 images for these three categories and most users are likely to only focus on the top 5 images. The proposed strategy em-phasizes the use of multi-feature descriptors to improve the precision rate and to reduce the computational com-plexity. Only 17 elements are used to characterize a re-gion and the maximum number of features is therefore

85 (a limitation of 5 regions per query iteration). Com-pared to the color histogram approach with 256 gray levels, the proposed strategy uses 66% less memory and the computational complexity is greatly reduced.

In the third experiment, the improved retrieval accu-racy achieved with the proposed relevance feedback mechanism is verified. The selected regions of the initial retrieval results are first compared with the matching templates, i.e., the image ID associated with the associa-tion rules; relevant images are then first recommended. Users’ selections are fed back to the system and subse-quently used to mine additional association rules off-line. The new association rules can subsequently be used to improve the retrieval efficiency for future queries. To verify this methodology, four images, taken from dataset DS2, with the categories flower, mountain, beach and car, are used as query images. The proposed relevance feed-back mechanism is compared with the retrieval ap-proaches based on both 256 gray level histograms and the Haar wavelet transform. The precision and recall rates of the top 10 returned images are shown in Table 3. Table 3. The precision/recall value between different ap-proaches and the proposed model employing the relevance feedback mechanism in submitting different images from various categories as the queries and taking top 10 images returned from the dataset DS2.

Query Image Categories

Retrieval Model

Flower Beach Mountain Car

256 Gray Level Histogram

0.7／0.14 0.5／0.1 0.3／0.06 0.5／0.1

Haar Wavelet Transform

0.6／0.12 0.4／0.08 0.4／0.08 0.3／0.06

Proposed Method (Initial Retrieval)

0.8／0.16 0.8／0.16 0.7／0.14 0.6／0.12

Proposed Method (First Feedback with association

rules)

0.9／0.18 0.9／0.18 0.8／0.16 0.8／0.16

The proposed model is more accurate on average, es-

pecially for the flower and beach categories. The re-trieval effectiveness can be further improved by the proposed relevance feedback. After the initial retrieval round, the 2nd, 3rd and 6th images from the car category are fed back to the system as shown in Fig. 9(b).

The query images are darkened 10% and brightened 10%. The retrieval results of top 5 matches from DS3 are shown in Fig. 10(a) and Fig. 10(b), respectively. For the alternative query image, the proposed strategy can re-trieve 3 same images out of 5, especially the top 2 re-turns are in same position. Nevertheless, these returns are visual features relevant to the query image.


300

Figure 9. The retrieval improvement by the proposed feedback mechanism, (a) the initial query image is not in the database, (b) the initial retrieval results, (c) first feedback retrieval re-sults while taking the 2nd, 3rd, and 6th feedback images from (b). Note that the retrieval results are ranked from left to right and from top to bottom.

(a) The query image on the left was darkened by 10%.

(b) The query image on the left was brightened by 10%.

Figure 10. (a) The query image was darkened by 10% and the top 5 matches from DS3 are shown from left to right. (b) The query image on the left was brightened by 10% and top 5 matches from DS3 are shown from left to right.

For genetic algorithm approach, possible solutions, i.e., candidate images are represented by chromosomes whose genes consist of image features. An initial popu-lation P(0) of C chromosomes is generated by selecting sets of images consisting of feature vectors. The objec-tive function used to estimate the fitness values of all chromosomes, is defined as the root mean square of cross-correlations between feature vectors belonging to the gene of the respective chromosome and the database images. The computational complexity is )( NCO for an image collection with N images and the number of chromosome C is limited up to 120. Fitness values

iFitness of chromosomes iC (for sizepopi _,...,2,1 ) are ranked in ascending order since the objective func-tion is to be minimized. Following a proportionate scheme for parent selection, a set of new chromosomes (offspring) is produced by mating the parent chromo-somes and applying crossover operations. This proce-dure is repeated in an iterative way until a convergence meets an optimal solution of the problem. The observed average query time for returning top 50 images is almost less than 1.2 seconds.

6. Conclusion and Future Works

A GA-based image retrieval approach with a novel

data mining-based recommender strategy is proposed. The proposed GA-based image retrieval model addresses two major issues in retrieving images. First, the chro-mosomes represent the possible region combinations. The population pool consists of potential solutions to the query image such that similar images are retrieved from the image database. Second, the GA-based model is ca-pable of retrieving images effectively even though the objects may be located at different positions in similar images. We conduct several experiments to evaluate the performance of this system. The experimental results verify that no single feature can exactly represent the content of an image. In addition, the automatically ex-tracted single low-level features cannot fully capture the semantic concepts of image content. Therefore, multiple features are combined. Users retrieve images by com-bining regions from different images. An image region is characterized by only 17 features that provide better performance in retrieving images. Furthermore, the search time is reduced by combining association rules with relevance feedback.

The current approach can be extended to enhance the image retrieval performance. For instance, the spatial relationship between regions was not considered in this paper. Furthermore, current study does not address the semantic gap issue. Future work will include combining the GA-based recommendations with semantic informa-tion.

Acknowledgment

This work was supported in part by Ministry of Sci-ence and Technology, Taiwan under Grants NSC 102- 2221-E-027-083- and NSC 102-2218-E-002- 009-MY2, and in part by joint project between National Taipei Uni-versity of Technology and Mackay Memorial Hospital under Grants NTUT-MMH-103-01 and NTUT- MMH-102-03.

References [1] X. Wan and C.-C. Kuo, “A new approach to image

retrieval with hierarchical color clustering,” IEEE Trans. on Circuits and Systems for Video Technol-ogy, vol. 8, no. 5, pp. 628-643, 1998.

[2] Y. Deng, B. S. Manjunath, and H. Shin, “Color image segmentation,” in Proc. of Int. Conf. on Computer Vision and Pattern Recognition, Fort Collins, CO, U.S.A., 1999, pp. 446-451.

[3] A. Laine and J. Fan, “Texture classification by wavelet packet signatures,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 15, no. 11, pp. 1186-1191, 1993.

[4] A. Del Bimbo and P. Pala, “Visual image retrieval


301

by elastic matching of user sketches,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 19, no. 2, pp. 121-132, 1997.

[5] J. W. Hsieh and W. E. L. Grimson, “Spatial tem-plate extraction for image retrieval by region matching,” IEEE Trans. on Image Processing, vol. 12, no. 11, pp. 1404-1415, 2003.

[6] W.-C. Chen and M.-S. Wang, “Region-based and content adaptive skin detection in color images,” Int. Journal of Pattern Recognition and Artificial Intelligence, vol. 21, no. 5, pp. 831-853, 2007.

[7] Y.-P. Huang, C.-L. Chiou, and F. E. Sandnes, “An intelligent strategy for the automatic detection of highlights in tennis video recordings,” Expert Sys-tems with Applications, vol. 36, no. 6, pp. 9907-9918, 2009.

[8] Y.-P. Huang, Y.-Z. Li, and C.-C. Lin, “Smart phone-based fuzzy modeling to examine facial skin quality,” in Proc. of IEEE Int. Conf. on System Science and Engineering, Budapest, Hungary, 2013, pp. 69-74.

[9] M.-J. Hsiao, Y.-P. Huang, T. Tsai, and T.-W. Chiang, “A general and effective two-stage ap-proach for region-based image retrieval,” Life Sci-ence Journal, vol. 7, no. 3, pp. 73-80, 2010.

[10] K. S. Tan and N. A. M. Isa, “Color image segmen-tation using histogram thresholding– fuzzy c-means hybrid approach,” Pattern Recognition, vol. 44, no. 1, pp. 1-15, 2011.

[11] M. Arevalillo-Herráeza, F. J. Ferri, and J. Domingo, “A naive relevance feedback model for con-tent-based image retrieval using multiple similarity measures,” Pattern Recognition, vol. 43, no. 3, pp. 619-629, 2010.

[12] J. Li and J. Z. Wang, “Real-time computerized an-notation of pictures,” IEEE Trans. on Pattern Analysis and Machine Intelligence, vol. 30, no. 6, pp. 985-1002, 2008.

[13] Y. Chen, J. Z. Wang, and R. Krovetz, “CLUE: Cluster-based retrieval of images by unsupervised learning,” IEEE Trans. on Image Processing, vol. 14, no. 8, pp. 1187-1201, 2005.

[14] Y. Chen and J. Z. Wang, “Image categorization by learning and reasoning with regions,” Journal of Machine Learning Research, vol. 5, pp. 913-939, 2004.

[15] S.-C. Cheng, M.-Y. Chen, H.-Y. Chang, and T.-C. Chou, “Semantic-based facial expression recogni-tion using analytical hierarchy process,” Expert Systems with Applications, vol. 33, no. 1, pp. 86-95, 2007.

[16] M. M. Rahman, P. Bhattacharya, and B. C. Desai, “A framework for medical image retrieval using machine learning and statistical similarity matching

techniques with relevance feedback,” IEEE Trans. on Information Technology in Biomedicine, vol. 11, no. 1, pp. 58-69, 2007.

[17] Z. Stejic, Y. Takama, and K. Hirota, “Variants of evolutionary learning for interactive image re-trieval,” Soft Computing, vol. 11, no. 7, pp. 669-678, 2007.

[18] D. C. Tao, X. L. Li, and S. J. Maybank, “Negative samples analysis in relevance feedback,” IEEE Trans. on Knowledge and Data Engineering, vol. 19, no. 4, pp. 568-580, 2007.

[19] J. Y. Chai, C. Zhang, and R. Jin, “An empirical investigation of user term feedback in text-based targeted image search,” ACM Trans. on Informa-tion Systems, vol. 25, no. 1, pp. 1-25, 2007.

[20] K. Wu and K.-H. Yap, “Content-based image re-trieval using fuzzy perceptual feedback,” Multime-dia Tools and Applications, vol. 32, no. 3, pp. 235-251, 2007.

[21] W.-Y. Ma and B. S. Manjunath, “NETRA: a tool-box for navigating large image database,” Multi-media Systems, vol. 7, no. 3, pp. 184-198, 1999.

[22] L. Chen, J. Zou, and C. L. P. Chen, “Kernel spatial showed c-means for image segmentation,” Int. Journal of Fuzzy Systems, vol. 16, no. 1, pp. 46-56, 2014.

[23] R. C. Gonzalez and R. E. Woods, Digital Image Processing, Addision-Wesley Publishing Co., New York, U.S.A., 1992.

[24] Y.-P. Huang, Y.-T. Chang, S.-L. Hsieh, and F. E. Sandnes, “An adaptive knowledge evolution strat-egy for finding near-optimal solutions of specific problems,” Expert Systems with Applications, vol. 38, no. 4, pp. 3806-3818, 2011.

Tsun-Wei Chang is an associate professor and the dean of the Department of Computer Science and Information Engineering at De-Lin Institute of Technology, Taiwan. His research interests include intelligent systems, data mining, information retrieval and neural network. He received his Ph.D. in the Department of Computer Science and

Engineering of Tatung University, Taiwan.

Yo-Ping Huang received his Ph.D. in electrical engineering from Texas Tech University, Lubbock, TX, USA. He is currently a Professor in the Department of Electrical Engineering at National Taipei University of Technology (NTUT), Taiwan. He also serves as CEO of Joint Commission of Technological and Vocational College Admission Com-

mittee in Taiwan and Chairman of IEEE SMC Taipei Chapter.


302

He was the secretary general at NTUT and Chairman of IEEE CIS Taipei Chapter. He was professor and Dean of College of Electrical Engineering and Computer Science, Tatung Univer-sity, Taipei, before joining NTUT. His research interests in-clude medical knowledge mining, intelligent control systems, handheld device application systems design. Prof. Huang is a senior member of the IEEE and a fellow of the IET.

Frode Eika Sandnes received a B.Sc. in computer science from the University of Newcastle upon Tyne, U.K., and a Ph.D. in computer science from the University of Reading, U.K. Sandnes is a Professor in the Institute of Information Technology, Faculty of Technology, Art and Design at Oslo and Akershus University College of Applied Sciences,

Norway. He he currently serves as pro-Rector of research and internationalisation at Oslo and Akerhus University College of Applied Sciences. His research interests include error-correction, intelligent systems, human computer interaction and universal design.

a ga-based fuzzy recommender system for region … ga-based fuzzy recommender system for...

Documents