Relevance Feedback Content-Based Image Retrieval Using Query Distribution Estimation Based on

Maximum Entropy Principle

Irwin King and Zhong Jin

Nov. 2001

1. Introduction

Content-Based Image retrieval (CBIR) :

The selection of images from a collection via primitive visual features representing color, shape, and texture extracted from images themselves.

Successful CBIR systems require the integration of various techniques in the fields of :

• Pattern Recognition (PR)• Digital Image Processing (DIP)• Information Retrieval (IR)

1. Introduction (Cont ’d)

Relevance feedback (RF) :

an iterative and interactive process for query reformulation based on user's feedback.

• query moving technique• similarity function re-weighting technique

RF techniques include mainly:

1. Introduction (Cont ’d)

Problems:

Since the retrievals under the commonly used nearest-neighbor rule cannot reflect the query distribution function properly , most of relevance feedback techniques may fail under the following assumption:

• The number of relevant retrievals is small• The number of iterations is required to be small

2. Background Review

RF technique in CBIR system can be regarded as a form of two-stage automatic learning for the unknown query distribution function:

• Estimate the query distribution function by using the Expectation-Maximization (EM) algorithm or by the classical statistics.

• Generate the inquiries to be returned to the user, where the nearest-neighbor rule is commonly used.

2. Background Review (Cont ’d)

Limitation of estimation theories:

• EM has its limitations in CBIR Because of a small number of labeled data in RF.

• Only relevant informaton can be utilized by classical statistical theory.

2. Background Review (Cont ’d)

Limitation of the nearest-neighbor rule:

• the retrievals generated cannot wholly reflect the Query Distribution Function (QDF) because the underlying QDF may not be isotropic in nature.

Note: QDF is the statistical distribution function deformed by all the images similar to the given query image in high dimensional feature space.

2. Background Review (Cont ’d)

Shannon’s Entropy

Maximum Entropy Principle (MEP): To obtain estimations by determining a probability distribution

associated with a random variable over a discrete space which has the greatest entropy subject to constraints on the expectations of a given set of functions of the variable.

ki

iiik ppppH

11 log),,(

The Maximum Entropy (MAXENT) solution with no bias (or constraints) is

),,1(1

kik

pi

2. Background Review (Cont ’d)

Some work on IR by MEP:

• In the early 80's, Cooper et al. made a strong case for applying the maximum entropy approach to the problems of information retrieval.

• Kantor extended the analysis of the MEP in the context of information retrieval.

• Recently, Greiff and Ponte took a fresh look at modeling approaches to information retrieval and analyzed classical probabilistic IR models in light of the MEP.

3. Proposed Framework

Our novel framework for image retrieval includes the following stages:

• Estimation stage -- Estimate the query mean and the query covariance matrix by using accumulative relevance retrieval information and irrelevant retrieval information.

• Generation stage -- Generate a set of inquiries for relevance selection based on MEP.

3. Proposed Framework (Cont ’d)

Estimation stage

'

11

1

rt

T

trt XX

T

T

trtXT 1

1

When the number T of relevant retrieval is less than the dimension M of the feature space, it is assumed that 22

1 ,, Mdiag

When T=1, an estimation can be givenby an equal-probability constrain 6745.0

ii

d

2

1 iiiiii dyPdyP

3. Proposed Framework (Cont ’d)

Generation stage

• For K number of retrievals, K+1 points can be determined

according to the following equal-probability conditions:

),,1(1

22

1 21

22

1

KiK

dzezM

zMz

zM

i

i

• all similar images in the database can be divided in the following K subsets:

),,1(,1 KiSIzIXZzIS iii

where IXIXIXZ 1'

4. Experiments and Analysis

DatabaseThere are 1,400 trademark images with 128*128. Here are ten samples:

4. Experiments and Analysis (Cont ’d)

Here are 10 deformation transformations:

4. Experiments and Analysis (Cont ’d)

100 Test Images:

4. Experiments and Analysis (Cont ’d)

Feature Extraction: 7 dimensional invariant moment

4. Experiments and Analysis (Cont ’d)

Experimental Aim: to evaluate the efficiency of the proposed generation

stage, Generation MAXENT.

The retrieval performance is measured using the following Average Retrieval Precision (ARP):

%100K

RetrievedRelevantARP

where K=10

4. Experiments and Analysis (Cont ’d)

In order to compare Generation MAXENT with Euclidean distance and Mahalanobis distance, a set of three-step experiments are designed as follows:

Step 1: For a query image, return K retrievals by Euclidean distance.

Step 2: Perform the estimation stage and return retrievals by

Euclidean distance, Mahalanobis distance, and

Generation MAXENT respectively.

Step 3: Perform the estimation stage and return K retrievals by using the Mahalanobis distance;

4. Experiments and Analysis (Cont ’d)

Experimental results:

Http://www.cse.cuhk.edu.hk/~miplab/MAXENT

4. Experiments and Analysis (Cont ’d)

Result analysis:

• According to the ARP's in Step 3 in Table 1, the proposed generation stage Generation MAXENT outperforms the commonly used Euclidean distance and Mahalanobis distance

• The proposed generation stage Generation Based On MAXENT aims to retrieve image samples which can reflect the query distribution function. This is the reason why the ARP of MAXENT in Step 2 in Table 1 is lower than those of Euclidean distance and Mahalanobis distance.

5. Conclusion

Novel two-stage relevance feedback framework for content-based image retrieval based on query estimation and the Maximum Entropy Principle is shown to be succeful in improving accuracy and speed on a trademark image database.

Future work: to overcome the difficulty in image retrieval for high-dimensional features in large image databases

Acknowledgment

This paper is supported in part by an Earmarked Grant from the Hong Kong Research Grants Council #CUHK4407/99E.