relevance feedback based on parameter estimation of target distribution k. c. sia and irwin king...

Relevance Feedback Relevance Feedback based on Parameter based on Parameter Estimation of Target Estimation of Target

DistributionDistribution

K. C. SiaK. C. Sia and Irwin King and Irwin KingDepartment of Computer Science & EngineeringDepartment of Computer Science & Engineering

The Chinese University of Hong KongThe Chinese University of Hong Kong

15 May15 May

IJCNN 2002IJCNN 2002

Relevance Feedback Based on Parameter Estimation of Target Distribution

AgendaAgenda

Introduction to content based image Introduction to content based image retrieval (CBIR) and relevance retrieval (CBIR) and relevance feedback (RF)feedback (RF)

Former approachesFormer approaches Tackling the problemTackling the problem

Parameter estimation of target Parameter estimation of target distributiondistribution

ExperimentsExperiments Future works and conclusionFuture works and conclusion


Content Based Image Content Based Image RetrievalRetrieval

How to represent an image?How to represent an image? Feature extractionFeature extraction

Colour histogram Colour histogram (RGB)(RGB) Co-occurrence matrix texture analysisCo-occurrence matrix texture analysis Shape representationShape representation

Feature vectorFeature vector Map images to points in hyper-spaceMap images to points in hyper-space Similarity is based on distance Similarity is based on distance

measuremeasure


Feature Extraction ModelFeature Extraction Model

R

B

G


Relevance FeedbackRelevance Feedback

Relevance feedbackRelevance feedback Architecture to capture user’s target Architecture to capture user’s target

of searchof search Learning processLearning process

Two stepsTwo steps FeedbackFeedback – how to learn from the – how to learn from the

user’s relevance feedbackuser’s relevance feedback DisplayDisplay – how to select the next set of – how to select the next set of

documents and present to userdocuments and present to user


1st iteration

UserFeedback

Display

2nd iteration

Display

UserFeedback

Estimation &Display selection

Feedbackto system


Former ApproachesFormer Approaches

Multimedia Analysis and Retrieval System Multimedia Analysis and Retrieval System (MARS)(MARS) Yong Rui et al. Relevance feedback: A powerful tool for Yong Rui et al. Relevance feedback: A powerful tool for

interactive content-based image retrieval. - 1998interactive content-based image retrieval. - 1998 Using weight to capture user’s preferenceUsing weight to capture user’s preference

Pic-HunterPic-Hunter Ingemar J. Cox et al. The Bayesian image retrieval Ingemar J. Cox et al. The Bayesian image retrieval

system, pichunter, theory, implementation, and system, pichunter, theory, implementation, and psychophysical experiments. - 2000psychophysical experiments. - 2000

Images are associated with a probability Images are associated with a probability being the user’s targetbeing the user’s target

Bayesian learningBayesian learning


ComparisonComparison

MARSMARS Pic-HunterPic-Hunter Our approachOur approach

Capturing Capturing user’s target user’s target of searchof search

Weight on Weight on different feature different feature and dimensionand dimension

Probability Probability associated associated with imageswith images

Estimated Estimated parameter of parameter of target cluster target cluster

UpdatingUpdating Counting and Counting and variancevariance

Bayes’ ruleBayes’ rule EM algorithmEM algorithm

DisplayDisplay Most likelyMost likely Maximum Maximum Entropy Entropy PrinciplePrinciple

Maximum Maximum Entropy Entropy PrinciplePrinciple


The ModelThe Model Feature ExtractionFeature Extraction

II - raw image data - raw image data - set of feature extraction method- set of feature extraction method ff - feature extraction operation - feature extraction operation

Images Images data point in hyper-space data point in hyper-space (R(Rdd)) Problem scope is narrowed down to a particular Problem scope is narrowed down to a particular

featurefeature

FeedbackFeedback


Inconsistence in FeedbackInconsistence in Feedback

User tells liesUser tells lies

Too many false positive or false Too many false positive or false negativenegative

Conflict of feedback in each Conflict of feedback in each iteration by careless mistakeiteration by careless mistake


Resolving ConflictsResolving Conflicts

How to deal with inconsistent user How to deal with inconsistent user feedback?feedback? Maintain a relevance measure for Maintain a relevance measure for

each data pointseach data points Relevance measure > 0 counted as Relevance measure > 0 counted as

relevant and use in estimationrelevant and use in estimation


Estimating Target Estimating Target DistributionDistribution

User’s target is a cluster User’s target is a cluster Assume it follows a Gaussian Assume it follows a Gaussian

distributiondistribution Model a distribution that fits Model a distribution that fits

the relevant data pointsthe relevant data points Based on the parameterBased on the parameter

of distribution, systemof distribution, systemlearns what user wantslearns what user wants

Data points selected as relevant

Red


Expectation MaximizationExpectation Maximization

Fitting a Gaussian distribution function using Fitting a Gaussian distribution function using feedback data pointsfeedback data points By expectation maximizationBy expectation maximization

Distribution represent user’s targetDistribution represent user’s target Expectation function match the display modelExpectation function match the display model


Updating ParametersUpdating Parameters

Estimated mean is the averageEstimated mean is the average Estimated variance by Estimated variance by

differentiationdifferentiation Iterative approachIterative approach

DisplayDisplay


Maximum Entropy DisplayMaximum Entropy Display

Why maximum entropy display?Why maximum entropy display?

ReasonReason: fully utilize information : fully utilize information contained in user feedback to reduce contained in user feedback to reduce number of feedback iterationnumber of feedback iteration

ResultResult: near boundary images will be : near boundary images will be selected to fine tune parametersselected to fine tune parameters


Maximum Entropy DisplayMaximum Entropy Display

How to simulate maximumHow to simulate maximumentropy display in ourentropy display in ourmodel?model? Data points 1.18 Data points 1.18 away away

from from are selected are selected Why 1.18?Why 1.18?

2P(2P(+1.18+1.18)=P()=P())

Querytargetclustercenter

Selectedby knnsearch

Selectedby Max.Entropy


ExperimentExperiment

Synthetic data generated by MatlabSynthetic data generated by Matlab

Mixture of GaussiansMixture of Gaussians Class label of data points shown for Class label of data points shown for

reference to give feedbackreference to give feedback Dose it works and works better?Dose it works and works better?


ConvergenceConvergence

Is the estimated parameter (mean Is the estimated parameter (mean and variance) converge to the and variance) converge to the actual parameter of target actual parameter of target distribution?distribution?

Is the maximum entropy display Is the maximum entropy display correctly done?correctly done?


Estimated mean RMS error along each iteration

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

1 2 3 4 5 6 7 8 9 10

iteration

RM

S e

rro

r o

f es

tim

ated

mea

n 4 dimension

6 dimension

8 dimension


Estimated standard deviation RMS error along iteration

0

0.05

0.1

0.15

0.2

0.25

1 2 3 4 5 6 7 8 9 10

iteration

RM

S e

rro

r o

f es

tim

ated

sta

nd

ard

dev

iati

on

4 dimension

6 dimension

8 dimension


No. of feedback given along each iteration

0

2

4

6

8

10

12

14

16

18

1 2 3 4 5 6 7 8 9

iteration

No

. of

feed

bac

k g

iven

4 dimension

6 dimension

8 dimension


PerformancePerformance

Compares to Rui’s intra-weight Compares to Rui’s intra-weight updating modelupdating model Nearest neighbour search performed Nearest neighbour search performed

after several feedbacks (6-7 after several feedbacks (6-7 iterations)iterations)

Data points outside 2 Data points outside 2 are discarded are discarded in our algorithmin our algorithm

Precision-Recall graphPrecision-Recall graph


Precision vs Recall

0

0.2

0.4

0.6

0.8

1

1.2

0 0.05 0.1 0.15 0.2 0.25 0.3

Recall

Pre

cis

ion

Expectation Maximization

Rui's weight updating


Precision vs Recall

0

0.2

0.4

0.6

0.8

1

1.2

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35

Recall

Pre

cis

ion




Precision vs Recall

0

0.2

0.4

0.6

0.8

1

1.2

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35 0.4 0.45 0.5

Recall

Prec

isio

n




Precision vs Recall

0

0.2

0.4

0.6

0.8

1

1.2

0 0.05 0.1 0.15 0.2 0.25

Recall

Pre

cis

ion




Precision vs Recall

0

0.2

0.4

0.6

0.8

1

1.2

0 0.05 0.1 0.15 0.2 0.25 0.3 0.35

Recall

Pre

cis

ion




Future WorksFuture Works

Modification to learn from Modification to learn from information contained in non-information contained in non-relevant setrelevant set

To capture correlation in different To capture correlation in different featuresfeatures

Apply in CBIR system for Apply in CBIR system for performance measurementperformance measurement


ConclusionConclusion

Proposed an approach to interpret Proposed an approach to interpret the feedback information from the feedback information from user and learn his target of searchuser and learn his target of search

Compares our approach with Rui’s Compares our approach with Rui’s intra-weight updating methodintra-weight updating method

ENDEND

Presentation file available atPresentation file available at http://www.cse.cuhk.edu.hk/~kcsia/research/http://www.cse.cuhk.edu.hk/~kcsia/research/

relevance feedback based on parameter estimation of target distribution k. c. sia and irwin king...

Documents

steps feedback

image retrieval cbir

system slide

user slide

conclusion slide

user display

retrieval system mars

distance measure slide