1

Evaluations in information retrieval

2

Evaluations in information retrieval:summary

• The following gives an overview of approaches that are applied to assess the quality of

»information retrieval systems, and more concretely of search systems

»the resulting set of records obtained after performing a query in an information retrieval system

• Note: This should not be confused with assessing the quality and value of the content of an information source.

3

Evaluations in information retrieval:introduction

• The quality of the results, the outcome of any search using any retrieval system depends on many components / factors.

• These components can be evaluated and modified to increase the quality of the results more or less independently.

4

Evaluations in information retrieval:important factors

• The information retrieval system ( = contents + system)

• The user of the retrieval system and the search strategy applied to the system

Result of a searchResult of a search

5

Evaluations in information retrieval:why? (Part 1)

• To study the differences in outcome/results when a component of a retrieval system is changed, such as

»the user interface

»the retrieval algorithm

»addition by the database of uncontrolled, natural language keywords versus keywords selected from a more rigid, controlled vocabulary

6

Evaluations in information retrieval:why? (Part 2)

• To study the differences in outcome/results when a search strategy is changed.

• To study the differences in outcome/results when searches are performed by different groups of users, such as

»children versus adults

»inexperienced users versus more experienced, professional information intermediaries/professionals

7

Evaluations in information retrieval: the simple Boolean model

Boolean model: # items in database = # items selected + # items not selected

# Items selected =

# relevant items + # irrelevant items

Relevant Yes

1In

IrrelevantNo0

Out

8

Relevant items in a database: scheme

Dependent on the aims, independent of the search strategy

Relevant items!(In most cases the small subset)

Irrelevant / NOT relevant items(In most cases the large subset)

9

Selecting relevant items by searching a database: scheme

Dependent on the aims, independent of the search strategy

Selected and relevant!

Selectedbut not relevant

Not selected but relevant

Not selectedand not relevant

Dependent on the aims and dependent on the search strategy

10

Recall: definition and meaning

• Definition: # Of selected relevant items “Recall” = ------------------------------------------------- * 100% Total # of relevant items in database

• Aim: high recall

• Problem: in most practical cases, the total # of relevant items in a database cannot be measured.

11

Selecting relevant items: recall

Selected and relevant!

Selectedbut not relevant

Not selected but relevant

Not selectedand not relevant

14

!? Question !? Task !? Problem !?

How can you change your search strategy

to increase the recall?

How can you change your search strategy

to increase the recall?

15

Precision: definition and meaning

• Definition:

# Of selected relevant items“Precision” = --------------------------------------- * 100% Total # of selected items

• Aim: high precision

16

Selecting relevant items: precision

Selected and relevant!

Selectedbut not relevant

Not selected but relevant

Not selectedand not relevant

17

!? Question !? Task !? Problem !?

How can you change your search strategy

to increase the precision?

How can you change your search strategy

to increase the precision?

18

!? Question !? Task !? Problem !?

When you change your search strategy to increase the precision,

which consequence do you expect for the recall, in most cases?

When you change your search strategy to increase the precision,

which consequence do you expect for the recall, in most cases?

19

Relation between recall and precision of searches

100%

Recall

0 0 Precision 100%

Ideal = Impossibleto reach in most systems

Ideal = Impossibleto reach in most systems

Search (results)

20

!? Question !? Task !? Problem !?

Indicate on the figure that a user improves a search.

Indicate on the figure that a user improves a search.

21

!? Question !? Task !? Problem !?

Indicate on the figure that a database producer

and / or the retrieval system improves the retrieval quality.

Indicate on the figure that a database producer

and / or the retrieval system improves the retrieval quality.

22

!? Question !? Task !? Problem !?

Indicate the relation between the recall and precision

in a classical information retrieval system in the form of a figure. Indicate in that figure

a good and a bad search.

Indicate the relation between the recall and precision

in a classical information retrieval system in the form of a figure. Indicate in that figure

a good and a bad search.

23

Recall and precision should be considered together

Examples:

• Increase in retrieved number of relevant items may be accompanied by an impractical decrease in precision.

• Precision of a search close to 100% may NOT be ideal, because the recall of the search may be too low. Make search / query broader to increase recall !

• Poor (low) precision is more noticeable than bad (low) recall.

24

Evaluation in the case of systems offering relevance ranking

• Many modern information retrieval systems offer output with relevance ranking.

• This is more complicated than simple Boolean retrieval, and the simple concepts of recall and precision cannot be applied.

• To compare retrieval systems or search strategies, decide to consider for comparison a particular number of items ranked highest in each output.This brings us to for instance: “first-20 precision”.

25

!? Question !? Task !? Problem !?

Give examples of retrieval systems that offer relevance ranking.

Give examples of retrieval systems that offer relevance ranking.