Using HTML Metadata to Retrieve Relevant Images from the World Wide

Web

Ethan V. Munson

University of Wisconsin-Milwaukee

Why is image search important?

• The Web is becoming the world’s primary information source

• Images are one of the Web’s key features• Few WWW image search engines exist currently• Using textual search engines to find images

manually is laborious

A Requirement for Web Image Search

• We need an efficient method of discovering and indexing image content.

• Two main sources of information about image content:– image processing

– associated text• text content

• markup

Related work

• QBIC (the IBM Almaden Research Center)– indexes and retrieves images according to:

– shape

– color

– texture

– object layout

– queries are formulated through visual examples – a sample image

– user provided sketches

Related work QBIC system

Related work QBIC system

Related work QBIC system

QBIC: Advantages and Disadvantages

• Advantages– well-developed visual query language

– interesting GUI

– queries are based on image appearance

• Disadvantages– works only at the primitive feature level (color, texture,

shape)

– doesn’t recognize semantics of image• very sensitive to camera viewpoint

– doesn’t scale up to the Web

Related work

• WebSeek (J. Smith & S. Chang, Columbia University)

– performs a semi-automated classification of the images• automatically extracts keywords from image file names

• computes the keyword histogram

• manually creates a subject hierarchy

• manually maps the images into the subject hierarchy

– User can• browse the categories

• search the categories by keyword

• search the database using image features – color content

Webseek: Advantages/Disadvantages

• Advantages– Large index of Web images

– Supports both text and image search

• Disadvantages– Not clear that database can scale up

• Manual categorization is very expensive

– Relevance feedback mechanism is computationally expensive

Related work

• WebSeer (M. Swain et al., The University of Chicago) – uses associated text and markup to supplement

information derived from analyzing image content

– uses multiple kinds of metadata• image file names

• alternate text

• text of a hyperlink

– decides which images are photographs, portraits, or computer generated drawing

– research emphasized categorization, not metadata-based search

Why seek new image retrieval methods?

• The number of WWW documents is growing rapidly and constantly changing

• We need fast and efficient methods for finding images

• Image processing is– complex

– computationally expensive

– limited (misses true image semantics)

– unnecessary

Research Goals

• Show that images can be found using HTML “metadata”– textual content

– HTML tag structure

– attribute values

• Determine which metadata features are the best clues to image content

The URL Filter• assembles a list of URLs from the results returned by Alta

Vista– parses the first page returned by Alta Vista

– follows the URLs of results pages, retrieves these pages, and parses them

– extracts list of URLs from the results pages

The Crawler• retrieves the pages

• saves each page’s HTML source code in a separate file

“Tidy”• converts arbitrary and probably ill-formed HTML into

XHTML

XHTML Parser• parses an XHTML document

• builds an XHTML parse tree

The Document Analyzer

• scans the parse tree for image URLs– an image URL appears in either an image or anchor

element

• converts relative URLs into absolute URLs• uses various heuristics to determine which URLs

point to relevant images

Search Strategies

• Image’s file name

• Textual content of the TITLE element

• Value of the ALT attribute of IMG elements

• Textual content of anchor elements

• Value of the title attribute of anchor elements

• Textual content of the paragraph surrounding an image

• Textual content of any paragraph located within the same center element as the image

• Textual content of heading elements

Image Retrieval Experiment

Experimental Questions

• Which HTML features reveal the most information about image? – Do particular patterns of HTML structure carry useful

information?

• Do image search results depend on the type of query?

Informal Experiments

• Conducted extensive informal testing– to check software correctness

– to investigate possible metadata clues

– to determine rules for filtering out images based on size• images smaller than 65 pixels in either dimension almost never

contained useful content

• reduced the number of images we had to classify

Metadata Clues

1 Image’s file name

2 Textual content of the TITLE element

3 Value of the ALT attribute of IMG elements

4 Textual content of anchor elements

5 Value of the title attribute of anchor elements

6 Textual content of the paragraph surrounding an image

7 Textual content of any paragraph located within the same center element as the image

8 Textual content of heading elements

Query Categories• Famous people

“Gorbachev”, “Yeltsin”, and “Streisand”

• Non-famous people“Yelena” and “Ekaterina”

• Famous places “Paris” and “London”

• Less-famous places

“Bremen” and “Spokane”

• Phenomena“Explosion”, “Sunset”, and “Hurricane”

Experimental Procedure

• For each of the 12 queries– Alta Vista returned 200 URLs (20 groups of 10)

– We used first, middle, and last groups (30 URLs)

– Downloaded pages and all images on pages• excluding small images (< 65 pixels in either dimension)

• 276 pages and 1578 images were accessible

– Manually determined relevance of each image

– Used our system to determine the effectiveness of each of the 8 metadata clue

• standard information retrieval measures: precision and recall

Information Retrieval Measures

• Recall = B/(A + B)– Warning: our study does not really test recall

• We need a controlled sample of the Web, but instead, we are using Alta Vista’s biased sample

• Precision = B/(B + D)

Relevant, not retrieved A

Relevant, retrieved B

Nonrelevant, not retrieved C

Nonrelevant, retrieved D

Recall Table

Precision Table

Key Results

• Image file name has poor recall for people’s names and excellent recall for less-famous cities

• Famous names have poorer precision than non-famous and place names

Image file name

Textual content of TITLE

Value of ALT

Overall percent of recallOverall percent of precision

43.5 % 62.1 % 13.7 %

70.7 % 58.2 % 87.5 %

Problems with this study

• This is a single, small study– results must be replicated

• No standard corpus for testing Web image search– our “recall” results are not reliable or truly sound

• Our choice of tools may bias our results– Title tag may be important only because Alta Vista

considers it important

– Tidy may remove some clues• What is the structure of “<P> Text <IMG>”?

– Analysis of “header” clue is questionable

Body Body

P IMG P

IMG

Conclusion

• Existing content-based image retrieval systems are not good models for Web image search

• HTML metadata is useful for Web image search– Image file name and document title are most useful

– Alternate text is extremely precise, when present

• HTML metadata should provide faster image search than image processing approaches– no need to download and analyze images

– can take advantage of existing search engines

Using HTML Metadata to Retrieve Relevant Images from the Web

Ethan V. Munson

Dept. of Electrical Engineering & Computer Science

University of Wisconsin - Milwaukee

munson@cs.uwm.edu

http://www.cs.uwm.edu/~multimedia