using html textual and structural data for web image search cheng thao, ethan munson, jim dabrowski,...
TRANSCRIPT
Using HTML Textual and Structural Data Using HTML Textual and Structural Data for Web Image Searchfor Web Image Search
Cheng Thao, Ethan Munson,Cheng Thao, Ethan Munson,
Jim Dabrowski, Nikolas D. Bohne
University of Wisconsin-MilwaukeeUniversity of Wisconsin-Milwaukee
Which image is George Bush or has George Bush?
Which images are similar to this image?
<tr><td width=400 bgcolor=ffffff><center><FONT FACE"Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/bill%20cosby.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Bill Cosby</center></FONT></em></td><td width=210 bgcolor=ffffff><center><FONT FACE="Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/betty%20white.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Betty White</center></FONT></em></td></tr><tr><td width=400 bgcolor=ffffff><center><FONT FACE"Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/tom%20brokaw.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Tom Brokaw</center></FONT></em></td><td width=210 bgcolor=ffffff><center><FONT FACE="Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/george%20bush.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Pres. George Bush</center></FONT></em></td></tr><tr><td width=400 bgcolor=ffffff><center><FONT FACE"Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/ed%20mcmahon.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Ed McMahon</center></FONT></em></td><td width=210 bgcolor=ffffff><center><FONT FACE="Arial, Helvetica" SIZE=-1><IMG SRC="http://www.gopbi.com/community/groups/bigband/images/bob%20barker.jpg" alt="" BORDER=0 VSPACE=3 HSPACE=3></FONT></center><em><center><FONT FACE="Arial, Helvetica" SIZE=-1>Bob Barker</center></FONT></em></td></tr>
Does the HTML source tell which images is George Bush?
IntroductionIntroduction
-image search is difficult
- performance is slow
- image identification is a complex, inaccurate task
-most research on image search has emphasized analysis of image content
-few Web image search engines
- commercial: Alta Vista, Google
- research: WebSeek
-little research in textual image search
HTML overview
-HTML document composed of:-head
-title-meta
-body-paragraph, -table,-text,-link,-image, …
<html> <head> <title>Sample HTML</title> <meta keywords="html, html elements"> <meta description="showing a simple html and some html elements"> </head> <body>
<h1>HTML overiew</h1>
<p> first paragraph</p>
<table border="1"> <caption> Simple Table</caption> <tr><td>1<td>2<td>3 <tr><td>4<td>5<td>6 <tr><td>7<td>8<td>9 </table>
Here is a photo of George Bush. <br> <img src="g-bush.jpg">
</body></html>
Previous work - Previous work - Yelena Tsymbalenko
-studied HTML constructs and determine what can be used in image search.
-found the following to be effective- title of the page- image filename- image alt attribute
Research GoalsResearch Goals
-What HTML features make good clues to the content of images?
- Structural features (document, table)
- File names or URLs
- Formatting of material (bold, heading)
- How can clues be combined into a single relevance rating?
Image Search Study ProcessImage Search Study Process
-Downloading pages with matching text- Use existing search engine to identify matches- These pages provide a corpus of images- We download pages so that our corpus remains static
-Download acts as a snap shot
-Clue extraction-Analyze each page in corpus for all possible clues to image content
-Human relevance ratings-human rates if an image is relevant to the query-Statistical analysis to find clue-based relevance functions
Downloading Software
query
URLs
Process: Downloading Web PagesProcess: Downloading Web Pages
queriesSearch Engine
Web Pages
<query>
images
Web pages and images are saved to local disk.
<query><query>
<engine><engine> 11 </engine></engine>
<method><method> 22 </method></method>
<word><word> GeorgeGeorge </word></word>
<word><word> BushBush </word></word>
</query></query>
<query><query>
<engine><engine> 22 </engine></engine>
<method><method> 33 </method></method>
<word><word> BillBill </word></word>
<word><word> ClintonClinton </word></word>
</query></query>
Design: Queries in XMLDesign: Queries in XML
Multiple queries are stored in an XML fileMultiple queries are stored in an XML fileEngine:
1=Altavista, 2=Excite, 3=Hotbot, 4=Google
Method:
1 =or, 2 = and, 3 = expression
Search for George Bush using Alta Vista and must have all the words
Search for Bill Clinton using Hotbot and search for exact expression
Process: Clue ExtractionProcess: Clue Extraction
Extraction Software
cluesqueries<query>
Clues Extraction Software clues
Data to be analyzed
• For each page– Query used to find page– Source URL– For each image
• Source URL• Attributes• Position in document
– For each clue• Whether clue feature occurs in document at all• If feature occurs with text matching the query
– Position in document for each occurrence
Relevant Rating Software
Query & image
Relevant/not
Process: Relevance RatingProcess: Relevance Rating
queriesHuman<query>
Presents images from each query to the user from the database, and record the human relevance rating back to the database.
Clues : globalClues : global
Global Clues - clues that apply every image on the page
- filename of page
- path of page
- host of page
- title element of the web page
- keywords found in meta element
- description found in meta element
Why do we break the URL into three clues?
Different parts of the URL contributes different relevance factor to the overall relevance of the image in that page.
Clues : globalClues : global
<HEAD><TITLE>Apple</TITLE>
<META NAME="keywords" CONTENT="Apple Computer, Power Macintosh, PowerBook, AppleWorks, WebObjects, iMovie, QuickTime, Desktop Movies, Software, Operating Systems, Mac OS, iMac, iBook">
<META NAME="Description" CONTENT="Visit www.apple.com for the latest news, the hottest products, and technical support resources from Apple Computer, Inc.">
<META HTTP-EQUIV="Expires" CONTENT="Fri, 26 Mar 1999 23:59:59 GMT"><META NAME="Date.Modified" CONTENT="19992109">
</HEAD>
Clues : image fileClues : image file
Image file properties - external properties
- filename
- path
- host
An image can be from another host, and have different paths.
Clues : common attributesClues : common attributes
Elements have common attributes
-title - describe what the element is
-id - used in identifying the element
-name - same as id, older HTML
Clues that use these attributes:
link, image, object, table, cell, row
Clues: Image ContainerClues: Image Container
Link to an image <a>
- text enclosed within the link element
Embed image element <img>
-alt attribute (usually describes what the image is)
Object element <object>
-text that enclosed within the object element
Clues: tableClues: table
Table (<table>):
- summary attribute - describes the table content
- caption - describes table content
- row heading
- row
- column heading
- column
- cell
- neighboring cells (above, below, right, left)
Clues: tableClues: table
Clues: tableClues: table
Clues: tableClues: table
Clues: tableClues: table
Clues: tableClues: table
Clues: tableClues: table
Clues: tableClues: table
Clues: headingsClues: headings
Heading elements(h1, h2, ..h6)
- headings above image
- headings below image
<h1>header above image</h1>
<img src=“sample.jpg”>
<h2>header below image</h2>
Heading can indicate a topic and images below the heading maybe relate to the heading. Some use headings as caption above images, and sometimes below images. Some headers are used where fonts should be used or bold should be used.
In this photo, the heading comes after the image. Often if it is used as a topic, it usually comes before the image. But some images have heading as caption below the images.
Clues : textClues : text
Emphasized text elements
- bold
- italic
- underline
- strong
- emphasis
- big
Body text - text that surrounds the image
- distance
Current Project StatusCurrent Project Status
- Prototype download and clue extraction software nearly complete
- now testing implementation
- data (without human relevance ratings) in early November
- Recruiting students to build on-line relevance rating system
- hope to get students outside lab to help with ratings via Web interface
Challenges for image search systemsChallenges for image search systems
- computing word distance from image
- Stylesheet used for presentation
- table pattern
- pattern of HTML elements usage
- CGI returned images
- structural boundaries
-patterns in Web page design
-HTML generators
Cheng Thao, [email protected]