Content Extraction from HTML Documents

A. Rahman H. Alam R. Hartono

Document Analysis and Recognition Team (DART)

BCL Computers Inc. Santa Clara, Calif, USA

Current need?

• Viewing website using small screen handheld devices

• Since web sites are written using HTML codes, we need to translate these to systems that the wireless devices can support.

Current Solutions

• Handcrafting: – Custom Web Sites are typically crafted by

hand by a set of content experts

• Transcoding:– Thranscoding replaces HTML tags with

suitable device specific tags (HDML, WML etc)

Handcrafting

• Automation– Use of XML.

• There is no standard XML tagset (Document Type Definition – DTD) in use by vendors.

• XML has been available to web designers for the last 10 years. Examination of websites shows little use of document structural elements.

– Web masters see themselves as artists rather than programmers.

– XML may meet the same fate as SGML, an earlier attempt to create structured documents.

Handcrafting

• Take an existing website and make it available to wireless access. Aether Systems, Mshift and 2Roam currently offer these types of solutions.

• Use a proprietary graphical interface to ease the development of wireless applications from scratch. Covigo and iConverse offer these type of solutions.

• Let the user do all coding in languages such as C++ or Java. ThinAirApps offers this type of solution.

Handcrafting

• Labor intensive

• Expensive.

• Typically less than 1% of a web site gets converted to wireless content.

Transcoding

• Most web pages have a loose repeating visual structure. The wireless user gets the same repeating information with every screen

• Browsing is an unfriendly experience • Transcoding sends all the information to the

wireless device, making it substantially slow on the wireless network

Transcoding

• Transcoding was introduced in Japan during 1999-2000. It was widely rejected by the Japanese users.

• Recently, Google and Pixo introduced this solution for the US market, but have so far failed to attract attention of end users.

The Alternate Solution

• Separate the content into smaller segments

• Generate a summary of these segments

• Prioritize these summaries from individual segments

• Put together to form a summary of the overall document

Steps to Content Extraction

• Structural analysis: Understanding the relationship of the various segments with the document

• Decomposition: Breakdown on these segments into operational units

• Contextual Analysis: Employment of context to revise the segmentation

(Continued=>)

Steps to Content Extraction (Continued)

• Labeling => Segment Summary: Extraction of a low level summary of the segment

• Priority: Estimating importance of these segments

• Table of Content (TOC) => Document Summary: Putting together a summary of the document

Content Extraction

• Proximity Analysis: Relational analysis of content between segments

• Content Classification: callification into various types, i.e. [stories], [navigation], [links], [images], [forms] etc.

• Relationship Analysis– Contextual grammar (Natural Language)– Knowledge modes– Information retrieval techniques

Content Extraction: Why do we need it?

• Viewing any website: Any solution to web browsing has to be universal

• High network access: Any transformation has to be fast and on-the-fly

• Network Usage: Network traffic should increase because of these systems

(Continued=>)

Content Extraction: Why do we need it (continued)?

• Easy Configurability: Any such system should be easiliy configurable

• Rapid Deployment: Should be rapidly deployable• Non-intrusive Design: Should be possible to

transform web sites without modifying the actual web site

• Multiple Views: System Integrators should be able to create multiple views of the same site

Advantages of Content Extraction

• Displays size• Locating information • Important content can be on top• Multiple levels of abstraction can be created• The browsing can use a demand-driven model• Faster download• More efficient use of small display areas• Mapping of the importance of content from the

original document

Supported Devices and Formats

• PDAs (HTML3.2)

• Cell phones – USA/Europe:

• WAP

– Japan• iMode (NTT DoCoMo)

• J-Sky (J-Phone)

• EZWeb (KDDI)

Conclusion• Content from web documents can be extracted

based on the – HTML structure– Proximity analysis– Logical relationship analysis– Information retrieval techniques

• Content can be used effectively to summarize web documents– Better option compared to handcrafting or transcoding – Produces faster browsing experience