scalable hybrid keyword search on distributed database

Scalable Hybrid Keyword Search on Distributed

Database

Jungkee KimFlorida State University

Community Grids Laboratory, Indiana University

Workshop on Autonomic Distributed Data and Storage Systems Management

(ADSM 2005)

Motivation

Internet

Where is the

Information?

Outline

Two Typical Search ParadigmsProblems of Current Search ApproachesLocal Hybrid Keyword SearchHybrid Search on Distributed Databases

Two Typical Search Paradigms

Searching over structured data

Relational Databases

Searching over unstructured data

Information Retrieval

Internet Environment

Semistructured Data – XML

Keyword Search in DB

Web Search Engines – Technologies from Information Retrieval

Hybrid Keyword Search ?

Current Approaches – Keyword-only Search

Web Search Engines Web crawlers visit Web pages and collect the

keyword based text indexes. Fast information retrieval

Keyword Search in databases Web integration on legacy DBMS Dynamic Web publication through embedded

DB Easy to use without knowledge of DB schema

Problems of Current Approaches – Keyword-based

Web Search Engines Can not collect every connected resource Query results are often unrelated

Keyword Search in Databases Losing the inherent meaning of the schema Query results are not based on semantic

schema

Current Approaches – Semantic

Semantic Web Multiple relation links with directed

labeled graphs and machines can understand the relationship between different resources

Describes metadata about resources To represent the relations of the objects

on the Web; the object terms defined under a specific description – an Ontology

Problems of Current Approaches – Semantic Web

Ontology design is sophisticatedLack of unified definition Limited adoption

Our Approach

Hybrid search mechanisms –Semantic metadata + Keyword search

Semantic SolutionSemantic Web might be better than Hybrid

search Hybrid search must be better than Web search

enginesSimplicity

Hybrid search is simpler than Semantic Web

Hybrid Keyword Search Service

A search service fetches target information data against a search query.

Unstructured dataA file containing data – MS Word, PDF, PS documents

Metadata: Structured or semistructured data – XML

We utilized an XML-enabled relational DBMS and a native XML DB along with a text search library (Apache Xindice + Jakarta Lucene) to address the search against metadata and text.

How to Combine? (1)

Two entity sets and a relationship in relational DBMS

We can obtain the hybrid search result using a nested subquery

How to Combine? (2)

A hash table is used for joining search results in non-DBMS based system (Apache Xindice + Lucene)

Local Query Processing – XML (1)

XML-enabled RDB DBLP XML record (1,000 – 10,000) Non indexed matches

except year match bound by the number of matches.

Combined query time depends on # of year query results

Average XML Query Time

Local Query Processing – XML (2)

Apache Xindice DBLP XML record (1,000 – 10,000) Indexed approximate

matches for text elements in XML instances as bad as non-indexed queries

Exact matches bound by the number of matches.

Average XML Query Time

Local Query Processing – Hybrid (1)

Hybrid search query performance measurement XML-enabled RDB For 100,000 XML instances and 100,000 text documents Small result set: 4 XML and a keyword matches Large result set: 7,752 XML and 41,889 documents (3,227)

Metadata Author Year

(Nested subquery)

Year

(Hash table)

Few

Keywords

0.04

Sec.

82.9 Sec. 5.70 Sec.

Many

Keywords

0.48

Sec.

Half hour 6.96 Sec.

Local Query Processing – Hybrid (2)

Hybrid search query performance measurement Apache Xindice + Jakarta Lucene For 10,000 XML instances and 10,000 text documents Small result set: 2 XML and a keyword matches Large result set: 192 XML and 4,562 documents (41)

Discussion – Local Hybrid Search

XML-enabled RDB provides proper response except some extreme query loads. Inefficient query plan and query optimization in an

old version – better performance in a newer version

A native XML DB (Apache Xindice) had very limited scalability. (No accurate query result over 16,000 XML instances)

We will generalize hybrid search to a distributed environment.

Hybrid Search on Distributed Databases Data Independence: logically and physically

independent; the same schema – no change, data encapsulation in each machine

Network Transparency: depends on MOM or P2P framework

No replication – restricted to a computer cluster Fragment: full partition; horizontal fragmentation The query result for the distributed databases is

the collection of query results from individual database queries.

Scalable Hybrid Search Architecture on DDBS

SearchService

MessageBroker

Client

SearchService

SearchService

Subscriber for a query topic

Publisher for a temporary topic

Publisher for a query topic

Subscriber for a temporary topic

QueryMessage

QueryMessage

ResultMessage

ResultMessage

Client Client

Cooperating Broker Network

Distributed Databases based on NaradaBrokering Network

Query Processing – DDBS (1)

100,000 XML and 100,000 Documents in 8 machines – 12,500 each

Few keyword match (1-3) on 1 machine only

RDB – 0.04 Sec. for few keyword match

Avg. response time for an author exact match queryover 8 search services

Query Processing – DDBS (2)

100,000 XML and 100,000 Documents in 8 machines – 12,500 each

RDB – half hour or 6.96 Sec. (Hash table)

Avg. response time for a year match queryover 8 search services

Coupling vs. Scalability

From ICDE 2002 Tutorial

Query Propagate and Results back on a P2P Network

Peer group architecture of the P2P Search

Conclusion

We addressed the semantic loss of keyword-only search while remaining a simpler solution than the Semantic Web

Our architecture contributed a performance improvement for some queries

Extension of the scalability of Xindice XML query limited to a small size on a single machine