1 selectivity estimation for exclusive query translation in deep web data integration fangjiao jiang...

1

Selectivity Estimation for Exclusive Query Translation

in Deep Web Data Integration

Fangjiao JiangRenmin University of China

Joint work with Weiyi Meng & Xiaofeng Meng

Selectivity Estimation for Exclusive Query Translation in Deep Web Data Integration (DASFAA2009)

2

The previous Web: things are just on the surface


3

The current Web: Getting “deeper”

A great deal of information is hidden behind query forms

Deep = not accessible through search engines


4

Why is it important?

More than 10 million distinct forms


5

Why is it important?

Up to 5,000 billions dynamic result pages


6

A Key Component: Query translation

ChallengeLarge-scaleHeterogeneityAutonomy

Integrated query interface

Web database query interfaces

Query translation


7

The Problem



8

Example

√

？？


9

Related work & the Challenge

A prominent solution for selectivity estimation —— histograms [Piatetsky+, Poosala+, Ioannidis+]

Categorical attribute

Infinite-value attribute

Another solution —— random sampling [Goodman+, Haas+, Oliken+, Vitter+, Dasgupta+]

Random sampling

ChallengeSelectivity estimation of infinite-value attribute


10



11

Two Observations

There exist different correlations between different attribute pairs

the word frequency of the values on an infinite-value attribute usually has a Zipf-like distribution

Weakest

Strongest

WeakerWeaker


12


Attribute Correlation calculation for a domain Selectivity estimation for a Web database

Correlation-based sampling

Word frequency probing

Zipf equation calculation

Selectivity estimation


13

Selectivity Estimation Challenges

1. Attribute Correlation calculationFind the least correlative attribute

Discover the word rank

2. Zipf equation calculationCalculate the parameters of Zipf equation

Estimate selectivity


14

Attribute Correlation Calculation


15

GoalRandom sample Word Rank

Attribute Correlation calculation

(1)

(2)


16

Discussion on Word rank

Word rank should be computed for each attribute


17

Zipf Equation Calculation


18

Zipf equation calculation

Zipf equantion

Frequency

RankRwi Rwx Rwj Rwy

Fwi

Fwj

Fwy

Fwx

Zipf Distribution


19

The parameters of Zipf equation


20

discussion on P, p and E


21

Experiments


22

Data Sets & Evaluation Method

Data sets

Evaluation method


23

Experimental Results

The average precision of selectivity estimations is high.


24

Summary


25

Contributions

Identify the selectivity estimation problem of infinite-value attribute for exclusive query translation

Propose correlation-base sampling approach to obtain the sample as random as possible

Propose Zipf-based selectivity estimation method

Verify the accuracy of our approach


26

Thanks(Q&A)

1 selectivity estimation for exclusive query translation in deep web data integration fangjiao jiang...

Documents

domainselectivity estimation

query formsdeep

word rank word rank

correlative attribute

word rank2

surfacethe current web

different correlations

approach thanksqa