how hard is this query? measuring the semantic complexity of schema-agnostic queries
Post on 17-Jul-2015
92 Views
Preview:
TRANSCRIPT
How hard is this query? Measuring the Semantic Complexity of
Schema-agnostic Queries
André Freitas, Juliano Efson Sales, Siegfried Handschuh, Edward Curry
IWCS, London 2015
Outline
• Motivation
• Query Semantic Complexity & Entropy
• Entropy Measures
• Validation & Analysis
• Conclusions
Motivation
Shift in the Database Landscape
Very-large and dynamic “schemas”.
10s-100s attributes1,000s-1,000,000s attributes
before 2000circa 2015
4 Brodie & Liu, 2010
Databases for a Complex WorldHow do you query data on this scenario?
5
Schema-agnosticism
Ab
stra
ctio
n
Laye
r
6
Who is the daughter of Bill Clinton?
Bill Clinton
Chelsea Clinton
child
Schema-agnostic queries
Query approaches over structured databases which
allow users satisfying complex information needs
without the understanding of the representation
(schema) of the database.
7
Semantic Parsing
Vocabulary Problem for Databases
Query: Who is the daughter of Bill Clinton married to?
Quantify the Semantic Gap
Possible representations
8
Core Questions
• Can we measure the semantic complexity of a query-DB mapping?
• What defines an “easy” or a “hard” query?
• Which are the best estimators?
9
Semantic Complexity & Entropy
Configuration space of semantic matchings
Quantify the Query-DB semantic gap
Not all queries are born equal!
11
Semantic Complexity & Entropy
Semantic Complexity & Entropy
• Structural/conceptual complexity
• Level of ambiguity/indeterminacy/vagueness
• Teminological gap
• Novelty
12
Semantic Configuration Space
mΣ(Q,DB)13
Semantic Entropy Measures
Semantic Entropy Measures
Hsyntax
15
?Hstruct
HtermHtermHmatching
In the scope of this work
• Entropy -> Entropy estimator, approximation.
16
Syntactic Entropy (Hsyntax)
• The syntactic entropy of a query is defined by thepossible syntactic configurations in which a querycan be interpreted under the database syntax.
• Estimate the uncertainty of the translation of thequery into the DB categories (IDB(Q)).
• Is a function of the probability of the syntacticinterpretation of a query.
17
Structural Entropy (Hstruct)
• The structural entropy defines the complexity of adatabase based on the possible facts that can beencoded under its schema.
• Pollard & Biermann, A measure of semanticcomplexity for natural language systems (2000).
18
Terminological Entropy (Hterm)
• The terminological entropy focuses on quantifying anestimate on the amount of ambiguity, synonymy andvagueness for the query or database terms.
• Translational Entropy (Htrans) as an estimator.
• Melamed, Measuring semantic entropy (1997).
• Translation probability based on parallel corpora.
19
Matching Entropy (Hmatching)
• Consists of measures which describe theuncertainty involved in the query-datamatching/alignment between query terms anddataset entities.
• Provides an estimate based on the set ofpotential alignments.
• Distributional entropy (Hdist): Estimator based ondistributional semantic models.
20
Query Features as Complexity Estimators
• Query features (reference to data model/query operator categories).– Contains instance reference (named entities)
– Contains class reference
– Contains complex class reference
– Contains property
– Contains value
– Yes/No question
– Contains operator
21
Validation & Analysis
Experimental Set-up
• Question Answering over Linked Data TestCollection (Unger et al. 2011).
• QALD 2011 & 2012.
• 150 natural language queries over DBpedia(RDF).
Dataset (DBpedia + YAGO classes): 45,768 properties288,316 classes9,434,677 instances128,071,259 triples
23
Query Analysis Example
24
Query Analysis Example
25
Experimental Set-up
• Linear regression between each entropymeasure and the f-measure of theparticipating QA systems.
• 4 QA systems:– QALD 2011: PowerAqua, Freya (κ = 0.501, 95% confidence
interval, ‘moderate’ agreement).
– QALD 2012: QAKis, MHE (κ= 0.236, 95% confidenceinterval, ‘fair’ agreement).
26
1st Analysis
• Linear regression model.
• Hsyntax, Hterm (Htrans), Hmatching (Hdist) and Hstruct
27
1st Analysis
• Higher correlation:
– Hsyntax (-)
– Hterm (Htrans) (-)
– Hmatching (Hdist) (-)
• Lower correlation:
– Hstruct
28
2nd Analysis
• Query features (reference to data model/query operator categories).– Contains instance reference (named entities)
– Contains class reference
– Contains complex class reference
– Contains property
– Contains value
– Yes/No question
– Contains operator
29
2nd Analysis
• Linear regression model.
30
2nd Analysis
• Higher correlation:
– References to instances (+)
– Presence of operators (-)
– Presence of complex classes (complex nominals) (-)
31
3rd Analysis
• Classification of the query-DBterminological gap for each datamodel category.
32
3rd Analysis
Lower terminological gap
Higher terminological gap
Query Classification
...34
Query Classification
• % of unanswered questions:
– Syntactic complexity (Hsyntax): 51.7%
– Vocabulary gap (Hmatching, Hterm): 68.9%
– No reference to instance (named entity) (Hstruct,Hterm): 20.6%
35
Limitations
• Validation of the regression model in adifferent test collection.
• Distributional entropy needs a moreprincipled definition.
36
Minimizing Semantic Entropy
Reflections on the Design of Schema-agnostic Query Mechanisms
Or ....
Minimizing the Semantic Entropy for the Semantic Matching
Definition of a semantic pivot: first query term to be resolved in the database.
Maximizes the reduction of the semanticconfiguration space (Hstruct , Hmatch).
38
Semantic Pivots (Hstruct , Hmatch)
• Who is the daughter of Bill Clinton married to?
437100,184 62,781
> 4,580,000
dbpedia:spouse dbpedia:children :Bill_Clinton
39
Minimizing the Semantic Entropy for the Semantic Matching
Definition of a semantic pivot: first query termto be resolved in the database.
Maximizes the reduction of the semanticconfiguration space (Hstruct , Hmatch).
Less prone to more complex synonymicexpressions and abstraction-level differences(Hterm , Hmatch).
40
Semantic Pivots
• Proper nouns tends to have high percentage of string
overlap for synonymic expressions.
William Jefferson Clinton
Bill Clinton
William J. Clinton
T. E. Lawrence
Thomas Edward Lawrence
Lawrence of Arabia
Who is the daughter of Bill Clinton married to?
41
Minimizing the Semantic Entropy for the Semantic Matching
Definition of a semantic pivot: first query term to be resolved in the database.
Maximizes the reduction of the semanticconfiguration space (Hstruct , Hmatch).
Less prone to more complex synonymic expressionsand abstraction-level differences (Hterm , Hmatch).
proper nouns >> nouns >> complex nominals >>adjectives , verbs.
42
Semantic Matching
• Hsyntax is a strong estimator of querycomplexity.
• Hmatching can be used as an estimator for thequality of the predicate alignment.
• Hterm can be used as a heuristic for matchingcomplexity.
43
Conclusions
• Both entropy (Hsyntax, Hterm, Hmatching) and query features(instances, complex classes, operators) can be used asestimators for query semantic complexity.
• This can be incorporated as heuristics into schema-agnostic query planning approaches (or approximatesemantic parsing) to maximize semantic matchingprobabilities.
• Need for the construction of better semantic entropyestimators.
44
top related