dynamically time-capped possibilistic testing of subclassof …tettaman/k-cap2015-slides.pdf ·...
TRANSCRIPT
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Dynamically Time-Capped Possibilistic Testing ofSubClassOf Axioms Against RDF Data to Enrich
Schemas
Andrea G. B. Tettamanzi, Catherine Faron-Zucker,and Fabien Gandon
Univ. Nice Sophia Antipolis, CNRS, Inria, I3S, UMR 7271, France
K-Cap 2015, Palisades, NY
1 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Introduction: Ontology Learning
Top-down construction of ontologies has limitations
aprioristic and dogmatic
does not scale well
does not lend itself to a collaborative effort
Bottom-up, grass-roots approach to ontology and KB creation
start from RDF facts and learn OWL 2 axioms
Recent contributions towards OWL 2 ontology learning
FOIL-like algorithms for learning concept definitions
statistical schema induction via association rule mining
light-weight schema enrichment (DL-Learner framework)
All these methods apply and extend ILP techniques.
2 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Introduction: Ontology validation, Axiom Scoring
Need for evaluating and validating ontologies
General methodological investigations, surveys
Tools like OOPS! for detecting pitfalls
Integrity constraint validation
Ontology learning and validation rely on axiom scoring
We have recently proposed a possibilistic scoring heuristic[A. Tettamanzi, C. Faron-Zucker, and F. Gandon. “Testing OWL Axioms
against RDF Facts: A possibilistic approach”, EKAW 2014]
Computationally heavy, but there is evidence that testing timetends to be inversely proportional to score
Research Question:
1 Can time capping alleviate the computation of the heuristicwithout giving up the precision of the scores?
3 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Content
Content of an Axiom
Definition (Content of Axiom φ)
We define content(φ), as the finite set of formulas, which can betested against an RDF dataset K, constructed from theset-theoretic formulas expressing the direct OWL 2 semantics of φby grounding them.
E.g., φ = dbo:LaunchPad v dbo:Infrastructure
∀x ∈ ∆I , x ∈ dbo:LaunchPadI ⇒ x ∈ dbo:InfrastructureI
content(φ) = dbo:LaunchPad(r)⇒ dbo:Infrastructure(r) :r is a resource occurring in DBPedia
By construction, for all ψ ∈ content(φ), φ |= ψ.
4 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Content
Confirmation and Counterexample of an Axiom
Given ψ ∈ content(φ) and an RDF dataset K, three cases:
1 K |= ψ: −→ ψ is a confirmation of φ;
2 K |= ¬ψ: −→ ψ is a counterexample of φ;
3 K 6|= ψ and K 6|= ¬ψ: −→ ψ is neither of the above
Selective confirmation: a ψ favoring φ rather than ¬φ.φ = Raven v Black −→ ψ = a black raven (vs. a green apple)
Idea
Restrict content(φ) just to those ψ which can be counterexamplesof φ. Leave out all ψ which would be trivial confirmations of φ.
5 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Support, Confirmation, and Counterexample
Support, Confirmation, and Counterexample of an Axiom
Definition
Given axiom φ, let us define
uφ = ‖content(φ)‖u+φ = the number of confirmations of φ
u−φ = the number of counterexamples of φ
Some properties:
u+φ + u−φ ≤ uφ (there may be ψ s.t. K 6|= ψ and K 6|= ¬ψ)
u+φ = u−¬φ (confirmations of φ are counterexamples of ¬φ)
u−φ = u+¬φ (counterexamples of φ are confirmations of ¬φ)
uφ = u¬φ (φ and ¬φ have the same support)
6 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Possibility Theory
Possibility Theory
Definition (Possibility Distribution)
π : Ω→ [0, 1]
Definition (Possibility and Necessity Measures)
Π(A) = maxω∈A
π(ω);
N(A) = 1− Π(A) = minω∈A1− π(ω).
For all subsets A ⊆ Ω,
1 Π(∅) = N(∅) = 0, Π(Ω) = N(Ω) = 1;2 Π(A) = 1− N(A) (duality);3 N(A) > 0 implies Π(A) = 1, Π(A) < 1 implies N(A) = 0.
In case of complete ignorance on A, Π(A) = Π(A) = 1.7 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Possibility and Necessity of an Axiom
Possibility and Necessity of an Axiom
Π(φ) = 1−
√√√√1−
(uφ − u−φ
uφ
)2
N(φ) =
√√√√1−
(uφ − u+
φ
uφ
)2
if Π(φ) = 1, 0 otherwise.
0 20 40 60 80 100
0.0
0.2
0.4
0.6
0.8
1.0
no. of counterexamples
poss
ibili
ty
0 20 40 60 80 100
0.0
0.2
0.4
0.6
0.8
1.0
no. of confirmations
nece
ssity
8 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Possibility and Necessity of an Axiom
Acceptance/Rejection Index
Combination of possibility and necessity of an axiom:
Definition
ARI(φ) = N(φ)− N(¬φ) = N(φ) + Π(φ)− 1
−1 ≤ ARI(φ) ≤ 1 for all axiom φ
ARI(φ) < 0 suggests rejection of φ (Π(φ) < 1)
ARI(φ) > 0 suggests acceptance of φ (N(φ) > 0)
ARI(φ) ≈ 0 reflects ignorance about the status of φ
9 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
OWL 2 → SPARQL
To test axioms, we define a mapping Q(E , x) from OWL 2expressions to SPARQL graph patterns such that
SELECT DISTINCT ?x WHERE Q(E , ?x)
returns [Q(E , x)], all known instances of class expression E and
ASK Q(E , a)
checks whether E (a) is in the RDF base.
For an atomic concept A (a valid IRI),
Q(A, ?x) = ?x a A .
10 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Concept Negation: Q(¬C , ?x)
Problem
Open-world hypothesis, but no ¬ in RDF!
We approximate an open-world semantics as follows:
Q(¬C , ?x) = ?x a ?dc .
FILTER NOT EXISTS ?z a ?dc . Q(C , ?z)
(1)
For an atomic class expression A, this becomes
Q(¬A, ?x) = ?x a ?dc .
FILTER NOT EXISTS ?z a ?dc . ?z a A .
(2)
11 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
SubClassOf(C D) Axioms
To test SubClassOf axioms, we must define their logical contentbased on their OWL 2 semantics:
(C v D)I = CI ⊆ DI
≡ ∀x x ∈ CI ⇒ x ∈ DI
Therefore, following the principle of selective confirmation,
uCvD = ‖D(a) : K |= C (a)‖,
because, if C (a) holds,
C (a)⇒ D(a) ≡ ¬C (a) ∨ D(a) ≡ ⊥ ∨ D(a) ≡ D(a)
12 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Support, Confirmations and Counterexamples of C v D
uCvD can be computed by
SELECT (count(DISTINCT ?x) AS ?u) WHERE Q(C , ?x).
As for the computational definition of u+CvD and u−CvD :
confirmations: a s.t. a ∈ [Q(C , x)] and a ∈ [Q(D, x)];
counterexamples: a s.t. a ∈ [Q(C , x)] and a ∈ [Q(¬D, x)].
Therefore,
u+CvD can be computed by
SELECT (count(DISTINCT ?x) AS ?numConfirmations)
WHERE Q(C , ?x) Q(D, ?x)
u−CvD can be computed by
SELECT (count(DISTINCT ?x) AS ?numCounterexamples)
WHERE Q(C , ?x) Q(¬D, ?x) 13 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Test a SubClassOf axiom (plain version, w/o time cap)
Input: φ, an axiom of the form SubClassOf(C D);Output: Π(φ), N(φ), confirmations, counterexamples.1: Compute uφ using the corresponding SPARQL query;2: compute u+
φ using the corresponding SPARQL query;
3: if 0 < u+φ ≤ 100 then
4: query a list of confirmations;5: if u+
φ < uφ then
6: compute u−φ using the corresponding SPARQL query;
7: if 0 < u−φ ≤ 100 then8: query a list of counterexamples;9: else
10: u−φ ← 0;
11: compute Π(φ) and N(φ) based on uφ, u+φ , and u−φ .
14 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Comparison with a Probability-Based Score
−1.0 −0.5 0.0 0.5 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Acceptance/Rejection Index
Bühm
ann a
nd L
ehm
ann’s
Score
15 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Scalable Axiom Testing
T (φ) = O((1 + ARI(φ))−1
)or O (exp(−ARI(φ)))
−1.0 −0.5 0.0 0.5 1.0
020000
40000
60000
Acceptance/Rejection Index
Ela
psed T
ime (
min
)
16 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Scalable Axiom Testing
Time Predictor
How much time should we allow in order to be reasonably sure weare not throwing the baby out with the bathwater, while avoidingto waste time on hopeless tests?
Studying the elapsed times for accepted axioms, we observed thatthe time it takes to test C v D tends to be proportional to
TP(C v D) = uCvD · nicC ,
where nicC denotes the number of intersecting classes of C .A computational definition of nicC is the following SPARQL query:
SELECT (count(DISTINCT ?A) AS ?nic)
WHERE Q(C , ?x) ?x a ?A .
where A represents an atomic class expression.17 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Scalable Axiom Testing
T (C v D)/TP(C v D)
−1.0 −0.5 0.0 0.5 1.01e−
09
1e−
07
1e−
05
1e−
03
Acceptance/Rejection Index
Ela
psed T
ime to T
ime P
redic
tor
Ratio
18 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Scalable Axiom Testing
TP(C v D) as a function of the cardinality rank of C
0 100 200 300 400
1e+
01
1e+
03
1e+
05
1e+
07
Class Rank
Tim
e P
redic
tor
19 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Scalable Axiom Testing
Test a SubClassOf axiom (time-capped version)
Input: φ, an axiom of the form SubClassOf(C D);a, b, the coefficients of the linear time cap equation.
Output: Π(φ), N(φ), confirmations, counterexamples.1: Compute uφ and nic using the corresponding SPARQL queries;2: TP(φ)← uφ · nic;
3: compute u+φ using the corresponding SPARQL query;
4: if 0 < u+φ ≤ 100 then
5: query a list of confirmations;6: if u+
φ < uφ then
7: tmax(φ)← a + b · TP(φ)8: waiting up to tmax(φ) min do9: compute u−φ using the corresponding SPARQL query;
10: if time-out then11: u−φ ← uφ − u+
φ ;
12: else if 0 < u−φ ≤ 100 then
13: query a list of counterexamples;14: else15: u−φ ← 0;
16: compute Π(φ) and N(φ) based on uφ, u+φ , and u−φ .
20 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Experiments
Experimental Setup:
DBpedia 3.9 in English as RDF fact repository
Local dump (812,546,748 RDF triples) loaded into Jena TDB
Method coded in Java, using Jena ARQ and TDB
12 6-core Intel Xeon CPUs @2.60GHz (15,360 KB cache), 128GB RAM, 4 TB HD (128 GB SSD cache), Ubuntu 64-bit OS.
Systematically generate and test SubClassOf axioms involvingatomic classes only
For each of the 442 classes C referred to in the RDF store
Construct all C v D : C and D share at least one instance
Test these axioms in increasing time-predictor order
Compare with scores obtained w/o time cap
21 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Time-Capped Score vs. Probability-Based Score
−1.0 −0.5 0.0 0.5 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Acceptance/Rejection Index
Bühm
ann a
nd L
ehm
ann’s
Score
22 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Results
Speed-up:
Testing 722 axioms w/o time cap took 290 days of cpu timeWe managed to test 5,050 axioms in < 342 h 30’ (244s/axiom) w/ time cap142-fold reduction in computing time
Precision loss due to time capping:
632 axioms were tested both w/ and w/o time capOutcome different on 25 of them: a 3.96% error rate
Absolute accuracy:
Comparison to a gold standard of DBpedia OntologySubClassOf axioms + SubClassOf axioms that can beinferred from themOf the 5,050 tested axioms, 1,915 occur in the gold standard327 (17%) get an ARI < 1/334 (1.78%) get an ARI < −1/3
23 / 25
Introduction Principles Possibilistic Scoring Candidate Axiom Testing Subsumption Axiom Testing Experiments Conclusion
Conclusions & Future Work
Contributions
Axiom scoring heuristics based on possibility theory
A framework based on the proposed heuristics
Time capping to reduce the computational overhead
Results
Experiments strongly support the validity of our hypothesis
142-fold speed-up with < 4% increase of error rate
Human evaluation suggests most axioms accepted by mistakeare inverted subsumptions or involve ill-defined concepts
Future work
Extend experimental evaluation to other types of axioms
Enlarge test base w/ additional RDF datasets from the LOD
24 / 25