ontology search: an empirical evaluation
DESCRIPTION
Much of the recent work in Semantic Search is concerned with addressing the challenge of finding entities in the growing Web of Data. However, alongside this growth, there is a significant increase in the availability of ontologies that can be used to describe these entities. Whereas several methods have been proposed in Semantic Search to rank entities based on a keyword query, little work has been published on search and ranking of resources in ontologies. To the best of our knowledge, this work is the first to propose a benchmark suite for ontology search. The benchmark suite, named CBRBench, includes a collection of ontologies that was retrieved by crawling a seed set of ontology URIs derived from prefix.cc and a set of queries derived from a real query log from the Linked Open Vocabularies search engine. Further, it includes the results for the ideal ranking of the concepts in the ontology collection for the identified set of query terms which was established based on the opinions of ten ontology engineering experts.TRANSCRIPT
Ontology Search: An Empirical Evaluation
Anila Sahar ButtAnila Sahar Butt, , Armin HallerArmin Haller, Lexing Xie, Lexing XieThe Australian National UniversityThe Australian National University
[email protected]@anu.edu.au
Outline
Motivation – Ranking the Ontology Search
CBRBench – CanBeRra Ontology Benchmark
Observation
Recommendations
2
Motivation – Ontology Search
“An ontology is a formal, explicit specification of a shared conceptualization.” [Gruber 1992]
A central ingredient when building an ontology or defining data is the ability to effectively re-use existing ontologies, i.e. discovering the “right”
class or property to be used for a specific use case
3
4
5
Motivation
Terms are defined with differing: Perspectives Levels of detail Reuse and Extensions
How to rank classes and properties with different levels of modelling detail?
6
CBRBench: Benchmark Suite
CBRBench: CanBeRra Ontology Ranking Benchmark
1. Ontology Collection
2. Queries
3. Ground Truth
4. Effectiveness of eight state-of-the-art ranking models
7
CBRBench: Benchmark Suite
Ontology Collection Seed set: Prefix.cc
1022 Ontologies ~5.5 Millions Triple ~280K classes ~7.5K properties
8
CBRBench: Benchmark Suite
Benchmark Queries Search Log: LOV - Linked Open
Vocabularies Benchmark Queries Single keyword Compound words– ~11% of search queries log, no compound query
in top 200
– No relevant resources for top 1000 in the ontology collection.
Search Term Rank
Person 1
Name 2
Event 3
Title 5
Location 7
Address 8
Music 10
Organization 15
Author 16
Time 17
9
CBRBench: Benchmark SuiteEstablishing the Ground Truth:
Candidate Result set selection for queries through partial keyword match URI, rdfs:label, rdfs:comment, rdfs:description
Initial Screening by two experts Relevant or Irrelevant
Ranking by ten ontology engineers
Judge classes against definition from Oxford Dictionary Based on label, subclass, superclass and properties 5-point Likert-Scale: Extremely Useful, Useful, Relevant, Slightly Useful, Irrelevant.
10
Sample Rankings by Experts – “Person” query
Rank URI
1 http://xmlns.com/foaf/0.1/Person
2 http://data.press.net/ontology/stuff/Person
3 http://schema.org/Person
4 http://www.w3.org/ns/person#Person
5 http://www.ontotext.com/proton/protontop#Person
6 http://omv.ontoware.org/2005/05/ontology#Person
7 http://bibframe.org/vocab/Person
8 http://iflastandards.info/ns/fr/frbr/frbrer/C1005
9 http://models.okkam.org/ENS-core-vocabulary.owl#person
9 http://swat.cse.lehigh.edu/onto/univ-bench.owl#Person
11
Sample Rankings by Experts – “Event” query
Rank URI
1 http://data.press.net/ontology/event/Event
2 http://purl.org/vocab/bio/0.1/Event
3 http://linkedevents.org/ontology/Event
4 http://schema.org/Event
5 http://purl.org/dc/dcmitype/Event
6 http://www.ontologydesignpatterns.org/cp/owl/participation.owl#Event
7 http://semanticweb.cs.vu.nl/2009/11/sem/Event
8 http://www.loa-cnr.it/ontologies/DUL.owl#Event
8 http://www.ontologydesignpatterns.org/ont/dul/DUL.owl#Event
10 http://purl.org/tio/ns#Event
12
Baseline Ranking
Ranking Algorithms
Content-Based Ranking Models
Graph-Based Ranking Models
PageRank[Page1998]
Density Measure[Alani2006]
Semantic Similarity Measure [Alani2006]
Betweenness Measure[Alani2006]
TF-IDF [Salton1988]
BM25 [Robertson 1995]
Vector Space Model[Salton1975]
Class Match Measure[Alani2006]
13
Baseline Ranking - Performance
14
Observations
Content-based models slightly outperform graph-based models
Intuition behind document retrieval algorithms does not match ontology retrieval TF-IDF misses foaf:Person (162 Ontologies) in top 10 BM25 and VSM inherit wrong intuition of TF-IDF
BM25 ranks “domain ontologies” higher
15
Observations (cont’d)
Density Measure Model ranks “upper level” ontologies higher Terms with complex hierarchies (sub-classes and
super-classes) receive higher ranks
Graph-based Models like PageRank and Density Measure consider relationships (links) irrespective of their relevance to query terms
16
Observations (cont’d)
Semantic Similarity Measure considers the shortest path of two matched terms in an ontology
Model becomes irrelevant in case of single keyword query and single matched term
Ranking models based on term labels alone result in poor performance
Class Match Measure least performing model Address: Same relevance score for all partial matches
“Postal address” “Email address of specimen provider principal investigator”
17
Recommendations
1. Intended type vs. context resource “Name of the Person” Name – Extremely Useful to Useful Person – Slightly Useful
2. Query semantics for partial matches Word disambiguation Person, Personal – Relevant
1. Location, Dislocation – Irrelevant
18
Recommendations (cont’d)
3. Relevant relations vs. context relations• Address: “email address” in OBO
– “part of continuant at some time”, “geographic focus”, “is about”, “has subject area”, “concretized by at some time”, “date/time, value” and “keywords”.
4. Resource relevance vs. ontology relevance
19
Q&ACBRBench available at
http://zenodo.org/record/11121
20