Empirical findings on ontology metrics

Download Empirical findings on ontology metrics

Post on 04-Sep-2016




0 download

Embed Size (px)


<ul><li><p>a,1 Alcy, W</p><p>e paing ptolor inentslogs fo</p><p>The results show that the existing proposed metrics evaluated in general do not seem to be indicators of</p><p>1. Introduction</p><p>Ontologies (Gruber, 1995) are explicits. Morproblembinedare nod as at of sof</p><p>are capableof judgingqualityproperties and their interpretation. In-quiry on the relation of quality indicators and metrics is in conse-quence an important open challenge for a more systematicapproach todevelopingontology-based software systems. Theprob-lemwith ontologymetrics is that ontologies are very heterogeneousin their structure, objectives and level of formality. In consequence,</p><p>Section 3 describes the statistical analyses performed togetherwith some ndings. Finally, Section 4 concludes the paper and out-lines future research work.</p><p>2. Background</p><p>Ontologies have been a resource in the eld of articial intelli-gence since the 70s. However since the inception of the Semantic</p><p> Corresponding author.E-mail addresses: msicilia@uah.es (M.A. Sicilia), daniel.rodriguezg@uah.es (D.</p><p>Rodrguez), elena.garciab@uah.es (E. Garca-Barriocanal), salvador.sanchez@uah.es1 http://www.w3.org/TR/owl-guide/.</p><p>Expert Systems with Applications 39 (2012) 67066711</p><p>Contents lists available at ciVerse ScienceDirect</p><p>Expert Systems w</p><p>.e lsevier .com/locate /eswa(S. Snchez-Alonso).In the software engineering domain, metrics play an importantrole in designing, developing and identifying software defects of fu-ture maintenance problems. Like it happens in the software engi-neering tasks, generally the construction of ontologies follows aniterative and incremental approach and a number of metrics hasbeen proposed tomeasure different aspects such as reliability, reus-ability and cohesion (see for example Uschold &amp; Gruninger, 1996).However, unlike it happens in software engineering domain, thereis a lack of empirical studies that validate how the different metrics</p><p>guage (OWL),1 that was already used in a previous preliminary study(Manouselis, Sicilia, &amp; Rodrguez, 2010). OWL has been chosen as thelanguage is aW3C recommendation and has rapidly become the stan-dard language for expressing formal ontologies. The results of thestudy point out that current metrics are not directly interpretable ina general way, but there seems to be different categories of ontologiesthat could be discernible from the value of the metrics.</p><p>The rest of the paper is structured as follows. Section 2 coversthe background on ontology metrics and some related work. Next,main concepts and their relationshipdenes the conceptualisations of aconstraints on how terms can be cousing a formal language. Ontologiesent purposes in different systems, anbecome artifacts in the developmen0957-4174/$ - see front matter 2011 Elsevier Ltd. Adoi:10.1016/j.eswa.2011.11.094that ranking, but they can be helpful to identify particular kinds of ontologies. 2011 Elsevier Ltd. All rights reserved.</p><p>representations of do-e formally, an ontologym domain and a set ofto model the domain,</p><p>wadays used for differ-consequence, they havetware.:</p><p>there is a need to carry out exploratory studies that could eventuallyresult in insights about the metrics applicable to different kinds ofontologies and their interpretation for each of them.</p><p>This paper reports on such an exploratory study, carried out bymeasuring a large set of OWL ontologies obtained from theSwoogle repository together with a measure of their popularityusing OntoRank (Ding et al., 2005). The study has been carried outby implementing and using an open source software frameworkfor computing ontologymetrics expressed in theOntologyWebLan-ing ontology metrics from a large set of ontologies extracted from the Swoogle search engine. The Onto-Rank value used inside the Swoogle to rank search results is used as a contrast for some existing metrics.Empirical ndings on ontology metrics</p><p>M.A. Sicilia a,, D. Rodrguez a,b, E. Garca-BarriocanalaDepartment of Computer Science, University of Alcal, Ctra. Barcelona, Km. 31.6, 2887bDepartment of Computing and Communication Technologies, Oxford Brookes Universit</p><p>a r t i c l e i n f o</p><p>Keywords:OntologyMetricsSwoogleOntoRank</p><p>a b s t r a c t</p><p>Ontologies are becoming thinformation in several domhas become an engineerinmethods. As part of any ontifying possible problems oassessment that complemthere are a number of ontostudies that provide a basi</p><p>journal homepage: wwwll rights reserved.S. Snchez-Alonso a</p><p>al de Henares, Madrid, Spainheatley Campus, Oxford OX33 1HX, UK</p><p>referred way of representing, dealing and reasoning with large volumes ofs. In consequence, the creation, evaluation and maintenance of ontologiesrocess that needs to be managed and measured using sound and reliablegy engineering or revision process, metrics can play a role helping in iden-correct use of ontology elements, along with providing a kind of qualityreviews requiring expert inspection. However, in spite of the fact that</p><p>y metric proposals described in the literature, there is a lack of empiricalr their interpretation. This paper reports a systematic exploration of exist-ith ApplicationsS</p></li><li><p>Motwani, &amp; Winograd, 1999). That ranking metric is an externa</p><p>ith Applications 39 (2012) 67066711 6707Web, ontologies have become the principal recourse to integrateand deal with online information, and a new set of standards toformally represent them has been developed and gained wide-spread adoption. The OntologyWeb Language (OWL) is one of suchstandards that belongs to a family of knowledge representationlanguages created for the Semantic Web (Berners-Lee, Hendler, &amp;Lassila, 2001). The OWL has reached the status of W3C (WorldWide Web Consortium) recommendation. From a technical pointof view OWL extends the RDF (Resource Description Framework)and RDF-S (RDF Schema) allowing us to integrate a variety of appli-cations using XML as interchange syntax.</p><p>The OWL has evolved since its inception in 2004 to a new spec-ication in 2009 (OWL2) with different sub-languages dependingon the expressiveness and reasoning capabilities provided. Thereare however common elements and OWL ontologies are composedof (i) classes that can be dened as sets of individuals, (ii) individ-uals as instances of classes, i.e., objects of the domain and (iii)properties as binary relations between individuals. It is also possi-ble to specify property domains, cardinality ranges and reasoningon ontologies. From these elements a number of authors have pro-posed metrics to measure the quality of ontologies.</p><p>Several authors have proposed metrics related to ontologies oradapted from object-oriented software. For example Yao, Orme,and Etzkorn (2005) proposed a set on metrics to measure the cohe-sion in ontologies such as the number of root classes, number ofleaf classes and average depth of inheritance tree of all leaf nodes.Authors validated the metrics both theoretically and empirically.Their empirical validation consisted of comparing statistically theresults of the metrics and subjective evaluation of eighteenevaluators.</p><p>Tartir, Budak Arpinar, Moore, Sheth, and Aleman-Meza (2005)propose three orthogonal dimensions to evaluate the quality ofan ontology. The rst dimension is quality of the ontology schemarepresenting the real world. The second dimension evaluates thecontent of the ontology, i.e, how well populated is the ontologyand if it reects the real world. Finally, the third dimension evalu-ates if the set of instances and relations agree with the schema.Schema metrics proposed by the authors include relationship,attribute and inheritance richness. Instance metrics are dividedinto (i) knowledge base metrics such as class richness, averagepopulation and cohesion and (ii) class metrics which includeimportance, fullness, inheritance richness, relationship richnessconnectivity and readability. Authors analysed two general pur-pose ontologies SWETO and TAP as well as the GlycO (GlycomicsOntology) ontology.</p><p>Vrandecic and Sure (2007) propose a framework for designingontologies though a process of normalisation and considering sta-ble metrics, i.e., metrics considering the open world assumption.</p><p>Metrics and metric frameworks can help with processes ormethodologies for the design of ontologies or ontological tools. Awell-known methodology for the evaluation of ontological ade-quacy of taxonomic relationships is OntoClean dened by Guarinoand Welty (2002). Another process is the OntoMetric process de-ned by Lozano-Tello and Gmez-Prez (2004). OntoMetric is aprocess to select ontologies using a broad range of metrics, notonly those that can be collected form the actual ontologies.</p><p>A suite a of ontology metrics to measure design complexity hasbeen recently proposed (Zhang, Li, &amp; Kuan Tan, 2010), reporting ananalysis of desirable properties for metrics described by Weyuker,and providing results of an empirical evaluation in which the met-rics proposed are shown to be discriminating ontologies that areconsidered by the authors to be of different complexity.</p><p>In another direction Ding et al. (2004, 2005) have developed</p><p>M.A. Sicilia et al. / Expert Systems wSwoogle, a semantic search engine in which the popularity of aSemantic Web document is dened using the OntoRank measureadapted from the well-known PageRank algorithm (Page, Brin,criteria that can be used to contrast ontology metrics. The problemwith the ranking is that its interpretation as a quality measure isnot evident and has not been empirically studied. However, asother quality measures are not available for ontologies, it repre-sents an interesting opportunity to contrast existing metrics.</p><p>We also developed an open source framework to extract theOWL ontology metrics called OntoMetrics. The OntoMetrics is anopen source Java implementation that makes use the Java librariesof Protg2 an ontology editor that provides an application program-mers interface (API) for loading, saving and manipulating OWL andRDF les. In this paper we have use this framework to collect metricsfrom a large number of ontologies together with their popularitymeasured using OntoRank.</p><p>3. Empirical analysis of ontology data</p><p>In this section, the metrics gathered are described and then theresults of the statistical analysis is reported. Only basic metrics re-lated to the main elements of ontologies have been implementedand used. Particularly, metrics dealing with classes inferred (i.e.,dened classes) or axioms were not considered as a majority ofthe ontologies available for the study lacked these kinds ofconstructs.</p><p>3.1. Dataset</p><p>In order to empirically analyse the relationships between thepreviously dened metrics (Section 2), we downloaded around1500 ontologies dened in OWL using Swoogle,3 which is a searchengine for the Semantic Web. A crawler was developed ad hoc thatsubmits a blank query to Swoogle and then navigates the pagesdownloading automatically the OWL les. Only ontologies expressedin OWL were considered as they represent the majority of the avail-able ones, and allows for an homogeneous analysis.</p><p>Using the previously commented OntoMetrics framework, wecollected the following metrics:</p><p> No. of classes (noc) is a simple count of the number of classes(jCj) contained in the ontology.</p><p> No. of instances (noi) is a simple count of the number ofinstances (jIj) contained in the ontology.</p><p> No. of properties (nop) is a simple count of the number of prop-erties (jpj) contained in the ontology.</p><p> No. of root classes metric (norc) (Yao et al., 2005) corresponds tothe number of root classes (without superclasses) explicitlydened. Let C the classes in an ontology:norc jCij; : 9 CjjCiCj. The range of this metric is from 1 tojCj. There is at least one root and the larger the number of rootclasses, the more diverse the ontology will be.</p><p> No. of leaf classes metric (nolc) (Yao et al., 2005) is the sum of allleaf classes, i.e., those without subclasses, in an ontology. Let Cthe set of classes in an ontology: nolc jCij; : 9 CjjCjCi. A rootclass without inheritance can also be considered a leaf class, therange of this metrics is from 1 to jCj.</p><p> Average population metric (ap) (Tartir et al., 2005) measures theaverage distribution of instances across all classes. Formally, itis dened as: ap jIjjCj. According the authors, this metric is sup-posed to be used in conjunction with the Class Richness metricas an indication if there is enough information in the ontology.</p><p> Class richnessmetric (cr) (Tartir et al., 2005) is the ratio betweenthe number of classes that have instances (jC0 j) divided by the2 http://protege.stanford.edu/.3 http://swoogle.umbc.edu/.l</p></li><li><p>total number of classes. This metric provides an indication ofhow many instances are really related to classes dened inthe schema.</p><p> Explicit depth of subsumption hierarchy (dosh). This metric corre-sponds to the maximum length of hierarchical relationships inthe ontology explicitly dened, i.e., longer path from a rootclass.</p><p> Relationship richness metric (rr) (Tartir et al., 2005) is dened asthe ratio of the number of relationships divided by the sum ofthe number of subclasses plus the number of relationships:rr jPjjSCjjPj, where jPj is the number of relationships and SC isthe number of subclasses (or the number of inheritance rela-tionships). According to the authors, this metric reects thediversity of relations and placement of relations in the ontology.</p><p>An ontology that contains many relationships other than class-subclass relations (values close to 1) is richer than a taxonomywith only class-subclass relationships (values close to zero).</p><p> Inheritance richness metric (ir) (Tartir et al., 2005) is dened asthe average number of subclasses per class. Formally, ir isdened as:</p><p>ir P</p><p>Ci2C jHcC1;CijjCj</p><p>where jHc(C1, Ci)j is the number of subclasses (C1) for a class (Ci) andthe divisor (jCj) is the total number of classes. According to theauthors, this metric represents the distribution of informationacross different levels of the ontology inheritance tree and servesas an indicator of how well knowledge is grouped into different cat-egories and subcategories in the ontology. Values close to zeroindicate at or horizontal ontologies representing perhaps more</p><p>Table 1Summary of ontology metrics used in this work.</p><p>Metric Acronym</p><p>No. of classes nocNo. of instances noiNo. of properties nopNo. of root classes norcNumber of leaf classes nolcAverage population apClass richness crExplicit depth of subsumption hierarchy doshInheritance richness irRelationship richness metric rrOntoRank or</p><p>Table 2Descriptive statistics.</p><p>4</p><p>891</p><p>Fig. 2. Box-and-whisker plots.</p><p>6708 M.A. Sicilia et al. / Expert Systems with Applications 39 (2012) 67066711Avg Med Var StdDev Min</p><p>or 2.74 0.85 968.5 31.12 0.6ap 1.34 1 8.83 2.97 0cr 0.54 0.75 0.20 0.44 0dosh 2.54 1 5.75 2.4 1ir 0.34 0 0.17 0.42 0noc 36.11 6 6166.3 78.53 1noi 28.13 6 9522.8 97.59 0nolc 27.46 5 3815.8 61.77 0nop 24 0 2997.8 54.75 0norc 6.69 5 127.37 11.29 0rr 2.78 1 13.57 3.68 0Fig. 1. Histo71 71 5 156.8 1326.26grams.23 823 27 90.87 367.8052 952 23 107.11 690.5394 194 6 131.7 857.18Max Range Q3 StdSk...</p></li></ul>


View more >