Ontology Assessment – Ontology Assessment – Proposed Framework and Proposed Framework and

MethodologyMethodology

Which one is an ontology? Why/not? Which one is an ontology? Why/not?

Biological Classification Scheme

AMS Classification Scheme NASA Thesaurus Library of Congress Subject

Headings Dublin Core Metadata Scheme Organizational Chart ISO Country List Metadata repository scheme Master Data Repository Content architecture models

(OO models) SCORM

XML Schema for Directory Records Classification SchemeRecords Classification Scheme Social Network Social Network

RepresentationRepresentation FolksomonyFolksomony Domain Knowledge Map Domain Knowledge Map Visual representation of Visual representation of

concept clustersconcept clusters Financial ratios Economic indicators Mathematical formula XML structured electronic

journal issue WordNet

Why do we care whether one is or isn’t an ontology? What’s the point?

Personal RecommendationPersonal Recommendation

We need to distinguish between an ontology and applications that We need to distinguish between an ontology and applications that use ontologiesuse ontologies

We need to suspend our heavy reliance on different domain We need to suspend our heavy reliance on different domain terminologies that describe applications that use ontologies and terminologies that describe applications that use ontologies and adopt a neutral mental model adopt a neutral mental model

Minimize our references to dictionaries, synonym rings, Minimize our references to dictionaries, synonym rings, taxonomies, thesauri, class schemes, knowledge maps, content taxonomies, thesauri, class schemes, knowledge maps, content schema, etc. schema, etc.

Rather, compare the specific applications and standards to a neutral Rather, compare the specific applications and standards to a neutral framework – this will facilitate more intelligent conversations, and framework – this will facilitate more intelligent conversations, and will also help us to better communicate with others outside the fieldwill also help us to better communicate with others outside the field

Framework must reflect the multi-dimensionality of ontologies, Framework must reflect the multi-dimensionality of ontologies, though – a single, linear representation of applications does not though – a single, linear representation of applications does not serve as a framework serve as a framework

Goals of the FrameworkGoals of the Framework Before we can do this, though, we need to explicitly agree on the ‘end

game’ of a the framework

Do we agree that the goal is to develop a neutral, well defined, quantifiable, multidimensional framework against which any ‘thing’ that any one is calling an ontology could be evaluated?

Anyone who has anything they’re calling an ontology should be able to use the framework to judge:

whether it is or is not an ontology Which essential components it is missing Where it ranks on a scale of informal to formal ontologies What they can do to improve or enhance it

Enable assessment of any proposed ontology for the purpose of informing users about an ontology Providing developers with methodology for comparison and

improvement

Enable definition of: Minimum standards for what is/is not an ontology Thresholds for formal and informal ontologies

First StepsFirst Steps

If we agree on the goal, we need to start by defining the basic dimensions of an ontology.

Dimensionality proposed in the framework includes: Structure Expressiveness representational granularity intended use automated reasoning descriptive/prescriptive design methodology

Do the proposed framework dimensions accomplish this goal? Are these theoretical or practical dimensions? Do they work at a representational or on an analytical level? How easy would it be for people who are developing ontologies to understand them? Do they allow everyone who is working in an ontology space to play, or do they automatically

exclude some? Do they support the ‘end game’ of communication and use?

Framework RecommendationsFramework Recommendations

I would suggest that the framework still requires both simplification and I would suggest that the framework still requires both simplification and elaborationelaboration

Simplification in terms of how it groups factors, and elaboration in Simplification in terms of how it groups factors, and elaboration in terms of coverage of the factors that matter to those who are terms of coverage of the factors that matter to those who are developing or using ontologiesdeveloping or using ontologies

‘‘Pre-tested’ the framework with some colleagues in different areas of Pre-tested’ the framework with some colleagues in different areas of responsibility – none of them could understand the framework because responsibility – none of them could understand the framework because it was too theoreticalit was too theoretical

Need to bring it to a practical levelNeed to bring it to a practical level Need to describe the dimensions in terms of formal lists of factors and Need to describe the dimensions in terms of formal lists of factors and

concrete definitions for those factorsconcrete definitions for those factors

As we develop the framework, we must also define the analytical As we develop the framework, we must also define the analytical methodsmethods

The test of the framework is our ability to leverage it as an analysis method that The test of the framework is our ability to leverage it as an analysis method that allows us to neutrally characterize any ‘thing’ as an ontology and to be able to allows us to neutrally characterize any ‘thing’ as an ontology and to be able to explain the characterization so that anyone can understandexplain the characterization so that anyone can understand

Suggest that we should consider using a simple factor analysis for representation Suggest that we should consider using a simple factor analysis for representation and analysisand analysis

Simplify the FrameworkSimplify the Framework

I would suggest that the following framework accomplishes this ‘end game’ more effectively

Concepts – the nature of the content or values that are delivered or accessed through the ontology such as type, granularity, etc.

Relationships – nature, type, extent, specification of relationships, logic associated with relationships

Context – the context for which the ontology was developed and in which it may be used, including knowledge domain, application domain,

Governance – control and management of the concepts, relationships and context exercised by the developer or current user

Dimensions are orthogonal but yet sufficiently well defined that they allow Dimensions are orthogonal but yet sufficiently well defined that they allow us to include factors which are important to different kinds of ontology us to include factors which are important to different kinds of ontology applicationsapplications

Factor AnalysisFactor Analysis

Statistical method used to describe variability of factors in which the factors Statistical method used to describe variability of factors in which the factors are modeled as linear combinations. are modeled as linear combinations.

A single factor in the model would represent a set of ‘like’ variables which A single factor in the model would represent a set of ‘like’ variables which otherwise would be too complex to modelotherwise would be too complex to model

Factor analysis might help us to synthesize a set of variables into a single Factor analysis might help us to synthesize a set of variables into a single factor – to represent in this case a dimension of an ontologyfactor – to represent in this case a dimension of an ontology

Challenges: Challenges: Agree on dimensions (synthesis of factors)Agree on dimensions (synthesis of factors) Develop a method for quantifying factors appropriate to the dimensionDevelop a method for quantifying factors appropriate to the dimension Define the method of factor analysisDefine the method of factor analysis

Advantages: Advantages: it might help us to focus our discussions on actual factors and away from it might help us to focus our discussions on actual factors and away from

argumentationargumentation Allows everyone and anyone to play in the ontology spaceAllows everyone and anyone to play in the ontology space Allows everyone and anyone to characterize their ontology as a starting point for Allows everyone and anyone to characterize their ontology as a starting point for

conversations and interoperability conversations and interoperability We can keep the analysis simple since we are only using this to ‘characterize’ and We can keep the analysis simple since we are only using this to ‘characterize’ and

‘communicate’ – not to predict or to explain factors and ‘communicate’ – not to predict or to explain factors and

Example of Factor Analysis Example of Factor Analysis Methodology is currently used to calculate and visually display factors which

Contribute to the development or knowledge economies. Helps economists to compare and define knowledge economies.

Sample List of Innovation FactorsSample List of Innovation Factors

Let’s take as an example the ICT factor as it relates to Let’s take as an example the ICT factor as it relates to knowledge economiesknowledge economies

What factors might define the ICT Dimension?What factors might define the ICT Dimension? Access to computersAccess to computers Telecommunications developmentTelecommunications development Level of education achievedLevel of education achieved Investment in technology development (Tech R&D)Investment in technology development (Tech R&D)

Proposed Ontology Assessment Proposed Ontology Assessment MethodologyMethodology

Factor analysis for ontologies would involve …Factor analysis for ontologies would involve …

1.1. defining the essential dimensions of an ontologydefining the essential dimensions of an ontology2.2. defining those factors which characterize each dimension defining those factors which characterize each dimension 3.3. quantifying the factors quantifying the factors 4.4. analyzing the factors for any given application (factor analysis) or analyzing the factors for any given application (factor analysis) or

comparisoncomparison5.5. visually representing the analysis for a single ‘ontology’ and/or visually representing the analysis for a single ‘ontology’ and/or

for comparisons of ‘ontologies’for comparisons of ‘ontologies’

Let me explain how factor analysis might be usedLet me explain how factor analysis might be used If we can define the dimensions of an ontology, each dimension If we can define the dimensions of an ontology, each dimension

could then be represented as a composite measurecould then be represented as a composite measure The composite measure is made up of scores for a set of factors The composite measure is made up of scores for a set of factors

that define that dimension that define that dimension Having a composite score for each dimension would allow us to Having a composite score for each dimension would allow us to

use a very simple analytical method that would characterize or use a very simple analytical method that would characterize or compare specific ontology applicationscompare specific ontology applications

Representation of Ontological Representation of Ontological AssessmentsAssessments

Methodology could be used to generate a factor index for ontologies, to rank and compare ontologies.

Dimension 3 Dimension 2

Dimension 1

Dimension 4

Another Dimensionality Framework

Index of Factors

Index of Factors

Index of Factors

Index of Factors

Factor AnalysisFactor Analysis

Factor analysis could be conducted:Factor analysis could be conducted: At the component level on that subset of factorsAt the component level on that subset of factors At the ontology level, across all factorsAt the ontology level, across all factors

Developers or users could determine what the optimal Developers or users could determine what the optimal dimensionality was for their particular use dimensionality was for their particular use

Summit members and the Ontology community could Summit members and the Ontology community could identify minimun factor scores that define what is/is not an identify minimun factor scores that define what is/is not an ontology, and what constitutes a full, formal ontologyontology, and what constitutes a full, formal ontology

Ultimately, this may provide us with an ecumenical vs. Ultimately, this may provide us with an ecumenical vs. evangelical approach to ontological standards development evangelical approach to ontological standards development and assessment and assessment

Representation of Ontological Representation of Ontological AssessmentsAssessments

Structure

RepresetationalGranularity

Expressiveness

Descriptive vs.Prescriptive

Dimensionality Suggested in the Framework Paper

Intended Use

Use of AutomatedReasoning

Critical Question: Are these dimensions orthogonal, mutually exclusive and clean enough for analysis?

Representation of Ontological Representation of Ontological AssessmentsAssessments

Methodology could be used to generate an ontological factor index for ontologies, and to rank and compare ontologies.

Concepts Context

Relationships

Governance

Another Dimensionality Framework

Representation of Ontological Representation of Ontological AssessmentsAssessments

Methodology could be used to generate an ontological factor index for ‘ontological things’, and to rank and compare ontologies.

Concepts Context

Relationships

Governance

Sample assessment of a folksonomy

Representation of Ontological Representation of Ontological AssessmentsAssessments

Concepts Context

Relationships

Governance

Sample assessment of a medical disease classification scheme

Representation of Ontological Representation of Ontological AssessmentsAssessments

Concepts Context

Relationships

Governance

Sample assessment of an institutional records classification scheme

Defining and Quantifying FactorsDefining and Quantifying Factors For each component an orthogonal, independent set of For each component an orthogonal, independent set of

factors must be defined factors must be defined

Factors must be independent of any particular pre-existing Factors must be independent of any particular pre-existing ontology (neutral)ontology (neutral)

Each factor must have a quantifiable method of Each factor must have a quantifiable method of representation that lends itself to ‘scoring’, analysis and representation that lends itself to ‘scoring’, analysis and comparisoncomparison

Factors must have agreed upon definitions, be easily Factors must have agreed upon definitions, be easily interpreted by people and machines, and be inclusive in interpreted by people and machines, and be inclusive in their coverage of values/conditionstheir coverage of values/conditions

To illustrate the idea, selected examples are presented in To illustrate the idea, selected examples are presented in following slidesfollowing slides

Selected Examples of Concept FactorsSelected Examples of Concept Factors

Concept typesConcept types Data/numbersData/numbers Calculation/ratiosCalculation/ratios WordsWords Grammatical fragmentGrammatical fragment Logical statementLogical statement Rule expressionRule expression Engineering equationsEngineering equations

Degree of ambiguityDegree of ambiguity Context Context

sensitivity/insensitivity sensitivity/insensitivity of definitionof definition

Representational formRepresentational form Usable encoding Usable encoding

methodmethod Availability of Availability of

representational representational specifications (Strings specifications (Strings vs. syntax)vs. syntax)

Degree of Degree of conceptualization/ conceptualization/ specificationspecification Theoretical to commitalTheoretical to commital

What else…?What else…?

Selected Examples of Relationship Selected Examples of Relationship FactorsFactors

Simple expressive form Simple expressive form of relationshipsof relationships GrammaticalGrammatical MathematicalMathematical LogicalLogical

Relationship behavior Relationship behavior Membership dependenceMembership dependence Representation or Representation or

instanceinstance EquivalenceEquivalence Causal dependenceCausal dependence Derivational dependenceDerivational dependence

Degree of Relationship Degree of Relationship Validation/RigorValidation/Rigor Fully SubjectiveFully Subjective Grammatical validationGrammatical validation Mathematical validationMathematical validation Logical rigor/validationLogical rigor/validation

What else?What else?

Selected Examples of Context FactorsSelected Examples of Context Factors

Knowledge ContextKnowledge Context Formal vs. informal Formal vs. informal

knowledge domainknowledge domain

Application ContextApplication Context System vs. human System vs. human

application/ application/ consumptionconsumption

Managed/standardized Managed/standardized application vs. home application vs. home growngrown

Functional contextFunctional context SearchSearch Mathematical or Mathematical or

statistical analysisstatistical analysis Logical inferenceLogical inference ClassificationClassification Dynamic clusteringDynamic clustering Metadata representationMetadata representation Concept indexingConcept indexing

What else…?What else…?

Selected Examples of Governance Selected Examples of Governance FactorsFactors

Standards AvailabilityStandards Availability Published formal vs. Published formal vs.

guidelines vs. ad hoc guidelines vs. ad hoc conceptsconcepts

Published formal vs. Published formal vs. guidelines vs. ad hoc guidelines vs. ad hoc relationshipsrelationships

Prescriptive vs. Descriptive Prescriptive vs. Descriptive GovernanceGovernance Enforcement of standardsEnforcement of standards

Design GuidelinesDesign Guidelines Top-down (model) vs. Top-down (model) vs.

Bottom-up (empirical)Bottom-up (empirical)

ExtensibilityExtensibility Degree to which others Degree to which others

can add to or extend can add to or extend either the concepts or the either the concepts or the relationshipsrelationships

CurrencyCurrency Degree to which the Degree to which the

concepts and/or concepts and/or relationships represent our relationships represent our current view or knowledge current view or knowledge of the contextof the context

What else…?What else…?