An Empirical Study of Instance-Based Ontology Mapping

Download An Empirical Study of Instance-Based Ontology Mapping

Post on 08-Jan-2016

46 views

Category:

Documents

2 download

Embed Size (px)

DESCRIPTION

An Empirical Study of Instance-Based Ontology Mapping. Antoine Isaac, Lourens van der Meij, Stefan Schlobach , Shenghui Wang STITCH@CATCH funded by NWO Vrije Universiteit Amsterdam Koninklijke Bibliotheek Den Haag Max Planck Instutute Nijmegen. Metamotivation. Ontology mapping in practise - PowerPoint PPT Presentation

TRANSCRIPT

  • An Empirical Study of Instance-Based Ontology Mapping Antoine Isaac, Lourens van der Meij, Stefan Schlobach, Shenghui Wang

    STITCH@CATCH funded by NWO

    Vrije Universiteit AmsterdamKoninklijke Bibliotheek Den HaagMax Planck Instutute Nijmegen

    Title of the presentation

    ISWC 2007

    MetamotivationOntology mapping in practise Based on real problems in the host institution at the Dutch Royal Library

    Task-driven Annotation supportMerging of thesauri

    Real thesauri (100 years of tradition)Really messyConceptually difficultInexpressive Generic Solutions to Specific Questions & TasksUsing Semantic Web Standards (SKOSification)

    Title of the presentation

    ISWC 2007

    OverviewUse-caseInstance-based mappingEvaluationExperimentsResultsConclusions

    Title of the presentation

    ISWC 2007

    The Alignment Task: ContextNational Library of the Netherlands (KB)2 main collectionsLegal Deposit: all Dutch printed booksScientific Collections: history, languageEach described (indexed) by its own thesaurus

    Title of the presentation

    ScientificCollection

    Depot

    1Mbooks

    1.4Mbooks

    GTT

    Brinkman

    ISWC 2007

    A need for thesaurus mappingThe KB wants (Scenario 1) Possibly discontinue one of both annotation and retrieval methods.(Scenario 2) Possibly merge the thesauriWe try to explore mapping(Task 1) In case of single/new/merged retrieval system, find books annotated with old system, facilitated by using mappings(Task 2) Candidate terms for merged thesaurusWe make use of the doubly annotated corpus to calculate Instance-Based mappings

    Title of the presentation

    ISWC 2007

    OverviewUse-caseInstance-based mappingEvaluationExperimentsResultsConclusions

    Title of the presentation

    ISWC 2007

    Calculating mappings using Concept Extensions

    ISWC 2007

    Standard approach (Jaccard)Use co-occurrence measure to calculate similarity between 2 concepts: e.g.

    BGElements of BElements of G

    Joint ElementsSimilarity = 5/9 = 55 % (overlap, e.g. Degree of Greenness )Similarity = 1/7 = 14 % (overlap, e.g. Degree of Greenness )Set of books in the library

    Title of the presentation

    ISWC 2007

    Issues with this measure (sparse data)What is more reliable?

    We need more reliable measures Or thresholds (at least n doubly annotated books)

    Or?Jacc = 18/21 = 86 % Jacc = 1/1 = 100 % The second solution is worse: bB = {MemberOfParliament} and bG = {Cricket}

    Title of the presentation

    ISWC 2007

    Issue with measure (hierarchy): BGNon hierarchicalSet of books in the libraryHierarchical ElementsBJacc(B,G) = = 50%Jacc(B,G) = 2/6 = 33%Consider a hierarchy

    Title of the presentation

    ISWC 2007

    An empirical study of instance-based OMWe experimented with three dimensionsSimilarity measureThreshold HierarchyJaccardCorrected JaccardPointwise Mutual InformationLog Likelihood RatioInformation Gain 010YesNoWhy only 2 thresholds? Because of evaluation costs!

    Title of the presentation

    ISWC 2007

    OverviewUse-caseInstance-based mappingEvaluationExperimentsResultsConclusions

    Title of the presentation

    ISWC 2007

    Evaluation: building a gold standardGTTBrinkmanPossible Thesaurus relations (~ SKOS)

    Title of the presentation

    ISWC 2007

    User Evaluation Statistics3 evaluators with 1500 evaluations90% agreement ONLYEQIf some evaluator says "equivalent", 73% of other evaluators say the sameComparing two evaluators, correspondence in assignment is best for equivalence, followed by "No Link", "Narrower than", "Broader than", at or above 50% agreement, "Related To" has 35% agreement.There are correlations between evaluators.For example, Ev1 and Ev2 agreed much more on saying that there is no link than the Ev3.

    Title of the presentation

    ISWC 2007

    Evaluation Interpretation: What is a good mapping?

    Is use case specific. We considered:ONLYEQ: Only Equivalent answer correctNOTREL: EQ, BT,NT correctALL: EQ, BT, NT, RT correct

    ONLYEQ NOTREL ALL

    The question is obviously: do they produce the same results

    Title of the presentation

    ISWC 2007

    Evaluation: validity of the (different) methodsAnswer is: yes All evaluations produce the same results (in different scales)

    Title of the presentation

    ISWC 2007

    A remark about EvaluationUse of mappings strongly task dependant Scenario 1 (legacy data/annotation support) and Scenario 2 (thesaurus merging) require different mappings. Our evaluation is useful (correct) for Scenario 2 (intensional)Scenario 1 can be evaluated differently (e.g. cross-validation on test-data)

    See our paper at the Cultural Heritage Workshop.

    Title of the presentation

    ISWC 2007

    OverviewUse-caseInstance-based mappingEvaluationExperimentsResultsConclusions

    Title of the presentation

    ISWC 2007

    Experiments: Setup, Data and ThesauriWe calculated 5 different similarity measures withThreshold: 0 and 10Hierarchy: yes or no.

    Based on on 24.061 GTT concepts with 4.990 Brinkman concepts based on 243.886 books with double annotations

    Title of the presentation

    ISWC 2007

    Experiments: Result calculationAverage precision at similarity position i:Pi = Ngood,i/Ni (take the first i mappings, and return the percentage of correct ones)

    Example:

    This means that from the first 798 mappings 86% were correct Recall is estimated based on lexical mappingsF-measure is calculated as usual100%798th mapping86 %

    Title of the presentation

    ISWC 2007

    OverviewUse-caseInstance-based mappingEvaluationExperimentsResultsConclusions

    Title of the presentation

    ISWC 2007

    Results: Three research questions

    What is the influence of the choice of threshold?

    What is the influence of hierarchical information?

    What is the best measure and setting for instance-based mapping?

    Title of the presentation

    ISWC 2007

    What is the influence of the choice of threshold?Threshold needed for JaccardThreshold NOT needed for LLR

    Title of the presentation

    ISWC 2007

    What is the influence of hierarchical information?Results are inconclusive!

    Title of the presentation

    ISWC 2007

    Best measure and setting for instance-based mapping?10We have two winners!The corrected Jaccard measures

    Title of the presentation

    ISWC 2007

    ConclusionSummaryAbout 80% precision at estimated 80% recallSimple measures perform better, if statistical correction applied, (threshold or explicit statistical correction)Hierarchical aspects unresolvedSome measures really unsuited Future work: Generalize resultsOther use cases, web directories, Study other measures

    Title of the presentation

    ISWC 2007

    Thank you.

    Title of the presentation

    ISWC 2007

    Similarity measures Formulae

    Jaccard:

    Corrected Jaccard: assign a smaller score to less frequently co-occurring annotations.

    Title of the presentation

    ISWC 2007

    Information Theoretic MeasuresPointwise Mutual Information:Measures the reduction of uncertainty that the annotation of one concept yields for the annotation with another concept.

    -> disadvantage: inadequate for spare data

    LogLikelihoodRatio:

    Information Gain:Information gain is the difference in entropy,determine the attribute that distinguishes best between positive an negative example

    Title of the presentation

Recommended

View more >