Using Several Ontologies for Describing Audio-

Visual Documents:A Case Study in the Medical

Domain

Sunday 29th of May, 2005

Antoine Isaac1 & Raphaël Troncy2

Multimedia and the Semantic Web

2005/05/29 A. Isaac & R. Troncy - MSW'2005 2

Agenda

• Context and Aims• Corpus• Ontological Resources

– AV Ontology, Medical Ontology

• Annotating the Videos• Querying and Reasoning• Performing SW-inspired Reasoning• Conclusion

2005/05/29 A. Isaac & R. Troncy - MSW'2005 3

Describe AV documents

• Various uses / Different granularities– Identification, feature extraction, structural decomposition,

semantic description

• Description deep meaning cannot be accessed and processed by classical systems– Knowledge is often implicit: labels and comments in natural

language– Formalisation for description syntax, not semantics

• Formal semantics should be interesting– Reasoning with AV document descriptions– Interoperability with formal domain-specific ontologies, allowing

to mix AV and domain-related reasoning

2005/05/29 A. Isaac & R. Troncy - MSW'2005 4

Objectives

• Settle an mini-experiment to show the benefits of using semantic web technologies for annotating audiovisual content

• Show that the use of:– formal ontologies,– annotation pattern,– inference capabilities

… is highly desirable for a better access to AV content

2005/05/29 A. Isaac & R. Troncy - MSW'2005 5

Corpus

• Medicine-related TV documentaries– 30 documents, about 30 hours– 50% deal with heart (surgery) theme

• Description point of view: how AV features are used to popularize scientific notions

• Describe both the form and the content– AV-oriented parts (documentary structure)– Theme-oriented parts (medicine notions)

2005/05/29 A. Isaac & R. Troncy - MSW'2005 6

Ontological Resources

• Audio-Visual Core Ontology [Isaac & Troncy, 2004]

– Characterization of programs and sequences (AV genre)– Decomposition of programs and sequences– Ability to introduce description from the point of view of the

activities that constitute the context of AV documents• roles of people involved, way production and broadcast are

achieved, etc.

• Dual Legitimacy– Use: conceptualization grounded on observed purposes and

domain initiatives, study of 30 years of documentary practices– Conception: articulation with an upper-level ontology: DOLCE

[Gangemi, 2002]

2005/05/29 A. Isaac & R. Troncy - MSW'2005 7

Ontological Resources

<owl:Class rdf:ID="DialogSequence"> <rdfs:subClassOf rdf:resource="#SpokenSequence"/> <rdfs:subClassOf> <owl:Restriction> <owl:onProperty> <owl:ObjectProperty rdf:about="#hasParticipant"/> </owl:onProperty> <owl:minCardinality rdf:datatype="&xsd;int">2</owl:minCardinality> </owl:Restriction> </rdfs:subClassOf> </owl:Class>

2005/05/29 A. Isaac & R. Troncy - MSW'2005 8

Ontological Resources

• Extension of AV core with specific application notions– Exemplification, demonstration, etc.

• Re-use of Medical Ontologies– Menelas: domain of coronary pathologies

• Concepts dealing with heart surgery

– Alternative choices are possible• Galen (concepts dealing with surgical procedures)

• Articulation between the ontologies– No use of automatic alignment methods or tools– State by hand OWL axioms (equivalentClass)

2005/05/29 A. Isaac & R. Troncy - MSW'2005 9

Description Process

• Segmentation of the AV material– Selection of relevant documentary items

• Knowledge-based AV annotation– Documentary structure characterization– Segment content description

2005/05/29 A. Isaac & R. Troncy - MSW'2005 10

Segmenting the Videos

2005/05/29 A. Isaac & R. Troncy - MSW'2005 11

Describing the Videos

• Documentary patterns– Layered approach [Troncy, 2003]

– AV description language [Troncy & Carrive, 2004]

• Knowledge-based Annotation Process– The structure is described at the knowledge

level• Concepts and relations from the AV ontology are

manually introduced in the description

– Content description• Link to external world themes and entities

2005/05/29 A. Isaac & R. Troncy - MSW'2005 12

Describing the Videos

• Relational Indexing Pattern– Help for user: specify how concepts and relations

have to be used (annotation ‘design pattern’)– Important for ontology conception and use (with

reasoning knowledge)

• Simple pattern that can lead to complex descriptions– Recursive relational structure

2005/05/29 A. Isaac & R. Troncy - MSW'2005 13

Describing the Videos

2005/05/29 A. Isaac & R. Troncy - MSW'2005 14

Querying and Reasoning

• Example:« retrieve the programs that explain a disease and

show one of its causes »

• Need for the following inferences:– Subsumption

– Composition

Disease(x)x)CVDisease(

z),explains(x z),explains(yy)ence(x,hasSubSequ z)shows(x, z)shows(y, y)ence(x,hasSubSequ

2005/05/29 A. Isaac & R. Troncy - MSW'2005 15

Querying and Reasoning

explains

CVDisease:blueDisease

hasSubSequence

ExpertInterview:cvDiseasesItw

causes

Drawing:stenosisDrwg

hasEditingInsert

hasParticipant

Person: prof1

Program:JN_children

represents

stateOf

Heart: heart1

PulmonaryArtery: pa

part

roleExpertRole:prof1Role

explains

Stenosis:paStenosis

represents

Animated:animatedValue

hasAVFeature

expl

ains

representshasAVFeature

Explicit statementsInferred statements

2005/05/29 A. Isaac & R. Troncy - MSW'2005 16

Performing reasoning

• A layered complexity approach– RDFS: subsumption– OWL(DL): complex definitions + algebraic properties– Rules: horn clauses

• Concrete implementation– RDFS: Sesame Architecture [Broekstra, 2002]

– OWL DL: BOR Reasoner [Simov, 2002] – OWL-DLP [Grosof, 2003] + Rules : Sesame custom

inference module

2005/05/29 A. Isaac & R. Troncy - MSW'2005 17

Examples

• DL definitions

• (Composition) rule

),(),(),( zxrepresentszyrepresentsyxencehasSubSequ

))(

)((

))(

(

nRoleInstitutio

leHospitalRoalRoleProfessionleAcademicRoExpertRole

ExpertRolerolesomePersonpanthasParticisome

InterviewrviewExpertInte

2005/05/29 A. Isaac & R. Troncy - MSW'2005 18

Summary

Explicit triples

Inferred triples

All triples

RDF Model 129

AV Ontology 5231 10810 16041

Menelas Ontology

10534 26637 37171

Instances 276 1507 1783

Total 16041 38954 54995

2005/05/29 A. Isaac & R. Troncy - MSW'2005 19

Conclusion

• This experimentation:– Uses Semantic Web languages and tools for describing AV

contents– Uses different ontologies to capture the structure and the

content of the documents– Uses relational indexing pattern for the annotation

• Future work: thorough evaluation of those techniques involving real users

• A problem that cannot be generally solved: fixing a trade-off between expressivity and tractable computation– Ad hoc, according to the needs of the application targeted