![Page 1: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/1.jpg)
Knowledge-Driven Video Information Retrieval with LOD
Leslie F. Sikos, Ph.D., Flinders University
ESAIR ’15, 23 October 2015 Melbourne, VIC, Australia
![Page 2: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/2.jpg)
Knowledge-Driven Video IR
• Video Retrieval Challenges and Limitations
• Unstructured vs. Structured Annotations
• Bridging the Semantic Gap
• Multimedia Vocabularies and Ontologies
• Linked (Open) Data for multimedia
• Standardization of Video Annotations
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Outline
![Page 3: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/3.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Unstructured vs. Structured Data
Plain text
tags
XML/XSD
metadata
RDFS/OWL
annotations
![Page 4: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/4.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Vocabularies and Ontologies
• Dublin Core Concept definitions
• Creative Commons Classes, properties,
• Schema.org individuals
… Relationships
Rules
![Page 5: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/5.jpg)
Video Object Representation
• HTML5 Microdata
• JSON-LD
• RDFa
• Microformats
Lightweight Annotations
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
![Page 6: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/6.jpg)
Video Object Representation Lightweight Annotations
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
![Page 7: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/7.jpg)
Video Object Representation Lightweight Annotations
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
![Page 8: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/8.jpg)
Video Object Representation
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Vocabulary Standard Language
MPEG-7 ISO/IEC 15938 XSD
MPEG-21 ISO/IEC 21000 XSD
NewsML IPTC NewsML-G2 XSD
TV-Anytime ETSI TS 102 822 XSD
Semi-Structured Vocabularies
![Page 9: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/9.jpg)
Video Object Representation
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Vocabulary Standard Language
MPEG-7 ISO/IEC 15938 XSD
MPEG-21 ISO/IEC 21000 XSD
NewsML IPTC NewsML-G2 XSD
TV-Anytime ETSI TS 102 822 XSD
Semi-Structured Vocabularies
![Page 10: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/10.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
XSD to RDFS/OWL Mapping
MPEG-7
COMM
SWIntO
MPEG-7Ontos
…
![Page 11: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/11.jpg)
” “
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Issues Inherited from MPEG-7
• Strong focus on low-level descriptors
“color distribution feature values of an image
for red, black, and yellow still do not allow
the conclusion that the image shows a sunset”
Boll et al., 1998
![Page 12: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/12.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Issues Inherited from MPEG-7
![Page 13: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/13.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Issues Inherited from MPEG-7
![Page 14: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/14.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Issues Inherited from MPEG-7
![Page 15: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/15.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Issues Inherited from MPEG-7
![Page 16: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/16.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Issues Inherited from MPEG-7
• Strong focus on low-level descriptors
• Conceptual ambiguity
• Semantic interoperability issues
• Syntactic interoperability issues
• Structural complexity: 1,182 elements,
417 attributes, and 377 complex types
![Page 17: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/17.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Custom Ontologies
• Large Scale Concept Ontology for
Multimedia (LSCOM)
• Linked Movie Database Ontology
• Multimedia Metadata Ontology (M3O)
• Ontology for Media Resources
…
![Page 18: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/18.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Custom Ontologies
• T TBox: terminological knowledge
• A ABox: assertional knowledge
![Page 19: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/19.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Custom Ontologies
• T TBox: terminological knowledge
• A ABox: assertional knowledge
Knowledge
Base
![Page 20: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/20.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Custom Ontologies
• T TBox: terminological knowledge
• A ABox: assertional knowledge
• R RBox: role inclusion axioms
+ transitivity axioms
![Page 21: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/21.jpg)
Multimedia Ontologies
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Ontology Language DL
LinkedMDB RDFS AL
LSCOM OWL AL
M3O OWL SHIQ(D)
COMM OWL SHOIN(D)
Limited DL Expressivity
![Page 22: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/22.jpg)
Ontology Language DL
LinkedMDB RDFS AL
LSCOM OWL AL
M3O OWL SHIQ(D)
COMM OWL SHOIN(D)
Multimedia Ontologies
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Full DL Expressivity
Ontology Language DL
VidOnt OWL 2 SROIQ(D)
http://vidont.org
![Page 23: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/23.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Domain Ontologies
• OWL 2 ontologies + SWRL rules
Decidability
![Page 24: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/24.jpg)
Bridging the Semantic Gap
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Domain Ontologies
• OWL 2 ontologies + SWRL rules
Decidability
Solution: OWL 2 + DL Rules + DL-safe
rules: very expressive & still decidable
![Page 25: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/25.jpg)
Linked Data for Multimedia
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Rationale
• Global identification with URIs
• Linking to annotations or media fragments
in the Linked Data cloud
• Differentiate video objects and media
fragments from information resources
• Support access through SPARQL queries
![Page 26: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/26.jpg)
Video IR with LOD
Knowledge-Driven Video Information Retrieval with LOD ESAIR’15
Conclusions
• OWL 2 ontologies are needed
a) Alignment with standards
b) Exploit SROIQ(D) constructs
c) Define Rbox axioms
d) Use OWL 2+DL-safe rules
• LOD: video understanding, discovery, …
Advanced
inference &
reasoning
support }
![Page 27: Knowledge-Driven Video Information Retrieval with LOD · Knowledge-Driven Video Information Retrieval with LOD ESAIR’15 Issues Inherited from MPEG-7 •Strong focus on low-level](https://reader034.vdocuments.net/reader034/viewer/2022042303/5ece4f9a265e064ba527454c/html5/thumbnails/27.jpg)
Questions