isslod2011 - semantic multimedia

Semantic MultimediaIndian Summer School on Linked Data

Leipzig, 15 Sep. 2011

Dr. Harald SackHasso-Plattner-Institut for IT-Systems Engineering

University of Potsdam

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011

■ HPI was founded in October 1998 as a Public-Private-Partnership

■ HPI Research and Teaching is focussed onIT Systems Engineering

■ 10 Professors and 100 Scientific Coworkers■ 450 Bachelor / Master Students ■ HPI is winner of CHE-Ranking 2010

http://hpi.uni-potsdam.de/

http://hpi.uni-potsdam.de

http://hpi.uni-potsdam.de


■ Research Topics□ Semantic Web Technologies□ Ontological Engineering□ Information Retrieval□ Multimedia Analysis & Retrieval□ Social Networking□ Data/Information Visualization

■ Research Projects

Semantic Technologies & Multimedia Retrieval


Overview(1) Multimedia and Semantics(2) Multimedia Metadata and Ontologies(3) Semantic Multimedia Analysis(4) Semantic Multimedia Retrieval

Semantic MultimediaIndian Summer School on Linked Data, Leipzig, 15 Sep. 2011


1. Multimedia and Semantics

Communication is the activity of conveying meaningful information


Sender

Information

Encoding

Message

Receiver

Information

Decoding

Message

Channel


Claude E. Shannon: ,A mathematical theory of communication‘, Bell System Technical Journal, vol. 27, pp. 379–423, 623-656, July, October, 1948

Shannon‘s Model of Communication

Claude E. Shannon(1916-2001)

http://plan9.bell-labs.com/cm/ms/what/shannonday/shannon1948.pdf


http://en.wikipedia.org/wiki/Bell_System_Technical_Journal



Sender

Information

Encoding

Message

Receiver

Information

Decoding

Message

Channel


Claude E. Shannon: ,A mathematical theory of communication‘, Bell System Technical Journal, vol. 27, pp. 379–423, 623-656, July, October, 1948

Shannon‘s Model of Communication

Claude E. Shannon(1916-2001)

Media





Sender

Information

Encoding

Receiver

Information

Decoding


Message Message

Channel


Media


Message Message

Channel


Media

MEDIA: In communications, media (singular medium) are the storage and transmission channels or tools used to store and deliver information or data.

http://en.wikipedia.org/wiki/Data_storage_device

http://en.wikipedia.org/wiki/Data_storage_device

http://en.wikipedia.org/wiki/Data_transmission

http://en.wikipedia.org/wiki/Data_transmission

http://en.wikipedia.org/wiki/Recording

http://en.wikipedia.org/wiki/Recording

http://en.wikipedia.org/wiki/Information

http://en.wikipedia.org/wiki/Information

http://en.wikipedia.org/wiki/Data

http://en.wikipedia.org/wiki/Data


TEXT: In literary theory, a text is a coherent set of symbols that transmits some kind of informative message.

http://en.wikipedia.org/wiki/Literary_theory

http://en.wikipedia.org/wiki/Literary_theory

http://en.wikipedia.org/wiki/Symbol

http://en.wikipedia.org/wiki/Symbol



Text


TEXT: In literary theory, a text is a coherent set of symbols that transmits some kind of informative message. Images

Text


Text


Image

Text


Video / Audio

Image

Text


Video / Audio

Image

Text

InteractiveElements



Media



Media

time-independent

text

image



Media

time-dependent

audio

video / animation

time-independent

text

image



One Small Step ...This video shows Neil Armstrong climbing down the lunar module ladder to the lunar surface. The video compares existing footage with the partially restored video. The thumbnail image shows the new footage on the left and the old on the right.

• Information is encoded in media content• Media content contains implicite semantics


SEMANTIC MULTIMEDIA facilitates • explicite semantic annotation • of multimedia content • on different levels of abstraction w.r.t.

• time, • space, and • provenance.




dbpedia:Neil_Armstrong

Text

(1)Identify media fragment(2)Annotate with explicite semantics




dbpedia:Astronautdbpedia:Flag

Video

(1)Identify media fragment(2)Annotate with explicite semantics


2. Multimedia Metadata and Ontologies



Video





Video

How can we put (semantic) metadata at the appropriate place within the media?



What is ,Metadata‘?



What is ,Metadata‘?„Metadata is defined as data providing information about one or more aspects of the data“ (informal Definition, Wikipedia)




„Metadata is structured, encoded data that describe characteristics of information-bearing entities to aid in the identification, discovery, assessment, and management of the described entities.“ (W.R. Durell, 1985)




„Metadata is structured, encoded data that describe characteristics of information-bearing entities to aid in the identification, discovery, assessment, and management of the described entities.“ (W.R. Durell, 1985)

„Metadata is machine understandable information about web resources or other things.“ (T.Berners-Lee, Axioms of Web Architecture: Metadata, W3C, 1997)

http://www.w3.org/DesignIssues/Metadata

http://www.w3.org/DesignIssues/Metadata

Metadata



•Simple example: bibliographic metadata

Identification viaISBN / ISSNauthor(s)titel...

Classification viacategorieskeywordsabstract...

Structured Metadata• name-value pairs (e.g. author=‘Ernest Hemingway‘)

• typed (e.g. author is of type string)

• Meaning (semantics) of structured data is only implicite, i.e. it relies on mutual agreement about the proper usage of the data (e.g. Standardization for Dublin Core)



• Title: A name given to the resource. • Creator: An entity primarily responsible for making the resource. • Subject: The topic of the resource. • Description: An account of the resource. • Publisher: An entity responsible for making the resource available. • Contributor: An entity responsible for making contributions to the

resource.....

http://dublincore.org/documents/dces/



Structured Metadata• can also be structured hierarchically



Systema Naturae (1735)

Carl von Linné(1707-1787)

Structured Metadata• Classification Systems, as e.g. Dewey Decimal Classification



DDC 23 (2011)• 4 volumes• >4.000 pages• >45.000 classes• >96.000 registered terms

DDC 1 (1876)• 44 pages

10 Main DDC Classes000 Computer science, information & general works100 Philosophy & psychology200 Religion300 Social sciences400 Language500 Science600 Technology700 Arts & recreation800 Literature900 History & geography

Melvil Dewey(1851-1931)

http://www.oclc.org/dewey/



Unstructured Metadata• Text based metadata without a predefined structure, where the meaning

(semantics) is determined implicitely by the (natural language) content. • e.g. abstract/summary



Melville Louis Kossuth (Melvil) Dewey was an American librarian and educator, inventor of the Dewey Decimalsystem of library classification, and a founder of the Lake Placid Club.. Dewey was born in Adams Center, New York, the fifth and last child of Joel and Eliza Greene Dewey. He attended rural schools and determined early that his destiny was to be a reformer in educating the masses. At Amherst College he belonged to Delta Kappa Epsilon, earning a bachelor's degree in 1874 and a master's in 1877....

http://en.wikipedia.org/wiki/Dewey_Decimal_Classification

http://en.wikipedia.org/wiki/Dewey_Decimal_Classification

http://en.wikipedia.org/wiki/Lake_Placid_Club

http://en.wikipedia.org/wiki/Lake_Placid_Club

http://en.wikipedia.org/wiki/Adams_Center,_New_York

http://en.wikipedia.org/wiki/Adams_Center,_New_York

http://en.wikipedia.org/wiki/Amherst_College

http://en.wikipedia.org/wiki/Amherst_College

http://en.wikipedia.org/wiki/Delta_Kappa_Epsilon

http://en.wikipedia.org/wiki/Delta_Kappa_Epsilon


Authoritative vs. non-authoritative Metadata

Authoritative Metadataare generated by a reliable (authoritative) source, as e.g. • the author of the original information• a certified expert

Non-authoritative Metadataare created by an unreliable source, as e.g.

• the user• Social Tagging Systems


Collaborative Annotation -- Social Tagging


AuthorRessource

Users

authoritativeMetadata

apple

fruit

non-authoritativeMetadata

tasty

apple

fruit

breakfast

to buy © E.C. Publications, Inc.


Collaborative Annotation -- Folksonomies


http://www.wordle.net/


http://www.wordle.net

http://www.wordle.net



Semantic Metadata• can be structured or semi-structured metadata• the semantics of metadata is defined explicitely in a formal way (Ontologies) and

therefore machine readable (as well as machine understandable)

"An ontology is an explicit, formal specification of a shared conceptualization. The term is borrowed from philosophy, where an Ontology is a systematic account of Existence. For AI systems, what ‘exists’ is that which can be represented.“ (Thomas R. Gruber, 1993)

conceptualization: abstract model (domain, relevant terms, relations)explicit: semantics of all terms must be definedformal: machine understandableshared: consensus about ontology

Example for Semantic Metadata


publication




publication

• titel

• keywords

• ...

properties




publication

book

is a

• titel

• keywords

• ...

properties




publication

book

is a

journal

is a

• titel

• keywords

• ...

properties




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by1..n

1..n




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

1..n

1..n




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a1..n

1..n




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a

• surname

• first name

• street...

properties

1..n

1..n




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a

• surname

• first name

• street...

properties

Springer Verlag

is a

1..n

1..n




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a

• surname

• first name

• street...

properties

Springer Verlag

is a

Harald Sack

is a

1..n

1..n




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a

• surname

• first name

• street...

properties

Springer Verlag

is a

Harald Sack

is a

Internetworkingis a

1..n

1..n




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a

• surname

• first name

• street...

properties

Springer Verlag

is a

Harald Sack

is a

Internetworkingis a

1..n

1..n

male

femaleis a

is a




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a

• surname

• first name

• street...

properties

Springer Verlag

is a

Harald Sack

is a

Internetworkingis a

1..n

1..n

male

femaleis a

is a

≠




publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a

• surname

• first name

• street...

properties

Springer Verlag

is a

Harald Sack

is a

Internetworkingis a

1..n

1..n

male

femaleis a

is a

≠


entity



publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a

• surname

• first name

• street...

properties

Springer Verlag

is a

Harald Sack

is a

Internetworkingis a

1..n

1..n

male

femaleis a

is a

≠


entity

class



publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a

• surname

• first name

• street...

properties

Springer Verlag

is a

Harald Sack

is a

Internetworkingis a

1..n

1..n

male

femaleis a

is a

≠


entity

class

relation



publication

book

is a

journal

is a

publisherpublishes

• titel

• keywords

• ...

properties

Autorwrites

is written by

Personis a

address

has a

• surname

• first name

• street...

properties

Springer Verlag

is a

Harald Sack

is a

Internetworkingis a

1..n

1..n

male

femaleis a

is a

≠


entity

class

relation

axiom



Semantic Metadata• enable the definition of formal Axioms

• e.g. „It is not possible that the publishing date is earlier than the birth date of the author of the publication.“

• enable deduction of new facts• e.g. „All men are mortal.“

„Socrates is a man.“ „Therefore Socrates is mortal.“

• semantic Metadata enable to make implicitely giveninformation explicite with the help of deduction andinference

Raffael: The School of Athens, 1510


Multimedia Metadata Description Languages• for time-based media

• annotatation of temporal media fragments


time


Multimedia Metadata Description Languages• for media with spatial extend

• annotatation of spatial media fragments


metadata

metadata


Multimedia Metadata• MPEG-7

The MPEG-7 standard, formerly named „Multimedia Content Description Interface“, provides a rich set of standard tools to describe multimedia content. Both human users and automatic systems that process audiovisual information are within the scope of MPEG-7.

• Components of the MPEG-7 Standard• MPEG-7 Systems• MPEG-7 Description Definition Language• MPEG-7 Visual• MPEG-7 Audio• MPEG-7 Multimedia Description Schemes MDS• MPEG-7 Reference Software• MPEG-7 Conformance• MPEG-7 Extraction and Use of Descriptions



MPEG-7 Description of a Video Segment <Mpeg7 xmlns="..."><Description xsi:type="ContentEntityType"> ... <Video> <TemporalDecomposition> <VideoSegment> <CreationInformation>...</CreationInformation> <TextAnnotation> <KeywordAnnotation> <Keyword>mouse</Keyword> </KeywordAnnotation> </TextAnnotation> <MediaTime> <MediaTimePoint>T00:05:05:0F25</MediaTimePoint> <MediaDuration>PT00H00M31S0N25F</MediaDuration> </MediaTime> </VideoSegment> </TemporalDecomposition> </Video> ... </Description></Mpeg7>



MPEG-7 Description of a Still Image



MPEG-7 and the Semantic Web• MDS Upper Layer represented in RDF(S)

(2001: Hunter, later with link to ABC upper ontology)• MDS fully represented in OWL-DL

(2004: Tsinaraki et al., DS-MIRF model)• MPEG-7 fully represented in OWL-DL

(2005: Garcia & Celma, Rhizomik model)• MDS and Visual Parts represented in OWL-DL

(2007: Arndt et al., COMM model, re-engineering of MPEG-7 with DOLCE design patterns)



Example: Tagging with an MPEG-7 Ontology


Reg1

• Localize a region → Draw a bounding box

• Annotate the content → Interpret the content → Tag ,Astronaut‘

:Reg1 foaf:depicts dbpedia:Astronaut


Example: Tagging with an MPEG-7 Ontology


Reg1

mpeg7:image

mpeg7:depicts

Man on the Moon

mpeg7:spatial_decomposition Reg1

mpeg7:StillRegion

rdf:type

mpeg7:depicts

dbpedia:Astronaut

mpeg7:SpatialMask

mpeg7:polygon

mpeg7:Coords


Media Fragment Identification• Multimedia data has temporal and spatial dimension• pinpoint access on media fragments (on the web) with media fragment

identifiers• (W3C Media Fragments URI 1.0, Juli 2009, Working Draft)• simple examples

• requires different handling of media data by http client-server transactions


http://www.example.com/example.ogg#track=‘audio‘

http://www.example.com/example.ogg#track=‘audio‘&t=10s,20s

http://www.example.com/example.ogg#track=‘video‘&xywh=160,120,320,240

http://www.example.com/example.ogg#track=







How do we find something in a Multimedia Archive?


How does Google find a video?


How do you find something in an audiovisual archive?


Step 1: Digitalization of analogue data



Step 1: Digitalization of analogue data


Step 2: Annotation with (textbased) metadata


• Manual annotation of AV-content with descriptive metadata



...can this also be achieved in an automated way?


Automated AV-Media Analysis

automated content-based analysis is•difficult (error prone) and•complex




Genre Analysis




Face Detection

Genre Analysis




Face Detection

overlay text

Genre Analysis




Face Detection

overlay text

Logo Detection

Genre Analysis




Face Detection

overlay text

Logo Detection

Genre Analysis

scenetext




Face Detection

overlay text

Logo Detection

Genre Analysis

scenetext{

Audio-Mining

structuralanalysis transcription speaker

identification


• Result: Video segments with time-based metadata annotations

• Metadata consist of combined low level / high level feature descriptors• Metadata serve as a basis for traditional and semantic retrieval

Metadata Extractiontime



time

e.g., person xylocation yzevent abc

e.g., bibliographical data,geographical data,encyclopedic data, ..

Video Analysis /Metadata Extraction

Entity Recognition/ Mapping

Semantic Multimedia Analysis


Some Examples of Automated Video Analysis


• Structural Analysis• Intelligent Character Recognition (ICR)

• Character/Logo Detection• Character Filtering• Character Recognition

• Audio Analysis • Speaker Detection • Automated Speech Recognition (ASR)

• Genre Analysis / Categorization•graphic / real• indoor / outdoor•day / night•...

• Face/Body Detection, Tracking & Clustering


video

• Automated subdivision of AV media data by structural segmentation• Subdivision of data streams in contentual coherent segments

Structural Analysis


video

scenes


Structural Analysis


video

scenes

shots


Structural Analysis


video

scenes

shots

subhots


Structural Analysis


video

scenes

shots

subhots

frames


Structural Analysis


shots

• Shot Boundary Detection

• Identification of• Hard Cuts• Drop Outs• Soft Cuts, as e.g., Dissolve, Wipe, Cross-Fade, etc.

Analytical Shot Boundary Detection• Analysis of Luminance/Chrominance Histograms• Analysis of Edge Distribution• Analysis of Motion Vectors

Machine Learning• Classification of Hard/Soft Cuts based on Image Features• K-Nearest Neighbor• Random Forrest • Support Vector Machines

Histogram Difference Analysis

Motion Vector Analysis

Structural Analysis


shots


• Identification of• Hard Cuts

91930 91931 91932919299192891927

Feature Analysis• Luminance Histogram Difference• Chrominance Histogram Difference• Edge Distribution

Structural Analysis


shots


• Identification of• Hard Cuts• Drop Outs

Drop Out

Histogram/Chrominance Difference Analysis

Structural Analysis


shots



Fade Out

Fade In

Structural Analysis


shots



Analytical Shot Boundary Detection• Analysis of Luminance/Chrominance Histograms• Analysis of Edge Distribution• Analysis of Motion Vectors

Machine Learning• Classification of Hard/Soft Cuts based on Image Features• K-Nearest Neighbor• Random Forrest • Support Vector Machines

Histogram Difference Analysis

Motion Vector Analysis

Structural Analysis


Face-DetectionFace ClusteringFace Tracking

Character DetectionCharacter Recognition

Logo-Detection

Genre Detection



Intelligent Character Recognition• Preprocessing• Character Identification• Text Preprocessing

• Text Filtering• Adaption of script geometry (Deskew)• Image quality enhancement

• Optical Character Recognition (OCR)• Standard OCR software (OCRopus)

• Postprocessing• Lexical analysis • Statistical / context based filtering Ermittlungen nach

Bombenfunden



• Preprocessing• Character Identification

Filtering• Local Binary Patterns (LBP)• Histogram of Oriented Gradients

Intelligent Character Recognition


Intelligent Character Recognition• Preprocessing

• Character Identification• Text Preprocessing

• Text Filtering• Adaption of script geometry (Deskew)• Image quality enhancement

• Optical Character Recognition (OCR)• Standard OCR software (OCRopus)

• Postprocessing• Lexical analysis • Statistical / context based filtering Ermittlungen nach

Bombenfunden

Automatisierte Audio- und Videoanalyse


Original Image Bounding Box



Advanced Image Enhancement



Standard OCR (OCRopus)



Context-based Spell Correction



timeVideo Analysis /Metadata Extraction






Named Entity Recognition

Main Problem in NER: Ambiguity of Terms




jaguar

Example: „Jaguar“ in different contexts



rainforest


jaguar




rainforest

Steve McQueen


jaguar




rainforest

Steve McQueen


jaguar


apple



rainforest

Steve McQueen


jaguar


Context matters!

apple


Named Entity Recognition• Mapping keyterms (text) to semantic entities

• Context Analysis and Disambiguation





JaguarKeyterm / User Tag





JaguarKeyterm / User Tag


Jaguar (Car)

Jaguar (Cat)

Jaguar (OS)

Jaguar (Aircraft)

?

?

?

?

Semantic Entities


RDF graph to find relations between entities co-occurringin a text maintaining the hypothesis that disambiguationof co-occurring elements in a text can be obtained byfinding connected elements in an RDF graph [7]. In orderto regard the special compilation of non-textual data, staticand user-genrated metadata in audio-visual content our novelapproach combines the use of semantic technologies andLinked Data with linguistic methods.

III. METHOD

According to a study about structure and characteristicsof folksonomy tags [8] an average of 83% of user-generatedtags are single terms. Also, an average of 82% of thereviewed tags are nouns. Based on these study results, weignore tag practices, such as camel case (”barackObama”)and treat tags as subjects or categories describing a resource.As a tag could also be part of a group of nouns representingan entity or a name (”flying machine”,”albert einstein”) thetags stored as single words without any given order have tobe combined in term groups of two or more terms to findall appropriate entities. Hence, every tag or group of tagswithin a given context may represent a distinct entity. Theterm combination process and subsequent mapping of termsand term groups to entities are described in sect. III-B.

To disambiguate ambiguous terms we combine two meth-ods: a co-occurences analysis of the terms in the context inWikipedia articles and an analysis of the page link graph ofthe Wikipedia articles of entity candidates. The scores forboth analysis steps are calculated to a total score.

A. Context Definition

Metadata exists in a certain context and has to be inter-preted according to this context. For tags of audio-visualcontent we identified two dimensions:

• temporal dimension• user-centered dimensionIn the temporal dimension a context can be defined as the

entire video, a segment or a single timestamp in the video.The user-centered dimension classifies a context by howmany users created the concerning metadata - only tags by acertain user or all tags regardless of which user. Fig. 1 showsthe combinations of the two dimensions of contexts formetadata in audio-visual content the interpretation regardingthe significance of a context.

Audio-visual content also provides the opportunity tosupply spatial information. Thus, tags in the same regionof a video frame are considered as related to each other.In the current approach we did not consider this contextdimension.

To describe our approach we use a sample context of ourtest set (see sect. IV). This sample context is composed oftags by only one user at a certain timestamp in the video.The video containing this sample context is a presentation

Figure 1. Dimensions of context definition in audio-visual content

by Dr. Garik Israelian at the TED conference3 entitled ”Howspectroscopy could reveal alien life”4. Our sample contextconsists of the tags ”hubble”, ”spitzer”, ”carbon”, ”dioxide”,”methan”, ”co2”, and ”water”.

B. Preprocessing

Term Combination: Our combination algorithm takesall tags of a specified spatio-temporal context (at a certaintimestamp/in a certain segment of a video, of a singleURL/image and generates every possible combination of atmost three terms of the context in every possible order. Inthat way we make sure to rectify groups of single termsthat belong together. We chose to generate combinationsof three words to make sure to also hit named entitiesconsisting of more than two words, such as ”public keycryptography” or ”alberto santos dumont”. About 90% ofthe DBpedia [9] labels consist of at most three words, butless than 5% consist of 4 words. Due to these numbersand performance issues we decided to limit the number ofterms to be combined to three. Subsequently in this paperby terms we will refer to single terms as well as generatedterm groups. The number c of combinations is calcultaed byc =

�jk=1

n!(n�k)! .

For our sample context containing 7 tags and at most3 terms in a combination (j = 3), 259 combinations aregenerated.

Term Mapping: The terms then have to be mapped tosemantic entities. For our approach we use entities of theLinked Open Data Cloud [10], in particular of the DBpedia,version 3.5.1.

DBpedia provides labels for the identification of distinctentities in 92 languages. We use English and German aswell as Finnish labels, as we noticed that neither English northe German labels contain important acronyms as labels, butthe Finnish language version does. As tagging users prefer tokeep it simple and short[2], resources dealing with ”DomainName System” would rather be tagged with ”DNS” than”Domain Name System”.

After simple string matching of the terms of the contextto DBpedia URIs, the URIs are revised for redirects and

3http://www.ted.com4http://yovisto.com/play/14415

Context Analysis and DisambiguationWhat defines a Context in AV-Data?

• Temporal Coherence • Spatial Coherence• Provenance




III. METHOD











B. Preprocessing


�jk=1

n!(n�k)! .









Spatial Dimension



III. METHOD











B. Preprocessing


�jk=1

n!(n�k)! .









Temporal Dimension

Spatial Dimension



III. METHOD











B. Preprocessing


�jk=1

n!(n�k)! .









User-centered Dimension

Temporal Dimension

Spatial Dimension



Preprocessing

Term Combination

Term Mapping

Entity Candidate Disambiguation

Co-Occurence Analysis

Semantic Graph Analysis

Score Calculation

1956 Stevejaguar

McQueenrim wheel

Steve McQueen ../resource/Steve_McQueen

jaguar ../resource/Jaguar_Cars

wheel rim ../resource/rim_(wheel)

1956 ../resource/1956

NER Custom Workflow



Preprocessing

Term Combination

Term Mapping

Entity Candidate Disambiguation

Co-Occurence Analysis


Score Calculation

1956 Stevejaguar

McQueenrim wheel

Steve McQueen ../resource/Steve_McQueen

jaguar ../resource/Jaguar_Cars

wheel rim ../resource/rim_(wheel)

1956 ../resource/1956

NER Custom Workflow

only if there is no spatialinformation for compositeterms available


Semantic Multimedia AnalysisNamed Entity Recognition Workflow

Term Combination

1956Steve

wheeljaguar

McQueen

rim



Term Combination

1956Stevewheel jaguar

McQueenrim



Assigning Entity Candidates

1956Stevewheel jaguar

McQueenrim

7 entity candidates

2 entity candidates

36 entity candidates

1 entity candidate


Cooccurrence Analysis

„jaguar“http://dbpedia.org/resource/Jaguar_(Cats)

http://dbpedia.org/resource





1956 wheel rimsteve mcqueen

context tags:







context tags:

score: 0.00




„jaguar“http://dbpedia.org/resource/Jaguar_Cars







context tags:







context tags:

score: 0.87





jaguarKeyterm / User Tag

LOD Cloud


1956 Stevejaguar

McQueenrim wheel

context

Jaguar (Car)Steve McQueen

1956

Jaguar (Cat)Jaguar (OS)


Searching is not always just searching...


Searching is not always just searching

a simple example:

I‘m looking for a book by Earnest Hemingway with the title ,For Whom the Bell Tolls‘ in the first German edition...“


Wem die Stunde schlägt. - Ernest H E M I N G W A Y. (Stockholm usw., Bermann-Fischer Verlag, 1941) 560 S. 8“

II 1, 2506, 34548

Searching is not always just searching


...but what if...

I really liked the book ,For Whom the Bell Tolls‘ but I have no idea what I should read next....


Exploratory Search• What, if the user does not know, which query string to use?

• What, if the user is looking for complex answers ?

• What, if the user does not know the domain he/she is looking for?• What, if the user wants to know all(!) about a specific topic?

• ...,Browsing‘ instead of ,Searching‘• ...to find something by chance• ...serendipitous findings• ...to get an overview• ...enable content based navigation


How to implement an exploratory search?


time



Video Analysis /Metadata Extraction


Semantic Video Search


Data is a precious thing and will last longer than the systems themselves. (Tim Berners-Lee) http://linkeddata.org/

The Web of Data - The Semantic Web

http://linkeddata.org



http://dbpedia.org/




dbpedia:For_Whom_the_Bell_Tolls

What facts for dbpedia:For_Whom_the_Bell_Tollsare relevant?

http://dbpedia.org/page/For_Whom_the_Bell_Tolls

DBPedia - the Semantic Wikipedia

...use heuristics


Exploratory Search

dbpedia-owl:author

dbpedia:Ernest_Hemingwaydbpedia:For_Whom_the_Bell_Tolls


Exploratory Search

dbpedia-owl:author


dbpe

dia-

owl:a

utho

r


Exploratory Search

dbpedia-owl:author


dbpe

dia-

owl:a

utho

r

dbpedia-owl:author


Exploratory Search

dbpedia-owl:author


dbpe

dia-

owl:a

utho

r

dbpedia-owl:author

dbpedia-owl:author


dbpedia-owl:author


Exploratory Search


dbpedia-owl:author


dbpedia:Raymond_Carver

dbpedia-

owl:influenced_by

Exploratory Search


dbpedia-owl:author



dbpedia-

owl:influenced_by

dbpedia:Jack_Kerouac

dbpe

dia-

owl:i

nflu

ence

d_by

Exploratory Search


dbpedia-owl:author



dbpedia-

owl:influenced_by

dbpedia:Jack_Kerouac

dbpe

dia-

owl:i

nflu

ence

d_by

dbpedia-owl:influenced_by

dbpedia:Jerome_D._Salinger

Exploratory Search


dbpedia:Jack_Kerouac dbpedia:Raymond_Carverdbpedia:Jerome_D._Salinger

Exploratory Search



dbpedia-owl:notableWork

Exploratory Search



dbpedia-owl:notableWork dbpedia-owl:notableWork

Exploratory Search



dbpedia-owl:notableWork dbpedia-owl:notableWork dbpedia-owl:notableWork

Exploratory Search


...and how does an exploratory search look like?


29

http://mediaglobe.yovisto.com:8080

http://mediaglobe.yovisto.com:8080/

http://mediaglobe.yovisto.com:8080/


2929

Semantische SuchtechnologienExplorative Suche in audiovisuellen Daten

J. Waitelonis, H. Sack, Z. Kramer, J. Hercher:Semantically Enabled Exploratory Video Search, in Proc. of Semantic Search Workshop (SemSearch10) at the 19th Int. World Wide Web Conference (WWW2010), 26-30 April 2010, Raleigh, NC, USA, 2010.

http://km.aifb.kit.edu/ws/semsearch10/Files/video.pdf




http://km.aifb.kit.edu/ws/semsearch10


http://www2010.org/www/



29











29


29

Semantische SuchtechnologienExplorative Suche in audiovisuellen Daten


29

Harald Sack, Hasso-Plattner-Institute for IT-Systems Engineering, Indian Summer School on Linked Data, Leipzig, 12-18. Sep. 2011http://bit.ly/SeMEX

http://mediaglobe.yovisto.com:8080/mggui/%23start













http://bit.ly/SeMEX

http://bit.ly/SeMEX


Contact:Dr. Harald SackHasso-Plattner-Institut für SoftwaresystemtechnikUniversität PotsdamProf.-Dr.-Helmert-Str. 2-3D-14482 Potsdam

Homepage:http://www.hpi.uni-potsdam.de/meinel/team/sack.html http://www.yovisto.com/Blog: http://moresemantic.blogspot.com/E-Mail: [email protected] [email protected]: lysander07 / biblionomicon / yovisto

Thank you for

your Attention!

http://www.hpi.uni-potsdam.de/meinel/team/sack.html

http://www.hpi.uni-potsdam.de/meinel/team/sack.html

http://www.yovisto.com

http://www.yovisto.com

http://moresemantic.blogspot.com

http://moresemantic.blogspot.com

mailto:[email protected]




isslod2011 - semantic multimedia

Technology

linked dataleipzig

harald sackhassoplattner

imagesharald sack

semanticsmediaharald

image textharald sack

2011overview1 multimedia

semantics2 multimedia

kind of informative