large scale processing for multimedia analysis - introduction

4242 42 4224424242 42 4242Large Scale

Processing for Multimedia Analysis

Master Seminar, WIntersemester 2011/2012

Dr. Harald Sack, Dr. Peter TrögerJörg Waitelonis, Christian Hentschel,

Bernhard Quehl, Haojin Yang, Magnus KnuthHasso-Plattner-Institut for IT-Systems Engineering

University of Potsdam

Dienstag, 18. Oktober 11

Large Scale Processing for Multimedia Analysis, Master Seminar, WS 2011/2012, Harald Sack, Peter Tröger

(1) Dozenten(2) Organisatorisches(3) Worum geht es überhaupt(4) nächste Schritte

4242 42 4224424242 42 4242Large Scale Processing for

Multimedia AnalysisMaster Seminar, WIntersemester 2011/2012



■ Research Topics□ Semantic Web Technologies□ Ontological Engineering□ Information Retrieval□ Multimedia Analysis & Retrieval□ Social Networking□ Data/Information Visualization

■ Research Projects

Semantic Technologies & Multimedia Retrieval



Dr. Harald Sack

■ Senior Researcher am Hasso Plattner Institut□ Leiter der Forschungsgruppe

,Semantische Technologien & Multimedia Retrieval‘

□ Research Projects



■ Seit Februar 2010 Senior Researcher am HPIim Bereich „Verlässliche Many-Core Systeme“

■ Forschungsschwerpunkte:□ Verlässliche Systeme, Fehlervorhersage□ Skalierbare Programmierung paralleler Systeme

■ Intel Single Chip Cloud Computer (SCC)■ CiteMaster.net

MC0

MC1

MC2

MC3

System InterfaceVRC

Router

IA-32 Core0

L2$0256KB

L2$1256KB

IA-32 Core1

MPB16KB

Router Tile

2 core clusters in 6x4 2-D mesh

16B

Dr. Peter Tröger



Administratives:□Semesterwochenstunden: 4□ECTS: 6□Leistungsfeststellung: □Schriftliche Ausarbeitung zum Vortragsthema (Umfang ca. 20 Seiten)□Umsetzung einer vorgegebenen Implementierungsaufgabe im Team□Präsentation der Ergebnisse

(Zwischenpräsentation, Endpräsentation, Wochenbesprechungen)

□Projektteams mit je 2-3 Studenten bearbeiten eine Aufgabenstellung

Large Scale Processing for Multimedia Analysis




Termine:□Wöchentliches Seminargruppentreffen□Termin nach Vereinbarung

□Zwischenpräsentation der Projektergebnisse□Abschlusspräsentation der Ergebnisse□Termin in der letzten Semesterwoche





Blog zum Seminar:□ http://lsp4mat2011.blogspot.com/




http://lsp4mat2011.blogspot.com

http://lsp4mat2011.blogspot.com


Google Multimedia SearchDienstag, 18. Oktober 11


How does Google find Multimedia?


Large Scale Processing for Multimedia Analysis, Master Seminar, WS 2011/2012, Harald Sack, Peter TrögerDienstag, 18. Oktober 11


...<a href="/mission_pages/shuttle/shuttlemissions/sts134/multimedia/index.html">

<IMG WIDTH="100" ALT="Close-up view of Endeavour's crew cabin prior to docking with the International Space Station" TITLE="Close-up view of Endeavour's crew cabin prior to docking with the International Space Station" SRC="/images/content/549665main_2011-05-18_1600_100-75.jpg" HEIGHT="75" ALIGN="Bottom" BORDER="0" /></a><p><a href="/mission_pages/shuttle/shuttlemissions/sts134/multimedia/index.html">&rsaquo; STS-134 Multimedia</a></p>

...

‣Google Multimedia Search relies on link context

How does Google find Multimedia?



How do you find something in an audiovisual archive?



Step 1: Digitalization of analog data

Step 2: Annotation with (textbased) metadata

How to Search in Multimedia Archives?



How to Search in Multimedia Archives?• manual anotation with text-based

descriptive metadata



How to Search in Multimedia Archives?• manual anotation with text-based

descriptive metadata

...how to extract metadatain an automated way?



Automated Multimedia Analysis




Face Detection




Face Detection

Genre Analysis

Classification:StudioIndoor

News Show




Face Detection

overlay text

Genre Analysis


News Show




Face Detection

overlay text

Genre Analysis


News Show

scenetext




Face Detection

overlay text

Logo Detection

Genre Analysis


News Show

scenetext




Face Detection

overlay text

Logo Detection

Genre Analysis


News Show

scenetext

Audio-Mining

structuralanalysis

AutomatedSpeech

Recognitionspeaker

identification



• Visual Analysis• Structural Analysis• Intelligent Character

Recognition (ICR)• Character/Logo

Detection• Character Filtering• Character Recognition

• Genre Analysis &Categorization

• Face / Body / Object •Detection•Tracking•Clustering


• Audio Analysis • Structural Analysis • Speaker Detection • Automated Speech

Recognition (ASR)



video

• Decomposition of time-based media into meaningful media fragments of coherent content that can be used as basic element for indexing and classification

Structural Analysis



video


Structural Analysis

scenes



video


Structural Analysis

scenes

shots



video


Structural Analysis

scenes

shots

subshots



video


Structural Analysis

scenes

shots

subshots

frameskey frames



• Preprocessing• Character Identification• Text Preprocessing

• Text Filtering• Adaption of script geometry (Deskew)• Image quality enhancement

• Optical Character Recognition (OCR)• Standard OCR software (OCRopus)

• Postprocessing• Lexical analysis • Statistical / context based filtering

Ermittlungen nachBombenfunden

Intelligent Character Recognition



• Result: Video segments with time-based metadata annotations

• Metadata consist of combined low level / high level feature descriptors• Metadata serve as a basis for traditional and semantic retrieval

Metadata Extractiontime




time

e.g., person xylocation yzevent abc

e.g., bibliographical data,geographical data,encyclopedic data, ..

Video Analysis /Metadata Extraction

Entity Recognition/ Mapping

Semantic Multimedia Analysis



Anwendungsbeispiele:


http://mediaglobe.yovisto.com:8080/mggui/%23start










http://www.yovisto.com/


http://www.yovisto.com

http://www.yovisto.com

yovisto 2.0


29

http://mediaglobe.yovisto.com:8080Dienstag, 18. Oktober 11

http://mediaglobe.yovisto.com:8080/

http://mediaglobe.yovisto.com:8080/

Large Scale Processing for Multimedia Analysis, Master Seminar, WS 2011/2012, Harald Sack, Peter Trögerhttp://bit.ly/SeMEX















http://bit.ly/SeMEX

http://bit.ly/SeMEX


nächste Woche:□Kickstart - Einführung in die zugrundeliegende Technologie□Finden/Organisation der studentischen Arbeitsgruppen□Ausgabe/Zuweisung der Arbeitsthemen□Aufbereitung/Ausgabe der notwendigen Materialien

in 14 Tagen:□Treffen mit den einzelnen Arbeitsgruppen nach Vereinbarung□Einarbeitung, etc...








Thank you for

your Attention!


large scale processing for multimedia analysis - introduction

Technology