MSI

Multimedia DatabasesMultimedia Databases

Arif GhafoorArif GhafoorProfessorProfessor

Electrical and Computer EngineeringElectrical and Computer EngineeringPurdue UniversityPurdue University

22

MSI

Multimedia Database Laboratory Multimedia Database Laboratory Facilities at Purdue UniversityFacilities at Purdue University

Distributed Multimedia Systems Lab (ECE)Distributed Multimedia Systems Lab (ECE)• Ghafoor, Kashyap, KhanGhafoor, Kashyap, Khan

Multimedia Systems Lab (CS)Multimedia Systems Lab (CS)• Aref, Elmagarmid, PrabhakarAref, Elmagarmid, Prabhakar

Raid Lab (CS)Raid Lab (CS)• BhargavaBhargava

Video and Image Processing Lab (ECE)Video and Image Processing Lab (ECE)• DelpDelp

33

MSI

On-Going Research Projects in On-Going Research Projects in Multimedia DatabasesMultimedia Databases

Multimedia database managementMultimedia database management(NSF, Fuji Electric, AT&T)(NSF, Fuji Electric, AT&T)• Video modeling and managementVideo modeling and management• Multimedia document managementMultimedia document management

Distributed multimedia systemsDistributed multimedia systems(NSF, AFRL, IBM, Intel, Siemens)(NSF, AFRL, IBM, Intel, Siemens)

High-performance multimedia database High-performance multimedia database architecture for storage managementarchitecture for storage management(NSF, AT&T)(NSF, AT&T)

44

MSI

Example of Event-Based Video Example of Event-Based Video Data (Touch-down Scenario)Data (Touch-down Scenario)

55

MSI

Research Challenges for Video Research Challenges for Video DatabasesDatabases

Development of video processing techniques Development of video processing techniques for automatic object identification and motion for automatic object identification and motion tracking tracking

Development of data models with powerful Development of data models with powerful semantic expressivenesssemantic expressiveness

Designing efficient indexing, classification and Designing efficient indexing, classification and event-based searching techniques with high event-based searching techniques with high degree of precision degree of precision

Providing suitable video querying environmentProviding suitable video querying environment Addressing scalabilityAddressing scalability

66

MSI

Video Parsing and Segmentation

Object Recognition

Granularity

Lev

el o

f A

bst

ract

ion

Iconic-basedGrouping and

Browsing

Motion DetectionAnalysis

Knowledge-basedHigher-levelSemantics

FineCoarse

Low

High

Semantic Video Data Modeling Semantic Video Data Modeling ComplexityComplexity

77

MSI

RawVideo

Database

RawImage

Database

PhysicalObject

Database

Frame

Object Description

Image Features

Spatial-Semanticsof Objects

(human,building,…)

Sequence of Frames(indexed)

Object Identification and Tracking

Intra/Inter-Frame Analysis

(Motion Analysis)

Inter-Object Movement(Analysis)

Semantic Association(President, Capitol,...)

Spatio-Temporal Semantics:Formal Specification of

Event/Activity/Episode forContent-Based Retrieval

Spatial Abstraction

TemporalAbstraction

Object Definitions(Events/Concepts)

An Architecture for Video An Architecture for Video Database SystemDatabase System

88

MSIModel Querying

Mechanism &Attributes

Semantic ModelingApproch

Video ProcessingApproach

FuzzinessSupport

Zhang (HP) Visual Browsing - video parsing No

Liu (Princeton) Visual Browsing Graphical video parsing No

Tanaka (Tokyo) Visual SQL, textonly

Object-oriented Manual No

Weiss (MIT) Algebraic , textonly

Object-oriented Manual No

Golshani (ASU) Algebraic ,motion only

- Automatictracking

No

Chang (Columbia) Sketch & ImageFeatures

- Auitomatictracking

Moderate

Ghafoor (Purdue) Petri-net ,Textand motion

Spatio-temporallogic and Petri-

nets

Automatictracking

High

Comparison of Different Video Comparison of Different Video Database Management ProjectsDatabase Management Projects

99

MSI

Digestive System

Esophagus

Stomach

OmasumAbomasum

Protozoa

Salivary Glands

Small Intestine

Large Intestine

Rumen

Reticulum

Exit

Multimedia Browsing Graph

The Ruminant Digestive System

Four compartment stomach that: - utilizes complex carbohydrates such as cellulose - utilizes non-protein nitrogen sources - utilizes feeds of no nutritive value for humans

Rumen

The rumen, the first compartment of the ruminant stomach, is essentially non-functional at birth but develops rapidly when calves are given solid feeds such as hay and grain.

Sample Multimedia DocumentSample Multimedia Document

1010

MSI

Document Representation and Document Representation and Specification ComplexitySpecification Complexity

Media Types and Attributes Complexity

Structu

re C

omplexity

SGML/XML

HTMLPDF

Interleaf/Word

Framemaker

VRML

MPEG

HyTime

Style Complexity

1111

MSISpecificationSpecification

- Reliability - Resolution- Rate of Presentation- Display Area- Temporal Synchronization ( Intra/Inter )

Meta Data / User Interface

TranslationTranslation- - End-to-End End-to-End Delays Delays- Jitter Delay- Jitter Delay- Bandwidth- Bandwidth- Packet Loss- Packet Loss Rate Rate

- - Storage Storage Throughput/Throughput/ Bandwidth Bandwidth - Storage Delays - Storage Delays - Distributed - Distributed Database Database Coordination (QoS) Coordination (QoS)

- - CPU ThroughputCPU Throughput- Memory Overflow - Memory Overflow and Reliability and Reliability

- - Intrusion DetectionIntrusion Detection - Access Control - Access Control

Database SecurityOSNetwork

NegotiationNegotiation Dependency Model Analysis and QoS Adjustment

End-to-End End-to-End Run TimeRun TimeSchedulingScheduling

End-to-End Resource Allocation and Scheduling

End-to-End QoP / QoS ManagementEnd-to-End QoP / QoS Management

1212

MSI

Multimedia Document ManagementMultimedia Document Management

Quality of presentation requirements Quality of presentation requirements • Resolution, reliability, rateResolution, reliability, rate

Synchronization requirementsSynchronization requirements• Temporal, spatial, and logical structure Temporal, spatial, and logical structure

specificationspecification Media processing requirementsMedia processing requirements

• Coloring, enhancements, dubbing, etc.Coloring, enhancements, dubbing, etc. Security attributesSecurity attributes

1313

MSI

Text

Video Image

Text Video

AudioAudio

1

2

3

4

Parameter SymbolDuration t

Size sDeadline p

Allowable Skew aDisplay Area A

Rate rReliability wContents

Petri-Net Based Model for Synchronization and Petri-Net Based Model for Synchronization and Quality Specification of Multimedia DocumentsQuality Specification of Multimedia Documents

1414

MSI

Rumen

Reticulum

Digestive System

Esophagus

Stomach

OmasumAbomasum

Protozoa

Salivary Glands

Small Intestine

Large Intestine

Image

Audio

Video

Text

Audio

Image

Image

Text

Text

Text

Multimedia Browsing Multimedia Browsing EnvironmentEnvironment

1515

MSI

Research Issues for Quality and Performance of Research Issues for Quality and Performance of Multimedia Document Management SystemsMultimedia Document Management Systems

Document modeling Document modeling (Dimensionality/Flexibility/Portability)(Dimensionality/Flexibility/Portability)• Languages (SGML/XML)Languages (SGML/XML)• Graphical (OCPN)Graphical (OCPN)

Transformation among models and heterogeneity Transformation among models and heterogeneity management of modelsmanagement of models

Development of efficient search and indexing Development of efficient search and indexing techniques for multimedia documents using these techniques for multimedia documents using these modelsmodels

Clustering of documents Clustering of documents • Based on contents for fast search Based on contents for fast search • Controlled accessControlled access

1616

MSI

Model Spatial/TemporalComposition

Facility

LogicalStructuring

Content &QoP

SpecificationFacilty

ConsistencyChecking forScheduling

MetaSchema

TranslationComplexity

StoragePlacement

Support

NetworkInterfaceSupport

HyTime/SGML

Both Yes Content Only No Difficult No No

Object-Oriented(VODAK)

Both Yes Both No Easy No No

OCPN Both Partial Both Yes Easy Yes Yes

Comparison of Different Comparison of Different Multimedia Document ModelsMultimedia Document Models

1717

MSIUser

Multimedia

Structuring Module

OODBMS and MQL Distributed Query Processor

HyTime/SGML/XML Translator

HyTime/SGMLXML Parser --> Multimedia Query Language

Document GeneratorTool Library

Temporal Synchronization Manager

Multimedia Meta-Data

VideoDatabase

TextDatabase

ImageDatabase

AudioDatabase

To Presentation Device

Software Architecture of a Multimedia Software Architecture of a Multimedia Database Management System Database Management System

1818

MSI

Presentation Device

AudioDBMS

Network A Network B

Image

Audio

Audio

TextVideoDBMSDBMS

DBMS

DBMS

DBMS

Distributed Multimedia Database Distributed Multimedia Database SystemsSystems

1919

MSI

Research Issues in Distributed Research Issues in Distributed Multimedia Database ManagementMultimedia Database Management

Distributed Object Management Distributed Object Management Efficient distributed query processing Efficient distributed query processing

techniquestechniques Network caching techniques for Network caching techniques for

composition of distributed objectscomposition of distributed objects Adaptive data filtering techniques to Adaptive data filtering techniques to

satisfy diverse user QoS requirementssatisfy diverse user QoS requirements

2020

MSI Multimedia Database Client

Visual Tool for Multimedia Document

Generation

Multimedia Database Interface

MultimediaPresentation Subsystem

Multimedia Database Server

MultimediaDatabase Server

MultimediaDatabase Server

...Text Image Video Audio

MetaData

DatabaseManagement

System

Media ServerSubsystem

Database Connectivity

Integrated Multimedia Information Server

Dir

ecto

ryM

anag

emen

t

Mul

tim

edia

Met

a D

ata

Man

agem

ent Distributed Query

ProcessorAPI for SBS Network

API for SBS Network

API for SBS Network

Architecture of a Distributed Architecture of a Distributed Multimedia Database ManagementMultimedia Database Management

2121

MSI

Planned ExperimentsPlanned Experiments

Development of video database Development of video database management system ( using database management system ( using database compute servers and storage system)compute servers and storage system)

Development of multimedia document Development of multimedia document management system management system

Development of distributed multimedia Development of distributed multimedia object management system using SBS object management system using SBS architecturearchitecture