new trends in multimedia - mpeg-21: concepts and applications 1 rik van de walleghent, april 2008...

Rik Van de Walle

Multimedia Lab

Ghent University, Belgium

e-mail: rik.vandewalle@ugent.be

web: multimedialab.ugent.be

MPEG-21: concepts and applications

media representation standardssome milestones

• Still imageso JPEG, JPEG2000, GIF, TIFF, ...

• CD-ROM storageo MPEG-1

• Digital TVo MPEG-2

• Video over PSTN and mobile networks(Public Switched Telephone Network)

o H.263• Object-based AV coding

o MPEG-4• State-of-the-art video coding

o MPEG-4 AVC/H.264o MPEG-21 Scalable Video Coding

08MPEG-1/2/4/7: technology for...

08MPEG-21: what is it about ?

• Started in October 1999

• General goalo infrastructure for delivery and consumption of MM contento 'the big picture'

• Some more specific objectiveso understand how the elements of the framework fit togethero where gaps in the framework exist: identify new standardso integration of different standards

• within MPEG

• outside MPEG

08MPEG-21: what is it about ?

A multimedia framework'to enable transparent and augmented useof multimedia resources'

08some MPEG-21 applications

• 'A billion content providers for a billion users...'

• Some exampleso digital libraries / archiving applications

(e.g., interaction with OAI)o interactive TV / broadcastingo publishingo event reporting / trade transactionso e-health (e.g., content adaptation)o mobile applications / lightweight applications

08who is the MPEG-21 User ?

• Definition within MPEG-21: Usero any entity interacting in the MPEG-21 environmento any entity using a Digital Itemo including

• individuals, organizations, corporations, communities• consortia, governments, standards bodies, ...

o User roles include• content creation, content consumption• content providers/distributors• rights holders• ...

08MPEG-21 parts

• Currently: quite a lot of parts...o Technical Report: Vision, Technologies and Strategyo Digital Item Declarationo Digital Item Identificationo Intellectual Property Management and Protectiono Rights Expression Languageo Rights Data Dictionaryo Digital Item Adaptationo Reference Software

08MPEG-21 parts

• Currently: quite a lot of parts...o File Formato Digital Item Processingo Evaluation Tools for Persistent Associationo Test Bed for MPEG-21 Resource Deliveryo Scalable Video Codingo Conformanceo Even Reportingo Binary Format

08declaring 'Digital Items' – why ?

• Currently, multimedia applications are based on transfer/processing/presentation/... of

o different media types, with different representations• still images (JPEG2000, GIF, PNG, ...)• video (MPEG-4, QuickTime, ASF, ...)• audio (WAV, MP3, ...)• text (txt, doc, ...)• ...

o metadata• descriptive information about actual data• DRM information (e.g., copyright statement)• configuration information• ...

08declaring 'Digital Items' – why ?

• Example: a digital music albumo without the concept of 'digital item declaration'

• a 'flat directory' containing a loose set of– audio files– lyrics files– a readme.txt file– a config.ini file

('to be used by advanced users only')– playTrack.exe

(only executable on a single platform/OS)– ...

• no 'structure'

declaring 'Digital Items' – structure

aria title: Nessun Dormatrack number: 04

nessunDorma.txttype: lyricscomposer: Giacomo Pucciniopera: Turandotcopyright: Ricordi & co...

nessunDorma.mp3type: audioformat: mp3duration: 200 sbitrate: 192 kbpscopyright: EMI...

aria title: O mio babbino carotrack number: 07

babbinoCaro.doctype: lyricscomposer:Giacomo Pucciniopera: Gianni Schiccicopyright: DECCA...

babbinoCaro.wavtype: audioformat: wavduration: 170 sbitrate: 128 kbpscopyright: DECCA...

ACA01039.jpgtype: album cover artformat: image/jpegsize: 300x400copyright: EMI...

title: concert recordingdate: July 2003

location: Covent Garden...

title: concert recordingdate: July 2003

location: Covent Garden...

concert.movtype: concert videotype: video/movduration: 4500 sbitrate: 500 kbpssize: 320x240copyright: DECCA...

declaring 'Digital Items' – MPEG-21 solutionbasic concepts

• A Digital Item is...o structured, with a standard representation, identification

and metadatao fundamental unit of distribution and transactiono expressed by Digital Item Declaration Language (DIDL)

• ISO/IEC 21000-2• XML schema

• Terminologyo resource individual assetso metadata info about or pertaining to the DIo structure relation between DI parts

declaring 'Digital Items' – MPEG-21 solution basic concepts

• As a result, MPEG-21 DIs are a composition of o different resources

• audio • video• textual• ...

o metadata describing• the content (e.g., MPEG-7)• terminal characteristics (e.g., MPEG-21 DIA)• rights expressions (e.g., MPEG-21 REL)

declaring 'Digital Items' – MPEG-21 solution basic concepts

resourcesresources metadatametadata

MPEG-1MPEG-2MPEG-4

MPEG-7

new resource formatsnew metadata formats(e.g., registration)

structurestructure

MPEG-21 Digital Item Declarationbasic concepts

• Digital Item Declaration Language (DIDL)o a generic container structureo a set of building blockso expressed in XML o allows declaration of any Digital Item

Descriptor

Container

Descriptor

Component

Resource

Descriptor

Component

Resource

MPEG-21 DIDexample

Prokofiev: Romeo and Juliet </Statement> </Descriptor> <Item> <Descriptor> <Statement mimeType='text/plain'>

Valery Gergiev </Statement> </Descriptor> <Component> <Resource ref='Prokofiev_RnJ.mp3' type='audio/mp3'/> </Component> </Item> … </Item></DIDL>

Prokofiev: Romeo and Juliet </Statement> </Descriptor> <Item> <Descriptor> <Statement mimeType='text/plain'>

Valery Gergiev </Statement> </Descriptor> <Component> <Resource ref='Prokofiev_RnJ.mp3' type='audio/mp3'/> </Component> </Item> … </Item></DIDL>

MPEG-21 DID Language (DIDL)building blocks

• DID technology is described in three normative sectionso model

• describes set of abstract terms and concepts• a Digital Item is the digital representation of 'a work'• a DI is the thing that is acted upon within the model• DIs are

– managed/handled/processed– described– exchanged– collected– ...

MPEG-21 DIDLbuilding blocks

o representation• DID elements are represented in XML• normative description of their syntax and semantics

o schema• normative XML schema• comprising entire grammar of the DID

• itemo grouping of sub-items and/or components that are

bound to relevant descriptorso descriptors contain information about the itemo may contain choices

• allow items to be customized or configuredo may be conditional

o NOTE• item that contains no sub-items can be considered a

logically indivisible work• item that does contain sub-items can be considered

a compilation

• containero structure allowing items and/or containers to be groupedo groupings of items and/or containers can be used to

form • logical packages (for transport or exchange)• logical shelves (for organization)

MPEG-21 DIDLbuilding blocks (example)

• componento binding of a resource to its descriptorso descriptors will typically contain control or structural

information about the resource• bit rate• character set• start points• encryption information

o NOTE • a component itself is not an item• components are building blocks of items

• descriptoro associates information with the enclosing elemento can be

• a component– thumbnail of an image– textual component

• a textual statement• resource

o individually identifiable asset• a video or audio clip• images, textual asset

o must be locatable via an unambiguous address

• statemento literal textual valueo contains information, but not an asseto examples:

• descriptive info• control info• revision tracking info

MPEG-21 DIDL building blocks (example)

<DIDL xmlns='urn:mpeg:mpeg21:2002:01-DIDL-NS'> <Container> <Item> <Item> <Component> <Descriptor> <Statement mimeType='text/plain'> Image </Statement> </Descriptor> <Resource ref='myFirstPicture.jpg' mimeType='image/jpg' /> </Component> </Item> <Item> <Component> <Resource ref='mySecondPic.bmp' mimeType='image/bmp' /> </Component> </Item> </Item> </Container></DIDL>

08MPEG-21 Digital Item Identification

• Scope: how too uniquely identify Digital Items and parts thereof

(including resources)o uniquely identify IP related to the Digital Items and parts

thereofo uniquely identify Description Schemeso use identifiers to link Digital Items with related

information such as descriptive metadatao identify different types if Digital Items

08MPEG-21 Digital Item Identification

• DII specification does NOTo specify new identification systems for content elements

for which identification and description schemes already exist

• does not attempt to replace the ISRC (as defined in ISO 3901) for sound recordings

• but allows ISRCs to be used within MPEG-21

• Identifiers can be associated with Digital Itemso by including them in a Statement element

MPEG-21 Digital Item Identification – example

MPEG-21Music Album

backcover.jpg

cover.jpg

covernotes.html

track1.aac

metadata

track2.aac

metadata

track2.aac

metadata

lyrics1.txt

metadata

lyrics2.txt

metadata

Identifier:ISRC FR-Z03-91-01234

Descriptions:lead singercomposerconcert hallconductor

Identifier:ISMN: M-2306-7118-7

Descriptions:composerlyrics writerlanguage

08MPEG-21 IPMP

• Defines how to manage and protect Digital Itemso taking IPMP in MPEG-4 as its starting pointo standardizes how IPMP tools can be retrieved from

remote locationso exchange of messages between IPMP tools and the

terminal

• Management and protection ofo rights expressionso Digital Item Adaptation descriptionso ...

08MPEG-21 REL and RDD

RELREL RDDRDD

based onXrML 2.0

based on<indecs>2rdd

consortium of companies

MPEG-21 REL and RDD are closely allignedstandards in 2003

08MPEG-21 REL and RDD

• Rights Expression Language (REL)o machine-readable languageo can declare rights and permissionso using the terms as defined in the Rights Data Dictionary

• Rights Data Dictionary (RDD)o provides a set of clear, consistent, structured,

and integrated definitions of termso terms may be used in the RELo provides framework for interoperability of terms used in

management of rights and permissions

simple/single digital objects

complex digital objects

LANL Use Casearchiving/publishing complex digital objects

Component<didl:Component>

Resource<didl:Resource>

Container<didl:Container>

<didl:Descriptor>

Item<didl:Item>

<didl:Descriptor> <didl:Descriptor>

Item<didl:Item>

Resource<didl:Resource>

Component<didl:Component>

LANL Use Case MPEG-21 DIDL – Data Model

LANL Use Case MPEG-21 DIDL – Descriptors

<didl:Descriptor> <didl:Statement mimeType='text/xml; charset=UTF-8'> <RDF:description>

<dc:title>Syntax for Semantic Enriching of Web Pages</dc:title> <dc:creator>Jeremy J. Carroll</dc:creator> <dc:subject>Metadata, Dublin Core element</dc:subject> <dc:publisher>OCLC Online Computer Library Center, Inc.</dc:publisher> <dc:format>text/html</dc:format> <dc:language>en</dc:language> <dc:date>2001-11-02</dc:date>

<RDF:description> </didl:Statement></didl:Descriptor>

RDF = Resource Description Frameworkdc = Dublin Core (metadata standard for library aplications)

LANL Use Case MPEG-21 DIDL – Rights

<didl:Item> … <didl:Descriptor> <didl:Statement mimeType='text/xml; charset=UTF-8'> <r:license xmlns:r='urn:mpeg:mpeg21:2003:01-REL-R-NS'>

<r:otherInfo> <dc:rights xmlns:dc='http://purl.org/dc/elements/1.1/'>

08MPEG-21 Digital Item Adaptation

• Description schemeso describing the environment in which digital items are usedo user preferenceso accessibility requirementso network and terminal characteristicso consumption environment (e.g., lighting conditions)

• Resource Adaptation

• Digital Item Adaptation toolso Session Mobility (see further)o (generic) Bitstream Syntax Description Language ((g)BSDL)

08MPEG-21 Digital Item Adaptation

DescriptorAdaptation

Engine

ResourceAdaptation

Engine

AdaptedDigitalItem

DigitalItem

08Digital Item Processing – why ?

• Scope of Digital Item Processingo declaration of a Digital Item

• defines 'structure' of a DI• static declaration

– what happens when a DI arrives at a terminal ?– nothing

o Digital Item Methods/Processing allow Users• to add functionality to a Digital Item Declaration

o on receipt of a DID• list of DI Methods that can be applied to the Digital Item is

presented to the User• User chooses a Method which is then executed• (poor) analogy: DVD menus

08Digital Item Processing – why ?

• Digital Item Methodso provide a way for DI Users of the DI

• to specify a selection of preferred procedures by which the DI should be handled

• at the level of the DI itselfo example:

• music album DI• 'AddTrack' DIM

o NOT intended to be utilized for implementing the processing of media resources themselves

Digital Item Processing – MPEG-21 solutionbasic concepts

• 'Methods' vs. 'Operations'o DIMs are defined as 'a list of operations'

o Digital Item Base Operations (DIBOs)• describe base operations on which DIMs are built• analogy

– DI Method = programming language– DIBOs = standard library of functions

Digital Item Processing – MPEG-21 solutionbasic concepts

• DIBOso specify high level normative interface to the basic

functionality of the MEPG‑21 Peer

• Digital Item Method Language (DIML)o specifies normative language for defining interoperable

DIMs and from which the DIBOs are able to be called

• Digital Item extension Operations (DIxO) o specify normative mechanism for enabling more

complex functionality in an efficient way

relationship DID – other parts of MPEG-21

Digital Item

DIM1DIM2DIM3...

DIME User

networklocal

storagelocal

platform

DIBO1 DIBO2 DIBO3 DIBO4 DIBO5 DIBO6

media resourcecodec

DIIengine

DIDengine

IPMPengine

RELengine

RDDengine

DIAengine

Digital Item

DIM1DIM2DIM3...

DIME User

networklocal

storagelocal

platform

DIBO1 DIBO2 DIBO3 DIBO4 DIBO5 DIBO6

media resourcecodec

DIIengine

DIDengine

IPMPengine

RELengine

RDDengine

DIAengine

...DII

engineDID

engineIPMPengine

RELengine

RDDengine

DIAengine

MPEG-21 Digital Item Processingbuilding blocks

• One DIML has been chosen: ECMAScript• One set of normative operations: DIBOs

o implemented in any language• any scripting language• any 'full' programming language• only APIs are normatively defined

o will be atomic operationso having access to MultiMedia Middleware API

(see further)o having ECMAScript bindings

• DIBOs can be called from within DIMso having DIxO bindings

• DIxOs can be called from within DIMs• DIxOs can call DIBOs as well

MPEG-21 Digital Item Processingbuilding blocks

• DIxOs have access too all normative DIBOso other DIxOs

• DIBO Extension Language:o multiple solutions possibleo already 1 solution identified: MPEG-J-based

• normative annex on how to call MPEG-J-based DIxOs

• informative annex on the execution environment for MPEG-J-based DIxOs

o include complex methods efficientlyo downloadable DIxOs are callable from DIML

Digital Item Method

DIBO DIBO DIBO DIBO

MultiMedia Middleware API (M3W API)

MPEG-21 Digital Item Processinghow does it work ?

Digital Item Method

DIBO DIBO DIBO DIBO

ECMAScript

native language

normative language

normativeAPI & semantics

non-normative implementation

Digital Item Method

DIBO DIBO DIBO DIBO DIxO DIxO DIxO

MultiMedia Middleware API (M3W API)

MPEG-21 Digital Item Processinghow does it work ?

Digital Item Method

DIBO DIBO DIBO DIBO

ECMAScript

identified platform

native language

DIxO DIxO DIxO

normative language

normativeAPI & semantics

normative platform

non-normativeAPI

normative DIxO invocation

non-normative implementation

POPE: Publish Once, Play Everywhere

08application – overview

• digital media - multimediao bits and byteso MPEG - Moving Picture Experts Group

• mobile multimedia applicationso problem statemento session mobilityo standardized solution: MPEG-21

• demo: prototype application (during lab visit)o streaming video o lightweight: PDA as a terminal

(Personal Digital Assistant)

08introduction

• The old analogue times - TV paradigmo video data modeled as a sequence of pictures with a

certain number of lineso one audio channel added to the video signalo video and audio have an analogue representationo user chooses among the available broadcast programso programs are consumed 'as is'

08mobile applications - Session Mobility

• Problem statemento increasing demand to access multimedia anywhere at

any timeo new powerful multimedia devices

• different terminal capabilities• different network capabilities

o increasing demand for• easy switching between different devices• without complex user interaction• i.e., 'session mobility'

o 'going mobile' is a key objective

08mobile applications - Session Mobility

student watchinga lesson on a set-

top box

student continuingthe session on a

finishing the e-learning session

on a tablet PC

tv/set-topbox

PocketPC

Session Transfer

Media stream

Tablet PC

Serverscontaining

onlineclasses

SM protocolssimple

• Device Ao collects information about the current sessiono stops the current session

• location of current media stream (e.g., the URI)• position in current media stream (e.g., 50 sec)• status of the current session (e.g., pause)• status of the application (e.g., full screen)

• Device A transfers session data to device B• Device B

o reconstructs session for the received data• load the media stream• go to the correct position• reconstruct the status of the session• reconstruct the status of the application

Device A Device B

2. send session data

1. collect session data

3. process session data

SM protocolssimple

dead period

end of session on device A

start of session on device B

SM protocolsadvanced

• Modified protocol to remove the dead periodo device A

• collects information about current session• does not stop the current session

o device A transfers session data to device Bo device B

• reconstructs session for the received data• requests updates of the session information• until both sessions are synchronized

o device A stops its session

SM protocolsadvanced

Device A Device B

2. send session data

1. collect session data

3. process session data

4. request updated session information

08SM and heterogeneous devices

• Different terminal capabilitieso cause difficulties for transferring sessionso example

• watching a movie encoded at CIF (352x288)• on a QCIF (176x144) display

• Different network capabilitieso example

• playing a movie encoded at 512 kbps• on a device with a 128 kbps network connection

• Requiremento common (proprietary or standardized) formato for storage of session information

• Solution: MPEG-21 technology

MPEG-21 Session Mobilityintroduction

• Provides a solution to the problems for session transfer between heterogeneous devices

o differences in capabilities • including choices that allow the configuration of a

Digital Item

o common format for the session information• standardized in Digital Item Adaptation

specification

prototype applicationarchitecture

Starting anMPEG-21multimedia

session

Resuming anMPEG-21multimedia

session

MPEG-21 Session Mobility

Content andstreaming

server

Media stream

Client A

Media stream

Client B

prototype applicationimplementation platforms

• Client A – Windows platformo .Net Framework (C#)o Pentium 4, 2.8 GHz running Windows XPo Windows Media Player 9

• Client B – Windows Mobile platformo .Net Compact Framework (C#)o iPAQ 5550, Intel XScale 400 MHz running Windows

Mobile 2003o Windows Media Player 9 for Pocket PC

prototype applicationperformance measurements

• 4 different measurementso parsing a Digital Itemo generating a Session Mobility Digital Itemo parsing a Session Mobility Digital Itemo transferring and resuming a session

prototype applicationperformance measurements

• Processing cost is insignificant on both platforms o parsing a Digital Item

• average iPAQ 219.2 mso generating a Session Mobility Digital Item

• average iPAQ 14.4 mso parsing a Session Mobility Digital Item

• average iPAQ 63.1 mso transferring and resuming a session

• average iPAQ 1270.9 ms

08conclusions

• MPEG-21 offers very generic framework for setting up and managing advanced multimedia applications

• Session Mobility between devices with different capabilitieso interoperability is a key requiremento MPEG-21 Session Mobility is offering such interoperabilityo by standardizing a format for session information

• Overhead introduced by MPEG-21o not significant compared to the consumption of the actual

multimediao MPEG-21 can easily be implemented on lightweight platforms

new trends in multimedia - mpeg-21: concepts and applications 1 rik van de walleghent, april 2008...

Documents

walle-1 - epikentro.gr · john van de walle ezÖóuets-...

tutorial illustrator walle

anillos - rik 2008

panduan rik 2014

rik houben

elect rik 22

rik and returns

walle powerpoint

rik veda sanscrito

1 mpeg-21 : goals and achievements ian burnett, rik van de...

beacons in action by thomas walle

elis – multimedia lab multimedia lab rik van de walle...

the open health data api rik smithies – rik@nprogram.co.uk...

an open data adapter with semantic output€¦ · an open...

bio list rik

lib.ugent.be · application of semantic web technologies...

romania e rik

rik-mar owners...

foundations of math de walle

welcome guide - universiteit gent...prof. dr. rik van de...