new trends in multimedia - mpeg-21: concepts and applications 1 rik van de walleghent, april 2008...
TRANSCRIPT
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
1
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
Rik Van de Walle
Multimedia Lab
Ghent University, Belgium
e-mail: [email protected]
web: multimedialab.ugent.be
MPEG-21: concepts and applications
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
2
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
media representation standardssome milestones
• Still imageso JPEG, JPEG2000, GIF, TIFF, ...
• CD-ROM storageo MPEG-1
• Digital TVo MPEG-2
• Video over PSTN and mobile networks(Public Switched Telephone Network)
o H.263• Object-based AV coding
o MPEG-4• State-of-the-art video coding
o MPEG-4 AVC/H.264o MPEG-21 Scalable Video Coding
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
3
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-1/2/4/7: technology for...
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
4
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21: what is it about ?
• Started in October 1999
• General goalo infrastructure for delivery and consumption of MM contento 'the big picture'
• Some more specific objectiveso understand how the elements of the framework fit togethero where gaps in the framework exist: identify new standardso integration of different standards
• within MPEG
• outside MPEG
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
5
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21: what is it about ?
A multimedia framework'to enable transparent and augmented useof multimedia resources'
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
6
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08some MPEG-21 applications
• 'A billion content providers for a billion users...'
• Some exampleso digital libraries / archiving applications
(e.g., interaction with OAI)o interactive TV / broadcastingo publishingo event reporting / trade transactionso e-health (e.g., content adaptation)o mobile applications / lightweight applications
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
7
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08who is the MPEG-21 User ?
• Definition within MPEG-21: Usero any entity interacting in the MPEG-21 environmento any entity using a Digital Itemo including
• individuals, organizations, corporations, communities• consortia, governments, standards bodies, ...
o User roles include• content creation, content consumption• content providers/distributors• rights holders• ...
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
8
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21 parts
• Currently: quite a lot of parts...o Technical Report: Vision, Technologies and Strategyo Digital Item Declarationo Digital Item Identificationo Intellectual Property Management and Protectiono Rights Expression Languageo Rights Data Dictionaryo Digital Item Adaptationo Reference Software
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
9
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21 parts
• Currently: quite a lot of parts...o File Formato Digital Item Processingo Evaluation Tools for Persistent Associationo Test Bed for MPEG-21 Resource Deliveryo Scalable Video Codingo Conformanceo Even Reportingo Binary Format
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
10
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08declaring 'Digital Items' – why ?
• Currently, multimedia applications are based on transfer/processing/presentation/... of
o different media types, with different representations• still images (JPEG2000, GIF, PNG, ...)• video (MPEG-4, QuickTime, ASF, ...)• audio (WAV, MP3, ...)• text (txt, doc, ...)• ...
o metadata• descriptive information about actual data• DRM information (e.g., copyright statement)• configuration information• ...
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
11
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08declaring 'Digital Items' – why ?
• Example: a digital music albumo without the concept of 'digital item declaration'
• a 'flat directory' containing a loose set of– audio files– lyrics files– a readme.txt file– a config.ini file
('to be used by advanced users only')– playTrack.exe
(only executable on a single platform/OS)– ...
• no 'structure'
declaring 'Digital Items' – structure
aria title: Nessun Dormatrack number: 04
...
aria title: Nessun Dormatrack number: 04
...
nessunDorma.txttype: lyricscomposer: Giacomo Pucciniopera: Turandotcopyright: Ricordi & co...
nessunDorma.txttype: lyricscomposer: Giacomo Pucciniopera: Turandotcopyright: Ricordi & co...
nessunDorma.mp3type: audioformat: mp3duration: 200 sbitrate: 192 kbpscopyright: EMI...
nessunDorma.mp3type: audioformat: mp3duration: 200 sbitrate: 192 kbpscopyright: EMI...
aria title: O mio babbino carotrack number: 07
...
aria title: O mio babbino carotrack number: 07
...
babbinoCaro.doctype: lyricscomposer:Giacomo Pucciniopera: Gianni Schiccicopyright: DECCA...
babbinoCaro.doctype: lyricscomposer:Giacomo Pucciniopera: Gianni Schiccicopyright: DECCA...
babbinoCaro.wavtype: audioformat: wavduration: 170 sbitrate: 128 kbpscopyright: DECCA...
babbinoCaro.wavtype: audioformat: wavduration: 170 sbitrate: 128 kbpscopyright: DECCA...
ACA01039.jpgtype: album cover artformat: image/jpegsize: 300x400copyright: EMI...
ACA01039.jpgtype: album cover artformat: image/jpegsize: 300x400copyright: EMI...
title: concert recordingdate: July 2003
location: Covent Garden...
title: concert recordingdate: July 2003
location: Covent Garden...
concert.movtype: concert videotype: video/movduration: 4500 sbitrate: 500 kbpssize: 320x240copyright: DECCA...
concert.movtype: concert videotype: video/movduration: 4500 sbitrate: 500 kbpssize: 320x240copyright: DECCA...
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
13
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
declaring 'Digital Items' – MPEG-21 solutionbasic concepts
• A Digital Item is...o structured, with a standard representation, identification
and metadatao fundamental unit of distribution and transactiono expressed by Digital Item Declaration Language (DIDL)
• ISO/IEC 21000-2• XML schema
• Terminologyo resource individual assetso metadata info about or pertaining to the DIo structure relation between DI parts
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
14
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
declaring 'Digital Items' – MPEG-21 solution basic concepts
• As a result, MPEG-21 DIs are a composition of o different resources
• audio • video• textual• ...
o metadata describing• the content (e.g., MPEG-7)• terminal characteristics (e.g., MPEG-21 DIA)• rights expressions (e.g., MPEG-21 REL)
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
15
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
declaring 'Digital Items' – MPEG-21 solution basic concepts
resourcesresources metadatametadata
MPEG-1MPEG-2MPEG-4
MPEG-7
new resource formatsnew metadata formats(e.g., registration)
structurestructure
MPEG-21 Digital Item Declarationbasic concepts
• Digital Item Declaration Language (DIDL)o a generic container structureo a set of building blockso expressed in XML o allows declaration of any Digital Item
Descriptor
Container
Descriptor
Item
Component
Resource
Descriptor
Item
Component
Resource
MPEG-21 DIDexample
<DIDL> <Item> <Descriptor> <Statement mimeType='text/plain'>
Prokofiev: Romeo and Juliet </Statement> </Descriptor> <Item> <Descriptor> <Statement mimeType='text/plain'>
Valery Gergiev </Statement> </Descriptor> <Component> <Resource ref='Prokofiev_RnJ.mp3' type='audio/mp3'/> </Component> </Item> … </Item></DIDL>
<DIDL> <Item> <Descriptor> <Statement mimeType='text/plain'>
Prokofiev: Romeo and Juliet </Statement> </Descriptor> <Item> <Descriptor> <Statement mimeType='text/plain'>
Valery Gergiev </Statement> </Descriptor> <Component> <Resource ref='Prokofiev_RnJ.mp3' type='audio/mp3'/> </Component> </Item> … </Item></DIDL>
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
18
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 DID Language (DIDL)building blocks
• DID technology is described in three normative sectionso model
• describes set of abstract terms and concepts• a Digital Item is the digital representation of 'a work'• a DI is the thing that is acted upon within the model• DIs are
– managed/handled/processed– described– exchanged– collected– ...
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
19
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 DIDLbuilding blocks
o representation• DID elements are represented in XML• normative description of their syntax and semantics
o schema• normative XML schema• comprising entire grammar of the DID
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
20
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 DIDLbuilding blocks
• itemo grouping of sub-items and/or components that are
bound to relevant descriptorso descriptors contain information about the itemo may contain choices
• allow items to be customized or configuredo may be conditional
o NOTE• item that contains no sub-items can be considered a
logically indivisible work• item that does contain sub-items can be considered
a compilation
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
21
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 DIDLbuilding blocks
• containero structure allowing items and/or containers to be groupedo groupings of items and/or containers can be used to
form • logical packages (for transport or exchange)• logical shelves (for organization)
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
22
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 DIDLbuilding blocks (example)
<DIDL xmlns='urn:mpeg:mpeg21:2002:01-DIDL-NS '> <Container> <Item> <Item> . . . </Item> <Item> . . . </Item> </Item> </Container></DIDL>
<DIDL xmlns='urn:mpeg:mpeg21:2002:01-DIDL-NS '> <Container> <Item> <Item> . . . </Item> <Item> . . . </Item> </Item> </Container></DIDL>
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
23
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 DIDLbuilding blocks
• componento binding of a resource to its descriptorso descriptors will typically contain control or structural
information about the resource• bit rate• character set• start points• encryption information
o NOTE • a component itself is not an item• components are building blocks of items
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
24
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 DIDLbuilding blocks
• descriptoro associates information with the enclosing elemento can be
• a component– thumbnail of an image– textual component
• a textual statement• resource
o individually identifiable asset• a video or audio clip• images, textual asset
o must be locatable via an unambiguous address
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
25
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 DIDLbuilding blocks
• statemento literal textual valueo contains information, but not an asseto examples:
• descriptive info• control info• revision tracking info
MPEG-21 DIDL building blocks (example)
<DIDL xmlns='urn:mpeg:mpeg21:2002:01-DIDL-NS'> <Container> <Item> <Item> <Component> <Descriptor> <Statement mimeType='text/plain'> Image </Statement> </Descriptor> <Resource ref='myFirstPicture.jpg' mimeType='image/jpg' /> </Component> </Item> <Item> <Component> <Resource ref='mySecondPic.bmp' mimeType='image/bmp' /> </Component> </Item> </Item> </Container></DIDL>
<DIDL xmlns='urn:mpeg:mpeg21:2002:01-DIDL-NS'> <Container> <Item> <Item> <Component> <Descriptor> <Statement mimeType='text/plain'> Image </Statement> </Descriptor> <Resource ref='myFirstPicture.jpg' mimeType='image/jpg' /> </Component> </Item> <Item> <Component> <Resource ref='mySecondPic.bmp' mimeType='image/bmp' /> </Component> </Item> </Item> </Container></DIDL>
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
27
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21 Digital Item Identification
• Scope: how too uniquely identify Digital Items and parts thereof
(including resources)o uniquely identify IP related to the Digital Items and parts
thereofo uniquely identify Description Schemeso use identifiers to link Digital Items with related
information such as descriptive metadatao identify different types if Digital Items
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
28
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21 Digital Item Identification
• DII specification does NOTo specify new identification systems for content elements
for which identification and description schemes already exist
• does not attempt to replace the ISRC (as defined in ISO 3901) for sound recordings
• but allows ISRCs to be used within MPEG-21
• Identifiers can be associated with Digital Itemso by including them in a Statement element
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
29
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 Digital Item Identification – example
MPEG-21Music Album
backcover.jpg
cover.jpg
covernotes.html
track1.aac
metadata
track2.aac
metadata
track2.aac
metadata
lyrics1.txt
metadata
lyrics2.txt
metadata
Identifier:ISRC FR-Z03-91-01234
Descriptions:lead singercomposerconcert hallconductor
Identifier:ISMN: M-2306-7118-7
Descriptions:composerlyrics writerlanguage
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
30
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21 IPMP
• Defines how to manage and protect Digital Itemso taking IPMP in MPEG-4 as its starting pointo standardizes how IPMP tools can be retrieved from
remote locationso exchange of messages between IPMP tools and the
terminal
• Management and protection ofo rights expressionso Digital Item Adaptation descriptionso ...
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
31
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21 REL and RDD
RELREL RDDRDD
based onXrML 2.0
based on<indecs>2rdd
consortium of companies
MPEG-21 REL and RDD are closely allignedstandards in 2003
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
32
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21 REL and RDD
• Rights Expression Language (REL)o machine-readable languageo can declare rights and permissionso using the terms as defined in the Rights Data Dictionary
• Rights Data Dictionary (RDD)o provides a set of clear, consistent, structured,
and integrated definitions of termso terms may be used in the RELo provides framework for interoperability of terms used in
management of rights and permissions
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
33
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
simple/single digital objects
complex digital objects
LANL Use Casearchiving/publishing complex digital objects
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
34
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
Component<didl:Component>
Resource<didl:Resource>
Container<didl:Container>
<didl:Descriptor>
Item<didl:Item>
<didl:Descriptor> <didl:Descriptor>
<didl:Descriptor> <didl:Descriptor>
<didl:Descriptor> <didl:Descriptor>
Item<didl:Item>
Item<didl:Item>
Resource<didl:Resource>
Resource<didl:Resource>
Resource<didl:Resource>
Component<didl:Component>
Component<didl:Component>
LANL Use Case MPEG-21 DIDL – Data Model
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
35
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
LANL Use Case MPEG-21 DIDL – Descriptors
<didl:Descriptor> <didl:Statement mimeType='text/xml; charset=UTF-8'> <RDF:description>
<dc:title>Syntax for Semantic Enriching of Web Pages</dc:title> <dc:creator>Jeremy J. Carroll</dc:creator> <dc:subject>Metadata, Dublin Core element</dc:subject> <dc:publisher>OCLC Online Computer Library Center, Inc.</dc:publisher> <dc:format>text/html</dc:format> <dc:language>en</dc:language> <dc:date>2001-11-02</dc:date>
<RDF:description> </didl:Statement></didl:Descriptor>
RDF = Resource Description Frameworkdc = Dublin Core (metadata standard for library aplications)
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
36
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
LANL Use Case MPEG-21 DIDL – Rights
<didl:Item> … <didl:Descriptor> <didl:Statement mimeType='text/xml; charset=UTF-8'> <r:license xmlns:r='urn:mpeg:mpeg21:2003:01-REL-R-NS'> <!-- optionally, specific rights can be added here.-->
<r:otherInfo> <dc:rights xmlns:dc='http://purl.org/dc/elements/1.1/'>
Copyright2003; American Physical Society</dc:rights> </r:otherInfo> </r:license> </didl:Statement> </didl:Descriptor> …</didl:Item> MPEG-21 r:license
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
37
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21 Digital Item Adaptation
• Description schemeso describing the environment in which digital items are usedo user preferenceso accessibility requirementso network and terminal characteristicso consumption environment (e.g., lighting conditions)
• Resource Adaptation
• Digital Item Adaptation toolso Session Mobility (see further)o (generic) Bitstream Syntax Description Language ((g)BSDL)
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
38
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08MPEG-21 Digital Item Adaptation
DescriptorAdaptation
Engine
ResourceAdaptation
Engine
AdaptedDigitalItem
DigitalItem
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
39
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08Digital Item Processing – why ?
• Scope of Digital Item Processingo declaration of a Digital Item
• defines 'structure' of a DI• static declaration
– what happens when a DI arrives at a terminal ?– nothing
o Digital Item Methods/Processing allow Users• to add functionality to a Digital Item Declaration
o on receipt of a DID• list of DI Methods that can be applied to the Digital Item is
presented to the User• User chooses a Method which is then executed• (poor) analogy: DVD menus
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
40
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08Digital Item Processing – why ?
• Digital Item Methodso provide a way for DI Users of the DI
• to specify a selection of preferred procedures by which the DI should be handled
• at the level of the DI itselfo example:
• music album DI• 'AddTrack' DIM
o NOT intended to be utilized for implementing the processing of media resources themselves
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
41
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
Digital Item Processing – MPEG-21 solutionbasic concepts
• 'Methods' vs. 'Operations'o DIMs are defined as 'a list of operations'
o Digital Item Base Operations (DIBOs)• describe base operations on which DIMs are built• analogy
– DI Method = programming language– DIBOs = standard library of functions
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
42
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
Digital Item Processing – MPEG-21 solutionbasic concepts
• DIBOso specify high level normative interface to the basic
functionality of the MEPG‑21 Peer
• Digital Item Method Language (DIML)o specifies normative language for defining interoperable
DIMs and from which the DIBOs are able to be called
• Digital Item extension Operations (DIxO) o specify normative mechanism for enabling more
complex functionality in an efficient way
relationship DID – other parts of MPEG-21
Digital Item
DIM1DIM2DIM3...
DIME User
networklocal
storagelocal
platform
DIBO1 DIBO2 DIBO3 DIBO4 DIBO5 DIBO6
media resourcecodec
DIIengine
DIDengine
IPMPengine
RELengine
RDDengine
DIAengine
...
Digital Item
DIM1DIM2DIM3...
DIME User
networklocal
storagelocal
platform
DIBO1 DIBO2 DIBO3 DIBO4 DIBO5 DIBO6
media resourcecodec
DIIengine
DIDengine
IPMPengine
RELengine
RDDengine
DIAengine
...DII
engineDID
engineIPMPengine
RELengine
RDDengine
DIAengine
...
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
44
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 Digital Item Processingbuilding blocks
• One DIML has been chosen: ECMAScript• One set of normative operations: DIBOs
o implemented in any language• any scripting language• any 'full' programming language• only APIs are normatively defined
o will be atomic operationso having access to MultiMedia Middleware API
(see further)o having ECMAScript bindings
• DIBOs can be called from within DIMso having DIxO bindings
• DIxOs can be called from within DIMs• DIxOs can call DIBOs as well
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
45
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 Digital Item Processingbuilding blocks
• DIxOs have access too all normative DIBOso other DIxOs
• DIBO Extension Language:o multiple solutions possibleo already 1 solution identified: MPEG-J-based
• normative annex on how to call MPEG-J-based DIxOs
• informative annex on the execution environment for MPEG-J-based DIxOs
o include complex methods efficientlyo downloadable DIxOs are callable from DIML
Digital Item Method
DIBO DIBO DIBO DIBO
MultiMedia Middleware API (M3W API)
MPEG-21 Digital Item Processinghow does it work ?
Digital Item Method
DIBO DIBO DIBO DIBO
ECMAScript
native language
normative language
normativeAPI & semantics
non-normative implementation
Digital Item Method
DIBO DIBO DIBO DIBO DIxO DIxO DIxO
MultiMedia Middleware API (M3W API)
MPEG-21 Digital Item Processinghow does it work ?
Digital Item Method
DIBO DIBO DIBO DIBO
ECMAScript
identified platform
native language
DIxO DIxO DIxO
normative language
normativeAPI & semantics
normative platform
non-normativeAPI
normative DIxO invocation
non-normative implementation
POPE: Publish Once, Play Everywhere
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
51
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08application – overview
• digital media - multimediao bits and byteso MPEG - Moving Picture Experts Group
• mobile multimedia applicationso problem statemento session mobilityo standardized solution: MPEG-21
• demo: prototype application (during lab visit)o streaming video o lightweight: PDA as a terminal
(Personal Digital Assistant)
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
52
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08introduction
• The old analogue times - TV paradigmo video data modeled as a sequence of pictures with a
certain number of lineso one audio channel added to the video signalo video and audio have an analogue representationo user chooses among the available broadcast programso programs are consumed 'as is'
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
53
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08mobile applications - Session Mobility
• Problem statemento increasing demand to access multimedia anywhere at
any timeo new powerful multimedia devices
• different terminal capabilities• different network capabilities
o increasing demand for• easy switching between different devices• without complex user interaction• i.e., 'session mobility'
o 'going mobile' is a key objective
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
54
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08mobile applications - Session Mobility
student watchinga lesson on a set-
top box
student continuingthe session on a
PDA
finishing the e-learning session
on a tablet PC
tv/set-topbox
PocketPC
Session Transfer
Session Transfer
Media stream
Media stream
Media stream
Tablet PC
Serverscontaining
onlineclasses
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
55
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
SM protocolssimple
• Device Ao collects information about the current sessiono stops the current session
• location of current media stream (e.g., the URI)• position in current media stream (e.g., 50 sec)• status of the current session (e.g., pause)• status of the application (e.g., full screen)
• Device A transfers session data to device B• Device B
o reconstructs session for the received data• load the media stream• go to the correct position• reconstruct the status of the session• reconstruct the status of the application
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
56
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
Device A Device B
2. send session data
1. collect session data
3. process session data
SM protocolssimple
dead period
end of session on device A
start of session on device B
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
57
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
SM protocolsadvanced
• Modified protocol to remove the dead periodo device A
• collects information about current session• does not stop the current session
o device A transfers session data to device Bo device B
• reconstructs session for the received data• requests updates of the session information• until both sessions are synchronized
o device A stops its session
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
58
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
SM protocolsadvanced
Device A Device B
2. send session data
1. collect session data
3. process session data
4. request updated session information
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
59
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08SM and heterogeneous devices
• Different terminal capabilitieso cause difficulties for transferring sessionso example
• watching a movie encoded at CIF (352x288)• on a QCIF (176x144) display
• Different network capabilitieso example
• playing a movie encoded at 512 kbps• on a device with a 128 kbps network connection
• Requiremento common (proprietary or standardized) formato for storage of session information
• Solution: MPEG-21 technology
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
60
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
MPEG-21 Session Mobilityintroduction
• Provides a solution to the problems for session transfer between heterogeneous devices
o differences in capabilities • including choices that allow the configuration of a
Digital Item
o common format for the session information• standardized in Digital Item Adaptation
specification
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
61
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
prototype applicationarchitecture
Starting anMPEG-21multimedia
session
Resuming anMPEG-21multimedia
session
MPEG-21 Session Mobility
Content andstreaming
server
Media stream
Client A
Media stream
Client B
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
62
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
prototype applicationimplementation platforms
• Client A – Windows platformo .Net Framework (C#)o Pentium 4, 2.8 GHz running Windows XPo Windows Media Player 9
• Client B – Windows Mobile platformo .Net Compact Framework (C#)o iPAQ 5550, Intel XScale 400 MHz running Windows
Mobile 2003o Windows Media Player 9 for Pocket PC
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
63
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
prototype applicationperformance measurements
• 4 different measurementso parsing a Digital Itemo generating a Session Mobility Digital Itemo parsing a Session Mobility Digital Itemo transferring and resuming a session
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
64
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08
prototype applicationperformance measurements
• Processing cost is insignificant on both platforms o parsing a Digital Item
• average iPAQ 219.2 mso generating a Session Mobility Digital Item
• average iPAQ 14.4 mso parsing a Session Mobility Digital Item
• average iPAQ 63.1 mso transferring and resuming a session
• average iPAQ 1270.9 ms
new
tren
ds in
mul
timed
ia -
MP
EG
-21:
con
cept
s an
d ap
plic
atio
ns
65
Rik
Van
de
Wal
leG
hent
, Apr
il 20
08conclusions
• MPEG-21 offers very generic framework for setting up and managing advanced multimedia applications
• Session Mobility between devices with different capabilitieso interoperability is a key requiremento MPEG-21 Session Mobility is offering such interoperabilityo by standardizing a format for session information
• Overhead introduced by MPEG-21o not significant compared to the consumption of the actual
multimediao MPEG-21 can easily be implemented on lightweight platforms