addressing and annotating multimedia fragments

Addressing and annotating multimedia fragments

Raphaël Troncy <[email protected]>

mailto:[email protected]

Use Case

10/04/2012 - Séminaire Muséologie - Le Web devient audiovisuel - 2

htpp://www.eurecom.fr/

http://www.ukoln.ac.uk/jisc-ie/blog/2010/09/15/consuming-and-producing-linked-data-in-a-content-management-system/

Use Case

Aidem received on her Facebook wall a status message containing a Media Fragment URI Use a ‘#’ ! Highlight a

video sequence

Highlight a region to pay attention to



What are Media Fragments?

t 0 20 35 temporal media fragment

spatial media fragment

track media fragment



Temporal Media Fragments


Fragment beginning Fragment end Playback progress

Original resource length


Spatial Media Fragments


semi-opaque overlay

highlighted fragment


Media Fragments Use Cases

Bookmark / Share parts (fragments) of audio/video content

Annotate media fragments

Mash-ups

Conserve bandwidth


http://www.w3.org/TR/media-frags-reqs/


http://www.w3.org/TR/media-frags-reqs/

Requirements

r01: Temporal fragments: a clipping along the time dimension from a start to an end time that

are within the duration of the media resource

r02: Spatial fragments: a clipping of an image region, only consider rectangular regions

r03: Track fragments: a track as exposed by a container format of the media resource

r04: Named fragments: A temporal media fragment that has been given a name through

some sort of annotation mechanism



Media URIs

Using URI query part:

Using URI fragment part:

Mixing both:

http://www.example.org/video.ogv?t=60,100

http://www.example.org/video.ogv#t=60,100

http://www.example.org/video.ogv?t=60,100#t=10,15



The media fragment URI syntax can be used for URI queries

We will focus on URI fragments

URI Fragments vs. URI Queries

#t=20,30 ?t=20,30 secondary resource,

notion of context primary resource,

no notion of context extraction needs to be

expressible in byte ranges no adaptation restrictions

no provisions for communi-cating

fragments to the server

key-value pairs are sent to the server

potentially cacheable not cacheable



Media Fragments Resolution

For the URI query part: The media file is only processed on server side The UA receives a new video file

For the URI fragment part: Smart UA may strip out the fragment definition and

encode it into custom http headers (Range header) (Media) Servers will handle the request, slice the media

content and serve just the fragment (corresponding byte ranges) … while old ones will serve the whole resource



Influence of Media Formats

Fragment extraction needs to be expressible in terms of byte ranges

Requirements for the different axes temporal: presence of intra-coded frames

(i.e., random access points) spatial: presence of independently coded spatial regions track: need to be identifiable by a name

Conclusion: temporal and track axes are realistic, spatial fragments can hardly be expressed in terms of byte ranges



Can I use Media Fragments now?


<video id='v' src='AudioAPI.webm#t=50,100' onloadedmetadata='update()' onpause='update()' onplay='update()' onseeked='update()' controls></video>

Firefox 9+ (or WebKit)

No bandwidth saving


http://www.bluishcoder.co.nz/mediafrag/



Sysnote over YouTube

No bandwidth saving




Ninsuna

Bandwidth saving


http://ninsuna.elis.ugent.be/MFPlayer/html5

Examining HTTP Traffic

HTTP request: retrieving URI parsing key=values pairs from fragment part setting Range header

HTTP response: checking Content-Type

and Content-Range-Mapping headers values attaching custom playback controls to page creating spatial dimension overlay (if specified)



Example: Requesting a Time Fragment

A web developer specifies a video source with a temporal fragment URI:

key=value pair is analyzed, fragment begin and end time are matched

Media Fragments Extension analyses the fragment part, retrieves beginning and end time and sets proper Range header value:

http://ninsuna.elis.ugent.be/DownloadServlet/mfwg/fragf2f.ogv#t=5,15

t=5,15

Range: t:npt=5-15




The NinSuna server responds with the 206 Partial Content response and Content-Range-Mapping header showing the mapped time ranges and media fragment in the message payload:

HTTP/1.1 206 Partial Content Content-Type: video/ogg Accept-Ranges: bytes, t, track, id Content-Range: bytes 629578-1690588/4055466 Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33}={bytes 629578-690588/4055466}




... and it won’t work!

Because the player does not have the media file header




Player needs to be initialized, thus bare fragment is not playable

To fix this, we add ;include-setup to the Range header value:

Range: t:npt=5-15;include-setup




The response from the server is slightly different:

HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End Content-Range-Mapping: {t:npt 4.8-14.8/0-38.33;include-setup} ={bytes 0-5998,629578-1690588/4055466} ... --End Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 {binary data} --End Content-Type: video/ogg Content-Range: bytes 629578-1690588/4055466 {binary data} --End --End--




This is why we need to attach a Stream Listener to the HTTP channel:

HTTP/1.1 206 Partial Content Content-Type: multipart/byteranges;boundary=End ... --End Content-Type: video/ogg Content-Range: bytes 0-5997/4055466 {binary data} --End Content-Type: video/ogg Content-Range: bytes 629578-1690588/4055466 {binary data} --End --End--

HTTP/1.1 206 Partial Content Content-Type: video/ogg ... {binary data} {binary data}



Spatial Fragments

Without additional information send to server

Overlay is created by appending four additional DIVs to the webpage

They are styled to create the impression of semi-opaque layer over the video element



NinSuna: Overview

Fully integrated media adaptation and delivery platform media adaptation and media packaging core is

independent of media formats based on model-driven content adaptation & delivery technique

only high-level adaptation operations such as scene selection and frame rate scaling no transcoding is applied ideal candidate for MF implementation

more information: http://ninsuna.elis.ugent.be



http://ninsuna.elis.ugent.be/

Support for Media Fragment URI queries both HTTP and RTSP implementation try some URIs at http://ninsuna.elis.ugent.be/MediaFragmentsServer#Test http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_t

railer_2.mp4?track=5;6

Support for Media Fragment URI fragments i.e., support for the MF-specific HTTP headers

Combining Media Fragment URI queries and fragments e.g., http://foo.com/media.mp4?t=10,40#t=5,10

MF-specific Features of NinSuna



http://ninsuna.elis.ugent.be/MediaFragmentsServer

http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_trailer_2.mp4?track=5;6

http://ninsuna.elis.ugent.be/DownloadServlet/apple/10,000_BC_trailer_2.mp4?track=5;6

http://foo.com/media.mp4?t=10,40

Updating existing infrastructure is not trivial web servers need to be extended with media extractors MF-specific HTTP communication

existing web caches are only effective with HTTP byte ranges specialized media caches need to be developed in the future to

cache the other cases

user agents need to be extended with a Media Fragment URI parser and interpreter MF-specific HTTP communication MF-specific visualization

Impact on the Existing Web Infrastructure



Goal: make existing media resources, served by generic HTTP Web servers, available as Media Fragments

Biggest problem for content providers willing to support Media Fragments media extractor (dependent on underlying media

formats)

Solution: an approach that works with existing HTTP Web servers (apache, IIS) works with existing Web caches works with not so smart user agents

Media Fragments Proxy



-get (only) the header info -try to find a fragment-to-byte mapping -construct the redirect response

Media Fragments Proxy

HTTP server

MF Proxy MF User

Agent

http://foo.com/video.ogv#t=11,19

GET /?url=http://foo.com/video.ogv HTTP/1.1 Host: MFProxy.com Accept: video/* Range: t:npt=11-19 Accept-Range-Redirect: bytes

HTTP/1.1 307 Temporary Redirect Location: http://foo.com/video.ogv Accept-Ranges: bytes, t, track Content-Length: 0 Content-Type: video/ogg Content-Range-Mapping: t:npt 10-20/0-50 Range-Redirect: 24000-32000 Vary: Accept-Range-Redirect

GET /video.ogv HTTP/1.1 Host: foo.com Accept: video/* Range: bytes=24000-32000

http://ninsuna.elis.ugent.be/MFProxy?url=<mediaURI>



Annotating Media Fragments


yuma.min.js


http://dme.ait.ac.at/annotation/

Annotating Media Fragments


Benchmarking: Sphinx, HTK, Julius

NER + full text index with the transcription

Interlinking with the Linked Data Cloud to enable semantic search




http://blog.dailymotion.com/2011/12/22/sous-titrer-une-video-devient-un-jeu-denfant-3/

http://www.slideshare.net/troncy

10/04/2012 - - 32 Séminaire Muséologie - Le Web devient audiovisuel


http://www.slideshare.net/troncy/

addressing and annotating multimedia fragments

Technology

web devient audiovisuel

media fragment uri syntax

media fragments resolution

media file

media uris

time fragment

temporal fragments

spatial fragments