Building a Dash-264 Player

Jeff Tapper / Michael LabriolaDigital Primates

jtapper@digitalprimates.netmlabriola@digitalprimates.netTwitter: @jtapper / @mlabriola

Jeff Tapper

• Senior Consultant at Digital Primates– Building next generation client applications

• Built video applications for clients including…

• Developing Internet applications for 17 years• Author of 12 books on Internet technologies

Who am I?

• Michael Labriola• Senior Consultant at Digital Primates– Building next generation client applications

• Built video applications for clients including…well lots…

• Developing Internet applications for 17 years• Author of 5 books on Internet technologies

Agenda

• Video and the Internet today• Understanding HTTP Streaming• What is DASH• Flash as a video platform• The inner workings of DASH• The challenge • Getting video bytes info flash• Introducing the Digital Primates DASH-264 Player• Questions

Video is dominating the Internet• Desktop: Video makes up 50% of traffic at peak periods – notably 30% from Netflix and 11% from YouTube

• Mobile: Video traffic is growing exponentially

Fixed Internet Mobile Internet

HTTP Adaptive Streaming

Media Capture & Encoding

Media Origin Servers HTTP Cache Servers Client Devices

001010100001010

010101010001110

01110100011010101

001010100001010

010101010001110

01110100011010101

001010100001010

010101010001110

01110100011010101

001010100001010

010101010001110

01110100011010101

Encode each segment at multiple bitrates

2 Split the video into small segments

1

Make each segment addressable via a HTTP-URL3

Client makes decision on which segment to download

4

Client splices together and plays back5

HTTP Streaming Landscape

• Apple’s HTTP Live Streaming (HLS)• Microsoft’s Smooth Streaming • Adobe’s HTTP Dynamic Streaming (HDS)• And many more…

The challenge

• Most agree that HTTP Streaming is the most efficient choice

• Different devices support different streaming protocols

• No one standard is currently supported ubiquitously

• Results in media being served in several different formats to support the broadest range of devices

What is MPEG-DASH

DASH – Dynamic Adaptive Streaming via HTTP International open standard, developed and

published by ISO Addresses both simple and advanced use cases Enables highest-quality multiscreen distribution

and efficient dynamic adaptive switching Enables reuse of existing content, devices and

infrastructure Attempts to unify to a single standard for HTTP

Streaming

DASH and codecs

• The DASH specification is codec agnostic• Any existing or future codec can work with

DASH• DASH manifest describes which codec is used– Different codecs store the actual video data

differently

DASH264

• H.264 is dominant format today• Many vendors and service providers are

committed to supporting/enabling DASH264• Provides support for today’s requirements

such as DRM• H.264 is backed by rigorous testing and

conformance

DASH Industry Forum

• Addressing the dramatic growth of broadband video by recommending a universal delivery format that provides end users with the best possible media experience by dynamically adapting to changing network conditions.

DASH Industry Forum

• Objectives: – promote and catalyze market adoption of MPEG-DASH – publish interoperability and deployment guidelines– facilitate interoperability tests – collaborate with standard bodies and industry consortia

in aligning ongoing DASH standards development and the use of common profiles across industry organizations

• Currently 46 members• Visit http://dashif.org for more information

Members

Building a Desktop DASH player

• There are many technologies available for playing video in modern web browsers– Flash– SilverLight– HTML5– Etc.

• We focused on Flash for our first DASH player– Largest percentage of users can support it

Building Video Players in Flash

• Flash Player includes a native video player– flash.media.Video

• Video accepts a source from a Camera or a NetStream– flash.media.Camera– flash.net.NetStream

• Netstream supports:– Streaming FLV file– Manually appending FLV Bytes into its buffer

Flash Video (continued)

• NetStream and Video classes can be manipulated manually

• Open Source frameworks are available for easing the process– Adobe’s Open Source Media Framework (OSMF)– Open Video Player (OVP)

• We have chosen to build on OSMF

OSMF

• Extensible framework with plugin architecture• Native support for DRM• Contains classes to work with HTTP Streaming

technologies– Only HDS is supported natively– Extensible architecture makes it possible to

support new technologies

How to DASH in Flash

• Extract Video Bytes from DASH segments• Wrap bytes in format understood by flash

(FLV)• Append FLV Bytes into Netstream

Understanding DASH structure

• Three types of files– Manifest (.mpd)• XML file describing the segments

– Initialization file• Contains headers needed to decode bytes in segments

– Segment Files• Contains playable media• Includes:

– 0…many video tracks– 0…many audio tracks

DASH Manifest

• Manifest contains:– Program Info (including title)– 1 or more periods

• Periods contain 1 adaptation set per video stream and • Periods contain 1 adaptation set per audio stream• Adaptation Sets contain:

– Content Composition nodes (for each video or audio track)– 1 or more Representation node

» Each representation describes a single bitrate» Representations contain data on finding the actual segments» Different ways a representation can describe segments

Describing Representations

• SegmentBase– Describes a stream with only a single Segment per bitrate– Can be used for Byte Range Requests

• SegmentList– A SegmentList will contain a specific list of each SegmentURL

(individual HTTP packet with media data)– Can be used for Byte Range Requests

• SegmentTemplate– Defines a known url for the fragment with wildcards resolved

at runtime to request a segments (see bbb.mpd)– Alternatively, can specify a list of segments based on duration

SegmentList<Representation id="h264bl_hd" mimeType="video/mp4" codecs="avc1.42c01f" width="1280" height="720" startWithSAP="1" bandwidth="514864"> <SegmentList timescale="1000" duration="10000"> <Initialization sourceURL="mp4-main-multi-h264bl_hd-.mp4"/> <SegmentURL media="mp4-main-multi-h264bl_hd-1.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-2.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-3.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-4.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-5.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-6.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-7.m4s"/> <SegmentURL media="mp4-main-multi-h264bl_hd-8.m4s"/>

SegmentTemplate fixed segment duration

<AdaptationSet> <ContentComponent id="1" contentType="video"/> <SegmentTemplate initialization="BigBuckBunny_720p_1800kbps_44khz_track1_dash.mp4"/> <Representation id="1" mimeType="video/mp4“ codecs="avc1.64001f" width="1280" height="720“ startWithSAP="1" bandwidth="1809954"> <SegmentTemplate timescale="1000" duration="13809" media="bbb_seg_BigBuckBunny_720p_1800kbps_44khz_track1$Number$.m4s" startNumber="1"/> </Representation></AdaptationSet>

SegmentTemplate variable segment duration

<AdaptationSet group="2" mimeType="video/mp4" par="16:9“ minBandwidth="475000“ maxBandwidth="6589000" minWidth="176" maxWidth="1680" minHeight="99" maxHeight="944“ segmentAlignment="true“ startWithSAP="1"> <SegmentTemplate timescale="1000"

initialization="dash/ateam-video=$Bandwidth$.dash"media="dash/ateam-video=$Bandwidth$-$Time$.dash">

<SegmentTimeline> <S t="0" d="4171" /> <S d="2503" /> <S d="2961" /> <S d="2461" /> <S d="2127" r="2" />…

Understanding MP4 – H.264 structure

• Each segment is binary, with a series of boxes contained within

• First 8 bytes of every box defines– Box type– Number of bytes for the box.

Initialization Segment

• FTYP – file type and compatibility• FREE – free space• MOOV – container of all metadata– MVHD – movie header– 1.* TRAK – container for individual stream data• TKHD – track header• MDIA – container for all media info

– Lots of child boxes which contain more detailed stream info

Content Segments

• STYP - segment type• SIDX - segment index• MOOF -movie fragment– MFHD - movie fragment header– TRAF - track fragment

• TFHD - track fragment header• TFDT - track fragment decode time• TRUN - track fragment run

• MDAT - media data container (actual media data)– Contains one or more samples (smallest described piece of

content)

Pulling it all together

• Using OSMF, we wrote a DASH specific IndexHandler and FileHandler

• IndexHandler responsible for deciding url of next http request

• FileHandler responsible for parsing data from segments and returning FLV Bytes

DashIndexHandler

• When OSMF determines a new fragment needs to be loaded, it calls the DashIndexHandler.

• When playing, getNextFile is called• When seeking, getFileForTime is called• Both return an HTTPStreamRequest, which

OSMF uses to request the proper segment

DashFileHandler

• When the requested file is returned, it is handed to the DashFileHandler for processing– initializeProcessing – gets decoder– processFileSegment – asks decoder to process data

• We are using a Decoder called MP4Decoder• MP4Decoder returns FLVBytes for the data

within the MDAT• processFileSegment returns ByteArray (FLVBytes)

to OSMF, for adding to the NetStream buffer

MP4Decoder

• processData() – iterates through all boxes in segment– Calls getNextBox()

• getNextBox knows how to process each specific box type

• When processing MDAT, individual samples are wrapped with FLVBytes

• FLVBytes wrap specific binary header around data, as required by NetStream.appendBytes()

Digital Primates DASH Player

• Currently our DASH player is not commercially available

• We use it as the basis for DASH players we build for our clients

• Initially built as a learning exercise to better understand DASH

Current DP Dash Player

• We currently support SegmentLists, SegmentTemplates and SegmentBase in manifest– Byte Range Requests only currently supported via

Akamai• We support Dynamic Adaptive Streaming for

VOD Streams• We have tested with streams up to 1080p• We have tested with streams from several

different encoders

DP Dash Player plans

• 4Q12 – Basic Seek Fixes– Improved multi-channel

audio support– Support for Live

Streams– Continue to test and fix

streams from more encoding partners

• 1Q13 – Initial JavaScript Player– Support for DRM– Support for Enhanced

seeking• 2Q13 – Investigate HEVC (h.265)– Initial Android Player

• 3Q13 – Initial iOS Player

Resources

• GPAC – http://gpac.wp.mines-telecom.fr– Provides baseline test streams– Provides baseline player

• MP4Parser– http://code.google.com/p/mp4parser/ – Open Source java project – Allows for display of contents within boxes

• DASH Industry Forum– http://www.dashif.org

Questions?

?