quality of experience of web-based adaptive http streaming clients in real-world environments using...

Quality of Experience of Web-‐based Adap8ve HTTP Streaming Clients in Real-‐World Environments using Crowdsourcing

Benjamin Rainer and Chris8an Timmerer

Alpen-‐Adria-‐Universität Klagenfurt (AAU) w Faculty of Technical Sciences (TEWI) w Department of Informa8on Technology (ITEC) w Mul8media Communica8on (MMC) w Sensory Experience Lab (SELab)

h"p://blog.+mmerer.com w h"p://dash.itec.aau.at/ w h"p://selab.itec.aau.at mailto:[email protected]‐klu.ac.at

December 2, 2014

Slides: hVp://www.slideshare.net/chris8an.8mmerer

Outline

•  Introduc+on •  How to evaluate DASH and QoE •  Methodology •  Results •  Conclusions

December 2, 2014 VideoNext 2014, Sydney 2

Mul+media is Predominant on the Internet

•  Real-‐+me entertainment –  Streaming video and audio – More than 50% of Internet traffic at peak periods

•  Popular services –  NeVlix (34.9%), YouTube (14.0%), Amazon Video (2.6%), Hulu (1.4%)

–  All delivered over-‐the-‐top (OTT)

– MPEG Dynamic Adap+ve Streaming over HTTP


Global Internet Phenomena Report: 2H 2014

Over-‐The-‐Top – Adap+ve Media Streaming

•  In a nutshell …


C. Timmerer and A. C. Begen, “Over-‐the-‐Top Content Delivery: State of the Art and Challenges Ahead”, In Proceedings of the ACM interna+onal conference on Mul+media (MM '14), Orlando, FL, USA, Nov. 2014. h"p://www.slideshare.net/chris+an.+mmerer/over-‐the-‐top-‐content-‐delivery-‐state-‐of-‐the-‐art-‐and-‐challenges-‐ahead

Adapta8on logic is within the client, not norma8vely specified

by the standard, subject to research and development

MPEG Dynamic Adap+ve Streaming over HTTP

What is specified – and what is not?


Media Presenta+on on HTTP Server

DASH-‐enabled Client Media Presenta8on Descrip8on

. . .

Segment

…

.

.

. Segment

…

. . .

Segment

…

.

.

. Segment

…

…

Segments located by HTTP-‐URLs

DASH Control Engine

HTTP/1.1 HTTP Client

MPD Parser

Media Engine

On-‐8me HTTP requests to segments

DASH Data Model


Segment Info

Initialization Segment http://bitmov.in/500/init.mp4

Media Presentation

Period, start=0s

…

Period, start=100s

…

Period, start=200s

…

…

Period start=100 baseURL=http://bitmov.in/

AdaptationSet 1 500-1500 kbit/s

…

AdaptationSet 2 1500-3000 kbit/s

…

Media Segment 1 start=100s http://bitmov.in/500/seg-1.m4s




AdaptationSet 1 width=640-1280 height=360-720 …

Representation 1 500 Kbit/s

Representation 2 1500 Kbit/s

…

Representation 2 bandwidth=1500 kbit/s width=960, height=540

Segment Info duration=2s

Template: 500/seg-$Number$.m4s

Initialization: 500/init.mp4

…

How to evaluate DASH? •  Methodology

– Dataset, tools (see backup slides for details) –  Common evalua+on setup –  Bandwidth traces (real/synthe+c) vs. models

•  Metrics – Average media bitrate/throughput at the client – Number of representa+on/quality switches – Number of stalls (in seconds) – buffer level


C. Mueller, S. Lederer, C. Timmerer, “An Evalua+on of Dynamic Adap+ve Streaming over HTTP in Vehicular Environments”, In Proceedings of the Fourth Annual ACM SIGMM Workshop on Mobile Video (MoVid12), Chapel Hill, North Carolina, February 2012.

Quality of Experience •  Quality of Experience

–  “… is the degree of delight or annoyance of the user of an applica+on or service…”

–  Factors influencing / features of QoE may lead to applica+on-‐specific defini+ons

•  Subjec+ve quality assessments –  Laboratory environment [ITU-‐T B.500 / P.910] –  Crowdsourcing with special plaVorms or social networks

•  QoE of DASH-‐based services –  Startup delay (low) –  Buffer underrun / stalls (zero) –  Quality switches (low) and media throughput (high)


P. Le Callet, S. Möller and A. Perkis, eds., “Qualinet White Paper on Defini+ons of Quality of Experience (2012)”, European Network on Quality of Experience in Mul>media Systems and Services (COST Ac>on IC 1003), Lausanne, Switzerland, Version 1.2, March 2013."

Methodology •  Quality of Experience …

–  Mean Opinion Score [0..100] –  [other objec+ve metrics:

start-‐up +me, throughput, number of stalls] •  … Web-‐based Adap+ve HTTP Streaming Clients …

–  HTML5+MSE: DASH-‐JS (dash.itec.aau.at), dash.js (DASH-‐IF, v1.1.2), YouTube •  … Real-‐World Environments …

–  DASH-‐JS, dash.js hosted at ITEC/AAU (~ 10Gbit/s) –  YouTube hosted at Google data centers –  Content: Tears of Steel @ 144p (250 kbit/s), 240p (380 kbit/s), 360p (740 kbit/

s), 480p (1308 kbit/s), and 720p (2300 kbit/s); segment size: 2s –  Users access content over the open Internet

•  … Crowdsourcing –  Campaign at Microworker plaVorm (others also possible: Mechanical Turk,

social networks) limited to Europe, USA/Canada, India –  Screening Techniques: Browser fingerprin+ng, s+mulus presenta+on +me,

QoE ra+ngs and pre-‐ques+onnaire


Results: QoE


Results: Media Throughput


Results: Start-‐Up Time


Results: Number of Switches


Results: Number of Stalls


Results: Summary •  DASH-‐JS

–  High start-‐up +me –  Low number of stalls –  Good throughput, QoE

•  dash.js –  Low start-‐up +me –  High # stalls –  Low throughput –  Low QoE

•  YouTube –  Low start-‐up +me –  Low number of stalls –  Best throughput, QoE


Conclusions •  QoE evalua+on of DASH-‐like systems in real-‐world environments

using crowdsourcing –  Detailed methodology described in the paper –  Results indicate that the delivered representa+on bitrate (media

throughput) and the number of stalls are the main influence factors on the QoE

–  Results confirmed by previous evalua+ons but within controlled environments

–  Evidence about QoE aspects of DASH-‐enabled Web clients within real-‐world environments

–  Feasibility of using crowdsourcing for subjec+ve quality assessments •  Future work

–  Comprehensive evalua+on of various adapta+on logics (both objec+ve and subjec+ve) and

–  the impact of dedicated delivery infrastructures aiming to improve DASH-‐based services


Thank you for your a"en+on

... ques+ons, comments, etc. are welcome …

Priv.-‐Doz. Dipl.-‐Ing. Dr. Chris+an Timmerer Associate Professor

Klagenfurt University, Department of Informa+on Technology (ITEC) Universitätsstrasse 65-‐67, A-‐9020 Klagenfurt, AUSTRIA

[email protected]‐klu.ac.at h"p://research.+mmerer.com/

Tel: +43/463/2700 3621 Fax: +43/463/2700 3699 © Copyright: Chris>an Timmerer

19 December 2, 2014 VideoNext 2014, Sydney

BACKUP SLIDES


End-‐to-‐End DASH System Aspects •  (Distributed) dataset

–  Full movie length in high quality

–  Various bitrate, resolu+ons, segment lengths (2-‐15s), (sub-‐)segments

–  Distributed: ini+al 3 sites, now 9 in Europe, USA, Taiwan

•  DASH encoder –  Encoding + Mul+plexing +

MPD genera+on –  Fully configurable using a

configura+on file –  Enables batch processing –  x264/ffmpeg + GPAC MP4Box


S. Lederer, C. Müller, C. Timmerer, “Dynamic Adap+ve Streaming over HTTP Dataset”, In Proceedings of the ACM Conference on Mul+media Systems 2012, Chapel Hill, North Carolina, February 2012. // S. Lederer, C. Mueller, C. Timmerer, C. Concolato, J. Le Feuvre, K. Fliegel, “Distributed DASH Dataset”, In Proceedings of the ACM Conference on Mul+media Systems 2013, Oslo, Norway, 2013.

End-‐to-‐End DASH System Aspects •  Playback

–  VLC plugin (first implementa+on) –  DASH-‐JS (HTML5 + MSE) –  libdash / qtsampleplayer

•  MPD valida+on –  XML schema valida+on –  Xlink resolver & processing –  Addi+onal valida+on rules

(Schematron) •  Experimental

–  DASH over Content-‐Centric Networks (CCN)

–  VLC + libdash


C. Müller and C. Timmerer, “A VLC Media Player Plugin enabling Dynamic Adap+ve Streaming over HTTP”, In Proceedings of the ACM Mul+media 2011, Sco"sdale, Arizona, November 2011. // B. Rainer, S. Lederer, C. Müller, C. Timmerer, “A Seamless Web Integra+on of Adap+ve HTTP Streaming”, In Proceedings of the 20th European Signal Processing Conference 2012, Bucharest, Romania, August 2012.

h"p://records.sigmm.ndlab.net/2013/04/open-‐source-‐column-‐dynamic-‐adap+ve-‐streaming-‐over-‐h"p-‐toolset/

quality of experience of web-based adaptive http streaming clients in real-world environments using...

Technology

m4smedia segment

segment infoduration

kbitsmedia segment

methodology dataset