a quest for an internet video quality-of-experience metric a. balachandran, v. sekar, a. akella, s....
TRANSCRIPT
A Quest for an Internet VideoQuality-of-Experience Metric
A. Balachandran, V. Sekar, A. Akella, S. Seshan, I. Stoica and H. Zhang
In Proceedings of the 11th ACM Workshop on Hot Topics in Networks
Seattle, WA, USAOctober 29-30, 2012
Introduction
• Content delivery costs down, subscription-based services up Internet video traffic predicted to increase– Possibly surpassing television-based viewership in the future
• Many players: content providers, content delivery networks, video player designers, and users
• All face challenge: lack of standardized approach to measure Quality of Experience (QoE)
• Need one to allow objective comparison of competing designs
New Notions of Quality for Internet Video
• Measuring quality – Internet video using HTTP over CDN– Largely reliable, so “loss” (PSNR) not so relevant– Instead: buffering, bitrate, frame rate, bitrate
switching, startup delay• Measuring experience – with ads and
subscriptions, opinions in controlled study != engagement for business– Instead: fraction of video played, number of visits to
provider
Today, Metrics Fall Short
• Adaptive video players do ad hoc tradeoffs between bitrate, startup delay and buffering [16, 20, 32]
• Frameworks for multi-CDN optimization use primitive QoE metrics that only capture buffering, not bitrate switching [28, 29]
• Content providers have no systematic way to evaluate cost-performance tradeoffs from different CDNs [1]
Robust QoE Measure
• Need unified understanding– Set of quality metrics together affect impact
• Rather than each in isolation
– Natural since there are tradeoffs for individual metrics• E.g. lower bitrate means lower buffering but reduces
quality
• Need quantitative understanding– Beyond simple “metric M impacts engagement”– Instead: “changing metric M from x to y changes
engagement from a to b”
Key Factors for Internet Video QoE
• Complex relationships – relationship between metrics and user experience complex, even counter intuitive– E.g., higher bitrate not always highest quality
• Metric dependencies – metrics subtle interdependencies and tradeoffs– E.g., switching quality reduces buffering, but can annoy users
• Impact of content – nature of content can confound factors– E.g., live versus video-on-demand have different viewing
patterns.– E.g., users interest in content affects tolerance.
Goal
• Identify feasible roadmap to robust, unified, quantitative QoE metric
• Cast QoE measure as machine learning problem– Build appropriate model to predict engagement
(e.g. play time) as a function of quality metrics• Content-induced effects addressed using
domain-specific measurements
Preliminary Results
• Decision tree-based classifier provides 50% accuracy in predicting engagement
• Carefully setting up inputs and features could lead to 25% gain in accuracy
Outline
• Introduction (done)• Use Cases for Video QoE (next)• Challenges in Measuring QoE• Predictive Model for QoE Inference• Preliminary Results• Discussion• Conclusion
Use Cases for Video QoE• Netflix objectively evaluate
CDN– Also Multi-CDN optimizers
• CDNs efficiently distribute resources for users
• Video player to make tradeoffs (bitrate vs. buffer)
• Users to make choices beyond content. Also, some ISPs have bandwidth quotas
• Industry agreement?– Set of quality metrics– Need for “in the wild” data, not
controlled studies
Internet Video Ecosystem
Outline
• Introduction (done)• Use Cases for Video QoE (done)• Challenges in Measuring QoE (next)• Predictive Model for QoE Inference• Preliminary Results• Discussion• Conclusion
Challenges in Measuring QoE
Approach• Example from 2 large
content providers– One serves TV episodes– One serves live sports events
• Industry standard QoE metrics [6]– Errors, delays, video quality
Mini-outline• Complex relationships• Interaction between metrics• Externalities
Complex Relationships• Counter-intuitive effects
– Higher quality should have higher engagement– But! lower quality led to longer play times– Why? For live sports in the background, low quality meant low CPU.
When high, terminated• Non-monotone effects
– Higher average bitrate not always higher quality– Bitrate values in discrete steps. When average “between” steps, means
a switch – annoys users• Threshold effects
– Rates up to 0.5 switches/minute no effect on engagement– Higher rates, users quit early
Interaction Between Metrics
• Switching versus Buffering– Should switch proactively to avoid buffering– But can annoy users (see previous graph)
• Join time versus Bitrate– Higher bitrate implies higher quality– But means takes longer to start (to fill buffer)
Externalities (1 of 2)• Confounding external factors that affect user
engagement• Genre– Live similar to VoD in terms of quality (right)– But user engagement different (left)
Externalities (2 of 2)• User interest– Sample videos and quit, independent of quality issues– Regional issues for live sports
• Quality the same (right)• But locals watch avg 10 minutes longer (left)
Outline
• Introduction (done)• Use Cases for Video QoE (done)• Challenges in Measuring QoE (done)• Predictive Model for QoE Inference (next)• Preliminary Results• Discussion• Conclusion
Towards Predictive Model for QoE Inference
Engagement = f({QualityMetrici})Engagement: e.g. playtime, visits to website
QualityMetrici: e.g. buffering ratio, bitrate
• Dependencies & hidden relationships handled through machine learning– As long as there are sufficiently large datasets
• Fortunately, content providers gather (e.g. Netflix)
• Confounding effects tackled by domain specific insights– Select input data and feed in– Or identify confounding features and let algorithm handle
Outline
• Introduction (done)• Use Cases for Video QoE (done)• Challenges in Measuring QoE (done)• Predictive Model for QoE Inference (done)• Preliminary Results (next)• Discussion• Conclusion
Confirm Intuition of Approach
• 1 month of video viewership, 10 million video sessions
• 10-fold cross-validation– Divide data into 10 equal sized pieces– Train on 9 pieces, test on 1 piece– Repeat 10 times
• Two solutions– Strawman– Domain-specific refinement
Strawman Solutions (1 of 2)
• Use “play time” as engagement• Classes, based on fraction of video viewed– E.g.: 5 classes [0-20%, 20-40%, 40-60%, 60-80%,
80-100%]• Varied learning algorithms (standard)– Naïve Bayes– Simple regression– Classic binary decision tree
Strawman Solutions (2 of 2)
• Decision trees best• Bayes tends to be
better when independent
• Simple regression not good when not linear, e.g., non-monotonic
• All worse with more classes (finer granularity)
Domain-specific Solutions (1 of 3)
• Decision trees can capture some complexity, but not confounding effects– Refine with measurements
• Genre-specific refinement: live and VoD different, so segment into two parts and run separately
• User interest-based refinement: since users tend to “sample” video, ignore early quitters (those < 5 minutes)
Outline
• Introduction (done)• Use Cases for Video QoE (done)• Challenges in Measuring QoE (done)• Predictive Model for QoE Inference (done)• Preliminary Results (done)• Discussion (next)• Conclusion
Discussion (1 of 2)• Metrics – for engagement, need more than playtime
– Ad impressions, user loyalty to return, total number of videos viewed
– Past work [19] suggests quality affects engagement differently, depending upon metric• e.g. delay may not affect specific viewing, but may hurt likelihood to
return• May need to weigh different engagement metrics
• Externalities – is everything covered?– User ISP or device viewing may have impact?– Individual user preference has impact?– Motivates more measurement studies– May need more feature selection – May need user profile information
Discussion (2 of 2)
• Intuitive models – inferring cause for lower engagement tough given confounding factors (user interest, tolerance for low quality)– But need intuitive model so designers and practitioners can
make sense of tradeoffs– But machine learning models can be black-box (e.g.
Principle Component Analysis – PCA)– Fortunately, decision trees have techniques to turn into
more intuitive explanations [27] and equations [25]• Validation – how to validate if useful? – Could have test group with system driven by metric
Conclusions
• Many industries suffer where lack of understandings lead to deceptive marketing– Quote individual metrics to look good, without
explanation to grander scheme (e.g. clock speed for CPU, or megapixels for camera)
• With proliferation of quality factors, Internet video could suffer similar fate
• Goal: robust, unified, quantified QOE metric• Preliminary results – reason to be hopeful