developing a predictive model for internet video quality-of-experience
DESCRIPTION
Developing a Predictive Model for Internet Video Quality-of-Experience. Athula Balachandran , Vyas Sekar , Aditya Akella , Srinivasan Seshan , Ion Stoica , Hui Zhang. QoE $$$ . CDN. Better Quality Video. $$$. Content Providers. Users. Higher Engagement. The QoE model. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/1.jpg)
1
Developing a Predictive Model for Internet Video Quality-of-Experience
Athula Balachandran, Vyas Sekar,Aditya Akella, Srinivasan Seshan,
Ion Stoica, Hui Zhang
![Page 2: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/2.jpg)
2
QoE $$$
CDN
UsersContentProviders
$$$Better QualityVideo
HigherEngagement
The QoE model Diagram courtesy: Prof. Ramesh Sitaraman, IMC 2012
![Page 3: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/3.jpg)
3
Adapting video bitrates quicker
Picking the best server
Why do we need a QoE model?
Comparing CDNs
![Page 4: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/4.jpg)
4
Traditional Video Quality Metrics
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion Score)
Does not capture new effects (e.g., buffering, switching
bitrates)
User studies not representative of “in-the-wild” experience
![Page 5: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/5.jpg)
5
Internet Video is a new ball game
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
Engagement(e.g., fraction of video viewed)
Quality metrics
![Page 6: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/6.jpg)
6
Commonly used Quality MetricsJoin Time
Average Bitrate
Buffering ratio
Rate of buffering
Rate of switching
![Page 7: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/7.jpg)
7
Which metric should we use?
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
Engagement(e.g., fraction of video viewed)
Quality metrics Buffering Ratio, Average bitrate?
Today:QualitativeSingle-metric
![Page 8: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/8.jpg)
8
Unified and Quantitative QoE Model
Objective Scores (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
Engagement(e.g., fraction of video viewed)
Quality metrics Buffering Ratio, Average bitrate?
ƒ (Buffering Ratio, Average bitrate,…)
![Page 9: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/9.jpg)
9
Outline
• What makes this hard?
• Our approach
• Conclusion
![Page 10: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/10.jpg)
Complex Engagement-to-metric RelationshipsEn
gage
men
t
Quality Metric
10
Non-monotonic
E
ngag
emen
t
Average bitrate
En
gage
men
t
Rate of switching
Threshold
[Dobrian et al. Sigcomm 2011]
Ideal Scenario
![Page 11: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/11.jpg)
11
Complex Metric Interdependencies
Join
Tim
e
Average bitrate
Average bitrate
Rate
of b
uffer
ing
Join Time Bitrate
Buffering ratio
Rate of buffering
Rate of switching
![Page 12: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/12.jpg)
Confounding Factors
12
Confounding Factors can affect:1) Engagement
Live and Video on Demand (VOD) sessions have different viewing patterns.
CDF
( %
of u
sers
)
Engagement
Type of Video Live
VOD
![Page 13: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/13.jpg)
Confounding Factors
13
Confounding Factors can affect:1) Engagement2) Quality Metrics
Type of Video Live
VOD
CDF
( %
of u
sers
)
Join Time
Live and Video on Demand (VOD) sessions had different join time distribution.
![Page 14: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/14.jpg)
Confounding Factors
14
Confounding Factors can affect:1) Engagement2) Quality Metrics3) Quality Metric
Engagement
Type of Video
Connectivity DSL/Cable
Wireless (3G/4G)
Eng
agem
ent
Rate of buffering
Users onwireless connectivity weremore tolerant to rate of buffering.
![Page 15: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/15.jpg)
Confounding Factors
15
Type of Video
Connectivity
Device
Location
Popularity
Time of day
Day of week
Need systematic approach to identify and incorporate confounding factors
![Page 16: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/16.jpg)
16
Summary of Challenges
1. Capture complex engagement-to-metric relationships and metric-to-metric dependencies.
2. Identify confounding factors3. Incorporate confounding factors
![Page 17: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/17.jpg)
17
Outline
• What makes this hard?
• Our approach
• Conclusion
![Page 18: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/18.jpg)
18
Challenge 1: Capture complex relationships
![Page 19: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/19.jpg)
19
Cast as a Learning Problem
MACHINE LEARNING
Engagement Quality Metrics
QoE Model
Decision Trees performed the best.Accuracy of 40% for predicting within a 10% bucket.
![Page 20: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/20.jpg)
20
Challenge 2: Identify the confounding factors
![Page 21: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/21.jpg)
21
Test Potential Factors
Engagement
Confounding Factors
Quality Metrics
![Page 22: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/22.jpg)
22
Test Potential Factors
Test 1: Relative Information Gain
Engagement
Confounding Factors
Quality Metrics
![Page 23: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/23.jpg)
23
Test Potential Factors
Test 1: Relative Information GainTest 2: Decision Tree StructureTest 3: Tolerance Level
Engagement
Confounding Factors
Quality Metrics
![Page 24: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/24.jpg)
24
Identifying Key Confounding Factors
Factor Relative Information Gain
Decision TreeStructure
Tolerance Level
Type of video ✓ ✓ ✓Popularity ✗ ✗ ✗Location ✗ ✗ ✗Device ✗ ✓ ✓Connectivity ✗ ✗ ✓Time of day ✗ ✗ ✓Day of week ✗ ✗ ✗
![Page 25: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/25.jpg)
25
Identifying Key Confounding Factors
Factor Relative Information Gain
Decision TreeStructure
Tolerance Level
Type of video ✓ ✓ ✓Popularity ✗ ✗ ✗Location ✗ ✗ ✗Device ✗ ✓ ✓Connectivity ✗ ✗ ✓Time of day ✗ ✗ ✓Day of week ✗ ✗ ✗
![Page 26: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/26.jpg)
26
Challenge 3: Incorporate the confounding factors
![Page 27: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/27.jpg)
Refine the Model
27
MACHINE LEARNING
EngagementQuality Metrics
QoE Model
Confounding Factors
Adding as a feature Splitting the dataConfounding Factors 1e.g., Live, Mobile
ML
EngmntQualityMetrics
Model 2
ML
EngmntQualityMetrics
Model 1
ML
EngmntQualityMetrics
Model 3
Confounding Factors 2e.g., VOD, Mobile
Confounding Factors 3e.g., VOD, TV
QoE Model
![Page 28: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/28.jpg)
28
Comparing Candidate Solutions
Final Model: Collection of decision treesFinal Accuracy- 70% (c.f. 40%) for 10% buckets
![Page 29: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/29.jpg)
29
Summary of Our Approach
1. Capture complex engagement-to-metric relationships and metric-to-metric dependencies
Use Machine Learning2. Identify confounding factors Tests 3. Incorporate confounding factors
Split
![Page 30: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/30.jpg)
30
Evaluation: Benefit of the QoE Model
Preliminary results show that using QoE model to select bitrate leads to 20% improvement in engagement
![Page 31: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/31.jpg)
31
Conclusions• Internet Video needs a unified and quantitative QoE model
• What makes this hard?– Complex relationships– Confounding factors (e.g., type of video, device)
• Developing a model– ML + refinements => Collection of decision trees
• Preliminary evaluation shows that using the QoE model can lead to 20% improvement in engagement
• What’s missing?– Coverage over confounding factors– Evolution of the metric with time
![Page 32: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/32.jpg)
32
EXTRA SLIDES
![Page 33: Developing a Predictive Model for Internet Video Quality-of-Experience](https://reader036.vdocuments.net/reader036/viewer/2022081604/56816376550346895dd45377/html5/thumbnails/33.jpg)
34
Choice of ML Algorithm Matters
ML algorithm must be expressive enough to handle the complex relationships and interdependencies
• Classify engagement into uniform classes• Accuracy = # of accurate predictions/ # of cases