1 a quest for an internet video quality-of-experience metric athula balachandran, vyas sekar, aditya...
TRANSCRIPT
![Page 1: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/1.jpg)
1
A Quest for an Internet VideoQuality-of-Experience Metric
Athula Balachandran, Vyas Sekar,Aditya Akella, Srinivasan Seshan,
Ion Stoica, Hui Zhang
![Page 2: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/2.jpg)
2
Internet Video is taking off
Improve Users’ Quality of Experience
![Page 3: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/3.jpg)
3
Video Quality Metrics: The State of the Art
Objective Score (e.g., Peak Signal to Noise Ratio)
Subjective Scores(e.g., Mean Opinion
Score)
![Page 4: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/4.jpg)
4
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
![Page 5: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/5.jpg)
5
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
![Page 6: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/6.jpg)
6
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
![Page 7: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/7.jpg)
7
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
![Page 8: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/8.jpg)
8
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
![Page 9: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/9.jpg)
9
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
![Page 10: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/10.jpg)
10
Problem 1: New Effects, New Metrics
PLAYERSTATES
EVENTS
Joining Playing Buffering Playing
Bufferfilled up
Bufferempty
Bufferfilled up
Switchbitrate
Join Time Buffering RatioRate of buffering
Rate of switchingAverage bitrate
![Page 11: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/11.jpg)
11
Problem 2: Opinion Scores Engagement
Opinion Scores- Not representative of “in the wild”
experience- Combinatorial explosion of parameters
Engagement as replacement for opinion score. (e.g., Play time, customer return rate)
![Page 12: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/12.jpg)
12
Internet Video QoE
Objective ScoresPSNR
Subjective ScoresMOS
![Page 13: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/13.jpg)
13
Internet Video QoE
Objective ScoresPSNR
Subjective ScoresMOS
Engagement(e.g., Fraction of video viewed)
![Page 14: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/14.jpg)
14
Internet Video QoE
Objective ScoresPSNR
Join Time, Avg. bitrate, …?
Subjective ScoresMOS
Engagement(e.g., Fraction of video viewed)
![Page 15: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/15.jpg)
15
Internet Video QoE
Objective ScoresPSNR
Join Time, Avg. bitrate, …?f(Join Time, Avg. bitrate, …)
Subjective ScoresMOS
Engagement(e.g., Fraction of video viewed)
![Page 16: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/16.jpg)
16
Internet Video QoE
Objective ScoresPSNR
Join Time, Avg. bitrate, …?f(Join Time, Avg. bitrate, …)
Subjective ScoresMOS
Engagement(e.g., Fraction of video viewed)
![Page 17: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/17.jpg)
17
Outline
• Need for a unified QoE
• What makes this hard?
• Our proposed approach
![Page 18: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/18.jpg)
18
Challenge: Complex Engagement-to-metric Relationships
Enga
gem
ent
Quality Metric
![Page 19: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/19.jpg)
[Dobrian et al. Sigcomm 2011] 19
Challenge: Complex Engagement-to-metric Relationships
Enga
gem
ent
Quality Metric
Non-monotonic
E
ngag
emen
t
Average bitrate
![Page 20: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/20.jpg)
[Dobrian et al. Sigcomm 2011] 20
Challenge: Complex Engagement-to-metric Relationships
Enga
gem
ent
Quality Metric
Non-monotonic
E
ngag
emen
t
Average bitrate
En
gage
men
t
Rate of switching
Threshold
![Page 21: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/21.jpg)
21
Challenge: Complex Metric Interdependencies
Join Time Bitrate
Rate of buffering
Rate of switching
Buffering Ratio
![Page 22: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/22.jpg)
22
Challenge: Complex Metric Interdependencies
Join Time Bitrate
Rate of buffering
Rate of switching
Buffering Ratio
![Page 23: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/23.jpg)
23
Challenge: Complex Metric Interdependencies
Join Time
Rate of buffering
Rate of switching
Buffering Ratio
Bitrate
![Page 24: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/24.jpg)
24
Challenge: Complex Metric Interdependencies
Join Time Avg. bitrate
Rate of buffering
Rate of switching
Buffering Ratio
![Page 25: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/25.jpg)
25
Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies
![Page 26: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/26.jpg)
26
Casting as a Learning Problem
MACHINE LEARNING
Engagement Quality Metrics
QoE Model
Need to learn these complex engagement-to-metric relationships and metric-to-metric dependencies
![Page 27: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/27.jpg)
27
Impact of the ML algorithm• Classify engagement into uniform classes• Accuracy = # of accurate predictions/ # of cases
ML algorithm must be expressive enough to handle the complex relationships and interdependencies
![Page 28: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/28.jpg)
28
Challenge: Confounding Factors
Live and VOD sessions experience similar quality
![Page 29: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/29.jpg)
29
Challenge: Confounding Factors
However, user viewing behavior is very different
![Page 30: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/30.jpg)
30
Challenge: Confounding Factors
Devices User InterestConnectivity
Need systematic approach to identify and handle confounding factors
![Page 31: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/31.jpg)
31
Domain-specific Refinement
MACHINE LEARNING
Engagement Quality Metrics
QoE Model
![Page 32: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/32.jpg)
32
Domain-specific Refinement
MACHINE LEARNING
Engagement Quality Metrics
QoE Model
Confounding Factors
![Page 33: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/33.jpg)
33
Improved prediction accuracy
Refined ML models can handle confounding factors
![Page 34: 1 A Quest for an Internet Video Quality-of-Experience Metric Athula Balachandran, Vyas Sekar, Aditya Akella, Srinivasan Seshan, Ion Stoica, Hui Zhang](https://reader036.vdocuments.net/reader036/viewer/2022062318/5519d19c55034649768b490d/html5/thumbnails/34.jpg)
34
Concluding Remarks• Internet Video needs unified quantitative QoE
• What makes this hard?– Complex engagement-to-metric relationships– Complex metric-to-metric interdependencies– Confounding factors (e.g., genre, device)
• Promising start– Machine learning + domain-specific refinements
• Open Challenges– Coverage over confounding factors– System Design