qoe++: shifting from ego- to eco-system? qcman 2015 keynote

Prof. Dr. Tobias Hoßfeld

Chair of Modeling of Adaptive Systems (MAS)Institute for Computer Science and Business Information Systems (ICB)University of Duisburg-Essen

www.mas.wiwi.uni-due.de

QoE++: Shifting from Ego- to Eco-System?

IFIP/IEEE QCMan 2015Ottawa, 11 May 2015

1. Current Status: Managing the QoE Ego-System

2. Some Observations on QoE

3. QoE++: The QoE Eco-System

Interest in QoE over the last 10 years

• Number of publications per year when searching for „QoE“

• Academic interest is increasing! Industrial Employment?

mas.wiwi.uni-due.de 3

200420052006200720082009201020112012201320140

50

100

150

200

250

300

350

400

450

2 5 8 1853

89

149

194

298 286

393

IEEE Xplore (Metadata)

Year

#Pub

licati

ons /

Yea

r

20042005

20062007

20082009

20102011

20122013

20140

500

1000

1500

2000

2500

3000

3500

4000

547 513709 709

11001370

1700

2190

2730

34103570

Google Scholar

Year

#Pub

licati

ons /

Yea

r

Research Communities

mas.wiwi.uni-due.de 4

0 1 2 3 4 5 61

2

3

4

5

number of stallings

MO

S

crowdsourcinglaboratoryQoE

MultimediaEncoder

Decoder

0 20 40 60 80 1000

5

10

15

20

25

30

ratio of buffering events

play

time

(min

)

Engagement

Application Control Plane Application

ControllerNetwork Control Plane

Data Plane

ApplicationNetworking

Current Status: QoE Management

mas.wiwi.uni-due.de 5

• Application level, end user site

• Within network, …

• Cross-layer approaches• Realization, e.g. SDN, …

• Parametric models

• Machine learning• …

• Subjective & objective tests • Crowdsourcing• …

Key Influence Factors

QoE Model

QoE Monitor-

ing

QoE Manage-

ment

Concept of QoE Management

Cloud / DC

Access Network

Core Network

Access Network

Cloud service providerEnd user

Cloud / DC

QoE Management requires1. QoE Model2. QoE Monitoring3. QoE Control

Network provider

mas.wiwi.uni-due.de 6

The QoE Ego-System

• Main focus– in-session– short-time scale– single user QoE– single apps– user perspective

• Typical research questions– What are the key QoE influence factors?

– How and where to monitor QoE and its influence factors?

– How to deliver contents and control traffic management?

– How to adapt contents and media to current network situation?

– How to exchange information between network and application to overcome QoE issues?

mas.wiwi.uni-due.de 7

SOME OBSERVATIONS

QoE Models: Complexity and Generic Relationships

mas.wiwi.uni-due.de 9

• Model is intended to fulfill acertain goal

$$$

• Generic relationships need to be considered, e.g. IQX

Subjective Testing

• Subjective Experiments– Quantifying QoE of improved system– Challenging: proper test design,

implementation, analysis– Limited by pool of test subjects

• Crowdsourcing– Access to large pool of humans– Challenging: remote conduction

of tests, statistical analysis

mas.wiwi.uni-due.de 10

What is ?

Crowdsourced QoE: Best Practices

Conceptual aspects

Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., & Tran-Gia, P. (2014). Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. Multimedia, IEEE Transactions on, 16(2), 541-558.

Pseudo reliable crowd

Lab Tester

Filtering- Demographics- Hardware requirements- Reliability- …

Training

Phase 1

QoE - Test - Software based screening mechanisms

- Content questions, reliability checks

- Incentive design, variable payments

- …

Postprocessing

Phase 2

- Statistical analysis- …

Practical aspects

Tobias Hoßfeld, Matthias Hirth, Judith Redi, Filippo Mazza, Pavel Korshunov, et al.. Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force "Crowdsourcing, 2014. https://hal.archives-ouvertes.fr/hal-01078761/

mas.wiwi.uni-due.de 11

Do we need QoE?

Can we utilize QoE for network & service management? Is it more appropriate to consider other means?

Measurement Studies for HTTP Video Streaming

mas.wiwi.uni-due.de 13

0 1 2 3 4 5 61

2

3

4

5

number of stallings

MO

S

crowdsourcinglaboratory

QoE

0 20 40 60 80 1000

5

10

15

20

25

30

ratio of buffering events

play

time

(min

)

Engagement

Engagement data: Dobrian, F., Sekar, V., Awan, A., Stoica, I., Joseph, D., Ganjam, A., Zhan, J. & Zhang, H. (2011). Understanding the impact of video quality on user engagement. ACM SIGCOMM Computer Communication Review, 41(4), 362-373.

System Model

QoE data: Hoßfeld, T., Schatz, R., Biersack, E., & Plissonneau, L. (2013). Internet video delivery in YouTube: from traffic measurements to quality of experience. InData Traffic Monitoring and Analysis (pp. 264-301). Springer Berlin Heidelberg.

Output: stalling pattern

Input: network and video

characteristics

User Behavior and QoE

• Example: QoE and User Engagement in HTTP Video Streaming

• Different video bufferdurations investigated

• Stakeholderinterested in watch time, e.g. sellingadvertisements

• Strong relationship,but complementary approach

mas.wiwi.uni-due.de 14

What are proper QoE models?

How can we extend existing QoE models to take into account the service provider's perspective, individual user perceptions?

Beyond Mean Opinion Scores (MOS)

• MOS is one measure for QoE!

• Confidence intervals show statistical significance, but not reliability!

• Reliability metrics quantify how reliable your data is.

• Standard deviation quantifies the user diversity.

• Quantiles are of interest for service providers.

mas.wiwi.uni-due.de 16

Excellent!

Bad!

Fair!Good!

Poor!

Æ

Fair = 3

Limitations of MOS

• Results from subjective experiments on video QoE

• Service providersdefines a threshold of acceptable quality

• Probability ofdissatisfied users: .

• But: Service providerwants to satisfy majority of users e.g. quantiles

mas.wiwi.uni-due.de 17

Individual QoE Profiles per User?

mas.wiwi.uni-due.de 18

QoE Model for MOSSystem Model

Do we need user profiles?Do we need usage scenarios?

Parameterization of QoE

Impact of user profile

Consequences for QoE Management

Parameterized wrt. user profile Impact of buffer size,

video bitrate, network conditions

Talk later by Christian Moldovan:

To Each According to his Needs: Dimensioning Video Buffer for Specific User Profiles and Behaviorby C. Moldovan, C. Schwartz, T. Hossfeld

Users more or less sensitive to delays and stalling

Is context more important than QoE?

Which context factors are relevant or are such context-factors even more important for network & service management, e.g. in order to foresee and react on flash crowds? 

Example: HTTP Adaptive Streaming with Context

• Use context and context predictors in adaptive streaming strategies• Predict bandwidth and buffer state based on location, connectivity state

(3G, WiFi, upcoming vertical/horizontal handovers), social (e.g. flash crowds), mobility (tunnel)

• Include context information– for buffering and quality

level selection strategy

– for caching decisions

mas.wiwi.uni-due.de 20

User performs QoE management?!

QOE++: THE QOE ECO-SYSTEM

Transition to QoE Eco-System

• QoE eco-system– in-session vs. global– short- vs. long-time scale– single vs. multi-user QoE– single vs. concurrent apps– user vs. business perspective– all key stakeholder goals

• Requirements– Extend current QoE models by the

different stakeholder perspectives of the QoE eco-system– Incorporate user behavior as part of the model– Identify and include relevant internal and external context factors

including physical, cultural, social, economic context.

Content ProviderISPs

CDNs

$$$

$$$ $$$

$$$

AdsData

analysis…

mas.wiwi.uni-due.de 22

Comprehensive Framework: QoE and User Behavior

mas.wiwi.uni-due.de 23

Reichl, P.; Egger, S.; Möller, S.; Kilkki, K.; Fiedler, M.; Hossfeld, T.; Tsiaras, C.; Asrese, A.: Towards a comprehensive framework for QoE and user behavior modelling. QoMEX 2015

An abstract view

mas.wiwi.uni-due.de 24

Quality of Experience Network Layers

ManagementApplication /

Service

Network

QoE++

Technical realization, e.g. SDNMonitoring

Model

Cross-layer approach, interaction

of control loops,economic

traffic management

Viewpoint

Top down: theoretical framework

Met

hodo

logy

Bottom up: use-case & technologydriven

Intermediate players, e.g.

cloud

……

QoE++ Research Directions

• Can we utilize QoE for network & service management? – User engagement and user behavior– Context factors

• How to realize QoE management?– Cross-layer optimization: application demands vs. network capabilities– SDN as technology path

• Can we transform QoE into business models, SLAs, etc.? – Or is it possible to 'trade' QoE? For example, offering WiFi sharing at home, a

user may get improved service delivery and QoE by its ISP. 

• Do we understand QoE as well as fundamental models and natural relationships? – Extend existing QoE models – Relationship between QoE and user behavior?

• Theoretical user-centric performance evaluation approaches

mas.wiwi.uni-due.de 25

THANKS

Additional Pointers (and references therein…) for HTTP Streaming QoE

Overview on HTTP Adaptive Streaming and HAS QoE. Seufert, M.; Egger, S.; Slanina, M.; Zinner, T.; Ho0feld, T.; Tran-Gia, P., "A Survey on Quality of Experience of HTTP Adaptive Streaming," Communications Surveys & Tutorials, IEEE , vol.17, no.1, pp.469,492, 2015doi: 10.1109/COMST.2014.2360940HTTP Streaming QoE Model: Total Stalling, Stalling frequency. Hoßfeld, T., Schatz, R., Biersack, E., & Plissonneau, L. (2013). Internet video delivery in YouTube: from traffic measurements to quality of experience. In Data Traffic Monitoring and Analysis (pp. 264-301). Springer Berlin Heidelberg.

HTTP Streaming model: initial delay, : Total Stalling, Stalling frequency. Tobias Hoßfeld, Christian Moldovan, Christian Schwartz: To Each According to his Needs: Dimensioning Video Buffer for Specific User Profiles and Behavior. In: QCMAN 2015. Ottawa, Canada 2015.

Time on high layer in HAS: Subjective Study. Hoßfeld, T., Seufert, M., Sieber, C., & Zinner, T. (2014). Assessing Effect Sizes of Influence Factors Towards a QoE Model for HTTP Adaptive Streaming. In Proceedings of the 6th International Workshop on Quality of Multimedia Experience (QoMEX 2014), Singapore.

HTTP Adaptive Streaming model: Total Stalling, Stalling frequency and quality adaptation. Hossfeld, Tobias; Skorin-Kapov, Lea; Haddad, Yoram; Pocta, Peter; Siris, Vasilios A. ;Zgank, Andrej; Melvin, Hugh;: Can context monitoring improve QoE? A case study of video flash crowds in the Internet of Services. In: QCMAN 2015 - Third IFIP/IEEE International Workshop on Quality of Experience Centric Management. Ottawa, Canada 2015.

Concrete HAS Implementation. Sieber, C.; Hossfeld, T.; Zinner, T.; Tran-Gia, P.; Timmerer, C., "Implementation and user-centric comparison of a novel adaptation logic for DASH with SVC," Integrated Network Management (IM 2013), 2013 IFIP/IEEE International Symposium on , vol., no., pp.1318,1323, 27-31 May 2013

Benchmarking Framework: Optimial HAS QoE. Hoßfeld, T., Seufert, M., Sieber, C., Zinner, T., & Tran-Gia, P. (2015). Identifying QoE optimal adaptation of HTTP adaptive streaming based on subjective studies. Computer Networks, 81, 320-332.

mas.wiwi.uni-due.de 27

Literature References from the Keynote

Conceptual aspects: Crowdsourced QoE. Hoßfeld, T., Keimel, C., Hirth, M., Gardlo, B., Habigt, J., Diepold, K., & Tran-Gia, P. (2014). Best practices for QoE crowdtesting: QoE assessment with crowdsourcing. Multimedia, IEEE Transactions on, 16(2), 541-558.

Practical aspects: Crowdsourced QoE. Tobias Hoßfeld, Matthias Hirth, Judith Redi, Filippo Mazza, Pavel Korshunov, et al.. Best Practices and Recommendations for Crowdsourced QoE - Lessons learned from the Qualinet Task Force "Crowdsourcing, 2014. https://hal.archives-ouvertes.fr/hal-01078761/

HTTP Streaming model: Total Stalling, Stalling frequency. Hoßfeld, T., Schatz, R., Biersack, E., & Plissonneau, L. (2013). Internet video delivery in YouTube: from traffic measurements to quality of experience. InData Traffic Monitoring and Analysis (pp. 264-301). Springer Berlin Heidelberg.

Beyond MOS: Quantiles and SOS for Service Providers. Hoßfeld, Tobias; Heegard, Poul; Varela, Martin: QoE beyond the MOS: Added Value Using Quantiles and Distributions. QoMEX 2015, Costa Navarino, Greece 2015.

QoE and User Behavior Model - Conceptual approach. Reichl, Peter; Egger, Sebastian; Möller, Sebastian; Kilkki, Kalevi; Fiedler, Markus; Hossfeld, Tobias; Tsiaras, Christos;Asrese, Alemnew: Towards a comprehensive framework for QoE and user behavior modelling. In: QoMEX 2015. Costa Navarino, Greece 2015.

User profiles and QoE / HTTP Streaming model for initial delay and stalling. Tobias Hoßfeld, Christian Moldovan, Christian Schwartz: To Each According to his Needs: Dimensioning Video Buffer for Specific User Profiles and Behavior. In: QCMAN 2015. Ottawa, Canada 2015.

mas.wiwi.uni-due.de 28