Perceptual Quality Assessment of P2P Assisted Streaming Video for Chunk-level Playback Controller Design

Tom Z.J. Fu, CUHK

W. T. Leung, CUHK

P. Y. Lam, CUHK

Dah Ming Chiu, CUHK

Zhibin Lei, ASTRI

PV 2010, Hong Kong

Introduction and motivation

Chunk-level impairment model

Experiments with various Chunk Receiving Patterns (CRP)

Heuristic on satisfaction function

Future work and conclusion

Outline

Internet streaming service becomes popular

1. C/S mechanism,

2. P2P mechanism, mostly implemented. CDN, single/multiple tree-based application

layer multicast, peer-to-peer streaming (live streaming /

VoD). The evaluation for the above two mechanisms

are quite different.

Introduction and motivation

Introduction and motivation

1. For the C/S mechanism, The simple end-to-end link model is suitable for

abstraction, link condition metrics: Packet loss rate End-to-end packet transmission delay etc.

计算机

End-to-End Li nk Model

Packet-level impairments are considered

Introduction and motivation

2. For the P2P assisted mechanism, The simple end-to-end link model is not suitable:

The transmission pattern is dynamic and complicated.

P2P mechanism forms overlay topology

Introduction and motivation

2. For the P2P assisted mechanism, The simple end-to-end link model is not suitable:

1. The transmission pattern is dynamic and complicated.

2. The granularity of packet-level impairment is too fine, causing mismatch with system design:a) Important building blocks of P2P mechanism are based on ch

unks: chunk selection, peer selection, buffer map management affecting the changing of the overlay topology

b) System-wide performance metrics are chunk level: Average playback (dis) continuity; Buffer filling probability, etc

c) In each client side, Playback decision is on top of chunk (How to make decisions with QoE considered?)

Therefore, we need chunk-level impairment model !

Chunk-level impairment model

K+1 K……… K+2K+3K+4

Cl i ent buff er

Streami ng di recti on

Medi a pl ayer

Decoder

UserInterface

Video encoder– Different media codec, transmission rate could be chosen at the video

encoder component Network transmission – chunk level impairment module

Chunk maker– responsible for organizing video stream packets into

chunks. Chunk-level distortion generator

– three different ways are designed to implement chunk-level distortion generator

Chunk buffer manager– manages and keeps the received chunks in a local chunk-

level buffer (serving other peers later) Playback controller (client software)

– make playback decision for each chunk. Video decoder

– After being decoded by the video decoder component, the processed videos (PVS) are then displayed in the monitors to the users.

Chunk-level impairment model

Various distortion sources:a) Peers’ dynamic behaviors

b) Peers’ network condition (Uplink and Downlink)

c) Chunk/peer selection strategy

d) Scheduling algorithm, etc……

Consider these factors too complicated What we can do make abstraction Chunk receiving pattern (CRP) is the

equivalent distortion effects of all the distortion sources considered together.

Chunk-level impairment model

General model of chunk-receiving patterna) ri(x) – download percentage of chunk, any non-decreasing function

b) Li – chunk size (all equal to l in the example below)

c) Xis – starting downloading time

d) Xic – complete downloading time

e) Xid – desired playback time, depend on previous chunk

f) Xip – real play back time, playback decision

Three conditions:r1(x): non-delayed chunk, playb

ack normally.r2(x): delayed chunk, wait and

playback after completionr3(x): delayed chunk, wait and

playback before completion

Chunk-level impairment model

Three ways to generate chunk-receiving patterns

1. Live experimentsCollecting and recording the CRP for each chunk a real-life experiment (not repeatable)

2. Simulation resultsCollecting and record CRPs from the simulation trace in a large network with a large number of users. (Simulation is repeatable under same settings, CRPs are following certain distributions)

3. Artificial generatingManually generate CRPs (by implementing ri(x) with certain increasing curves and parameters, completely repeatable) suitable for subjective testing studies

Chunk-level impairment model

The parameter space for subjective testing should not be too large!

In the previous work, we start from the simplest form of CRP: the step function

X3c

X3s

X1c

X2s

l

X3pX2

p

X2c

X1d

X1s

r3(x)r2(x)r1(x)

x

r(x)

0.2

0.4

0.6

0.8

1.0

X2d X3

d

X1p

l

In the previous experiment: Step function assumes that Xi

s = Xi

c for all chunks.

Two types of chunks:non-delayed (Xi

c <= Xid) and d

elayed (Xic > Xi

d);

Simple playback strategy:Xi

P = Xid + LWT

Non-delayed,Normal playback

Delayed, wait andnormal playback

Delayed, wait and skip

Experiments under various CRPs: I

Experiments under various CRPs: I

In the previous work, we start from the simplest form of CRP: the step function

Fixed LWT = 3 seconds Two types of chunk delay D (= Xi

c – Xid) distribution

1. Short delay: uniformly distributed in [0, 2] seconds; or 2. Long delay: all equals to 3 seconds, ( = LWT).

X3c

X3s

X1c

X2s

l

X3pX2

p

X2c

X1d

X1s

r3(x)r2(x)r1(x)

x

r(x)

0.2

0.4

0.6

0.8

1.0

X2d X3

d

X1p

l LWT

D2 D3

Experiments under various CRPs: I

In the previous work, we start from the simplest form of CRP: the step function

Fixed LWT = 3 seconds Two types of chunk delay D (= Xi

c – Xid) distribution

1. Short delay: uniformly distributed in [0, 2] seconds; or 2. Long delay: all equals to 3 seconds, ( = LWT).

Number of delay chunks is determined by applying different values of average discontinuity (d = 1 - c): 0, 5%, 10%, 15%, 20%, 25%, 30%, 40%, 50%, 60%

Experiment settings I:1. 50 source video clips (News, Music videos, Movie Trailers and Sports)

with average length of 30 seconds;2. 30 subjects (16 males and 14 females), age range (18 - 28);3. Absolute Category Rating (ACR) with hidden reference as assessment

Experiments under various CRPs: I

In the previous work, we start from the simplest form of CRP: the step functionSubjective assessment results for each processed video sequence MOS value (left), DMOS value (right):

The meaning for Mean Opinion Score (MOS) and DMOS:

Experiments under various CRPs: I

In the previous work, we start from the simplest form of CRP: the step function

Comparison between short and long chunk delay distribution

Insights from the comparison:1. PVSes with long delay distribution obtain higher MOSes than

those with short delay distribution when average d is same.2. Subjects care more about the number of screen freezing

events than the duration of each freezing event.

In this work, we are trying more complicated form of CRPs: the piecewise linear function

In this experiment, we assume: Two types of chunks:

non-delayed (Xic <= Xi

d) and delayed (Xi

c = Xid + LWT > Xi

d);

For delayed chunks, three piecewise linear patterns;

Playback strategy with D:Xi

p = Xid + D (D <= LWT)

R(D): Chunk completeness when playback (depend on D and Pattern)

Experiments under various CRPs: II

LWT

RA(D)

RB(D)

RC(D)

Xid Xi

c =Xid + LWTXi

p = Xid + D

In this work, we are trying more complicated form of CRPs: the piecewise linear function

In this experiment: Fixed probability of delayed

chunks = 0.1; i.e., 3 out of 30 chunks are delayed;

Fixed LWT = 4 seconds; Fixed Xi

c = Xid + LWT;

5 playback decisions: D = Xi

p – Xid = 0, 1, 2, 3, 4

seconds. Real implementation of

R(D):

Experiments under various CRPs: II

Xid Xi

c =Xid + LWT

LWT

Xip = Xi

d + D

RA(D)

RB(D)

RC(D)

In this work, we are trying more complicated form of CRPs: the piecewise linear function

Experiments under various CRPs: II

Experiment settings II:1. 56 source video clips (News, Music videos, Movie Trailers and Sports)

with average length of 30 seconds;2. 42 subjects (19 males and 23 females), age range (18 - 28);3. Absolute Category Rating (ACR) with hidden reference as assessment

In this work, we are trying more complicated form of CRPs: the piecewise linear function

Experiments under various CRPs: II

Insights from the comparison among four video categories:1. News earns the highest scores in all patterns, followed by MV. This

is due to the nature of these two categories.2. The overall difference among three patterns are not very big,

probably because we fixed the number of delayed chunks.

Patten A Patten BPatten C

We try to model the trade off between Waiting time D and Chunk completeness R.

Heuristic Satisfaction function: S(R, D) = alog (R) + bDc, a >= 0, b <=0, c > 0

So we have:

This is one possible choice, other forms may investigate in future.

Heuristic on satisfaction function

After applying the three patterns to the heuristic satisfaction function with proper parameters:

Experiments under various CRPs: II

Insights from the comparison among S-function and experiment results:1. The S-function does not match the results so well, more proper form of S-

function needs to be studies and investigated.2. However, the optimal D values for three patterns are similar as shown in

both figures. It indicates that, usually, for pattern like A, small waiting time is better while patterns like C, longer waiting time is more preferred.

Future work Carry out more experiments with different chunk

receiving patterns and parameter settings, e.g. number of delayed chunks, LWT, etc.

Investigate various forms of S-functions to fit the results.

Consider layered coding in P2P streaming (another form of D and R)

Conclusion Chunk-level impairment model is proposed for P2P

mechanism. We introduce two sets of QoE Experiments by

applying chunk-level impairment model. The results are preliminary but still get some

interesting insights.

Future work and conclusion

The end

Thanks!

Q & A