Real-time smoothing for network adaptive video streaming

Kui Gao, Wen Gao, Simin He, Yuan Zhang

J. Vis. Commun. Image R. 16 (2005)

Outline

MotivationThe architecture of network-adaptive streaming systemA Real-time dynamic smoothing algorithm

How to select and schedule packetsHow to retransmit the lost packet

Simulation resultsConclusion

MotivationProblems:

Variation of network bandwidthPacket may lost

The current InternetProvide best-effort servicesDo not provide QoS

Goals:Provide a smoothing quality at a client end (real-time smoothing)Maximizing the utilization of the variable network bandwidth (adaptive video streaming)

• Propose a dynamic real-time smoothing algorithm to select and scheduling the packet

– Use FGS streaming to maintaining a very flexible and simple video-coding structure

The architecture of network-adaptive streaming system

Server

Client

Use a real-time smoothing algorithm

Feedback information includes: Packet loss rate, RTT(round-trip time)

How to select and schedule the packet

ConsiderationsPlayback distortion (lost, damaged)Network bandwidthPacket Deadline (arrive before the playback time)Packet dependency

SummaryDifferent portions of video bitstream have different contributions to the video quality.The important packets (with large distortions) can be transmitted earlier.

• The packets of base layer are the most important.

Feedback information

When packet loss, how to recover

Use retransmission scheme to recover lost packetBut not all lost packet can be recovered

Due to the available network bandwidth and the delivery deadline constraint

Summary:The lower layer has more chances to retransmit its packets.

The characteristics of the transmission buffer (1/2)

The packet in transmission buffer has two states: In the ready states:

• Can be scheduled for transmissionIn the blocked states:

• Can not be scheduled until packet loss or its timeout RTO≦tout

How to select a packet to transmit at tcur from the transmission buffer

Sbuf is the set of all the ready packets Sbuf = {pi,j|ai,j≦tcur and tcur+ci,j≦di and in ready state}pi,j: the packet of the jth layer in frame Fi

ai,j: release time (the earliest time the packet in the buffer)

ci,j: process time of pi,j

di: deadline (the latest time pi,j should be sent to client)

The characteristics of the transmission buffer (2/2)

How to selectFirst step

• Server selects the ready packets of base layer with earliest deadline firstly (EDF)

Second step• If no base layer packets in ready state, the serer

select and schedule the ready packets of enhancement layers

• In enhancement layer– According to rate-distortion decision– If the packets have same distortion, packet with

earliest deadline are served first

The transmission flow

Base 0

Layer 1

Layer 2

Layer 3

Base 0

Layer 1

Layer 2

Layer 3

Base 0

Layer 1

Layer 2

Layer 3

Base 0

Layer 1

Layer 2

Layer 3

… 4 3 2 1 (Frames)

(1) According to feedback to decide ai (release time)

(2) Put packets into buffer according to ai

Transmit buffer

ai=4 ai=3 ai=2 ai=1

Ready state

Block state

client

Base 0Base 0Layer 1Layer 1Layer 2Layer 2Layer 3Layer 3

Feedback information & (ACK or NAK)

(3) Base layer transmit first

Enhancement layer transmit according to the distortion

ai: the earliest time the packet in the bufferFeedback includes: packet loss rate & RTT(round-trip time)ACK: packet unlossNAK: packet loss

A Real-time smoothing algorithm

n frames, a scalable media stream F = {F0,F1,…,Fn-1}m: the number of layers pi,j (0≦i≦n-1, 0≦j≦m): the packet of the jth layer in frame Fiai: release-time (the earliest time pi,j in the transmission buffer)di: deadline (the latest time pi,j should be sent to client)

assume the packets in the same frame have the same ai and di

ploss(t): packet loss rateRTT(t): round-trip-timeRTO(t): retransmission timeoutThe processing time of pi,j: ci,j = bi,j/X(t)

bi,j: the size of the packet pi,jX(t): the available network bandwidth at time t

The fulfill-time of pi,j : fi,j = si,j + ci,j si,j: the scheduler time of pi,j

(fulfill-time = scheduler time + processing time)

feedback information

How to decide the release time (ai)MPEG-4 video can accept an error rate of 10-5 or lowerHow to decide the release time (the earliest time pi,j into the transmission buffer)

If the release time is too early• The transmission buffer must be very large

If the release time is too late• There is no chance to retransmission

It is important to select the retransmission times K for lossy packets of the base layer

Let be the total loss probability of the pi,0 through K times retransmission, we get the smallest K that ≦10-5

ai = di – K * RTO (K & RTO decided by feedback)• di: deadline (the latest time pi,j should be sent to client)

)(0, Ki)(0, Ki

Constraint of playback (In enhancement layer) (1/2)

It can be Arbitrarily truncated (decided by the available channel bandwidth)Tolerate the channel errors

A packet pi,j can be selected and scheduled if satisfied

The current time is later than release time (ai ≦ tcur )The fulfill-time is earlier than deadline(ai ≦ tcur and tcur +ci,j ≦ di ) p.s: ci,j = process time of pi,j

According to rate-distortion optimized decision

Constraint of playback (In enhancement layer) (2/2)

Assume the quantization of a frame is Qif the maximum number bit-plane of a frame is m, and the last bit-plane is z, then Q=2m-z

A distortion model built for uniform quantizer (UQ) is

The distortion of packet pi,j is

The total quality is

when J is maximize, the total quality is the best

12,)(2

whereQ

QD

Fp pp

yxlossjilossji

ji jiyx

ppppQDJ, ,,

))(1(*))(1(*)( ,,,

jiyx pp

yxlossjilossjiji ppppQDpD,,

))(1(*))(1(*)()( ,,,,

Simulation results (1/3)Use TFRC protocol to decide the sending rate

s: the packet size, R is RTT (round-trip time), tRTO is retransmission timeout value, and p is the pack loss ratio.

The sequences include Foreman, Coastguard and AkiyoEncoded with 30 frames per secondThere are 7 enhancement layerThe playback frame rate is 30Hz

Simulation results (2/3)

Simulation results (3/3)

Conclusion

Propose a real-time smoothing algorithm

According to feedback information• to select and schedule the packet

– under a rate-distortion optimized

The algorithm improves the utility of the bandwidth smoothes the playback quality