Introduction to Middleware

The protocols used for conference media streams The protocols to setup and control various aspects of a conference, such as membership, session information, media activity, floor control and so on.

Introduction to Middleware

Here middleware is the glue software try to combine the multimedia system together, normally it includes Transport ProtocolsSession Directories, Advertisement and Invitation Protocols; Conference Control Specifications.

Transport ProtocolsDifference between TCP and UDP at the

viewpoint of multimediaMultimedia transport protocols vs.

conventional data servicesA basic framing service is neededMultiplexing may be neededTimely delivery and trade off against

reliabilitySynchronization

Transport ProtocolsRTP, the preferred carrier/framing

protocol for multimedia traffic, is carried on top of UDP. It has the potential for further multiplexing information, since it carries source identifiers.

RTCP, the Real Time Control Protocol, is used to convey additional information such as participant details and statistics of packet arrivals and loss.

RoadmapThe transport protocol for multimedia

provides some functions that are different than those for elastic reliable data transfer

RTP provides payload identification, multiplexing, feedback from receivers to senders of reception conditions and hooks for synchronization. This chapter will also talk about transportation reliability of multimedia.

TCP Adaption Algorithms TCP will smoothly degrade performance of

everyone as load increases(rather than blocking access). This is done by adaption algorithms.

Adaption in TCP is both to RTT and send rate. Based on the idea of adaption, multimedia

protocols adapt to delay at the receiver to: Alter adaptive playout buffer to smooth playbackSynchronize streams from different sources at a

receiver

TCP Adaption Algorithms Algorithm named Exponential Weighted Moving

Average: Assume we measure the arrival time for each (i'th)

packet as IATi, then the simple average would be: Av(IATi) = [ sumi=1 to n ( IATi) ]/n. But since the average is not fixed, the most recent measurements are more confidential, so use:

Av(IATi) = IATi*alpha +(1- alpha)*Av(IATi-1), where 1> alpha > 0.5

Once have a mean IAT, required playback buffer can be calculated( roughly twice of the variation of IAT)

TCP Adaption Algorithms Synchronization between two sites by exchange

of packetssend a packet from s to d with source time in t1, its

arrives at d when clock of d reads t2send the packet back to s with t1, t2 and t3(the time

at d when sending)s gets the packet at the time when its clock is at t4 if s and d have clocks that differ by offset, and

network delay is delay t4=t3+offset+delay, t2=t1-offset+delay so: delay=(t4+t2-t3-t1)/2, offset=(t4+t1-t2-t3)/2

Playout Algorithms For a variety of reasons, samples may arrive at

different times than they were sent. The variation of arrival time is a second order effect compared with the delivery delay. This can be coped with playout buffer with the size of the variation of the mean of the arrival time. Audio, video and other interactive application

receivers generally use an adaptive playout scheme.Senders are generally adaptive to reported network

conditions. The report should be constrained to be a constant percentage of the total session.

MPEG Systems MPEG systems is the control part of the

MPEG standard. It addresses the combining of one or more streams of video and audio as well as other data into a single or multiple streams which are suitable for storage or transmission. The stream is Packetised Elementary Stream(PES).

Transport and Program Streams Transport Stream(TS), can carry multiple

programs simultaneously, and is optimized for use in applications where data loss may be likely.

Program Stream(PS), is optimized for multimedia applications, for performing systems processing in software and for MPEG-1 compatibility.

Transport and Program Streams Synchronization

The basic principle of MPEG system coding is the use of time stamps provided by RTP, which specify the decoding and display time of audio/video and the time of reception of the multiplexed coded data at the decoder.

This method allows a great deal of flexibility in such areas as decoder design, the number of streams, multiplex packet lengths, video/audio frame rates digital storage medium or network performance.

RTP RTP is the Internet Standard for conveying

media streams between interactive participants.RTP Packet FormatRTP Header CompressionRTP MultiplexingRTCP Packet FormatPayloadsRTCP Scaling Properties and Timer

Considerations

RTP Packet Format 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P|X| CC |M| PT | sequence number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | timestamp | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | synchronization source(SSRC) identifier | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | contributing source(CSRC) identifiers | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

The first twelve octets are present in every RTP packet, while the list of CSRC identifiers is present only when inserted by a mixer.

RTP Header Compression The combination of the IP,UDP and RTP control

information adds up to a significant overhead for small media samples. IP(20 bytes)+UDP(8 bytes)+RTP(12 bytes)=40 bytes

A technique for reducing the overhead of packets, designed for compressing TCP/IP headers specified in RFC 1144 by Jacobson, was adapted by Casner and Jacobson to RTP headers, shows a typical reduction to a header size of 3-4 bytes for all control overhead.

RTP Header Compression The technique consists two parts:

Noting fields in the packet headers that do not change over the life of a flow. These are the second order information which can stored in the compression state

Noting that there are few flows at the edge of the network, so such information can be conveyed over the first hop by a single packet, and subsequently referred to by a short “connection identifier, which serves to index the full state so that the first op router can reconstruct the full packet.

RTP Header Compression The features of the header compression:

The compression for RTP header is basically the same as TCP/IP except in exceptional circumstances

The compression state is a soft state. It can be recovered if lost since the packet conveys enough implicit information that end-to-end checksums are still computed, and hop-wise recomputed from the state vector and from the remaining data in the compressed header. That is to say, when the router resets/changes or the network alters states, the invalid checksum causes a reset of the compressed state so that a new packet is created again.

RTP Multiplexing There are a number of circumstances that

multiple media flows within a single RTP data payload between two points is needed. There are at least two ways to multiplex data:Assuming all samples have the same payload types, so

just use a mapping table to save offsets for each end-to-end flow.

A more generic method suggested by Mark Handley is to adapt the ideas from previous section about RTP header compression and to allow for multiple compressed headers within a single RTP packet, one for each of the samples.

RTCP Packet Format 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ |V=2|P| RC | PT=SR=200 | length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | SSRC of sender | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | NTP timestamp, most significant Word (sender info ) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | NTP timestap, lest significant word (sender info ) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | RTP timestamp | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | sender’s packet count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | sender’s octet count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

RTCP Packet Format +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | SSRC_1 (SSRC of first source ) (report block 1 ) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | fraction lost | cumulative number of packets lost | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | extended highest sequence number received | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | interarrival jitter | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | last SR(LSR) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | sender’s packet count | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | delay since last SR(DLSR) | +=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+=+ | SSRC_2 (SSRC of second source) ... (report block 2 ) | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Payloads There are quite a few payloads defined for

RTP, ranging from owens designed to carry video/audio formerly available on hardware CODECs only, such as MPEG, H.261, and the various CCITT/ITU voice coding schemes, through to some more generic payloads designed to carry redundantly coded audio such as that used by INRIA’s Freephone.

RTCP Scaling Properties & Timer Considerations

The intention and original design goal of RTCP messages was to act as a distributed source of lightweight session data that would allow a range of highly fault tolertant, and seasonable scale mechanisms to be built including:Membership RTCP reports carry the

identifier of participants.Loss statistics RTCP reports convey

information about packet losses.

RTCP Scaling Properties & Timer Considerations

RTCP reports are designed to be sent periodically, with a frequency inversely proportional to the number of members. Though this was set to constrain the bandwidth consumed by RTCP to be a small fixed fraction of the total capacity of the session, there are three circumstances where this approach creates a problem:A flood of initial packets of reports at the startThe flood of “BYE” packet at the end of sessionThe reports arrives at all participants

Synchronization Some kinds of synchronization:

Intra-stream Synchronization inside a stream, for media like video/audio, whiteboard inside a stream

Inter-stream To see two streams’ reactions to the occurrence of a third stream

inter-media Just lip-synch

Intra-stream Synch Intra-stream synchronization is a base

part of the H.261 and MPEG coding systems. H.221 and MPEG systems specify an encapsulation of multiple streams as well as carrying timing information in the stream.

In the Internet, the RTP media specific timestamp provides a general way of carrying out the same above function.

Inter-stream Synch The easiest way of synchronizing between streams

at different sites is based on providing a globally synchronized clock.Use a single clock which is propagated around the

networkUse a clock synchronization protocol, such as

NTP(Network Time Protocol), DTS(Digital Time Service). The network continually exchanges messages about clock offsets and network delays.

Or use timestamps to adjust clocks and measure delays accordingly, and then insert a baseline delay into the adaptive playout algorithms.

Inter-media Synch Use options like Inter-stream Synch approaches More generally, encapsulate the media in the same

transmission stream. It is effective but may cause expensive computation at the receive side to decode all media at the same time.

Alternatively, use much the same scheme as to synchronize different sources from different places. This approach uses timestamps, assuming the media are timestamped accurately at the “real” source points which the receiver can use to synchronize multiple media.

Reliable Multicast Transport “Reliable Multicast” is not people normally

think to be a single protocol at a single ‘layer’ of a protocol stack. The differences are possibly:Fate sharingPerformanceSemantics

In this section we will also talk about congestion avoidance for reliable multicast applications and reliable multicast framework.

Fate Sharing Fate sharing in unicast applications means

that if two applications has set up connection, then TCP can keep the connection as long as they like. However, if either party fails, the connection certainly fails.

Fate sharing between multicast endpoints is a more subtle idea. It is a application-oriented question. Normally we must permit any recipient timeouts/Fails.

Performance The performance measurement is

different for a talks to b, from a talks to b,c,d,e,f.The previous situation is limited by 1 path,

what can be done to improve the throughput(or delay bound) is done by IP.

For the second situation, the throughput or delay may not be that sustainable even in the slowest transferring state.

Semantics Applications such as software distribution are

cited as classic 1-n service, telemetry is given as a n-1 reliable protocol, shared whiteboards are cited as examples of n-m.

1-n and n-1 services are those where reliability can be dealt with by rounding up the missing bits at the end of the transfer.

N-m processes need timely recovery. For each recipient, the timely order is very important.

Congestion Avoidance for Reliable Multicast

One of the big research areas not quite complete is how to achieve flow and congestion control for reliable multicast applications.

There are a lot options when confronted with a congested link and sending to a group. The best option depends on the nature of the application. The options can be viewed as sending at the slowest,average or fastest rate.

The global congestion picture is not at all clear for either reliable or unreliable multicast.

Reliable Multicast Framework Handley proposed a framework about reliable

multicast protocol. He described the categories of parameters of the system:Number of source: 1-to-n, n-to-m or other.Transmitting start time: Is there such a time? Is there

one for each receiver marked at join time?Real time-Consideration: About the latency constraints

on delivery of data.Consistency.

He listed congestion control and ordering as non-factors.

Reliable Multicast Framework He grouped applications into five main areas:

Bulk Transfer(1-n,relaxed real time, synchronized start time, object level consistency)

Live data feeds(1 source, no start time, quasi real time, may need synchronized consistency)

Resilient(1 source, no start time, nearly real time, non consistency)

Share applications(distributed data set, relaxed consistency, many changing sources)

Hybrid - some applications maybe a hybrid of all of the above.

This will be discussed further in chapter eight.