Download - L6 Internet Streaming
Lecture 6:
Internet Streaming Media
A/Prof. Jian ZhangNICTA & CSE UNSW
Dr. Reji MathewEE&T UNSW
COMP9519 Multimedia SystemsS2 2010
COMP9519 Multimedia Systems – Lecture 6 – Slide 2 –J. Zhang & R. Mathew
Background
� So now you can code video (and audio)
� But how to transfer this video to your mates ?
� Possibilities� File Transfer – TCP
� Streaming for real-time delivery – TCP, UDP
� Protocols for Enabling Streaming � RTP – Transport
� RTCP – Control
� SDP - Description
� RTSP - Signaling
COMP9519 Multimedia Systems – Lecture 6 – Slide 3 –J. Zhang & R. Mathew
Lecture Outline
� Introduction to Streaming
� Streaming Protocols� RTP/RTCP
� System Overview� System for streaming video and audio on the Internet
� Overview of important components
� Sender side issues
� Receiver side issues
� Client Architecture : An Example
� Session Description Protocol� How to describe a multimedia session ?
� SDP example
COMP9519 Multimedia Systems – Lecture 6 – Slide 4 –J. Zhang & R. Mathew
Introduction
� Facts on TCP (Transmission Control Protocol)� Data sent on a TCP connection is delivered
� reliably and
� in order at the destination.
� Transmission made reliable via � use of sequence numbers and
� acknowledgments.
� Little control over the timely arrival of data� Arbitrary number of retransmissions
� Not suitable for real-time streaming� Real-time applications have stringent limits on acceptable delay
� eg video conferencing
COMP9519 Multimedia Systems – Lecture 6 – Slide 5 –J. Zhang & R. Mathew
Introduction
� Facts on UDP (User Datagram Protocol)� Datagram mode of communication � Send messages with a minimum of protocol mechanism� Unordered datagram service� delivery and duplicate protection are not guaranteed
� UDP is unreliable � But used when timeliness is more important
� you have control over when data is sent� Preferred for streaming
� TCP is more reliable� Used when reliability is more important (than timeliness)
� Eg, file download
COMP9519 Multimedia Systems – Lecture 6 – Slide 6 –J. Zhang & R. Mathew
Introduction
� File Transfer� Examples : HTTP, FTP
� Using TCP
� Wait for file to be downloaded
� Lengthy delay for large files
� Not suitable for real-time applications
HTTPinternet
HTTP
Server
Request for a file
Download whole file
Play File
COMP9519 Multimedia Systems – Lecture 6 – Slide 7 –J. Zhang & R. Mathew
Introduction
� Streaming� Using TCP (not preferred)
� Wait until some content is downloaded to a buffer
� Start playing out from buffer while server/client run recive/ack procedure
� If data rate drops then will need to stop play out� Re-transmission due to congestion and packets loss
� Frequent start and stop of play out for Resync. Play Out Buffer
HTTP
d bits
r bits/sinternet
HTTP
Server File Size = D bits
File Viewing Time = T s(D - d) / r < T
TCP
Ack/receiver
procedure
COMP9519 Multimedia Systems – Lecture 6 – Slide 8 –J. Zhang & R. Mathew
Introduction
� Streaming� Using UDP (preferred)
� Transmit data rate matches coded rate of media� Eg Transmit rate = R = D/T
� Start playing out as long as data arriving� Small delay (smaller than when using TCP)
� Playing out from buffer regardless the traffic congestion and packets loss
� Play out rate = R
Play Out Buffer
UDPinternet
File Size = D bits
File Viewing Time = T s
Tx
File
COMP9519 Multimedia Systems – Lecture 6 – Slide 9 –J. Zhang & R. Mathew
Introduction
� So we choose UDP/IP (over TCP/IP) for streaming
� But multimedia data have other requirements� Sequencing:
� to reorder packets at the receiver that are out of order
� to detect lost packets
� Intra-media synchronization:� timing information for play out of successive packets
� Eg, period of time to show/hold a single video frame
� Inter-media synchronization:� timing information to synchronize multiple media streams
� Eg, synchronize audio and video streams (lip sync)
COMP9519 Multimedia Systems – Lecture 6 – Slide 10 –J. Zhang & R. Mathew
Introduction
� Payload identification: � Communicate the codec being currently used
� Allow codec switching to adjust to bandwidth or user capabilities
� Frame indication: � To indicate start and end of a frame or access unit
� Helps in delivery of data to higher layers
COMP9519 Multimedia Systems – Lecture 6 – Slide 11 –J. Zhang & R. Mathew
Internet Streaming Protocols
� How to meet requirements for multimedia streaming� Solution - Real Time Transport Protocol (RTP)
- Real Time Control Protocol (RTCP)
� RTP/RTCP� Defined in a generic way
� Developed by IETF
� RFC 3350 RTP: A Transport Protocol for Real-Time Applications
� More info ftp://ftp.rfc-editor.org/in-notes/rfc3550.pdf
� Designed to be multi-cast friendly
� Generally used with UDP� Although designed to be independent of underlying transport
� And network layers
COMP9519 Multimedia Systems – Lecture 6 – Slide 12 –J. Zhang & R. Mathew
RTP Overview
IP
fragment
RTP
Attach RTP
Header
UDP UDP
RTP
Decoder• Time stamping
• Sequence numbering
• Payload ID
• Frame indication
compressed bit stream
COMP9519 Multimedia Systems – Lecture 6 – Slide 13 –J. Zhang & R. Mathew
RTP Header Fields
� Marker (M) : 1 bit
Intended to allow
significant events such
as frame boundaries to
be marked in the packet stream
MPEG-4 Example : set to 1 to indicate last RTP packet of a VOP.
M=0 M=0 M=1
One compressed frame or VOP
Broken up into 3 packets
The first twelve bytes (octets) are
present in every RTP packet.
COMP9519 Multimedia Systems – Lecture 6 – Slide 14 –J. Zhang & R. Mathew
RTP Header Fields
� Payload Type : 7 bits
� PT
This field identifies the
format of the RTP payload
(eg payload is MPEG-4 video or u-law audio ….)
Static mapping
Payload Type value codec
COMP9519 Multimedia Systems – Lecture 6 – Slide 15 –J. Zhang & R. Mathew
RTP Header Fields
� Sequence Number : 16 bits
The sequence number is incremented by one for each RTP packet sent. Usually made to start at a random number.
Sequence number can be used by the receiver to reorder out of order packets and detect packet loss.
COMP9519 Multimedia Systems – Lecture 6 – Slide 16 –J. Zhang & R. Mathew
RTP Header Fields
� Time Stamp : 32 bits
The timestamp specifies the sampling instant of the first octet in the RTP data packet.
Example : two audio samples in one packet, timestamp refers to the sampling instant of the first sample.
one RTP packet
Timestamp of packet
= sample time of first packet
COMP9519 Multimedia Systems – Lecture 6 – Slide 17 –J. Zhang & R. Mathew
RTP Header Fields
� Time Stamp : 32 bits
The timestamp is derived from a clock that increments monotonically and linearly in time.
The clock frequency is dependent on the payload format.This means RTP timestamps are in units which are meaningful for the format of media being carried.
Examples :
� 8KHz sampled u-law audio would use an 8KHz sample driven clock for timestamps.
� MPEG video would use the MPEG 90KHz clock.
COMP9519 Multimedia Systems – Lecture 6 – Slide 18 –J. Zhang & R. Mathew
RTP Header Fields
� Time Stamp : 32 bits
RTP Timestamps can be used
at the receiver for intra media synchronization.
At the receiver inter media synchronization uses RTP Timestamps with the help of RTCP.
� clocks for audio and video streams from the same source use different timebases.
� RTCP relates these different clocks to a reference clock (wall clock), so audio-video synchronization can be performed.
Random start
timestamp
audio clock = 8kHz
video clock = 90kHz
x x + num_cycles …
y + num_cycles …y
COMP9519 Multimedia Systems – Lecture 6 – Slide 19 –J. Zhang & R. Mathew
RTP Header Fields
� SSRC : 32 bits
� Synchronization source� identifies an RTP sender
� value chosen randomly
� identify sources in multi-sender sessions
id2
id1
ssrc = id2
ssrc = id1
AMR
AMR
u-law
u-law
translator mixerssrc = id3
ssrc = id3
csrc = id1, id2
COMP9519 Multimedia Systems – Lecture 6 – Slide 20 –J. Zhang & R. Mathew
RTP Header Fields
� CSRC : 32 bits each
� Contributing Sources
� Identifies contributing sources
� number of contributing sources specified by CC field
� CSRC identifiers are inserted by mixers
� using SSRC of contributing sources
id2
id1
ssrc = id2
ssrc = id1
AMR
AMR
u-law
u-law
translator mixerssrc = id3
ssrc = id3
csrc = id1, id2
COMP9519 Multimedia Systems – Lecture 6 – Slide 21 –J. Zhang & R. Mathew
RTP
� RTP Header Fields
� CSRC : 32 bits each
� Contributing Sources
� Example use of CSRC
Audio conference where multiple audio streams are mixed together to form an output packet stream.
The CSRC list identifies all the sources that have been mixed together thereby allowing identification of current participants or talkers in the session.
Note that mixers will generate its own timing for the combined stream, therefore a mixer is identified by its own SSRC
COMP9519 Multimedia Systems – Lecture 6 – Slide 22 –J. Zhang & R. Mathew
RTCP
� RTP – allows for transport of media data
� But….� How do we know the packets are being received ?
� packet loss rate ?
� available bandwidth ?
� What are the bounds on delay and jitter ?
� Need some feedback on network performance
� Real-Time Control Protocol (RTCP)� Used in conjunction with RTP
� Primary function – feedback on quality of data distribution
COMP9519 Multimedia Systems – Lecture 6 – Slide 23 –J. Zhang & R. Mathew
RTCP� Control information exchanged between session members by RTCP packets
� Five RTCP packet types defined
� RTCP packets begin with a fixed part� Similar to RTP data packets
� RTCP packets are sent periodically � from all participants of a session� multicast example
� RTCP packets are stackable – compound packets� concatenate multiple packets
� Usually sent over UDP (same as RTP)� Using a port number = RTP_port + 1
COMP9519 Multimedia Systems – Lecture 6 – Slide 24 –J. Zhang & R. Mathew
RTCP
� RTCP packet types to carry control information
� RTCP SDES : Source description items� sent by all session members� includes a persistent identifier of sources in the session
� RTCP SR : Sender Report� sent by active senders� report of transmission and reception statistics
� RTCP RR : Receiver Report� Sent by receivers (not active senders)� report of reception statistics
� RTCP BYE : Packet to indicate end of participation� RTCP APP : Application specific functions
COMP9519 Multimedia Systems – Lecture 6 – Slide 25 –J. Zhang & R. Mathew
RTCP
� RTCP SDES Packet : � Source description items� Source identifier item called CNAME is mandatory� CNAME – canonical name� Receivers map SSRC and CNAME
� Note : SSRC can change but CNAME is persistent� CNAME also required by receivers to associate multiple streams from a single participant (eg synchronize auido + video)
� Other descriptions items include� NAME : of user� EMAIL : of user� PHONE : number of user� LOC : geographic user location, depends on application� TOOL : name of application generating the stream
COMP9519 Multimedia Systems – Lecture 6 – Slide 26 –J. Zhang & R. Mathew
RTCP
� RTCP Sender Report Packet
SSRC ID for the originator of
this Sender Report
Network Time Protocol
timestamp (wallclock time) when
this report was sent
Associated RTP time stamp
(same instant as NTP timestamp).
Used for synch at the receiver
Total number of RTP packets
sent since start of transmissionTotal number of payload octets
sent since start of transmission
COMP9519 Multimedia Systems – Lecture 6 – Slide 27 –J. Zhang & R. Mathew
RTCP
� RTCP Receiver Report Packet
PT=RR=201
Fraction of RTP packets lost
since last Receiver Report
Total number of RTP packets
lost since start of transmission
Highest RTP Sequence
number of packets
received
RTP packet
inter-arrival time
RTP timestamp of last
Sender Report
received from source
Delay between receiving
Sender Report and sending
this Receiver Report
SSRC ID for the originator of this RR SSRC ID of sender
COMP9519 Multimedia Systems – Lecture 6 – Slide 28 –J. Zhang & R. Mathew
RTCP
� RTCP Bandwidth Utilization
If transmission rate of
RTCP packets were fixed
then RTCP traffic would
grow linearly with number of receivers
� RTCP rate dependents on the number of participants
� Increase in participants (eg receivers) then decrease rate
� Need to keep control traffic to a small fraction at all times
� So as not to impair the primary function of data transport
� Need to have common known rules to follow in calculating the bandwidth used for control traffic
� So that each participant can independently calculate its share of the control bandwidth
COMP9519 Multimedia Systems – Lecture 6 – Slide 29 –J. Zhang & R. Mathew
RTCP
� RTCP Bandwidth Utilization (continued)
� RTCP rate dependents on the number of participants
� Try to keep control traffic to 5% session bandwidth
� Senders collectively allocated 25% of control bandwidth
� Receivers equally share the remaining 75% of control bandwidth
� Interval between RTCP packets is varied randomly � To avoid unintended synchronization of all participants.
� Varied [0.5,1.5] x interval
� Interval between RTCP packets is required to be > 5 s
COMP9519 Multimedia Systems – Lecture 6 – Slide 30 –J. Zhang & R. Mathew
System Overview
IP
UDP
RTP
Decoder
• Reorder packets
• Identify lost packets
• Detect payload type
• Detect participants
• Packets to Frames
fragment
RTP
Attach RTP
Header
UDP
compressed bit stream
• Time stamping
• Sequence numbering
• Payload ID
• Frame indication
• Source ID
COMP9519 Multimedia Systems – Lecture 6 – Slide 31 –J. Zhang & R. Mathew
System Overview
IP
UDP
RTP
Decoder
packetize
RTP
UDP
RTCP
RTCP
MPEG-4
File / EncoderMPEG-4
VideoSender Side Issues
-Conforming to Network Bandwidth
-Adaptive Rate Control
-Error Resilience
-Packetization strategy
Receiver Side Issues
-Buffering input data
-Error concealment
-Inter media synchronization
COMP9519 Multimedia Systems – Lecture 6 – Slide 32 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Conforming to Network Bandwidth� Example : live video encoding
� Rate Control to match available bandwidth
� Example : Pre-recorded video
� Multiple copies of content at various bit rates� Choose appropriate version to match bandwidth
MPEG-4 Encoder
Rate
Control
64 k bits/s
128 k bits/s
256 k bits/s
64 k bits/s 128 k bits/s 256 k bits/s
COMP9519 Multimedia Systems – Lecture 6 – Slide 33 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Conforming to Network Bandwidth� For multiple participants with different bandwidth access� Scalable coding is an option� Example : 3 layer coding
� Base Layer + 2 Enhancement layers
� MPEG-4 scalability optionsTemporal, SNR, Spatial
� Rate control for each layerto ensure combined rates meet bandwidth
MPEG-4 Encoder
Rate
Control
internet
64 k bits/s
90 k bits/s
110 k bits/s
COMP9519 Multimedia Systems – Lecture 6 – Slide 34 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Adaptive Rate Control� Even though rate control meets nominal bandwidth rates
� Fluctuations of available network bandwidth still occur
� Schemes to dynamically control output rate are required � Make rate control dynamically adjust to network conditions
� RTCP provides some feedback about network
� Therefore can formulate adaptive rate control using RTCP
� Example :
[a] “ On End-to-End Transport Architecture for MPEG-4 Video Streaming over the Internet ” , D Wu et al, IEEE CSVT, Vol. 10, No. 6, September 2000
COMP9519 Multimedia Systems – Lecture 6 – Slide 35 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Adaptive Rate Control
� Use packet loss ratio returned in the RTCP packet
� This is the packets loss experienced by the receiver since the last RTCP packet
MPEG-4 Encoder
Rate
Control
RTP
RTCP (SR)
RTCP (RR)
IF small loss ratio
increase output rate
ELSE
reduce output rate
IF (Ploss <= Pthreshold)
r = min{ ( r+∆), Peak_Rate }
ELSE
r = max{ (α x r), Min_Rate }
COMP9519 Multimedia Systems – Lecture 6 – Slide 36 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Adaptive Rate Control
� Trying to adjust output rate (r) to maintain packet loss ratio (Ploss) below a threshold (Pthreshold)
� Additive increase in rate – conservative increase� If there is no congestion and hence low loss ratio
� Multiplicative decrease in rate – fast adaptation� Swift reduction necessary to shorten congestion period and reduce packet loss.
IF (Ploss <= Pthreshold)
r = min{ ( r+∆), Peak_Rate }
ELSE
r = max{ (α x r), Min_Rate }
COMP9519 Multimedia Systems – Lecture 6 – Slide 37 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Adaptive Rate Control
� Results from [a] for peer-to-peer network.
Source rate and link capacity
Rate (kbps) vs Time (s)
Link utilization and packet loss ratio
Percentage(%) vs Time (s)
COMP9519 Multimedia Systems – Lecture 6 – Slide 38 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Error Resilience� Packet loss causes degradation of perceptual quality at the receiver
� Need to ensure source coding includes mechanisms to be resilient to error
� For MPEG-4 examples of limiting the effects of packet loss include � [a] regular intra frame coding or � [b] intra block refresh schemes.
[a] [b]
� Intra frame stops the propagation of error � Intra block refresh schemes also limit the propagation of error but with a “smoother” bit rate allocation among the frames
I P P P P I P…… I P
…P P
…
COMP9519 Multimedia Systems – Lecture 6 – Slide 39 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Packetization Strategy
� Must meet network MTU limits � Maximum Transmission Unit
� Minimum MTU of all links from sender to receiver
� Minimize number of packets required� Thereby minimizing overhead
� Each packet overhead – IP, UDP and RTP header
� Minimize dependency between consecutive packets� For better error resilience
COMP9519 Multimedia Systems – Lecture 6 – Slide 40 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Packetization Strategy
� Must meet network MTU limits� Example : MTU = 1500 bytes
� Packets larger than MTU will result in IP fragmentation� Resulting in overhead for each fragmented packet
� Loss of one fragment will corrupt the whole packet
� MPEG-4 : Try to allocate a single VOP (frame) per packet� If packet is greater than MTU then break into multiple packets
� Can break any where - arbitrary byte positions
� Best to break at GOB or slice boundaries
COMP9519 Multimedia Systems – Lecture 6 – Slide 41 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Packetization Strategy
� Minimize number of packets required� Less packets means less bits spend on packet headers� Example :
� 20 byte IP Header� 8 byte UDP Header� 12 byte RTP Header
� Minimize dependency between packets� So that even if one packet is lost, the next packet can still be decoded
� Example : � MPEG-4 – use VOP boundaries to start / stop packets
COMP9519 Multimedia Systems – Lecture 6 – Slide 42 –J. Zhang & R. Mathew
System Overview : Sender Side Issues
� Packetization Strategy� Minimize dependency between packets (continued)
VOP 1 VOP 2 VOP 3
loss
VOP 1 VOP 2 VOP 3
VOP 1 VOP 2 VOP 3
COMP9519 Multimedia Systems – Lecture 6 – Slide 43 –J. Zhang & R. Mathew
System Overview : Receiver Side Issues
� Buffering Input Data
Packets
Generated
Packets
Received
Received Data
Played out
Time
Number
of Packets
Constant
Transmit
Rate
Constant
Play Out
Rate
Mean
Receive
Rate
TN TB
TN : Network transfer delay
TB : Buffer Delay
COMP9519 Multimedia Systems – Lecture 6 – Slide 44 –J. Zhang & R. Mathew
System Overview : Receiver Side Issues
� Buffering Input Data
� Jitter can be removed by buffering received packets
� Receiver plays out packets TN + TB (milliseconds) after generation of the packet at the sender
� There exists a tradeoff between delay (due to buffering) and packet loss � Packets received late (> TN + TB) are ignored (i.e. lost)� Live streaming applications have more stringent requirements on delay – so this limits the amount of buffering
COMP9519 Multimedia Systems – Lecture 6 – Slide 45 –J. Zhang & R. Mathew
System Overview : Receiver Side Issues
� Error Concealment
� A streaming service will always be affected by lost packets.
� How to conceal errors due to missing packets ?� So as to minimize loss of quality
� Reduce impact to user perceived quality of image, video or audio service
� Error Resilience� Techniques employed at the receiver to conceal errors
� These techniques may vary from receiver to receiver
COMP9519 Multimedia Systems – Lecture 6 – Slide 46 –J. Zhang & R. Mathew
System Overview : Receiver Side Issues
� Error Concealment
� Example : MPEG-4� One possibility is to replace a missing group of blocks / slice of the current frame with group of blocks / slice from the previous frame.
� Not part of the MPEG-4 Standard� Various techniques proposed and used by decoder developers
Missing
blocks
Copy from
previous framett-1
COMP9519 Multimedia Systems – Lecture 6 – Slide 47 –J. Zhang & R. Mathew
System Overview : Receiver Side Issues
� Inter Media Synchronization
� Need to synchronize multiple media streams
� Synchronize video and audio streams – lip synch
� Synchronize video, audio and text
� RTP time stamps
� Start at a random number � random offset for each stream
� May advance at different rates� Audio and video clock rates could be different
� To synchronize streams at the receiver
� Requires information sent in the RTCP SR (sender reports)
COMP9519 Multimedia Systems – Lecture 6 – Slide 48 –J. Zhang & R. Mathew
System Overview : Receiver Side Issues
� Inter Media Synchronization
Video De-packetizer
and Decoder
Audio De-packetizer
and Decoder
16001760
19202080
2240
SR
1680 : 15:2:32.00
99000 108000 117000
Audio clock = 8000Hz
(eg u-law)
1760 = 32.00 + 0.01
1920 = 32.00 + 0.03
2240 = 32.00 + 0.07
----------------------------
Video clock = 90000Hz
(eg MPEG-4)
108000 = 32.01 + 0.02
117000 = 32.01 + 0.12SR
106200 : 15:2:32.01
COMP9519 Multimedia Systems – Lecture 6 – Slide 49 –J. Zhang & R. Mathew
System Overview : Receiver Side Issues
� Inter Media Synchronization
� RTCP SR� Maps RTP timestamps with the NTP time stamps
� Relates to the instance when the SR is sent
� More information on Network Time Protocol
� Synchronizing audio with Video� Must meet human perception limits
� Example :� Audio can lead video by 40 ms
� Video can lead audio by 120 ms
COMP9519 Multimedia Systems – Lecture 6 – Slide 50 –J. Zhang & R. Mathew
System Overview
IP
UDP
RTP
Video
Decoder
packetize
RTP
UDP
RTCP
RTCP
MPEG-4
Video
RTCP
packetize
RTPRTCP
AMR
Audio
RTP
Audio
Decoder
RTP
Audio
Decoder
RTP
Audio
Decoder
RTP
Audio
Decoder
Synchronized MPEG-4 Video and
AMR Audio Presentation
COMP9519 Multimedia Systems – Lecture 6 – Slide 51 –J. Zhang & R. Mathew
System Overview
� Inter Media Synchronization
� Separate RTP & RTCP streams for video and audio
� Synchronized display at receiver� Both streams part of one session
� Information on AMR Audio� IETF, Request for Comments: 3267
� “Real-Time Transport Protocol (RTP) Payload Format and File Storage Format for the Adaptive Multi-Rate (AMR) and Adaptive Multi-Rate Wideband (AMR-WB) Audio Codecs”
� ftp://ftp.rfc-editor.org/in-notes/rfc3267.txt
COMP9519 Multimedia Systems – Lecture 6 – Slide 52 –J. Zhang & R. Mathew
Client Architecture : An Example
� Reference
� “Client architecture for MPEG-4 streaming”, Haifeng Xu Diamand, J. Luthra, A.
IEEE Multimedia, April-June 2004, Vol 11, Issue: 2,
pages 16-23
� Review of � Streaming Video, RTP, MPEG-4
� Client Architecture for streaming MPEG-4 video� Multilevel buffer system
COMP9519 Multimedia Systems – Lecture 6 – Slide 53 –J. Zhang & R. Mathew
Client Architecture : An Example
“Client architecture for MPEG-4 streaming”, Haifeng Xu Diamand, J. Luthra, A.
IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages 16-23
Multiple buffer architecture
to allow efficient control of media
processing and presentation.
Three Modules
• Network Module : Packet buffer
• Codec Module : Frame Buffer
• Presentation Module : Image Buffer
COMP9519 Multimedia Systems – Lecture 6 – Slide 54 –J. Zhang & R. Mathew
Client Architecture : Multiple Buffers
“Client architecture for MPEG-4 streaming”, Haifeng Xu Diamand, J. Luthra, A.
IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages 16-23
Packet Buffer
Frame Buffer
Image Buffer
COMP9519 Multimedia Systems – Lecture 6 – Slide 55 –J. Zhang & R. Mathew
Client Architecture : Multiple Buffers
� Packet Buffer� Reorder packets� Identify missing packets (allow for easy insertion)� Ignores delayed packets (outside moving window)
� Frame Buffer� Stores video packets (smoothing out network jitter)� Allows dropping of frame (when lacking CPU resources)� Identify frames to be dropped (B, P)
� Image Buffer� Stores decoded frames for rendering � Smoothing decoders speed
COMP9519 Multimedia Systems – Lecture 6 – Slide 56 –J. Zhang & R. Mathew
Client Architecture : Rendering
Rendered frames rates with and without the image buffer
“Client architecture for MPEG-4 streaming”, Haifeng Xu Diamand, J. Luthra, A.
IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages 16-23
COMP9519 Multimedia Systems – Lecture 6 – Slide 57 –J. Zhang & R. Mathew
Client Architecture : Conclusion
� The MPEG-4 MLBA subsystem facilitates three player-related activities:
� precise A/V synchronization,
� client-based QoS management, and
� improved rendering performance through an image buffer.
“Client architecture for MPEG-4 streaming”, Haifeng Xu Diamand, J. Luthra, A.
IEEE Multimedia, April-June 2004, Vol 11, Issue: 2, pages 16-23
COMP9519 Multimedia Systems – Lecture 6 – Slide 58 –J. Zhang & R. Mathew
Introduction to SDP (Control Info)� How to describe content and session info to client
� Session Description Protocol -- SDP
� Example : An existing live multicast session� Video and Audio streams� Transport - RTP/UDP/IP, Control – RTCP/UDP/IP
� A new client wanting to join the multicast session� Needs to know
� multicast IP address and port� Media streams in a session (e.g. video only or audio + video)� Payload format (e.g. MPEG-4 video, AMR audio)� Initialization data for video and audio decoders� Transport protocol used� Other information ….
� Need a way to describe a multimedia session� To enable new clients to easily join the session
COMP9519 Multimedia Systems – Lecture 6 – Slide 59 –J. Zhang & R. Mathew
SDP
� Session Description Protocol (SDP)� IETF – RFC2327
� www.ietf.org/rfc/rfc2327.txt
� For describing multimedia sessions� To communicate the existence of a session
� To convey sufficient information to join a session
� Simple text format
� Defined to be general purpose� Can be used for a wide range of network environments
� And applications
COMP9519 Multimedia Systems – Lecture 6 – Slide 60 –J. Zhang & R. Mathew
SDP
� SDP includes
� Session name and purpose
� Time the session is active
� The media comprising the session
� Information to receive media (addresses, ports, formats)
� Information about bandwidth to be used
� Contact information of a person responsible for the session
� SDP is used by other signaling / initiation protocols� SIP : Session Initiation Protocol
� RTSP : Real-time Streaming Protocol
COMP9519 Multimedia Systems – Lecture 6 – Slide 61 –J. Zhang & R. Mathew
SDP
v= (protocol version)
o= (owner and session identifier)
s= (session name)
i=* (session information)
u=* (URI of description)
e=* (email address)
p=* (phone number)
c=* (connection information)
b=* (bandwidth information)
z=* (time zone adjustments)
k=* (encryption key)
a=* (zero or more session attribute)
t= (time the session is active)
r=* (zero or more repeat times)
m= (media & transport address)
i=* (media title)
c=* (connection information)
b=* (bandwidth information)
k=* (encryption key)
a=* (zero or more media attribute)
session level description
media level description
SDP session description consists of
a number of lines of text of the form
<type>=<value>
<type> is always exactly one
character and is case-significant.
COMP9519 Multimedia Systems – Lecture 6 – Slide 62 –J. Zhang & R. Mathew
SDP
v=0
o=NICTA 2890844526 2890842807 IN IP4 129.94.135.201
s=Camera ONE
i=Video stream for realtime surveillance
u=http://www.nicta.com/mmvc/demos/SurveillanceVideo.pdf
[email protected] (Jian Zhang)
c=IN IP4 225.0.0.37/2
t=0 0
a=recvonly
m=video 20000 RTP/AVP 98
a=rtpmap:98 MP4V-ES/90000
a=fmtp:98 profile-level-id=1; config=000001b001000001…
a=orient:portrait
Version number
Originator information
<username> <session id> <version> <network type> <address type> <address>
(address of machine from which the session was created)Session name
Session Information
(text description / title for session)URL for more information about the session
Contact person e-mail
(person responsible not necessarily the creator)Connection Details
<network type> <address type> <connection address>
IN (internet) IP4 (IP v4) address (multicast address/TTL)Time Information
<start time> <end time>
(ntp time, “0,0” implies permanent session)
Session level attribute
(operate in receive only mode)Media Announcement
<media> <port> <transport> <fmt list>
(fmt : refers to media format specific information)
(e.g. Dynamic payload number)
Media attribute (rtp map)
<payload type> <encoding name>/<clock rate>
(map RTP dynamic payload number to media format and clock rate)Media attribute (format specific parameters)
(info regarding mpeg-4 media, profile, level & initialization data)Media attribute (orientation)
(only used in some applications, example – landscape or portrait)
increased when a modification is made
to the session data.
Recommended that an NTP timestamp
is used
COMP9519 Multimedia Systems – Lecture 6 – Slide 63 –J. Zhang & R. Mathew
System Overview
� Example – Using SDP to join a multicast session� Request SDP file via HTTP
� Retrieve information from downloaded SDP file
� Receive RTP streams on SDP specifed address & port
� Decode and display specified media
HTTP GET
SDP file
http://www.nicta.com/lecture.sdp
RTP Media Stream
RTCPMedia Server
Client
COMP9519 Multimedia Systems – Lecture 6 – Slide 64 –J. Zhang & R. Mathew
System Overview
� Example – Traffic Monitoring� Continuous multicast streaming of video
� A client can receive the stream � by downloading SDP via http (web browser)
� Provide SDP file to QuickTime player
� Player initializes and waits for stream data
� Easily support multiple client
COMP9519 Multimedia Systems – Lecture 6 – Slide 65 –J. Zhang & R. Mathew
References and Further Reading
� RTP / RTCP : IETF, RFC 3350 RTP: A Transport Protocol for Real-Time Applications
� Streaming Video over the Internet: Approaches and Directions, Dapeng Wu, Yiwei Thomas Hou, Wenwu Zhu, Ya-Qin Zhang, and Jon M. Peha, IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, VOL. 11, NO. 3, MARCH 2001
� On End-to-End Transport Architecture for MPEG-4 Video
Streaming over the Internet , D Wu et al, IEEE TRANSACTIONS ON CIRCUITS AND SYSTEMS FOR VIDEO TECHNOLOGY, Vol. 10, No. 6, September 2000