![Page 1: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/1.jpg)
Video Coding for Streaming Media Delivery on the InternetConklin, G. et al, 2000
Jake Wires
![Page 2: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/2.jpg)
Motivation
Original standards designed for constant bit rate encoding
Constant bit rate encoding does not account for the channel variability inherent in many network applications
Large demand for streaming multimedia—over 350,000 hours of live content broadcast per week
![Page 3: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/3.jpg)
History of Streaming Media Technologies
VivoActive 1.0HTTP server-client transactions via TCPH.263 video codec, G.723 audio codecMultiplexing protocol to combine audio &
videoVery simple
![Page 4: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/4.jpg)
History: VivoActive
ProblemsHTTP’s lack of flow control and TCP’s
guaranteed delivery mechanisms resulted in large fluctuations of fragment delivery times
Necessitated large preroll buffer (5 – 20 sec)Rebuffering (filling empty buffer) diminished user
expereinceSession Control (i.e. f.f., rew.) difficult
![Page 5: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/5.jpg)
History: RealAudio 1.0
Introduced in March 1995Relied on Progressive Networks
Architecture—a suite of dedicated network protocolsUsed 2 connections:
TCP-based session control connectionUDP-based audio transmission connection
![Page 6: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/6.jpg)
History: RealAudio 1.0
Use of UDP-based transmission introduced possibility for lost, delayed, and out of order packets Automatic Repeat-Request mechanism allowed
clients to request retransmission of lost packets Frame interleaving technique minimized perceptual
damage Send packets of interleaved (rather than continuous)
audio frames Dropped packet results in multiple small disruptions rather
than single large disruption
![Page 7: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/7.jpg)
History: RealVideo
RealVideo design features: Produced compressed data that can be streamed
at a fixed bit rate Non-uniform nature of video clips (scene changes,
transitions, etc.) results in unequal distribution of bits between frames when encoded at same level of distortion
To maintain fixed bit rate, encoder must introduce unequal distortion between frames, neglect encoding some frames, or both
Use of preroll buffer increases bit rate flexibility—only average rate must be kept constant
Variable-bit-rate (VBR) rate-control algorithm
![Page 8: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/8.jpg)
History: RealVideo
Facilitated random access by inserting Intra frames
Employed unequal error protectionLarge video frame size precluded simple
interleaving techniques Instead, RealVideo combined forward error
correction codes for the most sensitive data with various error concealment mechanisms
![Page 9: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/9.jpg)
History: RealSystem G2
SureStream technology improved adaptive streaming capabilities: A single SureStream file contains multiple streams
of the original content Different streams optimized for different channel
capacities Client of stream monitors connection performance,
dynamically requests optimal stream from server
Used IETF standardized RTP and RTSP protocols
![Page 10: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/10.jpg)
History: Network Strategies
Multicasting reduces amount of redundant data transferred
Streaming media delivery networks allow for multiple, geographically scattered servers (achieved either by splitting or caching)
Dedicated networks guarantee bandwidth Multiple-access transmissions over Internet—
requested info sent from several different locations simultaneously
![Page 11: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/11.jpg)
Streaming Media
Three modes in which video information can be streamed:Live broadcastingOn-demand streamingConferencing
![Page 12: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/12.jpg)
Streaming Media: Live Broadcasting System Structure
![Page 13: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/13.jpg)
Streaming Media: Live Broadcasting
Live video encoded on-the-fly, passed to server, which disperses compressed info to all splitters and/or clients Splitters are additional servers that can be used to
distribute server workload over a number of machines
Push splitting: initiated by server Pull splitting: initiated by a client connecting to a local
splitter; request propagated to nearest active splitter Multiple-access splitting: lower-tier splitters can connect to
multiple upper-tier splitters
![Page 14: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/14.jpg)
Streaming Media: On-demand System Structure
![Page 15: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/15.jpg)
Streaming Media: On-demand
Distribution system similar to live broadcasting system, except:No direct connection between encoder and
serverRemote proxy servers can cache popular
filesClients allowed to rewind, fast forward, etc.Server proxy transfers only initiated by client
![Page 16: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/16.jpg)
Video Coding Problems
Channel capacity fluctuations Adaptive client-driven serving
Allow client to inform server of network performance Congestion control Error control: use bandwidth prescribed by congestion
control algorithm to minimize effects of packet loss
Dynamic prediction Combined with sufficient preroll buffer, a broad set of
statistical techniques can (possibly) be used to throttle transfer rate to improve user experience
![Page 17: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/17.jpg)
Video Coding Problems
Random accessCan provide random access by inserting
I-frames at the boundaries of small video intervals
Reduces achievable compression ratesProcessing resources
Diverse range of client machinesLive encoding often done on computer with
limited resources
![Page 18: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/18.jpg)
Video Coding Algorithms for Streaming
Original motion-compensated hybrid video coding algorithms produced single-rate encoded versions of input signals Inadequate for many network applications
Need scalable coding algorithms
![Page 19: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/19.jpg)
Scalable Streaming Techniques
Simulcast: produce several, redundant, independently coded streams, each optimized for specific channel conditions Not suitable for splitting, where the combined
representation of all streams should be as small as possible
Scalable coding Based on spatio-temporal pyramid decomposition
of source signal Temporal, spatial, and Signal to Noise Ratio (quantizer
size) scalability modes
![Page 20: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/20.jpg)
Scalable Streaming Techniques
Multiple-description coding for use in multiple-access systemsAchieved by replacing scalar quantization
with multiple-description quantization scheme
![Page 21: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/21.jpg)
RealSystem 8
Adaptive channel encoding of streaming content to match bit rate with bandwidth and loss statisticsChannel statistics not available until time of
streaming, so server must manage adaptive strategies
Infeasible for server to encode on demand, so encoding techniques are required which allow server to efficiently choose between different bit rates
![Page 22: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/22.jpg)
RealSystem 8
SureStream components:Adaptive Stream Management (ASM)SureStream file formatSource and channel coding algorithms
![Page 23: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/23.jpg)
RealSystem 8
ASM Allows clients to communicate which type of
encoding should be ‘synthesized’ by server File format plug-in uses ASM rules to produce
various combinations of encoded streams from compressed media files
Rules expressed by fully programmable syntax Client subscribes to appropriate rule based on
channel characteristics ASM is format-independent
![Page 24: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/24.jpg)
RealSystem 8
RealVideo 8 encoding module:
![Page 25: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/25.jpg)
RealVideo 8 Algorithm
Input filters: remove noise and potential artifacts introduced by edits and conversions of video signal Remove low-energy spatial noise De-interlace analog signals to produce progressive
frames without “jaggy”-shaped artifacts Inverse telecine filter: remove redundant 5.97 frames
introduced by NTSC video
![Page 26: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/26.jpg)
RealVideo 8 Algorithm
Spatial resampler: downscale input frames to spatial resolutions suitable for streaming
Codecs: motion-compensated hybrid scheme Rate control determines which frames to
encode Low bit rates: perceived quality impacted by
relationship between framerate and frame quality High bit rates: must maintain framerate, want to
maintain appearance of uniform high quality Parameterized preroll buffer size Rate control important in providing resiliency and
random access
![Page 27: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/27.jpg)
RealVideo 8 Algorithm
CPU scalability control module: throttles complexity of different components of algorithm to maintain best quality level for encoding live presentations
![Page 28: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/28.jpg)
RealVideo 8 Algorithm
Client-side video postprocessing: filters aimed at improving subjective quality of decoded signals Frame rate upsampler: special temporal filter that
attempts to interpolate intermediate frames Especially useful when many original frames were
regularly skipped to decrease bit rate of encoding
![Page 29: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/29.jpg)
Pointers
Helix Community “The Helix community is a collaborative
effort among Real, independent developers, and leading companies to extend the Helix DNA™ platform, the first open multi-format platform for digital media creation, delivery and playback.”
https://helixcommunity.org/
![Page 30: Video Coding for Streaming Media Delivery on the Internet Conklin, G. et al, 2000](https://reader034.vdocuments.net/reader034/viewer/2022051821/56814d20550346895dba5548/html5/thumbnails/30.jpg)
Conclusion
Streaming multimedia poses many challenges to video encoders Variability of channel performance High server workloads
Need to adapt coded bit rate to match channel capacities; must be done efficiently and on the fly
“RealSystem 8 provides an open and extensible platform, capable of accommodating various future needs of streaming media infrastructure on the Internet, and in particular, new demands for improved video-coding techniques.”