mpeg for the past, present and future of television
DESCRIPTION
MPEG for the past, present and future of television. Leonardo Chiariglione Keynote Speech at ATSC Annual Meeting Washington, DC – 2012/05/08. MPEG for the past of television. System Transport Stream DSM-CC Timed metadata Video MPEG-2 High Profile Audio MPEG-1 Audio Layer II - PowerPoint PPT PresentationTRANSCRIPT
MPEG for the past, present and
future of television
Leonardo ChiariglioneKeynote Speech at ATSC Annual
MeetingWashington, DC – 2012/05/08
MPEG for the past of television
SystemTransport StreamDSM-CCTimed metadata
VideoMPEG-2 High Profile
AudioMPEG-1 Audio Layer IIMPEG-2 AAC
2012/05/02 2MPEG for the past, present and future of TV
An assessmentMPEG has provided the means for the television
business on air, cable and satellite to migrate from the analogue to the digital age
It has been a collective effortsRepresentatives from the entire spectrum of the
broadcasting world provided their requirementsPackages of patent licences were created (outside of
MPEG)Today there is virtually no broadcasting system that
is not based, at least partially, on MPEG standards
MPEG for the past, present and future of TV 32012/05/08
MPEG for the present of television
SystemsCarriage of MPEG-4 on MPEG-2 TSDynamic Adaptive Streaming over HTTP (DASH)
VideoMPEG-4 Advanced Video Coding (AVC)3D extensions
AudioVarious forms of Advanced Audio Coding (AAC)Universal Speech and Audio (USAC)
2012/05/08 4MPEG for the past, present and future of TV
An assessmentThe deployment of 2nd generation digital
broadcasting systems continues successfullyUse of the AVC on MPEG-2 TS allows packing twice
as many programs as MPEG-2 Video in an analogue channel
With IPTV the notion of broadcasting is expandingMPEG is helping the broadcasting industry
To enrich the broadcasted user experience Through standard technologies that serve multiple
fields
MPEG for the past, present and future of TV 52012/05/08
MPEG for the future of television
Well, that’s what I am supposed to talk about…
2012/05/08 6MPEG for the past, present and future of TV
Broadcasting is a great business to be in…
Broadcasting is a proven business with a constant revenue flow
Many technologies appear thatCan help new competing businesses but Can also be used to extend the broadcasting
businessMPEG is a neutral body devoted to the
development of standard technologies for user communities to exploit
MPEG for the past, present and future of TV 72012/05/08
MPEG-H: a video standard for
future broadcastingISO/IEC 23008 High Efficiency Coding and Media
Delivery in Heterogeneous EnvironmentsA suite of standards for media coding and
deliveryCurrently scheduled to appear in 3 parts
MPEG Media TransportHigh Efficiency Video Coding3D Audio
Timeline
MPEG for the past, present and future of TV 82012/05/08
Part CD DIS FDISMMT 12/07 13/01 13/07HEVC 12/02 12/07 13/01
3D Audio
14/04 14/10 15/01
Service composition in MMT
3D Earth
UD contents
Bird is flying…
Bird is flying…
<subtitle> composition information friendly with html5
legacy broadcasting future broadcasting
MMT Package
9MPEG for the past, present and future of TV 2012/05/02
CloudsMMT Encapsulator
- Service Component List- Component location- Composition relationship- QoE requirement- How to deliver- How to consume
Webvideo
file
- Configuration- Adaptive delivery- Sync, Delay, Loss control
MMT Frame
pull pull
Smart delivery in Clouds
102012/05/08
Mash-up service
Service 1 (MMT Package1) Service 2 (MMT Package2)
MMT assets
Audio Video1 widget Web pageVideo2
112012/05/08MPEG for the past, present and future of TV
Second screen management
Additional Information with Live Content
(HTML)
Video content on 2nd device
composition information <device 1>…<device2>…
MMT Package
122012/05/08MPEG for the past, present and future of TV
MPEG-H part 2 High Efficiency Video
CodingPrimary goal: ≥ 50% better compression than AVC
For large displays (e.g. 4kx2k)For mobile (e.g. WVGA)
Developed jointly with ITU-T by Joint Collaborative Team (JCT) on Video Coding
Technically in the same stream of past MPEG Video Coding standards (AVC and prior standards)
Currently at Committee Draft level
MPEG for the past, present and future of TV 132012/05/08
HEVC vs AVC Subjective Performance
Measurements Preliminary subjective tests carried out
for HEVC Committee Draft Reasonably small and non-overlapping
confidence intervals Average rate savings at same
(interpolated) MOS points HEVC with 67% lower rate in class B (Full HD) HEVC with 49% lower rate in class C (WVGA) HEVC with 58% lower rate overall
2012/05/08MPEG for the past, present and future of TV 14
A performance sample
2012/05/08MPEG for the past, present and future of TV 15
For those technology aware…
Block-based/Variable block sizesBlock motion compensationFractional-pel motion vectorsSpatial intra predictionSpatial transform of residual differenceInteger-based transform designsArithmetic or VLC-based entropy codingIn-loop filtering to form final decoded picture
2012/05/08MPEG for the past, present and future of TV 16
Where we are (May 2012)Activity: level of 1,000 documents per meeting crossed Very diverse participation from industry/academiaSignificant technical advance over prior standardsChallenge: computational/implementation complexityDeliverables:
SpecReference softwareConformance
Currently only one profile
2012/05/08MPEG for the past, present and future of TV 17
HEVC Scalability Extensions/1
Serve the needs of heterogeneous environment of future digital TV distributionMultiple devices: SHDTVs, set-top boxes, tablets,
smart phones, PCs etc.Wide range of processing powers, display sizes and
power consumption needsMultiple networks: Cable, Satellite, Terrestrial, CDN,
Cable Modem, xDSL, WiFi, 4GWide range of channel bandwidths and QoS
Multiple protocols: MPEG-2 TS, DASH, HLS, RTP, UDPMultiple services: Broadcasting, On demand,
Streaming, Over The Top
2012/05/08MPEG for the past, present and future of TV 18
HEVC Scalability Extensions/2
Options for backward compatible migration pathsFrom 720p60 to 1080p60From1080p60 to 4kx2kx60From AVC to HEVC
Time LineMay 2012 Preliminary Call for Proposals (public) July 2012 Final Call for Proposals (public)October 2012
Evaluation of proposals & start of collaborative design phaseDevelopment of the time line associated with standardisation
phases (WD, CD/PDAM, FDIS/FDAM, DIS/AMD)
2012/05/08MPEG for the past, present and future of TV 19
Service Transitions1950s: Color TV
Analog, backward compatible1990s: Digital TV
New infrastructure requiredTransitions from SD to HD
2010s: 3DMixed results from services introduction isNot a single format across all services
2012/05/08MPEG for the past, present and future of TV 20
Current Status of 3D VideoIncreasing investments in 3D production and
servicesIncreasingly capable 3D displays in the market
(many competing and emerging technologies)Market is primarily stereo
(mix of different formats being deployed right now)Focus of current MPEG work: development of a new
3D format and associated compression techniques that could facilitate generation of multiview output
2012/05/08MPEG for the past, present and future of TV 21
Existing 3DV Coding Formats
Pack pixels from left and right views into a single frame (loss of resolution)
Leverage existing infrastructure and equipment, only require additional signaling to de-interleave frame
Embraced by broadcasters for first phase of 3D broadcast services
Full-resolution coding of stereo and MV video as extension of AVC
High coding efficiency achieved via inter-view prediction techniques
2D compatibility supported Adopted as format for 3D Blu-ray
Disc Being considered for second phase
of broadcast standards
2012/05/08MPEG for the past, present and future of TV 22
Left Right
Left View Right View
Effective prediction between views for high compression performance
Left View Right View
Effective prediction between views for high compression performance
Targets of Future 3DV Format
2012/05/08MPEG for the past, present and future of TV 23
Data Format
Data Format
Constrained Rate(based on distribution)
LimitedCamera Inputs
• Wide viewing angle• Large number of
output views
Left
Right
Auto-stereoscopicN-view displays
Stereoscopic displays• Variable stereo baseline• Adjust depth perception
3DV Reference FrameworkLimited # Video
Inputs(e.g., 2 or 3 views)
DepthEstimation
Video/DepthEncoder
Decoder and View Synthesis
Larger #Output Views
Binary Representation& Reconstruction
Process
+
1010001010001
242012/05/08MPEG for the past, present and future of TV
Overview of Call For Proposals
Test Material & Conditions8 test sequences (1920x1080 & 1024x768)4 target bit rates per sequence2-view and 3-view test scenariosAVC and HEVC compatible test categories
23 responses (12 AVC + 11 HEVC)Evaluation in Nov 2011
Objective quality (PSNR of synthesized views)Subjective assessment (stereo and auto-stereo)
2012/05/08MPEG for the past, present and future of TV 25
3DV Testing Scenarios
2012/05/08MPEG for the past, present and future of TV 26
Left
Right
Center
Depth Estimation
3D Video Codec
Multiview Rendering
virtual views
Output toAuto-Stereo
N-view Display
Left
Right
Center
Depth Estimation
3D Video Codec
Multiview Rendering
virtual views
Output toAuto-Stereo
N-view Display
Left
Right
Depth Estimation
3D Video Codec
Multiview Rendering
virtual view
Output toStereo Display
Left
Right
Depth Estimation
3D Video Codec
Multiview Rendering
virtual view
Output toStereo Display
2-view Configuration
3-view Configuration
Tool CategoriesTexture coding
Independent of depth E.g., inter-view prediction of color view, inter-view prediction
of motion parameters and residual dataUsing depth data
E.g., view synthesis prediction, motion prediction
Depth coding Independent of texture
E.g., depth modeling modes, weighted prediction, reduced resolution coding
Using texture dataE.g., motion parameter inheritance, intra prediction
2012/05/08MPEG for the past, present and future of TV 27
3DV Current Status Extensive subjective evaluation
of 20+ proposals to the 3DV CfP Considered both AVC and HEVC
compatible
Many new coding tools proposed for MV texture and depth coding Substantial rate savings
compared to capabilities of existing standards
AVC and HEVC extensions standardisation under wayd Add support for depth New AVC-based coding tools MV extensions of HEVC Hybrid architectures
2012/05/08MPEG for the past, present and future of TV 28
Hybrid ArchitecturesFrom a pure compression efficiency point of view, it
is always best to use the most advanced codecHowever, when introducing new services, providers
must also consider capabilities of existing receivers and transition plan
Use Case: Many terrestrial broadcast systems based on MPEG-2US cable systems based on mix of MPEG-2 and AVCMay not be easy to simply switch codecs in near term
2012/05/08MPEG for the past, present and future of TV 29
Multiview Extensions of HEVC
HEVC Stereo/MV extensions being consideredExtensions expected to be completed
approximately 12 months after the base specification, i.e., Jan 2014
Reportedly simple extensions of HEVC can achieve 30-40% bit rate reduction compared to HEVC simulcast
Further gains expected if block-level changes to codec are considered
2012/05/08MPEG for the past, present and future of TV 30
MPEG-H part 3: 3D AudioWell, there is no Systems and Video without
Audio…
2012/05/08MPEG for the past, present and future of TV 31
3D Audio requirements/1High quality: quality of decoded sound perceptually
transparentLocalization and Envelopment: Accurate sound
localization and very high sense of sound envelopment within a targeted listening area (“sweet spot”)
Flexible Loudspeaker Placement: the transmitted audio program to be mapped to a rendering setup in which loudspeakers are in alternate locations
Interactivity: Interactive modification of the sound scene rendered from the coded representation
MPEG for the past, present and future of TV 322012/05/08
3D Audio requirements/2Rendering on setups with fewer
loudspeakers (including headphones): ability to derive a signal from the transmitted program material for reproduction with fewer loudspeakers
Audio/visual alignment & consistency: ability to adapt the rendered acoustic scene to be consistent with the visual
Efficiency for decoding on different setups: ability to be rendered on all reproduction loudspeaker setups or headphone
MPEG for the past, present and future of TV 332012/05/08
3D Audio requirements/3Appropriate computational complexity and
system latency for the target application (Broadcasting, Spatial two-(n-)way communication / telepresence
Transcoding for low bandwidth devices: A lower bandwidth signal can be extracted from the original program material
Backward compatibility: (e.g. to 5.1 channel programs and decoders or transmission of HRTF encoded signals)
MPEG for the past, present and future of TV 342012/05/08
Envisioned Architecture
Inputs Audio Channels
“normal” content Audio Objects
“helicopter overflight” Audio Scene
Ambisonics
35
3D Audio Encoder
500 kb/sbitstream
3D Audio Encoder
Transcoder 64 kb/sbitstream
22.2 Channel Audio Program
Audio Objects
Audio Scene
SmartPhone TV
Home Theatre
Tablet TV
22.2 Loudspeakers
Spatialized on Headphones
In-Frame Speakers
Spatialized on Headphones
Spatialized on Headphones
• Outputs– Loudspeakers– Headphones
• Spatialized sound– In-frame
loudspeakers• “Sound Bar”
• Channel– 500 kb/s primary
channel– 64 kb/s cellular
channel
MPEG for the past, present and future of TV 2012/05/08
Audio, part of an Audio-visual
ExperienceWhat is new in video?
Larger, high-resolution displaysGreater sense of envelopmentPossibly closer viewing distance
More efficient compressionWireless transmission to portable devices
Immersive and enveloping audio experience
362012/05/08MPEG for the past, present and future of TV
Use Case 1Home Theatre
Many loudspeakers10.122.2
“3D” ExperienceHeight loudspeakers for greater envelopmentSense that audio objects are near or distantWhen near, they are consistent with video image
372012/05/08MPEG for the past, present and future of TV
22.2 multichannel sound system (NHK)
2012/05/08MPEG for the past, present and future of TV 38
TV Screen
Top layer9 channels
Middle layer10 channels
Bottom layer3 channels
LFE2 channels
TpFL TpFC TpFR
TpSiLTpC
TpSiR
TpBLTpBC
TpBR
BLBC
BR
SiL
FLFLc FC FRc
FR
SiR
BtFLBtFC
BtFRLFE1 LFE2
TV Screen
Top layer9 channels
Middle layer10 channels
Bottom layer3 channels
LFE2 channels
TpFL TpFC TpFR
TpSiLTpC
TpSiR
TpBLTpBC
TpBR
BLBC
BR
SiL
FLFLc FC FRc
FR
SiR
BtFLBtFC
BtFRLFE1 LFE2
Home Theater – IssuesMaintain compatibility with legacy systems: 5.1,
stereo Will content providers really step up to 22.2?Will consumers adopt 22.2?: 2 front highHow to compress 22.2 for transmission?
MPEG AAC 64 kb/s/channel: -> 1.5 mb/sHow to render for legacy setups?Relatively slow-growth market
2012/05/08MPEG for the past, present and future of TV 39
Use Case 2Personal TV
Small but high-resolution display“super-tablet”
Local wireless communicationsTo cable or fiber home hub
Possible audio presentationHeadphonesLoudspeakers around perimeter of display
Issue: To what extent can the user get an enveloping experience from only “front” loudspeakers?
40MPEG for the past, present and future of TV 2012/05/08
Use Case 3Mobile TV
Hand-held display: SmartphoneHeadphone listening:
Stereo, perhaps with binauralisationFastest growing market: Quick rollout and adoption
of standardWhat is new?
USAC for stereo; MPEG Surround for binauralisationWhat is the “Wow” factor?
To spur adoption of new technology
41MPEG for the past, present and future of TV 2012/05/08
User Experience
42
Home Theatre
Tablet TV
Spatialized on headphones
2012/05/08MPEG for the past, present and future of TV
Flexible RenderingContent providers create 22.2 programConsumed on stereo, 5.1, 10.1, 22.2 layoutsConsumed on “wrong” layouts
Mis-positioned surroundsMissing surroundsNon-standard layouts
2 front high, 5.1 mid
43MPEG for the past, present and future of TV 2012/05/08
3D Audio Test PlatformNHK “Loudspeaker
Array Frame” (LAF)Suggestions for
Alternative platform for assessing “Home Theatre” use case
Model for assessing “Personal TV” use case
44MPEG for the past, present and future of TV 2012/05/02
SummaryProvide compelling immersive audio experience
for audio/visual programsCreate content once, present on many different
loudspeaker layouts or on headphonesProvide high compression and high-quality audio
presentation
45MPEG for the past, present and future of TV 2012/05/08
MPEG-V: Humans have more than two senses
2012/05/08MPEG for the past, present and future of TV 46
MARBle
MediaServers
ServiceServers
User
LocalSensors
& Actuators
RemoteSensors & Actuators
ARAF file or
stream
Local Real World
Environment
Remote Real World
Environment
Augmented Reality Application Format (ARAF)
MPEG for the past, present and future of TV 2012/05/08 47
Augmentation Region in ARAF
Augmentation Region
Broadcaster
AR service provider A
AR service provider B
User A
User B
MPEG for the past, present and future of TV 2012/05/02 48
The past, the present and the future, again
25 years ago MPEG selected the digital media way bringing innovation and interoperability
…and thrivedThis year the broadcasting industry has started
a move in future broadcast services with interoperability at its core
…and it is bound to thriveMPEG looks forward to continuing a cooperation
with the broadcasting industry providing the necessary standard technologies
2012/05/08MPEG for the past, present and future of TV 49
502012/05/08MPEG for the past, present and future of TV
http://mpeg.chiariglione.org/