the mpeg-4 part 10 (iso/iec 14496-10) | itu-t h.264 ... · mobile video players (sony psp, apple...

The MPEG-4 Part 10

(ISO/IEC 14496-10) | ITU-T H.264

Advanced Video Coding (AVC)

Standard and its Extensions

Gary J. Sullivan

Chair/co-chair: JVT, JCT-VC, VCEG, MPEG Video

Video Architect: Microsoft Windows Core Media Processing

[email protected]

IEEE Signal Processing Society

Denver Section Lecture22 February 2010Denver, CO, USA

Outline

• Background, organizations, and concepts

• H.264/MPEG-4 Advanced Video Coding (AVC)

• Scalable Video Coding (SVC)

• Multi-view Video Coding (MVC) and Stereo Video

• Next-generation video coding work

Gary SullivanThe MPEG-4 Part 10 | H.264 Advanced Video Coding (AVC) Standard and its Extensions

IEEE Signal Processing Society22 February 2010

Denver, CO, USA

1990 1996 20021992 1994 1998 2000

H.263(1995-2000+)

MPEG-4

Visual(1998-2001+)

MPEG-1 (1993)

ISO

/IE

CIT

U-T

H.120(1984-1988)

H.261(1990+)

H.262 /

MPEG-2 (1994/95-1998+)

H.264 /

MPEG-4 AVC

(2003-2009)

Chronology of International

Video Coding Standards

2004



Denver, CO, USA

Video Coding Standards Organizations

ISO/IEC MPEG = “Moving Picture Experts Group”(ISO/IEC JTC 1/SC 29/WG 11 = International Standardization Organization and International Electrotechnical Commission, Joint Technical Committee Number 1, Subcommittee 29, Working Group 11)

ITU-T VCEG = “Video Coding Experts Group”(ITU-T SG16/Q6 = International Telecommunications Union –Telecommunications Standardization Sector (ITU-T,a United Nations Organization, formerly CCITT),Study Group 16, Working Party 3, Question 6)

JVT = “Joint Video Team” collaborative team of MPEG & VCEG

Recently, SMPTE (Society for Motion Picture and Television Engineers) also standardized “VC-1”, based on Microsoft‟s WMV 9 (but this talk does not focus on SMPTE or Microsoft).

New: JCT-VC = “Joint Collaborative Team on Video Coding” team of MPEG & VCEG, continuing the collaborative relationship for a new project (established January 2010)



Denver, CO, USA

The Scope of Picture and Video

Coding Standardization

Only the Syntax and Decoder are standardized:

• Permits optimization beyond the obvious

• Permits complexity reduction for implementability

• Provides no guarantees of Quality

Pre-Processing EncodingSource

DestinationPost-Processing

& Error Recovery

Decoding

Scope of Standard



Denver, CO, USA

A Basic Building Block of Video Coding:

Interframe Motion Prediction

Large areas of images stay the same from frame to frame, changing

mostly due to motion

Conditional Replenishment: Can signal to leave a block area of the

image unchanged, or replace it with new data

Interframe Difference Coding: Could encode a refinement to the

value of an area

Displaced Frame Difference Coding: Can predict an image area by

copying some nearby part of the previous image (motion

compensation) and optionally adding some refinement



Denver, CO, USA

A Basic Building Block of Video Coding:

The Discrete Cosine Transform (DCT)

Picture coding using the Discrete Cosine Transform (DCT) (early „70s, ITU+ISO JPEG approved „92):

• Analyze 8x8 blocks of image according to DCT

frequency content (images tend to be smooth) (‟74)

• Round off (“quantize”) the measured frequency

content to scaled integer values („50s, „60s, ...)

• Scan through the coefficients in a zig-zag pattern to

order them in the bitstream (‟70s - Tescher)

• Send integer approximations to decoder using

“Huffman” variable-length codes (VLC, early „50s)



Denver, CO, USA

H.264 / 14496-10 AVC Structure

Entropy

Coding

Scaling & Inv.

Transform

Motion-

Compensation

Control

Data

Quant.

Transf. coeffs

Motion

Data

Intra/Inter

Coder

Control

Decoder

Motion

Estimation

Transform/

Scal./Quant.-

Input

VideoSignal

Split intoMacroblocks

16x16 pixels

Intra-frame

Prediction

Deblocking

Filter

Output

VideoSignal



Denver, CO, USA

H.264 / 14496-10 Design Objectives

• Primary technical objectives:– Significant improvement in coding efficiency

– High loss/error robustness

– “Network Friendliness” (carry it well on MPEG-2 or RTP or H.32x or in MPEG-4 file format or MPEG-4 systems or …)

– Low latency capability (better quality for higher latency)

– Exact match decoding

• Initial extension objectives:– Professional applications (more than 8 bits per sample,

4:4:4 color sampling, etc.)

– Higher-quality high-resolution video

– Alpha plane support (a degree of “object” functionality)

– Extended color gamut support



Denver, CO, USA

H.264 / AVC Key Technical Advances

• Prediction: Quarter-sample motion precision

• Prediction: 6-tap motion interpolation filtering

• Prediction: Multi-frame motion compensation referencing

• Prediction: Inter-picture prediction partitioning: 16x16, 16x8, 8x16, 8x8, 8x4, 4x8, 4x4

• Prediction: Intra-picture spatial prediction 16x16, 8x8, 4x4

• Transform: 4x4 and 8x8 exact-match low-complexity integer ops

• Quantization: Logarithmic step size control, 16 bit friendly

• Entropy coding: Exp-Golomb, CABAC, CAVLC

• Deblocking filtering

• Syntax structuring for robustness and flexibility

– Parameter sets & slices

– Coded picture buffer management (HRD / VBV)

– Decoded picture buffer management

– Start codes and emulation prevention

– Supplemental enhancement information



Denver, CO, USA

Example Compression Comparison

0 100 200 300

28

30

32

34

36

38

40

Rate [kbit/s]

PSNR

[dB] Half-pel

motion compensation(MPEG-1 1993 MPEG-2 1994)

Integer-pel

motioncompensation(H.261, 1991)

Variable block size

(16x16 – 8x8)(H.263, 1996) +

quarter-pel

motion compensation(MPEG-4, 1998)

Variable block size

(16x16 – 4x4) +quarter-pel +multi-frame

motion compensation(H.264/AVC, 2003)

Foreman

10 Hz, QCIF

100 frames

Good

Picture

Quality

Bad

Picture

Quality



Denver, CO, USA

H.264 / 14496-10 AVC Structure

Entropy

Coding

Scaling & Inv.

Transform

Motion-

Compensation

Control

Data

Quant.

Transf. coeffs

Motion

Data

Intra/Inter

Coder

Control

Decoder

Motion

Estimation

Transform/

Scal./Quant.-

Input

VideoSignal


16x16 pixels

Intra-frame

Prediction

Deblocking

Filter

Output

VideoSignal



Denver, CO, USA

Entropy

Coding

Scaling & Inv.

Transform

Motion-

Compensation

Control

Data

Quant.

Transf. coeffs

Motion

Data

Intra/Inter

Coder

Control

Decoder

Motion

Estimation

Transform/

Scal./Quant.-

Input

VideoSignal


16x16 pixels

Intra-frame

Prediction

De-blocking

Filter

Output

VideoSignal

Motion Compensation Accuracy

Motion vector accuracy 1/4 sample

(6-tap filter)

8x8

0

4x8

0 10 1

2 3

4x48x4

1

08x8

Types

0

16x16

0 1

8x16

MBTypes

8x8

0 1

2 3

16x8

1

0



Denver, CO, USA

Entropy

Coding

Scaling & Inv.

Transform

Motion-

Compensation

Control

Data

Quant.

Transf. coeffs

Motion

Data

Intra/Inter

Coder

Control

Decoder

Motion

Estimation

Transform/

Scal./Quant.-

Input

VideoSignal


16x16 pixels

Intra-frame

Prediction

De-blocking

Filter

Output

VideoSignal

Motion

Data

Output

VideoSignal

Multiple Reference Frames

Multiple Reference Frames

Generalized B Frames



Denver, CO, USA

Entropy

Coding

Scaling & Inv.

Transform

Motion-

Compensation

Control

Data

Quant.

Transf. coeffs

Motion

Data

Intra/Inter

Coder

Control

Decoder

Motion

Estimation

Transform/

Scal./Quant.-

Input

VideoSignal


16x16 pixels

Intra-frame

Prediction

De-blocking

Filter

Output

VideoSignal

Intra Prediction Directional spatial prediction

(9 types for 4x4 luma pred,

4 types for 16x16 luma pred,

4 types for 8x8 chroma pred)

e.g., Mode 4: diagonal down/right prediction

a, f, k, p are predicted by

(A + 2M + I + 2) >> 2

M A B C D E F G H

I a b c d

J e f g h

K i j k l

L m n o p

1

8

3 4

5

6

70



Denver, CO, USA

Entropy

Coding

Scaling & Inv.

Transform

Motion-

Compensation

Control

Data

Quant.

Transf. coeffs

Motion

Data

Intra/Inter

Coder

Control

Decoder

Motion

Estimation

Transform/

Scal./Quant.-

Input

VideoSignal


16x16 pixels

Intra-frame

Prediction

De-blocking

Filter

Output

VideoSignal

DCT-Like Transform Coding

4x4 Block Integer Transform

Hierarchical transform of DC coeffs

for 8x8 chroma and 16x16 Intra

luma blocks

1 1 1 1

2 1 1 2

1 1 1 1

1 2 2 1

H



Denver, CO, USA

Deblocking Filter

Entropy

Coding

Scaling & Inv.

Transform

Motion-

Compensation

Control

Data

Quant.

Transf. coeffs

Motion

Data

Intra/Inter

Coder

Control

Decoder

Motion

Estimation

Transform/

Scal./Quant.-

Input

VideoSignal


16x16 pixels

Intra-frame

Prediction

Deblocking

Filter

Output

VideoSignal



Denver, CO, USA

Entropy Coding

Entropy

Coding

Scaling & Inv.

Transform

Motion-

Compensation

Control

Data

Quant.

Transf. coeffs

Motion

Data

Intra/Inter

Coder

Control

Decoder

Motion

Estimation

Transform/

Scal./Quant.-

Input

VideoSignal


16x16 pixels

Intra-frame

Prediction

Deblocking

Filter

Output

VideoSignal

•UVLC/Exp-Golomb +

•CABAC or

•CAVLC



Denver, CO, USA

H.264 / AVC Adoption & Deployment

Video conferencing (Polycom, Tandberg, Sony, Layered Media)

HDTV satellite (DirecTV, BSkyB, Echostar, …)

Terrestrial TV (France, Estonia, Norway, Brazil, …)

IPTV (AT&T, France Telecom, British Telecom, Deutsche Telekom, KPN, Belgacom, …)

Mobile TV (Korea, Japan, Italy, USA, Qatar, Malaysia, …)

Mobile phones (Nokia, Samsung, SonyEricsson, Apple, …)

Mobile video players (Sony PSP, Apple iPod, …)

Blu-ray disc and HD DVD (Sony, Samsung, Toshiba, LG, …)

AVC HD Camcorders (Panasonic, Sony, Hitachi, …)

Internet video streaming (Apple Quicktime, Adobe Flash-YouTube, Microsoft Silverlight, …)

Video application area where H.264/AVC is not present: Digital Cinema



Denver, CO, USA

H.264/14496-10 AVC Milestones

• ITU-T VCEG call for H.26L proposals February 1998

• First draft of H.26L August 1999

• MPEG call for proposals December 2000

• MPEG proposal evaluations July 2001

• Joint Video Team created December 2001

• First version of standard completed May 2003

• Fidelity Range Extensions (inc. High profile): Fall 2004

• Fast-forward to late 2008/ early 2009:– August 2008: ATAS Primetime Emmy Engineering Award

– January 2009: Paired NATAS Tech & Eng Emmy Awards

• OK, but is there anything newsworthy since 2004?



Denver, CO, USA

H.264/14496-10 MPEG-4 AVC

Milestones since 2004

• Extended-gamut color spaces: Mid 2006

• Professional profiles: Mid 2006

• Scalable Video Coding (SVC) Extension: Fall 2007

• Multi-view Video Coding (MVC) Extension: Fall 2008

• Constrained Baseline Profile: Spring 2009

• Stereo High Profile (and Frame Packing Arrangement

SEI for Stereo Video) Extension: Fall 2009



Denver, CO, USA

Scalable Video Coding (SVC)

scene

SVC

encoder

SVC

decoder

SVC

decoder

SVC

decoder

H.264/AVC

decoder

128

kbit/s

256

kbit/s

512

kbit/s

1024

kbit/s

CIF@

30 Hz

CIF@

15 Hz

QCIF@

7,5 Hz



Denver, CO, USA

1st Concept: Temporal Hierarchy

I P P P P P P P P

B0 B0 B0 B0

Temporal

Scalability

B0 B0B1 B1 B1 B1

B0B1 B1B2 B2 B2 B2

N=1

I P P P P

N=2

I P P

N=4

I P

N=8



Denver, CO, USA

2nd New SVC Concept: Use other

data

• Older scalable designs (MPEG-2, H.263+, MPEG-4 Part 2) used only decoded pictures to predict enhancement layers

• Here, other information is also used– Motion vectors

– Partitioning

– Residual signal

• Intra- and Inter-picture coded regions treated distinctly

• Result:– Capability enhancement

– Less need for decoded picture values

– Opportunity for complexity reduction



Denver, CO, USA

3rd New Concept: Single-loop Decode

• Past spatial scalable video standards: – Inter-layer intra-picture prediction requires that base layer is

completely reconstructed

– Decode using multiple motion compensation loops(and deblocking filter applications)

– Decode complexity greater than simulcast: multi-layer full decoding plus inter-layer prediction

• In SVC: Inter-layer intra prediction is restricted to macroblocks for which the co-located base layer signal is intra-coded– No need to decode multiple motion compensation loops

– Complexity comparable to single-layer coding

– Common building blocks with non-scalable design



Denver, CO, USA

Multiview Video Coding (MVC)

• Increasing use of high-quality 3D production

• In 2006, 200 movie theatres were equipped with 3D technology using glasses,

• Roughly 1000 in 2007

• Major releases (e.g. Beowulf) in 2008, …

• Autostereoscopic display technology developed– Becoming practical for consumer use

– More than 2 views emitted by the display

– E.g. Display by Philips: 9 views

• Lots of 3D at CES 2009; Dominating CES 2010

• What is the best way to code Multi-View Video?



Denver, CO, USA

Multiview Video Concepts

Multiple camera views of

the same scene

Large amount of inter-view

statistical dependencies



Denver, CO, USA

Temporal Coding (Simulcast)



Denver, CO, USA

Inter-view Prediction for Key Pictures



Denver, CO, USA

Inter-view Prediction for All Pictures



Denver, CO, USA

Coding Efficiency of MVC

Ballroom

30

32

34

36

38

100 300 500 700 900

Avgerage Rate [kb/s]

Avg

era

ge Y

-PS

NR

[d

B]

Simulcast (IBBP)

Simulcast (Hierarchical B)

MVC Inter-view Prediction

Sample comparison of simulcast with inter-view prediction(majority of gains due to inter-view prediction at I-picture locations)

~25%



Denver, CO, USA

Key Elements of MVC Design

• Syntax– Does not require any changes to lower-level syntax (slice and lower),

so very compatible with single-layer AVC hardware

– Base layer required and easily extracted from video bitstream (identified by NAL unit type)

• Inter-view prediction– Enabled through flexible reference picture management

– Allow decoded pictures from other views to be inserted and removed from reference picture buffer

– Core decoding modules do not need to be aware of whether reference picture is a time reference or multiview reference

• Small changes to high-level syntax– E.g., specify view dependency, random access points



Denver, CO, USA

MVC and Stereo Video

Remarks and Directions

• Stereo video is an important special case of MVC

• New “Stereo High Profile” provides full-resolution stereo using MVC – Now is a feature of Blu-Ray Disc specification

• “Frame-compatible” (e.g., side-by-side) encoding is another way to enable stereo video – new “frame packing arrangement” supplemental enhancement information standardized

• Classical MVC (i.e. coding of N views) is efficiently achieved through backward-compatible H.264/MPEG-4 AVC extension using only high-level syntax changes

• Further improvements can be achieved by additional tools, but improvements need to be substantial to be justified

• For efficient transmission, still must minimize number of views

• For more substantial benefits, classical MVC coding may be extended by inter-view interpolation (e.g. using depth maps)

• New MPEG exploration activity nearing “Call for Proposals” Stage



Denver, CO, USA

Beyond Current Multi-View Video

• Synthesize a continuum of views based on a limited set of

decoded views

• Specify a format that fixes the rate, but allows arbitrarily large

number of views to be rendered

Data Format

Data Format

Fixed Rate

LimitedCamera

Input ArbitrarilyLarge Number of Output Views

MPEG activity referred to as “3D Video”

(within “FTV” exploration project)



Denver, CO, USA

High Performance Video Coding (HVC) /

Next-Generation Video Coding (NGVC)

Project Scope and Goals

• Goals

– Substantial improvement of compression capability

– Moving beyond current HD for entertainment

– Moving toward 720p and higher for mobile

• Substantial bit rate reduction (e.g. 50%) for equal quality

– E.g., relative to Baseline profile with relatively low complexity

– E.g., relative to High profile for high quality entertainment

• Complexity range (e.g. from 50% to 3x) vs. current High

profile

• Picture size range QVGA (320x240) to 8k x 4k +

• Very broad range of intended applications



Denver, CO, USA

High Performance Video Coding (HVC) /

Next-Generation Video Coding (NGVC)

Latest News

• Requirements documents drafted in both organizations

• New Joint Collaborative Team on Video Coding

(JCT-VC) established January 2010

• Final Call for Proposals issued January 2010

• 31 pre-registered proponents

• Proposals due 22 February 2010

• Subjective evaluation testing March-April 2010

• 1st meeting of JCT-VC: Evaluations 15-23 April 2010

Dresden



Denver, CO, USA

Review

• H.264/14496-10 MPEG-4 AVC a widely deployed state-of-the-art standard technology

• Multiple powerful extensions packages:

– Fidelity Range Extensions (High Profile, etc.)

– Professional Profiles

– Extended Color Gamut

– Scalable Video Coding (SVC)

– Constrained Baseline Profile

– Multiview Video Coding (MVC)

– Stereo High Profile

• VCEG, MPEG, JCT-VC actively investigating new technology developments



Denver, CO, USA

Acknowledgements & Links

• Acknowledgements:

– JVT, MPEG, and VCEG experts

– Thomas Wiegand & Fraunhofer HHI

(presentation material)

– Anthony Vetro of MERL

(some MVC presentation material)

• Links

– http://ftp3.itu.int/av-arch/jvt-site (JVT)

– http://ftp3.itu.int/av-arch/video-site (VCEG)

– http://mpeg.chiariglione.org (MPEG)



Denver, CO, USA

http://ftp3.itu.int/av-arch/jvt-site





http://ftp3.itu.int/av-arch/video-site





http://mpeg.chiariglione.org/

For Further Information• JVT, JCT-VC, VCEG, MPEG video chairs:

– Gary J. Sullivan ([email protected])

– Jens-Rainer Ohm ([email protected])

• H.264/AVC literature references:– IEEE Transactions on Circuits and Systems for Video Technology Special

Issue on H.264/AVC (July 2003)(Luthra, Sullivan, Wiegand, Eds.)

– IEEE Transactions on Circuits and Systems for Video Technology Special Issue on Scalable Video Coding (September 2007)(Wiegand, Sullivan, Ohm, Luthra, Eds.)

– IEEE Transactions on Circuits and Systems for Video Technology Special Section on Multiview Video Coding (November 2007)(He, Ostermann, Tanimoto, Smolic, Eds.)

– Tutorial/overview paper in Proceedings of IEEE January 2005 (Sullivan & Wiegand)

– I. Richardson, H.264 and MPEG-4 Video Compression, 2003

– Overview incl. FRExt: SPIE Aug 2004 (Sullivan, Topiwala, & Luthra)

– Paper at SPIE VCIP 2005: Meta-overview and deployment

– Paper in IEEE Communications Magazine, Aug 2006(Marpe, Wiegand, Sullivan)

– Professional extensions paper, IEEE ICIP, Sept 2007 (Sullivan et al.)

– Wikipedia H.264/AVC page



Denver, CO, USA

mailto:[email protected]




the mpeg-4 part 10 (iso/iec 14496-10) | itu-t h.264 ... · mobile video players (sony psp, apple...

Documents