lec13, video iii (video coding standards),...

Multimedia SystemsMultimedia Systems

Video IIIVideo III

(Video Coding Standards)(Video Coding Standards)

Course PresentationCourse Presentation

(Video Coding Standards)(Video Coding Standards)

Mahdi Amiri

April 2013

Sharif University of Technology

Video Coding StandardsStandardization OrganizationsStandardization Organizations

Two organizations have dominated video compression

standardization.

ITU-T Video Coding Experts Group (VCEG)

International Telecommunications Union –Telecommunications

Standardization Sector (ITU-T, a United Nations Organization,

Multimedia Systems, Mahdi Amiri, Video IIIPage 1

Standardization Sector (ITU-T, a United Nations Organization,

formerly CCITT), Study Group 16, Question 6

ISO/IEC Moving Picture Experts Group (MPEG)

International Standardization Organization and International

Electrotechnical Commission, Joint Technical Committee Number

1, Subcommittee 29, Working Group 11

Video Coding StandardsDynamicsDynamics

VCEG is older and more focused on conventional (esp. low-delay)

video coding goals (e.g. good compression and packet-loss/error

resilience)

MPEG is larger and takes on more ambitious goals (e.g. “object

oriented video”, “synthetic-natural hybrid coding”, and digital


oriented video”, “synthetic-natural hybrid coding”, and digital

cinema)

Sometimes the major organizations team up (e.g. ISO, IEC and ITU

teamed up for both MPEG-2 and JPEG)

Relatively little industry consortium activity (DV and organizations

that tweak the video coding standards in minor ways, such as DVD,

3GPP, 3GPP2, SMPTE, IETF, etc.)

Growing activity for internet streaming media outside of formal

standardization (e.g., Microsoft, Real Networks, Quicktime)

Video Coding StandardsThe Scope of Picture and Video Coding StandardizationThe Scope of Picture and Video Coding Standardization

A Video standard specifically do not define an encoder; rather, they

define the output that an encoder should produce.

A decoding method is defined in each standard (only the Bitstream

Syntax and Decoding Process are standardized):

e.g. use IDCT, but not how to implement the IDCT.


e.g. use IDCT, but not how to implement the IDCT.

Permits optimization beyond the obvious.

Permits complexity reduction for implementability.

Provides no guarantees of Quality - only interoperability.

Ensuring Ensuring interoperabilityinteroperability::

Enabling communication between Enabling communication between

devices made by different devices made by different

manufacturersmanufacturers

Video Quality EvaluationSubjectiveSubjective

A human “subject” rates the video on a scale

Double Stimulus Continuous

Quality Scale Method

Hidden scale of 0-100


Hidden scale of 0-100

Difference is calculated as

the actual rating

Video Quality EvaluationObjectiveObjective

A computer algorithm judges the distortion between

videos

Attempts to model a human observer

There is currently no standard method


There is currently no standard method

Video Quality EvaluationObjective Metrics: PSNRObjective Metrics: PSNR

Peak Signal-To-Noise Ratio (PSNR)

Used widely in evaluating coding performance

Purely mathematical difference

Can be tricked quite easily


Can be tricked quite easily

Root Mean Squared Root Mean Squared

Error (RMSE)Error (RMSE)

255 255 = = 22^n ^n –– 11

n: the number of bits per image samplen: the number of bits per image sample

Video Quality EvaluationPSNR, ExamplePSNR, Example


OriginalOriginal PSNR PSNR 3535..4 4 [dB][dB] PSNR PSNR 2929..0 0 [dB][dB]

ABCABAB

OriginalOriginal PSNR PSNR 4545..53 53 [dB][dB]


PSNR PSNR 3636..81 81 [dB][dB] PSNR PSNR 3131..45 45 [dB][dB]

Video Quality EvaluationABCABC



ReminderReminder

CIFCIF--size image size image

352352××288288


Video Quality EvaluationABCABCR-D Performance of MPEG Codecs

44

46

48

50


32

34

36

38

40

42

350 450 550 650 750 850 950 1050

Bit rate (kbps)

PSNR (Y)

MPEG-1 MPEG-2 MPEG-4 H.264

Video Quality EvaluationObjective Metrics: PSNRObjective Metrics: PSNR

How to trick PSNR

Take a natural image

Give more bits to areas you look at more

Give less bits to areas you look at less


Give less bits to areas you look at less

Subjective rating will be high, PSNR low

OriginalOriginal Attention Map ExampleAttention Map Example TestTest(High subjective rating, low PSNR)(High subjective rating, low PSNR)

Video Coding StandardsGary J. Sullivan, Ph.D.

Gary J. Sullivan, Ph.D.

Rapporteur

Rapporteur| C

hair

| Chair

Rapporteur

Rapporteur| Co

| Co--Chair

Chair

Rapporteur

Rapporteur| C

o| C

o--Chair

Chair




ITU

ITU--T VCEG

T VCEG Rapporteur

Rapporteur

ISO/IEC M

PEG Video

ISO/IEC M

PEG Video Rapporteur

Rapporteur

ITU/ISO/IEC JVT

ITU/ISO/IEC JVT Rapporteur

Rapporteur

Video Coding StandardsHistoryHistory


en.wikipedia.org/wiki/Data_compressionThe Society of Motion Picture and Television Engineers, SMPTE (pron. simpti) is an internationally

recognized standards organizations founded in 1916 (en.wikipedia.org/wiki/SMPTE).

Video Coding StandardsMajor Video Compression ApplicationsMajor Video Compression Applications


Video Coding StandardsMoving Picture Experts Group (MPEG)Moving Picture Experts Group (MPEG)

A working group of ISO/IEC in charge of the

development of standards for coded

representation of digital audio and video and

related data.


related data.

Established in 1988

23 years of activity

The number of independent standards: more than

125

Video Coding StandardsMPEG-1 The standard on which such products as Video CD and MP3 are based

MPEG-2 The standard on which such products as Digital Television set top boxes and DVD are

based;

MPEG-4 The standard for multimedia for the fixed and mobile web;

MPEG-7 The standard for description and search of audio and visual content;

MPEG-21 The Multimedia Framework;

MPEG-A The standard providing application-specific formats by integrating multiple MPEG

technologies;

MPEG-B A collection of Systems specific standards

MPEG-C A collection of Video specific standards


MPEG-C A collection of Video specific standards

MPEG-D A collection of Audio specific standards

MPEG-E A standard (M3W) providing support to download and execution of multimedia

applications

MPEG-H A standard (HEVC) providing a significantly increased video compression performance

MPEG-M A standard (MXM) for packaging and reusability of MPEG technologies

MPEG-U A standard for rich-media user interface

MPEG-V A standard for interchange with virtual worlds

Video Coding StandardsVideo Coding Experts Group (VCEG)Video Coding Experts Group (VCEG)

Part of study group 16 (Multimedia coding, systems and

applications) of the ITU-T. Established in 1984

H.120 The first digital video coding standard

H.261 Was the first practical digital video coding standard.


H.261 Was the first practical digital video coding standard.

H.262 It is identical in content to the video part of the ISO/IEC MPEG-2 standard.

H.263 Provided a suitable replacement for H.261 at all bitrates.

H.263v2 Also known as H.263+, Enhanced robustness against data loss in the transmission channel.

H.264 The ITU-T H.264 standard and the ISO/IEC MPEG-4 Part 10 standard (formally, ISO/IEC 14496-

10) are technically identical.

H.265 Not yet developed; expected 2012 or later.

H.271 Video back channel messages for conveyance of status information and requests from a video

receiver to a video sender.

Video Coding StandardsH.H.120120

The First Digital Video Coding Standard

ITU-T (ex-CCITT) Rec. H.120: 1984

v1 (1984) had conditional replenishment, DPCM, scalar

quantization, variable-length coding, switch for quincunx sampling

v2 (1988) added motion compensation and background prediction


v2 (1988) added motion compensation and background prediction

Operated at 1.544 (NTSC) and 2.048 (PAL) Mbits/s

Few units made, essentially not in use today

Conditional ReplenishmentConditional Replenishment: Can signal to leave a block area of the image : Can signal to leave a block area of the image

unchanged, or replace it with new data (using a threshold value).unchanged, or replace it with new data (using a threshold value).

Quincunx samplingQuincunx sampling: In a digital video system, a sampling structure with : In a digital video system, a sampling structure with

an array of samples where alternate rows of pixel samples are displaced an array of samples where alternate rows of pixel samples are displaced

horizontally in the grid by half of the pitch of the pixel samples along the horizontally in the grid by half of the pitch of the pixel samples along the

remaining rows.remaining rows.


ITU-T, completed in 1990, The first widespread

practical success

Video telephony and teleconferencing over ISDN

(Integrated Services Digital Network)


Embodying typical structure that dominates today

Combination of DPCM and DCT

Motion Compensation

p x 64kbps (64-2048 kbps)

Still in use, although mostly as a backward-compatibility

feature – overtaken by H.263

Video Coding StandardsMPEGMPEG--11

MPEG-1: “Coding of moving pictures and associated audio for

digital storage media” (1992)

Group of Pictures, Motion Estimation and Motion Compensation,

Differential Coding, DCT, Quantization, Entropy Coding

Video on digital storage media (CD-ROM)


Video on digital storage media (CD-ROM)

Target was VHS Quality at 1.5MBits/s (at 352x240 resolution)

Basis of Video-CD

MP3 (MPEG-1 Layer 3)

16 bits

Samping rate - 32, 44.1, or 48 kHz

Bitrate – 32 to 320 kbps

De facto - 44.1 kHz sample rate, 192 kbps bitrate


Only supports progressive pictures.

Adds bi-directional motion prediction to H.261 design.

Adds half-pixel motion estimation (See next slide).

Slice-structured coding.


Slice-structured coding.

DC-only “D” pictures.

Superior quality to H.261 when operated a higher bit

rates ( > 1 Mbps for CIF 352x288 resolution).

Now mostly overtaken by MPEG-2.

Video Coding StandardsMPEGMPEG--11, Half, Half--Pixel MEPixel ME

Half-Pixel (coarse-fine) Motion Estimation Algorithm

1) Coarse step: Perform integer motion estimation on blocks; find best integer-pixel MV

2) Fine step: Refine estimate to find best half-pixel MV

a) Spatially interpolate the selected region in reference frame

b) Compare current block to interpolated reference frame block

c) Choose the integer or half-pixel offset that provides best matchTypically, bilinear Typically, bilinear


Typically, bilinear Typically, bilinear

interpolation is used for interpolation is used for

spatial interpolationspatial interpolation


MPEG-2: “Generic coding of Moving Pictures and

Associated Audio”.

Broadcasting and storage.

Satellite TV, DVD, Digital TV.


Ubiquity in hardware implies that it will be here for a

long time.

Transition to HDTV has taken over 10 years and is not finished yet

Different profiles and levels allow for quality control.

Bitrates: Typ. 4-9 MBits/s (Not especially useful below 4 Mbps,

normal range of use 5-30 Mbps).


Support for interlaced scan, various picture

sampling formats, user defined quantization

matrix.

Essentially same as MPEG-1 for progressive-


Essentially same as MPEG-1 for progressive-

scan pictures.

Various forms of scalability (SNR, Spatial,

Temporal and hybrid).

Base Layer: Basic quality requirement, For SDTV.

Enhanced Layer: High quality service, For HDTV.

Video Coding StandardsMPEGMPEG--2 2 Profiles and LevelsProfiles and Levels

Goal: To enable more efficient implementations for different

applications (interoperability points)

Profile: Subset of the tools applicable for a family of applications

Level: Bounds on the complexity for any profile


Video Coding StandardsBitrateBitrate allocationallocation

CBR – Constant BitRate

Streaming media uses this.

Easier to implement.

VBR – Variable BitRate


VBR – Variable BitRate

DVD’s use this.

Allocate more bits for complex scenes.

Usually requires 2-pass coding.

This is worth it, because you assume that you encode

once, decode many times.

Video Coding StandardsMPEG Container FormatMPEG Container Format

Container format is a file format that can

contain data compressed by standard codecs.

2 types for MPEG

Program Stream (PS) – Designed for


Program Stream (PS) – Designed for

reasonably reliable media, such as disks.

Transport Stream (TS) – Designed for lossy

links, such as networks or broadcast antennas.

Video Coding StandardsMPEGMPEG--3 3 ??

Originally developed for HDTV, but abandoned when

MPEG-2 was determined to be sufficient.



ITU-T Rec. H.263 (v1: 1995): The next generation of

video coding performance, developed by ITU-T – the

current premier ITU-T video standard (has overtaken

H.261 as dominant videoconferencing codec)

Video telephony over PSTN (public switched telephone


Video telephony over PSTN (public switched telephone

network)

Wins by a factor of two at very low rates

Version 2 (late 1997 / early 1998) & version 3 (2000)

later developed with a large number of new features

H.263+ & H.263++ (Extensions to H.263)


MPEG-4: “Coding of audio-visual objects”.

Started as very low-bitrate project.

Contains the H.263 baseline design and adds many

creative new extras:


creative new extras:

Coding of media objects (Segmented coding of shapes).

Bitrate: variable.

Synthetic/Semi-synthetic objects.

XMT: Like HTML, but to build videos.

First standard with Intellectual Property Management.


Part Number Title Description

Part 1 ISO/IEC 14496-1 SystemsDescribes synchronization and multiplexing of video and audio. For example Transport

stream.

Part 2 ISO/IEC 14496-2 VisualA compression codec for visual data (video, still textures, synthetic images, etc.). One of

the many "profiles" in Part 2 is the Advanced Simple Profile (ASP).

Part 3 ISO/IEC 14496-3 AudioA set of compression codecs for perceptual coding of audio signals, including some

variations of Advanced Audio Coding (AAC) as well as other audio/speech coding tools.

Part 4 ISO/IEC 14496-4 Conformance Describes procedures for testing conformance to other parts of the standard.

Part 5 ISO/IEC 14496-5 Reference Software Provides software for demonstrating and clarifying the other parts of the standard.

Delivery Multimedia


Part 6 ISO/IEC 14496-6

Delivery Multimedia

Integration Framework

(DMIF).

Part 7 ISO/IEC 14496-7Optimized Reference

Software

Provides examples of how to make improved implementations (e.g., in relation to Part

5).

Part 8 ISO/IEC 14496-8 Carriage on IP networks Specifies a method to carry MPEG-4 content on IP networks.

Part 9 ISO/IEC 14496-9 Reference HardwareProvides hardware designs for demonstrating how to implement the other parts of the

standard.

Part 10 ISO/IEC 14496-

10

Advanced Video Coding

(AVC)A codec for video signals which is technically identical to the ITU-T H.264 standard.

http://en.wikipedia.org/wiki/MPEGhttp://en.wikipedia.org/wiki/MPEG--44

Video Coding StandardsMPEGMPEG--44, Object Based Coding, Object Based Coding

Extension of MPEG-1/2-type algorithms to code arbitrarily shaped objects


Basic Idea: Extend BlockBasic Idea: Extend Block--DCT and BlockDCT and Block--ME/MCME/MC--

prediction to code prediction to code arbitrarily shaped arbitrarily shaped objects.objects.

[MPEG Committee][MPEG Committee]

Video Coding StandardsMPEGMPEG--44, Sprite Coding, Sprite Coding

Sprite: Large background image.

Hypothesis: Same background exists for many

frames, changes resulting from camera motion and

occlusions.


One possible coding strategy:

1. Code & transmit entire sprite once.

2. Only transmit camera motion parameters for each

subsequent frame.

Significant coding gain for some scenes.

Video Coding StandardsMPEGMPEG--44, Sprite Coding, Sprite Coding


[MPEG Committee][MPEG Committee]

Video Coding StandardsH.H.264 264 or MPEGor MPEG--4 4 Part Part 10 10 (AVC)(AVC)

MPEG-4 Part 10: Advanced Video Coding / H.264.

Designed by a Joint MPEG and VCEG.

Claims 50% bitrate savings to MPEG-2, 30% over MPEG-4!

Bitrate: 10’s to 100’s kb/s.

Variable Block Size, Multiple Reference Frames, Integer Transform,


Variable Block Size, Multiple Reference Frames, Integer Transform,

Intra Prediction, In-loop Deblocking Filtering, 1/4-pel Resolution

Motion Estimation, ASO (Arbitrary Slice Ordering), FMO (Flexible

Macroblock Ordering).

Enhanced entropy coding.CAVLC (Context Adaptive Variable Length Coding)

CABAC (Context Adaptive Binary Arithmetic Codes)

Increased complexity relative to prior standards.

Video Coding Standards, H.264Integer TransformInteger Transform

MPEG-2, MPEG-4


MPEG-4 AVC

Find out more about integer

transform by looking for

"lifting scheme"

Video Coding Standards, H.264Variable Block SizeVariable Block Size

The fixed block size may not be suitable for all motion objects

Improve the flexibility of comparison

Reduce the error of comparison

7 types of blocks for selection


00

10

2 3

0 11

16 x 16 16 x 8 8 x 16 8 x 8

00

10

2 3

0 11

8 x 8 8 x 4 4 x 8 4 x 4

Video Coding Standards, H.264Variable Block SizeVariable Block Size

Residual (without MC) showing optimum choice of partitions


Video Coding Standards, H.264Multiple Reference FramesMultiple Reference Frames

The neighboring frames are not the most similar in some cases.

The B-frame can be reference frame.

B-frame is close to the target frame in many situations.


Video Coding Standards, H.264DeblockingDeblocking FilterFilter

There are severe blocking artifacts.

4*4 transforms and block-based motion compensation.

Result in bit rate savings of around 6~9%.

Improve subjective quality and PSNR of the decoded picture.


Without Filter With AVC Deblocking Filter

Video Coding Standards, H.264FMO (Flexible FMO (Flexible MacroblockMacroblock Ordering)Ordering)

Slice (composed in FMO) �� Enhance Robustness to data loss


Subdivision of a picture into Subdivision of a picture into

slices when slices when not using FMOnot using FMO

Subdivision of a QCIF frame into Subdivision of a QCIF frame into

slices when slices when utilizing FMOutilizing FMO

H.264, ProfilesABCABC

ABC

http://en.wikipedia.org/wiki/MPEG

http://en.wikipedia.org/wiki/MPEG--44_AVC.htm

_AVC.htm




H.264, ProfilesABCABC

ABC


Video Coding Standards, H.265H.H.265 265 or HEVCor HEVC

High Efficiency Video Coding (HEVC) / H.265.

Under joint development by MPEG and VCEG.

Claims 50% bitrate savings to H.264.

Two of the key features where HEVC was improved compared to H.264/MPEG-4

AVC was:

Support for higher resolution video.

MP: Main Profile

HP: High Profiel

ASP: Advanced Simple Profile

HLP: High Latency Profile


Support for higher resolution video.

Improved parallel processing methods. en.wikipedia.org/wiki/High_Efficiency_Video_Coding

Scalable Video CodingMotivationMotivation

Basic situation:

1. Diverse receivers may request the same video

Different bandwidths, spatial resolutions, frame rates, computational capabilities

2. Heterogeneous networks and a priori unknown network conditions

Wired and wireless links, time-varying bandwidths

When you originally code the video you don’t know which client or network situation


When you originally code the video you don’t know which client or network situation

will exist in the future.

Probably have multiple different situations, each requiring a different compressed

bitstream.

Need a different compressed video matched to each situation.

Possible solutions:

1. Compress & store MANY different versions of the same video.

2. Real-time transcoding (e.g. decode/re-encode).

3. Scalable coding.

Scalable Video CodingType of ScalabilityType of Scalability


The basic types of scalability in video codingThe basic types of scalability in video coding

Scalable Video CodingTemporal ScalabilityTemporal Scalability

Based on the use of B-frames to refine the temporal resolution.

B-frames are dependent on other frames.

However, no other frame depends on a B-frame.

Each B-frame may be discarded without affecting other frames.


Scalable Video CodingSpatial ScalabilitySpatial Scalability

Based on refining the spatial resolution

Base layer is low resolution version of video.

Enhanced (Enh1) contains coded difference between

upsampled base layer and original video.

Also called: Pyramid coding.


Scalable Video CodingQuality ScalabilityQuality Scalability

Based on refining the amplitude resolution.

Base layer uses a coarse quantizer.

Enh1applies a finer quantizer to the difference between the original DCT

coefficients and the coarsely quantized base layer coefficients.

Also called: SNR Scalability.


Note: Base & enhancement layers Note: Base & enhancement layers

are at the are at the samesame spatial resolution.spatial resolution.

Video Coding StandardsIndependent playersIndependent players

Efforts from independent companies in video

compression.

Main competitors:RealNetworks (www.real.com)


RealNetworks (www.real.com)

Microsoft Windows Media (WM) (www.microsoft.com)

Apple Quicktime (www.apple.com)

On2 (www.on2.com) (now acquired by Google)

MPEG & Multimedia ApplicationMPEGMPEG--77

MPEG-7: “Multimedia Content Description

Interface” (2001)

Describing audio/video

Applications:


Applications:

Indexing of video databases.

Search & Retrieval.

Browsing.

Independence between description and content.

MPEGMPEG--2121

Aims at defining an open framework for multimedia

applications.

MPEG-21 is based on two essential concepts:

Definition of a Digital Item (a fundamental unit of distribution

and transaction).

MPEG & Multimedia Application


and transaction).

Users interacting with Digital Items.

Main objective of the MPEG-21:

Is to define the technology needed to support users to exchange,

access, consume, trade or manipulate Digital Items in an efficient

and transparent way.

MPEGMPEG--21 21 and MPEGand MPEG--7 7 ExampleExample

Metadata and Identifiers

within an MPEG-21 Music

Album

MPEG & Multimedia Application


Thank You

Multimedia SystemsMultimedia Systems

Video III (Video Coding Standards)Video III (Video Coding Standards)


Thank You

1. http://ce.sharif.edu/~m_amiri/

2. http://www.dml.ir/

FIND OUT MORE AT...

Next Session: Multimedia Networks INext Session: Multimedia Networks I

lec13, video iii (video coding standards),...

Documents