lec 10 video coding standard and system -...

Spring 2017: Multimedia Communication

Lec 10

Video Coding Standard and System - HEVC

Zhu Li

Course Web:

http://l.web.umkc.edu/lizhu/

Z. Li Multimedia Communciation, Spring 2017 p.1

http://l.web.umkc.edu/lizhu/teaching/2016sp.video-communication/main.html

Outline

Lecture 09 Video Signal Processing ReCap

Course Project Topic: Light Field Compression

Video Coding Standard and Systems

HEVC History

HEVC Performance

HEVC Key Coding Features


Coding Structure

Fast Block Motion Estimation:

Video Signal Processing


I P P P … P P P

……

GoP

1 2 3 4 5 6 7

I B B P B B P

GoP

MV coding modes in HEVC

3 MV coding modes

key ideas, predicting from current block’s spatio-temporal neighbors

new merge mode: saving MV coding bits by combine small blocks into larger blocks.


Intra Prediction in HEVC

Much more modes DC mode: copy DC values from neighbor

Planar mode: top row or left col average

Angular: pixels on certain line

Ref: Jani Lainema, Frank Bossen, Woojin Han, Junghye Min, Kemal Ugur, Intra Coding of the HEVC Standard. IEEE Trans. Circuits Syst. Video Tech. 22(12): 1792-1801 (2012)


Like a sparse transform basis!

Deblocking Filter

Reduce blocking artifact in the reconstructed frames

Can improve both subjective and objective quality

Filter in H.261:

[1/4, 1/2, 1/4]: Applied to non-block-boundary pixels in each block.

A low-pass smoothing filter.

In H.264 (and H.263v2), this is used in the prediction loop to

improve motion estimation accuracy. Decoder needs to do the

same. Also called loop filter.

Before…. and After

H.265: 8x4 block level

H.264: 4x4 block level


Sample Adaptive Offset (SAO) Filter

Classify pixels on block edge as one of the four categories

Offset its pixel value accordingly


Scalable Video Coding

Scalable video coding is desired in heterogeneous networks:

Types of scalability:


HW-3: Motion Estimation

Fractional Pel Motion Compensation

Use bi-linear interpolation to scale up the frames 2x2

Do ½ pel motion estimation and compensation

Compare block size and pel accuracy effects on the residual MSE

B-Frames

Implement a special version of B-frames, have forward and backward motion vectors

Compare the results with P frame motion estimation

Fast Motion Estimation

Implement Diamond search pattern, compare with the exhaustive search

Data set:

“foreman” sequence frames 150, 151, and 152.


Outline




HEVC History

HEVC Performance



What is Light Field ?

Capturing all lights in an observing volume


Light Field Acquisition and Format

Lytro System

• Images were acquired using a Lytro Illum camera

• LFR (light field raw) file format as defined by Lytro contains the raw image

data:

Output of camera sensor, overlaid with ‘GRBG’ Bayer filter;

7728×5368 samples - 10-bit little endian integer values;

• A demosaicing process is required to obtain the color information from

the sensor data;

Demosaicing: raw data → RGB image;

From Prof. Fernando Pereira’s San Diego JPEG Presentation


View Extraction Processes

Compression Code Point:

• Light fields were extracted from the LFR files using the Light Field Toolbox

v0.4 made available by Donald Dansereau, QUT, AU;

• http://lightfield-forum.com/tag/lightfield-toolbox/

• The process of obtaining a specific view from the full light field involves

two main stages:

Coding

Multi-perspectives

Multi-focus

From Prof. Fernando Pereira’s San Diego JPEG Presentation


Lenslet (sensor) vs Sub-aperture (synthesis) Images


Lenslet sensor capture Sub-aperture image

Light Field Data Set

EPFL data set (6.6GB) : http://mmspg.epfl.ch/EPFL-light-field-image-dataset


a) Bikes b) Desktop c) Flowers

d) Friends 1 e) Magnets 1 f) Stone Pillars Outside

Light Field Compression Test

View Point Compression

Pre-selected 9 perspective view point in the light field


Naïve Solution – Direct Coding of Lenslet

R-D curves: Frenando Pereira/IST results:


JPEG JPEG 2000

H.264/AVC

Intra

HEVC

Intra

Naïve Solution – Codec Performances

JPEG, JPEG2K, AVC Intra, HEVC Intra

Fernando Pereira, at JPEG meeting in San Diego, 2016.02


Bikes Desktop Flowers

Re-Focusing Point Performance

Re-Focusing Operating Points


Re-Focusing Point R-D curves

Also across different codec, naïve solution (Fernando Pereira, IST)


JPEG JPEG 2000

H.264/AVC

Intra

HEVC

Intra

LF Coding Discussions

Redundancy in the lenslet images

Across lenslet prediction ?

o Optical correction ?

o Affine block matching ?

R-D optimization for certain view perspective ?

Across focal lens R-D optimization ?


Outline




HEVC History

HEVC Performance



HEVC/H.265 History

HEVC Objective:


Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial

New Design Features

HEVC (H.265) vs AVC (H.264)



HEVC Development History

Long hard fight, many casualties…


HEVC Performance – PSNR

HEVC vs AVC


HEVC Performance – Subjective Score

HEVC vs AVC


HEVC Coding Pipeline

HEVC Diagram


Credit: Prof. Oscar Au, Tutorial on HEVC

HEVC Coding Structure

Quad Tree Decomposition:


Slide Credit: Vivienne Sze & Madhukar Budagavi, ISCAS 2014 Tutorial

Ref:G. Schuster, PhD Thesis, 1996: Optimal Allocation of Bits Among Motion, Segmentation and Residual

HEVC Coding Structure

CTU- Coding Tree Unit

Upto 64x64 pel block

Macro Block equivalent

Aka LCU: Largest Coding Unit

CU – Coding Unit

Square NxN pel

Like Block

PU – Prediction Unit

Inter Prediction

Intra Prediction


Prediction Unit

PU:

Basic Unit for Transform & Quantization


Transform Unit (TU)

TU Square blocks for transform & quantization/coding, size: 4x4, 8x8,

16x16, 32x32

MaxTU size: 32 for luma, 16 for chroma

Min TU size: 4x4 for both luma and chroma

TU size can be larger than PU for inter PU case (MV merge)


HEVC Transforms

Transform + Quant:


HEVC Intra-Prediction

Intra-Prediction Modes


Intra-Predicted Basis

As if it is a 1-non zero coefficient transform…


Ref:J. Laniema and W.-J. Han, “Intra Picture Prediction in HEVC”, Chapter in, Springer-Velag Book on High Efficiency Video Coding (HEVC): Algorithms and Architectures, Springer, 2014. Ed. V. Sze et. Al.

Pre-Processing for Intra Prediction

Smoothing filtering before intra prediction


Post-Processing for Intra Prediction

Remove discontinuity:


HEVC Intra as Still Image Coding Standard

Advancing the state or art by quite a bit


HEVC Inter Prediction

Supports 4x4 block ¼ pel accuracy Motion Compensation


Interpolation Filters

Quite a bit of memory and computing cost:


Fast Motion Estimation in HEVC HM

Diamond Search


HEVC Entropy Coding

Binary Arithmetic Coding:


HEVC CABAC Throughput Improvement

Context Bypassing Mode:


Parallel Processing Tools: Slice/Tile



Profiles, Levels and Tiers

Operating HEVC in Profiles/Levels/Tiers


HEVC Resources

Main Spec: http://www.itu.int/ITU-T/recommendaBons/rec.aspx?rec=11885

T-CSVT Special Issue: 2012: Combined Issue on HEVC Standard and Research: http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6403920

2016: Special Issue on HEVC Extensions and Efficient HEVC Implementations:

http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=7372356

Springer Book: V. Sze, M. Budagavi, G. J. Sullivan (Editors), “High

Efficiency Video Coding (HEVC): Algorithms and Architectures,” Springer, 2014,http://www.springer.com/engineering/signals/book/978-3-319-06894-7

HM (open source software): https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSovware/

HARP: A python HM access tool: http://www.lms.lnt.de/HARP/


http://ieeexplore.ieee.org/xpl/tocresult.jsp?isnumber=6403920

Summary

Light Field Compression

Sensor data at lenslet

Support new perspective and re-focusing

Interesting R-D optimization issue at new perspective and focusing point in the light field

Calls for new coding tools, based on HEVC intra, e.g,

HEVC

Crown Jewel of 20+ years of research and engineering

Significant performance gains over the previous state of art

Worthwhile to master this tool for a variety of research and biz opportunities.


lec 10 video coding standard and system -...

Documents