1 h.264 fast mode decision multimedia systems team project 林展意 695410074 林志仁 695410063

1

H.264 Fast Mode Decision

Multimedia Systems Team Project

林展意 695410074林志仁 695410063

2

• H.264 intra-prediction and inter-prediction– “Fast Macro-block Mode Determination to Reduce

H.264 Complexity”

– Fast Inter-mode Decision in H.264/AVC Video Coding

– “Feature-Based Intra-Prediction Mode Precision for H.264”

• Fast intra 4*4 block mode decision

– “Fast Mode Decision for H.264”• Fast P frame encode

3

Outline

Introduction

Fast mode decision

Comparison and Conclusion

Reference

Frame Encoding

H.264 Frame

4

Outline

Introduction

Fast mode decision


Reference

H.264 Frame

Frame Encoding

5

H.264 has Certain distinct features such as

variable block size Motion Estimation, multiple

reference frames, and so on.

Introduction

H.264 uses rate distortion optimization technique to

get the best coding result in terms of maximizing

coding quality and minimizing resulting data bits.

7

However, the complexity of the codec and the

computational cost increase tremendously, how to

reduce the complexity of the algorithm is a

valuable problem here.

Introduction

8

Outline

Introduction

Fast mode decision


Reference

Frame Encoding

H.264 Frame

I - FrameP - Frame

9

I - Frame

P - Frame

Frame EncodingIntra Frame

For Every

Macro-Block

For every 8*8 block

DCT

Quantization

Entropy Coding

110010110….

YCb

Cr

RGB

• Macro-block– 16*16 RGB macro-block

for YCbCr transformation– 4:2:0 chromance format – Block size 8*8 for DCT

transformation– Fixed quantization step

size– Context free entropy

coding• Huffman coding

10

I - Frame

P - Frame

Frame Encoding

• Motion vector Block matching algorithm

• Sequential search

• Binary search

11

I - Frame

P - Frame

Frame Encoding

Target frame

Reference frame

16

16

MV

Current macro-block

Best match- Y

Cb

Cr

Difference macro-block

For every 8*8 block

DCTQuantization

Entropy Coding

110010110….

This process of finding the best match is known as Motion Estimation

12

Outline

Introduction

Fast mode decision


Reference

Frame Encoding

H.264 Frame

Intra-prediction Inter-prediction

13

16 168 8

81616

8

888

8

44

44

The method of partition macro-blocks into motion compensated sub-blocks of varying size is known as tree structured motion compensation.

14Block sizes chosen after H.264 RDO. Ex. CIF sequence “Paris”

15

Fast mode decision for H.264

• P frame encode– “Fast mode decision for H.264”

• Decision skip mode or inter-prediction or intra-prediction– 1. Inter-prediction

• Temporal

• Skip mode , or block size decision

– 2. Intra-prediction• Spatial

• H.264 can do it

• but other codec using JPEG-like encode

16

High computation need

降低計算量且維持 H.264 quality : fast mode decision • Macro-block 16*16block size decison

• Intra-prediction : mode decision to find optimal residue matrix and block size decision

– 16*16 or 4*4

– Intra-prediction optimal SAD(SATD) cost

– Intra-prediction optimal RD cost

• Inter-prediction : optimal motion vector decision to find residue matrix and block size decision

– 16*16,16*8,8*16,8*8,8*4,4*8,4*4

– Using residue matrix

– Inter-prediction optimal SAD cost

– Inter-prediction optimal RD cost

Inter RD cost

Intra RD cost

Inter SATD

Inter SAD

Intra SAD Intra SATD

17

Intra-prediction prediction:

spatial redundancy • Encoder decide intra-

prediction– 1. 16*16 and 4*4 mode

decision• SAD or SATD cost• RD cost

– 2 . decide block size• Block size

– 16*16 or 16 個 4*4• Mode decision

– 16*16 have 4 mode– 4*4 have 9 mode decision– (fast 4*4mode decision)

• “feature-based intra-prediction mode decision for H.264”

18

Inter-prediction mode

• 1. find combination block size optimal motion vector and residue matrix

• Residue matrix to calculate– SAD , low complexity to

calculate quality– Or RD cost, high

complexity to calculate quality

• 2. decide block size combination

19

How to find the best matching block ?

Mean Square Error (MSE)

Mean Absolute Error (MAE)

Sum of absolute Difference (SAD)

Rate Distortion Optimization (RDO)

s=1, MAE ; s=2, MSE

1M N

M ¡ 1P

k=0

N ¡ 1P

l=0jC(x +k;y+ l) ¡ R(x + i +k;y+ j + l)js

M ¡ 1P

k=0

N ¡ 1P

l=0j(C(x + k;y+ l) ¡ R(x + i + k;y+ j + l))sj

20

Outline

Introduction

Fast mode decision


Reference

Frame Encoding

H.264 Frame

21

Fast Macro-block Mode Determination to Reduce H.264 Complexity

Ki-Hun HAN, Yung-Lyul LEE

IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences Issue 3 (March 2005) 800-804

OXFORD Journal

22

Using the RDO technology to decision the macro-block mode in the H.264 encoder.

The proposed method reduces the encoder complexity by 55% and reduces approximately 10 % of bit-rate.

The RDO mechanism is used to select the best MB mode among all of the blocks mode.

Summary

23

Rate Distortion Optimization (RDO)

RDcost : To choose the best MB mode that has the minimum rate distortion value.

Distortion = MSE

¸mode

RDcost = Distortion+ ¸modeRates

, the Lagrangian parameter , is a function related to the H.264 quantization parameter (QP), whose value ranges from 0 to 51.=

0:85¢2(Q P ¡ 12)

3

24

P-Frame

Inter mode decision

Intra mode decision

Skip

Inter 16*16

Inter 16*8

Inter 8*16

Inter 8*8

Intra 16*16

Intra 4*4

Inter 8*8

Inter 8*4

Inter 4*8

Inter 4*4

P-Frame mode decision

25


If the RDcost value of the Inter-16*16 mode is the smallest of the three mode, it is preferable to select the Inter-16*16 MB mode.

For each Macro-block

ME of Inter-16*16, Inter16*8, Inter 8*16 modeME & RDcost computation

If RDcost of Inter-16*16 mode

is minimum among 3 modes

P8*8 mode ME & RDcost computation

Best Inter mode decision

Inter-8*8 SkipNo

Yes

26


Intra mode decision

Vertical Spatial Prediction Errors (VSPE)

Horizontal Spatial Prediction Errors (HSPE)

27


Intra mode decision

Intra-16*16If ((SPE > TPE) and (SPE >= 20))

Intra-16*16 mode RDcost computation

Intra-16*16 Skip

No

Yes

TPE is the sum of the SAD between the current MB and predicted MB with respect to the best Inter MB mode and the motion vector bits.

28


Intra mode decision

Intra-4*4

Intra_RD = (Sum of the RDcost of all MBs encoded in the Intra-4*4 mode that has computed until now) / n

If Best_Inter_mode_RDcost <

Intra_RD

Intra-4*4 mode RDcost computation

Intra-4*4 Skip

No

Yes

31

Intra_RD = CurRD+n¢Intra RDn+1

CurRD is the RDcost value when the current MB is selected as the Intra 4*4 mode.

n denotes the number of intra 4*4 MBs.

32

Result

Recommended common experimental conditions by H.264 standard group

33

Experimental results

ETime : total reduction in computation timeResult

34

Fast Intermode Decision in H.264/AVC Video Coding

D. Wu, F. Pan, K. P. Lim, S. Wu, Z. G. Li, X. Lin, S. Rahardja, and C. C. Ko

IEEE Journal Circuits and Systems for Video Technology, IEEE Transactions onVolume 15, Issue 7, July 2005 Page(s):953 - 958

35

It makes use of the spatial homogeneity and the temporal stationarity characteristics of video objects.

This paper proposes a fast algorithm is able to reduce on the average 30% encoding time and a bit rate increment of 0.6%.


spatial homogeneity

temporal stationarity

36

Homogeneous blocks in the picture would have similar motion and are very seldom split into smaller blocks.

There is an average of 35% homogeneous in a typical video frame, and these areas are suitable for larger size inter-mode coding, and RD cost calculation of small size modes can be avoided.

Specifically, spatial homogeneity of a MB is decided based on the MB’s edge intensity.


37

(i, j)

(i-1, j+1)

(i, j+1)(i+1, j+1)

(i-1, j-1) (i, j-1)

(i, j+1)

dxi ;j =+ 2 + - - 2 - + 2 + - - 2 -

dyi ;j =

(i, j)(i-1, j)

(i+1, j)

(i+1, j-1)(i-1, j-1)

(i-1, j+1)

(i+1, j+1)

If the sum of magnitude of the edge vectors at all pixel locations in the block is less than the block homogeneous threshold, ThdH :

ThdH is set to 20,000 for 16*16 block and 5,000 for 8*8 block .

Homogeneous Regions Determination

38

Stationary Regions Determination

We can use the sum of absolute difference to check if this MB changes or not.

Di® =i=1;j =1P

16;16abs(M [i; j ]¡ N [i; j ])

If the change between the two MBs is less than a threshold, Ts, the MB is classified as stationary and 16*16 mode is used for motion estimation.

Based on the experimental results, we set the threshold Ts to 200.

39

Perform edge detection on one image

Generate the edge direction histogram.

Predict if the current

MB has zero motion

Compute the MB difference.

If the sum is greater than the threshold

Yes

No

No

Yes

40

Perform RDOon the

16*16 block

If the MB is homogeneous

Perform RDO on the 16*16 blockand selectively on 16*8 or

8*16 block

NoYes

Perform RDO on the 16*16, 16*8 and 8*16 blocks. Save the best

mode of the three

41

Perform RDO on the 8*8 blockAnd the best type is selected

to be 8*8

For each 8*8 block in theMB, if is homogeneous

No

Yes

Perform RDO on 8*8, 8*4, 4*8 and 4*4 blocks and choose the best block subtype

All 8*8 blockhave selected

best mode

No

Yes

42

Determine the best mode between P8*8 type and the

best type from P16*16

Select the ultimate best mode among the intra-mode,

inter-mode and Skip mode

43

Result

Average -30.27 -0.0315 0.65

44

IEEE Conference Proceeding Volume 2, 24-27 Oct. 2004 Page(s):769 - 772 Vol.2

Feature-based intra-prediction mode decision for H.264

Changsung Kim, Hsuan-Huei Shih,C. C. Jay Kuo

45

“feature-based intra-Prediction mode decision for H.264” outline

• Quality calculation – RDO ( RD optimization ) method must try to all coding modes at

every block– STD,STAD , RD quality calculation

• Quality calculation filtering heuristic– STADSTADRD

• Flow chart of the proposed method– Intra-prediction mode decision schme

• Experiment results– Simulation condition – Computation evaluation– Quality evaluation

46

Quality calculation filtering heuristic• Two method decide quality

– Low computation but quality evaluation is not always good

• Calculate SAD or SATD

– High computation but quality evaluation ,quality evaluation is good

• Calculate SSD

• I(x,y): 原本的 pixel 值，• P(x,y): predict 的 pixel 值• D(x,y) :difference 的值• T{D(x,y)} : 對 D(x,y) 做 Hadamard transformation• SAD: summation of absolute difference• SATD: summation of transformation absolute

difference• Bias: 其他的 overhead• RD :rate distortion • SSD :the sum of the squared difference of s and c

– S : original signal– c: : reconstructed signal– 選擇這個 mode 的 overhead

• QP: quantization parameter– Decision by bit rate

Low complexity calculation

High complexity calculation

47

透過 SAD 還有 STAD 計算、利用 threshold 判斷是否再向下計算 RD cost 的 mode decision 的效果。利用設定 threshold 再和其比較，大則要算 RD cost 、反之即可 mode decision

Using (x,y)=(SAD,SATD)

statistically 93~95 % RDO mode fall in the of 3*3 window

48

Multi-stage 4*4 block mode decision降低計算量且維持品質的方法

• Filtering heuristic• Design early termination thresholds

– For hold quality not just calculate STAD

• 由觀察 SATD 比 STD 做 mode decision 更可能和 RDO 所做的 mode decision 相同、用此概念做 filtering ，降低計算量而且可以維持品質

• Threshold 值 T’th,Tth 由實驗值去決定

• 1. 先做 SATD 選出最小的二個 mode的 SATD 求出此 T’ ， if T< T’th,mode decision else step2

• 2 再做 SAD 計算挑出 mode 最小的 2個 mode 求出 T, if T<Tth ,mode decision else step3

• 3. 由 SAD 還有 SATD 的計算找出2~3 mode 再去做 RDO 計算做 mode decision

• RDO: rate distortion optimization

49

Simulation and calculation

• H.264 Jm7.3a codec • 以全部用 RD 做

mode decision 做比較– Quality 差異不大– 效能可提升 10~15 倍

50

IEEE Conference Proceeding Volume 2, 27-30 June 2004 Page(s):1131 - 1134 Vol.2

Fast mode decision for H. 264

Jeyun Lee, Byeungwoo Jeon

51

“Fast mode decision for H.264” outline

Flow chart of the proposed method in P frame encode– Skip mode decision at early stage in 16*16 block inter-

prediction– Heuristic method

• Selective intra mode decision• Decide to do intra-prediction or not

• Experiment results– Simulation condition – Performance evaluation

52

Selective intra mode decision(1)

• AE: average rate– The number of bits for

texture data without header– The average number of bits

consumed to encode the motion –compensated residual data under the best inter mode as an indicative of degree of temporal correlation

– Decide best inter-prediction– Calculate RD cost and

calculate AR

For Every

Macro-Block

Using 4:2:0 sub-sampling

YCb

8*8 block

53

Selective intra mode decision(2)

• Intra-prediction uses the pixels of already encoded neighboring block

• Boundary 和 block 邊緣的值相差不大則用此做預測效果好– 可以有較小的 residue 值

• ABE: average boundary error• 1 個 116*16 的 luma block 和二個 8*8 Cb,Cr block• ABE<AR 決定做 intra-predicton 且做出的結果和 inter-predicon 最

好的結果比

Case4 Left: available , Upper : available

54

Proposed algorithm

• 1.Inter Skip mode decision at early stage

– 先找出 16*16 的 block 是否可設為 skip mode

– Reference frame is just one previous one

– Transform coefficients are all quantized to zero

– 如果成立可以忽略其他的block size RD cost 計算

• 2. 計算 inter 下的 AR 值• 3. Selective intra mode decision

– Spatial – AR :average rate– ABE: average boundary– Omit all intra modes if AR<ABE– Else find out the best intra mode

and compares it with that of the best inter mode

55

Experiment: performance and quality• Computing RD cost 降 72%• Encoding time 降 30%• Simulation condition • S: the percent saving in number

of computing RD cost• • means the percent processing

time difference• BD-Bit rate• BD-PSNR• Other paper metrtic

56

Outline

Introduction

Fast mode decision


Reference

Frame Encoding

H.264 Frame

57

Stefan (CIF)

Foreman (QCIF)

News

(QCIF)

Slient

(QCIF)

Container

(QCIF)

Paris (CIF)

58

PSNR(dB)

[1]

[2] [3] [4]

Sequence

Foreman (QCIF) -0.06 -0.062 -0.13 -0.065

News (QCIF) -0.04 -0.065 0.000

Container (QCIF)

-0.3 -0.012 -0.067

Slient (QCIF) -0.04 -0.022 -0.028

Paris (CIF) -0.04 -0.040 -0.048

Mobile (CIF) -0.05 -0.005 -0.41 -0.003

Average -0.08 -0.0343 -0.27 -0.0351





59

Time(%)

[1]

[2] [3] [4]

Sequence

Foreman (QCIF) -48.82 -25.18 -11.316 -20.24

News (QCIF) -59.36 -42.62 -36.11

Container (QCIF)

-33.95 -36.25 -36.48

Slient (QCIF) -58.49 -45.16 -33.75

Paris (CIF) -63.03 -31.90 -34.69

Mobile (CIF) -60.76 -9.97 -13.805 -24.11

Average -54.06 -30.27 -12.56 -30.89





60

Outline

Introduction

Fast mode decision


Reference

Frame Encoding

H.264 Frame

61

Main Paper Reference

Chang, A. Wong, P.H.W. Yeung, Y.M. Au, O.C. Fast multi-block selection for H.264 video coding IEEE 2004 III- 817-20 Vol.3

Dajun Wu, Feng Pan, Keng Pang Lim, Si Wu, Zhengguo Li, Xiao Lin, Susanto Rahardja, Chi Chung Ko Fast Intermode Decision in H.264/AVC Video Coding IEEE Journal Volume 15, Issue 7, July 2005 Page(s):953 - 958

62

Main Paper Reference

Jeyun Lee, Byeungwoo Jeon Fast mode decision for H. 264IEEE Conference Proceeding Volume 2, 27-30 June 2004 Page(s):1131 - 1134 Vol.2

Changsung Kim, Hsuan-Huei Shih,C. C. Jay Kuo Feature-based intra-prediction mode decision for H.264IEEE Conference Proceeding Volume 2, 24-27 Oct. 2004 Page(s):769 - 772 Vol.2

63

Paper Reference

Ki-Hun Han, Yung-Lyul Lee Fast Macro-block Mode Determination to Reduce H.264 Complexity IEICE Issue 3 (March 2005) 800-804

Wei, Zhenyu; Ngan, King NgiA Fast Macro-block Mode Decision Algorithm for H.264IEEE Conference Proceeding 4-7 Dec. 2006 Page(s):772 - 775

64

Iain E. G. Richardson “H.264 and MPEG-4 video compressionvideo coding for next-generation multimedia,” The Robert Gordon University, Aberdeen, UK, WILEY 2003

Book Reference

Li. Drew 原著徐永煜‧莊舜清‧陳岳宏‧曾政龍‧賴柏伸‧鄭士賢‧謝宗儒‧呂宜玲‧沈揚智‧李卓育‧陳建榮‧蔡富評‧聶政邦譯傅心家審閱譯多媒體概論 (Fundamentals of Multimedia) 高立圖書有限公司

65

Report Reference

資工所 P76924194 蘇琬婷多媒體通訊期末報告 - 快速動作估計演算法 Fast Block Motion Estimation

曾紹崟 Shau-Yin Tseng 許志高 Ji-Gao Hsu多媒體壓縮標準 H.264 面面觀多媒體應用技術部 SoC Technical Journal

郭其昌H.264 先進視訊編解碼標準工研院電通所

66

E N D

T hanks for y Uo r lis nte

67

Appendix

1 h.264 fast mode decision multimedia systems team project 林展意 695410074 林志仁 695410063

Documents