next generation video coding‐ h.265/hevc and its extensions · apsipa asia-pacific signal and...
TRANSCRIPT
![Page 1: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/1.jpg)
APSIPAAsia-Pacific Signal and Information Processing Association
APSIPA Distinguished Lecture Series www.apsipa.org
NextGenerationVideoCoding‐H.265/HEVCanditsextensions
Oscar C. Au (PhD, Princeton Univ.)Dept. of Electronic and Computer EngineeringHong Kong University of Science and TechnologyClearwater Bay, Hong KongEmail: [email protected]
1
![Page 2: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/2.jpg)
OscarC.Au• BS,Toronto.MA/PhD,Princeton.Postdoc,Princeton.• Professor,HKUST.Director,MultimediaTechCenter.• SteeringCommittee,ICME/TMM.• IEEE/HKIEFellow.BoG,APSIPA.• BestPaperAwards:SiPS/PCM/MMSP/ICIP• AE ofjournals:TCSVT,TIP,TCAS1,JVCIR,JSPS,TSIP,JMM,JFI.• Chairof3TC:CASMSATC,SPSMMSPTC,APSIPAIVMTC.• MemberofTC:CASVSPS/DSP,SPSIVMSP/IFS,ComSoc MMC.• 400+papers.H‐index=29.100+patentsfiled.20granted.• 80+standardcontribution(MPEG/VCEG/JCTVC/AVS).
![Page 3: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/3.jpg)
Outline
• Introduction• Lossless Compression• Image Compression• Video Compression• Simulation Results• Conclusion
3
![Page 4: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/4.jpg)
Whatis“Image”?• “An image is worth a thousand words.”• Camera – “an optical instrument that records images that can be stored directly” (wikipedia)
• Black & white photo, Color photo• Analog camera (film), DC (card), DSLR, smart phone• Storage card: SD, xD, SDHC, USB, • auto‐focus (AF), auto‐exposure (AE), auto‐white‐balancing (AWB)• ISO, shutter speed, aperture, metering• Lens: wide‐angle, zoom, fish‐eye, vibration reduction/image stablizer,
4
![Page 5: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/5.jpg)
Whatis“Video”?• Video is sequence of images• Transmission: TV, CCTV, broadcasting TV, movie, camcorder, • Storage: VHS, betamax, SVHS, video8, LD, VCD, DVD, Blu‐Ray• PAL/SECAM (625 lines, 25 fps), NTSC (525 lines, 30 fps), movie• SDTV: 720x576 (PAL/SECAM), 720x480 (NTSC)• HDTV: 1920x1080 (1080i/p), 1280x720 (720i/p)• UHDTV: 7680x4320• digital cinema: 2048x1080 (2K), 4096x2160 (4K)• CCD/CMOS sensor, Bayer color filter array, demosaicking• AF/AE/AWB
5
![Page 6: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/6.jpg)
Whyvideocoding?Whypossible?
Why? • Raw video data is huge. • Channel capacity is limited• Video coding to reduce data rate
Why possible? • Signal representation is naturally redundant.• Some signal details irrelevant to observer.• Video coding by removing redundancy and irrelevancy• Redundancy removal is reversible (lossless)• Irrelevancy removal is irreversible. 6
![Page 7: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/7.jpg)
WhyCompression?Storage.
a) A 3000 line text file • 3000 line x 80 char/line x 1 byte/char = 240 KB
b) A 3‐minute song on CD• 3 x 60 sec x 44100 Hz x 2 byte/sample x 2 channel = 32 MB
c) A 150 minute NTSC movie (3.4 TB for HDTV, 60fps)• 150 x 60 sec x 30 fps x 480 x 720 pixel x 3 color = 273 GB
• Storage capacity:• SD card/Flash drive: 1/2/4GB, 8/64/128GB• iPod/hard‐disk: 8GB/16GB, 100GB/500GB/1TB• CD‐R: 650MB/700MB• DVD+‐R: ~4.7GB 7
![Page 8: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/8.jpg)
WhyCompression?Communication.
a) Speech over telephone• 8000 Hz x 8 bit/sample = 64 kbit/s
b) CD quality music/audio over network• 44.1 x 1000 Hz x 16 bit/sample x 2 channels = 1.4 Mbit/s
c) NTSC Digital Video over network (3.0 Gb/s for HDTV)• 30 fps x 480 x 720 pixel x 8 bit/pixel x 3 color = 250 Mbit/s
• Channel capacity• GPRS: 120/20 kb/s 3G (SS): 384/160 kb/s• HSPA: 28/42 Mb/s, 4G (OFDMA/MIMO): 1000/100 Mb/s• Bluetooth: 1‐3 Mb/s• WiFi: 12/54 Mb/s, WiMax: 1 Gb/s 8
![Page 9: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/9.jpg)
WhyPossible?
• Compression is performed to remove the redundancy inherent in the image/video• Spatial, temporal, statistical and psychovisual redundancies.
• Compression methods• Lossless: Redundancy removal is reversible;• Lossy: Redundancy removal is irreversible.
9
![Page 10: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/10.jpg)
Lossy coding:Distortionmeasure
Distortion measures• Sum of absolute error (SAD), Sum of square error (SSE)• Peak‐signal‐to‐noise ratio (PSNR)
Rate‐distortion trade‐off• Lower bit rate =>
higher distortion• Higher bit rate =>
lower distortion
10
![Page 11: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/11.jpg)
Lossy codingissues
• Bit rate (R)• Distortion (D)• Coding delay (important for real time systems)• Complexity (important for software/hardware cost)• Sensitivity to model deviation (robustness)• Sensitivity to channel errors (robustness, error concealment)
11
![Page 12: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/12.jpg)
Twocodingapproaches
• Waveform coding• Represent a signal such that after source coding, a good approximation of original waveform remains.
• E.g. DPCM, JPEG, MPEG‐1/2/4, H.261/3/4, JPEG2000, GIF, MP3• Typically distortion is due to quantization
• Parametric coding• Assume signal generated from model with parameters.• Encoder extract/encode model parameters from • Good approximation of waveform may not be obtained, but the result could be sound/image that resemble the original perceptually
• E.g. fractals, model‐based coding, LPC, CELP, 12
![Page 13: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/13.jpg)
Videocodingstandards?
• What is a standard? • International standards• National standards• Industry standards• Defacto standards
• Mandatory? Optional? MP3? JPEG?• Why standard?• The standard war ..
13
![Page 14: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/14.jpg)
StandardOrganizations
• Three main international organizations:• ITU‐T: International Telecommunication Union (Telegraphy section);
• ISO: International Standards Organization;• IEC: International Electrotechnical Commission.
• These organizations form different groups to develop various standards:• ITU‐T forms the VCEG (Video Coding Experts Group);• ISO and IEC form the MPEG (Moving Picture Experts Group);• ITU‐T and ISO form the JPEG (Joint Photographic Experts Group).
14
![Page 15: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/15.jpg)
ISO/IECwithsubgroupsforvideo
15
![Page 16: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/16.jpg)
StandardizationHistory
16
Video
Image
![Page 17: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/17.jpg)
Outline
• Introduction• Lossless Compression• Image Compression• Video Compression• Simulation Results• Conclusion
17
![Page 18: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/18.jpg)
Entropycoding
• Entropy is a measure of randomness• Entropy H is non‐negative.
• Theorem: Average codeword length S of a uniquely decodable binary code satisfies S>=H.
• Theorem: A prefix‐condition code for groups of M symbols exists with an average no. of bits per symbol S<H(q_i)+1/M
18
N
jjji ppqH
1log
![Page 19: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/19.jpg)
LosslessCompressionTechniques
• Huffman Coding• Arithmetic Coding• LZW Coding• Run‐length Coding• DPCM Coding• …
19
![Page 20: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/20.jpg)
HuffmanCoding
• Properties• Fixed‐length input, variable‐length output;• Sensitive to channel errors;• Encode/decode using table lookup;• Sensitive to changes in signal statistics.
• Applications• Modified Huffman coding in JPEG and H.261;• 2D and 3D Huffman coding in MPEG and H.263;
20
![Page 21: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/21.jpg)
21
Examples(onesymbol)
• P(A)=3/4, P(B)=3/16, P(C)=1/16• Average length per symbol (S) is
• S =1x3/4 + 2x3/16 + 2x1/16• = 1.25 bit/symbol
• where entropy (H) is defined as
• Theoretically, S>=H. S can be arbitrarily close to H by coding N symbols at a time. (meaningful 20% improvement possible)
012.1161log
161
163log
163
43log
43
H
ii ppH log
A
B
C
3/4
3/16
1/161/4
0
110
A 0B 10C 11
![Page 22: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/22.jpg)
22
HuffmanCodingExample
A 0.6B 0.2C 0.1D 0.05E 0.025F 0.0125G 0.00625H 0.00625
• Ave. codeword length = 1x0.6+2x0.2+3x0.1+4x0.05+5x0.025+6x0.0125+7x0.00625x2 = 1.7875 bit/symbol
• Entropy=H=1.7585
![Page 23: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/23.jpg)
23
Example(twosymbols)
AA 9/16AB 9/64AC 3/64BA 9/64BB 9/256BC 3/256CA 3/64CB 3/256CC 1/256
• Ave. length = 2.09 bit/codeword = 1.045 bit/symbol• Much closer to H=1.012 !!
• Code multiple symbols at a time => lower average length/symbol
![Page 24: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/24.jpg)
ArithmeticCoding
• Properties• Fixed‐length input, variable‐length output• Optimal minimum average code word length• Sensitive to channel errors;• Adapt to changing of signal statistics.
• Application• Image coding standards like JBIG, JPEG, JPEG 2000;• Video coding standards like H.263 and H.264/MPEG‐4 AVC
24
![Page 25: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/25.jpg)
25
ArithmeticCoding
• flexible, can use adaptive model of signal statistics• optimal, minimum average codeword length• slower than Huffman and Ziv‐Lempel• no random access• potential unbounded output delay• need to indicate end‐of‐file• poor error resistance; error propagates• compress better than Huffman in JPEG, more
complicated
![Page 26: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/26.jpg)
26
BasicalgorithmofAC
1. Initialize the current interval [L, H) to [0, 1).2. For each symbol of the file, do:
a) subdivide current interval into subintervals, one for each possible alphabet symbol. The size of a symbol’s subinterval is proportional to the estimated probability that the symbol will be the next symbol in the file, according to the model of the input.
b) select the subinterval corresponding to the symbol that actually occurs next in file, and make it the current interval.
3. Output enough bits to distinguish the final current interval from all other possible final intervals. 1log iP
![Page 27: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/27.jpg)
27
Example(Arithmeticcoding)
P(a)=0.5, P(b)=0.4, P(c=EOF)=0.1
Current Interval
Action Subinterval In-put
a b c
[0.0, 1.0) Subdivide [0.0, 0.5) [0.5. 0.9) [0.9, 1.0) a
[0.0, 0.5) Subdivide [0.0, 0.25) [0.25, 0.45) [0.45, 0.5) a
[0.0, 0.25) Subdivide [0.0, 0.125) [0.125, 0.225) [0.225, 0.25) b
[0.125, 0.225) Subdivide [0.125, 0.175) [0.175, 0.215) [0.215, 0.225) c
[0.215, 0.225) End-of-file
![Page 28: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/28.jpg)
28
Example(Arithmeticcoding)
• To encode the interval [0.215, 0.225), • need a number with enough precision to specify the interval.
• Width of interval=0.01. • Half width=0.005.• Need 8 bits of precision to specify half‐width:
• 2‐8 =0.0039<0.005, 2‐7 =0.0078>0.005
![Page 29: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/29.jpg)
29
Example(Arithmeticcoding)
Target interval: [0.215, 0.225)0 0.5 [0.0,0.5) / [0.5, 1.0) 0 0.25 [0.0, 0.25) / [0.25, 0.5)1 0.125 [0.0, 0.125) / [0.125, 0.25)1 0.0625 [0.125, 0.1875) / [0.1875, 0.25)1 0.03125 [0.1875, 0.21875) / [0.21875, 0.25)0 0.015625 [0.21875, 0.234375) / [0.234375, 0.25)0 0.0078125 [0.21875, 0.22656) / [0.22656, 0.234375)0 0.00390625 [0.21875, 0.2227125) / [0.2227125,
0.22656) (completely within=> stop)Codeword = 00111000 = 0.21875
![Page 30: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/30.jpg)
30
ModifiedArithmeticCoding
a) if new subinterval is not entirely within one of these intervals: [0, 1/2), [1/4, 3/4), or [1/2, 1), then stop iterating and return.
b) if new subinterval lies entirely within [0, 1/2), then output 0 and any 1’s left over from previous “follow”symbols (one 1 for each “follow” symbol), and double the size of the interval [0, 1/2) expanding from left boundary (0) towards the right.
![Page 31: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/31.jpg)
31
ModifiedArithmeticCoding
c) if new subinterval lies entirely within [1/2, 1), then output 1 and any 0’s left over from previous “follow” symbols (one 0 for each “follow”), and double the size of current interval by expanding it from right boundary (1) towards the left. For example: interval [0.72, 0.8) will become [0.44, 0.6), because 2*dist(0.72,1)=0.56, 1‐0.56=0.44 => left point; new width=2*(0.8‐0.72)=0.16; 0.44+0.16=0.6 => right point.
![Page 32: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/32.jpg)
32
ModifiedArithmeticCoding
d) if new subinterval lies entirely within [1/4, 3/4), apply a “follow” symbol and double the size of current interval by expanding it in both directions away from midpoint (1/2). For example: interval [0.48, 0.588) will become [0.46, 0.676), because 2*dist(0.48,0.5)=0.04, 0.5‐0.04=0.46 => left point; 2*dist(0.588,0.5)=0.176, 0.5+0.176=0.676 => right point.
e) goto (a).
![Page 33: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/33.jpg)
33
Example(ModifiedAC)
P(a)=0.4, P(b)=0.2, P(c)=0.4. Assume a=EOF.
Current Interval
Action Subinterval In-put
a b c
[0.0, 1.0) Subdivide [0.0, 0.4) [0.4. 0.6) [0.6, 1.0) b
[0.4, 0.6) Expand (Follow)
[0.3, 0.7) Expand (Follow)
[0.1, 0.9) Subdivide [0.1, 0.42) [0.42, 0.58) [0.58, 0.9) a
[0.1, 0.42) Expand (Output 0, 1, 1)
[0.2, 0.84) End (Output 01 or 10)*
![Page 34: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/34.jpg)
34
Example(ModifiedAC)
• Therefore, • codeword = 01110 or 01101.• cw(01110)=0.25+0.125+0.0625=0.4375• cw(01101)=0.25+0.125+0.003125=0.40625
* Need to output 10 or 01 for input x, where P(x1 )=0.2, P(x2 )=0.64, P(x3 )=0.16
![Page 35: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/35.jpg)
35
Example(oldmethod‐ AC)
• P(a)=0.4, P(b)=0.2, P(c)=0.4
Current Interval
Action Subinterval In-puta b c
[0.0, 1.0) Subdivide [0.0, 0.4) [0.4. 0.6) [0.6, 1.0) b
[0.4, 0.6) Subdivide [0.4, 0.48) [0.48, 0.52) [0.52, 0.60) a
[0.4, 0.48) End-of-File
![Page 36: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/36.jpg)
36
Example(oldmethod–AC)
Target interval: [0.4, 0.48)0 0.5 [0.0, 0.5) / [0.5, 1.0)1 0.25 [0.0, 0.25) / [0.25, 0.5)1 0.125 [0.25, 0.375) / [0.375, 0.5)1 0.0625 [0.375, 0.4375) / [0.4375, 0.5)0 0.03125 [0.4375, 0.46875) / [0.46875, 0.5)
(entirely within [0.4, 0.48))
• codeword = 01110 (can also be 01101)• Same as modified AC.
![Page 37: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/37.jpg)
37
Example(ModifiedACDecode)
P(a)=0.4, P(b)=0.2, P(c)=0.4. Assume a=EOF. cw(01110)=0.4375Current Interval
Action Subinterval In-put
a b c
[0.0, 1.0) Subdivide [0.0, 0.4) [0.4. 0.6) [0.6, 1.0) b
[0.4, 0.6) Expand (Follow) cw=0.5-2*(0.5-0.4375)=0.375
[0.3, 0.7) Expand (Follow) cw=0.5-2*(0.5-0.375)=0.25
[0.1, 0.9) Subdivide [0.1, 0.42) [0.42, 0.58) [0.58, 0.9) a
Stop decoding because a=end-of-file. (cw=codeword)
![Page 38: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/38.jpg)
38
Example(ModifiedACDecode)P(a)=0.4, P(b)=0.2, P(c)=0.4. Assume a=EOF. cw(01101)=0.40625
Current Interval
Action Subinterval In-put
a b c
[0.0, 1.0) Subdivide [0.0, 0.4) [0.4. 0.6) [0.6, 1.0) b
[0.4, 0.6) Expand (Follow) cw=0.5-2*(0.5-0.40625)=0.3125
[0.3, 0.7) Expand (Follow) cw=0.5-2*(0.5-0.3125)=0.125
[0.1, 0.9) Subdivide [0.1, 0.42) [0.42, 0.58) [0.58, 0.9) a
Stop decoding because a=end-of-file. (cw=codeword)
![Page 39: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/39.jpg)
Outline
• Introduction• Lossless Compression• Image Compression• Video Compression• Simulation Results• Conclusion
39
![Page 40: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/40.jpg)
ImageCompression
• Applications: Internet, digital photography, medical imaging, remote sensing, surveillance, facsimile, etc.
• Standardization of image compression• JPEG became the International Standards in 1992.• JPEG 2000 was finalized in 2002.• General Structure of Image Coding Standards:
40
![Page 41: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/41.jpg)
ImageStandards
• JPEG (lossy and lossless): ITU‐T T.81, ISO/IEC 10918‐1• JPEG extensions: ITU‐T T.84• JPEG‐LS (lossless, improved): ITU‐T T.87, ISO/IEC 14495‐1• JBIG (lossless, bi‐level pictures, fax): ITU‐T T.82, ISO/IEC 11544• JBIG2 (bi‐level pictures): ITU‐T T.88, ISO/IEC 14492• JPEG 2000: ITU‐T T.800, ISO/IEC 15444‐1• JPEG 2000 extensions: ITU‐T T.801• JPEG XR (formerly called HD Photo prior to standardization) : ITU‐T T.832, ISO/IEC 29199‐2
41
![Page 42: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/42.jpg)
JPEG
• History• Starting from the mid‐1980s;• Designed for compressing grayscale and color still images;• Became an international standard in 1992.
• Application• The JPEG coding standard still serves as the most widely used compression algorithm today.
• Its application can be found in diverse storage and transmission domains, such as the Internet, digital professional and consumer photography, and video.
42
![Page 43: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/43.jpg)
43
GoalsofJPEGFour modes of operation:
a) Sequential encoding: image encoded in single left‐right, top‐bottom scan (include baseline sequential codec)
b) Progressive encoding: image encoded in multiple scans encoding with progressively refined details, for image reconstruction in multiple coarse‐to‐clear passes
c) Lossless encoding: image encoded to guarantee exact recovery of every source image sample value (though compress ratio low compared to lossy modes)
d) Hierarchical encoding: image encoded at multiple resolutions s.t. lower resolution image accessible without decompressing higher or even full resolution
![Page 44: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/44.jpg)
44
SequentialModeofJPEG
• 3 major steps: DCT, quantization, entropy coding
Image Source FDCT Quantizer Entropy
Encoder
Compressed Image
Dequantizer Entropy Decoder
IDCTReconstructed
Image
Table
![Page 45: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/45.jpg)
45
DiscreteCosineTransform(DCT)
where , for u=0 and v=0,
, otherwise.
7
0
7
041
7
0
7
041
1612cos
1612cos,
1612cos
1612cos,,
u v
x y
vyuxvCuCyxf
vyuxyxfvCuCvuF
This image cannot currently be displayed.
2
1,2
1 vCuC
1,1 vCuC
![Page 46: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/46.jpg)
46
Quantization
• quantization is a many‐to‐one mapping and thus lossy –the principal source of distortion in DCT‐based encoder
• quantization defined as division of each DCT coefficients by its corresponding quantizer step size followed by rounding to nearest integer (normalized by the quantizer step size):
vuQvuFroundvuFQ
,,,
![Page 47: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/47.jpg)
47
EntropyCoding
• each quantized DC coefficient encoded as difference from DC term of previous block in encoding order;
• this special treatment is worthwhile as DC coefficients frequently contain a significant fraction of total image energy.
![Page 48: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/48.jpg)
48
a) Input image
158158158163161161162162157157157162163161162162157157157160161161161161155155155162162161160159159159159160160162161159156156156158163160155150156156156156156153151144155155155155153149144139
00000000000000000000000000000000000000010000001100000012000001079
0000000000000000000000000000000000000001400000013140000001224000001001264
160161163164164163161160157158160161162161160160154156158160161161161160155156158161162162162162157158160162163163162162158159160161161159158157156156157156155153150149154154153152150147144142
9910310011298959272101120121103877864499211310481645535247710310968563722186280875129221714566957402416131455605826191412126151402416101116
b) Forward DCT coefficients c) Quantization table
d) quantized DCT coefficients e) dequantized DCT coefficients f) Reconstructed image
DCT/QuantizationExample
4.06.02.19.18.18.36.16.28.01.17.15.05.13.04.03.10.10.15.18.03.06.12.08.1
3.16.07.01.06.15.18.06.03.001.09.05.12.09.11.71.06.09.02.05.16.13.99.102.14.01.09.22.32.65.176.22
3.17.27.11.22.51.120.16.1259
![Page 49: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/49.jpg)
49
JPEGLosslessMode
1. JPEG discovered that a DCT‐based lossless mode was difficult to define as a practical standard against which encoders and decoders could be independently implemented, without placing severe constraints on both encoder and decoder implementations. • Instead, JPEG has chosen a simple predictive method
which is wholly independent of DCT processing.
![Page 50: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/50.jpg)
50
JPEGLosslessMode
2. The predictive method produces results which, in light of its simplicity, are surprisingly close to the state of the art for loss‐less continuous tone compression.
• Lossless codecs typically produce around 2:1 compression for color images with moderately complex scenes.
![Page 51: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/51.jpg)
51
JPEGLosslessMode
3. A predictor combines values of up to 3 neighbouringsamples (A, B, C) to form a prediction of the sample indicated by X, and the difference is encoded losslesslyby either Huffman or arithmetic coding. • The encoder can use any source image precision from
2 to 16 bits per sample, and can use any of the predictors except selection value 0.
![Page 52: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/52.jpg)
52
![Page 53: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/53.jpg)
53
JPEGProgressiveMode
• Each image component is encoded in multiple scans rather than in a single scan.
• The first scan encodes a rough but recognizable version of the image which can be transmitted quickly in comparison to the total transmission time
• Subsequent scans refined the image progressively to finally reach the level of picture quality that was established by the quantization tables.
![Page 54: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/54.jpg)
54
JPEGProgressiveMode
• To achieve this requires the addition of an image‐sized buffer memory at the output of the quantizer before the input to the entropy encoder.
• The buffer memory must be sufficiently large to store the image as quantized DCT coefficients each of which is 3 bits larger in size than the source image samples.
![Page 55: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/55.jpg)
55
JPEGProgressiveMode
• There are two complementary method by which a block of quantized DCT coefficients may be partially encoded.
• In the first method, only N low frequency DCT coefficients need to be encoded. This is called spectral selection. The other high frequency coefficients are sent in succeeding scans.
![Page 56: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/56.jpg)
56
JPEGProgressiveMode
• In the second method, the coefficients need not be encoded to their full (quantized) accuracy in a given scan
• Initially the N most significant bits are encoded. • In subsequent scans, the less significant bits can be
encoded. • This is called successive approximation.
![Page 57: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/57.jpg)
57
![Page 58: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/58.jpg)
58
JPEGHierarchicalMode(pyramidalcoding)
• image is filtered and down‐sampled (decimated) by the desired number of multiple of 2 in each dimension.
• Encode the reduced‐size image using one of the sequential DCT, progressive DCT or lossless encoders. (step b)
![Page 59: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/59.jpg)
59
JPEGHierarchicalMode
• Decode this reduced sized image, interpolate and up‐sample by a factor of 2 horizontally and vertically, using an identical interpolation filter which the receiver must also use.
• Use this up‐sampled image as a prediction of the original at this resolution and encode the difference image using one of the sequential DCT, progressive DCT or lossless encoder. (step d)
• repeat until full resolution of image is encoded.
![Page 60: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/60.jpg)
60
JPEGHierarchicalMode
• encoding in steps (b) and (d) may be done using only DCT‐based processes, only lossless processes or DCT‐based processes with a final lossless process for each component.
• useful in applications in which a very high resolution image must be accessed by a lower‐resolution device, which does not have the buffer capacity to reconstruct the image at its full resolution and then scale it down for lower‐resolution display.
![Page 61: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/61.jpg)
61
JPEGHierarchicalMode
![Page 62: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/62.jpg)
JPEG• Key technology
• Lossless coding scheme: predictive coding method using neighboring pixel values
• Lossy mode: the well‐known DCT, serves as the base‐line of JPEG• Block artifacts
62
Block based DCT
![Page 63: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/63.jpg)
JPEG
63
Cameraman, 8 bits/pixel
Cameraman, 0.5 bits/pixel Cameraman, 0.8 bits/pixel
Cameraman, 0.15 bits/pixel
![Page 64: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/64.jpg)
JPEG
64
Lena, 8 bits/pixel
Lena, 0.5 bits/pixel Lena, 0.8 bits/pixel
Lena, 0.15 bits/pixel
![Page 65: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/65.jpg)
JPEG2000
• History• Started in 1998;• Aimed at improving the quality and capability of JPEG;• Approved as the international standards in 2002
• Application• Remote sensing, color fax, printing, scanning, digital photography, medical imagery, digital libraries/archives, Internet, e‐commerce, etc.
65
![Page 66: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/66.jpg)
JPEG2000Tiling
• Each color component divided into rectangular tiles (E.g. 64x64 non‐overlapping blocks)
• Each tile encoded independently. For each tile, apply wavelet transform, quantization, form precinct and code blocks, EBCOT, AC.
• Bad: Slightly lower compression efficiency with tiling than without.
• Good: lower memory requirement, random access • Arithmetic coding used: MQ‐coder (also used in JBIG2, similar to QM‐coder in JPEG)
![Page 67: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/67.jpg)
Block Diagram of JPEG2000(Encoder)
![Page 68: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/68.jpg)
• Preprocessing• subtract input value (unsigned) by 128
• Forward Inter‐component Transform• RGB to YUV (reversible) or YCbCr (irreversible)• Decorrelation, possible 2:1 subsampling in UV or CbCr• Useful data reduction method in JPEG, but not so much in JPEG2000 (can discard HL, LH, HH)
Reversible Irreversible
![Page 69: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/69.jpg)
• Forward Intra‐component Transform• Discrete Wavelet Transform (DWT) of non‐overlapping tiles • Daubechies 9‐tap/7‐tap wavelet filter for irreversible/ lossytransform
• 5‐tap/3‐tap for reversible/lossless transform (allow repetitive encoding/decdding without additional loss)
• implemented by convolution or lifting using periodic symmetric extension to handle boundary effect
![Page 70: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/70.jpg)
Imagewavelettransform
![Page 71: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/71.jpg)
• Quantization• Step size calculated from rate control
• Precinct partitioning• 3 spatially consistent rectangles grouped to form a packet partition location or a precinct
• Each precinct divided into non‐overlapping code‐blocks, scanned with a particular order
![Page 72: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/72.jpg)
• Tier‐1 Encoder• Bit plane of coefficients within code‐block entropy encoded using embedded block coding with optimal truncation (EBCOT)
• EBCOT: Arithmetic Coding (AC) followed by post‐compression rate‐distortion (PCRD) optimized truncation
• certain ROI can be coded at higher quality• Tier‐2 Encoder
• Referred to as packetization.
![Page 73: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/73.jpg)
EBCOT(EmbeddedBlockCodingwithOptimalTruncation)
![Page 74: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/74.jpg)
EBC(EmbeddedBlockCoding)
• Each bit plane of a code‐block is encoded by 3 coding passes• Significant pass (AC or direct)• Refinement pass (AC or direct)• Cleanup pass (AC)
• Exception: highest bit plane encoded with Cleanup pass only
![Page 75: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/75.jpg)
![Page 76: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/76.jpg)
![Page 77: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/77.jpg)
JPEG2000
77
Cameraman, 8 bits/pixel
Cameraman, 0.5 bits/pixel Cameraman, 0.8 bits/pixel
Cameraman, 0.15 bits/pixel
![Page 78: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/78.jpg)
JPEG2000
78
Lena, 8 bits/pixel
Lena, 0.5 bits/pixel Lena, 0.8 bits/pixel
Lena, 0.15 bits/pixel
![Page 79: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/79.jpg)
ComparisonofJPEGandJPEG2000
• Compared with JPEG
79
Advantages Disadvantages
• Better compression performance at low bit rate
• Spatial and quality scalability• No blocking artifacts
• No substantial improvement at medium and high bit rate
• More complex than JPEG
![Page 80: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/80.jpg)
JPEGDerivedIndustryStandards
• JFIF (JPEG File Interchange Format, XXX.jpg);• JTIP (JPEG Tiled, Pyramid Format);• TIFF (Tagged Image File Format);• SPIFF (Still Picture Interchange File Format, JPEG Part 3);• FlashPix
• Developed by Kodak, Hewlett‐Packard, Microsoft (1996);• Widely used in digital still cameras.
80
![Page 81: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/81.jpg)
Outline
• Introduction• Lossless Compression• Image Compression• Video Compression• Simulation Results• Conclusion
81
![Page 82: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/82.jpg)
VideoCompression
• Applications• DVD, digital TV, HDTV, video telephony, and teleconferencing
• Standardization process
82
VCEG MPEG
H.263
H.261
H.264/MPEG‐4 AVC
H.262/MPEG‐2
MPEG‐4
MPEG‐1
HEVC
![Page 83: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/83.jpg)
83
Twotypesofapplications1. Asymmetric applications: infrequent use of compressor
(complication) but frequent use of decompressor (simple).• e.g. electronic publishing, education and training, travel
guidance, videotext, point‐of‐sale, video games, entertainment (movies), video‐on‐demand (VOD), etc.
2. Symmetric applications: essentially equal use of compressor and decompressor.
• e.g. video mail, videophone, video‐conferencing, generation of material for playback only applications, etc.
* MPEG is for asymmetric applications. H.261/263 are for symmetric applications.
![Page 84: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/84.jpg)
84
Requirementsforcompressedvideoondigitalstoragemedia1. Random access : any frame decodable in less than 0.5 second. Need
access point, i.e. segments of information coded only with reference to themselves.
2. Fast Forward/Reverse Searches : possible to scan a compressed bit stream, display selected frames to obtain a fast forward or fast reverse effect. (a more demanding form of random access)
3. Reverse Playback : impossible without an extreme additional cost in memory.
4. Audio‐video synchronization : need mechanism to resynchronize audio and video should they be derived from slightly different clocks.
![Page 85: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/85.jpg)
85
Requirementsforcompressedvideoondigitalstoragemedia5. Robustness to errors : avoid catastrophic behavior in the presence of
errors in storage media or transmission channels.
6. Coding/Decoding delay : video‐conferencing applications need to maintain total system delay under 150ms in order to maintain conversation. Publishing applications can allow long encoding delay, but short interactive threshold decoding delay of 1 sec.
7. Editability : editing units of a short time duration and coded only with reference to themselves needed for editability in compressed form.
8. Format flexibility : allow for a large flexibility of format in terms of raster size (width, height) and frame rate.
9. Cost tradeoff : decoder implementable in small number of chips.
![Page 86: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/86.jpg)
VideoCompression
• General structure of the video codec• Intra‐frame: exploit the spatial correlation to predict the signal• Inter‐frame: exploits the temporal correlation to further reduce the redundancies.
86
![Page 87: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/87.jpg)
87
TemporalRedundancyReductionThree types of pictures :
1) Intra‐pictures(I): provide access point for random access, but only moderate compression(~10:1)
2) Predictive Pictures(P): coded with reference to a past picture(I or P); used as a reference for future P pictures; higher compression (~20:1)
3) Bidirectional predicted pictures(B): provide highest amount of compression (~40:1) but require both a past and future reference for prediction; not used as a reference.
![Page 88: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/88.jpg)
MPEG‐1/2/4VideoCompression• Raw video is huge in size for transmission/storage (e.g. 150 GB for a
120 minute movie). Need compression.
• Compression achieved by exploiting 4 characteristics:
• statistical redundancy, perceptual irrelevancy, spatial redundancy ‐
used in image compression
• temporal redundancy (Motion Estimation) unique and most
important for video compression
![Page 89: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/89.jpg)
Frame 21 of tennis sequenceFrame 20 of tennis sequence
Frame difference without motion estimation with motion estimation
TemporalRedundancyReduction
![Page 90: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/90.jpg)
90
AdvantageofDPCMoverPCM
![Page 91: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/91.jpg)
91
Block‐basedMotionEstimationGoal : To establish blockwise correspondence between two frames
sec/108.5sec
30396961512 9 opsxframeframeblock
blocksearch
searchops
![Page 92: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/92.jpg)
MEtoocomputationintensive!
Computational distribution in MPEG-4 encoder
![Page 93: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/93.jpg)
93
ErrorSurface
• Multi‐model
• Easily trapped in local minimum
![Page 94: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/94.jpg)
94
ErrorSurface
![Page 95: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/95.jpg)
FastMotionEstimation
Full Search (41.79dB, slow) PMVFAST (41.85dB, speedup=1125)
“Foreman” test sequence, medium bit rate 512 kbit/s, medium resolution CIF, 15 fps, SA32
![Page 96: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/96.jpg)
96
AdaptiveQuantization/RateControl
cij = ijth DCT coefficient of an 8x8 block,cijq = quantized cij,Qij = ijth entry in the quantization table,Qp = extra quantization step.
• For a given frame, Qij or the quantization table is fixed. But Qp can be changed on a block‐by‐block basis.
pij
ijqij QQ
croundC 8
![Page 97: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/97.jpg)
97
Errorconcealment
a) Two types of I‐frame errors: loss of HP cells resulting in loss of headers, DC and low order AC
coefficients, which results in serious degradation of video signal.
loss of SP cells resulting in loss of higher order AC coefficients, which results in loss of detail in the blocks being reconstructed.
b) Three types of P and B‐frame errors loss of an HP cell resulting in total loss of information pertaining
to image region represented by the coded bits loss of SP cell containing motion information loss of SP cell containing no motion information (ignore)
![Page 98: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/98.jpg)
98
c) DC synthesis (I‐frames) DC values are synthesized by bilinear interpolation from the
nearest blocks in the top and bottom macroblocks. Since a cell loss generally causes loss of data in a series of macroblocks, the horizontal neighbors are not used for synthesis.
d) AC synthesis (I‐frames) In order to reduce the effect of distinct block boundaries,
some of the low order AC coefficients have to be synthesized. The five lowest order AC coefficients (in zigzag order) are synthesized using some method.
Errorconcealment
![Page 99: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/99.jpg)
99
e) Prediction mode (P and B‐frames) For P‐frame, if top or bottom macroblock is coded as forward
predicted, the damaged macroblock is assigned forward mode.
If both neighbors are coded in intra mode, the damaged macroblock is assigned intra mode. Similar for B‐frames.
f) Motion vectors (P and B‐frames) If both top and bottom vectors are defined, then the average
of motion vectors is used for the synthesized macroblock. If only one of the vertical neighbors has valid motion vector(s)
defined, then this vector(s) is used. If no motion available, the macroblock is synthesized by intra
frame technique as in I‐frames.
Errorconcealment
![Page 100: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/100.jpg)
VideoStandards
• CCIR 601 (ITU‐T)• H.261 (ITU‐T)• H.263 (ITU‐T)• H.264/MPEG‐4 AVC (ITU‐T + ISO)• H.265• M‐JPEG (ISO)• MPEG‐1 (ISO)• MPEG‐2 (ITU‐T + ISO)• MPEG‐4 (ISO)• VC‐1 (SMPTE)• AVS 100
![Page 101: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/101.jpg)
H.261
• Target• International standard for ISDN picture phones and for video conferencing systems (1990)
• Image format: CIF (352x288) or QCIF (176*144), frame rate 7.5 ... 30 fps• Bit‐rate: multiple of 64 kbps, typically 128 kbps including audio.• Picture quality: for 128 kbps acceptable with limited motion in the scene
• Basic Properties• The very first one of the H.26x standards in the domain of VCEG• I‐frame + P‐frame• I‐frame: basically the same as JPEG• P‐frame: motion estimation/compensation + JPEG• Motion estimation/compensation.• Loop filter.
101
![Page 102: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/102.jpg)
H.261
• Ratified in November 1988 • The first widespread practical success• Still in use, mostly as a backward‐compatibility feature overtaken by H.263
• Designed to operate at video bit rates between 40kbit/s and 2Mbit/s
• Still used as backward‐compatibility mode in some video conferencing systems and for some types of internet video
102
![Page 103: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/103.jpg)
MPEG‐1
• Target• Target bit‐rate about 1.5 Mbit/s • Typical image format CIF, no interlace• Frame rate 24 ... 30 fps• Main application: video storage for multimedia (e.g., on CD‐ROM)
• Basic Properties• Designed for CD‐ROM application by MPEG• I‐frame + P‐frame + B‐frame• P‐frame: unidirectional motion compensation• B‐frame: bi‐directional motion compensation• Half‐pixel ME 103
![Page 104: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/104.jpg)
MPEG‐1
• Final standard was approved in November 1992• Use was fairly widespread, but mostly overtaken by MPEG‐2• Can provide approximately VHS quality between 1‐2 Mbps• Application:
• MP3• VCD• DVD• CD‐ROM• DVB (Digital Video Broadcasting)• DAB (Digital Audio Broadcasting)
104
![Page 105: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/105.jpg)
H.262/MPEG‐2
• Target• Extension for interlace, optimized for TV resolution (NTSC: 704 x 480 Pixel)
• Image quality similar to NTSC, PAL, SECAM at 4 ‐8 Mbit/s• HDTV at 20 Mbit/s
• Basic Information• Meet the need of entertainment TV for transmission media.• Frame/field picture;• The scalability tools as functionality tools were first defined.
105
![Page 106: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/106.jpg)
106
SNRscalability
• EI and EP have same resolution as I and P
• EI predicted from I
• EP predicted from P or previous EI or EP
![Page 107: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/107.jpg)
107
Spatialscalability
• Similar to SNR scalability except that I and P are half-size of EI and EP
• Prediction from I and P involves enlarging them by a factor of 2
![Page 108: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/108.jpg)
108
Multilayerscalability
![Page 109: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/109.jpg)
H.262/MPEG‐2
• First video compression codec released in 1995• Now in wide use for DVD standard and DTV
• The most commonly used video coding standard• Range of use normally 2‐20 Mbps• Application
• DVD (NTSC & PAL)• HDV (High‐definition video on DV cassette tape)• MOD and TOD (Digital tapeless camcorders)• VOD (Video On Demand)• ATSC• XDCAM• ISDB‐T• DVB
109
![Page 110: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/110.jpg)
H.263
• Target• International standard for picture phones over analog subscriber lines (1995)
• Image format usually CIF, QCIF or Sub‐QCIF, frame rate usually below 10 fps
• Bit‐rate: arbitrary, typically 20 kbps for PSTN• Picture quality: with new options as good as H.261 (at half rate)• Widely used as compression engine for Internet video streaming
• Basic Properties• Designed for video conferencing at a low bit rate in the mobile wireless communication scenario.
• 8x8 motion compensation;• 8x8 block DCT
110
![Page 111: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/111.jpg)
H.263
• Ratification in March 1996• Application
• Widely used as compression engine for Internet video streaming (YouTube, Google Video, Myspace)
• Also found use in H.323 (RTP/IP based video conferencing), RTSP , SIP (IP‐based videoconferencing) solutions.
• Low bit‐rate compressed format• MMS (mobile multimedia message)• Video telephony
111
![Page 112: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/112.jpg)
MPEG‐4
• Target• Object based coding;• Wide‐range of applications, with choices of interactivity, scalability, error resilience, etc.
• Basic Properties• A lot of new coding tools:
• Interactive graphics;• Object and shape coding;• Scalable video coding.
• Robust transmission;• Can encode mixed media data. 112
![Page 113: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/113.jpg)
MPEG4
• Introduced in late 1998, still a developing standard• Efficient across a variety of bit‐rates ranging from a few kbps to tens of mbps.
• Support variety of bitrates• Application
• Internet video streaming• Wireless video• Studio editing• Video database• Interactive video• Video conferencing/e‐mail• Games• Education
113
![Page 114: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/114.jpg)
H.264/MPEG‐4AVC
• Target• Reduce half of the bit‐rate compared to MPEG‐2, H.263 or MPEG‐4 Part 2
• Basic Properties• Finalized in 2003• Variable block‐size motion compensation• Quarter‐pixel precision for motion compensation• Powerful entropy coding techniques:
• Context‐Adaptive Binary Arithmetic Coding (CABAC)• Context‐Adaptive Variable‐Length Coding (CAVLC)
• Scalable Video Coding (SVC)• Multiview Video Coding (MVC)• Integer‐based transform
114
![Page 115: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/115.jpg)
H.264/MPEG‐4AVC
• First drafting work was completed in May 2003• Broad Application
• Blu‐ray Discs• Streaming internet source (Vimeo, YouTube, iTunes Store)• Web software (Adobe Flash Player, Microsoft Silverlight)• HDTV broadcasts (ATSC, DVB‐T, DVB‐C DVB‐S)• CCTV (Closed Circuit TV) and Video surveillance
115
![Page 116: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/116.jpg)
116
AdvancedTechniquesofH.264
• New Intra Prediction Method• Advance Inter Prediction Method
• Multiple Reference Frames• Multiple Block Size ME (e.g. 16x8, 8x8, 4x4)• Better ME accuracy (1/4 pixel)
• 4x4 integer transform• In‐Loop Deblocking Filter• Better entropy encoding (CABAC)
![Page 117: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/117.jpg)
117
4x4Intra‐prediction(luma)
![Page 118: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/118.jpg)
118
4x4IntraPredictionModes
0
1
43
57
8
6
![Page 119: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/119.jpg)
119
![Page 120: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/120.jpg)
120
16x16IntraPrediction(luma)
Mode 0 (Vertical) Mode 1 (Horizontal)
Mode 2 (DC) Mode 3 (Plane)
Complicated!
![Page 121: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/121.jpg)
121
![Page 122: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/122.jpg)
122
8x8intra‐prediction(chroma)
![Page 123: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/123.jpg)
123
Codingintra‐predictionmode
![Page 124: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/124.jpg)
124
ME:Multiplereferenceframes
Frame NN-1N-2N-3N-4N-5
![Page 125: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/125.jpg)
125
ME:Multipleblocksize
![Page 126: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/126.jpg)
126
ME:Multipleblocksize
• Totally 7 block mode• Motion estimation• 4x4 integer transform• Advantage
• Save bits (~ 15%, 7 modes)• Disadvantage
• Computation increase
![Page 127: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/127.jpg)
127
Transform
• 4x4 array for luma DC coefficients
• 2x2 array for chroma DC coefficients
•
![Page 128: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/128.jpg)
128
4x4DCT(withHadamard transformforDC)
![Page 129: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/129.jpg)
129
4x4IntegerDCT
![Page 130: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/130.jpg)
130
Fast4x4DCT
![Page 131: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/131.jpg)
131
QuantizationofDCTcoeff.
![Page 132: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/132.jpg)
132
QPandQStep
![Page 133: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/133.jpg)
133
QuantizationinH.264referencesoftware
![Page 134: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/134.jpg)
HEVCvideocompression
• Target• Target at HDTV or ultra‐HDTV compression, with substantially improved coding efficiency compared to H.264/AVC, i.e. 50% bit rate reduction
• Focus on the increasing need for parallel processing
• Basic Properties• Finalized in 2013• Large Block structure• Quad‐tree based block partition• Asymmetric mode partition• Sample adaptive offset• Tile
134
![Page 135: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/135.jpg)
HEVCTimeline
• 2010.01: Formal joint CfP from VCEG and MPEG
• 2010.04: JCT‐VC team, HEVC joint project, full proposals
• 2010.07: TMuC SW ready, tool experiments (TE)
• 2010.10: HM SW ready, core experiments (CE)
• 2011.02: WD
• 2012.02: CD
• 2012.07: DIS
• 2012.10: SoDIS (Study of Draft International Standard)
• 2013.01: FDIS
• Mid‐2013 – mid 2014: Extensions/amendments, such as Scalable, 3D, 4:x:x, bit‐depth > 10, color…
135
![Page 136: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/136.jpg)
HEVCInvolvedCompanies
136
![Page 137: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/137.jpg)
MajorApplicationsSummaryField Bandwidth Video Standards
Digital Television Broadcasting
2 ... 6 Mbps (10 ... 20 Mbps for HD)
• H.262/MPEG‐2• H.264/MPEG‐4 AVC
Blue‐ray DVD video 6 ... 8 Mbps • H.262/MPEG‐2• H.264/MPEG‐4 AVC
Internet video streaming 20 ... 200 kbps • H.263• H.264/MPEG‐4 AVC
Videoconferencing, Videotelephony
20 ... 320 kbps • H.261• H.263• H.264/MPEG‐4 AVC
Video over 3G wireless 20 ... 200 kbps • H.263• H.264/MPEG‐4 AVC
137
![Page 138: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/138.jpg)
AVSvideocompression
• AVS: Audio Video Standard• Founded by the China Audio Video Coding Standard Working Group in June 2002.
• Aimed at reducing the foreign technology dependence.• Two AVS standards are finalized or to be finalized
• AVS 1• Finalized in 2008• Provide the coding efficiency two times higher than MPEG‐2, comparable to H.264/MEPG‐4 AVC
• The complexity is only 30%, and 70% compared to H.264/MPEG‐4 AVC encoder and decoder.
• AVS 2• To be finalized in Dec 2013• Expected to improve coding efficiency by two times compared to AVS1, under high definition or higher resolution conditions
138
![Page 139: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/139.jpg)
Outline
• Introduction• Lossless Compression• Image Compression• Video Compression• Simulation Results• Conclusion
139
![Page 140: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/140.jpg)
ImageStandardsComparison
• JPEG and JPEG 2000 simulations are conducted;• Several images are tested under various compression rate;
• Tools: 1520x1200• Bike: 2048x2560• Cafe: 2048x2560• Woman: 2048x2560
140
![Page 141: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/141.jpg)
ImageStandardsComparison
141
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.830
31
32
33
34
35
36
37
Bit Rate bits/pixel
PSN
R d
B
Tools
JPEGJPEG2000
0 0.2 0.4 0.6 0.8 1 1.2 1.430
32
34
36
38
40
42
44
Bit Rate bits/pixel
PSN
R d
B
Bike
JPEGJPEG2000
0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.830
31
32
33
34
35
36
37
Bit Rate bits/pixel
PSN
R d
B
Cafe
JPEGJPEG2000
0 0.2 0.4 0.6 0.8 1 1.2 1.430
32
34
36
38
40
42
44
Bit Rate bits/pixel
PSN
R d
B
Woman
JPEGJPEG2000
![Page 142: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/142.jpg)
VideoStandardsComparison
• Various video standards simulations are conducted under various quality levels:• H.261, H.263, MPEG‐1, MPEG‐2, MPEG‐4 are conducted using the FFMPEG software;
• H.264 uses the latest JM 18.4;• HEVC uses the latest HM 11.0;
• Two sequences are tested:• Foreman: 176x144• RaceHorses: 832x480
• All Intra condition is tested:• All the frames are encoded as I frame;• JPEG and JPEG 2000 are also included in the comparison;
• Low‐Delay condition is tested:• IPPP structure are used;
142
![Page 143: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/143.jpg)
VideoStandardsComparison
143
All Intra Comparison
0 5000 10000 1500028
30
32
34
36
38
40RaceHorses All Intra
Bit−Rate(kbps)
PS
NR
(dB
)
HEVCH264MJPEG2000MPEG4MPEG1MPEG2MJPEG
0 200 400 600 800 1000 1200 140026
28
30
32
34
36
38
40
42Foreman All Intra
Bit−Rate(kbps)
PS
NR
(dB
)
HEVCH264MPEG4MPEG1MPEG2H263H261MJPEG2000MJPEG
![Page 144: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/144.jpg)
VideoStandardsComparison
144
Low Delay Comparison
0 50 100 150 200 250 300 350 400 45024
26
28
30
32
34
36
38Foreman Low Delay
Bit−Rate(kbps)
PS
NR
(dB
)
HEVCH264MPEG4H263MPEG1MPEG2H261
0 1000 2000 3000 4000 5000 6000 700027
28
29
30
31
32
33
34
35
36
37RaceHorses All Intra
Bit−Rate(kbps)
PS
NR
(dB
)
HEVCH264MPEG4MPEG1MPEG2
![Page 145: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/145.jpg)
Outline
• Introduction• Lossless Compression• Image Compression• Video Compression• Simulation Results• Conclusion
145
![Page 146: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/146.jpg)
Conclusion
• Image coding standards: JPEG and JPEG 2000, both standards have their own advantages and enjoy popularity under certain circumstances.
• Video coding standards: MPEG‐1, MPEG‐2, MPEG‐4, H.261, H.263, H.264/MPEG‐4 AVC and HEVC, key features and applications introduced
• Experimental results: every evolution of the coding algorithms contributes greatly the compression performance
• This field is developing rapidly and its application can be found in various situations, and continuous effort on improving the coding algorithm will bring about a promising future for image and video compression. 146
![Page 147: Next Generation Video Coding‐ H.265/HEVC and its extensions · APSIPA Asia-Pacific Signal and Information Processing Association APSIPA Distinguished Lecture Series Next Generation](https://reader034.vdocuments.net/reader034/viewer/2022042113/5e8f61b24e36db26353b1e56/html5/thumbnails/147.jpg)
WhatNext?
• H.266?• MPEG‐5?• Wavelet transform?• Big Block?• SSIM v.s. PSNR?
147