Lecture 4:Lecture 4:Data Compression TechniquesData Compression Techniques
TSBK01 Image Coding and Data CompressionTSBK01 Image Coding and Data Compression
Jörgen AhlbergDiv. of Sensor Technology
Swedish Defence Research Agency (FOI)
OutlineOutline
Huffman codingHuffman coding Arithmetic codingArithmetic coding
Application: JBIGApplication: JBIG
Universal codingUniversal coding LZ-codingLZ-coding
LZ77, LZ78, LZWLZ77, LZ78, LZW
Applications: GIF and PNGApplications: GIF and PNG
RepetitionRepetition
Coding:Coding: Assigning binary codewords to Assigning binary codewords to (blocks of) source symbols.(blocks of) source symbols.
Variable-lengthVariable-length codes (VLC) and codes (VLC) and fixed-fixed-lengthlength codes. codes.
Instantaneous codes Instantaneous codes ½½ Uniqely decodable Uniqely decodable codes codes ½½ Non-singular codes Non-singular codes ½½ All codes All codes
Tree codesTree codes are instantaneous. are instantaneous. Tree code Tree code ,, Kraft’s Inequality. Kraft’s Inequality.
Creating a Code: The Data Creating a Code: The Data Compression ProblemCompression Problem
Assume a source with an alphabet Assume a source with an alphabet AA and and known symbol probabilities known symbol probabilities {{ppii}}..
GoalGoal: Chose the codeword lengths as to : Chose the codeword lengths as to minimize the bitrate, i.e., the average number minimize the bitrate, i.e., the average number of bits per symbol of bits per symbol llii ¢¢ ppii..
Trivial solution:Trivial solution: llii = 0 = 0 88 ii.. Restriction:Restriction: We want an instantaneous code, We want an instantaneous code,
so so 22-l-l
ii ·· 1 1 (KI) must be valid. (KI) must be valid. Solution Solution (at least in theory)(at least in theory):: llii = – = – loglog p pii
In practice…In practice…
Use some nice algorithm to find the code Use some nice algorithm to find the code treetree– Huffman codingHuffman coding– Tunnstall codingTunnstall coding
Huffman CodingHuffman Coding
Two-step algorithm:Two-step algorithm:1.1. Iterate:Iterate:
– Merge the least probable symbols.Merge the least probable symbols.– Sort.Sort.
2.2. Assign bits.Assign bits.
a
d
b
c
0.5
0.25
0.125
0.125
0.5
0.25
0.5
0.25
0.5
Merge
Sort
Assign
0
1
0
10
11
0
10
110
111Get code
Coding of the BMSCoding of the BMS
Trick:Trick: Code blocks of symbols (extended source). Code blocks of symbols (extended source). Example:Example: pp11 = ¼ , = ¼ , pp22 = ¾. = ¾.
Applying the Huffman algorithm directly:Applying the Huffman algorithm directly:1 bit/symbol1 bit/symbol..
BlockBlock PP(block)(block) CodeCode
0000 9/169/16 00 ))
0101 3/163/16 1010 approx 0.85 approx 0.85
1010 3/163/16 110110 bits/symbolbits/symbol
1111 1/161/16 111111
Huffman Coding: Pros and ConsHuffman Coding: Pros and Cons
++ Fast implementations.Fast implementations.++ Error resilient: resynchronizes in Error resilient: resynchronizes in ~ ~ ll2 2 steps.steps.- The code tree grows exponentially when the The code tree grows exponentially when the
source is extended.source is extended.- The symbol probabilities are built-in in the The symbol probabilities are built-in in the
code.code.Hard to use Huffman coding for extended Hard to use Huffman coding for extended sources / large alphabets or when the sources / large alphabets or when the symbol probabilities are varying by time.symbol probabilities are varying by time.
Arithmetic CodingArithmetic Coding
Shannon-Fano-EliasShannon-Fano-Elias Basic idea:Basic idea: Split the interval [0,1] according Split the interval [0,1] according
to the symbol probabilities.to the symbol probabilities. Example:Example: AA = {a,b,c,d}, = {a,b,c,d}, PP = {½, ¼, 1/8, 1/8}. = {½, ¼, 1/8, 1/8}.
b
c
a
0.6
0.5
0.5
0.20.8
0.2
0.2 Start in b.Code the sequence c c a.
cba 0.9
Code the sequence c c a.) Code the interval [0.9, 0.96]
Bit Interval Decoder
1 c0.5 - 1
1 0.75 - 1
1 0.875 - 1
0 0.875 - 0.9375
1 c a0.90624 - 0.9375
b c0.5 10
cba
0.9 10.96 0.98
An Image Coding ApplicationAn Image Coding Application
Consider the image content in a local environment Consider the image content in a local environment of a pixel as of a pixel as a state in a Markov modela state in a Markov model..
Example (binary image):Example (binary image):
Such an environment is called a Such an environment is called a contextcontext.. A probability distribution for A probability distribution for XX can be estimated for can be estimated for
each state. Then arithmetic coding is used.each state. Then arithmetic coding is used. This is the basic idea behind the JBIG algorithm This is the basic idea behind the JBIG algorithm
for binary images and data.for binary images and data.
X
0 0 11 0
Flushing the CoderFlushing the Coder
The coding process is ended (restarted) and The coding process is ended (restarted) and the coder flushedthe coder flushed– after a given number of symbols (FIVO)after a given number of symbols (FIVO)
oror– When the interval is too small for a fixed When the interval is too small for a fixed
number of output bits (VIFO).number of output bits (VIFO).
Universal CodingUniversal Coding
A universal coder doesn’t need to know the A universal coder doesn’t need to know the statistics in advance. Instead, estimate from statistics in advance. Instead, estimate from data.data.
Forward estimation:Forward estimation: Estimate statistics in a Estimate statistics in a first pass and transmit to the decoder.first pass and transmit to the decoder.
Backward estimation:Backward estimation: Estimate from already Estimate from already transmitted (received) symbols.transmitted (received) symbols.
Universal Coding: ExamplesUniversal Coding: Examples
1.1. An adaptive arithmetic coderAn adaptive arithmetic coder
2.2. An adaptive dictionary techniqueAn adaptive dictionary technique– The LZ coders The LZ coders [Sayood 5][Sayood 5]
3.3. An adaptive Huffman coder An adaptive Huffman coder [Sayood 3.4][Sayood 3.4]
Arithmeticcoder
Statisticsestimation
Ziv-Lempel Coding (ZL or LZ)Ziv-Lempel Coding (ZL or LZ)
Named after J. Ziv and A. Lempel (1977).Named after J. Ziv and A. Lempel (1977). Adaptive dictionary technique.Adaptive dictionary technique.
– Store previously coded symbols in a buffer.Store previously coded symbols in a buffer.– Search for the current sequence of symbols to Search for the current sequence of symbols to
code.code.– If found, transmit buffer offset and length.If found, transmit buffer offset and length.
LZ77LZ77
a b c a b d a c a b d e e e fc
Search buffer Look-ahead buffer
Output triplet <offset, length, next>
123456783
8 0 13 d 0 e 2 f
2
Transmitted to decoder:
If the size of the search buffer is N and the size of the alphabet is Mwe need
bits to code a triplet.
Variation:Variation: Use a VLC to code the triplets!PKZip, Zip, Lharc,PNG, gzip, ARJ
Drawback with LZ77Drawback with LZ77
Repetetive patterns with a period longer Repetetive patterns with a period longer than the search buffer size are not found.than the search buffer size are not found.
If the search buffer size is 4, the sequenceIf the search buffer size is 4, the sequence a b c d e a b c d e a b c d e a b c d e …a b c d e a b c d e a b c d e a b c d e …will be expanded, not compressed.will be expanded, not compressed.
LZ78LZ78
Store patterns in a dictionaryStore patterns in a dictionary Transmit a tuple <Transmit a tuple <dictionary indexdictionary index, , nextnext>>
LZ78LZ78
Output tuple <dictionary index, next>
Dictionary:
1 a
2 b
3 c
4 a b
5 a b c
a b c a b a b c
0 aTransmitted to decoder: 0 b 0 c 1 b 4 c
Decoded: a b c a a bb c
Strategy needed for limiting dictionary size!Strategy needed for limiting dictionary size!
LZWLZW
Modification to LZ78 by Terry Welch, 1984.Modification to LZ78 by Terry Welch, 1984. Applications: GIF, v42bisApplications: GIF, v42bis Patented by UniSys Corp.Patented by UniSys Corp. Transmit only the dictionary index.Transmit only the dictionary index. The alphabet is stored in the dictionary in The alphabet is stored in the dictionary in
advance.advance.
LZWLZW
Output: dictionary index
Encoder dictionary:
1 a
2 b
3 c
4 d
5 a b
a b c a b a b c
1
Transmitted:
2 3 5 5
Decoded:
a b c a b a b
6 bc
7 ca
8 aba
9 abc
Decoder dictionary:
1 a
2 b
3 c
4 d
5 a b
6 bc
7 ca
8 aba
Input sequence:
And now for some applications:GIF & PNG
GIFGIF
CompuServe Graphics Interchange Format (1987, CompuServe Graphics Interchange Format (1987, 89).89).
Features:Features:– Designed for up/downloading images to/from BBSes via Designed for up/downloading images to/from BBSes via
PSTN.PSTN.– 1-, 4-, or 8-bit 1-, 4-, or 8-bit colour palettescolour palettes..– Interlace for Interlace for progressive decodingprogressive decoding (four passes, starts (four passes, starts
with every 8th row).with every 8th row).– Transparent colourTransparent colour for non-rectangular images. for non-rectangular images.– Supports multiple images in one file (”animated GIFs”).Supports multiple images in one file (”animated GIFs”).
GIF: MethodGIF: Method
Compression by LZW.Compression by LZW. Dictionary size 2Dictionary size 2bb+1+1 8-bit symbols 8-bit symbols
– bb is the number of bits in the palette. is the number of bits in the palette.
Dictionary size doubled if filled (max 4096).Dictionary size doubled if filled (max 4096). Works well on computer generated images.Works well on computer generated images.
GIF: ProblemsGIF: Problems
Unsuitable for natural images (photos):Unsuitable for natural images (photos):– Maximum 256 colors (Maximum 256 colors ()) bad quality). bad quality).– Repetetive patterns uncommon (Repetetive patterns uncommon ()) bad bad
compression).compression).
LZW patented by UniSys Corp.LZW patented by UniSys Corp. Alternative: PNGAlternative: PNG
PNG: Portable Network GraphicsPNG: Portable Network Graphics
Designed to replace GIF.Designed to replace GIF. Some features:Some features:
– Indexed Indexed oror true-colour images true-colour images ((·· 16 bits per plane) 16 bits per plane)..– Alpha channel.Alpha channel.– Gamma information.Gamma information.– Error detection.Error detection.
No support for multiple images in one file.No support for multiple images in one file.– Use MNG for that.Use MNG for that.
Method:Method:– Compression by LZ77 using a 32KB search buffer.Compression by LZ77 using a 32KB search buffer.– The LZ77 triplets are Huffman coded.The LZ77 triplets are Huffman coded.
More information:More information: www.w3.org/TR/REC-png.html www.w3.org/TR/REC-png.html
SummarySummary
Huffman codingHuffman coding– Simple, easy, fastSimple, easy, fast– Complexity grows exponentially with the block lengthComplexity grows exponentially with the block length– Statistics built-in in the codeStatistics built-in in the code
Arithmetic codingArithmetic coding– Complexity grows linearly with the block sizeComplexity grows linearly with the block size– Easily adapted to variable statistics Easily adapted to variable statistics )) used for coding of Markov used for coding of Markov
sourcessources Universal codingUniversal coding
– Adaptive Huffman or arithmetic coderAdaptive Huffman or arithmetic coder– LZ77: Buffer with previously sent sequences <offset,length,next>LZ77: Buffer with previously sent sequences <offset,length,next>– LZ78: Dictionary instead of buffer <index,next>LZ78: Dictionary instead of buffer <index,next>– LZW: Modification to LZ78 <index>LZW: Modification to LZ78 <index>
Summary, contSummary, cont
Where are the algorithms used?Where are the algorithms used?– Huffman coding:Huffman coding: JPEG, MPEG, PNG, … JPEG, MPEG, PNG, …– Arithmetic coding: Arithmetic coding: JPEG, JBIG, MPEG-4, …JPEG, JBIG, MPEG-4, …– LZ77:LZ77: PNG, PKZip, Zip, gzip, … PNG, PKZip, Zip, gzip, …– LZW:LZW: compress, GIF, v42bis, … compress, GIF, v42bis, …
FinallyFinally
These methods work best if the source These methods work best if the source alphabet is small and the distribution alphabet is small and the distribution skewed.skewed.– TextText– GraphicsGraphics
Analog sources (images, sound) require Analog sources (images, sound) require other methodsother methods– complex dependenciescomplex dependencies– accepted distortionaccepted distortion