information theoryhamada/it/l08-it.pdf4. fano code． the fano code is a technique for constructing...

Information Theory

Mohamed Hamada

Software Engineering LabThe University of Aizu

Email: [email protected]: http://www.u-aizu.ac.jp/~hamada

Today’s Topics

• Source Coding Techniques• Fano Code• Coding Examples• Shannon Code• Code Comparison

1

Channel

Source Coding Techniques

InformationSource

User of Information

Source Encoder

Channel Encoder

Modulator

Source Decoder

Channel Decoder

De-Modulator

1. Huffman Code．

2. Two-pass Huffman Code．

4. Fano code．

5. Shannon Code．

6. Arithmetic Code．

3. Lemple-Ziv Code．

2

Sender Receiver


1. Huffman Code．

2. Two-pass Huffman Code．

4. Fano code．

5. Shannon Code．



3


1. Huffman Code．

2. Two-path Huffman Code．

4. Fano Code．

5. Shannon Code．



4

4. Fano Code．

The Fano code is a technique for constructing a prefix code

It is suboptimal in the sense that it does not achieve the lowest possible expected code word length like Huffman coding.

However unlike Huffman coding, it does guarantee that all code word lengths are within one bit of their theoretical ideal log(1/P(x)).

5

Fano coding is used in the IMPLODE compression method, which is part of the ZIP file format.

(Also called Shannon-Fano code)

4. Fano Code．

The Fano code is performed as follows:

1. arrange the information source symbols in order of decreasing probability

2. divide the symbols into two equally probable groups, as possible as you can

3. each group receives one of the binary symbols (i.e. 0 or 1) as the first symbol

4. repeat steps 2 and 3 per group as many times as this is possible.

5. stop when no more groups to divide

6

Example 1:

4. Fano Code．

Symbol Probability Fano CodeA 1/4

B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

1. arrange the information source symbols in order of decreasing probability

7

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32


8

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32


00

111

11

111

9

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32


00

111

11

111

10

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111


11

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111


01

12

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

13

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01


14

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01


001

11

111

15

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111


16

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111


17

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111


01

18

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

01

19

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

01


20

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

01


0

01

111

21

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

010

01

111


22

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

010

01

111


0

1

23

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

010

01

111

0

1


24

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

010

01

111

0

1


0

011

25

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

010

01

111

0

10

011


26

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

010

01

111

0

10

011


0

1

27

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

010

01

111

0

10

011

0

1


28

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

010

01

111

0

10

011

0

1


01

29

Example 1:

4. Fano Code．


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

00

111

11

111

01

001

11

111

010

01

111

0

10

011

0

1

5. stop when no more groups to divide

01

30

4. Fano Code．

Note that: If it was not possible to divide precisely the probabilities into equally probable groups, we should try to make the division as good as possible, as we can see from the following example.

Symbol Probability Fano Code

T 1/3

U 1/3

V 1/9

W 1/9

X 1/9

Example 2:

1

0

00

0

1

1 1

1 1 1

0

32

Example 3:

4. Fano Code．

Symbol Probability Fano CodeA 0.39 00

B 0.18 01

C 0.15 10

D 0.15 110

E 0.13 111

31

0

0 1

0 1 0 1

1

0 1

0 1 01

0 1

Tree form:


1. Huffman Code．

2. Two-path Huffman Code．

4. Fano Code．

5. Shannon Code．



33

5. Shannon Code．

The Shannon code is the coding method used to proveShannon's noiseless coding theorem

It constructs a uniquely decodable code

It is suboptimal in the sense that it does not achieve the lowest possible expected code word length like Huffman coding.

34

5. Shannon Code．

The Shannon code is performed as follows:

1. calculate a series of cumulative probabilities

k

iik pq

1, k=1,2,…,n

2. calculate the code length for each symbol using log（1\pi ）≤ li < log （1\pi ）+ 1

3. write qk in the form c1 ２-１ + c2 2-2 + … + cli 2-li where each ci is either 0 or 1

35

Example 4:

Symbol Probability qk Length li Shannon CodeA 1/4

B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

1. calculate a series of cumulative probabilities

5. Shannon Code．

0+

1/4+

1/2+5/8+

3/4+

13/16+7/8+

29/32+

15/16+

31/3236


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

2. calculate the code length for each symbol using

5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

log（1\pi ）≤ li < log （1\pi ）+ 1

Example 4:

37


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32


5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

log（1\pi ）≤ li < log （1\pi ）+ 1

Log(1/(1/4)) ≤ l1 < Log(1/(1/4)) + 1

2 ≤ l1 < 2 + 1

Example 4:

38


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32


5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

log（1\pi ）≤ li < log （1\pi ）+ 1

Log(1/(1/4)) ≤ l1 < Log(1/(1/4)) + 12 ≤ l1 < 2 + 1

l1 = 22

Example 4:

39


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32


5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

log（1\pi ）≤ li < log （1\pi ）+ 1

223

3

445

555

Example 4:

40


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

223

3

445

555


Example 3:

41


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

223

3

445

555


qk ≤ c1 ２-1 + c2 2-2

Example 4:

42


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

223

3

445

555


qk ≤ c1 ２-1 + c2 2-2

0 ≤ c1 ２-1 + c2 2-2

Example 4:

43


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

223

3

445

555


qk ≤ c1 ２-1 + c2 2-2

0 ≤ c1 ２-1 + c2 2-2

c1 = 0, c2 = 0

Example 4:

44


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

223

3

445

555


qk ≤ c1 ２-1 + c2 2-2

0 ≤ c1 ２-1 + c2 2-2

c1 = 0, c2 = 0

00

Example 4:

45


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

223

3

445

555


0001

100101

11001101

11100111011111011111

Example 4:

46


B 1/4

C 1/8

D 1/8

E 1/16

F 1/16

G 1/32

H 1/32

I 1/32

J 1/32

5. Shannon Code．

0

1/41/25/8

3/413/16

7/8

29/3215/1631/32

223

3

445

555

0001

100101

11001101

11100111011111011111

Example 4:

47

Example

Symbol Probability qk Length li Shannon Code Fano code

W 0.4 0 2 00 0

X 0.3 0.4 2 01 10

Y 0.2 0.7 3 101 110

Z 0.1 0.9 4 1110 111

5. Shannon Code．

Note that:

from examples 1 and 4 one may conclude that Fanocoding and Shannon coding produce the same code, however this is not true in general as we can see from the following example.

48

information theoryhamada/it/l08-it.pdf4. fano code． the fano code is a technique for constructing...

Documents