Fountain CodesD.J.C MacKay

IEE Proceedings Communications, Vol. 152, No. 6, December 2005

2

Introduction Fountain Codes Intermission LT Codes Raptor Codes

Outline

3

Erasure Channel:◦ Files are transmitted in multiple small packets.◦ Each packet is either received without error or

loss.◦ Such as the Internet.

How to deal with packet loss?◦ Some simple retransmission protocols:

ACKs: for missing packets. ACKs: for received packets.

◦ Erasure Correcting Codes.

Introduction

4

Why Erasure Correcting Codes? Retransmission are wasteful when erasure is

serious: ACKs: for missing packets.

ACKs would be enormous.

ACKs: for received packets. Would lead to multiple copies.

Broadcast Channel with erasure:

Introduction

server

ADC

FE

5

Erasure Correcting Codes: Block Code, such as (N, K) Reed-Solomon Code:

Any K of the N transmitted symbols are received, then the

original K source symbols can be recovered.

High Complexity: O( K(N-K) log2N)

Estimate the erasure probability f first, then choose the

code rate R = K/N before transmission.

Ex. If loss-rate = 50%, then set code rate R = 1/(1-50%) =

1/2 = K/N.

( N = 2K )

Introduction

loss-rate = 50%

6

Erasure Correcting Codes: Block Code, such as (N, K) Reed-Solomon Code:

If f is larger than expected (decoder receives fewer than K

symbols):

Ex. We thought loss-rate is 50%, and set the code rate R =

1/2 ( N = 2K ); however, the actual loss-rate = 66.7%, the

proper code rate R should be lower: R = 1/3 ( N = 3K )

We would like a simple way to extend the code on the fly to

create a lower-rate (N’, K) code.

Introduction

loss-rate = 66.7%

No way!

Fountain Codes are rateless: The number of encoded packets generated can be

determined on the fly.

Fountain Codes

7

Fountain Codes are rateless: The number of encoded packets generated can be

determined on the fly.

Fountain Codes can also have fantastically small encoding and decoding complexities. Depends on the careful choice of Degree Distribution.

Fountain Codes

9

Balls–and–Bins Problem:◦ Imagine that we throw N balls independently at

random into K bins, what probability of one bin have no balls in it?

Intermission

…K bins

…N throws

10

Balls–and–Bins Problem:◦ After N balls have been thrown, what probability

of one bin have no ball in it? The probability that one particular bin is empty after

N balls have been thrown:

Intermission

…K

bins

11

Balls–and–Bins Problem:◦ After N balls have been thrown, what probability of

one bin have no ball in it? The probability that one particular bin is empty after

N balls have been thrown:

The expected number of empty bins: δ =

,which roughly implies: the probability of all bins have a ball is (1- δ) only if:

Intermission …K

bins

12

Luby Transform (LT) Codes:◦ Encoding process:

For the ith encoded packet, select degree di by carefully chosen Degree Distribution (Robust Soliton Distribution).

Choose di source data. Perform XOR on chosen data.

◦ Decoding process: Decode degree-one encoded packets. Remove degree-one edges iteratively.

LT Codes

Degree 1 2 3 … k

probability μ1 μ2 μ3 … μk

…

x1 x2 x3 x4 x5

y1 y2 y3 y4 y5

x1

x3

x2

x2

x5

x4

x6

x3

x5

x6

13

Designing the Degree Distribution:◦ A few encoded packets must have high degree.

To ensure that every source data are connected to encoded packets.

◦ Many encoded packets must have low degree. So that decoding process can get started, and keep

going. Also the total number of XOR operations involved in

the encoding and decoding is kept small.

LT Codes

x1 x2 x3 x4 x5

y1 y2 y3 y4 y5

14

Some properties of Degree Distribution:◦ The complexity (both encoding and decoding) are

scaled linearly with the number of edges in the graph.

◦ Key factor: The average degree of the encoded packets.

How small (number of edges) can this be?◦ Recall: Balls–and–Bins Problem.

Balls: linked edges. Bins: source data.

LT Codes

x1 x2 x3 x4 x5

y1 y2 y3 y4 y5

How small number of edges can assure that every source data must have at least one edge on it? (all bins have a ball)

Some properties of encoder:◦ Encoder throws edges into source data at

random. The number of edges must be at least of order : K

lnK. Balls–and–Bins Problem:

The expected number of empty bins: δ =

,which roughly implies: the probability of all bins have a

ball is (1- δ) only if:

LT Codes

15

For decoder:◦ If decoder received optimal K encoded packets, the

average degree of each encoded packet must be at least: lnK The number of edges must be at least of order: K lnK. The complexity (both encoding and decoding) of an LT

code will definitely be at least: K lnK Luby showed that this bound on complexity

can indeed be achieved by a careful choice of Degree Distribution.

LT Codes

16

Ideally, to avoid redundancy:◦ We would like just one check node has degree

one at each iteration. Ideal Soliton Distribution:

The expected average degree under this distribution is roughly: lnK

LT Codes

17

In practice, this distribution works poorly:◦ Fluctuations around the expected behavior:

Sometimes in the decoding process there will be no degree-one check node.

◦ A few source data will receive no connections at all.

Some small modification fixes these problems.◦ Robust Soliton Distribution:

More degree-one check node. A bit more high-degree check node.

LT Codes

18

19

Robust Soliton Distribution: Two extra parameters: c and δ The expected number of degree-one check node

(through out decoding process) is about:

δ: a bound on the decoding failure probability. Decoding fails to run to completion after K’ of encoded

packets have been received. c: a free parameters smaller than 1.

Luby’s key result.

LT Codes

20

Luby defines a positive function:

, then adds the Ideal Soliton Distribution ρ to τ and normalize to obtain the Robust Soliton Distribution μ:

LT Codes

, where ※ Receiver once receives K' = KZ encoded packets

ensures that the decoding can run to completion with probability at least 1 - δ.

21

LT Codes

τ( k/S )

※ High-degree ensures every source data is likely to be connected to a check.

※ Small-degree of τ ensures the decoding process gets started.

22

LT Codes※ Histograms of the number of encoded packets N required in order to recover source data K = 10,000

23

LT Codes ※ Practical performance of LT codes

- Three experimental decodings are shown.

※ All codes created with c = 0.03, δ = 0.5 (S= 30, K/S = 337, Z = 1.03), and K = 10,000

an overhead of 10%

24

Complexity cost:◦ LT Codes: O(K lnK), where K: the number of

original data. Average degree of the encoded packets: lnK Encoding and decoding complexity: lnK per

encoded packet◦ Raptor Codes: Linear time encoding and

decoding. Concatenating a weakened LT Code with an outer

code. Average degree of weakened LT code ≒ 3

Raptor Codes

25

Weakened LT Code:◦ Average degree of encoded packets ≒ 3◦ A fraction of source data will receive no

connections at all. What fraction?

◦ Balls–and–Bins Problem:

Raptor Codes

Also the fraction of empty bins

≒ 5%

26

Shokrollahi’s trick: ※ Encoder:

◦ Find a outer code can correct erasures if the erasure rate is:, then pre-code K source data into:

◦ Transmit this slightly enlarged data using a weaken LT Code. ※ Decoder:

◦ Once slightly more than K encoded packets been received, can recover of the pre-coded packets (roughly K packets).

◦ Then use the outer code to recover all the original data.

Raptor Codes

27

Raptor CodesK = 16

K’ = 20

N = 18

※ Schematic diagram of a Raptor Code

Pre-Coding

Weaken LT

covered = 17

28

Raptor Codes※ The idea of a weakened LT Code.

※ LT codes created with c = 0.03, δ = 0.5 and truncated at degree 8 (thus average degree = 3)

Thank you :)

30

Soliton Distribution Ideal Soliton Distribution:

The average degree is roughly: lnK

)ln( 1

)1(111

)(

1

2

1

Kd

ddd

K

ddd

K

d

K

d

K

d

31

Soliton Distribution Robust Soliton Distribution μ:

The average degree is roughly: lnK

)ln( )ln(1)ln(

)ln(1

1

) (

) () (

1

1

1

2

KOSK

SKS

d

ddd

ddddd

d

RK

d

K

d

d

d

d

, where