Andrea Montanari and Ruediger UrbankeTIFR

Tuesday, January 6th, 2008

Phase Transitions in Coding, Communications, and Inference

Outline

Outline

1) Thresholds in coding, the large size limit (definition and density evolution characterization)            

2) The inversion of limits (length to infty vs size to infty)                  

Outline

1) Thresholds in coding, the large size limit (definition and density evolution characterization)            

2) The inversion of limits (length to infty vs size to infty)                  

                  3) Phase transitions in measurements                     (compressed sensing versus message passing,  dense versus sparse matrices)

4) Phase transitions in collaborative filtering          (the low-rank matrix model)

Model

Shannon ’48

Model

Shannon ’48

binary erasures channelcapacity: R≤1-ε

Model

Shannon ’48

binary symmetric channelcapacity: R≤1-h(ε)

Channel Coding

Channel Coding

codeC={000, 010, 101, 111}

Channel Coding

codeC={000, 010, 101, 111}

n ... blocklength

Channel Coding

codeC={000, 010, 101, 111}

n ... blocklength

Channel Coding

codeC={000, 010, 101, 111}

n ... blocklength

Channel Coding

code

decoding

C={000, 010, 101, 111}

n ... blocklength

xMAP(y)=argmaxX in C p(x | y)

xiMAP(y)=argmaxXi p(xi |y)

Factor Graph Representation of Linear Codes

Factor Graph Representation of Linear Codes

every linear codeparity-check matrix

Factor Graph Representation of Linear Codes

(7, 4) Hamming code

every linear codeparity-check matrix

Factor Graph Representation of Linear Codes

(7, 4) Hamming code

every linear code

Tanner, Wiberg, Koetter, Loeliger, Frey

parity-check matrix

Low-Density Parity Check Codes

Gallager ‘60

Low-Density Parity Check Codes

(3, 4)-regular codes

Gallager ‘60

Low-Density Parity Check Codes

(3, 4)-regular codes

Gallager ‘60

Low-Density Parity Check Codes

(3, 4)-regular codes

Gallager ‘60

Low-Density Parity Check Codes

(3, 4)-regular codes

Gallager ‘60

Low-Density Parity Check Codes

(3, 4)-regular codes

Gallager ‘60

number of edges is linear in n

Ensemble

Ensemble

Ensemble

Ensemble

Ensemble

Variations on the Theme

Variations on the Theme

degree distributions as well as structure

Variations on the Theme

irregular LDPC ensemble

(Luby, Mitzenmacher, Shokrollahi, Spielman, and Stehman)

Variations on the Theme

regular RA ensemble

Divsalar, Jin, and McEliece

Variations on the Theme

irregular RA ensemble

Jin, Khandekar, and McEliece

Variations on the Theme

irregular MN ensemble

Davey, MacKay

Variations on the Theme

ARA ensemble

Abbasfar, Divsalar, Kung

Variations on the Theme

irregular LDGM ensemble

Variations on the Theme

turbo code

Berrou and Glavieux

Variations on the Theme

protograph

Thorpe, Andrews, Dolinar

Message-Passing Decoding -- BEC

?

?

0

Message-Passing Decoding -- BEC

?

?

00

0

?

?

?

Message-Passing Decoding -- BEC

?

?

0 0

0

?

?

?

Message-Passing Decoding -- BEC

?

?

0

0+?

0

0?

?

?

Message-Passing Decoding -- BEC

?

?

0

0+? =??

0

0?

?

?

Message-Passing Decoding -- BEC

?

?

0

?

0

0

?

?

?

Message-Passing Decoding -- BEC

?

?

0 0

?

??

?

Message-Passing Decoding -- BEC

?

?

0 0?

??

0=00

?

Message-Passing Decoding -- BEC

?

?

0 0

?

??

0

?

Message-Passing Decoding -- BEC

?

?

0

??

?

?

0

Message-Passing Decoding -- BEC

?

?

0

?

?

?

?

0

Message-Passing Decoding -- BEC

?

?

0

?

?

?

?

0 0decoded

Message-Passing Decoding -- BEC

?

?

00

0

?0

?

decoded

Message-Passing Decoding -- BEC

?

?

0 0

0

?

0

?

decoded

Message-Passing Decoding -- BEC

?

?

0 0

0

?

0?

decoded

0+0

Message-Passing Decoding -- BEC

?

?

0 0

0

?

0?

decoded

0+0 =00

Message-Passing Decoding -- BEC

?

?

0 0

0

?

0

?

decoded

0

Message-Passing Decoding -- BEC

?

?

0 0

0

?

0

?

decoded

decoded 0

Message-Passing Decoding -- BSCGallager Algorithm

Asymptotic Analysis: Computation Graph

Asymptotic Analysis: Computation Graph

Asymptotic Analysis: Computation Graph

Asymptotic Analysis: Computation Graph

Asymptotic Analysis: Computation Graph

Asymptotic Analysis: Computation Graph

Asymptotic Analysis: Computation Graph

Asymptotic Analysis: Computation Graph

Asymptotic Analysis: Computation Graph

probability that computation graphof fixed depth becomes tree

tends to 1 as n tends to infinity

Asymptotic Analysis: Computation Graph

Asymptotic Analysis: Computation Graph

Asymptotic Analysis: Density Evolution -- BEC

Luby,Mitzenmacher, Shokrollahi, Spielman,

and Steman ‘97

Asymptotic Analysis: Density Evolution -- BEC

x x x

Luby,Mitzenmacher, Shokrollahi, Spielman,

and Steman ‘97

Asymptotic Analysis: Density Evolution -- BEC

x

1-(1-x)r-1

x x

Luby,Mitzenmacher, Shokrollahi, Spielman,

and Steman ‘97

Asymptotic Analysis: Density Evolution -- BEC

x

1-(1-x)r-1

x x

ε (1-(1-x)r-1)l-1

Luby,Mitzenmacher, Shokrollahi, Spielman,

and Steman ‘97

Asymptotic Analysis: Density Evolution -- BEC

x

1-(1-x)r-1

x x

ε (1-(1-x)r-1)l-1

ε

Luby,Mitzenmacher, Shokrollahi, Spielman,

and Steman ‘97

Asymptotic Analysis: Density Evolution -- BEC

ε

Asymptotic Analysis: Density Evolution -- BEC

ε

phase transition: εBP so that xt → 0 for ε< εBP

xt → x∞>0 for ε> εBP

Asymptotic Analysis: Density Evolution -- BEC

ε

phase transition: εBP so that xt → 0 for ε< εBP

xt → x∞>0 for ε> εBP

Asymptotic Analysis: Density Evolution -- BSC, Gallager Algorithm

xt =ε (1-p+(xt-1))+(1-ε) p-(xt-1)

p+(x)=((1+(1-2x)r-1)/2) l-1 p-(x)=((1-(1-2x)r-1)/2) l-1

Asymptotic Analysis: Density Evolution -- BSC, Gallager Algorithm

phase transition: εBP so that xt → 0 for ε< εBP

xt → x∞>0 for ε> εBP

xt =ε (1-p+(xt-1))+(1-ε) p-(xt-1)

p+(x)=((1+(1-2x)r-1)/2) l-1 p-(x)=((1-(1-2x)r-1)/2) l-1

Asymptotic Analysis: Density Evolution -- BSC, Gallager Algorithm

xt =ε (1-p+(xt-1))+(1-ε) p-(xt-1)

p+(x)=((1+(1-2x)r-1)/2) l-1 p-(x)=((1-(1-2x)r-1)/2) l-1

Asymptotic Analysis: Density Evolution -- BP

Inversion of Limits

Inversion of Limits

size versus number of iterations

Density Evolution Limit

Density Evolution Limit

Density Evolution Limit

Density Evolution Limit

Density Evolution Limit

Density Evolution Limit

Density Evolution Limit

Density Evolution Limit

Density Evolution Limit

Density Evolution Limit

Density Evolution Limit

“Practical” Limit

“Practical” Limit

“Practical” Limit

“Practical” Limit

“Practical” Limit

“Practical” Limit

“Practical” Limit

The Two Limits

Easy: (Density Evolution Limit)

Hard(er): (“Practical Limit”)

Binary Erasure Channel

Binary Erasure Channel

DE Limit

Binary Erasure Channel

DE Limit

Binary Erasure Channel

DE Limit

Binary Erasure Channel

DE Limit

“Practical” Limit

implies

What about “General” Case

expansion

probabilistic methods

Korada and U.

Expansion

expansion ~ 1-1/l

Expansion

Miller and Burshtein: Random element of LDPC(l, r, n) ensemble is expander with expansion close to 1-1/l

with high probability

expansion ~ 1-1/l

Why is Expansion Useful?

Setting: Channel

Setting: Ensemble

Setting: Algorithm

Aim: Show for this setting that ...

DE Limit

“Practical” Limit

implies

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Linearized Decoding Algorithm

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Combine with Density Evolution

Combine with Density Evolution

Combine with Density Evolution

Combine with Density Evolution

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Correlation and Interaction

Correlation and Interaction

Correlation and Interaction

Correlation and Interaction

0

Correlation and Interaction

0 1

Correlation and Interaction

0 1

Correlation and Interaction

0 1

0

Correlation and Interaction

0 1

00

Correlation and Interaction

0 1

000

Correlation and Interaction

0 1

1 000

Correlation and Interaction

0 1

1 000

Correlation and Interaction

0 1

1 000Expected growth:

Correlation and Interaction

0 1

1 000Expected growth:

(r-1)(r-1)

Correlation and Interaction

0 1

1 000Expected growth:

(r-1) 2 ε 2 ε

Correlation and Interaction

0 1

1 000Expected growth:

(r-1) 2 ε ?< 1

Correlation and Interaction

0 1

1 000Expected growth:

(r-1) 2 ε ?< 1

Problem: interaction correlation

Correlation and Interaction

Correlation and Interaction

Correlation and Interaction

Correlation and Interaction

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Witness

Witness

Witness

Witness

Witness

Witness

Witness

Witness

Witness

Witness

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Monotonicity

Monotonicity

Monotonicity

Monotonicity

Monotonicity

Monotonicity

Monotonicity

Randomizing the Noise Outside

Randomizing the Noise Outside

Randomizing the Noise Outside

→

Randomizing the Noise Outside

→

←⁄

Randomizing the Noise Outside

randomizing noise outside the witness increases the probability of error

FKG≤

Proof Outline

linearize the algorithm combine with density evolution correlation and interaction witness randomizing noise outside the witness sub-critical birth and death process

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

0

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

0 1

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

0 1

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

0 1

0

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

0 1

00

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

0 1

000

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

0 1

1 000

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

0 1

1 000

Expansion

random graph has expansion close to expansion of a treewith high probability

⇒this limits interaction

0 1

1 000

References

For a list of references see:http://ipg.epfl.ch/doku.php?id=en:courses:2007-2208:mct

Results

Open Problems

0.0

0.4

0.3

0.2

0.1

0.2 0.4 0.6 0.8

Pb

channel entropy

Open Problems

0.0

0.4

0.3

0.2

0.1

0.2 0.4 0.6 0.8

Pb

channel entropy

Open Problems

0.0

0.4

0.3

0.2

0.1

0.2 0.4 0.6 0.8

Pb

channel entropy

Open Problems

0.0

0.4

0.3

0.2

0.1

0.2 0.4 0.6 0.8

Pb

channel entropy