searching for low weight pseudo-codewordscnls.lanl.gov/~chertkov/talks/fec/pseudo20min.pdfoutline...

Outline

Searching for low weight pseudo-codewords

Michael Chertkov1 & Mikhail Stepanov 2,1

1Theory Division, LANL and 2University of Arizona, Tucson

Jan 30, 2007, ITA workshop, UCSD

Michael Chertkov, Los Alamos Searching for low weight pseudo-codewords

Outline

Outline

1 IntroductionLDPCBP vs LP & Bethe Free EnergyError-floor

2 Instanton-amoeba (general)

3 Pseudo-Codeword-Search (LP)

4 Dendro-LDPC

5 Simulations: FER vs SNR & pseudo-codeword spectrumTanner codeMargulis p=7 code

6 Conclusions


IntroductionInstanton-amoeba (general)

Pseudo-Codeword-Search (LP)Dendro-LDPC

Simulations: FER vs SNR & pseudo-codeword spectrumConclusions

LDPCBP vs LP & Bethe Free EnergyError-floor

Error Correction

Scheme:

Example of Additive White Gaussian Channel:

P(xout |xin) =Y

i=bits

p(xout;i |xin;i )

p(x|y) ∼ exp(−s2(x − y)2/2)

Channelis noisy ”black box” with only statistical information available

Encoding:use redundancy to redistribute damaging effect of the noise

Decoding:reconstruct most probable codeword by noisy (polluted) channel






Low Density Parity Check Codes

N bits, M checks, L = N − M information bitsexample: N = 10, M = 5, L = 5

2L codewords of 2N possible patterns

Parity check: Hv = c = 0example:

H =

0BBB@

1 1 1 1 0 1 1 0 0 00 0 1 1 1 1 1 1 0 00 1 0 1 0 1 0 1 1 11 0 1 0 1 0 0 1 1 11 1 0 0 1 0 1 0 1 1

1CCCA

LDPC = graph (parity check matrix) is sparse






Decoding=Statistical InferenceMaximum Likelihood Maximum-a-Posteriori

ML=arg maxσ=codeword

P(xout|σ) MAPi =sign

(∑σ σiP(xout|σ)∑σ P(xout|σ)

)

MAP≈BP=Belief-Propagation (Bethe-Pieirls) Gallager ’61

Exact on a tree

Trading optimality for reduction in complexity: ∼ 2L →∼ L

BP = solving equations on the graph:

ηjα = hj +j∈β∑β 6=α

tanh−1

(i∈β∏i 6=j

tanh ηiβ

)Message Passing = iterative BP






Bethe free energy: variational approach(Yedidia,Freeman,Weiss ’01 - inspired by Bethe ’35, Peierls ’36)

F = −Pi

hiPσi

σibi (σi ) +Pα

Pσα

bα(σα) ln bα(σα)−Pi(qi − 1)

Pσi

bi (σi ) ln bi (σi )

constraints:

∀ i , α : 0 ≤ bi (σi ), bα(σα) ≤ 1

∀ α :X

σα

bα(σα) = 1

∀ i ; α ∈ i : bi (σi ) =X

σα\σi

bα(σα)

Belief-Propagation Equations:

δFδb

��constr.

= 0

• Relaxation to minimum of the Bethe Free energy enforces convergence of iterative BP (Stepanov, Chertkov ’06)

LP decoding Feldman, Wainwright, Karger ’03

LP decoding = minimization of a linear function over a bounded domaindescribed by linear constraints

”Large SNR” limit of BP: F ≈ E = −Pi

hiPσi

σibi (σi )

“small” polytope = get rid of bα






Error-Floor

T. Richardson, Allerton ’03

BER vs SNR = measure ofperformance

Waterfall ↔ Error-floor

Suboptimal decoding causeserror-floor: at s2 = Es/N0 →∞,

FERML ∼ exp(−dMLs2/2) vs

FERsub ∼ exp(−dsubs2/2) where

dML ≥ dsub

Monte-Carlo is useless atFER . 10−8

Need an efficient method toanalyze error-floor





Pseudo-codewords and Instantons

Error-floor is caused by Pseudo-codewords:

Wiberg ’96; Forney et.al’99; Frey et.al ’01;

Richardson ’03; Vontobel, Koetter ’04-’06

Instanton = optimal conf of the noise

BER =

∫d(noise) WEIGHT (noise)

BER ∼ WEIGHT

(optimal confof the noise

)optimal confof the noise

=Point at the ESclosest to ”0”

Instantons are decoded to Pseudo-Codewords

Instanton-amoeba

= optimization algorithmStepanov, et.al ’04,’05

Stepanov, Chertkov ’06





“0”-codeword

Error-Surface

dangerouspseudo-codeword

Weighted Median: xinst = argminx P(0|x)|E(x;σ)=0

xinst =σ

2

∑i σi∑i σ

2i

, d =(∑

i σi )2∑

i σ2i

, FER ∼ exp(−d ·s2/2)

Wiberg ’96; Forney et.al ’99; Vontobel, Koetter ’03,’05

Pseudo-Codeword-Search Algorithm Chertkov, Stepanov ’06

Start: Initiate x(0).

Step 1: x(k) is decoded to σ(k).

Step 2: Find y(k) - weighted median

between σ(k), and ”0”

Step 3: If y(k) = y(k−1), k∗ = k End.Otherwise go to Step 2 with

x(k+1) = y(k) + 0.





“0”-codeword

Error-Surface

dangerouspseudo-codeword

Weighted Median: xinst = argminx P(0|x)|E(x;σ)=0

xinst =σ

2

∑i σi∑i σ

2i

, d =(∑

i σi )2∑

i σ2i

, FER ∼ exp(−d ·s2/2)

Wiberg ’96; Forney et.al ’99; Vontobel, Koetter ’03,’05

Pseudo-Codeword-Search Algorithm Chertkov, Stepanov ’06

000

xxx(0)

xxx(1)

xxx(2)

xxx(3)

σσσ(0)

σσσ(1)

σσσ(2,3)

xxx(0) start

step 2

step 3

σσσ(2) = σσσ(3) end

Start: Initiate x(0).

Step 1: x(k) is decoded to σ(k).

Step 2: Find y(k) - weighted median

between σ(k), and ”0”

Step 3: If y(k) = y(k−1), k∗ = k End.Otherwise go to Step 2 with

x(k+1) = y(k) + 0.

Multiple repetitions ⇒ instanton frequency spectra





LP complexity grows exponentially with check degree

Current solutions:

Adaptive LP (Taghavi, Siegel ’06)

BP-style relaxation of LP (Vontobel, Koetter ’06)

Dendro-trick = Graph Modification (our solution)Chertkov,Stepanov’07

MAP solutions are identical

Set of Pseudo-codewords are identical

Instanton spectra are very alike, ≈





Tanner codeMargulis p=7 code

1 2 3 4 5

1

10

10

10

10

10

–2

–4

–6

–8

–10

SNR

Fram

e-E

rror

-Rat

e (F

ER

)

5

5 5 5

7

3

3 3

4 4

1 1

BP, 4 iter

BP, 1024 iter

LP

BP, 8 iter

BP, 128 iter

0

0.2

0.4

0.6

0.8

1

10 15 20 25 30di

stri

butio

n fu

nctio

nd

dML

LP

BP, 8 iter

BP, 4 iter

dmin;inst < dML = 20

Dangerous instantons are frequent





Tanner codeMargulis p=7 code

1 2 3 4 5

1

10

10

10

10

10

–2

–4

–6

–8

–10

SNR

Fram

e-E

rror

-Rat

e (F

ER

)

BP, 4 iter

LP

BP, 128 iter

BP, 8 iter

BP, 1024 iter 0

0.2

0.4

0.6

0.8

1

10 15 20 25 30 35 40 45 50di

stri

butio

n fu

nctio

nd

dML

BP, 8 iter

BP, 4 iter

LP

dmin;inst;LP > dML = 16

Dangerous codewords are rare ⇒ emergence of a steeptransient asymptotic of FER vs SNR





Conclusions:

BP is faster but LP is easier to analyze

Instanton-amoeba and especially Pseudo-codeword-search areof a practical value for the error-floor domain exploration

Dendro-LDPC is a convenient trick reducing complexity of LP

Future Challenges:

Improving LP/BP (Facet Guessing of Dimakis,Wainwright ’06 and

LP-erasure Chertkov,Chernyak ’06 are good candidates)

Analyzing really long codes

Analyzing error-floor in correlated channels (e.g. 1d and 2d ISI

with and without coding)

Design of LPDC codes (with reduced error-floor)





Bibliography:

M. Chertkov, M.G. Stepanov, Searching for low weight pseudo-codewords, 2007 Information Theory andApplication Workshop, proceedings.

M. Chertkov, M.G. Stepanov, Pseudo-codeword Landscape, submitted for proceeding of ISIT 2007, June2007, Nice, cs.IT/0701084.

M. Chertkov, M.G. Stepanov, An Efficient Pseudo-Codeword Search Algorithm for Linear ProgrammingDecoding of LDPC Codes, arXiv:cs.IT/0601113, submitted to IEEE Transactions on Information Theory.

M.G. Stepanov, M. Chertkov, Instanton analysis of Low-Density-Parity-Check codes in the error-floorregime, arXiv:cs.IT/0601070, ISIT 2006 (July 2006, Seattle, WA)

M.G. Stepanov, M. Chertkov, Improving convergence of belief propagation decoding, arXiv:cs.IT/0607112,44th Allerton Conference (September 27-29, 2006, Allerton, IL)

M.G. Stepanov, M. Chertkov, The error-floor of LDPC codes in the Laplacian channel, Proceedings of 43rdAllerton Conference (September 28-30, 2005, Allerton, IL), arXiv:cs.IT/0507031

M.G. Stepanov, V. Chernyak, M. Chertkov, B. Vasic, Diagnosis of weakness in error correction: a physicsapproach to error floor analysis, Phys. Rev. Lett. 95, 228701 (2005) [See alsohttp://www.arxiv.org/cond-mat/0506037 for extended version with Supplements.]

V. Chernyak, M. Chertkov, M. Stepanov, B. Vasic, Error correction on a tree: An instanton approach,Phys. Rev. Lett. 93, 198702-1 (2004).

All papers are available at http://cnls.lanl.gov/∼chertkov/pub.htm


searching for low weight pseudo-codewordscnls.lanl.gov/~chertkov/talks/fec/pseudo20min.pdfoutline...

Documents