# error-correcting codes: progress & challenges madhu sudan microsoft/mit 02/17/20101ecc:...

of 31
/31

Embed Size (px)

TRANSCRIPT

- Slide 1
- Error-Correcting Codes: Progress & Challenges Madhu Sudan Microsoft/MIT 02/17/20101ECC: Progress/Challenges (@CMU)
- Slide 2
- Communication in presence of noise Noisy Channel Sender Receiver We are not ready We are now ready If information is digital, reliability is critical 02/17/20102ECC: Progress/Challenges (@CMU)
- Slide 3
- Shannons Model: Probabilistic Noise Noisy Channel Probabilistic Noise: E.g., every letter flipped to random other letter of w.p. p Focus: Design good Encode/Decode algorithms. Encode (expand) Decode (compress?) Sender Receiver E: k n D: n k 02/17/20103ECC: Progress/Challenges (@CMU)
- Slide 4
- Hamming Model: Worst-case error Errors: Upto t worst-case errors Focus: Code: C = Image(E) = {E(x) | x k } (Note: Not encoding/decoding) (Note: Not encoding/decoding) Goal: Design code to correct every possible pattern of t errors. of t errors. 02/17/20104ECC: Progress/Challenges (@CMU)
- Slide 5
- Problems in Coding Theory, Broadly Combinatorics: Design best possible error- correcting codes. Combinatorics: Design best possible error- correcting codes. Probability/Algorithms: Design algorithms correcting random/worst-case errors. Probability/Algorithms: Design algorithms correcting random/worst-case errors. 02/17/20105ECC: Progress/Challenges (@CMU)
- Slide 6
- Part I (of III): Combinatorial Results 02/17/20106ECC: Progress/Challenges (@CMU)
- Slide 7 0 Can transmit at rate R = 1 H q (p) - , 8 > 0 Converse Coding Theorem: Converse Coding Theorem: Can not transmit at rate R = 1 H q (p) + Can not transmit at rate R = 1 H q (p) + So: No mysteries? So: No mysteries? 02/17/201018ECC: Progress/Challenges (@CMU) If R = 1 H q (p) - , then for every n and k = Rn, there exist E: k n and D: n k s.t. Pr Channel,x [D(Channel(E(x)) x] exp(-n).">
- Recall Shannon 1948 -symmetric channel w. error prob. p: -symmetric channel w. error prob. p: Transmits 2 as w.p. 1-p; Transmits 2 as w.p. 1-p; and as 2 - {} w.p. p/(q-1). and as 2 - {} w.p. p/(q-1). Shannons Coding Theorem: Shannons Coding Theorem: Can transmit at rate R = 1 H q (p) - , 8 > 0 Can transmit at rate R = 1 H q (p) - , 8 > 0 Converse Coding Theorem: Converse Coding Theorem: Can not transmit at rate R = 1 H q (p) + Can not transmit at rate R = 1 H q (p) + So: No mysteries? So: No mysteries? 02/17/201018ECC: Progress/Challenges (@CMU) If R = 1 H q (p) - , then for every n and k = Rn, there exist E: k n and D: n k s.t. Pr Channel,x [D(Channel(E(x)) x] exp(-n).
- Slide 19
- Shannons functions: Shannons functions: E random, D brute force search. E random, D brute force search. Can we get poly time E, D? Can we get poly time E, D? [Forney 66]: Yes! (Using Reed-Solomon codes correcting -fraction error + composition.) [Forney 66]: Yes! (Using Reed-Solomon codes correcting -fraction error + composition.) [Sipser-Spielman 92, Spielman 94, Barg- Zemor 97]: Even in linear time! [Sipser-Spielman 92, Spielman 94, Barg- Zemor 97]: Even in linear time! Still didnt satisfy practical needs. Why? Still didnt satisfy practical needs. Why? [Berrou et al. 92] Turbo codes + belief propagation: [Berrou et al. 92] Turbo codes + belief propagation: No theorems; Much excitement No theorems; Much excitement Constructive versions 02/17/201019ECC: Progress/Challenges (@CMU)
- Slide 20
- What is satisfaction? Articulated by [Luby,Mitzenmacher,Shokrollahi,Spielman 96] Articulated by [Luby,Mitzenmacher,Shokrollahi,Spielman 96] Practically interesting question: Practically interesting question: n = 10000; q = 2, p =.1; n = 10000; q = 2, p =.1; Desired error prob. = 10 -6 ; k = ? Desired error prob. = 10 -6 ; k = ? [Forney 66]: Decoding time: exp(1/(1 H(p) (k/n))); [Forney 66]: Decoding time: exp(1/(1 H(p) (k/n))); Rate = 90% ) decoding time 2 100; Rate = 90% ) decoding time 2 100; Right question: reduce decoding time to poly(n,1/ ); where = 1 H(p) (k/n) Right question: reduce decoding time to poly(n,1/ ); where = 1 H(p) (k/n) 02/17/201020ECC: Progress/Challenges (@CMU)
- Slide 21
- Current state of the art Luby et al.: Propose study of codes based on irregular graphs (Irregular LDPC Codes). Luby et al.: Propose study of codes based on irregular graphs (Irregular LDPC Codes). No theorems so far for erroneous channels. No theorems so far for erroneous channels. Strong analysis for (much) simpler case of erasure channels (symbols are erased); decoding time = O(n log (1/ )) Strong analysis for (much) simpler case of erasure channels (symbols are erased); decoding time = O(n log (1/ )) (Easy to get composition based algorithms with (Easy to get composition based algorithms with decoding time = O(n poly(1/ )) decoding time = O(n poly(1/ )) Do have some proposals for errors as well (with analysis by Luby et al., Richardson & Urbanke), but none known to converge to Shannon limit. Do have some proposals for errors as well (with analysis by Luby et al., Richardson & Urbanke), but none known to converge to Shannon limit. 02/17/201021ECC: Progress/Challenges (@CMU)
- Slide 22
- Part III: Correcting Adversarial Errors 02/17/201022ECC: Progress/Challenges (@CMU)
- Slide 23
- Motivation: As notions of communication/storage get more complex, modeling error as oblivious (to message/encoding/decoding) may be too simplistic. As notions of communication/storage get more complex, modeling error as oblivious (to message/encoding/decoding) may be too simplistic. Need more general models of error + encoding/decoding for such models. Need more general models of error + encoding/decoding for such models. Most pessimistic model: errors are worst-case. Most pessimistic model: errors are worst-case. 02/17/201023ECC: Progress/Challenges (@CMU)
- Slide 24
- Gap between worst-case & random errors In Shannon model, with binary channel: In Shannon model, with binary channel: Can correct upto 50% (random) errors. Can correct upto 50% (random) errors. ( 1-1/q fraction errors, if channel q-ary.) ( 1-1/q fraction errors, if channel q-ary.) In Hamming model, for binary channel: In Hamming model, for binary channel: Code with more than n codewords has distance at most 50%. Code with more than n codewords has distance at most 50%. So it corrects at most 25% worst-case errors. So it corrects at most 25% worst-case errors. ( (1 1/q) errors in q-ary case.) ( (1 1/q) errors in q-ary case.) Shannon model corrects twice as many errors: Shannon model corrects twice as many errors: Need new approaches to bridge gap. Need new approaches to bridge gap. 02/17/201024ECC: Progress/Challenges (@CMU)
- Slide 25
- Approach: List-decoding Main reason for gap between Shannon & Hamming: The insistence on uniquely recovering message. Main reason for gap between Shannon & Hamming: The insistence on uniquely recovering message. List-decoding: Relaxed notion of recovery from error. Decoder produces small list (of L) codewords, such that it includes message. List-decoding: Relaxed notion of recovery from error. Decoder produces small list (of L) codewords, such that it includes message. Code is (p,L) list-decodable if it corrects p fraction error with lists of size L. Code is (p,L) list-decodable if it corrects p fraction error with lists of size L. 02/17/201025ECC: Progress/Challenges (@CMU)
- Slide 26
- List-decoding Main reason for gap between Shannon & Hamming: The insistence on uniquely recovering message. Main reason for gap between Shannon & Hamming: The insistence on uniquely recovering message. List-decoding [Elias 57, Wozencraft 58]: Relaxed notion of recovery from error. Decoder produces small list (of L) codewords, such that it includes message. List-decoding [Elias 57, Wozencraft 58]: Relaxed notion of recovery from error. Decoder produces small list (of L) codewords, such that it includes message. Code is (p,L) list-decodable if it corrects p fraction error with lists of size L. Code is (p,L) list-decodable if it corrects p fraction error with lists of size L. 02/17/201026ECC: Progress/Challenges (@CMU)
- Slide 27
- What to do with list? Probabilistic error: List has size one w.p. nearly 1 Probabilistic error: List has size one w.p. nearly 1 General channel: Need side information of only O(log n) bits to disambiguate [Guruswami 03] General channel: Need side information of only O(log n) bits to disambiguate [Guruswami 03] (Altly if sender and receiver share O(log n) bits, then they can disambiguate [Langberg 04]). (Altly if sender and receiver share O(log n) bits, then they can disambiguate [Langberg 04]). Computationally bounded error: Computationally bounded error: Model introduced by [Lipton, Ding Gopalan L.] Model introduced by [Lipton, Ding Gopalan L.] List-decoding results can be extended (assuming PKI and some memory at sender) [Micali et al.] List-decoding results can be extended (assuming PKI and some memory at sender) [Micali et al.] 02/17/201027ECC: Progress/Challenges (@CMU)
- Slide 28
- List-decoding: State of the art [Zyablov-Pinsker/Blinovskii late 80s] [Zyablov-Pinsker/Blinovskii late 80s] There exist codes of rate 1 H q (p) - \epsilon that are (p,O(1))-list-decodable. There exist codes of rate 1 H q (p) - \epsilon that are (p,O(1))-list-decodable. Matches Shannons converse perfectly! (So cant do better even for random error!) Matches Shannons converse perfectly! (So cant do better even for random error!) But [ZP/B] non-constructive! But [ZP/B] non-constructive! 02/17/201028ECC: Progress/Challenges (@CMU)
- Slide 29
- Algorithms for List-decoding Not examined till 88. Not examined till 88. First results: [Goldreich-Levin] for Hadamard codes (non-trivial in their setting). First results: [Goldreich-Levin] for Hadamard codes (non-trivial in their setting). More recent work: More recent work: [S.96, Shokrollahi-Wasserman 98, Guruswami-S.99, Parvaresh-Vardy 05, Guruswami-Rudra 06] Decode algebraic codes. [S.96, Shokrollahi-Wasserman 98, Guruswami-S.99, Parvaresh-Vardy 05, Guruswami-Rudra 06] Decode algebraic codes. [Guruswami-Indyk 00-02] Decode graph- theoretic codes. [Guruswami-Indyk 00-02] Decode graph- theoretic codes. 02/17/201029 ECC: Progress/Challenges (@CMU)
- Slide 30
- Results in List-decoding q-ary case: q-ary case: [Guruswami-Rudra 06] Codes of rate R [Guruswami-Rudra 06] Codes of rate R correcting 1 R - fraction errors correcting 1 R - fraction errors with q = q( ) with q = q( ) Matches Shannon bound (except for q( ) ) Matches Shannon bound (except for q( ) ) Binary case: Binary case: c ! 3 : I mp l i e db y P arvares h - V ar d y 05 c = 4 : G uruswam i e t a l. 2000 9 C o d eso f ra t e c correc t i ng 1 2 f rac t i onerrors. 02/17/201030ECC: Progress/Challenges (@CMU) c = 3 : G uruswam i Rudra
- Slide 31
- Few lines about Guruswami-Rudra Code = Collated Reed-Solomon Code + Concatenation. Code = Collated Reed-Solomon Code + Concatenation. C o d emaps K ! N f or N q = C. M essage: D egree C K po l ynom i a l over F q. A l p h a b e t = F C q ;q ! 1, C cons t an t. 02/17/201031ECC: Progress/Challenges (@CMU)
- Slide 32
- Few lines about Guruswami-Rudra Special properties: Special properties: Is this code combinatorially good? Is this code combinatorially good? Algorithmically good!! (uses ideas from [S96,GS98,PV05 + new ones]. Algorithmically good!! (uses ideas from [S96,GS98,PV05 + new ones]. Can concatenate to reduce alphabet size. Can concatenate to reduce alphabet size. S i = f i C = ; ;:::; C 1 g. sa t i s esx q = xmo dh ( x ) f or i rre d uc i bl e h o fd egree CK. o f K, S i s D o B a ll so f ra d i us ( 1 o ( 1 )) ( N K ) h ave f ewco d ewor d s ? 02/17/201032ECC: Progress/Challenges (@CMU)
- Slide 33
- Few lines about Guruswami-Rudra Warnings: K, N, partition all very special. Warnings: K, N, partition all very special. C o d emaps K ! N f or N q = C. M essage: D egree C K po l ynom i a l over F q. A l p h a b e t = F C q ;q ! 1, C cons t an t. Encoding: \\ \indent First partition $\F_q$ into {\red special} sets $S_0,S_1,\ldots,S_N$, \\ \indent \indent with $|S_1| = \cdots = |S_N| = C$. \\ \indent Say $S_1 = \{\alpha_1,\ldots,\alpha_C\}$, $S_2 = \{\alpha_{C+1},\ldots,\alpha_{2C}\}$ etc.\\ \indent Encoding of $P$\\ \indent $\langle \langle P(x_1),\ldots,P(x_C) \rangle, \langle P(x_{C+1}),\ldots,P(x_{2C}) \rangle \cdots \rangle$ 02/17/201033ECC: Progress/Challenges (@CMU)
- Slide 34
- Major open question N o t e: I f runn i ng t i me i spo l y ( 1 = ) t h en t h i s i mp l i esa so l u t i on t o t h eran d omerrorpro bl emaswe ll. C ons t ruc t ( p ; O ( 1 )) l i s t - d eco d a bl e b i naryco d e o f ra t e 1 H ( p ) w i t h po l y t i me l i s t d eco d i ng.. 02/17/201034ECC: Progress/Challenges (@CMU)
- Slide 35
- Conclusions Coding theory: Very practically motivated problems; solutions influence (if not directly alter) practice. Coding theory: Very practically motivated problems; solutions influence (if not directly alter) practice. Many mysteries remain in combinatorial setting. Many mysteries remain in combinatorial setting. Significant progress in algorithmic setting, but many more questions to resolve. Significant progress in algorithmic setting, but many more questions to resolve. 02/17/201035ECC: Progress/Challenges (@CMU)
- Slide 36
- LDPC Codes D e nes E : f 0 ; 1 g k ! f 0 ; 1 g n. n l e f t ver t i cesn k r i g h t ver t i ces 010001111 C o d ewor d = 0 / 1 ass i gnmen tt o l e f t i f ne i g hb or h oo d o f r i g h t ver t i ces h aveevenpar i t y. R i g h t ver t i cesarepar i t yc h ec k s. G rap hh as l ow d ens i t y. H ence L ow- D ens i t y- P ar i t y- C h ec k C o d es. 02/17/201036ECC: Progress/Challenges (@CMU)
- Slide 37
- LDPC Codes 02/17/201037ECC: Progress/Challenges (@CMU)
- Slide 38
- LDPC Codes D e nes E : f 0 ; 1 g k ! f 0 ; 1 g n. n l e f t ver t i cesn k r i g h t ver t i ces 010001111 C o d ewor d = 0 / 1 ass i gnmen tt o l e f t i f ne i g hb or h oo d o f r i g h t ver t i ces h aveevenpar i t y. R i g h t ver t i cesarepar i t yc h ec k s. G rap hh as l ow d ens i t y. H ence L ow- D ens i t y- P ar i t y- C h ec k C o d es. 02/17/201038ECC: Progress/Challenges (@CMU)
- Slide 39
- LDPC Codes 010001111 D eco d i ng I n t u i t i on: P ar i t yc h ec kf a i l s ) somene i g hb orcorrup t e d. F ewne i g hb ors ) ass i gn i ng bl amewor k s. [ G a ll ager 63... S i pser- S p i e l man 92 ] : C orrec t ( 1 ) f rac t i onerrors. C urren t h ope: P i c k i ng d egreescare f u ll y w i lll ea d t oco d e / a l gor i t h m correc t i ngp f rac t i onran d omerrors 02/17/201039ECC: Progress/Challenges (@CMU)