elec 619a selected topics in digital communications ...agullive/coding3.pdf · there are simple...
TRANSCRIPT
-
ELEC 619ASelected Topics in Digital Communications:
Information Theory
Error Correcting Codes II
What is Possible?
-
Notation and Examples
An (n,k,d) - code C is a linear code such that• n - is the length of the codewords
• k - is the number of data symbols in a codeword
• d - is the minimum distance in C
Examples:C1 = {00, 01, 10, 11} is a (2,2,1)-code.
C2 = {000, 011, 101, 110} is a (3,2,2)-code.
C3 = {00000, 11100, 00111, 11011} is a (5,2,3)-code.
A good (n,k,d) code has small n and large k and d.
2
-
(5,2,3) Code
• k x n Generator matrix
•
3
-
4
-
Advantages of Linear Block Codes1. The minimum distance h(C) is easy to compute if C is a linear code.
2. Linear codes have simple specifications.
• To specify a non-linear code usually all codewords have to be listed.
• To specify a linear [n,k] -code it is enough to list k linearly independent codewords.
Definition A k n matrix whose rows form a basis for a linear [n,k]-code (subspace) C is said to be a generator matrix for C.
Example A generator matrix for the code
and for the code
3. There are simple encoding/decoding procedures for linear codes.
}111{ is 111
000
C
5
-
Important Linear Block Codes
There are many classes of practical linear block codes:
• Hamming codes
• Cyclic codes
• BCH codes
• Reed-Solomon codes
• LDPC codes
• Turbo codes
• …
6
-
Equivalence of Linear CodesDefinition Two linear codes are called equivalent if one can be obtained from the other by the following operations:
(a) permutation of the positions of the code;
(b) multiplication of symbols in a fixed position by a non-zero scalar.
Theorem Two k n matrices generate equivalent linear (n,k) codes if one matrix can be obtained from the other by a sequence of the following operations:
(a) permutation of the rows
(b) multiplication of a row by a non-zero scalar
(c) addition of one row to another
(d) permutation of columns
(e) multiplication of a column by a non-zero scalar
Proof Operations (a) - (c) just replace one basis by another. The last two operations convert a generator matrix to one of an equivalent code.
7
-
Equivalent Codes
Distances between codewords are unchanged by equivalence operations (a) - (b). Consequently, equivalent linear codes have the same parameters (n,k,d) (and correct the same number of errors).
8
-
Systematic Codes• For a systematic block code the dataword appears
unaltered in the codeword – usually at the start
• The generator matrix has the structure
P | I
..1..00
................
..0..10
..0..01
G
21
22221
11211
kknkk
kn
kn
ppp
ppp
ppp
k n-k
• P is often referred to as parity matrix
9
-
Systematic CodesExample
][
1110001
1100010
1010100
0111000
4 PIG
0101011010 1011
1000111100 0111
mmGm
mmGm
k n - k
information digits redundancy(parity-check digits)
10
-
Systematic Codes
Theorem Let G be a generator matrix of an [n,k] -code. Then by
operations (a) - (e) the matrix G can be transformed into the form
[ Ik | A ]
where Ik is the k k identity matrix, and A is a k (n - k) matrix.
Example
11
-
Encoding with a Systematic Code
• vectormatrix multiplication
• Encoding of a message m = (m1, … ,mk) with C
Example Let C be a (7,4) code with generator matrix
12
| ( | )kc m G m I P m b
-
Example
A message (m1, m2, m3, m4) is encoded as:
(0 0 0 0) is encoded as ………………………….. ?
(1 0 0 0) is encoded as ………………………….. ?
(1 1 1 0) is encoded as ………………………….. ?
13
-
Dual Codes
Example
^
Theorem Suppose C is a linear (n,k) code, then the dual code C^ is a linear (n,n-k) code.
111
000 then
101
110
011
000
66
CC
14
-
Dual Codes
For the (n,1) repetition code C with generator matrix
G = [1,1, … ,1]
the dual code C^ is an (n,n-1) SPC code with generator matrix G^
described by
^
The rows of G are orthogonal to the rows of G^
1...0001
..
0...0101
0...0011
G
15
-
Parity Check MatricesDefinition A parity-check matrix H for an (n,k) code C is a generator matrix of C^.
Theorem If H is a parity-check matrix of C, then
C = {x V(n,2) | xHT = 0}
and therefore any linear code is completely specified by a parity-check matrix.
Example Parity-check matrix for
The rows of a parity check matrix are parity checks on codewords. They specify that certain linear combinations of the coordinates of every codeword are zero.
16
6 is 1 1 1C
-
Systematic Parity Check Matrices
If G = (Ik | P) is the standard form (systematic) generator matrix of a binary (n,k) code C, then a parity check matrix for C is H = (PT | In-k)
Example
1011
1110
0111
110
011
111
101
34 IHIG
17
-
Decoding Linear Codes• One possibility is a ROM look-up table
• In this case the received word r is used as an address
• Example – Even single parity check codeAddress Entry
000000 0
000001 1
000010 1
000011 0
……… .
• The output is an error flag, i.e., 0 – codeword valid
• If no error, the dataword is the first k codeword bits
• For an error correcting code, the ROM can also store datawords after correction
18
-
Standard Array
• The Standard Array is constructed as follows
c1 (all zero)
e1
e2
e3
….
en-k
c2+e1
c2+e2
c2+e3
……
c2+en-k
c2
cM-1+e1
cM-1+e2
cM-1+e3
……
cM-1+en-k
cM-1
……
……
……
……
……
…… cM
cM+e1
cM+e2
CM+e3
…..
cM+en-k
• The array has 2k columns (equal to the number of valid codewords) and 2n-k rows (the number of correctable error patterns)
19
-
Standard Array
• The standard array is formed by initially choosing ei to be– All 1 bit error patterns
– All 2 bit error patterns
……
• Ensure that each new error pattern is not already in the array
• All correctable error patterns will appear as coset leaders
20
-
Standard Array Decoding
• Assume that the received word is r = c2 + e3(shown in bold in the standard array)
• The most likely codeword is the one at the top of the column containing c2 + e3
• The corresponding error pattern is the cosetleader at the start of the row containing c2 + e3
• Can be implemented using a look-up table (ROM) which maps all words in the array to the most likely codeword (the one at the top of the column containing the received word).
21
-
Standard Array for the (5,2,3) Code
22
-
Syndromes• For a received word r, we need a simpler method
to determine the codeword estimate
• To do this we use the Syndrome, s of a received word r
s = rHT
• If c was corrupted by an error vector, e, then
r = c + e
and
s = rHT =(c + e) HT = cHT + eHT
s = 0 + eHT
• The syndrome depends only on the error vector e
23
-
Syndromes and Cosets
Definition Suppose H is a parity-check matrix of an (n,k) code C. Then for any r V(n,q) the following word is called the syndrome of r
S(r) = rHT
Lemma Two words have the same syndrome iffthey are in the same coset.
Proof
S(r) = rHT = (c+e)HT = eHT
and e is the coset leader
24
-
Syndromes• We can add the same error pattern to different
codewords and get the same syndrome.– There are 2(n - k) syndromes but 2n error patterns
– Example: for a (5,2) code there are 8 syndromes and 32 error patterns
– Example: a (7,4) code has 8 syndromes and 128 error patterns• 8 syndromes provides values to indicate single errors in any of
the 7 bit positions as well as the zero value to indicate no errors
• Only need to determine which error pattern corresponds to the syndrome
25
-
Syndromes for the (5,2,3) Code
26
-
Syndrome Decoding
• This block diagram shows the syndrome decoder implementation
Compute syndrome
Look-up table
+r
s e
c
27
-
Syndrome Decoding• For systematic linear block codes
G = [ I | P] so H = [-PT | I]
• Example (7,4,3) code
1111000
0110100
1010010
1100001
P|I G
1001011
0101101
0011110
I|P- H T
28
-
Syndrome Decoding - Example• For a received word r = (1101001)
• So r is a valid codeword
000
100
010
001
111
011
101
110
1001011rH s T
29
-
Syndrome Decoding - Example• The same codeword, this time with an error
in the first bit position r = (1101000)
100
100
010
001
111
011
101
110
0001011rH s T
• Syndrome 001 indicates an error in bit 7 of the received word
30
-
Single Error Correcting Codes
31
-
Hamming Codes• (7,4,3) Hamming code
1111000
0110100
1010010
1100001
P|I G
1001011
0101101
0011110
I|P- H T
32
-
Hamming Code Parameters
33
-
Comments about H• The minimum distance of the code is equal
to the minimum number of columns of H which sum to zero
• For any codeword c
0d...dd
d
.
d
d
],...,,[cH 1n11100
1
1
0
110
T
n
n
n cccccc
where do, d1, …, dn-1 are the column vectors of H
• cHT is a linear combination of the columns of H
34
-
Comments about H
• For a codeword of weight w (w ones), cHT is a linear combination of w columns of H.
• Thus we have a one-to-one mapping between weight w codewords and linear combinations of w columns of H that sum to 0.
• The minimum value of w which results in cHT=0, i.e., codeword c with weight w,determines that dmin = w
35
-
Example
• For the (7,4,3) code, a codeword with weight dmin = 3 is given by the first row of G, i.e., c = (1000011)
• The linear combination of the first and last 2 columns in H gives
(011)+(010)+(001) = 0
• Thus a minimum of 3 columns (= dmin) are required to get a zero value for cHT
36
-
37
Hamming codesDefinitionDefinition Let m be an integer and H be an m (2m - 1) matrix with columns which are the non-zero distinct words from V(m,2). The code having H as its parity-check matrix is called a binary Hamming code and is denoted by Ham(m,2).
The Hamming code Ham(m,2) is a (2m - 1, 2m – 1 – m,3) code
m = n - k
The coset leaders are precisely the words of weight 1. The syndrome of the word 0…010…0 with 1 in j -th position and 0 otherwise is the transpose of the j -th column of H.
111101
0112,2
GHHam
-
38
Decoding Hamming Codes
For the case that the columns of H are arranged in order of increasing binary numbers that represent the column numbers 1 to 2m - 1
• Step 1 Given r compute the syndrome S(r) = rHT
• Step 2 If S(r) = 0, then r is assumed to be the codeword sent
• Step 3 If S(r) 0, then assuming a single error, S(r) gives the binary position of the error
-
39
ExampleFor the Hamming code given by the parity-check matrix
the received wordr = 1101011
has syndromeS(r) = 110
and therefore the error is in the sixth position.
Hamming codes were originally used to deal with errors in long-distance telephone calls.
1010101
1100110
1111000
H
-
The Main Coding Theory Problem
A good (n,M,d) code has small n, large M and large d.
The main coding theory problem is to optimize one of the parameters n, M, d for given values of the other two.
For linear codes, a good (n,k,d) code has small n, large k and large d.
The main coding theory problem for linear codes is to optimize one of the parameters n, k, d for given values of the other two.
40
-
Hamming Bound
The number of word of length n and weight i is
!( )
!( )!
ni
n
i n i
n n n n0 1 2 t
Let be a word of length , for 0 , the number
of words of length a distance at most from is
( ) ( ) ( ) .... ( )
v n t n
n t v
41
-
Hamming Bound• For C a code of length n, distance d = 2t+1 or 2t+2
• Example: Give an upper bound on the size of a linear code C of length n=6 and distance d=3
• This gives but the size of a linear code C must be a power of 2 so
n
n n n n
0 1 2 t
2C
( ) ( ) ( ) .... ( )
62 64C
6 6 7
0 1
| | 8C
| | 9C
42
-
Hamming Bound for Binary Linear Codes
• A code that meets the Hamming bound with equality is called perfect
• Which codes meet this bound?
43
k n n n n n
0 1 2 t
t n n-k
i i=0
2 ( ) ( ) ( ) .... ( ) 2
( ) 2
-
Hamming Bound for Nonbinary Linear Codes
44
k n n n 2 n t n
0 1 2 t
t n i n-k
i i=0
q ( ) ( )(q 1) ( )(q 1) .... ( )(q 1) q
( )(q 1) q
-
Golay Codes
• Marcel Golay considered the problem of perfect codes in 1949
• He found three possible solutions to equality for the Hamming bound– q = 2, n = 23, t = 3
– q = 2, n = 90, t = 2
– q = 3, n = 11, t = 2
• Only the first and third codes exist [Van Lint and Tietäväinen, 1973]
45
-
• There exists a code of length n, distance d, and
M codewords with
• The bound also holds for linear codes
– Let k be the largest integer such that
then an (n,k,d) code exists
Gilbert Bound
n
d-1 n
jj=0
2M=
( )
nk
d-1 n
jj=0
22
( )
46
-
• For linear codes, the Gilbert bound can be improved
• There exists a linear code of length n, dimension k and
distance d if
• Proof: construct a parity check matrix based on this condition.
• Thus if k is the largest integer such that
then an (n,k,d) code exists
Gilbert-Varshamov Bound
n-1 n-1 n-1 n-k 0 1 d-2( )+( )+.....+( ) 2
nk
d-2 n-1
jj=0
22
( )
47
-
• Does there exist a linear code of length n=9, dimension k=2, and distance d=5? Yes, because
• Give a lower and an upper bound on the
dimension, k, of a linear code with n=9 and d=5.
• G-V lower bound: but |C| is a power of 2
so |C| ≥ 4
Example
292128933
8
2
8
1
8
0
8
9k 22 5.55
93
48
-
Example (Cont.)• Hamming upper bound:
|C| ≤
but |C| is a power of 2 so |C| ≤ 8• From the tables, the optimal codes are
– (9,2,6) • so the G-V bound is exceeded – sufficient but not necessary
– (9,3,4)• so the Hamming bound is necessary but not sufficient for a
code to exist
49
13.113691
512
2
9
1
9
0
9
29
-
• Does a (15, 7, 5) linear code exist?
• Check the G-V bound
• G-V bound does not hold, so it does not tell us whether or
not such a code exists.
• But actually such a code does exist - the (15,7,5) BCH code
G-V Bound and Codes
2562247036491141
3
14
2
14
1
14
0
14
2
1...
1
1
0
1
715
kn
d
nnn
50
-
The Nordstrom-Robinson Code
• Adding an overall parity check to the (15,7,5) code gives a (16,7,6) linear code
– This code is optimal
– The G-V bound says a (16,5,6) code exists
• A (16,256,6) nonlinear code exists
– Twice as many codewords as the best linear code
51
-
Singleton Bound• Singleton bound (upper bound)
For any (n, k, d) linear code, d-1≤ n-k
k ≤ n-d+1 or |C| ≤ 2n-d+1
Proof: the parity check matrix H of an (n,k,d) linear code is an n-k by n matrix such that every d-1 columns of H are independent Since the columns have length n-k, we can never have more than n-k independent columns. Hence d-1 ≤ n-k.
• Theorem
For an (n, k, d) linear code C, the following are equivalent:• d = n-k+1
• Every n-k columns of the parity check matrix are linearly independent
• Every k columns of the generator matrix are linearly independent
• C is Maximum Distance Separable (MDS) (definition: if d=n-k+1)
52
-
Reed-Muller Codes
• Parameters:
• Example: m = 3, r = 1
n = 8, d = 4, k = 1 + 3 = 4
(8,4,4) code
53
r
m m-r
i=0
n=2 d=2 k= ( )mi
1 1 1 1 1 1 1 1
0 0 0 0 1 1 1 1
0 0 1 1 0 0 1 1
0 1 0 1 0 1 0 1
G
-
Reed-Muller Codes• r-th order code, length 2m, 0≤r≤m, RM(r, m)
• RM(0, m) = {00…0, 11…1} → repetition code
• RM(m, m) = V(2,m) → vector space
• RM(r, m) =
• Examples
{( , ) | ( , 1), ( 1, 1)},0x x y x RM r m y RM r m r m
2
4
(0, 0) {0,1}
(0,1) {00,11} (1,1) {00, 01,10,11}
(0,2) {0000,1111} (2,2)
(1,2) {( , ) | {00, 01,10,11}, {00,11}}
RM
RM RM K
RM RM K
RM x x y x y54
-
Reed-Muller Codes
• RM(1,2) → (4,3,2) code
• Codewords: 0000 0011
0101 0110
1010 1001
1111 1100
55
1 1 1 1
0 0 1 1
0 1 0 1
G
-
• Example: G(1,3)
( , 1) ( , 1)0
-
• RM(r-1,m) is contained in RM(r,m), r>0
• Example: m=3, r=1
• RM(1,3) → (8,4,4) extended Hamming code contains RM(0,3) → (8,1,8) repetition code
• The dual code of RM(r,m) is RM(m-1-r,m) r< m
• Example: m=3,r=0
• RM(0,3) → (8,1,8) repetition code
• RM(2,3) → (8,7,2) SPC code
Properties of Reed-Muller Codes
57
-
1971: Mariner 9Mariner 9 used a (32,6,16) Reed-Muller code to transmit its grey images of Mars.
camera rate:100,000 bits/second
transmission speed:16,000 bits/second
58
-
1979+: Voyager I and IIVoyager I and II used a [24,12,8] extended Golay code to send its color images of Jupiter and Saturn.
Voyager 2 traveled further to Uranus and Neptune. Because of the higher error rate it switched to a more robust Reed-Solomon code.
59
-
Modern Codes
More recently Turbo codeswere invented, and are used in 3G cell phones, (future) satellites,and in the Cassini-Huygens spaceprobe [1997–].
Other modern codes: Expander, Fountain, RaptorLDPC are being proposed for future communications standards
60
-
NASA Error Correcting CodesIm
pe
rfe
ctn
ess
, dB
-Sig
nal
to
No
ise
Rat
io
61