1/11/05 1 capacity of noiseless and noisy two-dimensional channels paul h. siegel electrical and...

1/11/05 1

Capacity of Noiseless and Noisy Two-Dimensional Channels

Paul H. Siegel

Electrical and Computer Engineering

Center for Magnetic Recording Research

University of California, San Diego

1/11/05 2LANL Workshop

Outline

• Shannon Capacity

• Discrete-Noiseless Channels• One-dimensional

• Two-dimensional

• Finite-State Noisy Channel• One-dimensional

• Two-dimensional

• Summary


Claude E. Shannon


The Inscription


The Formula on the “Paper”

Capacity of a discrete channel with noise [Shannon, 1948]

For noiseless channel, Hy(x)=0, so:

Gaylord, MI: C = W log (P+N)/NBell Labs: no formula on paper

(“H = – p log p – q log q” on plaque)

(x))H – (H(x)Max C y

H(x)Max C


Discrete Noiseless Channels(Constrained Systems)

• A constrained system S is the set of sequences generated by walks on a labeled, directed graph G.

Telegraph channel constraints [Shannon, 1948]

DOT

DASH

DOT

DASH

LETTER SPACE

WORD SPACE


Magnetic Recording Constraints

Forbidden words F={101, 010}

01

0 0

1 1Biphase

Even

10

0 0

1 1

1

0

1

0

Forbidden word F={11}

Runlength constraints (“finite-type”: determined by finite

list F of forbidden words)

Spectral null constraints (“almost-finite-type”)


(d,k) runlength-limited constraints

• For , a (d,k) runlength-limited

sequence is a binary string such that:

• F={11} forbidden list corresponds to

1 0 0 0 1 0 1 0 0 1 0 1 0 0 0 1 0 1 0 0 0 0 1 0

kd0

),1(),( kd

ksd s1' econsecutivbetween '0#


Practical Constrained Codes

Finite-state encoder Sliding-block decoder (from binary data into S) (inverse mapping from S to data)

m data bits n code bits

Encoder Logic

Rate m:n

(states)

Decoder Logic

n bits

m bitsWe want: high rate R=m/n low complexity


Codes and Capacity

• How high can the code rate be?• Shannon defined the capacity of the constrained system S:

where N(S,n) is the number of sequences in S of length n.

Theorem [Shannon,1948] : If there exists a decodable code at rate R= m/n from binary data to S, then R C.

Theorem [Shannon,1948] : For any rate R=m/n < C there exists a block code from binary data to S with rate km:kn, for some integer k 1.

S,n Nn

Cn

log1

lim


Computing Capacity:Adjacency Matrices

• Let be the adjacency matrix of the graph G representing S.

• The entries in correspond to paths in G of length n.

GA

nGA

0

0

1

01

11GA


Computing Capacity (cont.)

• Shannon showed that, for suitable representing graphs G ,

where , i.e.,the spectral radius of the matrix .

• Assigning “transition probabilities” to the edges of G, the constrained system S becomes a Markov source x, with entropy H(x). Shannon proved that

and expressed the maximizing probabilities in terms of the spectral radius and corresponding eigenvector of .

GA ρlogC

GG lue of Aan eigenva is: λ λ maxAρ GA

x HmaxC

GA


Maxentropic Measure

• Let denote the largest real eigenvalue of , with corresponding eigenvector

• Then the maxentropic (capacity-achieving) transition probabilities are given by

• The stationary state distribution is expressed in terms of corresponding left and right eigenvectors.

MBBB ,,1

ij

i

jij

A

B

BP

GA


Computing Capacity (cont.)

• Example:

• More generally, , where is the

largest real root of the polynomial

and

),1(),( kd

6942.02

51log

C

kdkd logC ,, kd ,

kxxxxf dkkkd for ,1)( 1

,

. 1for ,12 ,1, dCC ddd


Constrained Coding Theorems

• Stronger coding theorems were motivated by the problem of constrained code design for magnetic recording.

Theorem[Adler-Coppersmith-Hassner, 1983] Let S be a finite-type constrained system. If m/n C, then there exists a rate m:n sliding-block decodable, finite-state encoder. (Proof is constructive: state-splitting algorithm.)

Theorem[Karabed-Marcus, 1988]Ditto if S is almost-finite-type.(Proof not so constructive…)


Two-Dimensional Constrained Systems

• Band-recording and page-oriented recording technologies require 2-dimensional constraints, for example:

• Two-Dimensional Optical Storage (TwoDOS) - Philips

• Holographic Storage - InPhaseTechnologies

• Patterned Magnetic Media – Hitachi, Toshiba, …

• Thermo-Mechanical Probe Array – IBM


TwoDOS

Courtesy of Wim Coene, Philips Research


Constraints on the Integer Lattice Z2

• : constraint in x - y directions:

),1(),( kd

Hard-Square ModelHard-Square Model

11 11

11

11

11

11

1111

11

11

11

11

1111

11

11 Independent SetsIndependent Sets

11,

1

1F

,1sqS


(d,k) Constraints on the Integer Lattice Z2

• For 2-dimensional (d,k) constraints , the capacity is given by:

• The only nontrivial (d,k) pairs for which is known precisely are those with zero capacity, namely [Kato-Zeger, 1999] :

mn

N limC

kdnm

nm

kd,,

,

,

kdC ,

2 , 0, dkC kd

kdsqS ,

01, ddC , d>0, d>0


(d,k) Constraints on Z2 – Capacity Bounds

• Transfer matrix methods provide numerical bounds on [Calkin-Wilf, 1998] , [Nagy-Zeger, 2000]

• Variable-rate “bit-stuffing” encoders for yield best known lower bounds on for d >1 [Halevy, et al., 2004]:

,dC

6858789116180 7558789116170 ,1 .. C

,dsqS

,1C

d Lower bound d Lower bound

2 0.4267 4 0.2858

3 0.3402 5 0.2464

)1(121

)(max lim /},min{12210,

,dnmdpnm

d oppdp

phC


2-D Bit-Stuffing RLL Encoder

• Source encoder converts binary data to i.i.d bit stream (biased bits) with , rate penalty .

• Bit-stuffing encoder inserts redundant bits which can be identified uniquely by decoder.

• Encoder rate R(p) is a lower bound of the capacity. (For d=1, we can determine R(p) precisely.)

Data Source

Bit-stuffingDecoder

SourceEncoder

B it-stuffingEncoder

DiscreteNoiselessChannel

SourceDecoder

Data S ink

pp, 10Pr 1Pr )( ph

),( d


2-D Bit-Stuffing (1,∞) RLL Encoder

• Biased sequence: 1 1 1 0 0 0 1 0 0 1 0 0 0 0 1 1 0 0 0

11 00

00

00

00

0000 11

11

11

11

11

00

00

00

11

00

00

00

00

00

00

00

00

00

00

00

0000

00

00

Optimal bias Pr(1) = p = 0.3556

R(p)=0.583056 (within 1% of capacity)

Optimal bias Pr(1) = p = 0.3556

R(p)=0.583056 (within 1% of capacity)


Enhanced Bit-Stuffing Encoder

• Use 2 source encoders, with parameters p0 , p1 .

00

00

00 00

00

11

Optimal bias

Pr(1) = p0 = 0.328167

Optimal bias

Pr(1) = p0 = 0.328167

Optimal bias

Pr(1) = p1 = 0.433068

Optimal bias

Pr(1) = p1 = 0.433068

R(p0 , p1)=0.587277 (within 0.1% of capacity)R(p0 , p1)=0.587277 (within 0.1% of capacity)


Non-Isolated Bit (n.i.b.) Constraint on Z2

• The non-isolated bit constraint is defined by the forbidden set:

• Analysis of the coding ratio of a bit-stuffing encoder yields:

0.91276 ≤ Csqnib ≤ 0.93965

1

101

1

,

0

010

0

F

nibsqS


Constraints on the Hexagonal Lattice A2

• : constraints:

),1(),( kd

Hard-Hexagon ModelHard-Hexagon Model

,1hexS

1

1,11,

1

1F

111

1

111

1

1


Hard Hexagon Capacity

• Capacity of hard hexagon model is known precisely! [Baxter,1980]*

212

4

22

3

22

2

21254511

1221

12211

12211

1134

aaaa

cccc

cccc

c

,1hexC

31

3131

21

31

)1()1(8

3

4

1

3311979

2501

11363

124

bbac

b

a

and where,log 4321,1

hhexC

So, So, 480767622.0,1 hexC


Hard Hexagon Capacity

• Alternatively, the hard hexagon entropy constant satisfies a degree-24 polynomial with (big!) integer coefficients.

• Baxter does offer this disclaimer regarding his derivation, however:

*“It is not mathematically rigorous, in that certain analyticity properties of κ are assumed, and the results of Chapter 13 (which depend on assuming that various large-lattice limits can be interchanged) are used. However, I believe that these assumptions, and therefore (14.1.18)-(14.1.24), are in fact correct.”

*“It is not mathematically rigorous, in that certain analyticity properties of κ are assumed, and the results of Chapter 13 (which depend on assuming that various large-lattice limits can be interchanged) are used. However, I believe that these assumptions, and therefore (14.1.18)-(14.1.24), are in fact correct.”


(d,k) Constraints on A2 – Capacity Bounds

• Zero capacity region partially known [Kukorelly-Zeger, 2001].

• Variable-to-fixed length “bit-stuffing” encoders for

yield best known lower bounds on for d>1

,dhexS

,dhexC

d Lower bound d Lower bound

2 0.3387 4 0.2196

3 0.2630 5 0.1901

[Halevy, et al., 2004]:[Halevy, et al., 2004]:

)1(31

)(max lim /},min{210,

,dnm

pnm

dhex o

pdp

phC


Practical 2-D Constrained Codes

• There is no comprehensive algorithmic theory for constructing encoders and decoders for 2-D constrained systems.

• Very efficient bit-stuffing encoders have been defined and analyzed for several 2-D constraints, but they are not suitable for practical applications [Roth et al., 2001] , [Halevy et al., 2004] , [Nagy-Zeger, 2004].

• Optimal block codes with m x n rectangular code arrays have been designed for small values of m and n, and some finite-state encoders have been designed, but there is no generally applicable method [Demirkan-Wolf, 2004] .

• There is no comprehensive algorithmic theory for constructing encoders and decoders for 2-D constrained systems.

• Very efficient bit-stuffing encoders have been defined and analyzed for several 2-D constraints, but they are not suitable for practical applications [Roth et al., 2001] , [Halevy et al., 2004] , [Nagy-Zeger, 2004].

• Optimal block codes with m x n rectangular code arrays have been designed for small values of m and n, and some finite-state encoders have been designed, but there is no generally applicable method [Demirkan-Wolf, 2004] .


Concluding Remarks

• The lack of convenient graph-based representations of 2-D constraints prevents the straightforward extension of 1-D techniques for analysis and code design.

• There are strong connections to statistical physics that may open up new approaches to understanding 2-D constrained systems (and, perhaps, vice-versa).


Noisy Finite-State ISI Channels (1-Dim.)

• Binary input process

• Linear intersymbol interference

• Additive, i.i.d. Gaussian noise

1

0

][][][][n

k

inkixkhiy

][ix

][ih

)( 2,0~][ Nin


Example: Partial-Response Channels

• Impulse response:

• Example: Dicode channel

1

0

)1)(1(][)(

NiN

i

DDDihDh

)1()( DDh

-1

10 0

00 11


Entropy Rates

• Output entropy rate:

• Noise entropy rate:

• Conditional entropy rate:

nn YH

nYH 1

1lim

02

1eNlogNH

NHXYHn

XYH nn

n

11 |1

lim|


Mutual Information Rates

• Mutual information rate:

• Capacity:

• Symmetric information rate (SIR): Inputs are constrained to be independent, identically distributed, and equiprobable binary digits.

NHYHX|YHYHY;XI

][ixX

Y;XImaxC

XP


Finding the Output Entropy Rate

• For one-dimensional ISI channel model:

and

where

n

nYH

nYH 1

1lim

nnn yYpEYH 111 log

nYYYY n ,2,11


Sample Entropy Rate

• If we simulate the channel N times, using inputs with specified (Markovian) statistics and generating output realizations

then

converges to with probability 1 as .N nYH 1

Nknyyyy kkkk ,,2,1,][,,]2[,]1[ )()()()(

N

k

kypN 1

)(log1


Computing Sample Entropy Rate

• The forward recursion of the sum-product (BCJR)algorithm can be used to calculate the probability p(y1

n) of a sample realization of the channel output.

• In fact, we can write

where the quantity is precisely the normalization constant in the (normalized) forwardrecursion.

n

i

ii

n y|yplogn

yplogn 1

111

11

11i

i y|yp


Computing Entropy Rates

• Shannon-McMillan-Breimann theorem implies

as , where is a single long sample realization of the channel output process.

ny1

YHyplogn .s.a

n 11

n


SIR for Partial-Response Channels


Computing the Capacity

• For Markov input process of specified order r , this technique can be used to find the mutual informationrate. (Apply it to the combined source-channel.)

• For a fixed order r , [Kavicic, 2001] proposed a Generalized Blahut-Arimoto algorithm to optimize the parameters of the Markov input source.

• The stationary points of the algorithm have been shown to correspond to critical points of the information rate curve [Vontobel,2002] .


Capacity Bounds for Dicode h(D)=1-D


Markovian Sufficiency

Remark: It can be shown that optimized Markovian processes whose states are determined by their previous r symbols can asymptotically achieve the capacity of finite-state intersymbol interference channels with AWGN as the order r of the input process approaches .

(This generalizes to 2 dimensional channels.)

[Chen-Siegel, 2004]


Capacity and SIR in Two Dimensions

• In two dimensions, we could estimate by calculating the sample entropy rate of a very large simulated output array.

• However, there is no counterpart of the BCJR algorithm in two dimensions to simplify the calculation.

• Instead, conditional entropies can be used to derive upper and lower bounds on .

YH

YH


Examples of Past{Y[i,j]}


Conditional Entropies

• For a stationary two-dimensional random field Y on the integer lattice, the entropy rate satisfies:

(The proof uses the entropy chain rule. See [5-6])

• This extends to random fields on the hexagonal lattice, via the natural mapping to the integer lattice.

j,i,kj,i YPastYHYH |


Upper Bound on H(Y)

• For a stationary two-dimensional random field Y,

where

1Ul,k

kHminYH

j,il,kj,i YPastYHYH |Ul,k 1


Two-Dimensional Boundary of Past{Y[i,j]}

• Define to be the boundary

of .

• The exact expression for

is messy, but the geometrical concept is

simple.

j,il,k YPast

j,il,kStrip Y

j,il,kStrip Y


Two-Dimensional Boundary of Past{Y[i,j]}


Lower Bound on H(Y)

• For a stationary two-dimensional hidden Markov field Y,

where

and is the “state information” for

the strip .

1Ll,k

kHmaxYH

j,iYl,kStX,j,il,kj,i YPastYHYH |Ll,k 1

j,iYl,kStX

j,iYl,kStrip


Computing the SIR Bounds

• Estimate the two-dimensional conditional entropies

over a small array.

• Calculate to get

for many realizations of output array.

• For column-by-column ordering, treat each row

as a variable and calculate the joint probability

row-by-row using the BCJR forward

recursion.

BAP

BAH

BPBAP ,,

iY

mYYYP ,,, 21


2x2 Impulse Response

• “Worst-case” scenario - large ISI:

• Conditional entropies computed from 100,000 realizations.

• Upper bound:

• Lower bound:

(corresponds to element in middle of last column)

5.05.0

5.05.0],[1 jih

1 ,log

2

1min 0

10,3,7,7,1,2 eNH U

01

0,3,7,7,1,2 log2

1eNH L


SIR Bounds for 2x2 Channel


Computing the SIR Bounds

• The number of states for each variable increases exponentially with the number of columns in the

array.

• This requires that the two-dimensional impulse response have a small support region.

• It is desirable to find other approaches to computing bounds that reduce the complexity, perhaps at the cost of weakening the resulting bounds.


Alternative Upper Bound

• Modified BCJR approach limited to small impulse response support region.

• Introduce “auxiliary ISI channel” and bound

where

and is an arbitrary conditional

probability distribution.

2Ul,kHYH

ydjilk

jiyqjilk

jiypH yPastyPastUlk

,

,,log,

,,, |2

,

ji

lkjiyq yPast ,

,, |


3x3 Impulse Response

• Two-DOS transfer function

• Auxiliary one-dimensional ISI channel with memory

length 4.

• Useful upper bound up to Eb/N0 = 3 dB.

011

121

110

101],[2 jih


SIR Upper Bound for 3x3 Channel


Concluding Remarks

• Recent progress has been made in computing information rates and capacity of 1-dim. noisy finite-state ISI channels.

• As in the noiseless case, the extension of these results to 2-dim. channels is not evident.

• Upper and lower bounds on the SIR of two-dimensional finite-state ISI channels have been developed.

• Monte Carlo methods were used to compute the bounds for channels with small impulse response support region.

• Bounds can be extended to multi-dimensional ISI channels.

• Further work is required to develop computable, tighter bounds for general multi-dimensional ISI channels.


References

1. D. Arnold and H.-A. Loeliger, “On the information rate of binary-input channels with memory,” IEEE International Conference on Communications, Helsinki, Finland, June 2001, vol. 9, pp.2692-2695.

2. H.D. Pfister, J.B. Soriaga, and P.H. Siegel, “On the achievable information rate of finite state ISI channels,” Proc. Globecom 2001, San Antonio, TX, November2001, vol. 5, pp. 2992-2996.

3. V. Sharma and S.K. Singh, “Entropy and channel capacity in the regenerative setup with applications to Markov channels,” Proc. IEEE International Symposium on Information Theory, Washington, DC, June 2001, p. 283.

4. A. Kavcic, “On the capacity of Markov sources over noisy channels,” Proc. Globecom 2001, San Antonio, TX, November2001, vol. 5, pp. 2997-3001.

5. D. Arnold, H.-A. Loeliger, and P.O. Vontobel, “Computation of information rates from finite-state source/channel models,” Proc.40th Annual Allerton Conf. Commun., Control, and Computing, Monticello, IL, October 2002, pp. 457-466.


References

6. Y. Katznelson and B. Weiss, “Commuting measure-preserving transformations,” Israel J. Math., vol. 12, pp. 161-173, 1972.

7. D. Anastassiou and D.J. Sakrison, “Some results regarding the entropy rates of random fields,” IEEE Trans. Inform. Theory, vol. 28, vol. 2, pp. 340-343, March 1982.

1/11/05 1 capacity of noiseless and noisy two-dimensional channels paul h. siegel electrical and...

Documents