info theory lectures

8/3/2019 Info Theory Lectures

1/75

Information Theory and Coding

Computer Science Tripos Part II, Michaelmas Term11 Lectures by J G Daugman

1. Foundations: Probability, Uncertainty, and Information

2. Entropies Defined, and Why they are Measures of Information

3. Source Coding Theorem; Prefix, Variable-, & Fixed-Length Codes

4. Channel Types, Properties, Noise, and Channel Capacity

5. Continuous Information; Density; Noisy Channel Coding Theorem

6. Fourier Series, Convergence, Orthogonal Representation

7. Useful Fourier Theorems; Transform Pairs; Sampling; Aliasing

8. Discrete Fourier Transform. Fast Fourier Transform Algorithms

9. The Quantized Degrees-of-Freedom in a Continuous Signal

10. Gabor-Heisenberg-Weyl Uncertainty Relation. Optimal Logons

11. Kolmogorov Complexity and Minimal Description Length


2/75


J G Daugman

Prerequisite courses: Probability; Mathematical Methods for CS; Discrete Mathematics

Aims

The aims of this course are to introduce the principles and applications of information theory.The course will study how information is measured in terms of probability and entropy, and therelationships among conditional and joint entropies; how these are used to calculate the capacityof a communication channel, with and without noise; coding schemes, including error correctingcodes; how discrete channels and measures of information generalise to their continuous forms;the Fourier perspective; and extensions to wavelets, complexity, compression, and efficient codingof audio-visual information.

Lectures

Foundations: probability, uncertainty, information. How concepts of randomness,redundancy, compressibility, noise, bandwidth, and uncertainty are related to information.Ensembles, random variables, marginal and conditional probabilities. How the metrics ofinformation are grounded in the rules of probability.

Entropies defined, and why they are measures of information. Marginal entropy, joint entropy, conditional entropy, and the Chain Rule for entropy. Mutual informationbetween ensembles of random variables. Why entropy is the fundamental measure of infor-mation content.

Source coding theorem; prefix, variable-, and fixed-length codes. Symbol codes.The binary symmetric channel. Capacity of a noiseless discrete channel. Error correctingcodes.

Channel types, properties, noise, and channel capacity. Perfect communicationthrough a noisy channel. Capacity of a discrete channel as the maximum of its mutualinformation over all possible input distributions.

Continuous information; density; noisy channel coding theorem. Extensions of thediscrete entropies and measures to the continuous case. Signal-to-noise ratio; power spectraldensity. Gaussian channels. Relative significance of bandwidth and noise limitations. The

Shannon rate limit and efficiency for noisy continuous channels. Fourier series, convergence, orthogonal representation. Generalised signal expan-

sions in vector spaces. Independence. Representation of continuous or discrete data bycomplex exponentials. The Fourier basis. Fourier series for periodic functions. Examples.

Useful Fourier theorems; transform pairs. Sampling; aliasing. The Fourier trans-form for non-periodic functions. Properties of the transform, and examples. NyquistsSampling Theorem derived, and the cause (and removal) of aliasing.

Discrete Fourier transform. Fast Fourier Transform Algorithms. Efficient al-gorithms for computing Fourier transforms of discrete data. Computational complexity.

Filters, correlation, modulation, demodulation, coherence.


3/75

The quantised degrees-of-freedom in a continuous signal. Why a continuous sig-nal of finite bandwidth and duration has a fixed number of degrees-of-freedom. Diverseillustrations of the principle that information, even in such a signal, comes in quantised,countable, packets.

Gabor-Heisenberg-Weyl uncertainty relation. Optimal Logons. Unification ofthe time-domain and the frequency-domain as endpoints of a continuous deformation. The

Uncertainty Principle and its optimal solution by Gabors expansion basis of logons.Multi-resolution wavelet codes. Extension to images, for analysis and compression.

Kolmogorov complexity. Minimal description length. Definition of the algorithmiccomplexity of a data sequence, and its relation to the entropy of the distribution fromwhich the data was drawn. Fractals. Minimal description length, and why this measure ofcomplexity is not computable.

Objectives

At the end of the course students should be able to

calculate the information content of a random variable from its probability distribution

relate the joint, conditional, and marginal entropies of variables in terms of their coupledprobabilities

define channel capacities and properties using Shannons Theorems

construct efficient codes for data on imperfect communication channels

generalise the discrete concepts to continuous signals on continuous channels

understand Fourier Transforms and the main ideas of efficient algorithms for them describe the information resolution and compression properties of wavelets

Recommended book

* Cover, T.M. & Thomas, J.A. (1991). Elements of information theory. New York: Wiley.


4/75


Computer Science Tripos Part II, Michaelmas Term

11 lectures by J G Daugman

1. Overview: What is Information Theory?

Key idea: The movements and transformations of information, just like

those of a fluid, are constrained by mathematical and physical laws.

These laws have deep connections with: probability theory, statistics, and combinatorics

thermodynamics (statistical physics)

spectral analysis, Fourier (and other) transforms

sampling theory, prediction, estimation theory

electrical engineering (bandwidth; signal-to-noise ratio) complexity theory (minimal description length)

signal processing, representation, compressibility

As such, information theory addresses and answers the

two fundamental questions of communication theory:

1. What is the ultimate data compression?(answer: the entropy of the data, H, is its compression limit.)

2. What is the ultimate transmission rate of communication?

(answer: the channel capacity, C, is its rate limit.)

All communication schemes lie in between these two limits on the com-

pressibility of data and the capacity of a channel. Information theory

can suggest means to achieve these theoretical limits. But the subjectalso extends far beyond communication theory.

1


5/75

Important questions... to which Information Theory offers answers:

How should information be measured?

How much additional information is gained by some reduction in

uncertainty?

How do the a priori probabilities of possible messages determine

the informativeness of receiving them?

What is the information content of a random variable?

How does the noise level in a communication channel limit its

capacity to transmit information? How does the bandwidth (in cycles/second) of a communication

channel limit its capacity to transmit information?

By what formalism should prior knowledge be combined with

incoming data to draw formally justifiable inferences from both?

How much information in contained in a strand of DNA?

How much information is there in the firing pattern of a neurone?

Historical origins and important contributions:

Ludwig BOLTZMANN (1844-1906), physicist, showed in 1877 that

thermodynamic entropy (defined as the energy of a statistical en-semble [such as a gas] divided by its temperature: ergs/degree) is

related to the statistical distribution of molecular configurations,

with increasing entropy corresponding to increasing randomness. He

made this relationship precise with his famous formula S= k logW

where S defines entropy, W is the total number of possible molec-

ular configurations, and k is the constant which bears Boltzmanns

name: k =1.38 x 1016

ergs per degree centigrade. (The aboveformula appears as an epitaph on Boltzmanns tombstone.) This is

2


6/75

equivalent to the definition of the information (negentropy) in an

ensemble, all of whose possible states are equiprobable, but with a

minus sign in front (and when the logarithm is base 2, k=1.) The

deep connections between Information Theory and that branch of

physics concerned with thermodynamics and statistical mechanics,hinge upon Boltzmanns work.

Leo SZILARD (1898-1964) in 1929 identified entropy with informa-

tion. He formulated key information-theoretic concepts to solve the

thermodynamic paradox known as Maxwells demon (a thought-

experiment about gas molecules in a partitioned box) by showing

that the amount of information required by the demon about the

positions and velocities of the molecules was equal (negatively) tothe demons entropy increment.

James Clerk MAXWELL (1831-1879) originated the paradox called

Maxwells Demon which greatly influenced Boltzmann and which

led to the watershed insight for information theory contributed by

Szilard. At Cambridge, Maxwell founded the Cavendish Laboratory

which became the original Department of Physics.

R V HARTLEY in 1928 founded communication theory with his

paper Transmission of Information. He proposed that a signal

(or a communication channel) having bandwidth over a duration

T has a limited number of degrees-of-freedom, namely 2T, and

therefore it can communicate at most this quantity of information.

He also defined the information content of an equiprobable ensembleofN possible states as equal to log2 N.

Norbert WIENER (1894-1964) unified information theory and Fourier

analysis by deriving a series of relationships between the two. He

invented white noise analysis of non-linear systems, and made the

definitive contribution to modeling and describing the information

content of stochastic processes known as Time Series.

3


7/75

Dennis GABOR (1900-1979) crystallized Hartleys insight by formu-

lating a general Uncertainty Principle for information, expressing

the trade-off for resolution between bandwidth and time. (Signals

that are well specified in frequency content must be poorly localized

in time, and those that are well localized in time must be poorlyspecified in frequency content.) He formalized the Information Di-

agram to describe this fundamental trade-off, and derived the con-

tinuous family of functions which optimize (minimize) the conjoint

uncertainty relation. In 1974 Gabor won the Nobel Prize in Physics

for his work in Fourier optics, including the invention of holography.

Claude SHANNON (together with Warren WEAVER) in 1949 wrote

the definitive, classic, work in information theory: MathematicalTheory of Communication. Divided into separate treatments for

continuous-time and discrete-time signals, systems, and channels,

this book laid out all of the key concepts and relationships that de-

fine the field today. In particular, he proved the famous Source Cod-

ing Theorem and the Noisy Channel Coding Theorem, plus many

other related results about channel capacity.

S KULLBACK and R A LEIBLER (1951) defined relative entropy

(also called information for discrimination, or K-L Distance.)

E T JAYNES (since 1957) developed maximum entropy methods

for inference, hypothesis-testing, and decision-making, based on the

physics of statistical mechanics. Others have inquired whether these

principles impose fundamental physical limits to computation itself. A N KOLMOGOROV in 1965 proposed that the complexity of a

string of data can be defined by the length of the shortest binary

program for computing the string. Thus the complexity of data

is its minimal description length, and this specifies the ultimate

compressibility of data. The Kolmogorov complexityKof a string

is approximately equal to its Shannon entropy H, thereby unifying

the theory of descriptive complexity and information theory.

4


8/75

2. Mathematical Foundations; Probability Rules;

Bayes Theorem

What are random variables? What is probability?

Random variables are variables that take on values determined by prob-

ability distributions. They may be discrete or continuous, in either their

domain or their range. For example, a stream of ASCII encoded text

characters in a transmitted message is a discrete random variable, with

a known probability distribution for any given natural language. An

analog speech signal represented by a voltage or sound pressure wave-

form as a function of time (perhaps with added noise), is a continuousrandom variable having a continuous probability density function.

Most of Information Theory involves probability distributions of ran-

dom variables, and conjoint or conditional probabilities defined over

ensembles of random variables. Indeed, the information content of a

symbol or event is defined by its (im)probability. Classically, there are

two different points of view about what probability actually means:

relative frequency: sample the random variable a great many times

and tally up the fraction of times that each of its different possible

values occurs, to arrive at the probability of each.

degree-of-belief: probability is the plausibility of a proposition or

the likelihood that a particular state (or value of a random variable)

might occur, even if its outcome can only be decided once (e.g. the

outcome of a particular horse-race).

The first view, the frequentist or operationalist view, is the one that

predominates in statistics and in information theory. However, by no

means does it capture the full meaning of probability. For example,

the proposition that "The moon is made of green cheese" is one

which surely has a probability that we should be able to attach to it.We could assess its probability by degree-of-belief calculations which

5


9/75

combine our prior knowledge about physics, geology, and dairy prod-

ucts. Yet the frequentist definition of probability could only assign

a probability to this proposition by performing (say) a large number

of repeated trips to the moon, and tallying up the fraction of trips on

which the moon turned out to be a dairy product....

In either case, it seems sensible that the less probable an event is, the

more information is gained by noting its occurrence. (Surely discovering

that the moon IS made of green cheese would be more informative

than merely learning that it is made only of earth-like rocks.)

Probability Rules

Most of probability theory was laid down by theologians: Blaise PAS-

CAL (1623-1662) who gave it the axiomatization that we accept today;

and Thomas BAYES (1702-1761) who expressed one of its most impor-

tant and widely-applied propositions relating conditional probabilities.

Probability Theory rests upon two rules:

Product Rule:

p(A,B) = joint probability ofbothA andB

= p(A|B)p(B)

or equivalently,= p(B|A)p(A)

Clearly, in case A and B are independent events, they are not condi-

tionalized on each other and so

p(A|B) = p(A)

and p(B|A) = p(B),

in which case their joint probability is simply p(A,B) = p(A)p(B).

6


10/75

Sum Rule:

If event A is conditionalized on a number of other events B, then the

total probability ofA is the sum of its joint probabilities with all B:

p(A) =Bp(A,B) =

Bp(A|B)p(B)

From the Product Rule and the symmetry that p(A,B) = p(B,A), it

is clear that p(A|B)p(B) = p(B|A)p(A). Bayes Theorem then follows:

Bayes Theorem:

p(B|A) = p(A|B)p(B)p(A)

The importance of Bayes Rule is that it allows us to reverse the condi-

tionalizing of events, and to computep(B|A) from knowledge ofp(A|B),

p(A), and p(B). Often these are expressed as priorand posteriorprob-

abilities, or as the conditionalizing of hypotheses upon data.

Worked Example:

Suppose that a dread disease affects 1/1000th of all people. If you ac-

tually have the disease, a test for it is positive 95% of the time, and

negative 5% of the time. If you dont have the disease, the test is posi-

tive 5% of the time. We wish to know how to interpret test results.

Suppose you test positive for the disease. What is the likelihood thatyou actually have it?

We use the above rules, with the following substitutions of data D

and hypothesis H instead ofA and B:

D = data: the test is positive

H= hypothesis: you have the diseaseH= the other hypothesis: you do not have the disease

7


11/75

Before acquiring the data, we know only that the a priori probabil-

ity of having the disease is .001, which sets p(H). This is called a prior.

We also need to know p(D).

From the Sum Rule, we can calculate that the a priori probability

p(D) of testing positive, whatever the truth may actually be, is:

p(D) = p(D|H)p(H) + p(D|H)p(H) =(.95)(.001)+(.05)(.999) = .051

and from Bayes Rule, we can conclude that the probability that you

actually have the disease given that you tested positive for it, is muchsmaller than you may have thought:

p(H|D) =p(D|H)p(H)

p(D)=

(.95)(.001)

(.051)= 0.019 (less than 2%).

This quantity is called the posterior probabilitybecause it is computed

after the observation of data; it tells us how likely the hypothesis is,

given what we have observed. (Note: it is an extremely common humanfallacy to confound p(H|D) with p(D|H). In the example given, most

people would react to the positive test result by concluding that the

likelihood that they have the disease is .95, since that is the hit rate

of the test. They confound p(D|H) = .95 with p(H|D) = .019, which

is what actually matters.)

A nice feature of Bayes Theorem is that it provides a simple mech-

anism for repeatedly updating our assessment of the hypothesis as more

data continues to arrive. We can apply the rule recursively, using the

latest posterior as the new prior for interpreting the next set of data.

In Artificial Intelligence, this feature is important because it allows the

systematic and real-time construction of interpretations that can be up-

dated continuously as more data arrive in a time series, such as a flowof images or spoken sounds that we wish to understand.

8


12/75

3. Entropies Defined, and Why They are

Measures of Information

The information content I of a single event or message is defined as the

base-2 logarithm of its probability p:

I= log2p (1)

and its entropyH is considered the negative of this. Entropy can be

regarded intuitively as uncertainty, or disorder. To gain information

is to lose uncertainty by the same amount, so I and H differ only in

sign (if at all): H= I. Entropy and information have units ofbits.

Note that I as defined in Eqt (1) is never positive: it ranges between

0 and as p varies from 1 to 0. However, sometimes the sign is

dropped, and I is considered the same thing as H (as well do later too).

No information is gained (no uncertainty is lost) by the appearance

of an event or the receipt of a message that was completely certain any-

way (p = 1, so I = 0). Intuitively, the more improbable an event is,

the more informative it is; and so the monotonic behaviour of Eqt (1)

seems appropriate. But why the logarithm?

The logarithmic measure is justified by the desire for information to

be additive. We want the algebra of our measures to reflect the Rules

of Probability. When independent packets of information arrive, wewould like to say that the total information received is the sum of

the individual pieces. But the probabilities of independent events

multiply to give their combined probabilities, and so we must take

logarithms in order for the joint probability of independent events

or messages to contribute additively to the information gained.

This principle can also be understood in terms of the combinatoricsof state spaces. Suppose we have two independent problems, one with n

9


13/75

possible solutions (or states) each having probability pn, and the other

with m possible solutions (or states) each having probability pm. Then

the number of combined states is mn, and each of these has probability

pmpn. We would like to say that the information gained by specifying

the solution to both problems is the sumof that gained from each one.This desired property is achieved:

Imn = log2(pmpn) = log2pm + log2pn = Im + In (2)

A Note on Logarithms:

In information theory we often wish to compute the base-2 logarithmsof quantities, but most calculators (and tools like xcalc) only offer

Napierian (base 2.718...) and decimal (base 10) logarithms. So the

following conversions are useful:

log2 X= 1.443 logeX= 3.322 log10 X

Henceforward we will omit the subscript; base-2 is always presumed.

Intuitive Example of the Information Measure (Eqt 1):

Suppose I choose at random one of the 26 letters of the alphabet, and we

play the game of 25 questions in which you must determine which let-

ter I have chosen. I will only answer yes or no. What is the minimum

number of such questions that you must ask in order to guarantee find-

ing the answer? (What form should such questions take? e.g., Is it A?Is it B? ...or is there some more intelligent way to solve this problem?)

The answer to a Yes/No question having equal probabilities conveys

one bit worth of information. In the above example with equiprobable

states, you never need to ask more than 5 (well-phrased!) questions to

discover the answer, even though there are 26 possibilities. Appropri-

ately, Eqt (1) tells us that the uncertainty removed as a result of solvingthis problem is about -4.7 bits.

10


14/75

Entropy of Ensembles

We now move from considering the information content of a single event

or message, to that of an ensemble. An ensemble is the set of outcomesof one or more random variables. The outcomes have probabilities at-

tached to them. In general these probabilities are non-uniform, with

event i having probability pi, but they must sum to 1 because all possi-

ble outcomes are included; hence they form a probability distribution:

ipi = 1 (3)

The entropy of an ensemble is simply the average entropy of all theelements in it. We can compute their average entropy by weighting each

of the logpi contributions by its probability pi:

H= I=

ipi logpi (4)

Eqt (4) allows us to speak of the information content or the entropy of

a random variable, from knowledge of the probability distribution that

it obeys. (Entropy does not depend upon the actual values taken by

the random variable! Only upon their relative probabilities.)

Let us consider a random variable that takes on only two values, one

with probability p and the other with probability (1 p). Entropy is a

concave function of this distribution, and equals 0 ifp = 0 or p = 1:

11


15/75

Example of entropy as average uncertainty:

The various letters of the written English language have the followingrelative frequencies (probabilities), in descending order:

E T O A N I R S H D L C ...

.105 .072 .066 .063 .059 .055 .054 .052 .047 .035 .029 .023 ...

If they had been equiprobable, the entropy of the ensemble would

have been log2(1

26) = 4.7 bits. But their non-uniform probabilities im-

ply that, for example, an E is nearly five times more likely than a C;

surely this prior knowledge is a reduction in the uncertainty of this ran-

dom variable. In fact, the distribution of English letters has an entropy

of only 4.0 bits. This means that as few as only four Yes/No questionsare needed, in principle, to identify one of the 26 letters of the alphabet;

not five.

How can this be true?

That is the subject matter of Shannons SOURCE CODING THEOREM

(so named because it uses the statistics of the source, the a priori

probabilities of the message generator, to construct an optimal code.)

Note the important assumption: that the source statistics are known!

Several further measures of entropy need to be defined, involving the

marginal, joint, and conditional probabilities of random variables. Somekey relationships will then emerge, that we can apply to the analysis of

communication channels.

Notation: We use capital letters X and Y to name random variables,

and lower case letters x and y to refer to their respective outcomes.

These are drawn from particular sets A and B: x {a1, a2, ...aJ}, and

y {b1, b2, ...bK}. The probability of a particular outcome p(x = ai)is denoted pi, with 0 pi 1 and

ipi = 1.

12


16/75

An ensemble is just a random variable X, whose entropy was defined

in Eqt (4). A joint ensemble XY is an ensemble whose outcomes

are ordered pairs x, y with x {a1, a2, ...aJ} and y {b1, b2, ...bK}.

The joint ensemble XY defines a probability distribution p(x, y) overall possible joint outcomes x, y.

Marginal probability: From the Sum Rule, we can see that the proba-

bility ofX taking on a particular value x = ai is the sum of the joint

probabilities of this outcome for X and all possible outcomes for Y:

p(x = ai) =yp(x = ai, y)

We can simplify this notation to: p(x) =yp(x, y)

and similarly: p(y) =xp(x, y)

Conditional probability: From the Product Rule, we can easily see that

the conditional probability that x = ai, given that y = bj, is:

p(x = ai|y = bj) =p(x = ai, y = bj)

p(y = bj)

We can simplify this notation to: p(x|y) =p(x, y)

p(y)

and similarly: p(y|x) =p(x, y)

p(x)

It is now possible to define various entropy measures for joint ensembles:

Joint entropy ofXY

H(X, Y) =x,yp(x, y)log

1

p(x, y)(5)

(Note that in comparison with Eqt (4), we have replaced the sign infront by taking the reciprocal ofp inside the logarithm).

13


17/75

From this definition, it follows that joint entropy is additive ifX and Y

are independent random variables:

H(X, Y) = H(X) + H(Y) iff p(x, y) = p(x)p(y)

Prove this.

Conditional entropy of an ensemble X, given that y = bj

measures the uncertainty remaining about random variable X after

specifying that random variableY

has taken on a particular valuey = bj. It is defined naturally as the entropy of the probability dis-

tribution p(x|y = bj):

H(X|y = bj) =xp(x|y = bj)log

1

p(x|y = bj)(6)

If we now consider the above quantity averaged over all possible out-

comes that Y might have, each weighted by its probability p(y), thenwe arrive at the...

Conditional entropy of an ensemble X, given an ensemble Y:

H(X|Y) =yp(y)

xp(x|y)log

1

p(x|y)

(7)

and we know from the Sum Rule that if we move the p(y) term fromthe outer summation over y, to inside the inner summation over x, the

two probability terms combine and become just p(x, y) summed over all

x, y. Hence a simpler expression for this conditional entropy is:

H(X|Y) =x,yp(x, y)log

1

p(x|y)(8)

This measures the average uncertainty that remains about X, when Yis known.

14


18/75

Chain Rule for Entropy

The joint entropy, conditional entropy, and marginal entropy for two

ensembles X and Y are related by:H(X, Y) = H(X) + H(Y|X) = H(Y) + H(X|Y) (9)

It should seem natural and intuitive that the joint entropy of a pair of

random variables is the entropy of one plus the conditional entropy of

the other (the uncertainty that it adds once its dependence on the first

one has been discounted by conditionalizing on it). You can derive the

Chain Rule from the earlier definitions of these three entropies.

Corollary to the Chain Rule:

If we have three random variables X,Y,Z , the conditionalizing of the

joint distribution of any two of them, upon the third, is also expressed

by a Chain Rule:

H(X, Y|Z) = H(X|Z) + H(Y|X, Z) (10)

Independence Bound on Entropy

A consequence of the Chain Rule for Entropy is that if we have many

different random variables X1, X2,...,Xn, then the sum of all their in-dividual entropies is an upper bound on their joint entropy:

H(X1, X2,...,Xn) n

i=1H(Xi) (11)

Their joint entropy only reaches this upper bound if all of the random

variables are independent.

15


19/75


20/75

Distance D(X, Y) between X and Y

The amount by which the joint entropy of two random variables ex-

ceeds their mutual information is a measure of the distance betweenthem:

D(X, Y) = H(X, Y) I(X;Y) (16)

Note that this quantity satisfies the standard axioms for a distance:

D(X, Y) 0, D(X,X) = 0, D(X, Y) = D(Y,X), and

D(X, Z) D(X, Y) + D(Y, Z).

Relative entropy, or Kullback-Leibler distance

Another important measure of the distance between two random vari-

ables, although it does not satisfy the above axioms for a distance metric,

is the relative entropy or Kullback-Leibler distance. It is also called

the information for discrimination. Ifp(x) and q(x) are two proba-

bility distributions defined over the same set of outcomes x, then theirrelative entropy is:

DKL(pq) =xp(x)log

p(x)

q(x)(17)

Note that DKL(pq) 0, and in case p(x) = q(x) then their distance

DKL(pq) = 0, as one might hope. However, this metric is not strictly a

distance, since in general it lacks symmetry: DKL(pq) = DKL(qp).

The relative entropy DKL(pq) is a measure of the inefficiency of as-

suming that a distribution is q(x) when in fact it is p(x). If we have an

optimal code for the distribution p(x) (meaning that we use on average

H(p(x)) bits, its entropy, to describe it), then the number of additional

bits that we would need to use if we instead described p(x) using an

optimal code for q(x), would be their relative entropy DKL(pq).

17


21/75

Venn Diagram: Relationship among entropies and mutual information.

Fanos Inequality

We know that conditioning reduces entropy: H(X|Y) H(X). It

is clear that ifX and Y are perfectly correlated, then their conditional

entropy is 0. It should also be clear that ifX is any deterministicfunction ofY, then again, there remains no uncertainty about X once

Y is known and so their conditional entropy H(X|Y) = 0.

Fanos Inequality relates the probability of error Pe in guessing X from

knowledge ofY to their conditional entropy H(X|Y), when the number

of possible outcomes is |A| (e.g. the length of a symbol alphabet):

Pe H(X|Y) 1

log |A|(18)

The lower bound on Pe is a linearly increasing function ofH(X|Y).

The Data Processing Inequality

If random variables X, Y, and Z form a Markov chain (i.e. the condi-tional distribution ofZ depends only on Y and is independent ofX),

which is normally denoted as X Y Z, then the mutual informa-

tion must be monotonically decreasing over steps along the chain:

I(X;Y) I(X;Z) (19)

We turn now to applying these measures and relationships to the study

of communications channels. (The following material is from McAuley.)

18


22/75

# % ' ( # 0 4 ( 6

7 8 @ B D D @ B H I P R T R W H B ` 8 a H R e a T 8 H @ B P I g 8 g R a q r H R @ 8 s D H R T R W H B ` 8 a u s a v R x

a R T 8 H H 8 H e a R e a 8 x 8 W P

a R T 8 H H s H q g R D B H 8 W 8 a s P 8 ` R W 8 s T I 8 ` 8 B H

H I R @ W a s

I B T s D D q P R 8 P I 8 a @ B P I s P @ R H P s P 8 u s a v R x

a R T 8 H H R a P I 8 s D

I s 8 P

r r r r j k B W n e a 8 o 7 8 T s W P I 8 W H R D x 8 R a P I 8 H P s P 8 R T T e

s W T q e H B W

R @ 8 e s P B R W H P I B H 8 s g

D 8 B H P a B x B s D

D, 1/8

C, 1/8

B, 1/4

A, 1/2 A, 1/2

B, 3/8

C, 1/8

D, 1/8

B, 1/4

C, 1/8

A, 1/4

E, 1/4

{ B e a 8 o | } a s

I H a 8

a 8 H 8 W P B W g 8 g R a q D 8 H H H R e a T 8 s W ` P @ R H P s P 8 u s a v R x

a R

T 8 H H

W 8 W 8 a s D P I 8 W @ 8 g s q ` 8 n W 8 s W q n W B P 8 H P s P 8

a R T 8 H H @ B P I H P s P 8 H

k r

@ B P I P a s W H B P B R W

a R s B D B P B 8 H 8 B W P I 8

a R s B D B P q R g R x B W a R g H P s P 8

P R H P s P 8

@ B P I P I 8 8 g B H H B R W R H R g 8 H q g R D { B a H P @ 8 T s W ` 8 n W 8 P I 8 8 W P a R

q

R 8 s T I H P s P 8 B W P I 8 W R a g s D g s W W 8 a |

D R

s W ` P I 8 W P I 8 8 W P a R

q R P I 8 H q H P 8 g P R 8 P I 8 H e g R P I 8 H 8 B W ` B x B ` e s D H P s P 8

8 W P a R

q x s D e 8 H @ 8 B I P 8 ` @ B P I P I 8 H P s P 8 R T T e

s W T q T s D T e D s P 8 ` a R g P I 8

R @

8 e s P B R W H |

D R

D 8 s a D q R a s H B W D 8 H P s P 8 r @ 8 I s x 8 P I 8 8 W P a R

q R P I 8 g 8 g R a q D 8 H H H R e a T 8

4 ( % ' ( 4 ( (

P 8 W @ 8 @ B H I P R 8 T B 8 W P D q a 8

a 8 H 8 W P P I 8 H q g R D H 8 W 8 a s P 8 ` q H R g 8 H R e a T 8

7 8 H I s D D T R W H B ` 8 a 8 W T R ` B W P I 8 H q g R D H s H B W s a q ` B B P H

o


23/75

Symbols EncodingSource

encoder

{ B e a 8 o | B H T a 8 P 8 g 8 g R a q D 8 H H H R e a T 8 s W ` 8 W T R ` 8 a

R W H B ` 8 a 8 W T R ` B W P I 8 H q g R D H

k r 8 W P a R

q r s H s n 8 ` D 8 W P I D R T v

B W s a q ` B B P H R 8 W H e a 8 @ 8 T s W ` 8 T R ` 8 P I 8 H 8 H q g R D H @ 8 W 8 8 ` |

s

R @ 8 a R

o R P I 8 a @ B H 8

@ I 8 a 8

B H P I 8 D s a 8 H P B W P 8 8 a D 8 H H P I s W I 8 B H P I 8 W B P H

8 a

H q g R D r s W ` s H @ 8 v W R @

P I 8 W I 8 8 T B 8 W T q R P I 8 T R ` B W

B H B x 8 W q |

7 I 8 W P I 8 H q g R D H s a 8 8 e B

a R s D 8 r s W ` B H s

R @ 8 a R P @ R r

o s W `

R P 8 P I s P B @ 8 P a q P R s T I B 8 x 8 B W P I B H T s H 8 @ 8 g e H P s D D R T s P 8 s P

D 8 s H P R W 8 8 W T R ` B W P R g R a 8 P I s W R W 8 H q g R D P I B H g 8 s W H P I s P @ 8 s a 8 B W T s

s D 8

R e W B e 8 D q ` 8 T R ` B W

P B D D @ B P I 8 e B

a R s D 8 H q g R D H r e P @ I 8 W B H W R P s

R @ 8 a R P @ R r P I B H T R ` B W

B H B W 8 T B 8 W P P R P a q P R R x 8 a T R g 8 P I B H @ 8 T R W H B ` 8 a H 8 e 8 W T 8 H R H q g R D H R D 8 W P I

s W ` T R W H B ` 8 a 8 W T R ` B W 8 s T I

R H H B D 8 H 8 e 8 W T 8 R D 8 W P I s H s D R T v R B W s a q

` B B P H r P I 8 W @ 8 R P s B W |

D R

o

@ I 8 a 8 B H W R @ P I 8 s x 8 a s 8 W e g 8 a R B P H

8 a H q g R D R P 8 P I s P s H 8 P H

D s a 8 r

o

W 8 W 8 a s D @ 8 ` R W R P I s x 8 8 e B

a R s D 8 H q g R D H r s W ` @ 8 @ R e D ` I R

8 P R s T I B 8 x 8

H R g 8 g R a 8 T R g

a 8 H H 8 ` R a g R 8 W T R ` B W q e H 8 R x s a B s D 8 D 8 W P I T R ` 8 H s W

8 s g

D 8 R H e T I s W 8 W T R ` B W B H u R a H 8 T R ` 8 ` s P B W a R g P I 8 ` s q H R P 8 D 8 a s

I H

7 8 T R W H B ` 8 a s s B W R e a H B g

D 8 R e a H q g R D s D

I s 8 P s W ` H R g 8

R H H B D 8 x s a B s D 8

D 8 W P I T R ` 8 H |

R ` 8 o R ` 8 R ` 8

o o

o o o

o o o o o o

o o o o o o o o


24/75

7 8 T R W H B ` 8 a 8 s T I T R ` 8 B W P e a W |

o H B W P I B H 8 W T R ` B W r @ 8 n W ` P I s P

a 8 H 8 W P 8 ` @ B P I s H 8 e 8 W T 8 D B v 8 o o r @ 8

` R W R P v W R @ @ I 8 P I 8 a P R ` 8 T R ` 8 s H

R a I B H g s v 8 H H e T I s T R ` 8

e W H s P B H s T P R a q { e a P I 8 a r B W 8 W 8 a s D r 8 x 8 W s T R ` 8 @ I 8 a 8 H e T I s W s g B e B P q

T R e D ` 8 a 8 H R D x 8 ` e W B e 8 D q q D R R v B W s P B P H e a P I 8 a s I 8 s ` B W P I 8 H P a 8 s g

s W ` s T v P a s T v B W B H e W H s P B H s T P R a q

H 8 a x 8 P I s P R a P I B H T R ` 8 r P I 8 T R ` B W a s P 8 r R a s x 8 a s 8 W e g 8 a R B P H

8 a

H q g R D r B H B x 8 W q |

o

@ I B T I B H D 8 H H P I s W P I 8 8 W P a R

q

I B H T R ` 8 B H e W B e 8 D q ` 8 T R ` s D 8 e a P I 8 a P I B H T R ` 8 B H B W P 8 a 8 H P B W B W P I s P

@ 8 T s W ` 8 T R ` 8 P I s P B H W R s T v P a s T v B W B H a 8 e B a 8 ` R W T 8

@ 8 I s x 8 P I 8 B P H R P I 8 8 W T R ` 8 ` H q g R D @ 8 T s W ` 8 T R ` 8 @ B P I R e P @ s B P B W

R a g R a 8 { e a P I 8 a P I B H s D H R H s P B H n 8 H P I 8 T R W ` B P B R W r P I s P B H P I 8 a 8 B H

W R T R ` 8 @ R a ` @ I B T I B H

a 8 n B 8 H s g 8 B P

s P P 8 a W R s D R W 8 a T R ` 8 @ R a `

W P I B H T s H 8 P I 8 T R ` B W a s P 8 B H 8 e s D P R P I 8 8 W P a R

q

7 I B D 8 P I B H B H ` 8 T R ` s D 8 s W ` T R ` B W a s P 8 B H 8 e s D P R P I 8 8 W P a R

q s s B W r

R H 8 a x 8 P I s P B P ` R 8 H W R P I s x 8 P I 8

a 8 n

a R

8 a P q s W ` B H W R P s W B W H P s W P s

W 8 R e H T R ` 8

I s W W R W H n a H P P I 8 R a 8 g B H P I 8 # $ % @ I B T I B H |

{ R a s ` B H T a 8 P 8 g 8 g R a q D 8 H H H R e a T 8 @ B P I n W B P 8 8 W P a R

q R a s W q

R H B P B x 8 ' B P B H

R H H B D 8 P R 8 W T R ` 8 P I 8 H q g R D H s P s W s x 8 a s 8 a s P 8

r H e T I P I s P |

'

{ R a

a R R H 8 8

I s W W R W 0 7 8 s x 8 a I B H B H s D H R H R g 8 P B g 8 H T s D D 8 ` P I 8 W R B H 8 D 8 H H

T R ` B W P I 8 R a 8 g s H B P ` 8 s D H @ B P I T R ` B W @ B P I R e P T R W H B ` 8 a s P B R W R W R B H 8

a R T 8 H H 8 H

B 8 B P T R a a e

P B R W 8 P T

I 8 8 W P a R

q e W T P B R W P I 8 W a 8

a 8 H 8 W P H s e W ` s g 8 W P s D D B g B P R W P I 8 W e g 8 a R B P H

R W s x 8 a s 8 a 8 e B a 8 ` P R a 8

a 8 H 8 W P P I 8 H q g R D H R P I 8 H R e a T 8

2 3

7 8 I s x 8 s D a 8 s ` q g 8 W P B R W 8 ` P I 8

a 8 n

a R

8 a P q @ 8 n W ` P I s P R a s

a 8 n T R ` 8

P R 8 B H P r B P g e H P H s P B H q P I 8 4 6 8 8 B W 8 e s D B P q I s P B H r s W 8 T 8 H H s a q W R P

H e T B 8 W P T R W ` B P B R W R a s T R ` 8 I s x B W B W s a q T R ` 8 @ R a ` H @ B P I D 8 W P I @

@

C C C

@ F P R H s P B H q P I 8

a 8 n T R W ` B P B R W B H |

F

H

o

P

o

o


25/75

R # ' ( ( S ( 6 U ( # # 4 ( U

7 8 I s x 8 T R W H B ` 8 a 8 ` P I 8 ` B H T a 8 P 8 H R e a T 8 r W R @ @ 8 T R W H B ` 8 a s T I s W W 8 D P I a R e I

@ I B T I @ 8 @ B H I P R

s H H H q g R D H 8 W 8 a s P 8 ` q H e T I s H R e a T 8 q H R g 8 s

a R

a B s P 8

8 W T R ` B W g 8 T I s W B H g @ 8 s D H R B W P a R ` e T 8 P I 8 B ` 8 s R W R B H 8 B W P R P I 8 H q H P 8 g P I s P

B H @ 8 T R W H B ` 8 a P I 8 T I s W W 8 D P R g R ` B q P I 8 B W

e P T R ` B W s W `

R H H B D q 8 W 8 a s P 8

H R g 8 g R ` B n 8 ` x 8 a H B R W

7 8 H I R e D ` ` B H P B W e B H I 8 P @ 8 8 W H q H P 8 g s P B T g R ` B n T s P B R W R P I 8 8 W T R ` 8 ` H q g R D H r

B 8 ` B H P R a P B R W r s W ` W R B H 8 B H P R a P B R W B H @ I 8 W s W B W

e P T R ` 8 s D @ s q H a 8 H e D P H B W P I 8

P I 8 H s g 8 R e P

e P T R ` 8 P I B H

a R T 8 H H T s W T D 8 s a D q 8 a 8 x 8 a H 8 ` R B H 8 R W P I 8 R P I 8 a

I s W ` B W P a R ` e T 8 H P I 8 8 D 8 g 8 W P R a s W ` R g W 8 H H B W P R P I 8 a 8 H e D P B W R e P

e P T R ` 8

Symbols Source

encoderDecoder

Symbols

ChannelX Y

{ B e a 8 o | R ` B W s W ` ` 8 T R ` B W R H q g R D H R a P a s W H 8 a R x 8 a s T I s W W 8 D

7 8 T R W H B ` 8 a s W B W

e P s D

I s 8 P W

X X `

k s W ` R e P

e P s D

I s 8 P a

c c e

k s W ` a s W ` R g x s a B s D 8 H

s W ` f @ I B T I a s W 8 R x 8 a P I 8 H 8 s D

I s 8 P H

R P 8 P I s P s W ` g W 8 8 ` W R P 8 P I 8 H s g 8 R a 8 s g

D 8 @ 8 g s q I s x 8 P I 8 B W s a q

B W

e P s D

I s 8 P

o k s W ` P I 8 R e P

e P s D

I s 8 P

o

i

k r @ I 8 a 8

i

a 8

a 8 H 8 W P H

P I 8 ` 8 T R ` 8 a B ` 8 W P B q B W H R g 8 8 a a R a I 8 ` B H T a 8 P 8 g 8 g R a q D 8 H H T I s W W 8 D T s W P I 8 W

8 a 8

a 8 H 8 W P 8 ` s H s H 8 P R P a s W H B P B R W

a R s B D B P B 8 H |

c p r X

f

c p r

X

I s P B H P I 8

a R s B D B P q P I s P B

X

B H B W v 8 T P 8 ` B W P R P I 8 T I s W W 8 D r

c p

B H 8 g B P P 8 `

cp

r X

B H P I 8 T R W ` B P B R W s D

a R s B D B P q I B H T s W 8 @ a B P P 8 W s H P I 8 $ V

% |

w

x

c

r X

c

r X

ce

r X

c r X

c r X

c e r X

c r X `

c r X `

c e r X `

R P 8 P I s P @ 8 I s x 8 P I 8

a R

8 a P q P I s P R a 8 x 8 a q B W

e P H q g R D r @ 8 @ B D D 8 P H R g 8

P I B W R e P |

e

p

H

c p r X

o


26/75

8 P @ 8 P s v 8 P I 8 R e P

e P R s ` B H T a 8 P 8 g 8 g R a q D 8 H H H R e a T 8 s H P I 8 B W

e P P R s T I s W

W 8 D

R @ 8 I s x 8 s H H R T B s P 8 ` @ B P I P I 8 B W

e P s D

I s 8 P R P I 8 T I s W W 8 D P I 8

a R s B D

B P q ` B H P a B e P B R W R R e P

e P a R g s g 8 g R a q D 8 H H H R e a T 8

X

o

k 7 8

P I 8 W R P s B W P I 8 v R B W P

a R s B D B P q ` B H P a B e P B R W R P I 8 a s W ` R g x s a B s D 8 H

s W `

f |

X c p

X

f

c p

c p r X

X

7 8 T s W P I 8 W n W ` P I 8 g s a B W s D

a R s B D B P q ` B H P a B e P B R W R r P I s P B H P I 8

a R s

B D B P q R R e P

e P H q g R D

c p

s

8 s a B W |

c p

f

c p

`

H

cp

r X

X

@ 8 I s x 8 g r @ 8 R P 8 W B ` 8 W P B q 8 s T I R e P

e P H q g R D s H 8 B W P I 8 ` 8 H B a 8 `

a 8 H e D P R H R g 8 B W

e P H q g R D a @ 8 g s q H 8 D 8 T P H R g 8 H e H 8 P R R e P

e P H q g R D H r

R a 8 s g

D 8 B W P I 8 B W

e P

o k s W ` R e P

e P

o

i

k 7 8 P I 8 W ` 8 n W 8 P I 8 s x 8 a s 8

a R s B D B P q R H q g R D 8 a a R a s H |

e

p

H

p

H

f

c p r

X

e

p

H

`

H

H

p

c p r X

X

s W ` T R a a 8 H

R W ` B W D q r P I 8 s x 8 a s 8

a R s B D B P q R T R a a 8 T P a 8 T 8

P B R W s H o

I 8 B W s a q H q g g 8 P a B T T I s W W 8 D I s H P @ R B W

e P s W ` R e P

e P H q g R D H e H e s D D q @ a B P

P 8 W

o k s W ` s T R g g R W

a R s B D B P q r r R B W T R a a 8 T P ` 8 T R ` B W R s W B W

e P

s P P I 8 R e P

e P P I B H T R e D ` 8 s H B g

D B H P B T g R ` 8 D R s T R g g e W B T s P B R W H D B W v r n

e a 8 s

R @ 8 x 8 a r P R e W ` 8 a H P s W ` P I 8 s x 8 a s B W

a R

8 a P q R P I 8 8 a a R a a s P 8

` 8 H T a B 8 `

s R x 8 r T R W H B ` 8 a P I 8 n e a 8 r @ I 8 a 8 @ 8 I s x 8 o H q g R D H r R @ I B T I P I 8 n a H P I s H

s

a R s B D B P q R 8 B W a 8 T 8 B x 8 ` B W 8 a a R a R

o r s W ` P I 8 a 8 g s B W ` 8 a s a 8 s D @ s q H

a 8 T 8 B x 8 `

8 a 8 T P D q I 8 W R H 8 a x B W P I s P g R H P R P I 8 P 8 a g H B W P I 8 H e g R W P I 8

a B I P R 8 e s P B R W s a 8 j 8 a R |

c r X k

X k

o

o l

o l m


27/75

0

11

0

p

p

1-p

1-p

X Y

0

11

0

YX

0.1

0.9

1.0

1.0

1.0

2

3

2

3

999999999999

{ B e a 8 | s B W s a q H q g g 8 P a B T T I s W W 8 D r s D 8 R P I 8 e W D e T v q H q g R D

n S % % U ( o 6

j P 8 W ` B W P I 8 B ` 8 s H R B W R a g s P B R W s W ` 8 W P a R

q R a P I 8 ` B H T a 8 P 8 H R e a T 8 r @ 8 T s W

W R @ T R W H B ` 8 a P I 8 B W R a g s P B R W s R e P

R P s B W 8 ` q R H 8 a x B W P I 8 x s D e 8 R P I 8

f 7 8 ` 8 n W 8 P I 8 8 W P a R

q s P 8 a R H 8 a x B W f

c p

|

W

r

f

c p

`

H

X r c p

D R

o

X

r c p

P I B H B H s a s W ` R g x s a B s D 8 r H R @ 8 T s W P s v 8 P I 8 s x 8 a s 8 s s B W |

W

r

a

e

p

H

W

r

f

c p

c p

e

p

H

`

H

X

r c p

D R

o

X r cp

c p

e

p

H

`

H

X

c p

D R

o

X r c p

W

r

a B H P I 8 7 8 P I 8 W @ a B P 8 P I 8 % " 6 % R

P I 8 T I s W W 8 D |

W a W W

r

a

I B H

a R x B ` 8 H e H @ B P I s g 8 s H e a 8 R P I 8 s g R e W P R B W R a g s P B R W R a e W T 8 a P s B W P q

a 8 ` e T 8 ` s R e P

s P 8 a R H 8 a x B W P I 8 R e P

e Pf

R s T I s W W 8 D 8 ` q

I 8

g e P e s D B W R a g s P B R W P 8 D D H e H H R g 8 P I B W s R e P P I 8 T I s W W 8 D

W s D P 8 a W s P B x 8 x B 8 @

R B W P B H P R P I B W v s R e P P I 8 T I s W W 8 D P R 8 P I 8 a @ B P I s T R a a 8 T

P B R W ` 8 x B T 8 8 ` @ B P I B W R a g s P B R W a R g R H 8 a x s P B R W H R R P I P I 8 B W

e P s W ` R e P

e P

R P I 8 T I s W W 8 D r 8 n e a 8 o W 8 g B I P s H v I R @ g e T I B W R a g s P B R W g e H P 8


28/75

Source

encoderDecoder

X Y XCorrection

Observer

Correction data

{ B e a 8 o | R a a 8 T P B R W H q H P 8 g

s H H 8 ` s D R W P I 8 T R a a 8 T P B R W T I s W W 8 D P R a 8 T R W H P a e T P P I 8 B W

e P P I B H P e a W H R e P P R

8

r

f

7 8 T s W e a P I 8 a a 8 @ a B P 8 P I 8 8 W P a R

q e W T P B R W W s H s H e g R x 8 a P I 8 v R B W P

a R s B D B P q ` B H P a B e P B R W |

W1

`

H

X

D R

X

o

`

H

X

D R

X

e

p

H

c p r X

`

H

e

p

H

c p r X

X

D R

X

`

H

e

p

H

X

c p

D R

X

8 W T 8 @ 8 R P s B W s W 8

a 8 H H B R W R a P I 8 g e P e s D B W R a g s P B R W |

W a

`

H

e

p

H

X

c p

D R

X

r c p

X

7 8 T s W ` 8 ` e T 8 x s a B R e H

a R

8 a P B 8 H R P I 8 g e P e s D B W R a g s P B R W |

o

W a z

R H I R @ P I B H r @ 8 W R P 8 P I s P

X

r c p

c p

X

c p

s W ` H e H P B P e P 8 P I B H

B W P I 8 8 e s P B R W s R x 8 |

W a

e

p

H

`

H

X c p

D R

X c p

X

c p

z

q e H 8 R P I 8 B W 8 e s D B P q @ 8 8 H P s D B H I 8 `

a 8 x B R e H D q


29/75

W a B

s W ` f s a 8 H P s P B H P B T s D D q B W ` 8

8 W ` 8 W P

s W ` f s a 8 B W ` 8

8 W ` 8 W P r

X

c p

X

c p

r I 8 W T 8 P I 8 D R P 8 a g

8 T R g 8 H j 8 a R

B H H q g g 8 P a B T r

W a

a W

H B W

X

r c p

c p

c p r X

X

r @ 8 R P s B W |

W a

`

H

e

p

H

X c p

D R

X r cp

X

`

H

e

p

H

X c p

D R }

c p r X

c p

~

a W

I 8

a 8 T 8 ` B W D 8 s ` H P R P I 8 R x B R e H H q g g 8 P a B T ` 8 n W B P B R W R a P I 8 g e P e s D

B W R a g s P B R W B W P 8 a g H R P I 8 8 W P a R

q R P I 8 R e P

e P |

W a a a

r

W

7 8 ` 8 n W 8 P I 8 v R B W P 8 W P a R

q R P @ R a s W ` R g x s a B s D 8 H B W P I 8 R x B R e H g s W

W 8 a r s H 8 ` R W P I 8 v R B W P

a R s B D B P q ` B H P a B e P B R W |

W

a

`

H

e

p

H

X

c p

D R

X

c p

I 8 g e P e s D B W R a g s P B R W B H P I 8 g R a 8 W s P e a s D D q @ a B P P 8 W |

W a W a W

a

R W H B ` 8 a P I 8 T R W ` B P B R W s D 8 W P a R

q s W ` g e P e s D B W R a g s P B R W R a P I 8 B W s a q H q g

g 8 P a B T T I s W W 8 D I 8 B W

e P H R e a T 8 I s H s D

I s 8 P W

o k s W ` s H H R T B s P 8 `

a R s B D B P B 8 H

o

o k s W ` P I 8 T I s W W 8 D g s P a B B H |

o

o

I 8 W P I 8 8 W P a R

q r T R W ` B P B R W s D 8 W P a R

q s W ` g e P e s D B W R a g s P B R W s a 8 B x 8 W q |

W o

W

r

a D R o D R o

W aC o

D R o D R o

{ B e a 8 s H I R @ H P I 8 T s

s T B P q R P I 8 T I s W W 8 D s s B W H P P a s W H B P B R W

a R s B D B P q

R P 8 P I s P P I 8 T s

s T B P q R P I 8 T I s W W 8 D ` a R

H P R j 8 a R @ I 8 W P I 8 P a s W H B P B R W

a R s

B D B P q B H o r s W ` B H g s B g B j 8 ` @ I 8 W P I 8 P a s W H B P B R W

a R s B D B P q B H R a o B @ 8

a 8 D B s D q P a s W H

R H 8 P I 8 H q g R D H R W P I 8 @ B a 8 @ 8 s D H R 8 P P I 8 g s B g e g s g R e W P

R B W R a g s P B R W P I a R e I { B e a 8 H I R @ H P I 8 8 8 T P R W g e P e s D B W R a g s P B R W R a

s H q g g 8 P a B T B W

e P s D

I s 8 P H


30/75

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1P(transition)

I(X;Y)

I(X;Y)

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.90.10.20.30.40.50.60.70.80.9

0

0.5

1

P(transition)

P(X=0)

{ B e a 8 | s

s T B P q R B W s a q T I s W W 8 D s H q g g 8 P a B T r s H q g g 8 P a B T

4 ( U ' o ' 6

7 8 @ B H I P R ` 8 n W 8 P I 8 T s

s T B P q R s T I s W W 8 D r e H B W P I 8 g R ` 8 D R s a 8 8 B W

e P

s D

I s 8 P s W ` ` 8

8 W ` 8 W P R e P

e P s D

I s 8 P B x 8 W q P I 8 T I s W W 8 D g s P a B 7 8

W R P 8 P I s P R a s B x 8 W T I s W W 8 D r B @ 8 @ B H I P R g s B g B j 8 P I 8 g e P e s D B W R a g s P B R W r

@ 8 g e H P

8 a R a g s g s B g B j s P B R W R x 8 a s D D

a R s B D B P q ` B H P a B e P B R W H R a P I 8 B W

e P

s D

I s 8 P 7 8 ` 8 n W 8 P I 8 $ G r ` 8 W R P 8 ` q r s H |

g s

W a o

7 I 8 W @ 8 s T I B 8 x 8 P I B H a s P 8 @ 8 ` 8 H T a B 8 P I 8 H R e a T 8 s H 8 B W g s P T I 8 ` P R P I 8

T I s W W 8 D

4 ( U '

R R x 8 a T R g 8 P I 8

a R D 8 g R W R B H 8 B W P I 8 H q H P 8 g r @ 8 g B I P T R W H B ` 8 a s ` ` B W

a 8 ` e W ` s W T q ` e a B W P I 8 8 W T R ` B W

a R T 8 H H P R R x 8 a T R g 8

R H H B D 8 8 a a R a H I 8 8

s g

D 8 H P I s P s a 8 e H 8 ` I 8 a 8 s a 8 a 8 H P a B T P 8 ` P R H R e a T 8 H @ I B T I @ R e D ` W s P e a s D D q 8

8 W T R ` 8 ` B W s W R B H 8 D 8 H H 8 W x B a R W g 8 W P s H n 8 ` H B j 8 D R T v T R ` 8 H B 8 s H R e a T 8 s D

I s 8 P W r @ I B T I I s H

8 e B

a R s D 8 H q g R D H I R @ 8 x 8 a r P I 8 ` B H T e H H B R W s

D B 8 H

P R g R a 8 8 W 8 a s D H R e a T 8 H s W ` x s a B s D 8 D 8 W P I T R ` B W H T I 8 g 8 H

W 8

s a P B T e D s a s H

8 T P P R 8 T R W H B ` 8 a 8 ` B W a 8 s D e H 8 H R T I s W W 8 D T R ` B W B H P I s P g s W q


31/75

H R e a T 8 H @ I B T I @ 8 s a 8 B W P 8 a 8 H P 8 ` B W 8 W T R ` B W R a P a s W H g B H H B R W H I s x 8 s H B W B n T s W P

s g R e W P R a 8 ` e W ` s W T q s D a 8 s ` q R W H B ` 8 a H 8 W ` B W s

B 8 T 8 R H q W P s T P B T s D D q T R a

a 8 T P s W ` H 8 g s W P B T s D D q g 8 s W B W e D j W D B H I R a T R g

e P 8 a

a R a s g P 8 P P I a R e I s

T I s W W 8 D @ I B T I a s W ` R g D q T R a a e

P 8 ` R W s x 8 a s 8 o B W o T I s a s T P 8 a H H e T I s H g B I P

8 B W P a R ` e T 8 ` q P a s W H g B H H B R W s T a R H H s a s P I 8 a H B T v D q 8 D 8 H q H P 8 g 8 |

o a B W a 8 B W R a T 8 g 8 W P H r @ 8 a 8 R B W P R s ` x s W T 8

P H 8 s H q P R a 8 T R W B H 8 H

8 8 T I

8 T R W H P a e T P B R W a R g P I 8 R D D R @ B W ` e 8 P R T R a a e

P B R W R o B W o T I s a s T P 8 a H @ R e D `

8 T R g

s a s P B x 8 D q H P a s B I P R a @ s a ` |

o a B j a 8 B W R a T 8 8 W P H r @ 8 a 8 R B W P R s ` x s W T 8

P H 8 s H q g R a 8 T R W B H 8 H

8 8 T I

R @ 8 x 8 a r @ I B D 8 P I 8 a 8 ` e W ` s W T q R P I B H H R e a T 8

a R P 8 T P H s s B W H P H e T I a s W ` R g

T I s a s T P 8 a 8 a a R a r T R W H B ` 8 a P I 8 8 a a R a ` e 8 P R s I e g s W g B H I 8 s a B W |

o a B W P I a 8 8 s W ` R e a

8 W T 8 r @ 8 a 8 R B W P R s ` s W T 8

P H 8 s H q P R @ a 8 T v s W B T 8

8 s T I

I 8 T R ` B W W 8 8 ` H P R T R W H B ` 8 a P I 8 8 a a R a T I s a s T P 8 a B H P B T H R P I 8 T I s W W 8 D s W ` ` 8

T R ` 8 a r s W ` P a q P R s T I B 8 x 8 s H B W B n T s W P ` B H P s W T 8 8 P @ 8 8 W

D s e H B D 8 8 W T R ` 8 `

g 8 H H s 8 H

W 8 R P I 8 H B g

D 8 H P T R ` 8 H B H s 7 8 P s v 8 s B W s a q H q g g 8 P a B T

T I s W W 8 D @ B P I s P a s W H B P B R W

a R s B D B P q P I B H B x 8 H s T I s W W 8 D T s

s T B P q

o D R o D R o I 8 W s P e a s D B W s a q 8 W T R ` B W R a P I B H B H P I 8 W

o k P I 8 a 8

8 P B P B R W T R ` 8 @ B D D a 8

8 s P P I 8 H 8 ` B B P H s W R ` ` W e g 8 a R P B g 8 H s W `

8 a R a g g s v R a B P q x R P B W

8 W T 8 @ 8 P a s W H g B P @ o B P H

8 a H q g R D r s W ` @ B D D R P s B W s W 8 a a R a B o

R a g R a 8 B P H s a 8 a 8 T 8 B x 8 ` B W 8 a a R a r P I s P B H |

H

o

o

l

R W H B ` 8 a s P a s W H B P B R W

a R s B D B P q R

o I 8 T I s W W 8 D T s

s T B P q s H B x 8 W q

8 e s P B R W o B H

o n e a 8 s I 8 T R ` 8 a s P 8 R P I 8 a 8

8 P B P B R W P 8 T I W B e 8

s s B W H P P I 8 a 8 H B ` e s D

a R s B D B P q R 8 a a R a B H ` 8 g R W H P a s P 8 ` B W n e a 8


32/75

0.9

0.91

0.92

0.93

0.94

0.95

0.96

0.97

0.98

0.99

1

1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1

Capac

ity

P(transition)

I(X;Y)(p)I(X;Y)(0.01)

1e-10

1e-09

1e-08

1e-07

1e-06

1e-05

0.0001

0.001

0.01

0.1

0.01 0.1 1

Res

idua

l

error

rate

Code rate

Repetitiion codeH(0.01)

{ B e a 8 | s s

s T B P q R B W s a q H q g g 8 P a B T T I s W W 8 D s P D R @ D R H H a s P 8 H r j

T B 8 W T q R a 8

8 P B P B R W T R ` 8 R a s P a s W H B P B R W

a R s B D B P q R

o

4 ( U 4 ( (

7 8 s a a B x 8 s P

I s W W R W H H 8 T R W ` P I 8 R a 8 g r P I 8 $ # $ % |

{ R a s T I s W W 8 D R T s

s T B P q s W ` s H R e a T 8 R 8 W P a R

q B r

P I 8 W R a s a B P a s a B D q H g s D D ' r P I 8 a 8 8 B H P H s T R ` B W H T I 8 g 8 H e T I P I s P

P I 8 H R e a T 8 B H a 8

a R ` e T 8 ` @ B P I s a 8 H B ` e s D 8 a a R a a s P 8 D 8 H H P I s W '

I s W W R W H

a R R R P I B H P I 8 R a 8 g B H s W 8 B H P 8 W T 8

a R R a s P I 8 a P I s W s g 8 s W H P R

T R W H P a e T P H e T I T R ` 8 H B W P I 8 8 W 8 a s D T s H 8 W

s a P B T e D s a P I 8 T I R B T 8 R s R R `

T R ` 8 B H ` B T P s P 8 ` q P I 8 T I s a s T P 8 a B H P B T H R P I 8 T I s W W 8 D W R B H 8 W

s a s D D 8 D @ B P I P I 8

W R B H 8 D 8 H H T s H 8 r 8 P P 8 a T R ` 8 H s a 8 R P 8 W s T I B 8 x 8 ` q T R ` B W g e D P B

D 8 B W

e P H q g R D H

R W H B ` 8 a s a s P I 8 a s a P B n T B s D T I s W W 8 D @ I B T I g s q a s W ` R g D q T R a a e

P R W 8 B P B W 8 s T I

D R T v R H 8 x 8 W e H 8 ` P R 8 W T R ` 8 H q g R D H B W P I 8 T I s W W 8 D @ 8 P s v 8 P I 8

a R s B D

B P q R s B P T R a a e

P B R W 8 x 8 W P B H P I 8 H s g 8 s H T R a a 8 T P a 8 T 8

P B R W 7 8 B W v 8 T P

8 e B

a R s D 8 B W

e P H q g R D H T D 8 s a D q

m

R a e W B e 8 ` 8 T R ` B W 7 I s P B H P I 8

T s

s T B P q R P I B H T I s W W 8 D

7 8 I s x 8

m

B W

e P s W ` R e P

e P

s P P 8 a W H R a s B x 8 W B W

e P

X

@ B P I B W s a q ` B B P

a 8

a 8 H 8 W P s P B R W

m

r @ 8 I s x 8 8 B I P 8 e B

a R s D 8 B 8 @ B P I o

a R s

B D B P q R e P

e P H q g R D H s W ` W R R P I 8 a H |


33/75

m

m

m

m

m

m

m

m

I 8 W T R W H B ` 8 a B W P I 8 B W R a g s P B R W T s

s T B P q

8 a H q g R D |

X

a a

r

W

o

p

c p r X

D R

o

c p r X

X

o

o

D R

o

o

o

}

D R

o

o

~

I 8 T s

s T B P q R P I 8 T I s W W 8 D B H B W R a g s P B R W B P H

8 a B W s a q ` B B P R P I 8

T I s W W 8 D T R ` B W s W @ 8 n W ` s g 8 T I s W B H g P R 8 W T R ` 8 B W R a g s P B R W B P H B W

T I s W W 8 D B P H H e v 8 T P P R P I 8 8 a a R a

a R

8 a P q ` 8 H T a B 8 ` s R x 8

I 8 s g g B W R ` 8

a R x B ` 8 H s % T R ` 8 P R

8 a R a g P I B H s H q H

P 8 g s P B T T R ` 8 B H R W 8 B W @ I B T I P I 8 R x B R e H B W s a q 8 W T R ` B W R P I 8 H R e a T 8 H q g R D H

B H

a 8 H 8 W P B W P I 8 T I s W W 8 D 8 W T R ` 8 ` R a g { R a R e a H R e a T 8 @ I B T I 8 g B P H s P 8 s T I P B g 8

B W P 8 a x s D o R o H q g R D H r @ 8 P s v 8 P I 8 B W s a q a 8

a 8 H 8 W P s P B R W R P I B H s W ` T R

q B P

P R B P H

r r

s W `

m

R P I 8 8 W T R ` 8 ` D R T v P I 8 a 8 g s B W B W B P H s a 8 B x 8 W q

r s W ` % q

|

-

-

m

s W ` r

- -

-

m

-

-

m

s W ` r

-

-

-

m

- -

m

s W ` r

-

- -

m

W a 8 T 8

P B R W B P I 8 B W s a q W e g 8 a

P I 8 W P I 8 a 8 B H W R 8 a a R a r 8 D H 8

B H P I 8 B P B W 8 a a R a

I B H s g g B W T R ` 8 e H 8 H B P H P R T R a a 8 T P

o 8 a a R a

s P P 8 a W H s W `

P a s W H 8 a e H 8 e D B P H W 8 W 8 a s D s s g g B W T R ` 8 e H 8 H B P H P R T R a a 8 T P

o

8 a a R a

s P P 8 a W H s W ` P a s W H 8 a

o e H 8 e D B P H I 8 s g g B W T R ` 8 H s a 8

T s D D 8 ` 6 s H P I 8 q e H 8 B P H P R T R a a 8 T P

o 8 a a R a H


34/75

I 8 s g g B W T R ` 8 H 8 B H P R a s D D

s B a H

o

l

s W ` ` 8 P 8 T P R W 8 B P 8 a a R a H

D H R P I 8 } R D s q T R ` 8 B H s

o D R T v T R ` 8 @ I B T I T R a a 8 T P H e

P R P I a 8 8 B P

8 a a R a H r s W e W W s g 8 ` T R ` 8 8 B H P H s P

@ I B T I T R a a 8 T P H e

P R P @ R B P 8 a a R a H

1e-20

1e-10

1

1e-08 1e-07 1e-06 1e-05 0.0001 0.001 0.01 0.1

Residualerrorrate

Mean error rate

{ B e a 8 | s g g B W T R ` 8 a 8 H B ` e s D 8 a a R a a s P 8

7 8 T s W P I 8 W T R W H B ` 8 a P I 8 g R a 8 8 W 8 a s D T s H 8 r @ I 8 a 8 @ 8 I s x 8 a s W ` R g B P 8 a a R a H

e W B R a g D q ` B H P a B e P 8 ` B 8 @ 8 s D D R @ P I 8

R H H B B D B P q R P @ R R a g R a 8 B P 8 a a R a H

8 a

B P D R T v

s B W @ 8 R P s B W P I 8

a R s B D B P q R a 8 H B ` e s D 8 a a R a s H P I 8 a 8 g s B W ` 8 a

R P I 8 B W R g B s D H 8 a B 8 H |

m

H

o m l

7 8 W R @ T R W H B ` 8 a P I 8 T s H 8 B W @ I B T I P I 8 H B W s D H R a g 8 H H s 8 H @ 8 @ B H I P R P a s W H 8 a

s a 8 T R W P B W e R e H D q x s a B s D 8 P I s P B H R P I P I 8 H q g R D H @ 8 @ B H I P R P a s W H g B P s a 8

T R W P B W e R e H e W T P B R W H r s W ` P I 8 s H H R T B s P 8 `

a R s B D B P B 8 H R P I 8 H 8 e W T P B R W H s a 8

B x 8 W q s T R W P B W e R e H

a R s B D B P q ` B H P a B e P B R W

7 8 T R e D ` P a q s W ` R P s B W P I 8 a 8 H e D P H s H s D B g B P B W

a R T 8 H H a R g P I 8 ` B H T a 8 P 8 T s H 8

{ R a 8 s g

D 8 r T R W H B ` 8 a s a s W ` R g x s a B s D 8

@ I B T I P s v 8 H R W x s D e 8 H

X p

X

r

o

r @ B P I

a R s B D B P q

X p

X

r B 8

a R s B D B P q ` 8 W H B P q e W T P B R W

X p

7 8 I s x 8 P I 8 s H H R T B s P 8 `

a R s B D B P q W R a g s D B j s P B R W |

p

X p

X

o

o


35/75

H B W R e a R a g e D s R a ` B H T a 8 P 8 8 W P a R

q s W ` P s v B W P I 8 D B g B P |

W1 D B g

k

p

X p

X

D R

}

o

X p

X

~

D B g

k

p

X p

D R

}

o

X p

~

X

D R

X

p

X

X

l

X

D R

}

o

X

~

X

} D B g

k

D R

X

~

l

X

X

W D B g

k

D R

X

I B H B H a s P I 8 a @ R a a q B W s H P I 8 D s P P 8 a D B g B P ` R 8 H W R P 8 B H P R @ 8 x 8 a r s H @ 8 s a 8

R P 8 W B W P 8 a 8 H P 8 ` B W P I 8 ` B 8 a 8 W T 8 H 8 P @ 8 8 W 8 W P a R

B 8 H B 8 B W P I 8 T R W H B ` 8 a s P B R W

R g e P e s D 8 W P a R

q R a T s

s T B P q r @ 8 ` 8 n W 8 P I 8

a R D 8 g s @ s q q e H B W P I 8 n a H P

P 8 a g R W D q s H s g 8 s H e a 8 R |

W

l

X

D R

}

o

X

~

X

7 8 T s W 8 P 8 W ` P I B H P R s T R W P B W e R e H a s W ` R g x 8 T P R a R ` B g 8 W H B R W @ T R W T 8 s D B W

P I 8 @ R D ` B W P 8 a s D 8 I B W ` x 8 T P R a W R P s P B R W s W ` R D ` P q

8 |

l

D R

}

o

~

7 8 T s W P I 8 W s D H R ` 8 n W 8 P I 8 v R B W P s W ` T R W ` B P B R W s D 8 W P a R

B 8 H R a T R W P B W e R e H ` B H

P a B e P B R W H |

D R

}

o

~

r

D R

}

~

r

D R

}

~

@ B P I |

X

c

{ B W s D D q r g e P e s D B W R a g s P B R W R T R W P B W e R e H a s W ` R g x s a B s D 8 H

s W ` f B H ` 8 n W 8 `

e H B W P I 8 ` R e D 8 B W P 8 a s D |

f

X c

D R

}

X r c

X

~

X

c

@ B P I P I 8 T s

s T B P q R P I 8 T I s W W 8 D B x 8 W q g s B g B j B W P I B H g e P e s D B W R a g s P B R W

R x 8 a s D D

R H H B D 8 B W

e P ` B H P a B e P B R W H R a


36/75

n o ( ( #

7 8 R P s B W x s a B R e H

a R

8 a P B 8 H s W s D s R e H P R P I 8 ` B H T a 8 P 8 T s H 8 W P I 8 R D D R @ B W

@ 8 ` a R

P I 8 R D ` P q

8 r e P 8 s T I ` B H P a B e P B R W r x s a B s D 8 8 P T H I R e D ` 8 P s v 8 W P R

8 R @ ` B g 8 W H B R W H

o j W P a R

q B H g s B g B j 8 ` @ B P I 8 e B

a R s D 8 H q g R D H

X

B H D B g B P 8 ` P R

H R g 8 x R D e g 8 B 8 B H R W D q W R W j 8 a R @ B P I B W P I 8 x R D e g 8 P I 8 W

X

B H g s

B g B j 8 ` @ I 8 W

X

o

X c

X

c

7 I s P B H P I 8 g s B g e g ` B 8 a 8 W P B s D 8 W P a R

q R a H

8 T B n 8 ` x s a B s W T 8 @ 8

T I R R H 8 P I B H s H P I 8 x s a B s W T 8 B H s g 8 s H e a 8 R s x 8 a s 8

R @ 8 a 8 W T 8 s a 8

H P s P 8 g 8 W P R P I 8

a R D 8 g B H P R n W ` P I 8 g s B g e g ` B 8 a 8 W P B s D 8 W P a R

q R a

s H

8 T B n 8 ` g 8 s W

R @ 8 a

R W H B ` 8 a P I 8 o ` B g 8 W H B R W s D a s W ` R g x s a B s D 8

r @ B P I P I 8 T R W H P a s B W P H |

X

X

o

X

X

X

@ I 8 a 8 |

X

X

X

I B H R

P B g B j s P B R W

a R D 8 g B H H R D x 8 ` e H B W s a s W 8 g e D P B

D B 8 a H s W ` g s

B g B j B W |

X

D R

X

X

X

X

X

@ I B T I B H R P s B W 8 ` q H R D x B W |

o D R

X

X

H R P I s P @ B P I ` e 8 a 8 s a ` R a P I 8 T R W H P a s B W P H R W P I 8 H q H P 8 g |

X

o

l

l

I 8 W T 8 |

o

D R

7 8 R H 8 a x 8 P I s P | B R a s W q a s W ` R g x s a B s D 8 f @ B P I x s a B s W T 8 r f

r B B P I 8 ` B 8 a 8 W P B s D 8 W P a R

q B H ` 8

8 W ` 8 W P R W D q R W P I 8 x s a B s W T 8 s W `

B H B W ` 8

8 W ` 8 W P R P I 8 g 8 s W r I 8 W T 8 B B B P I 8 a 8 s P 8 a P I 8

R @ 8 a r P I 8 a 8 s P 8 a

P I 8 ` B 8 a 8 W P B s D 8 W P a R

q

I B H 8 P 8 W ` H P R g e D P B

D 8 ` B g 8 W H B R W H B W P I 8 R x B R e H g s W W 8 a


37/75

n # ( U ( #

W P I 8 T s H 8 R T R W P B W e R e H H B W s D H s W ` T I s W W 8 D H @ 8 s a 8 B W P 8 a 8 H P 8 ` B W e W T P B R W H R

H s q P B g 8 r T I R H 8 W a R g s H 8 P R

R H H B D 8 e W T P B R W H r s H B W

e P P R R e a T I s W W 8 D r @ B P I

H R g 8

8 a P e a 8 ` x 8 a H B R W R P I 8 e W T P B R W H 8 B W ` 8 P 8 T P 8 ` s P P I 8 R e P

e P I 8 H 8

e W T P B R W H P I 8 W P s v 8 P I 8

D s T 8 R P I 8 B W

e P s W ` R e P

e P s D

I s 8 P H R P I 8 ` B H T a 8 P 8

T s H 8

7 8 g e H P s D H R P I 8 W B W T D e ` 8 P I 8

a R s B D B P q R s B x 8 W e W T P B R W 8 B W B W v 8 T P 8 `

B W P R P I 8 T I s W W 8 D @ I B T I a B W H e H P R P I 8 B ` 8 s R & % P I B H 8 W 8 a s D s a 8 s R

@ R a v B H v W R @ W s H % $

{ R a 8 s g

D 8 r T R W H B ` 8 a P I 8 8 W H 8 g D 8 H |

o

H B W

j s T I x s D e 8 R ` 8 n W 8 H s ` B 8 a 8 W P e W T P B R W r s W ` P R 8 P I 8 a @ B P I s

a R s B D B P q

` B H P a B e P B R W r H s q r @ 8 I s x 8 s W 8 W H 8 g D 8 R P 8 P I s P I 8 a 8 g s q 8

` B H T a 8 P 8 R a T R W P B W e R e H T R W H B ` 8 a

I s H 8 H I B P v 8 q B W

R W H B ` 8 a s H 8 P R a s W ` R g x s a B s D 8 H

|

o

k @ I 8 a 8 8 s T I

P s v 8 H R W s a s W ` R g x s D e 8 s T T R a ` B W P R s } s e H H B s W ` B H P a B e P B R W @ B P I

H P s W ` s a ` ` 8 x B s P B R W

P I 8 W |

k

H B W T

B H P I 8 @ I B P 8 W R B H 8 8 W H 8 g D 8 r s W ` D B g B P 8 ` P R 8 a P j s W ` @ B P I s x 8 a s 8

R @ 8 a

u R a 8 8 W 8 a s D D q r @ 8 I s x 8 R a a s W ` R g x s a B s D 8 H

X

k P I 8 8 W H 8 g D 8 R s W `

D B g B P 8 ` e W T P B R W H |

X

k

X

H B W T

@ I 8 a 8 R T R e a H 8 @ 8 a 8 g 8 g 8 a a R g P I 8 H s g

D B W P I 8 R a 8 g P I s P |

X

}

~

@ 8 s D H R T R W H B ` 8 a e W T P B R W H D B g B P 8 ` P R P B g 8 B W P 8 a x s D r P I 8 W @ 8 R

P s B W R W D q W R W j 8 a R T R 8 T B 8 W P H s W ` @ 8 T s W T R W H B ` 8 a P I 8 8 W H 8 g D 8

P R 8 a 8

a 8 H 8 W P 8 ` q s W @ ` B g 8 W H B R W s D @

a R s B D B P q ` B H P a B e P B R W

X X X

u R a 8 H

8 T B n T s D D q r B @ 8 T R W H B ` 8 a P I 8 8 W H 8 g D 8 R D B g B P 8 `

R @ 8 a q

r

s W ` D B g B P 8 ` P R

s W ` P B g 8 D B g B P 8 ` H B W s D H W R W j 8 a R R W D q B W B W P 8 a x s D

r @ 8 n W ` P I s P P I 8 8 W H 8 g D 8 B H a 8

a 8 H 8 W P 8 ` q s W @ ` B g 8 W H B R W s D

a R s B D B P q ` B H P a B e P B R W @ I B T I B H j 8 a R R e P H B ` 8 P I 8 @ H

I 8 a 8 a s ` B e H


38/75

q T R W H B ` 8 a B W P I 8 D s P P 8 a P q

8 H R 8 W H 8 g D 8 H r @ 8 T s W n P P I 8 g B W P R P I 8 n W B P 8

` B g 8 W H B R W s D T R W P B W e R e H ` B 8 a 8 W P B s D 8 W P a R

q ` 8 n W B P B R W H B x 8 W B W H 8 T P B R W

n 4 ( U o ' 6

7 8 T R W H B ` 8 a T I s W W 8 D H B W @ I B T I W R B H 8 B H B W v 8 T P 8 ` B W ` 8

8 W ` 8 W P D q R P I 8 H B W s D P I 8

s a P B T e D s a W R B H 8 H R e a T 8 R B W P 8 a 8 H P B H P I 8 H R T s D D 8 ` $ b

{ e a P I 8 a @ 8 a 8 H P a B T P T R W H B ` 8 a s P B R W H P R P I 8 n W s D T D s H H R 8 W H 8 g D 8

7 8 I s x 8 s H B W s D @ B P I s x 8 a s 8

R @ 8 a

r P B g 8 D B g B P 8 ` P R s W ` s W ` @ B ` P I

D B g B P 8 ` P R

7 8 P I 8 W T R W H B ` 8 a P I 8 @ H s g

D 8 H

p

s W ` f

p

P I s P T s W 8 e H 8 ` P R

T I s a s T P 8 a B H 8 R P I P I 8 B W

e P s W ` R e P

e P H B W s D H I 8 W P I 8 a 8 D s P B R W H I B

8 P @ 8 8 W

P I 8 B W

e P s W ` R e P

e P B H B x 8 W q |

f

p

p

p

r o

@

@ I 8 a 8

p

B H a R g P I 8 s W ` D B g B P 8 ` } s e H H B s W ` B H P a B e P B R W @ B P I j 8 a R g 8 s W s W `

x s a B s W T 8 |

k

@ I 8 a 8

k

B H P I 8

R @ 8 a H

8 T P a s D ` 8 W H B P q

H B H B W ` 8

8 W ` 8 W P R

@ 8 R P s B W P I 8 T R W ` B P B R W s D 8 W P a R

q s H 8 B W H R D 8 D q

` 8

8 W ` 8 W P R W P I 8 W R B H 8 H R e a T 8 r s W ` a R g 8 e s P B R W n W ` B P H x s D e 8 |

f

r

o

D R

k

8 W T 8 |

f f

I 8 T s

s T B P q R P I 8 T I s W W 8 D B H P I 8 W R P s B W 8 ` q g s B g B j B W P I B H R x 8 a s D D B W

e P

` B H P a B e P B R W H P I B H B H s T I B 8 x 8 ` q g s B g B j B W @ B P I a 8 H

8 T P P R P I 8 ` B H P a B e P B R W

R

H e v 8 T P P R P I 8 s x 8 a s 8

R @ 8 a D B g B P s P B R W |

w

p

x

H @ 8 I s x 8 H 8 8 W P I B H B H s T I B 8 x 8 ` @ I 8 W @ 8 I s x 8 s } s e H H B s W ` B H P a B e P B R W r I 8 W T 8

@ 8 n W ` R P I

s W ` f g e H P I s x 8 } s e H H B s W ` B H P a B e P B R W H W

s a P B T e D s a

I s H

x s a B s W T 8

s W ` f I s H x s a B s W T 8

k

7 8 T s W P I 8 W 8 x s D e s P 8 P I 8 T I s W W 8 D T s

s T B P q r s H |

f

o

D R

k

o

D R

k

o

D R } o

k

~


39/75

I B H T s

s T B P q B H B W B P H

8 a T I s W W 8 D H q g R D r H R @ 8 T s W R P s B W s a s P 8

8 a H 8 T R W ` r

q g e D P B

D B T s P B R W q @ r B 8 a R g @ r g e D P B

D B T s P B R W q |

D R

}

o

k

~

B P H

R

I s W W R W H P I B a ` P I 8 R a 8 g r P I 8 $ # $ % |

I 8 T s

s T B P q R s T I s W W 8 D s W ` D B g B P 8 ` P R 8 a P j r

8 a P e a 8 ` q

s ` ` B P B x 8 @ I B P 8 } s e H H B s W W R B H 8 R

R @ 8 a H

8 T P a s D ` 8 W H B P q

k

s W ` s W `

D B g B P 8 ` P R B H B x 8 W q |

D R

} o

k

~

B P H

@ I 8 a 8

B H P I 8 s x 8 a s 8 P a s W H g B P P 8 `

R @ 8 a

I 8 H 8 T R W ` P 8 a g @ B P I B W P I 8 D R B W 8 e s P B R W B H P I 8 H B W s D P R W R B H 8 a s P B R

o H 8 a x 8 P I s P P I 8 T s

s T B P q B W T a 8 s H 8 H g R W R P R W B T s D D q s W ` @ B P I R e P R e W `

s H P I 8

B W T a 8 s H 8 H

B g B D s a D q P I 8 T s

s T B P q B W T a 8 s H 8 H g R W R P R W B T s D D q s H P I 8 s W ` @ B ` P I B W T a 8 s H 8 H

e P P R s D B g B P H B W s q D R a H 8

s W H B R W R a D W |

D W o

C C C

@ 8 R P s B W |

k

D R

I B H B H R P 8 W a 8 @ a B P P 8 W B W P 8 a g H R # r r @ I B T I B H ` 8 n W 8 ` q

I 8 D B g B P B W x s D e 8 B H P I 8 W |

k

D R

I B H B H T s D D 8 ` P I 8

I s W W R W B g B P

I 8 T s

s T B P q R P I 8 T I s W W 8 D B H s T I B 8 x 8 ` @ I 8 W P I 8 H R e a T 8

D R R v H D B v 8 W R B H 8

I B H B H P I 8 s H B H R a H

a 8 s ` H

8 T P a e g P 8 T I W B e 8 H R g R ` e D s P B R W s W ` B W

s a P B T e D s a R ` 8 B x B H B R W u e D P B

D 8

T T 8 H H u


40/75

1. Foundations: Probability, Uncertainty, and Information2. Entropies Defined, and Why they are Measures of Information3. Source Coding Theorem; Prefix, Variable-, and Fixed-Length Codes4. Channel Types, Properties, Noise, and Channel Capacity5. Continuous Information; Density; Noisy Channel Coding Theorem6. Fourier Series, Convergence, Orthogonal Representation

7. Useful Fourier Theorems; Transform Pairs; Sampling; Aliasing

8. Discrete Fourier Transform. Fast Fourier Transform Algorithms9. The Quantized Degrees-of-Freedom in a Continuous Signal10. Gabor-Heisenberg-Weyl Uncertainty Relation. Optimal Logons11. Kolmogorov Complexity and Minimal Description Length.

Jean Baptiste Joseph Fourier (1768 - 1830)

37


41/75

Introduction to Fourier Analysis, Synthesis, and Transforms

It has been said that the most remarkable and far-reaching relationship in all of mathemat-ics is the simple Euler Relation,

ei + 1 = 0 (1)

which contains the five most important mathematical constants, as well as harmonic analysis.This simple equation unifies the four main branches of mathematics: {0,1} represent arithmetic, represents geometry, i represents algebra, and e = 2.718... represents analysis, since one wayto define e is to compute the limit of (1 + 1

n)n as n.

Fourier analysis is about the representation of functions (or of data, signals, systems, ...) interms of such complex exponentials. (Almost) any function f(x) can be represented perfectly asa linear combination of basis functions:

f(x) =k

ckk(x) (2)

where many possible choices are available for the expansion basis functions k(x). In the case

of Fourier expansions in one dimension, the basis functions are the complex exponentials:

k(x) = exp(ikx) (3)

where the complex constant i =1. A complex exponential contains both a real part and an

imaginary part, both of which are simple (real-valued) harmonic functions:

exp(i) = cos() + i sin() (4)

which you can easily confirm by using the power-series definitions for the transcendental functionsexp, cos, and sin:

exp() = 1 +

1!+

2

2!+

3

3!+ +

n

n!+ , (5)

cos() = 1 2

2!+

4

4!

6

6!+ , (6)

sin() = 3

3!+

5

5!

7

7!+ , (7)

Fourier Analysis computes the complex coefficients ck that yield an expansion of some functionf(x) in terms of complex exponentials:

f(x) =n

k=n

ck exp(ikx) (8)

where the parameter k corresponds to frequency and n specifies the number of terms (whichmay be finite or infinite) used in the expansion.

Each Fourier coefficient ck in f(x) is computed as the (inner product) projection of the functionf(x) onto one complex exponential exp(ikx) associated with that coefficient:

ck =1

T

+T/2

T/2f(x)exp(ikx)dx (9)

where the integral is taken over one period (T) of the function if it is periodic, or from to+

if it is aperiodic. (An aperiodic function is regarded as a periodic one whose period is

).

For periodic functions the frequencies k used are just all multiples of the repetition frequency;for aperiodic functions, all frequencies must be used. Note that the computed Fourier coefficientsck are complex-valued.

38


42/75

" % )

2 3 5 7 9 A C E E C G H P G H S C A V C E 9 3 A 9 Y a S 5 Y c 9 3 5 7 d f g i q 9 r C r a 3 E 7 3 v C x G 5 A g q d f g x i

d f g i r 9 c 3 S c H 3 7 7 3 a C 5 C E G H 9 c C c G C x r

C 3 7 C E P C c C 3 E c 3 3 5 G H 9 c V E 3 V C E c 9 C 7 3 a 7 9 5 f g i G 5 A Y 3 7 f g i a 3 E 9 5 c C C E 7

G 5 A k

l m o

Y 3 7 f g i Y 3 7 f g i 6 g

9 a

{ 3 c C E 9 7 C

l

m o

7 9 5 f g i 7 9 5 f g i g

9 a

{

3 c C E 9 7 C

l

m o

7 9 5 f g i Y 3 7 f g i 6 g ~ f i

C S 7 C c C E 3 5 C Y C E a S 5 Y c 9 3 5 c 3 v C G 5 k

{

3 c C E 9 7 C

C 5 c C 3 S E 9 C E C E 9 C 7 a 3 E d f g i 9 7 k

x

{

f x { Y 3 7 f g i { 7 9 5 f g i i f i

C E C c C 3 S E 9 C E 2 3 C Y 9 C 5 c 7 G E C k

x {

l

m o

d f g i Y 3 7 f g i g f i

{

l m o

d f g i 7 9 5 f g i g f i

C 3 V C c G c c C 3 S E 9 C E C E 9 C 7 V E 3 P 9 A C 7 G 5 C V G 5 7 9 3 5 3 a c C a S 5 Y c 9 3 5 d f g i 9 5

c C E v 7 3 a Y 3 7 9 5 C G 5 A 7 9 5 C c C E v 7 r

- -

C c

f g i C G 5 7 S v 3 a 7 9 5 C G 5 A Y 3 7 9 5 C c C E v 7 k

f g i

x

{

f x

{

Y 3 7 f g i

{

7 9 5 f g i i f i

G 5 A

f g i q c C c E S 5 Y G c C A 3 S E 9 C E C E 9 C 7 a 3 E G a S 5 Y c 9 3 5 d f g i k

f g i

x

{

f x { Y 3 7 f g i { 7 9 5 f g i i


43/75

C E C x{

G 5 A {

G E C c C 3 S E 9 C E Y 3 C Y 9 C 5 c 7 r 2 4 3 5 7 9 A C E c C 9 5 c C E G H 9 P 9 5 c C

A 9 7 Y E C V G 5 Y C c C C 5 d f g i G 5 A

{

f g i f G 7 7 S v 9 5 d f g i 9 7 C H H C G P C A C 5 3 S a 3 E

c C 9 5 c C E G H c 3 C 9 7 c i k

lm o

d f g i

{

f g i

m

g

9 Y 7 9 v V H 9 C 7 c 3 k

l

m o

d f g i

m

g

xm

{

f xm

{

m

{

i

f x

x

i m

{

f x

{

x {6 i m f

{

{ i m f i

3 c C c G c c C c C E v 7 9 5 P 3 H P 9 5 x G 5 A G E C G H H G 5 A P G 5 9 7 C 5 x

{

x {

G 5 A

{

{ r

d

- - -

C 3 S E 9 C E C E 9 C 7 G 5 A 3 S E 9 C E Y 3 C Y 9 C 5 c 7 G E C A C 5 C A G 7 G 3 P C r 3 C P C E C

v G C 5 Y 3 S 5 c C E 7 3 v C V E 3 H C v 7 k

r 9 5 c C E G H 7 9 5 C S G c 9 3 5 7 q a G 9 H c 3 C 9 7 c r C r r k

d f g i

g

3 E

d f g i

g E G c 9 3 5 G H

g 9 E E G c 9 3 5 G H

r G H c 3 S x { q { C 9 7 c q c C 7 C E 9 C 7 A 3 C 7 5 3 c Y 3 5 P C E C q

r C P C 5 c 3 S c C 7 C E 9 C 7 Y 3 5 P C E C 7 q c C E C 7 S H c 9 7 5 3 c d f g i

d f g i

g

g

c C 5 k

x { ~ {

l

o

7 9 5 f g i g

{

o

3 A A

C P C 5

7 3 k

d f g i

7 9 5 f g i

7 9 5 f g i

7 9 5 f g i

S c 7 C E 9 C 7 9 P C 7 d f i

r


44/75

-1.5

-1

-0.5

0

0.5

1

1.5

-1 0 1 2 3 4 5 6 7

fseries( 1,x)fseries( 7,x)fseries(63,x)

rect(x)

9 S E C k V V E 3 9 v G c 9 3 5 7 c 3 E C Y c G 5 S H G E V S H 7 C

-1.5

-1

-0.5

0

0.5

1

1.5

-0.2 -0.15 -0.1 -0.05 0 0.05 0.1 0.15 0.2

fseries( 63,x)fseries(163,x)

rect(x)

9 S E C k 9 7 V C 5 3 v C 5 3 5


45/75

3 C P C E q 9 5 E C G H H 9 a C C G v V H C 7 C 5 Y 3 S 5 c C E C A 9 5 7 9 5 G H V E 3 Y C 7 7 9 5 c 9 5 7 G E C 7 9 v

V H C E G 7 k

r a d f g i 9 7 3 S 5 A C A 9 5 f ~ i G 5 A V 9 C Y C 9 7 C Y 3 5 c 9 5 S 3 S 7 q c C 3 S E 9 C E 7 C E 9 C 7

Y 3 5 P C E C 7 c 3

d f g

i d f g i G c 9 5 c C E 9 3 E V 3 9 5 c 7 G 5 A

d f i d f

i

G c G 5 A r

r a d f g i 9 7 G H 7 3 Y 3 5 c 9 5 S 3 S 7 3 5 f ~ i q c C 7 S v 3 a c C 3 S E 9 C E C E 9 C 7 9 7 C S G H

c 3 d f g i G c G H H V 3 9 5 c 7 r

r x { G 5 A { c C 5 A c 3 C E 3 G c H C G 7 c G 7 a G 7 c G 7 r

-

C E 9 c C S 7 9 5

{

9 5 c C 3 P 9 3 S 7 G a E 3 v c C a 3 E v S H G C a 3 E 7 9 5 G 5 A Y 3 7 k

d f g i

x

{

f x { Y 3 7 f g i { 7 9 5 f g i i

x

{

x {

f

{

{

i

{

f

{

{

i

9

{

{

f i

9 c k

x

{ f x

{ !

{i

{ f x { ! { i

3 7 C E P C k

{

$

{

f % i

C E C

$

A C 5 3 c C 7 c C Y 3 v V H C Y 3 5 & S G c C q G 5 A k

{

lm o

d f g i

{

g f i

( -

2 3 5 7 9 A C E c C P G H S C 3 a G V C E 9 3 A 9 Y a S 5 Y c 9 3 5 d f g i G c ) A 9 7 Y E C c C C S G H H 7 V G Y C A

P G H S C 7 3 a g k

g 1 3 5 f 5

)

q 3 ~ ~ 8 8 8 ~ ) i

c E c 3 5 A Y 3 C Y 9 C 5 c 7

{ 7 S Y c G c k

d f 3 5 i

{

{

A

1 {

f i

C D E G I Q I R G E T G V W R Y R I a R a R E I V G e f g h e q h V h t v x y W R Y t v x Q E e t v x W R Y t v x


46/75

S H c 9 V H

A

1

G 5 A 7 S v 9 c E C 7 V C Y c c 3 3 k

1

d f 3 5 i

A

1

1

{

{

A

1 {

S c c C 7 S v 3 a G C 3 v C c E 9 Y 7 C E 9 C 7 G 5 A H G c G 5 c G 7 7 C E c 9 3 5 k

1

A

1 {

jl

l

l

m

l

l

ln

A

{

A

{

o

)

7 3 k

)

1

d f 3 5 i

A

1

f i

C E Y 9 7 C a 3 E c C E C G A C E k 7 3 9 5 7 C E c 9 5 C S G c 9 3 5 9 5 c 3 C S G c 9 3 5 7 G c 9 7 C 7

c C 3 E 9 9 5 G H C S G c 9 3 5 7 r

-5

-4

-3

-2

-1

0

1

2

3

4

0 2 4 6 8 10 12

saw(x)trig.pts.8trig.pts.64

9 S E C k % G 5 A V 3 9 5 c 9 5 c C E V 3 H G c 9 3 5 7 c 3 7 G c 3 3 c a S 5 Y c 9 3 5

C Y G 5 c E G 5 7 H G c C c 9 7 G Y 9 5 c 3 Y 3 7 G 5 A 7 9 5 7 C E 9 C 7 k

d f g 1 i x

m

{

f x { Y 3 7 f

3

info theory lectures

Documents