chapter 2 probability and stochastic processes

Chapter 2Probability and Stochastic Processes

WITS Lab, NSYSU.2

Table of ContentsProbability

Random Variables, Probability Distributions, and Probability DensitiesFunctions of Random VariablesStatistical Averages of Random VariablesSome Useful Probability DistributionsUpper Bounds on the Tail ProbabilitySums of Random Variables and the Central Limit Theorem

Stochastic ProcessesStatistical AveragesPower Density SpectrumResponse of a Linear Time-Invariant System to a Random Input SignalSampling Theorem for Band-Limited Stochastic ProcessesDiscrete-Time Stochastic Signals and SystemsCyclostationary Processes

WITS Lab, NSYSU.3

Introduction

Probability and stochastic process is important in:The statistical modeling of sources that generate the information;The digitization of the source output;The characterization of the channel through which the digital information is transmitted;The design of the receiver that processes the information-bearing signal from the channel;The evaluation of the performance of the communication system.

WITS Lab, NSYSU.4

2.1 Probability

Sample space or certain event of a die experiment:

The six outcomes are the sample points of the experiment.An event is a subset of S, and may consist of any number of sample points. For example:

The complement of the event A, denoted by , consists of all the sample points in S that are not in A:

6,5,4,3,2,1=S

4,2=AA

6,5,3,1=A

WITS Lab, NSYSU.5

Two events are said to be mutually exclusive if they have no sample points in common – that is, if the occurrence of one event excludes the occurrence of the other. For example:

The union (sum) of two events in an event that consists of all the sample points in the two events. For example:

6,3,1 ;4,2 == BA

SAA

CBDC

=

===

∪

∪ 6,3,2,13,2,1

events. exclusivemutually are and AA

2.1 Probability

WITS Lab, NSYSU.6

The intersection of two events is an event that consists of the points that are common to the two events. For example:

When the events are mutually exclusive, the intersection is the null event, denoted as φ. For example:

3,1== CBE ∩

φ=AA∩

2.1 Probability

WITS Lab, NSYSU.7

Associated with each event A contained in S is its probability P(A).Three postulations:

P(A)≥0.The probability of the sample space is P(S)=1. Suppose that Ai , i =1, 2, …, are a (possibly infinite) number of events in the sample space S such that

Then the probability of the union of these mutually exclusive events satisfies the condition:

,...2,1 ; =≠= jiAA ji φ∩

∑=⎟⎟⎠

⎞⎜⎜⎝

⎛

ii

ii APAP )(∪

2.1 Probability

WITS Lab, NSYSU.8

Joint events and joint probabilities (two experiments)If one experiment has the possible outcomes Ai , i =1,2,…,n, and the second experiment has the possible outcomes Bj , j =1,2,…,m, then the combined experiment has the possible joint outcomes(Ai ,Bj), i =1,2,…,n, j =1,2,…,m.Associated with each joint outcome (Ai ,Bj) is the joint probabilityP (Ai ,Bj) which satisfies the condition:

Assuming that the outcomes Bj , j =1,2,…,m, are mutually exclusive, it follows that:

If all the outcomes of the two experiments are mutually exclusive, then:

1),(0 ≤≤ ji BAP

∑=

=m

jiji APBAP

1)(),(

∑∑= =

=n

i

m

jji BAP

1 11),(

2.1 Probability

WITS Lab, NSYSU.9

Conditional probabilitiesThe conditional probability of the event A given the occurrence of the event B is defined as:

provided P(B)>0.)(

),()|(BP

BAPBAP =

)()|()()|(),( APABPBPBAPBAP ==

If two events and are mutually exclusive, , then ( | ) 0.

A B A BP A B

φ==

∩

.1)|( and have we, ofsubset a is If == BAPBBAAB ∩

. and of occurrence ussimultaneo thedenotes ),(is,That . ofy probabilit theas dinterprete is ),(BABAP

BABAP ∩

2.1 Probability

WITS Lab, NSYSU.10

Bayes’ theorem:

P(Ai) represents their a priori probabilities and P(Ai|B) is the a posteriori probability of Ai conditioned on having observed the received signal B.

i 1

If , 1, 2,..., , are mutually exclusive events such that

and is an arbitrary event with nonzero probability, then( , ) ( | )

( )

i

n

i

ii

A i n

A S

BP A B PP A B

P B

=

=

=

= =

∪

1

( | ) ( )

( | ) ( )

i in

j jj

B A P A

P B A P A=∑

2.1 Probability

WITS Lab, NSYSU.11

Statistical independence

When the events A and B satisfy the relation P(A,B)=P(A)P(B), they are said to be statistically independent.Three statistically independent events A1, A2, and A3 must satisfy the following conditions:

).()|( then , ofoccurrence on the dependnot does of occurrence theIf

APBAPBA=

)()()()|(),( BPAPBPBAPBAP ==

)()()(),,()()(),()()(),()()(),(

321321

3232

3131

2121

APAPAPAAAPAPAPAAPAPAPAAPAPAPAAP

====

2.1 Probability

WITS Lab, NSYSU.12

2.1.1 Random Variables, Probability Distributions, and Probability Densities

The function X(s) is called a random variable.Example 1: If we flip a coin, the possible outcomes are head (H) and tail (T), so S contains two points labeled H and T. Suppose we define a function X(s) such that:

Thus we have mapped the two possible outcomes of the coin-flipping experiment into the two points ( +1,-1) on the real line.Example 2: Tossing a die with possible outcomes S=1,2,3,4,5,6. A random variable defined on this sample space may be X(s)=s, in which case the outcomes of the experiment are mapped into the integers 1,…,6, or, perhaps, X(s)=s2, in which case the possible outcomes are mapped into the integers 1,4,9,16,25,36.

line. real on the numbers ofset a is range whoseand isdomain whose)(funciton a define we, elements

and space sample a having experimentan Given

SsXSs

S∈

⎩⎨⎧

==+

=T)(s 1-H)(s 1

)(sX

WITS Lab, NSYSU.13

The function F(x) is called the probability distribution functionof the random variable X.It is also called the cumulative distribution function (CDF).

∞<<∞≤=

≤∞∞≤

x-xXPxFxF

xXPxxX

X

),()( i.e., ,)(by simply it denote

and )( asevent thisofy porbabilit the writeWe).,(- interval in thenumber realany is where

event heconsider t uslet , variablerandom aGiven

1)(0 ≤≤ xF.1)( and 0)( =∞=−∞ FF


WITS Lab, NSYSU.14

Examples of the cumulative distribution functions of two discrete random variables.


WITS Lab, NSYSU.15

An example of the cumulative distribution function of a continuous random variable.


WITS Lab, NSYSU.16

An example of the cumulative distribution function of a random variable of a mixed type.


WITS Lab, NSYSU.17

The derivative of the CDF F(x), denoted as p(x), is called the probability density function (PDF) of the random variable X.

When the random variable is discrete or of a mixed type, the PDF contains impulses at the points of discontinuity of F(x):

∫∞−

∞<<∞−=

∞<<∞−=

x

xduupxF

xdx

xdFxp

,)()(

,)()(

∑=

−==n

iii xxxXPxp

1

)()()( δ


WITS Lab, NSYSU.18

( )

. range in the PDF under the area the

simply is event theofy probabilit The

)(

)()()( )()()(

)()()(

. where, , intervalan in fallsvariablerandomay that probabilitthegDeterminin

21

21

1221

2112

2112

1221

2

1

xXxxXx

dxxp

xFxFxXxPxXxPxFxF

xXxPxXPxXP

xxxxX

x

x

≤<≤<

=

−=≤<⇒≤<+=

≤<+≤=≤

>

∫


WITS Lab, NSYSU.19

Multiple random variables, joint probability distributions, and joint probability densities: (two random variables)

( )( ) ( ) ( ) .0,,, : thatNote

1,),(

PDFs. called are variables theofoneover gintegratin from obtained )( and )( PDFs The

)(),( )(),(

),(),( :PDFJoint

),(),(),( :CDFJoint

12

2- - 121

21

12212121

2121

2

21

2-

x

- 1212211211 2

=−∞=∞−=−∞∞−

=∞∞=

==

∂∂∂

=

=≤≤=

∫ ∫

∫∫

∫ ∫

∞

∞

∞

∞

∞

∞−

∞

∞−

∞ ∞

xFxFF

Fxddxxxp

xpxp

xpdxxxpxpdxxxp

xxFxx

xxp

udduuupxXxXPxxFx


marginal

WITS Lab, NSYSU.20

Multiple random variables, joint probability distributions, and joint probability densities: (multidimensional random variables)

( )( ) .0,...,,,,

).,...,,,(,...,,,,

),...,,(),...,,(

),...,,(...

),...,,( PDFJoint

...),...,,(...

),...,,(),...,,( CDFJoint . variablesrandom are 21 that Suppose

41

54141

413221

2121

21

- 2

x

- 121

221121

1 2

=−∞−∞=∞∞

=

∂∂∂∂

=

=

≤≤≤==

∫ ∫

∫ ∫ ∫

∞

∞−

∞

∞−

∞ ∞ ∞−

n

nn

nn

nn

n

n

n

x x

n

nnn

i

xxxFxxxxFxxxF

xxxpdxdxxxxp

xxxFxxx

xxxp

duudduuuup

xXxXxXPxxxF,...,n,,, iX

n


WITS Lab, NSYSU.21

Conditional probability distribution functions (consider two random variables):

( )

( )( )

( ) ( )

( ) ( )

( ) ( ))()(

,,

,,

)(

,|X

.on dconditione variablerandom that theyprobabilit theis |Xevent theofy Probabilit

222

22121

2222

21212121

22

2121

222211

222211

222211

222

1 1 222

2

22

1 2

22

xxFxFxxxFxxF

duupduup

duduuupduduuup

duup

duduuupxXxxxP

xXxxxXxXxxx

xxx

x x xxx

x

xx

x x

xx

∆−−∆−−

=

⎥⎦⎤

⎢⎣⎡ −

−=

=≤<∆−≤

≤<∆−≤≤<∆−≤

∫∫∫ ∫ ∫∫

∫∫ ∫

∆−

∞−∞−

∞− ∞−

∆−

∞−∞−

∆−

∞− ∆−


WITS Lab, NSYSU.22


( ) ( ) ( )( )

( )1 2

2 2

1 2

1 2 21 1 2 2 1 2

2 2

1 2 1 2

2

Divide both numerator and denominator by and let 0The conditional CDF of given the random variable is given by:

, /X | |

/

,

(

x x

x xX X

F x x xP x X x F x x

F x x

p u u du dux

p

−∞ −∞

∆ ∆ →

∂ ∂≤ = ≡ =

∂ ∂

⎡ ⎤∂ ⎢ ⎥⎣ ⎦∂

=∂

∫ ∫( )( ) ( )

( ) ( )

1

2

1 2 1

22 2

2

2 2

, *

)

Observe that | 0 and | 1.

x

x

p u x du

p xu dux

F x F x

−∞

−∞

=⎡ ⎤⎢ ⎥⎣ ⎦

∂

−∞ = ∞ =

∫∫


WITS Lab, NSYSU.23


( )

( ) ( )( )( )

( ) ( )

1

1 21 2

2

1 2

1 2 2 1

Differentiating Equation * with respect to , we obtain

, |

We may express the joint PDF , in terms of the

conditional PDFs, | or | , as:

x

p x xp x x

p x

p x x

p x x p x x

=

( ) ( )( ) ( )

1 2 1 2 2

2 1 1

, | ( )

|

p x x p x x p x

p x x p x

=

=


WITS Lab, NSYSU.24

( ) ( ) ( )

( )

1 2 1 2 1 1

1 2 1

Joint PDF of the random variables , 1,2,..., , , ,..., , ,..., | ,..., ,...,

Joint conditional CDF: , ,..., | ,...,

...

i

n k k n k n

k k n

X i np x x x p x x x x x p x x

F x x x x x

p

+ +

+

=

=

=( )

( )( ) ( )( )

1

1 2 1 1 2

1

2 1 2 1

2 1

, ,..., , ,..., ...

,...,

, ,..., , ,..., ,..., , ,...,

, ,..., | ,..., 0

kx x

k k n k

k n

k k n k k n

k k n

u u u x x du du du

p x x

F x x x x F x x x x

F x x x x

+−∞ −∞

+

+ +

+

∞ =

−∞ =

∫ ∫


Conditional probability distribution functions (consider multidimensional variables):

WITS Lab, NSYSU.25

Statistically independent random variables:If the experiments result in mutually exclusive outcomes, the probability of an outcome is independent of an outcome in any other experiment.The joint probability of the outcomes factors into a product of the probabilities corresponding to each outcome.Consequently, the random variables corresponding to the outcomes in these experiments are independent in the sense that their joint PDF factors into a product of marginal PDFs.The multidimensional random variables are statistically independent if and only if:

( ) ( ) ( ) ( )( ) ( ) ( ) ( )nn

nn

xpxpxpxxxpxFxFxFxxxF

...,...,,...,...,,

2121

2121

==


WITS Lab, NSYSU.26

Problem: given a random variable X, which is characterized by its PDF p(x), determine the PDF of the random variable Y=g(X), where g(X) is some given function of X.When the mapping g from X to Y is one-to-one, the determination of p(y) is relatively straightforward.However, when the mapping is not one-to-one, as in the case, for example, when Y=X2, we must be very careful in our derivation of p(y).

2.1.2 Functions of Random Variables

WITS Lab, NSYSU.27


Transformation of a random variable

WITS Lab, NSYSU.28

Example 2.1-1:Y=aX+b; where a and b are constant and assume that a>0.

( )( )

( ) ⎟⎠⎞

⎜⎝⎛ −

=

⎟⎠⎞

⎜⎝⎛ −

==

−≤=≤+=≤=

∫∞

abyp

ayp

abyFdxxp

abyXPybaXPyYPyF

XY

XX

Y

1

)()()()(

ab-y

-


WITS Lab, NSYSU.29

Example 2.1-1 (cont.):Y=aX+b; where a and b are constant and assume that a>0.


WITS Lab, NSYSU.30

Example 2.1-2:Y=aX3+b; a>0.The mapping between X and Y is one-to-one.

[ ] ⎥⎥⎦

⎤

⎢⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −

−=

⎥⎥⎦

⎤

⎢⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −

=⎥⎥⎦

⎤

⎢⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −

≤=

≤+=≤=

31

32

31

31

3

)(31)(

PDFs twoebetween th iprelationshdesired theyields respect to ation withDifferenti

)()()(

aby

XY

X

Y

pabya

yp

y

abyF

abyXP

ybaXPyYPyF


WITS Lab, NSYSU.31

Example 2.1-3:Y=aX2+b; a>0.The mapping between X and Y is not one-to-one.


WITS Lab, NSYSU.32

Example 2.1-3 (cont.):Y=aX2+b; a>0.

Differentiating with respect to y, we obtain the PDF of Y in terms of the PDF of X in the form:

⎟⎟⎠

⎞⎜⎜⎝

⎛ −−−⎟⎟

⎠

⎞⎜⎜⎝

⎛ −=

⎟⎟⎠

⎞⎜⎜⎝

⎛ −≤=≤+=≤=

abyF

abyFyF

abyXPybaXPyYPyF

XXY

Y

)(

)()()( 2

[ ] [ ][ ( ) ] [ ( ) ]

( )2 ( ) 2 ( )

X XY

p y b a p y b ap y

a y b a a y b a− − −

= +− −


WITS Lab, NSYSU.33

Example 2.1-3 (cont.):Y=g(X)=aX2+b; a>0 has two solutions:

pY(y) consists of two terms corresponding to these two solutions:

In general, suppose that x1,x2, …, xn are the real roots of g(x)=y. The PDF of the random variable Y=g(X) may be expressed as

abyx

abyx −

−=−

= 21 ,

( ) ( )( )∑

= ′=

n

i i

iXY xg

xpyp1

( ) ( )( )

( )( ) ]/[

]/[]/[]/[

2

2

1

1

abyxgabyxp

abyxgabyxp

yp XXY

−−=′

−−=+

−=′

−==


WITS Lab, NSYSU.34

Function of one random variable (discrete case):Suppose X is a discrete random variable that can have one of nvalues x1, x2, …, xn.Let g(X) be a scalar-valued function.Y=g(X) is a discrete random variable that can have one of m, m≤n, values y1, y2, …, ym.If g(X) is a one-to-one mapping, then m will be equal to n.If g(X) is many-to-one, then m will be smaller than n.The CDF of Y can be obtained easily from the CDF function of X as:

. tomap that of valuesallover is sum thewhere)()(

ii

ii

yxxXPyYP ∑ ===


WITS Lab, NSYSU.35

Function of one random variable (continuous case):If X is a continuous random variable, then the pdf of Y=g(X) can be obtained from the pdf of X.Let y be a particular value of Y and let x(1), x(2), …, x(n) be roots of the equation y=g(x). That is y=g(x(1))=g(x(2))= … = y=g(x(n)).

( ) ( )( ) [ ]yyxgyxP

yyyfyyYyP Y

∆+≤<=→∆∆=∆+≤<

: 0 as


WITS Lab, NSYSU.36

Function of one random variable (continuous case):


x

=y

WITS Lab, NSYSU.37

Function of one random variable (continuous case):( ) ( ) ( ) ( )( ) ( ) ( ) ( )( )

( ) ( ) ( )( )( )( ) ( ) ( )( ) ( ) ( )( ) ( )

1 1 1 2 2 2

3 3 3

1 1 2 2 3 3

X X X

P y Y y y P x X x x P x x X x

P x X x x

f x x f x x f x x

< < + ∆ = < < + ∆ + −∆ < <

+ < < + ∆

= ∆ + ∆ + ∆

( ) ( )( )

( )( )( )

( )( )( )

( )( )33

22

11

have we,/ is of slope theSince

xgyxxg

yxxgyx

xyxgxg

′∆=∆′

∆=∆′∆=∆

∆∆′

( )( )( )( )( )

( )( )( )( )

( )( )( )( )

( )( )( )( )( )∑

= ′=⇒

∆′

+∆′

+∆′

=∆

k

ii

iX

Y

XXXY

xgxfyf

yxgxfy

xgxfy

xgxfyyf

1

3

3

2

2

1

1


WITS Lab, NSYSU.38

Function of several random variables (continuous case):Goal: find the joint distribution of n random variables Y1, Y2, …, Yn given the distribution of n related random variables, X1, X2, …, Xn, where

Let’s start with a mapping of two random variables onto two other random variables:

( ) niXXXgY nii ,...,2,1 ,,...,, 21 ==

( ) ( )21222111 , , XXgYXXgY ==( ) ( )( ) ( )( )

( ) ( ) 222122112111

212122

211121

, and ,such that plane

, in theregion thefind toneed We.,y and,y of roots theare ,...,2,1 ,, Suppose

yyxxgyyyxxgy

xxxxgxxgkkixx ii

∆+<<∆+<<

===


WITS Lab, NSYSU.39

Function of several random variables (continuous case):There are k such regions as shown in the following figure for the case of k=3.


WITS Lab, NSYSU.40

Function of several random variables (continuous case):

( ) ( )( ) ( )

( )

( )( ) ( )( )

( ) ( )( )∑=

=

∂∂

∂∂

∂∂

∂∂

=

∆∆

k

iii

iiXX

YY

ii

xxJxxf

yyf

YY

xg

xg

xg

xg

xxJ

xxJxxJ

yy

1 21

21,21,

21

2

2

1

2

2

1

1

1

21

2121

21

,,

,

as and of pdfjoint obtain the weregions, all fromon contributi thesummingBy

,

as defined isation transform theofJacobian the

is , where,

toequal is ramparallelog

eachofareatheand ramparallelogaofconsistsregion Each

21

21


WITS Lab, NSYSU.41

Function of several random variables (continuous case):

( )( ) ( ) ( )( )

( )

( )

1 1 11 1 2 2

1 21

We can generalize this result to the -variate case as

, , ...,, , ...,

where denotes the Jacobian of the transformation defined by the

following determinant when

i i ik X n n

Y n ii

i

n

f x g x g x gf y y y

J

J

− − −

=

= = == ∑

( )

( ) ( ) ( ) ( ) ( )

( )

( ) ( ) ( )1 1 11 21 2

1

1 1 11 2 2 1 2 n

1 1 1

1 2

1 2 at , ,...,

evaluating at the solution

, , ..., of ( ) [ x ,g x , ..., g x ]:

i i in n

i

i in n

ni

n n n

n x g x g x g

i th x

g x g x g y g x g

g g gx x x

Jg g gx x x − − −

− − −

= = =

− =

= = = =

∂ ∂ ∂∂ ∂ ∂

=∂ ∂ ∂∂ ∂ ∂


WITS Lab, NSYSU.42

Example 2.1-4

Aybxybxybxp

yyypAJ

Ab

,...,n,, iYbXYAX

nnAAXY

niXaY

n

j

n

j

n

jjnjnjjjjX

nY

-ij

n

jjiji

n

jjiji

det1,...,,

),...,,(.det isation transform thisofJacobian The

.matrix inverse theof elements theare

21

matrix. an is e wher

,...,2,1 ,

1 1 12211

21

1

1

1

1

⎟⎟⎠

⎞⎜⎜⎝

⎛⋅==⋅==

=

==⇒=

×=

==

∑ ∑ ∑

∑

∑

= = =

⋅

=

−

=


WITS Lab, NSYSU.43

The mean or expected value of X, which characterized by its PDF p(x), is defined as:

This is the first moment of random variable X.The n-th moment is defined as:

Define Y=g(X), the expected value of Y is:

∫∞

∞−=≡ dxxxpmXE x )()(

dxxpxXE nn )()( ∫∞

∞−=

[ ] ∫∞

∞−== dxxpxgXgEYE )()()()(

2.1.3 Statistical Averages of Random Variables

WITS Lab, NSYSU.44

The n-th central moment of the random variable X is:

When n=2, the central moment is called the varianceof the random variable and denoted as :

In the case of two random variables, X1 and X2, with joint PDF p(x1,x2), we define the joint moment as:

( )[ ] ∫∞

∞−−=−= dxxpmxmXEYE n

xn

x )()()(

[ ] 22222

22

)()()(

)()(

xx

xx

mXEXEXE

dxxpmx

−=−=

−= ∫∞

∞−

σ

σ

2xσ

21212121 ),()( dxdxxxpxxXXE nknk ∫ ∫∞

∞−

∞

∞−=


WITS Lab, NSYSU.45

The joint central moment is defined as:

If k=n=1, the joint moment and joint central momentare called the correlation and the covariance of the random variables X1 and X2, respectively.The correlation between Xi and Xj is given by the joint moment:

[ ]∫ ∫∞

∞−

∞

∞−−−=

−−

21212211

2211

),()()(

)()(

dxdxxxpmxmx

mXmXEnk

nk

∫ ∫∞

∞−

∞

∞−= jijijiji dxdxxxpxxXXE ),()(


WITS Lab, NSYSU.46

The covariance between Xi and Xj is given by the joint central moment:

The n×n matrix with elements μij is called the covariance matrix of the random variables, Xi, i=1,2, …, n.

( )( )( )( )( )

jiji

jijijijijiji

jijijiji

jiijijiijji

jiijjii

jjiiij

mmXXE

mmmmmmdxdxxxpxx

mmdxdxxxpxx

dxdxxxpmmmxmxxx

dxdxxxpmxmx

mXmXE

j

j

−=

+−−=

−=

+−−=

−−=

−−≡

∫ ∫∫ ∫∫ ∫∫ ∫

∞

∞−

∞

∞−

∞

∞−

∞

∞−

∞

∞−

∞

∞−

∞

∞−

∞

∞−

)(

),(

),(

),(

),(

][µ


WITS Lab, NSYSU.47

Two random variables are said to be uncorrelated if E(XiXj)=E(Xi)E(Xj)=mimj.Uncorrelated → Covariance μij = 0.If Xi and Xj are statistically independent, they are uncorrelated.If Xi and Xj are uncorrelated, they are not necessarystatistically independently.Two random variables are said to be orthogonal if E(XiXj)=0.

Two random variables are orthogonal if they are uncorrelated and either one of both of them have zero mean.


WITS Lab, NSYSU.48

Characteristic functionsThe characteristic function of a random variable X is defined as the statistical average:

Ψ(jv) may be described as the Fourier transform of p(x).The inverse Fourier transform is:

First derivative of the above equation with respect to v:

∫∞

∞−=≡ dxxpejveE jvxjvX )()()( ψ

∫∞

∞−

−= dvejvxp jvx)(21)( ψπ

∫∞

∞−= dxxpxej

dvjvd jvx )()(ψ


WITS Lab, NSYSU.49

Characteristic functions (cont.)First moment (mean) can be obtained by:

Since the differentiation process can be repeated, n-thmoment can be calculated by:

Suppose the characteristic function can be expanded in a Taylor series about the point v=0:

0

)()()(=

−=v

n

nnn

dvjvdjXE ψ

0

)()(=

−==v

x dvjvdjmXE ψ

∑∑∞

==

∞

=

=⇒⎥⎦

⎤⎢⎣

⎡=

000 !)()()(

!)()(

n

nn

n

vnn

n

njvXEjv

nv

dvjvdjv ψψψ


WITS Lab, NSYSU.50

Characteristic functions (cont.)Determining the PDF of a sum of statistically independentrandom variables:

( )

[ ]nXY

i

n

iXYnn

nn

n

i

jvxn

i

jvX

n

ii

jvYY

n

ii

jvjv

X

jvjvxpxpxpxxxp

dxdxdxxxxpeeE

XjvEeEjvXY

i

ii

)()(

d)distributey identicall andnt (independe iid are If

)()( )()...()(),...,,(

t,independenlly statistica are variablesrandom theSince

...),...,,(...

exp)()(

12121

212111

11

ψψ

ψψ

ψ

=⇒

=⇒=

⎟⎟⎠

⎞⎜⎜⎝

⎛=⎥

⎦

⎤⎢⎣

⎡=

⎥⎦

⎤⎢⎣

⎡⎟⎠

⎞⎜⎝

⎛==⇒=

∏

∫ ∫ ∏∏

∑∑

=

∞

∞−

∞

∞−==

==


WITS Lab, NSYSU.51

Characteristic functions (cont.)The PDF of Y is determined from the inverse Fourier transform of ΨY(jv).Since the characteristic function of the sum of n statistically independent random variables is equal to the product of the characteristic functions of the individual random variables, it follows that, in the transform domain, the PDF of Y is the n-fold convolution of the PDFs of the Xi.Usually, the n-fold convolution is more difficult to perform than the characteristic function method in determining the PDF of Y.


WITS Lab, NSYSU.52

Characteristic functions (for n-dimensional random variables)

If Xi, i=1,2,…,n, are random variables with PDF p(x1,x2,…,xn), the n-dimensional characteristic function is defined as:

For two dimensional characteristic function:

nn

n

iii

n

iiin

dxdxdxxxxpxvj

XvjEjvjvjv

...),...,,(exp...

exp),...,,(

21211

121

∫ ∫ ∑

∑∞

∞−

∞

∞−=

=

⎟⎠

⎞⎜⎝

⎛=

⎥⎦

⎤⎢⎣

⎡⎟⎠

⎞⎜⎝

⎛=ψ

021

2211

21

212

21

2121)(

21

),()(

),(),(

==∂∂

∂−=

= ∫ ∫∞

∞−

∞

∞−

+

vvvv

jvjvXXE

dxdxxxpejvjv xvxvj

ψ

ψ


WITS Lab, NSYSU.53

2.1.4 Some Useful Probability Distributions

Binomial distribution:

Let

what is the probability distribution function of Y ?1

where the , 1,2,..., are statistically iid,n

i ii

Y X X i n=

= =∑

( ) ( ) pXPXP ==−== 110

( ) ( ) ( )

( ) ( )

)()1(

)( :is of PDF

!!! 1

0

0

kyppkn

kykYPypY

knkn

kn

ppkn

kYP

knkn

k

n

k

knk

−−⎟⎟⎠

⎞⎜⎜⎝

⎛=

−==

−≡⎟⎟

⎠

⎞⎜⎜⎝

⎛−⎟⎟

⎠

⎞⎜⎜⎝

⎛==

−

=

=

−

∑

∑

δ

δ

WITS Lab, NSYSU.54

Binomial distribution:The CDF of Y is:

where [y] denotes the largest integer m such that m≤y.The first two moments of Y are:

The characteristic function is:

( ) ( )[ ]

knky

kpp

kn

yYPyF −

=

−⎟⎟⎠

⎞⎜⎜⎝

⎛=≤= ∑ )1(

0

( )

)1()1()(

2

222

pnppnpnpYE

npYE

−=

+−=

=

σ

njvpepj )1()( +−=νψ


WITS Lab, NSYSU.55

Uniform DistributionThe first two moments of X are:


( )

22

222

)(121

)(31)(

)(21

ba

abbaYE

baXE

−=

++=

+=

σ( )

22

222

)(121

)(31)(

)(21

ba

abbaXE

baXE

−=

++=

+=

σ

)()(

abjveej

jvajvb

−−

=νψ


WITS Lab, NSYSU.56

Gaussian (Normal) DistributionThe PDF of a Gaussian or normal distributed random variable is:

where mx is the mean and σ2 is the variance of the random variable.The CDF is:

22 2/)(

21)( σ

σπxmxexp −−=

( ) ( )

)2

(211)

2(

21

21

121)(

2/2/ 222

σσ

πσπσσ

xx

mx tx mu

mxerfcmxerf

dteduexF xx

−−=

−+=

== ∫∫−

∞−

−

∞−

−−


( ) 2 2

xu m dut dtσ σ

−= =

WITS Lab, NSYSU.57

Gaussian (Normal) distribution:


WITS Lab, NSYSU.58

Gaussian (Normal) Distributionerf( ) and erfc( ) denote the error function and complementary error function, respectively, and are defined as:

erf(-x)=-erf(x), erfc(-x)=2-erfc(x), erf(0)=erfc(∞)=0, and erf(∞)=erfc(0)=1.For x>mx, the complementary error functions is proportional to the area under the tail of the Gaussian PDF.

( ) ( ) )(12 and 2 22

0xerfdtexerfcdtexerf

x

tx t −=== ∫∫∞ −−

ππ


WITS Lab, NSYSU.59

Gaussian (Normal) DistributionThe function that is frequently used for the area under the tailof the Gaussian PDF is denoted by Q(x) and is defined as:

2 / 21 1( ) 022 2

t

x

xQ x e dt erfc xπ

∞ − ⎛ ⎞= = ≥⎜ ⎟⎝ ⎠∫


WITS Lab, NSYSU.60

Gaussian (Normal) DistributionThe characteristic function of a Gaussian random variable with mean mx and variance σ2 is:

The central moments of a Gaussian random variable are:

The ordinary moments may be expressed in terms of the central moments as:

( ) ( ) ( ) 2222 2/12/

21 σσ

σπψ vjvmmxjvx xx edxeejv −∞

∞−

−− =⎥⎦

⎤⎢⎣

⎡= ∫

[ ]⎩⎨⎧ −⋅⋅⋅⋅

==− ) (odd 0)(even )1(31

)(kkk

mXEk

kk

xσ

µ

[ ] ikix

k

i

k mik

XE −=∑ ⎟⎟

⎠

⎞⎜⎜⎝

⎛= µ

0


WITS Lab, NSYSU.61

Gaussian (Normal) DistributionThe sum of n statistically independent Gaussian random variables is also a Gaussian random variable.

( ) ( )

. varianceand

mean with ddistribute-Gaussian is Therefore,

and where

2

1

22

1

2/

1

2/

1

1

2222

y

y

n

iiy

n

iiy

vjvmn

i

vjvmn

iXY

n

ii

mY

mm

eejvjv

XY

yyii

i

σ

σσ

ψψ σσ

∑∑

∏∏

∑

==

−

=

−

=

=

==

===

=


WITS Lab, NSYSU.62

Chi-square distributionIf Y=X2, where X is a Gaussian random variable, Y has a chi-square distribution. Y is a transformation of X.There are two type of chi-square distribution:

Central chi-square distribution: X has zero mean.Non-central chi-square distribution: X has non-zero mean.

Assuming X be Gaussian distributed with zero mean and variance σ2, we can apply (2.1-47) to obtain the PDF of Ywith a=1 and b=0;

( ) ( )( )

( )( ) ]/[

]/[]/[]/[

1

1

1

1

abyxgabyxp

abyxgabyxp

yp XXY

−−=′

−−=+

−=′

−==


WITS Lab, NSYSU.63

Central chi-square distributionThe PDF of Y is:

The CDF of Y is:

The characteristic function of Y is:

0 ,21)(

22/ ≥= − yey

yp yY

σ

σπ

( ) dueu

duupyF uyy

YY

22/

00

121)( σ

σπ−∫∫ ==

2/12 )21(1)(σ

ψvj

jvY −=


WITS Lab, NSYSU.64

Chi-square (Gamma) distribution with n degrees of freedom.


The inverse transform of this characteristic function yields the PDF:

.varianceandmean zero with variablesrandomGaussian (iid) ddistributey identicall

andt independenlly statistica are ,,...,2,1 , ,

2

1

2

σ

niXXY i

n

ii ==∑

=

2/2 )21(1)( nY vj

jvσ

ψ−

=

0

212

1)(2212

2/≥

⎟⎠⎞

⎜⎝⎛Γ

= − y,e ynσ

yp σ-y/n/

nnY


WITS Lab, NSYSU.65

Chi-square (Gamma) distribution with n degrees of freedom (cont.).

When n=2, the distribution yields the exponential distribution.

( )

ππ21

23

21

0 integer an !1-p)(

0 ,)(

:asdefinedfunction,gamma theis )(

0

1

=⎟⎠⎞

⎜⎝⎛Γ=⎟

⎠⎞

⎜⎝⎛Γ

>=Γ

>=Γ

Γ

∫∞ −−

pp

pdtetp

ptp


WITS Lab, NSYSU.66


The PDF of a chi-square distributed random variable for several degrees of freedom.


WITS Lab, NSYSU.67


The first two moments of Y are:

The CDF of Y is:

42

4242

2

22)(

)(

σσ

σσ

σ

nnnYE

nYE

y =

+=

=

( ) ∫ ≥⎟⎠⎞

⎜⎝⎛Γ

= −−y un

nnY ydueu

nyF

0

2/12/

2/0 ,

212

1 2σ

σ


WITS Lab, NSYSU.68


The integral in CDF of Y can be easily manipulated into the form of the incomplete gamma function, which is tabulated by Pearson (1965).When n is even, the integral can be expressed in closed form. Let m=n/2, where m is an integer, we can obtain:

0 ,)2

(!

11)( 2

1

0

2/ 2

≥−= ∑−

=

− yyk

eyF km

k

yY σ

σ


WITS Lab, NSYSU.69

Non-central chi-square distributionIf X is Gaussian with mean mx and variance σ2, the random variable Y=X2 has the PDF:

The characteristic function corresponding to this PDF is:

0 ),cosh(21)( 2

2/)( 22

≥= +− ymy

ey

yp xmyY

x

σσπσ

)21/(2/12

22

)21(1)( σ

σψ vjvjm

Yxe

vjjv −

−=


WITS Lab, NSYSU.70

Non-central chi-square distribution with n degrees of freedom

The characteristic function is:. toequal varianceidentical and ,,...,2,1 ,mean

with variablesrandomGaussian (iid) ddistributey identicall

andt independenlly statistica are ,,...,2,1 , ,

2

1

2

σnim

niXXY

i

i

n

ii

=

==∑=

⎟⎟⎟⎟

⎠

⎞

⎜⎜⎜⎜

⎝

⎛

−−=

∑=

21

2

2/2 21exp

)21(1)(

σσψ

vj

mjv

vjjv

n

ii

nY


WITS Lab, NSYSU.71


The characteristic function can be inverse Fourier transformed to yield the PDF:

( )

2 2( 2) / 4 ( ) / 2/ 2 12 2 2

2

2 2

1

1( ) ( ) ( ), 02

where, is called the non-centrality parameter:

and is the th-order modified Bessel function

n s yY n

n

ii

y sp y e I y ys

s

s m

I x

σ

α

σ σ

α

− − +−

=

= ≥

=∑

2

0

of thefirst kind, which may be represented by the infinite series:

( / 2) ( ) , 0! ( 1)

k

k

xI x xk k

α

α α

+∞

=

= ≥Γ + +∑


WITS Lab, NSYSU.72


The CDF is:

The first two moments of a non-central chi-square-distributed random variable are:

( )

∫ −+−

−

⎟⎠⎞

⎜⎝⎛=

y

nus

n

Y dusuIesuyF

0 212/2/)(

4/2

22 )(2

1)(22

σσσ

2242

2222242

22

42)(42)(

)(

snsnsnYE

snYE

y σσσ

σσσ

σ

+=

+++=

+=


WITS Lab, NSYSU.73


When m=n/2 is an integer, the CDF can be expressed in terms of the generalized Marcum’s Q function:

( )

( ) ( ) ( )

( ) ( ) ( )

2 2

2 2

2 2

1( ) / 2

1

1/ 21

0

/ 21

022 2

22

, ( )

,

where , , 0

By using and let , it is easily shown:

mx a

m mb

kma bk

kk

a bk

k

xQ a b x e I ax dxa

bQ a b e I aba

bQ a b e I ab b aa

u sx a σσ

−∞ − +

−

−− +

=

∞− +

=

⎛ ⎞= ⎜ ⎟⎝ ⎠

⎛ ⎞= + ⎜ ⎟⎝ ⎠⎛ ⎞= > >⎜ ⎟⎝ ⎠

= =

∫

∑

∑

( ) 1 ,Y mysF y Q

σ σ⎛ ⎞

= − ⎜ ⎟⎜ ⎟⎝ ⎠


WITS Lab, NSYSU.74

Rayleigh distributionRayleigh distribution is frequently used to model the statistics of signals transmitted through radio channels such as cellular radio.Consider a carrier signal s at a frequency ω0 and with an amplitude a:

The received signal sr is the sum of n waves:

)exp( 0tjas ω⋅=

[ ] [ ]

∑

∑

=

=

=

+≡+=

n

iii

n

iiir

jajr

tjrtjas

1

01

0

)exp()exp( where

)(exp)(exp

θθ

θωθω


WITS Lab, NSYSU.75

Rayleigh distribution

Because (1) n is usually very large, (2) the individual amplitudes ai are random, and (3) the phases θi have a uniform distribution, it can be assumed that (from the central limit theorem) x and y are both Gaussian variables with means equal to zero and variance:

θθ

θθ

θθθ

sin cos :where

sin and cos :have We

sincos)exp( :Define

22211

11

ryrxyxr

ayax

jyxajajr

n

iii

n

iii

n

iii

n

iii

==+=

≡≡

+≡+=

∑∑

∑∑

==

==

222 σσσ ≡= yx


WITS Lab, NSYSU.76

Rayleigh distributionBecause x and y are independent random variables, the joint distribution p(x,y) is

The distribution p(r,θ) can be written as a function of p(x,y) :

⎟⎟⎠

⎞⎜⎜⎝

⎛ +−== 2

22

2 2exp

21)()(),(

σπσyxypxpyxp

⎟⎟⎠

⎞⎜⎜⎝

⎛−=

=−

=∂∂∂∂∂∂∂∂

≡

=

2

2

2exp

2),(

cossinsincos

////

),(),(

σπσθ

θθθθ

θθ

θ

rrrp

rrr

yryxrx

J

yxpJrp


WITS Lab, NSYSU.77

Rayleigh distributionThus, the Rayleigh distribution has a PDF given by:

The probability that the envelope of the received signal does not exceed a specified value R is given by the corresponding cumulative distribution function (CDF):

⎪⎩

⎪⎨⎧ ≥==

−

∫otherwise 0

0 ),()(22 2/

22

0

rerdrprp

r

R

σπ

σθθ

0 ,exp1)(2222 2/

0

2/2 ≥−== −−∫ rdueurF r

ru

Rσσ

σ


WITS Lab, NSYSU.78

Rayleigh distributionMean:

Variance:

Median value of r is found by solving:

Monents of R are:

Most likely value:= max pR(r) = σ.

σπσ 2533.12

)(][0

==== ∫∞

drrrpRErmean

22

2

0

2222

4292.02

2

2)(][][

σπσ

πσσ

=⎟⎠⎞

⎜⎝⎛ −=

−=−= ∫∞

drrprREREr

( )∫=medianr

drrp02

1

σ177.1=medianr

( ) ⎟⎠⎞

⎜⎝⎛ +Γ=

212][ 2/2 kRE kk σ


WITS Lab, NSYSU.79

Rayleigh distribution


WITS Lab, NSYSU.80

Rayleigh distributionProbability That Received Signal Doesn’t Exceed A Certain Level (R)

⎟⎟⎠

⎞⎜⎜⎝

⎛−−=

⎟⎟⎠

⎞⎜⎜⎝

⎛−−=

⎟⎟⎠

⎞⎜⎜⎝

⎛−=

=

∫

∫

2

2

02

2

02

2

2

0

2exp1

2exp

2

exp

)()(

σ

σ

σσ

r

u

duuu

duuprF

r

r

r

R


WITS Lab, NSYSU.81

Rayleigh distributionMean value:

02 2 2

2 2 20 0

2 2

2 200

2

20

[ ] ( )

exp exp2 2

exp exp2 212 exp

2212 1.25332 2

meanr E R rp r dr

r r rdr rd

r rr dr

r dr

σ σ σ

σ σ

πσσπσ

ππσ σ σ

∞

∞ ∞

∞ ∞

∞

= =

⎛ ⎞ ⎛ ⎞= − = − −⎜ ⎟ ⎜ ⎟

⎝ ⎠ ⎝ ⎠⎛ ⎞ ⎛ ⎞

= − ⋅ − + −⎜ ⎟ ⎜ ⎟⎝ ⎠ ⎝ ⎠

⎛ ⎞= ⋅ −⎜ ⎟

⎝ ⎠

= ⋅ = =

∫

∫ ∫

∫

∫

0-0=0


WITS Lab, NSYSU.82

Rayleigh distribution:Mean square value:

2

02

22

2

2

002

22

2

2

0

2

02

2

2

3

0

22

22

exp2

2exp2

2exp

2exp

2exp

)(][

σσ

σ

σσ

σσσ

=⎟⎟⎠

⎞⎜⎜⎝

⎛−⋅−=

⎟⎟⎠

⎞⎜⎜⎝

⎛−⋅+⎟⎟

⎠

⎞⎜⎜⎝

⎛−⋅−=

⎟⎟⎠

⎞⎜⎜⎝

⎛−−=⎟⎟

⎠

⎞⎜⎜⎝

⎛−=

=

∞

∞∞

∞∞

∞

∫

∫∫

∫

r

drrrrr

rdrdrrr

drrprRE

0-0=0


WITS Lab, NSYSU.83

Rayleigh distributionVariance:

22

22

222

4292.02

2

)2

()2(

][][

σπσ

πσσ

σ

=⎟⎠⎞

⎜⎝⎛ −⋅=

⋅−=

−= REREr


WITS Lab, NSYSU.84

Rayleigh distributionMost likely value

Most Likely Value happens when: dp(r) / dr = 0

σσσσ

σσσσσ

σσ

6065.021exp

2exp)(

02

exp22

2exp1)(

2

2

2

2

2

4

2

2

2

2

=⎟⎠⎞

⎜⎝⎛−

=⎟⎟⎠

⎞⎜⎜⎝

⎛−=⇒

=⇒

=⎟⎟⎠

⎞⎜⎜⎝

⎛−⋅

⋅−⎟⎟⎠

⎞⎜⎜⎝

⎛−=

==

rr

rrrp

r

rrrdr

rdp


WITS Lab, NSYSU.85

Rayleigh distributionCharacteristic function

( )

( ) ( )

( ) ( ) ( )( ) ( ) ,...2,1,0 ,

!;;

:function etrichyupergeomconfluent theis ;21,1where

21

21;

21,1

sinrcosr

011

11

2/22211

0

2/20

2/2

2/

0 2

22

2222

22

−−≠+ΓΓΓ+Γ

=

⎟⎠⎞

⎜⎝⎛ −

+⎟⎠⎞

⎜⎝⎛ −=

+=

=

∑

∫∫

∫

∞

=

−

∞ −∞ −

−∞

ββα

βαβα

σπσ

σσ

σψ

σ

σσ

σ

k

k

v

rr

jvrrR

kkxkxF

aF

evjvF

drvrejdrvre

dreerjv


WITS Lab, NSYSU.86

Rayleigh distributionCharacteristic function (cont.)

( )∑∞

=

−

−−=⎟

⎠⎞

⎜⎝⎛ −

⎟⎠⎞

⎜⎝⎛ −

011

11

!12;

21,1

:as expressed

bemay ;21,1 shown that has (1990)Beaulieu

k

ka

kkaeaF

aF


WITS Lab, NSYSU.87

Generalized Rayleigh distribution

Y=R2 is chi-square distributed with n degrees of freedom.PDF is given by:

. variablesradomGaussianmean zero ddistributey identicall t,independen

llystatistica are ,,...,2,1 , where,Consider 1

2 niXXR i

n

ii == ∑

=

( )( )

0 ,

212

22 2/

2/2

1

≥⎟⎠⎞

⎜⎝⎛Γ

= −

−

−

ren

rrp r

nn

n

Rσ

σ


WITS Lab, NSYSU.88

Generalized Rayleigh distributionWhen n is an even number, i.e., n=2m, the CDF of R can be expressed in the closed form:

For any integer n, the k-th moment of R is:

( ) 0 ,2!

111

02

22/ 22

≥⎟⎟⎠

⎞⎜⎜⎝

⎛−= ∑

−

=

− rrk

erFm

k

kr

R σσ

( ) ( )( )

0 ,

21

21

2 2/2 ≥⎟⎠⎞

⎜⎝⎛Γ

⎥⎦⎤

⎢⎣⎡ +Γ

= kn

knRE kk σ


WITS Lab, NSYSU.89

Rice distributionWhen there is a dominant stationary (non-fading) signal component present, such as a line-of-sight (LOS) propagation path, the small-scale fading envelope distribution is Rice.

θθ

θωωωθω

sincos

)(

)](exp[)exp(])[( )exp()](exp['

22200

esdirect wav

0

wavesscattered

0

ryrAx

yAxr

tjrtjjyAxtjAtjrsr

==+

++=

+≡++≡++=


WITS Lab, NSYSU.90

Rice distributionBy following similar steps described in Rayleigh distribution, we obtain:

( )2 2

02 2 2

2

0 2 20

exp for A 0,r 0 ( ) 2

0 for 0where

1 cos exp2

is the modified zeroth

r r r

r r

r r A ArIp r

(r )

Ar ArI dπ

σ σ σ

θ θσ π σ

⎧ ⎛ ⎞ ⎛ ⎞+− ≥ ≥⎪ ⎜ ⎟ ⎜ ⎟= ⎨ ⎝ ⎠ ⎝ ⎠

⎪ <⎩

⎛ ⎞ ⎛ ⎞=⎜ ⎟ ⎜ ⎟

⎝ ⎠ ⎝ ⎠∫

00

-order Bessel function.

! 2

i

ii

xI (x)i

∞

=

⎛ ⎞= ⎜ ⎟⋅⎝ ⎠∑


WITS Lab, NSYSU.91

Rice distributionThe Rice distribution is often described in terms of a parameter K which is defined as the ratio between the deterministic signal power and the variance of the multi-path. It is given by K=A2/(2σ2) or in terms of dB:

The parameter K is known as the Rice factor and completely specifies the Rice distribution.As A 0, K -∞ dB, and as the dominant path decreases in amplitude, the Rice distribution degenerates to a Rayleighdistribution.

[dB] 2

log10)( 2

2

σAdBk ⋅=


WITS Lab, NSYSU.92

Rice distribution


WITS Lab, NSYSU.93

Rice distribution

[ ]

[ ][ ] [ ]

∑

∑

∑

=

=

=

=

+≡++≡++=

+≡

+⎥⎦

⎤⎢⎣

⎡=

++=

n

iii

n

iii

n

iiir

jajr

tjrtjjyAxtjAtjr

tjAtjjr

tjAtjja

tjAtjas

1

00

00

00

001

01

0

)exp()exp(' where

)(exp)exp()( )exp()(exp'

)exp()exp()exp('

)exp()()exp(

)exp()(exp

θθ

θωωωθω

ωωθ

ωωθ

ωθω


WITS Lab, NSYSU.94

Rice distribution

Because (1) n is usually very large, (2) the individual amplitudes ai are random, and (3) the phases θi have a uniform distribution, it can be assumed that (from the central limit theorem) x and y are both Gaussian variables with means equal to zero and variance:

θθ

θθ

θθθ

sin cos )( and

sin and cos :have We

sincos)exp(' :Define

22211

11

ryrAxyAxr

ayax

jyxajajr

n

iii

n

iii

n

iii

n

iii

==+++=

≡≡

+≡+=

∑∑

∑∑

==

==

222 σσσ ≡= yx


WITS Lab, NSYSU.95

Rice distributionBecause x and y are independent random variables, the joint distribution p(x,y) is

The distribution p(r,θ) can be written as a function of p(x,y) :

⎟⎟⎠

⎞⎜⎜⎝

⎛ +−== 2

22

2 2exp

21)()(),(

σπσyxypxpyxp

rrr

yryxrx

J

yxpJrp

=−

=∂∂∂∂∂∂∂∂

≡

=

θθθθ

θθ

θ

cossinsincos

////

),(),(


WITS Lab, NSYSU.96

Rice distribution

⎟⎠⎞

⎜⎝⎛

⎟⎟⎠

⎞⎜⎜⎝

⎛ +−=

⎟⎟⎠

⎞⎜⎜⎝

⎛ −+−=

⎟⎟⎠

⎞⎜⎜⎝

⎛ +−−=

⎟⎟⎠

⎞⎜⎜⎝

⎛ +−=

22

22

2

22

2

22

2

22

cosexp21

2exp

2cos2exp

2

2)sin()cos(exp

2

2exp

2),(

σθ

πσσ

σθ

πσ

σθθ

πσ

σπσθ

ArArr

ArArr

rArr

yxrrp


WITS Lab, NSYSU.97

Rice distribution

⎪⎩

⎪⎨⎧

≥⎟⎠⎞

⎜⎝⎛

⎟⎟⎠

⎞⎜⎜⎝

⎛ +−=

=

∫

∫

otherwise 0

0 cosexp21

2exp

),()(

:bygiven (pdf)function density y probabilitahason distributi RiceThe

2

022

22

2

2

0

rdArArr

drprp

π

π

θσ

θπσσ

θθ


WITS Lab, NSYSU.98

Rice distributionJust as the Rayleigh distribution is related to the central chi-square distribution, the Rice distribution is related to the non-central chi-square distribution.

2 21 2 1 2

2

To illustrate this relation, let , where and are statistically independent Gaussian random variables withmeans , 1, 2, and common variance .i

Y X X X X

m i σ

= +

=

( ) ( )2 2

2 2 21 2

/ 202 2

has a non-central chi-square distribution with non-centrality parameter . The PDF of Y, obtainedfrom Equation 2.1-118 for n 2, is:

1 2 , 02

s yY

Ys m m

p y e I y yσ

σ σ− +

= +=

⎛ ⎞= ≥⎜ ⎟⎝ ⎠


WITS Lab, NSYSU.99

Rice distribution

This PDF characterizes the statistic of the envelope of a signal corrupted by additive narrowband Gaussian noise.It is also used to model the signal statistics of signals transmitted through some radio channels.The CDF of R is obtained from Equation 2.1-124:

( ) ( ) 0 ,

:is variable,of change simple aby 140-2.1Equation fromobtained , of PDF The . variblerandom new a Define

202/

2

222

≥⎟⎠⎞

⎜⎝⎛=

=

+− rrsIerrp

RyR

srR σσ

σ

( ) 123)-2.1in given is (Q 0 ,,1 11 ≥⎟⎠⎞

⎜⎝⎛−= rrsQrFR σσ


WITS Lab, NSYSU.100

Generalized Rice distribution

. toequal variancesidentical

and ,,...,2,1, means with variablesrandomGaussian

tindependenlly statistica are ,...,2,1, where,

2

1

2

σ

nim

niXXR

i

i

n

ii

=

== ∑=

( ) ( )( ) 0 ,

:118-2.1Equation by given is PDF Its 119.-2.1Equation by given parameter

centrality-non and freedom of degreesn on with distributisquare-chi central-non a has variablerandom The

212/2/

2/22

2/

2

2

222

≥⎟⎠⎞

⎜⎝⎛=

=

−+−

− rrsIesrrp

s

YR

nsr

n

n

R σσσ


WITS Lab, NSYSU.101

Generalized Rice distributionThe CDF is:

In the special case when m=n/2 is an integer, we have:

The k-th moment of R is:

( ) ( ) ( ) ( ) ( )( ) 121.-2.1Equation by given is where

2

22

rF

rFrYPryPrRPrF

Y

YR =≤=≤=≤=

( ) 0 ,,1 ≥⎟⎠⎞

⎜⎝⎛−= rrsQrF mR σσ

( ) ( )( )

0 ;2

;2

,2

21

21

2 2

2

112/2/2 22

≥⎟⎟⎠

⎞⎜⎜⎝

⎛ +

⎟⎠⎞

⎜⎝⎛Γ

⎥⎦⎤

⎢⎣⎡ +Γ

= − ksnknFn

kneRE skk

σσ σ


WITS Lab, NSYSU.102

Nakagami m-distributionFrequently used to characterize the statistics of signals transmitted through multi-path fading channels.PDF is given by Nakagami (1960) as:

( ) ( )( )

( )

figure. fading thecalled moments, of ratio theas defined is mparameter The

21 ,

][

2

22

2

2

/12 2

≥Ω−

Ω=

=Ω

⎟⎠⎞

⎜⎝⎛ΩΓ

= Ω−−

mRE

m

RE

ermm

rp mrmm

R


WITS Lab, NSYSU.103

Nakagami m-distributionThe n-th moment of R is:

By setting m=1, the PDF reduces to a Rayleigh PDF.

( ) ( )( )

2/2/ nn

mmnmRE ⎟

⎠⎞

⎜⎝⎛ Ω

Γ+Γ

=

PDF for the Nakagami m-distribution.Shown with Ω=1.


WITS Lab, NSYSU.104

Lognormal distribution:

The PDF of R is given by:

The lognormal distribution is suitable for modeling the effect of shadowing of the signal due to large obstructions, such as tall buildings, in mobile radio communications.

. varianceandmean with ddistributenormally is where,lnLet

2σ

mXRX =

( )( ) ( )

( )

2 2ln / 21 02

0 0

r me rp r r

r

σ

πσ− −⎧ ≥⎪= ⎨

⎪ <⎩


WITS Lab, NSYSU.105

Multivariate Gaussian distributionAssume that Xi, i=1,2,…,n, are Gaussian random variables with means mi, i=1,2,…,n; variances σi

2, i=1,2,…,n; and covariances μij, i,j=1,2,…,n. The joint PDF of the Gaussian random variables Xi, i=1,2,…,n, is defined as

M denotes the n × n covariance matrix with elements μij;x denotes the n × 1 column vector of the random variables;mx denote the n × 1 column vector of mean values mi, i=1,2,…,n.M-1 denotes the inverse of M.x’ denotes the transpose of x.

( )( ) ( )

( ) ( )⎥⎦⎤

⎢⎣⎡ −′−−= −

xxnnxxxp mxMmxM

1

21exp

det21,...,, 2/12/21 π


WITS Lab, NSYSU.106

Multivariate Gaussian distribution (cont.)Given v the n-dimensional vector with elements υi, i=1,2,…,n, the characteristic function corresponding to the n-dimentional joint PDF is:

( ) ( ) ⎟⎠⎞

⎜⎝⎛ ′−′== ′ Mvvvmv xv

21exp X

j jeEjψ


WITS Lab, NSYSU.107

Bi-variate or two-dimensional GaussianThe bivariate Gaussian PDF is given by:

( )

( ) ( )( ) ( )( )( )( )

1 2 21 2

2 22 22 1 1 1 2 1 1 2 2 1 2 2

2 2 21 2

21 1 12

12 1 1 2 222 12 2

21 1 2

21 2 2

1,2 1

2 exp

2 1

, ,

, , 0 1

X

ijij ij

i j

p x x

x m x m x m x m

mE x m x m

m

i j

πσ σ ρ

σ ρσ σ σπσ σ ρ

σ µµ

µ σµ

ρ ρσ σσ ρσ σ

ρσ σ σ

=−

⎡ ⎤− − − − + −⎢ ⎥× −

−⎢ ⎥⎣ ⎦⎡ ⎤⎡ ⎤

= = = − −⎡ ⎤⎢ ⎥⎢ ⎥ ⎣ ⎦⎣ ⎦ ⎣ ⎦

= ≠ ≤ ≤

⎡ ⎤= ⎢⎣ ⎦

m M

M( )

21 2 1 2

22 2 21 2 11 2

1, 1

σ ρσ σρσ σ σσ σ ρ

− ⎡ ⎤−=⎥ ⎢ ⎥−− ⎣ ⎦

M


WITS Lab, NSYSU.108

Bi-variate or two-dimensional Gaussianρ is a measure of the correlation between X1 and X2.When ρ=0, the joint PDF p(x1,x2) factors into the product p(x1)p(x2), where p(xi), i=1,2, are the marginal PDFs.When the Gaussian random variables X1 and X2 are uncorrelated, they are also statistically independent. This property does not hold in general for other distributions.This property can be extended to n-dimensional Gaussian random variables: if ρij=0 for i≠j, then the random variables Xi, i=1,2,…,n, are uncorrelated and, hence, statistically independent.


WITS Lab, NSYSU.109

Linear transformation of n Gaussian random variablesLet Y=AX , X=A-1Y, where A is a nonsingular matrix.The Jacobian of this transformation is J=1/det A.The joint PDF of Y is:

The linear transformation of a set of jointly Gaussian random variables results in another set of jointly Gaussian random variables.

( )( ) ( )

( ) ( )

( ) ( )( ) ( )

AAMQAmm

myQmyQ

myAMmyAAM

y

1

1

′==

⎥⎦⎤

⎢⎣⎡ −′−−=

⎥⎦⎤

⎢⎣⎡ −

′−−=

−

−−−

and where21exp

det21

21exp

det det21

y

1/22/

112/12/

x

yyn

xxnp

π

π


WITS Lab, NSYSU.110

Performing a linear transformation that results in nstatistically independent Gaussian random variables

Gaussian random variables are statistically independent if they are pairwise-uncorrelated, i.e., if the covariance matrix Q is diagonal: Q=AMA’=D, where D is a diagonal matrix.Since M is the covariance matrix, one solution is to select A to be an orthogonal matrix (A’=A-1) consisting of columns that are the eigenvectors of the covariance matrix M. Then, D is a diagonal matrix with diagonal elements equal to the eigenvalues of M.To find the eigenvalues of M, solve the characteristic equation: det(M-λI)=0.


WITS Lab, NSYSU.111

2.1.5 Upper Bounds on the Tail Probability

Chebyshev inequalitySuppose X is an arbitrary random variable with finite mean mxand finite variance σx

2. For any positive number δ:

Proof:

( ) 2

2

δσδ x

xmXP ≤≥−

2 2 2

| |

2 2

| |

( ) ( ) ( ) ( )

( ) (| | ) x

x

x x xx m

xx m

x m p x dx x m p x dx

p x dx P X m

δ

δ

σ

δ δ δ

∞

−∞ − ≥

− ≥

= − ≥ −

≥ = − ≥

∫ ∫∫

WITS Lab, NSYSU.112

Chebyshev inequalityIt is apparent that the Chebyshev inequality is simply an upper bound on the area under the tails of the PDF p(y), where Y=X-mx, i.e., the area of p(y) in the intervals (-∞,δ) and (δ,∞).Chebyshev inequality may be expressed as:

2

2

2

2

1 [ ( ) ( )]

or, equivalently, as

1 [ ( ) ( )]

xY Y

xX x X x

F F

F m F m

σδ δδ

σδ δδ

− − − ≤

− + − − ≤


WITS Lab, NSYSU.113

Chebyshev inequalityAnother way to view the Chebyshev bound is working with the zero mean random variable Y=X-mx.Define a function g(Y) as:

Upper-bound g(Y) by the quadratic (Y/δ)2, i.e.

The tail probability

( ) ( )( ) ( )[ ] ( )δ

δδ

≥=⎩⎨⎧

<≥

= YPYgEYg and Y 0Y 1

( )2

⎟⎠⎞

⎜⎝⎛≤δYYg

( )[ ] ( )2

2

2

2

2

2

2

2

δσ

δσ

δδxyYEYEYgE ===⎟⎟

⎠

⎞⎜⎜⎝

⎛≤


WITS Lab, NSYSU.114

Chebychev inequalityA quadratic upper bound on g(Y) used in obtaining the tail probability (Chebyshev bound)

For many practical applications, the Chebyshev bound is extremely loose.


WITS Lab, NSYSU.115

Chernoff boundThe Chebyshev bound given above involves the area under the two tails of the PDF. In some applications we are interested only in the area under one tail, either in the interval (δ, ∞) or in the interval (-∞, δ).In such a case, we can obtain an extremely tight upper bound by over-bounding the function g(Y) by an exponential having a parameter that can be optimized to yield as tight an upper bound as possible.Consider the tail probability in the interval (δ, ∞).

( ) ( ) ( ) ( ) ( )( )

1 and is defined as

0 where 0 is the parameter to be optimized.

v Y Y δg Y e g Y g Y

Y δv

δ− ≥⎧⎪≤ = ⎨ <⎪⎩≥


WITS Lab, NSYSU.116

Chernoff bound

The expected value of g(Y) is

This bound is valid for any υ ≥0.

( )[ ] ( ) ( )( )δδ −≤≥= YveEYPygE


WITS Lab, NSYSU.117

Chernoff boundThe tightest upper bound is obtained by selecting the value of that minimizes E(eυ(Y-δ)).A necessary condition for a minimum is:

( )( ) 0=−δYveEdvd

( ) ( )

( )

( )

( ) ( )

[( ) ] [ ( ) ( )] 0

v Y v Y

v Y

v v Y vY

d dE e E edv dv

E Y ee E Ye E e

δ δ

δ

δ δ

δ

δ

− −

−

− −

=

= −

= − =

( ) ( ) 0 =−→ vYvY eEYeE δ


WITS Lab, NSYSU.118

Chernoff bound

An upper bound on the lower tail probability can be obtained in a similar manner, with the result that

( ) ( )Yvv eEeYP

v

ˆˆ :isy probabilit

tailsided-one on the boundupper thesolution, thebe ˆLet

δδ −≤≥

( ) ( ) 0 ˆˆ <≤≤ − δδ δ Yvv eEeYP


WITS Lab, NSYSU.119

Chernoff boundExample 2.1-6: Consider the (Laplace) PDF p(y)=e-|y|/2.

The true tail probability is:

( ) δ

δδ −∞ − ==≥ ∫ edyeYP y

21

21


WITS Lab, NSYSU.120

Chernoff boundExample 2.1-6 (cont.)

( )( ) ( )

( ) ( )( )( ) ( )

( ) ( )( ) ⎟

⎠⎞

⎜⎝⎛≤≥>>

++−≤≥⇒

++−=

=−+=−

−+=

−+=

−

+−

bound Chebyshevfor 1 2

:1for

112

positive) bemust ˆ( 11ˆ

02obtain we,0 Since

111

112

2

11

2

2

2

2

22

2

δδδδ

δδδ

δδ

δδδ

δ

δ

eYP

eYP

vv

vveEYeE

vveE

vvvYeE

vYvY

vYvY


WITS Lab, NSYSU.121

Chernoff bound for random variable with a nonzero mean:

Let Y=X-mx, we have:

For δ<0

( ) ( ) ( ) ( )mxx XPmXPmXPYP δδδδ ≥≡+≥=≥−=≥

( ) ( )( )( ) ( )

( ) ( )ˆ ˆ

1 Define:

0

and upper-bounded as: We can obtain the result:

m

m

m

m

v X

v vXm

Xg X

X

g X eP X e E e

δ

δ

δδ

δ

−

−

≥⎧⎪= ⎨ <⎪⎩≤

≥ ≤

( ) ( ) ( ) ( )( )( ) ( )Xvv

m

Xvmxx

eEeXP

eEXPmXPmXPm

m

ˆˆ δ

δ

δ

δδδ−

−

≤≤⇒

≤≤≡+≤=≤−


WITS Lab, NSYSU.122

2.1.6 Sums of Random Variables and the Central Limit Theorem

Sum of random variablesSuppose that Xi, i=1,2,…,n, are statistically independent and identically distributed random variables, each having a finite mean mx and a finite variance σx

2. Let Y be defined as the normalized sum, called the sample mean:

The mean of Y is

∑=

=n

iiX

nY

1

1

( ) ( )

x

n

iiy

m

XEn

mYE

=

== ∑=

11

WITS Lab, NSYSU.123

Sum of random variablesThe variance of Y is:

An estimate of a parameter (in this case the mean mx) that satisfies the conditions that its expected value converges to the true value of the parameter and the variance converges to zero as n→∞ is said to be a consistent estimate.

( ) ( ) ( )

( ) ( ) ( )

( )[ ] ( )nσmmnn

nmσn

n

mXEXEn

XEn

mXXEn

mYEmYEσ

xxxxx

x

n

i

n

jij

ji

n

ii

n

i

n

jxjixyy

222

222

2

2

1 12

1

22

1 1

22

22222

111

11

1

=−−++=

−+=

−=−=−=

∑∑∑

∑∑

=≠==

= =


WITS Lab, NSYSU.124

Weak law of large numbersThe tail probability of Y can be upper-bounded.The Chebyshev inequality applied to Y is:

In the limit as n→∞, the above equation becomes:

The probability that the estimate of the mean differs from the true mean mx by more than δ(δ>0) approaches zero as n approaches infinity. The upper bound converges to zero relatively slowly, i.e., inversely with n.

( ) 2

2

12

2 1 δσδ

δσ

δn

mXn

PmYP xx

n

ii

yy ≤⎟⎟

⎠

⎞⎜⎜⎝

⎛≥−⇒≤≥− ∑

=

01lim1

=⎟⎟⎠

⎞⎜⎜⎝

⎛≥−∑

=∞→

δx

n

iin

mXn

P


WITS Lab, NSYSU.125

The Chernoff bound

( )

( ) ( )[ ]( ). theof oneany is :Note

expexp

.0 and where

exp

1

1

11

11

1

i

nvXvn

i

vXvn

n

ii

vnm

n

ii

xm

m

n

iim

n

ii

x

n

iiy

XX

eEeeEe

XvEenXvE

m

nXvEnXP

mXn

PmYP

mim

m

δδ

δδ

δδδ

δδ

δδ

−

=

−

=

−

=

==

=

==

⎥⎦

⎤⎢⎣

⎡⎟⎠

⎞⎜⎝

⎛=

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

⎥⎦

⎤⎢⎣

⎡⎟⎠

⎞⎜⎝

⎛−

>+=

⎪⎭

⎪⎬⎫

⎪⎩

⎪⎨⎧

⎥⎦

⎤⎢⎣

⎡⎟⎠

⎞⎜⎝

⎛−≤⎟

⎠

⎞⎜⎝

⎛≥=

⎟⎠

⎞⎜⎝

⎛≥−=≥−

∏

∑∑

∑∑

∑


(*)

WITS Lab, NSYSU.126

The Chernoff bound (cont.)The parameter υ that yields the tightest upper bound is obtained by differentiating Equation (*) and setting the derivative equal to zero. This yields the equation:

The bound on the upper tail probability is

In a similar manner, the lower tail probability is upper-bounded as:

( ) ( ) 0=− vXm

vX eEXeE δ

( )[ ] xmnXvv

m

n

ii meEeX

nP m >≤⎟

⎠

⎞⎜⎝

⎛≥ −

=∑ δδ δ ,1 ˆˆ

1

( ) ( )[ ] xmnXvv

m mδeEeYP m <≤≤ − ,ˆˆδδ


WITS Lab, NSYSU.127

Central limit theoremConsider the case in which the random variables Xi, i=1,2,…,n, being summed are statistically independent and identically distributed, each having a finite mean mx and a finite variance σx

2.Define a normalized random variable Ui with a zero mean and unit variance.

Let

We wish to determine the CDF of Y in the limit as n→∞.

nimXUx

xii ,...,2,1 , =

−=

σ

nce.unit varia andmean zero has that Note .11

YUn

Yn

ii∑

=

=


WITS Lab, NSYSU.128

Central limit theoremThe characteristic function of Y is:

Where U denotes any of the Ui, which are identically distributed.

( ) ( ) 1

1

exp

i

n

ijvY i

Y

n

Ui

n

U

jv Ujv E e E

n

jvψn

jvn

ψ

ψ

=

=

⎡ ⎤⎛ ⎞⎢ ⎥⎜ ⎟⎢ ⎥⎜ ⎟= =⎢ ⎥⎜ ⎟

⎜ ⎟⎢ ⎥⎝ ⎠⎣ ⎦⎛ ⎞= ⎜ ⎟⎝ ⎠

⎡ ⎤⎛ ⎞= ⎢ ⎥⎜ ⎟⎝ ⎠⎣ ⎦

∑

∏


WITS Lab, NSYSU.129

Central limit theoremExpand the characteristic function of U in a Taylor series:

( ) ( ) ( )( )

( )

( ) ( )

( )

( )

322 3

3

2

2

1 ...2! 3!

Since 0,and 1, the above equation simplifies to:

1 1 ,2

where denotes the remainder. We note that

U

U

jvv v vj j E U E U E Unn n n

E U E U

v vj R v nn nn

R v,n /n R v

ψ

ψ

⎛ ⎞ = + − + −⎜ ⎟⎝ ⎠

= =

⎛ ⎞ = − +⎜ ⎟⎝ ⎠

( )

( ) ( ) ( ) ( )2 2

approaches zero as .

, ,1 ln ln 1

2 2

n

Y Y

,nn

R v n R v nv vjv jv nn n n n

ψ ψ

→∞

⎡ ⎤ ⎡ ⎤= − + ⇒ = − +⎢ ⎥ ⎢ ⎥⎣ ⎦ ⎣ ⎦


WITS Lab, NSYSU.130

Central limit theorem

The above equation is the characteristic function of a Gaussian random variable with zero mean and unit variance.We can conclude that the sum of statistically independent and identically distributed random variables with finite mean and variance approaches a Gaussian CDF as n→∞.

( )

( ) ( ) ( )

( ) ( ) 2/Yn

2

222

32

2

lim 21lnlim

...,22

1,2

ln

...31

211ln , of valuesmallFor

vYn

Y

ejvvjv

nnvR

nv

nnvR

nvnjv

xxxxx

−

∞→∞→=⇒−=

⎥⎥⎦

⎤

⎢⎢⎣

⎡+⎟⎟

⎠

⎞⎜⎜⎝

⎛+−−+−=

−+−=+

ψψ

ψ


WITS Lab, NSYSU.131

Central limit theoremAlthough we assumed that the random variables in the sum are identically distributed, the assumption can be relaxed provided that additional restrictions are imposed on the properties of the random variables.If the random variables Xi are i.i.d., sum of n = 30 random variables is adequate for most applications.If the functions fi(x) are smooth, values of n as low as 5 can be used.For more information, referred to Cramer (1946) and Papoulis “Probability, Random Variables, and Stochastic Processes”, pp. 214-221, 3rd Edition.


WITS Lab, NSYSU.132

2.2 Stochastic Processes

Many of random phenomena that occur in nature are functions of time.In digital communications, we encounter stochastic processes in:

The characterization and modeling of signals generated by information sources;The characterization of communication channels used to transmit the information;The characterization of noise generated in a receiver;The design of the optimum receiver for processing the received random signal.

WITS Lab, NSYSU.133


IntroductionAt any given time instant, the value of a stochastic process is a random variable indexed by the parameter t. We denote such a process by X(t).In general, the parameter t is continuous, whereas X may be either continuous or discrete, depending on the characteristics of the source that generates the stochastic process.The noise voltage generated by a single resistor or a single information source represents a single realization of the stochastic process. It is called a sample function.

WITS Lab, NSYSU.134

2.2 Stochastic ProcessesIntroduction (cont.)

The set of all possible sample functions constitutes an ensemble of sample functions or, equivalently, the stochastic process X(t).In general, the number of sample functions in the ensemble is assumed to be extremely large; often it is infinite.Having defined a stochastic process X(t) as an ensemble of sample functions, we may consider the values of the process at any set of time instants t1>t2>t3>…>tn, where n is any positive integer.

( )( ).,...,, PDFjoint by their lly statistica zedcharacteri

are ,,...,2,1, variablesrandom thegeneral,In

21 n

i

ttt

it

xxxp

nitXX =≡

WITS Lab, NSYSU.135


Stationary stochastic processes

When the joint PDFs are different, the stochastic process is non-stationary.

( )

( )1 2

Consider another set of random variables ,

1, 2,..., , where is an arbitrary time shift. These random

variables are characterized by the joint PDF , ,..., .

i

n

t t i

t t t t t t

n X X t t

i n t

p x x x

+

+ + +

≡ +

=

( ) ( ).sensestrict in the stationary be tosaid isit , all and allfor

,...,,,...,, wheni.e., identical, arey When theidentical. benot may or may

,21 and variablesrandom theof PDFsjont The

2121

nt

xxxpxxxp

,...,n,,iXX

ttttttttt

ttt

nn

ii

+++

+

=

=

WITS Lab, NSYSU.136

2.2.1 Stochastic ProcessesAverages for a stochastic process are called ensemble averages.

( ) ( )∫∞

∞−=

iiii

i

ttnt

nt

t

dxxpxXE

Xn

:as defined is variablerandom theof momentth The

.on depends of PDF theif instant timeon the depend willmomentth theof value thegeneral,In

iti tXtn

i ( ) ( )

time.oft independen is momentth thee,consequenca as and, time,oft independen is PDF theTherefore,

. allfor ,stationary is process When the

n

txpxpii ttt =+

WITS Lab, NSYSU.137

2.2.1 Stochastic ProcessesTwo random variables:

The correlation is measured by the joint moment:

Since this joint moment depends on the time instants t1 and t2, it is denoted by φ(t1 ,t2).φ(t1 ,t2) is called the autocorrelation function of the stochastic process.For a stationary stochastic process, the joint moment is:

Average power in the process X(t): φ(0)=E(Xt2).

( ) , 1, 2.it iX X t i≡ =

1 2 1 2 1 2( ) ( , ) ( ) ( )t tE X X t t t tφ φ φ τ= = − =

( ) ( )1 2 1 2 1 2 1 2,t t t t t t t tE X X x x p x x dx dx

∞ ∞

−∞ −∞= ∫ ∫

' '1 1 1 1 1 1

( ) ( ) ( ) ( ) ( )t t t t t tE X X E X X E X Xτ τ τ

φ τ φ τ+ + −− = = = =

WITS Lab, NSYSU.138

2.2.1 Stochastic Processes

Wide-sense stationary (WSS)A wide-sense stationary process has the property that the mean value of the process is independent of time (a constant) and where the autocorrelation function satisfies the condition that φ(t1,t2)=φ(t1-t2).Wide-sense stationarity is a less stringent condition than strict-sense stationarity.If not otherwise specified, any subsequent discussion in which correlation functions are involved, the less stringent condition (wide-sense stationarity) is implied.

WITS Lab, NSYSU.139

2.2.1 Stochastic Processes

Auto-covariance functionThe auto-covariance function of a stochastic process is defined as:

When the process is stationary, the auto-covariance function simplifies to:

For a Gaussian random process, higher-order moments can be expressed in terms of first and second moments. Consequently, a Gaussian random process is completely characterized by its first two moments.

[ ][ ] )()(),(

)()(),(2121

2121 21

tmtmtttmXtmXEtt tt

−=−−=

φµ

22121 )()()(),( mtttt −==−= τφτµµµ

WITS Lab, NSYSU.140

2.2.1 Stochastic ProcessesAverages for a Gaussian process

Suppose that X(t) is a Gaussian random process. At time instants t=ti, i=1,2,…,, the random variables Xti, i=1,2,…,n, are jointly Gaussian with mean values m(ti), i=1,2,…,n, and auto-covariances:

If we denote the n × n covariance matrix with elements μ(ti,tj) by M and the vector of mean values by mx, the joint PDF of the random variables Xti, i=1,2,…,n, is given by:

If the Gaussian process is wide-sense stationary, it is also strict-sense stationary.

( )( )[ ] njitmXtmXEtt jtitji ii,......,2,1, , )()(),( =−−=µ

( )( ) ( )

( ) ( )⎥⎦⎤

⎢⎣⎡ −′−−= −

xxnnxxxp mxMmxM

1

21exp

det21,...,, 2/12/21 π

WITS Lab, NSYSU.141

2.2.1 Stochastic ProcessesAverages for joint stochastic processes

Let X(t) and Y(t) denote two stochastic processes and let Xti≣X(ti), i=1,2,…,n, Yt’j

≣Y(t’j), j=1,2,…,m, represent the random variables at times t1>t2>t3>…>tn, and t’1>t’2>t’3>…>t’m , respectively. The two processes are characterized statistically by their joint PDF:

The cross-correlation function of X(t) and Y(t), denoted by φxy(t1,t2), is defined as the joint moment:

The cross-covariance is:

( )''2

'1,21

,...,, ,...,,mn tttttt yyyxxxp

∫ ∫∞

∞−

∞

∞−==

21212121),()(),( 21 ttttttttxy dydxyxpyxYXEtt φ

)()(),(),( 212121 tmtmtttt yxxyxy −= φµ

WITS Lab, NSYSU.142

2.2.1 Stochastic ProcessesAverages for joint stochastic processes

When the process are jointly and individually stationary, we have φxy(t1,t2)=φxy(t1-t2), and μxy(t1,t2)= μxy(t1-t2):

The stochastic processes X(t) and Y(t) are said to be statistically independent if and only if :

for all choices of ti and t’i and for all positive integers n and m.The processes are said to be uncorrelated if

' ' ' '1 1 1 1 1 1( ) ( ) ( ) ( ) ( )xy t t yxt t t t

E X Y E X Y E Y Xτ τ τφ τ φ τ+ − −

− = = = =

),...,,(),...,,(),...,,,,...,,( ''2

'121''

2'121 mnmn tttttttttttt yyypxxxpyyyxxxp =

⇒= )()(),(2121 ttxy YEXEttφ 0),( 21 =ttxyµ

WITS Lab, NSYSU.143

2.2.1 Stochastic ProcessesComplex-valued stochastic process

A complex-valued stochastic process Z(t) is defined as:

where X(t) and Y(t) are stochastic processes.The joint PDF of the random variables Zti≣Z(ti), i=1,2,…,n, is given by the joint PDF of the components (Xti, Yti), i=1,2,…,n. Thus, the PDF that characterizes Zti, i=1,2,…,n, is:

The autocorrelation function is defined as:

)()()( tjYtXtZ +=

),...,, , ,...,,(2121 nn tttttt yyyxxxp

( )( )

1 2 1 1 2 21 2

1 2 1 2 1 2 1 2

1 1( , ) ( )2 21 ( , ) ( , ) ( , ) ( , ) 2

zz t t t t t t

xx yy yx xy

t t E Z Z E X jY X jY

t t t t j t t t t

φ

φ φ φ φ

∗ ⎡ ⎤≡ = + −⎣ ⎦

⎡ ⎤= + + −⎣ ⎦

(**)

WITS Lab, NSYSU.144

2.2.1 Stochastic ProcessesAverages for joint stochastic processes:

When the processes X(t) and Y(t) are jointly and individually stationary, the autocorrelation function of Z(t) becomes:

φZZ(τ)= φ*ZZ(-τ) because from (**):

)()(),( 2121 τφφφ zzzzzz tttt =−=

' ' ' '1 1 1 1 1 1

1 1 1( ) ( ) ( ) ( ) ( )2 2 2zz t t zzt t t t

E Z Z E Z Z E Z Zτ τ τφ τ φ τ∗ ∗ ∗ ∗

− + += = = = −

WITS Lab, NSYSU.145

2.2.1 Stochastic ProcessesAverages for joint stochastic processes:

Suppose that Z(t)=X(t)+jY(t) and W(t)=U(t)+jV(t) are two complex-valued stochastic processes. The cross-correlation functions of Z(t) and W(t) is defined as:

When X(t), Y(t),U(t) and V(t) are pairwise-stationary, the cross-correlation function become functions of the time difference.

( )( )

1 2

1 1 2 2

1 2

1 2 1 2 1 2 1 2

1( , ) ( )2

1 21 ( , ) ( , ) ( , ) ( , ) 2

zw t t

t t t t

xu yu yu xu

t t E Z W

E X jY U jV

t t t t j t t t t

φ

φ φ φ φ

∗≡ ⋅

⎡ ⎤= + −⎣ ⎦

⎡ ⎤= + + −⎣ ⎦

' ' ' '1 1 1 1

1 1 1( ) ( ) ( ) ( ) ( )2 2 2i i

zw t t wzt t t tE Z W E Z W E W Zτ τ τ

φ τ φ τ∗ ∗ ∗ ∗− + +

= = = = −

WITS Lab, NSYSU.146

2.2.2 Power Density SpectrumA signal can be classified as having either a finite (nonzero) average power (infinite energy) or finite energy.The frequency content of a finite energy signal is obtained as the Fourier transform of the corresponding time function.If the signal is periodic, its energy is infinite and, consequently, its Fourier transform does not exist. The mechanism for dealing with periodic signals is to represent them in a Fourier series.

WITS Lab, NSYSU.147

2.2.2 Power Density SpectrumA stationary stochastic process is an infinite energy signal, and, hence, its Fourier transform does not exist.The spectral characteristic of a stochastic signal is obtained by computing the Fourier transform of the autocorrelation function.The distribution of power with frequency is given by the function:

The inverse Fourier transform relationship is:

( ) ( )∫∞

∞−

−=Φ ττφ τπ def fj2

( ) ( )∫∞

∞−Φ= dfef fj τπτφ 2

WITS Lab, NSYSU.148

2.2.2 Power Density Spectrum

Since φ(0) represents the average power of the stochastic signal, which is the area under Φ(f), Φ(f) is the distribution of power as a function of frequency.Φ(f) is called the power density spectrum of the stochastic process.If the stochastic process is real, φ(τ) is real and even, and, hence Φ(f) is real and even.If the stochastic process is complex, φ(τ)=φ*(-τ) and Φ(f) is real because:

( ) ( ) ( ) 00 2 ≥=Φ= ∫∞

∞− tXEdffφ

( ) ( ) ( )

( ) ( )

* * 2 * 2 '

2

' '

j f j f

j f

f e d e d

e d f

π τ π τ

π τ

φ τ τ φ τ τ

φ τ τ

∞ ∞ −

−∞ −∞

∞ −

−∞

Φ = = −

= = Φ

∫ ∫∫

(2.2-13)

WITS Lab, NSYSU.149

2.2.2 Power Density SpectrumCross-power density spectrum

For two jointly stationary stochastic processes X(t) and Y(t), which have a cross-correlation function φxy(τ), the Fourier transform is:

Φxy(f) is called the cross-power density spectrum.

If X(t) and Y(t) are real stochastic processes

( ) ( )∫∞

∞−

−=Φ ττφ τπ def fjxyxy

2

( ) ( ) ( )

( ) ( )fde

dedef

yxfj

yx

fjxy

fjxyxy

Φ==

−==Φ

∫∫∫

∞

∞−

−

∞

∞−

−∞

∞−

ττφ

ττφττφ

τπ

τπτπ

2

2*2**

( ) ( ) ( )* 2j fxy xy xyf e d fπ τφ τ τ

∞

−∞Φ = = Φ −∫ ( ) ( )yx xyf fΦ = Φ −→

(2.2-15)

WITS Lab, NSYSU.150

2.2.3 Response of a Linear Time-Invariant System to a Random Input Signal

Consider a linear time-invariant system (filter) that is characterized by its impulse response h(t) or equivalently, by its frequency response H(f), where h(t) and H(f) are a Fourier transform pair. Let x(t) be the input signal to the system and let y(t) denote the output signal.

Suppose that x(t) is a sample function of a stationary stochastic process X(t). Since convolution is a linear operation performed on the input signal x(t), the expected value of the integral is equal to the integral of the expected value.

The mean value of the output process is a constant.

( ) ( ) ( )y t h x t dτ τ τ∞

−∞= −∫

( ) ( ) ( )

( ) ( ) 0

y

x x

m E Y t h E X t d

m h d m H

τ τ τ

τ τ

∞

−∞

∞

−∞

= = −⎡ ⎤ ⎡ ⎤⎣ ⎦ ⎣ ⎦

= =

∫∫

WITS Lab, NSYSU.151


The autocorrelation function of the output is:

If the input process is stationary, the output is also stationary:

( ) ( )

( ) ( ) ( ) ( )[ ]( ) ( ) ( )∫ ∫

∫ ∫∞

∞−

∞

∞−

∞

∞−

∞

∞−

−+−=

−−=

=

βαβαφαβ

βααβαβ

φ

ddtthh

ddtXtXEhh

YYEtt

xx

ttyy

21*

2*

1*

*21

21

21,

21

( ) ( ) ( ) ( )*yy xxh h d dφ τ α β φ τ α β α β

∞ ∞

−∞ −∞= + −∫ ∫

WITS Lab, NSYSU.152


The power density spectrum of the output process is:

(by making τ0=τ+α-β)The power density spectrum of the output signal is the product of the power density spectrum of the input multiplied by the magnitude squared of the frequency response of the system.

( ) ( ) 2j fyy yyf e dπ τφ τ τ

∞ −

−∞Φ = ∫

( ) ( ) ( )* 2j fxxh h e d d dπ τα β φ τ α β τ α β

∞ ∞ ∞ −

−∞ −∞ −∞= + −∫ ∫ ∫

( ) ( ) 2xx f H f= Φ

WITS Lab, NSYSU.153

When the autocorrelation function φyy(τ) is desired, it is usually easier to determine the power density spectrum Φ yy(f) and then to compute the inverse transform.

The average power in the output signal is:

Since φyy(0)=E(|Yt|2) , we have:

( ) ( ) 2j fyy yy f e dfπ τφ τ

∞

−∞= Φ∫

( ) ( ) 2 2j fxx f H f e dfπ τ∞

−∞= Φ∫

( ) ( ) ( ) 20yy xx f H f dfφ

∞

−∞= Φ∫

( ) ( )∫∞

∞−≥Φ 02 dffHfxx


valid for any H(f ).

WITS Lab, NSYSU.154

Suppose we let |H(f )|2=1 for any arbitrarily small interval f1≤ f ≤f2 , and H( f )=0 outside this interval. Then, we have:

This is possible if an only if Φxx(f )≥0 for all f.

Conclusion: Φxx(f )≥0 for all f.

( )∫ ≥Φ2

1

0f

f xx dff


WITS Lab, NSYSU.155

Example 2.2-1Suppose that the low-pass filter is excited by a stochastic process x(t) having a power density spectrum Φxx(f)=N0/2 for all f. A stochastic process having a flat power density spectrum is called white noise. Let us determine the power density spectrum of the output process.The transfer function of the low-pass filter is:


( )RfLjfLjR

RfH/21

12 ππ +

=+

=

( )( )

2

2 2

11 2 /

H fL R fπ

=+

⇒

WITS Lab, NSYSU.156

Example 2.2-1 (cont.)The power density spectrum of the output process is:

The inverse Fourier transform yields the autocorrelation function:

( )( )

202 2

12 1 2 /yy

j fN e dfL R f

π τφ τπ

∞

−∞=

+∫


( )( )

02 2

12 1 2 /yy

NfL R fπ

Φ =+

( )/0

4R LRN e

Lτ−=

WITS Lab, NSYSU.157

Cross-correlation function between y(t) and x(t)

With t1-t2=τ, we have:

In the frequency domain, we have:

If the input process is white noise, the cross correlation of the input with the output of the system yields the impulse response h(t) to within a scale factor.

( ) ( ) ( ) ( ) ( )[ ]( ) ( ) ( )

( ) ( ) .stationaryjointly are and processes stochastic The

21

21,

2121

2*

1*

21 21

tYtX

ttdtth

dtXtXEhXYEtt

yxxx

ttyx

−=−−=

−==

∫

∫∞

∞−

∞

∞−

φααφα

αααφ


( ) ( ) ( )∫∞

∞−−= αατφατφ dh xxyx

( ) ( ) ( )fHff xxyx Φ=Φ

Function of t1-t2

WITS Lab, NSYSU.158

2.2.4 Sampling Theorem for Band-Limited Stochastic Processes

A deterministic signal s(t) that has a Fourier transform S(f) is called band-limited if S(f)=0 for |f|>W, where W is the highest frequency contained in s(t).A band-limited signal is uniquely represented by samples of s(t) taken at a rate of fs≥2W samples/s.The minimum rate fN=2W samples/s is called the Nyquist rate.Sampling below the Nyquist rate results in frequency aliasing.

WITS Lab, NSYSU.159


The band-limited signal sampled at the Nyquist rate can be reconstructed from its samples by use of the interpolation formula:

where s(n/2W) are the samples of s(t) taken at t=n/2W, n=0,±1, ±2,….Equivalently, s(t) can be reconstructed by passing the sampled signal through an ideal low-pass filter with impulse response h(t)=(sin2πWt)/2πWt.

( )⎟⎠⎞

⎜⎝⎛ −

⎥⎦

⎤⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −

⎟⎠⎞

⎜⎝⎛= ∑

∞

−∞=

WntW

WntW

Wnsts

n

22

22sin

2 π

π

WITS Lab, NSYSU.160


The signal reconstruction process based on ideal interpolation can be illustrated in the following figure:

WITS Lab, NSYSU.161


A stationary stochastic process X(t) is said to be band-limited if its power density spectrum Φ(f)=0 for |f|>W.Since Φ(f) is the Fourier transform of the autocorrelation function φ(τ), it follows that φ(τ) can be represented as:

where φ(n/2W) are the samples of φ(τ) taken at τ=n/2W, n=0,±1, ±2,….

( )⎟⎠⎞

⎜⎝⎛ −

⎥⎦

⎤⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −

⎟⎠⎞

⎜⎝⎛= ∑

∞

−∞=

WnW

WnW

Wn

n

22

22sin

2 τπ

τπφτφ

WITS Lab, NSYSU.162


If X(t) is a band-limited stationary stochastic process, then X(t) can be represented as:

where X(n/2W) are the samples of X(t) taken at t=n/2W, n=0,±1, ±2,…. This is the sampling representation for a stationary stochastic process.

( )⎟⎠⎞

⎜⎝⎛ −

⎥⎦

⎤⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −

⎟⎠⎞

⎜⎝⎛= ∑

∞

−∞=

WntW

WntW

WnXtX

n

22

22sin

2 π

π(A)

WITS Lab, NSYSU.163


The samples are random variables that are described statistically by appropriate joint probability density functions.The signal representation in (A) is easily established by showing that:

Equality between the sampling representation and the stochastic process X(t) holds in the sense that the mean square error is zero.

( ) 0

22

22sin

2

2

=

⎪⎪

⎭

⎪⎪

⎬

⎫

⎪⎪

⎩

⎪⎪

⎨

⎧

⎟⎠⎞

⎜⎝⎛ −

⎥⎦

⎤⎢⎣

⎡⎟⎠⎞

⎜⎝⎛ −

⎟⎠⎞

⎜⎝⎛− ∑

∞

−∞=n

WntW

WntW

WnXtXE

π

π

WITS Lab, NSYSU.164

2.2.5 Discrete-Time Stochastic Signals and Systems

Discrete-time stochastic process X(n) consisting of an ensemble of sample sequences x(n) are usually obtained by uniformly sampling a continuous-time stochastic process.The mth moment of X(n) is defined as:

The autocorrelation sequence is:

The auto-covariance sequences is:

[ ] ( )∫∞

∞−= nn

mn

mn dXXpXXE

( ) ( ) ( )∫ ∫∞

∞−

∞

∞−== knknknkn dXdXXXpXXXXEkn ,

21

21, **φ

( ) ( ) ( ) ( )*

21,, kn XEXEknkn −= φµ

WITS Lab, NSYSU.165


For a stationary process, we have φ(n,k)≣φ(n-k), μ(n,k)≣μ(n-k), and

where mx=E(Xn) is the mean value.A discrete-time stationary process has infinite energy but a finite average power, which is given as:

The power density spectrum for the discrete-time process is obtained by computing the Fourier transform of φ(n).

( ) ( )02 φ=nXE

( ) ( ) 2

21

xmknkn −−=− φµ

( ) ( )∑∞

−∞=

−=Φn

fnjenf πφ 2

WITS Lab, NSYSU.166


The inverse transform relationship is:

The power density spectrum Φ(f) is periodic with a period fp=1. In other words, Φ(f+k)=Φ(f) for k=0,±1,±2,….The periodic property is a characteristic of the Fourier transform of any discrete-time sequence.

( ) ( )∫− Φ=21

21

2 dfefn fnj πφ

WITS Lab, NSYSU.167


Response of a discrete-time, linear time-invariant system to a stationary stochastic input signal.

The system is characterized in the time domain by its unit sample response h(n) and in the frequency domain by the frequency response H(f).

The response of the system to the stationary stochastic input signal X(n) is given by the convolution sum:

( ) ( )∑∞

−∞=

−=n

fnjenhfH π2

( ) ( ) ( )∑∞

−∞=

−=k

knxkhny

WITS Lab, NSYSU.168


Response of a discrete-time, linear time-invariant system to a stationary stochastic input signal.

The mean value of the output of the system is:

where H(0) is the zero frequency [direct current (DC)] gain of the system.

( ) ( ) ( )

( ) ( ) 0

yk

x xk

m E y n h k E x n k

m h k m H

∞

=−∞

∞

=−∞

= = −⎡ ⎤ ⎡ ⎤⎣ ⎦ ⎣ ⎦

= =

∑

∑

WITS Lab, NSYSU.169


The autocorrelation sequence for the output process is:

By taking the Fourier transform of φyy(k) and substituting the relation in Equation 2.2-49, we obtain the corresponding frequency domain relationship:

Φyy(f), Φxx(f), and H(f) are periodic functions of frequency with period fp=1.

( ) ( ) ( )

( ) ( ) ( ) ( )

( ) ( ) ( )

12

12

yy

i j

xxi j

k E y n y n k

h i h j E x n i x n k j

h i h j k j i

φ

φ

∗

∞ ∞∗ ∗

=−∞ =−∞

∞ ∞∗

=−∞ =−∞

⎡ ⎤= +⎣ ⎦

⎡ ⎤= − + −⎣ ⎦

= − +

∑ ∑

∑ ∑

( ) ( ) ( ) 2yy xxf f H fΦ = Φ

WITS Lab, NSYSU.170

Cyclostationary Processes

For signals that carry digital information, we encounter stochastic processes with statistical averages that are periodic.Consider a stochastic process of the form:

where an is a discrete-time sequence of random variables with mean ma=E(an) for all n and autocorrelation sequence φaa(k)=E(a*nan+k)/2.The signal g(t) is deterministic.The sequence an represents the digital information sequence that is transmitted over the communication channel and 1/Trepresents the rate of transmission of the information symbols.

( ) ( )nn

X t a g t nT∞

=−∞

= −∑

WITS Lab, NSYSU.171

Cyclostationary ProcessesThe mean value is:

The mean is time-varying and it is periodic with period T.The autocorrelation function of X(t) is:

( ) ( ) ( )

( )

nn

an

E X t E a g t nT

m g t nT

∞

=−∞

∞

=−∞

= −⎡ ⎤⎣ ⎦

= −

∑

∑

( ) ( ) ( )

( ) ( ) ( )

( ) ( ) ( )

12

12

,

xx

n mn m

aan m

t t E X t X t

E a a g t nT g t mT

m n g t nT g t mT

φ τ τ

τ

φ τ

∗

∞ ∞∗ ∗

=−∞ =−∞

∞ ∞∗

=−∞ =−∞

⎡ ⎤+ = +⎣ ⎦

= − + −

= − − + −

∑ ∑

∑ ∑

WITS Lab, NSYSU.172

Cyclostationary ProcessesWe observe that

for k=±1,±2,…. Hence, the autocorrelation function of X(t) is also periodic with period T.Such a stochastic process is called cyclostationary or periodically stationary.Since the autocorrelation function depends on both the variables tand τ, its frequency domain representation requires the use of a two-dimensional Fourier transform.The time-average autocorrelation function over a single period is defined as:

( ) ( ), ,xx xxt kT t kT t tφ τ φ τ+ + + = +

( ) ( )2

2

1 ,T

xx xxTt t dt

Tφ τ φ τ

−= +∫

WITS Lab, NSYSU.173

Cyclostationary ProcessesThus, we eliminate the tie dependence by dealing with the average autocorrelation function.The Fourier transform of φxx(τ) yields the average power density spectrum of the cyclostationary stochastic process.This approach allows us to simply characterize cyclostationaryprocess in the frequency domain in terms of the power spectrum.The power density spectrum is:

( ) ( ) 2j fxx xxf e dπ τφ τ τ

∞ −

−∞Φ = ∫

chapter 2 probability and stochastic processes

Documents