probability & random variables

Principles of Communications I (Fall, 2007) Probability & Random Variables

NCTU EE 1

Probability & Random Variables Motivations:

Both messages (signals) & noises are random in nature

Some definitions: (a) random variable (r.v.): one random quantity (b) random sequence: sequence of random variable (c) random process: a (continuous-time) function whose value (at any time instant) is

a r.v.

Probability Space

Definition of prob. space (a) Relative Frequency --- experimental, intuitive, (b) Axiomatic Theory --- mathematical, rigorous, facilitate further derivation E.g. Tossing two fair coins

Relative Frequency Axiomatic Theory

Model:⎪⎩

⎪⎨

⎧

L

L

),(),(:Events of Prob.

B, A, :Events

BPAP

HTBHHA

≡≡

To find P(A) &P(B), we repeat experiment N times

41lim).(Pr =≡

∞→ NN

Aob A

N

21lim)or .(Pr =≡ ∪

∞→ NNBAob BA

N

Model: },,{ LHTHH=Ω Field:

},},{,},{},{{ φLL HTHHHTHH ∪=ℑ

2/1})({4/1})({

]1,0[:

=∪=

→ℑ

HTHHPHHP

P

Axioms of probability

field)- (a filed a forms and of subsets of collectionA events) elementary of collection (a space Sample

σΩ=ℑ=Ω


NCTU EE 2

Remarks: (1) Field= ),( ℑΩ , if

(a) ℑ∈φ , and ℑ∈Ω (b) For any A, B ℑ∈ , then ℑ∈∪ BA , ℑ∈∩ BA (c) For any ℑ∈A , then ℑ∈−Ω= AAC

(2) :field- σ countable infinite ∪ and ∩

If ℑ∈iA , ℑ∈∞

=U

1iiA and ℑ∈

∞

=I

1iiA

Probability space = ),,( PℑΩ

(i) ),( ℑΩ is a field- σ , and

(ii) Ρ is a probability measure

Remarks: ]1,0[: →ℑΡ is a probability measure on ℑ if

(a) For any ℑ∈A , 0)( ≥AP (b) 0)( =φP , 1)( =ΩP (c) For any ℑ∈BA, , if φ=∩ BA , )()()( BPAPBAP +=∪

(d) additivity- σ : For ℑ∈iA and if φ=∩ ji AA for all ,ji ≠

then ∑∞

=

∞

=

=11

)()(i

ii

i APAP U

e.g. ]1,0[=Ω , line segment. }],1,0[],1,2/1(],2/1,0{[ φ=ℑ . )(AP =length of the line segment A.


NCTU EE 3

Random Variable (R.V.) Def.: Given a probability space ),,( PℑΩ , a r.v. is a mapping (real-valued function):

RX →Ω: (real line) such that

(i) the set })(:{ BX ∈ζζ must be a legal event A in ℑ for any Borel set of R, B;

(ii) 0)()( =∞==−∞= XPXP .

Note: Borel set: Smallest field- σ that contains all of the open sets in R. (≈ all kinds of intervals on real line.)

Remarks:

(1) )},{,( PBR is a derived probability space (from the original probability space

),,( PℑΩ ).

If )(•X is properly defined (selected), )},{,( PBR reflects all the probabilis-tic properties of ),,( PℑΩ . But )},{,( PBR id often easier to handle, (be-cause of real line)

(2) A random variable )(•X is a function (mapping) not a simple value (3) Notations:

Capital letters ↔ random variable: X, Y, Θ…..

Lower-case letters ↔ (constant) values of random variable: L,,, θyx

E.g. Tossing a coin, define two r.v.’s Note: After all the abstraction and formality, from now on we mostly deal with )(•X ,

)(•Y directly, and do not care the underlying ),,( PℑΩ 1

1 Ref: Stock & Woods, Prob., Random Processes, and Estimation Theory for Engineering, 2nd ed., Prentice-Hall, 1994.


NCTU EE 4

Probability (Cumulative) Distribution Functions (PDF or cdf)

})(:{}{ e wher][)( xXxXxXPxFX ≤≡≤≤≡ ζζ

Properties of FX(x):

1. 1)(,0)( =∞=−∞ XX FF

2. Continuous from right: )()(lim 00

xFxF XXXX =+→

3. Nondecreasing: 2121 if )()( xxxFxF XX ≤≤

Probability Density Functions (pdf)

dxxdFxf X

X)()( =

Properties of f(x) (fX(x)):

1. 1)( =∫∞

∞−ξξ df

2. ][)()( xXPdfxFx

X ≤== ∫ ∞− ξξ

3. ][)()()( 21122

1

xXxPxFxFdf XX

x

x≤<=−=∫ ξξ


NCTU EE 5

Some common pdf’s

♦ Binomial distribution

For an n ≧ 1 and 0 < p < 1

Remark: Laplace approximation to the binomial distribution

]2

)(exp[2

1)(2

npqnpk

npqkPn

−−≅

π k integer

♦ Poisson distribution

For 0 < a

Tk

T ekTkP αα −=!)()( for k = 0, 1, 2, ….

Remark: When n is large, p is small

Kk

n ek

KkP −≅!)()( , where ][KEK =


NCTU EE 6

♦ Gaussian (normal) distribution

Joint cdf’s and pdf’s

Joint cdf: ),(),( yYxXPyxFXY ≤≤=

Joint pdf: yxyxFyxf XY

XY ∂∂∂

=),(),(

2

∫ ∫=≤<≤< 2

1

2

1

),(),( 2121

y

y

x

x XY dxdyyxfyYyxXxP

Marginal distribution:

∫∞

∞−=

∞≤=∞=∞≤=∞=

dyyxfxf

yXFyFyFYxFxFxF

XYX

XYXYY

XYXYX

),()(

),(),()(),(),()(


NCTU EE 7

Independent:

)()(),()()(),(

)()(),(

yfxfyxfyFxFyxF

yYPxXPyYxXP

YXXY

YXXY

==

≤≤=≤≤

Example: 2-D Gaussian:

Example: n-D Gaussian:


NCTU EE 8

Conditional Probability: a derived probability measure

Conditional cdf, pdf:

Conditional random variable:

Bayes’ Theorem


NCTU EE 9

Functions of Random Variables

Assume y = g(x) is a monotonically increasing function.

fX(x)|dx| = fY(y)|dy|

)(1

)()(ygx

XY dydxxfyf

−=

=

In general,

∑= = −

=N

n ygx

iiXY

iidydxxfyf

1 )(1

)()(


NCTU EE 10

For random vector X, with pdf fX(x),

Assume x = g-1(y) has an inverse,

Remark: dx = |J|dy

Example: X and Y are independent and Gaussian, zero mean and 2σ variance: N(0, 2σ )

Let

or


NCTU EE 11

Rayleigh pdf

Statistical Averages

♦ Mean (Weighted Average)

♦ rth Moment: of X, r = 0, 1, 2, …

♦ rth Central Moment: of X, r = 0, 1, 2, …

Variance:


NCTU EE 12

♦ rth Joint Moment: of X and Y, i, j = 0, 1, 2, …

Correlation

Note: Independent: FXY(x,y) = FX(x)FY(y) Uncorrelated: E((X-E(X))(Y-E(Y))) = 0 Orthogonal: E(XY) = 0

♦ rth Joint Central Moment: of X and Y, i, j = 0, 1, 2, …

Covariance:

Correlation Coefficient:

♦ Conditional Expectation: of X given Y = y

♦ Expectation of Functions of X: Y = g(X)


NCTU EE 13

♦ Moment Generating Functions

♦ Characteristic Functions


NCTU EE 14

Error function & Q-function

Normalize normal distribution ),( xxmn σ :

)1,0(),( nmn xx →σ : standard normal distribution

22

2

21

2)(

2 21

21 y

mx

x

ee x

x−

−−

→ππσ

σ

Q-function:

1for ,22

1)(2/

21 2

2

>>≈≡−∞ −

∫ uuedyeuQ

u

u

y

ππ

Error-function:

)2(212)(0

2

uQdyeuerfu y −=≡ ∫ −

π

Note:

∫∫ −

−

≡

−

− ==u

u

s

syu

u

y dsedyeuerf2

2

2/2 22

211)(ππ

)(1)( uerfuerfc −≡

Thus, )2

(21)( uerfcuQ =

)/(21]22

1[2)]()[(/

2/2

xa

y

xx aQdyeamXamPx

σπσ

−=−=+≤≤− ∫∞ −

probability & random variables

Documents