8/9/2019 ENCS 6161 - ch7

1/15

Chapter 7

Sums of Random Variables andLong-Term Averages

ENCS6161 - Probability and StochasticProcesses

Concordia University

8/9/2019 ENCS 6161 - ch7

2/15

Sums of Random VariablesLet X1, , Xn be r.v.s and Sn = X1 + + Xn, then

E[Sn] = E[X1] + + E[Xn]

V ar[Sn] = V ar[X1 + + Xn]= E

n

i=1(Xi Xi)

n

j=1(Xj Xj)

=ni=1

V ar[Xi] +ni=1

nj=1

i=j

Cov(Xi, Xj)

If Z = X + Y (n = 2),V ar[Z] = V ar[X] + V ar[Y] + 2Cov(X, Y)

ENCS6161 p.1/14

8/9/2019 ENCS 6161 - ch7

3/15

Sums of Random VariablesExample: Sum of n i.i.d r.v.s with mean and

variance 2.

E[Sn] = E[X1] + + E[Xn] = nV ar[Sn] = nV ar[Xi] = n

2

pdf of sums of independent random variables

X1, , Xn indep r.v.s and Sn = X1 + + Xn, thenSn(w) = E[e

jwSn ] = E[ejw(X1++Xn)]

= X1(w) Xn(w)and

fSn(s) = F1 {X1(w) Xn(w)}

ENCS6161 p.2/14

8/9/2019 ENCS 6161 - ch7

4/15

Sums of Random Variables

Example: X1 Xn indep and Xi N(mi, 2i ). Whatis the pdf of Sn = X1 + + Xn?For a Guassian r.v.

X N(, 2) X(w) = ejww22

2

(prove it by yourself)So

Sn(w) =ni=1

ejwmiw22i2 = ejw(m1++mn)w

2(21++2n)/2

Sn

N(m1

+

+ mn

, 2

1+

+ 2

n)

What if X1, , Xn are not indep?? (hint: useY = AX)

ENCS6161 p.3/14

8/9/2019 ENCS 6161 - ch7

5/15

Sums of Random Variablespdf of i.i.d r.v.s

Sn(w) = (X(w))n

Example:Find the pdf of the sum of n i.i.d exponentialr.v.s with parameter .

X(w) =

jw (see table 3.2 on page 101)

Sn(w) = ( jw )n

fSn(s) =

es(s)n1

(n 1)!, s > 0

This is the so called m-Erlang r.v.

ENCS6161 p.4/14

8/9/2019 ENCS 6161 - ch7

6/15

Sums of Random VariablesWhen dealing with non-negative integer-valued r.v.s,we use the probability generating function:

GN(z) = E[zN

] = n znPN(n)PN(n) =

1

n!

dn

dznGN(z)z=0

For N = X1 + + Xn where Xi are independent.GN(z) = E[z

X1++Xn]

= E[zX1

] E[zXn

]= GX1(z) GXn(z)

ENCS6161 p.5/14

8/9/2019 ENCS 6161 - ch7

7/15

Sums of Random VariablesExample: Find the pdf of the sum of n independent

Bernoulli r.v.s with p0 = 1 p = q and p1 = p.GX(z) = E[z

X

] = q +pz GN(z) = (q +pz)n

PN(k) =

n

kpkqnk, k = 0, 1

n

(See Table 3.1)

N Binomial(n, p)

ENCS6161 p.6/14

8/9/2019 ENCS 6161 - ch7

8/15

The Sample MeanLet X be a r.v. with mean and variance 2. X1, , Xndenote n independent, repeated measurement of X. That

is, Xis are i.i.d r.v.s with the same pdf as X. The samplemean is defined as

Mn =Sn

n=

X1 + + Xnn

=1

n

n

i=1Xi

The mean and variance of the sample mean are

E[Mn] = E[1

n

n

i=1Xi] =

1

n

n

i=1E[Xi] =

V ar[Mn] = E[(Mn )2] = E

Sn E(Sn)n

2

=1

n2E[(Sn E(Sn)2)] =

1

n2V ar[Sn] =

2

n

ENCS6161 p.7/14

8/9/2019 ENCS 6161 - ch7

9/15

The Laws of Large NumbersFrom chebyshev inequality for any > 0

P

{|Mn

|

}

V ar[Mn]

2

=2

n2

So P{|Mn | < } > 1 2

n2

The weak law of large numbers:

limnP{|Mn | < } = 1for any > 0.

The strong law of large numbers:

P

limn

Mn =

= 1

The proof is beyond the level of this course.

ENCS6161 p.8/14

8/9/2019 ENCS 6161 - ch7

10/15

The Central Limit Theorem

Let X1, , Xn be i.i.d r.v.s with , 2 andSn = X1 + + Xn. Let

Zn =Sn

n

nthen as n , the distribution of Zn tends tostandard Gaussian.

limn

P[Zn z] = 12

z

ex2

2 dx

= 1 Q(z) = (z)

ENCS6161 p.9/14

8/9/2019 ENCS 6161 - ch7

11/15

The Central Limit TheoremProof:

Zn(w) = E[ejwZn] = E

e

jw

n

ni=1(Xi)

= E

ni=1

ejw(Xi)

n

=

ni=1

E

ejw(Xi)

n

( indep)

= Ee jw(Xi)n n ( i.i.d)(to be continued)

ENCS6161 p.10/14

8/9/2019 ENCS 6161 - ch7

12/15

The Central Limit TheoremProof: (continues)

Eejw(Xi)

n = E1 +

jw

n(X

) +

(jw)2

2!n2

(X

)2 + R(w)

= 1 +jw

nE[X ]

=0 w

2

2n2E[(X )2]

=2+E[R(w)]

= 1 w2

2n+ E[R(w)]

E[R(w)] becomes negligible compared to w2

2n when

n , thereforelimn

Zn(w) = limn

1 w

2

2n

n= e

w2

2

So, when n , Zn N(0, 1)ENCS6161 p.11/14

8/9/2019 ENCS 6161 - ch7

13/15

Convergence of Sequence of R.V.sX1, Xn are r.v.s, how to define the convergence ofof r.v.s? Recall: a r.v. is a function: S R. SoX1(w), X2(w),

are functions.

If Xn(w) X(w) for all w, sure convergenceIf P{w|Xn(w) X(w)} = 1,almost sure convergence, Xn X a.s. (or w.p. 1)If E[(Xn(w) X(w))2] 0 as n mean square convergence, Xn X m.s.If > 0, P{|Xn(w) X(w)| > } 0, convergencein probability.

ENCS6161 p.12/14

8/9/2019 ENCS 6161 - ch7

14/15

Convergence of Sequence of R.V.sa.s. convergence convergence in probabilitym.s. convergence convergence in probabilityBut almost sure mean square.

Convergence in distribution

Xn has cdf Fn(x) and X has cdf F(x). IfFn(x)

F(x) for all x where F(x) is continuous. We

call Xn converge to X in distribution.

Convergence in prob. convergence in distribution.

ENCS6161 p.13/14

8/9/2019 ENCS 6161 - ch7

15/15

Convergence of Sequence of R.V.sNote:weak LLN: convergence in prob.

Mn

in prob.

strong LLN: almost sureMn a.s.

CLT: convergence in distribution

Zn Z N(0, 1) in dist.In fact Mn m.s. sinceE[(Mn )2] = V ar[Mn] =

2

n 0 , as n

ENCS6161 p.14/14