eece 522 notes_01a review of probability.pdf

8/9/2019 EECE 522 Notes_01a Review of Probability.pdf

1/28

1/28

Review of ProbabilityReview of Probability


2/28

2/28

Random Variable! Definition

Numerical characterization of outcome of a randomevent

!Examples

1) Number on rolled dice

2) Temperature at specified time of day3) Stock Market at close

4) Height of wheel going over a rocky road


3/28


4/28

4/28

Two Types of Random Variables

Random Variable

Discrete RV

• Die

• Stocks

Continuous RV

• Temperature

• Wheel height


5/28


6/286/28

Most Commonly Used PDF: Gaussian

m & & are parameters of the Gaussian pdf

m = Mean of RV X

= Standard Deviation of RV X (Note: > 0)

2 = Variance of RV X

22 2/)(

2

1)( &

' &

m x X e x p

(($

A RV X with the following PDF is called a Gaussian RV

Notation: When X has Gaussian PDF we say X ~ N (m,& 2)


7/287/28

" Generally: take the noise to be Zero Mean

22 2

2

1)( &

' &

x x e x p $

Zero-Mean Gaussian PDF


8/28

8/28

Small

Large

Small Variability

(Small Uncertainty)

Large Variability

(Large Uncertainty)

p X ( x)

x

Area within )1 of mean = 0.683= 68.3%

x = m

p X ( x)

p X ( x)

x

x

Effect of Variance on Guassian PDF


9/28

9/28

"Central Limit theorem (CLT)

The sum of N independent RVs has a pdf

that tends to be Gaussian as N +

"So What! Here is what : Electronic systems generate

internal noise due to random motion of electrons in electroniccomponents. The noise is the result of summing the random

effects of lots of electrons.

CLT applies Guassian Noise

Why Is Gaussian Used?


10/28

10/28

Describes probabilities of joint events concerning X and Y . For

example, the probability that X lies in interval [a,b] and Y lies in

interval [a,b] is given by:

),( y x p XY

, - # #$%%%%b

a

d

c

XY dxdy y x pd Y cb X a ),()(and)(Pr

This graph shows the Joint PDF

Graph from B. P. Lathi’s book: Modern Digital & Analog Communication Systems

Joint PDF of RVs X and Y


11/28

11/28

When you have two RVs… often ask: What is the PDF of Y if X isconstrained to take on a specific value.

In other words: What is the PDF of Y conditioned on the fact X is

constrained to take on a specific value.

Ex.: Husband’s salary X conditioned on wife’s salary = $100K?

First find all wives who make EXACTLY $100K… how are their

husband’s salaries distributed.

Depends on the joint PDF because there are two RVs… but it

should only depend on the slice of the joint PDF at Y =$100K. Now… we have to adjust this to account for the fact that the joint

PDF (even its slice) reflects how likely it is that X =$100K will

occur (e.g., if X =105 is unlikely then p XY (105, y) will be small); so…if we divide by p X (10

5) we adjust for this.

Conditional PDF of Two RVs


12/28

12/28

Conditional PDF (cont.)

Thus, the conditional PDFs are defined as (“slice and normalize”):

./

.0

12

$

otherwise,0

0)(,)(

),(

)|(| x p

x p

y x p

x y p X

X

XY

X Y

./

.0

12

$

otherwise,0

0)(,)(

),(

)|(| y p

y p

y x p

y x p Y

Y

XY

Y X

This graph shows the Conditional PDF

Graph from B. P. Lathi’s book: Modern Digital & Analog Communication Systems

y is heldfixed x is heldfixed

“slice and

normalize”

y is held fixed


13/28

13/28

Independence should be thought of as saying that:

neither RV impacts the other statistically – thus, thevalues that one will likely take should be irrelevant to the

value that the other has taken.

In other words: conditioning doesn’t change the PDF!!!

)(

)(

),()|(

)()(

),(

)|(

|

|

x p

y p

y x p y x p

y p x p

y x p

x y p

X

Y

XY yY X

Y X

XY x X Y

$$

$$

$

$

Independent RV’s


14/28

14/28

Independent and Dependent Gaussian PDFs

Independent

(zero mean)

Independent(non-zero mean)

Dependent

y

x

y

x

y

x

Contours of p XY ( x , y).

If X & Y are independent,

then the contour ellipsesare aligned with either the

x or y axis

Different slices

give

different

normalized curves

Different slicesgive

same normalized

curves


15/28


16/28


17/28

17/28

Motivation First w/ “Data Analysis View”

Consider RV X = Score on a test Data: x1, x2,… x N

Possible values of RV X : V0 V1 V2... V1000 1 2 … 100

This is called Data Analysis View

But it motivates the Data Modeling View

N i = # of scores of value V i N = (Total # of scores)3

$

n

i

i N

1

Test

Average

4 P( X = V i)

33$

$ $""

$$$100

0

10011001 ...

i

ii

n N i i

N

N V

N

V N V N V N

N

x x

Probability

Statistics

Motivating Idea of Mean of RV


18/28

18/28

Theoretical View of Mean

" For Discrete random Variables :

Data Analysis View leads to Probability Theory:

" This Motivates form for Continuous RV:

Probability Density Function

3$

$n

n

i X i x P x X E

1

)(}{

#

+

+($ dx x p x X E X )(}{

Probability Function

Notation: X X E $}{

Data Modeling

Shorthand Notation


19/28

19/28

Aside: Probability vs. Statistics

Probability Theory» Given a PDF Model

» Describe how the

data will likely behave

Statistics» Given a set of Data

» Determine how the

data did behave

3$$

n

ii x N Avg 1

1

Data

There is no DATA here!!!

The PDF models how

the data will likely behave

There is no PDF here!!!

The Statistic measures how

the data did behave

#

+

+($ dx x p x X E X )(}{

Dummy Variable

PDF

4“Law of LargeNumbers”


20/28


21/28

21/28

Motivating Idea of Correlation

Consider a random experiment that observes the

outcomes of two RVs:Example: 2 RVs X and Y representing height and weight, respectively

y

x

Positively Correlated

Motivate First w/ Data Analysis View

x

y


22/28


23/28

23/28

To capture this, define Covariance :

If the RVs are both Zero-mean :

)})({( Y Y X X E XY (($&

}{ XY XY 5$&

If X = Y : 22 Y X XY & & & $$

# # (($ dxdy y x pY y X x XY XY ),())((&

Prob. Theory View of Correlation

If X & Y are independent, then:0$

XY &


24/28

24/28

If 0)})({( $(($ Y Y X X E XY &

Then… Say that X and Y are “uncorrelated”

If 0)})({( $(($ Y Y X X E XY &

Then Y X XY E $

}{Called “Correlation of X & Y ”

So… RVs X and Y are said to be uncorrelated

if XY = 0

or equivalently… if E { XY } = E { X } E {Y }


25/28

25/28

X & Y are

IndependentImplies X & Y are

Uncorrelated

Uncorrelated

Independence

INDEPENDENCE IS A STRONGER CONDITION !!!!

)()(

),(

y f x f

y x f

Y X

XY

$ }{}{

}{

Y E X E

XY E

$

Independence vs. Uncorrelated

PDFs Separate Means Separate


26/28

26/28

Covariance :

)})({( Y Y X X E XY (($

&

Correlation : }{ XY E

Correlation Coefficient :

Y X XY XY & &

& 6 $

11 77( XY 6

Same if zero mean

Confusing Covariance and

Correlation Terminology

C i d C l ti F


27/28

27/28

Correlation Matrix :

, - , - , -

, - , - , -

, - , - , -88888

8

9

:

;;;;;

;

<

=

$$

N N N N

N

N

T

X X E X X E X X E

X X E X X E X X E

X X E X X E X X E

E

!

"#""

!

!

21

22212

12111

}{xxR x

T N X X X ][ 11 !$x

Covariance Matrix :

}))({( T E xxxxCx (($

Covariance and Correlation For

Random Vectors…


28/28

28/28

}{}{}{ Y E X E Y X E "$"

> ?> ?, -

> ? > ?, -> ?, - > ?, - , -

XY Y X

z z z z

z z z z

z z z

Y X E Y E X E

Y X Y X E

X X X Y X E

Y X Y X E Y X

& & & 2

2

2

where

}var{

22

22

22

2

2

""$

""$

""$

($"$

(("$"

./

.

0

1

"

""

$"eduncorrelatare&if ,

2

}var{22

22

Y X Y X

Y X

XY Y X

& &

& & &

#$ dx x p x f X f E X )()()}({}{}{ X aE aX E $

22

}var{ X aaX & $

A Few Properties of Expected Value

eece 522 notes_01a review of probability.pdf

Documents