chapter 2: introduction to probabilityhomepages.wmich.edu/~bazuinb/ece3800sw/sw_notes02.pdf ·...

Notes and figures are based on or taken from materials in the course textbook: Probability, Statistics and Random Processes for Engineers, 4th ed., Henry Stark and John W. Woods, Pearson Education, Inc., 2012.

B.J. Bazuin, Fall 2016 1 of 66 ECE 3800

Henry Stark and John W. Woods, Probability, Statistics, and Random Variables for Engineers, 4th ed.,

Pearson Education Inc., 2012. ISBN: 978-0-13-231123-6

Chapter 2: Introduction to Probability

Sections 2.1 Introduction 79 2.2 Definition of a Random Variable 80 2.3 Cumulative Distribution Function 83 Properties of FX(x) 84 Computation of FX(x) 85 2.4 Probability Density Function (pdf) 88 Four Other Common Density Functions 95 More Advanced Density Functions 97 2.5 Continuous, Discrete, and Mixed Random Variables 100 Some Common Discrete Random Variables 102 2.6 Conditional and Joint Distributions and Densities 107 Properties of Joint CDF FXY (x, y) 118 2.7 Failure Rates 137 Summary 141 Problems 141 References 149 Additional Reading 149

Random variable: A real function whose domain is that of the outcomes of an experiment (sample space, S) and whose actual value is unknown in advance of the experiment.

From: http://en.wikipedia.org/wiki/Random_variable

A random variable can be thought of as the numeric result of operating a non-deterministic mechanism or performing a non-deterministic experiment to generate a random result.

Unlike the common practice with other mathematical variables, a random variable cannot be assigned a value; a random variable does not describe the actual outcome of a particular experiment, but rather describes the possible, as-yet-undetermined outcomes in terms of real numbers.



2.2 Definition of a Random Variable

This section is strongly driven by theory and formality.

It assumes that you have heard about a “Cumulative Distribution Function” to define probability … so is a bit backwards (Sec 2.3 defines it).

So …

Cumulative Distribution Function (CDF):The probability of the event that the observed random variable X is less than or equal to the allowed value x.

xXxFX Pr

The defined function can be discrete or continuous along the x-axis. Constraints on the cumulative distribution function are:

xforxFX ,10

0XF and 1XF

XF is non-decreasing as x increases

1221Pr xFxFxXx XX

Returning to random variables …



For any experiment with a sample space, we map the outcomes into a numerical space, assigning a real number (or numbers). Therefore, it is a mapping from experiments and events to numbers with values. For consistency on a number line, we define the events in terms of segments beginning at -∞ up to the value x as xX .

All sets of engineering interest can be written as countable unions or intersections of events on the interval (-∞,x]. Thus, if X is a random variable, the set of points is an event.

Under the mapping X we have generated a new probability space (R1,B,PX), where R1 is the real line, B is the Borel σ-field of all subsets of R1 generated by all unions, intersections, and complements of the semi-infinite interval (-∞,x], and PX is a set of function assigning a number PX[A]≥0 to each set BA .

Figure 2.2-1 Symbolic representations of the action of the random variable X.

Enough of the theoretical semantics ….



2.3 Cumulative Distribution Function

In some text this is call the Probability Distribution Function but this is too close to another required term … the probability density function often referred to as the pdf.

The CDF is defined by

xPXXPxF XX ,:

You may also see

xXPxF XX

For discrete probability of discontinuous curves, the probability is “inclusive” of x or as the text book suggests; the CDF value immediately to the right of x.

Properties of the CDF, xFX

i. 0XF and 1XF

ii. XF is non-decreasing as x increases. For 2121 xFxFxx XX

iii. xFX is continuous from the right, that is,

0,lim0

xFxF XX

Derivable concepts

xforxFX ,10

1221 xFxFxXxP XX

Opened and close interval derivations: (note these are needed for discrete probability)

11221 xXPxFxFxXxP XX

21221 xXPxFxFxXxP XX

211221 xXPxXPxFxFxXxP XX



For discrete events, the probability density function, on the x-axis, consists of discrete steps “climbing” towards 1 at the appropriate points.

For a six-sided die,

6

161,Pr intint egereger aaX

The probability density function can be defined as:

For discrete events, 061,Pr intint egereger aaX or

061,Pr intintintint egerXegerXegereger aFaFaaX

There will be a difference for continuous events … coming soon.

Examples:

6

111Pr XFX

2

133Pr XFX

6

555Pr XFX

0.177Pr XFX

6

2

6

41414Pr XFX

6

3

6

2

6

52552Pr XX FFX

From: Probabilistic Methods of Signal and System Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press.



For continuous events, the CDF consists of a continuous, non-decreasing curve. For example:

10

xFX

1.0

0.0x

-10

Examples:

2

100Pr XFX

4

155Pr XFX

20

13103

20

133Pr XFX

20

3107

20

177Pr XFX

4

1

20

5105

20

11515Pr XFX

20

2101

20

1101

20

11111Pr XX FFX

What about 0Pr X ?

20

210

20

110

20

1Pr

XX FFX

020

02

20

2limPrlim

00

X

As there are an infinite number of points in any region of the x axis, the probability of any specific point for a continuous distribution is zero.

From: Probabilistic Methods of Signal and System Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press.



Example 2.3-2: Waiting for a bus

A bus arrives at random in the interval T,0 . For the random variable related to the bus arrival, the bus is equally likely of coming at any time during the interval (uniformly distributed). Then

tT

TtT

t

t

tFX

,1

0,

0,0

Figure 2.3-2 Cumulative distribution function of the uniform random variable X of Example 2.3-2.



Example 2.3-3: Binomial Distribution Function

Flipping 4 coins and counting the number of heads, but in the case, each of the coins is unfair with a probability p=0.6 of coming up heads.

x

k

knkX pp

k

nxF

0

1

Note: The textbook is in error, use the above formula (k should be j in text).

Figure 2.3-3 Cumulative distribution function for a binomial RV with n = 4, p = 0.6.

see MATLAB: Ex_2_3_3.m Discrete probability density function 2.560% of the time get 0 heads 15.360% of the time get 1 heads 34.560% of the time get 2 heads 34.560% of the time get 3 heads 12.960% of the time get 4 heads Discrete cumulative distribution function 2.560% of the time get 0 or fewer heads 17.920% of the time get 1 or fewer heads 52.480% of the time get 2 or fewer heads 87.040% of the time get 3 or fewer heads 100.000% of the time get 4 or fewer heads



Example 2.3-4: Binomial Distribution Function Computations Discrete probability density function 2.560% of the time get 0 heads 15.360% of the time get 1 heads 34.560% of the time get 2 heads 34.560% of the time get 3 heads 12.960% of the time get 4 heads Discrete cumulative distribution function 2.560% of the time get 0 or fewer heads 17.920% of the time get 1 or fewer heads 52.480% of the time get 2 or fewer heads 87.040% of the time get 3 or fewer heads 100.000% of the time get 4 or fewer heads

(a) ?35.1 XPX Note the upper less than sign instead of less than or equal to and non-integer:

1335.13335.1 XXXXXXX FPFFPFXP

3456.01792.03456.087040.035.1 XPX

(b) ?30 XPX Note the lower greater than or equal to sign:

00330 XXXX PFFXP

8704.00256.00256.08704.030 XPX

(c) ?8.12.1 XPX

Note the non-integer values:

112.18.18.12.1 XXXXX FFFFXP

01792.01792.08.12.1 XPX

(c) ?399.1 XPX Note the upper less than sign instead of less than or equal to and non-integer:

99.1133

99.199.133399.1

XXXX

XXXXX

PFPF

PFPFXP

3456.001792.03456.08704.0399.1 XPX

Note: Equation on top of p.88 is wrong … forgot the combinatorial! See p. 86.



2.4 Probability Density Function (the pdf)

The derivative of the cumulative distribution function

dx

xdFxFxFxf XXX

X

0lim

Assumption …the CDF is continuous and differentiable.

Properties of the pdf, if it exists, include

1. xforxf X ,0

2. 1

XXX FFdxxf

3. xXPduufxFx

XX

4. 1221

2

1

Pr xFxFdxxfxXx XX

x

x

X

From: http://en.wikipedia.org/wiki/Probability_density_function

In mathematics, a probability density function (pdf) serves to represent a probability distribution in terms of integrals. A probability density function is everywhere non-negative and its integral from −∞ to +∞ is equal to 1. If a probability distribution has density f(x), then intuitively the infinitesimal interval [x, x + dx] has probability f(x) dx.

An interpretation is

xFdxxFdxxXxPdxxf XXX

This helps with discrete functions ….

Observe that if xf X exists, meaning that it is bounded and has at most a finite number

of discontinuities, then xFX is continuous and therefore 0 xXP (except at the discontinuities).



ProbabilityMassFunction(pmf)

The probability that a discrete random variable takes on an exact value is defined as the pmf.

xXxf X Pr

xFxFxf XXX

Note that for discrete random variables, the cumulative distribution function, CDF, is not continuous at the discrete inputs of interest.

Properties of the pdf include

1. xforxf X ,0

2. 1

u

X uf

3.

x

u

XX ufxF

4.

2

1

21Prx

xuX ufxXx

From: http://en.wikipedia.org/wiki/Probability_mass_function

In probability theory, a probability mass function (abbreviated pmf) gives the probability that a discrete random variable is exactly equal to some value. A probability mass function differs from a probability density function in that the values of the latter, defined only for continuous random variables, are not probabilities; rather, its integral over a set of possible values of the random variable is a probability.

Note: The textbook does not differentiate between the probability density function and probability mass function. Notice that in the definitions, the pmf represents the actual probability while the pdf is defined in terms of the derivative of the “distribution” function (CDF).

If you wish to pursue correct mathematical derivations, use pmf and pdf and CDF. If you just intend to apply this concept to engineering problems, you can do it like the textbook. …



Examples:

1 2 3 4 5 6

xf X1.0

0.0x

1/6

Cumulative Distribution Function (CDF) Probability Mass Function (pmf)

10

xFX

1.0

0.0x

-10 10

xf X

1.0

0.0x

-10

1/20

Cumulative Distribution Function (CDF) Probability Density Function (pdf)



Examples:

Given a CDF of

xfor

xFX ,

5tan

21

2

1 1

Define the pdf …

The derivative of the CDF is the pdf. Therefore,

xfor

xxf X ,

5

1522

Math hint: dx

du

uu

dx

d

221

1

1tan

From: Probabilistic Methods of Signal and System Analysis (3rd ed.) by George R. Cooper and Clare D. McGillem; Oxford Press, 1999. ISBN: 0-19-512354-9

-100 -80 -60 -40 -20 0 20 40 60 80 1000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Probability Distribution Function (PDF)

-100 -80 -60 -40 -20 0 20 40 60 80 1000

0.01

0.02

0.03

0.04

0.05

0.06

0.07Probability Density Function (pdf)



UpdatingPreviousExamples

Experiment: Flip two Coins and count the number of heads

HHHTTHTTSPair ,,, 2,1,0S

For xXxFX Pr

xfor

xfor

xfor

xfor

xFX

2,1

21,43

10,41

0,0

And the probability mass function, xXxf X Pr , is then

else

xfor

xfor

xfor

xf X

,0

2,41

1,42

0,41

1 2 3 4

xFX

1.0

0.0x

0-1 1 2 3 4

xf X

1.0

0.0x1/4

0-1

1/2

Cumulative Distribution Function (CDF) Probability Mass Function (pmf)

Note: The pmf corresponds to Bernoulli trials of 0, 1, and 2 occurrences in 2 trials with a probability of 50%.

knkn qp

k

nkptrialsnintimeskoccuringA

Pr

4

15.0

0

20 2

2

p

4

25.0

1

21 2

2

p

4

15.0

2

22 2

2

p



First Look describing “named” random variables

Note: there are documents describing specific discrete and continuous pmf, pdf and PDF or CDF on the password web site.

UniformRandomVariables

The uniform random variable arises in situations where all values in an interval of the real line are equally likely to occur. The uniform random variable U in the interval [a,b] has pdf:

bxandx

bxaabxfU

0,0

,1

bx

bxaab

ax

x

xFU

,1

,

0,0

xFX

x

xf X

x

Note: this is the “generic derivation”. It is applicable for all cases! Some classic problems: Arrival time: Uniform density on 0 to T or on –T1 to + T2 Random Phase angles: Uniform from 0 to 360 degrees or 0 to 2 pi or –pi to pi.



ExponentialRandomVariables

The exponential random variable arises in the modeling of the time between occurrence of events (e.g., the time between customer demands for call connections), and in the modeling of the lifetime of devices and systems. The exponential random variable X with parameter l has pdf

0,exp

0,0

xx

x

dz

xdFxf X

X

0,exp1

0,0

xx

xxFX

Expected waiting times for “services” (computer, traffic, stores, etc.) The modeling the lifetimes of devices and systems. (probability of failure increases exponentially in time). Interesting note … this distribution has a “memoryless” property. As time or values passes, the probability remains exponential …

Fx(x

)

f x(x)



TheGaussianprobabilitydensityfunction(pdf)

The Gaussian or Normal probability density function is defined as:

xforx

xf X ,2

exp2

12

2

where is the mean and is the variance

The Gaussian Cumulative Distribution Function (CDF)

dv

vxF

x

v

X

2

2

2exp

2

1

The CDF can not be represented in a closed form solution!

Not yet proven reasons for importance:

1. It provides a good mathematical model for a great many different physically observed random phenomena that can be justified theoretically in many ways.

2. It is one of the few density functions that can be extended to handle an arbitrarily large number of random variables conveniently.

3. Linear combinations of Gaussian random variables lead to new random variables that are also Gaussian. This is not true for most other density functions.

4. The random process from which Gaussian random variables are derived can be completely specified, in a statistical sense, from a knowledge of the first and second moments. This is not true for other processes. All higher level moments are sums, products and/or powers of the mean and variance.

5. In system analysis, the Gaussian process is often the only one for which a complete statistical analysis can be carried through in either the linear or nonlinear situation.

6. The function is infinitely differentiable (all the derivatives exist).

-8 -6 -4 -2 0 2 4 6 80

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Gaussian PDF and pdf



Gaussian or Normal Distribution

http://en.wikipedia.org/wiki/Normal_distribution http://en.wikipedia.org/wiki/Normal_distribution#Occurrence

To summarize, here is a list of situations where approximate normality is sometimes assumed. For a fuller discussion, see below.

In counting problems (so the central limit theorem includes a discrete-to-continuum approximation) where reproductive random variables are involved, such as

Binomial random variables, associated to yes/no questions; Poisson random variables, associated to rare events;

In physiological measurements of biological specimens: The logarithm of measures of size of living tissue (length, height, skin

area, weight); The length of inert appendages (hair, claws, nails, teeth) of biological

specimens, in the direction of growth; presumably the thickness of tree bark also falls under this category;

Other physiological measures may be normally distributed, but there is no reason to expect that a priori;

Measurement errors are assumed to be normally distributed, and any deviation from normality must be explained;

Financial variables The logarithm of interest rates, exchange rates, and inflation; these

variables behave like compound interest, not like simple interest, and so are multiplicative;

Stock-market indices are supposed to be multiplicative too, but some researchers claim that they are Levy-distributed variables instead of lognormal;

Other financial variables may be normally distributed, but there is no reason to expect that a priori;

Light intensity The intensity of laser light is normally distributed; Thermal light has a Bose-Einstein distribution on very short time scales,

and a normal distribution on longer timescales due to the central limit theorem.



TheGaussianCumulativeDistributionFunctionis

dv

vxF

x

v

X

2

2

2exp

2

1


xforx

xf X ,2

exp2

12

2

Important notes on the curve:

1. There is only one maximum and it occurs at the mean value.

2. The density function is symmetric about the mean value.

3. The width of the density function is directly proportional to the standard deviation, . The width of 2 occurs at the points where the height is 0.607 of the maximum value. These are also the points of the maximum slope. Also note that:

683.0Pr X

955.022Pr X

4. The maximum value of the density function is inversely proportional to the standard deviation, .

2

1Xf

5. Since the density function has an area of unity, it can be used as a representation of the impulse or delta function by letting approach zero. That is

2

2

0 2exp

2

1lim

xx



ConversionoftheGaussiantothestandardnormal

The CDF is tabulated for a zero mean, unit variance in Table 1 page G-3 of the appendix. For these values, it is often described as “normalized” and is defined as

dyy

2exp

2

1 2

The distribution function is then converted based on the normalizing relationship

x

y

x

yxFX

When using Appendix D, the negative values of u are derived from the positive as

uu 1

AnotherwaytofindvaluesfortheGaussian

The error function, defined as (Note: this is not your textbook definition of erf !!)

duuxerf

x

u

0

2exp2

22

1

2

1

21

2

11

xerf

xerfxFX

For multiple bounds

22

1

2

1

22

1

2

11

aerf

berfFbFbXaP XXX

22

1

22

11

aerf

berfFbFbXaP XXX

For some reason, the textbook has defined erf() differently than MATLAB and EXCEL and WIKIPEDIA. Other sources do it the author’s way …

This may be a problem … to be determined!!



MeanandVarianceofarandomvariable(ImportantNotEmphasizedinText)

In chapter 4 we will be doing this a lot (like for every density function used), but in the meantime …

The mean value of a random variable is defined as

dxxfxXEX X

For a discrete random variable, integration is replaced by summation and

i

iXii

iXi xPxxfxXEX

The variance of a random variable is defined as

dxxfxXEXX X2222

For a discrete random variable, integration is replaced by summation and

i

iXii

iXi xPxxfxXEXX 22222

Often when we talk about values we say

That is to say that we expect the result x to be

x



2.5 Continuous, Discrete and Mixed R.V.

see “Continuous RV” and Discrete RV” on the solution web site ….

derived from the ECE 5820 textbook: Alberto Leon-Garcia, “Probability, Statistics, and Random Processes For Electrical Engineering, 3rd ed., Pearson Prentice Hall, 2008, ISBN: 013-147122-8.

A Continuous R.V. as a continuous density (pdf) and distribution (CDF).

integrals and derivatives are used

A Discrete R.V. as a CDF composed of steps and a pdf composed of delta functions with magnitude.

summations or differences are used Dirac delta functions (another area where engineers drive mathematicians crazy)

A Mixed R.V. has both constructs present. There will be cases where a continuous R.V. becomes a Mixed R.V. due to conditionals or other “knowledge gained” about an event.

mixed math is likely required (integrals and sums along with derivatives and differences)

Some Continuous RV Densities



Defining proper CDF/pdf functions

Example: xxFX1tan1

Determine the values of and .

Known values at 0XF and 1XF and monotonically increasing, meaning that the derivative is positive.

0tan1 1 XF

02

1

XF

02

1

02

1

2

1tan1 1 XF

12

1

XF

12

21

2

1

Therefore,

xxFX

1tan2

12

1

and

21

11

xxf X

As a check, it is positive for all selections of x!

This is a Cauchy Random Variable density and distribution! (It has really odd properties.)



Exercise 2-3.1Cooper and McGillem: A probability density function

xuexf xKX 5

Determine the value of K

dxxf X1

0

51 dxe xK

0

151 xKe

K

5111

51 0

Ke

Ke

KKK

5K

Therefore xuexf xX 55

As an extension or generalization, note that xueKxf xKX

Probability X>1

1

11Pr dxxfX X

1

0

5511Pr dxeX x

1

0

5

5

511Pr xeX

0067.01111Pr 55 eeX

Probability X≤0.5

5.0

0

555.0Pr dxeX x

5.0

0

5

5

55.0Pr xeX

9179.01115.0Pr 5.25.2 eeX



Example: Archery target shooting with

A Rayleigh distribution - two dimensional Gaussian

Archer capability described by 4

125.0 YX in feet

0,0

0,2

exp2

2

2

rfor

rforrr

rfR

0,0

0,2

exp12

2

rfor

rforr

rFR

0,0

0,8exp16 2

rfor

rforrrrfR

0,0

0,8exp1 2

rfor

rforrrFR

Assume a 1 foot radius target with a 1 inch radius Bulls-eye

The archers expected performance can be described by ….

Probability of a Bulls-eye (1 inch radius)

0540.0144

8exp1

12

18exp1

12

1 2

RF

Probability of missing the target (1 foot radius)

42 1035.38exp18exp1111 RF



Some Common Continuous Random Variables

1.Uniform

otherwise

bxaabxfU

,0

,1

xb

bxaab

ax

ax

xFU

,1

,

,0

2. Triangle

xfor

xforx

xforx

xfor

xf X

1,0

10,1

01,1

1,0

xfor

xforxx

xforxx

xfor

xFX

1,1

10,2

1

2

01,2

1

2

1,0

2

2

3. Exponential

0,exp

1

0,0

0,exp

0,0

xx

x

xx

xxf X

0,exp1

0,0

0,exp1

0,0

xx

x

xx

xxFX



4. Laplacian

0,2

exp2

1

xxf X

5. Rayleigh

0,0

0,2

exp2

2

2

rfor

rforrr

rfR

0,0

0,2

exp12

2

rfor

rforr

rFR



Triangular density function example

Assume that a random variables probability density function is triangular and can be described as

xfor

xforx

xforx

xfor

xf X

1,0

10,1

01,1

1,0

Find the cumulative distribution function.

The definition

x

v

XX dvvfxF

For 1 x 0xFX

For 01 x

x

v

X dvvxF

1

1

x

Xv

vxF

1

2

2

2

1

22

11

2

22

x

xxxxFX

For 10 x

x

v

X dvvxF

0

12

1

x

Xv

vxF

0

2

22

1

2

1

222

1 22

x

xxxxFX

For x1 1xFX



Therefore,

xfor

xforxx

xforxx

xfor

xFX

1,1

10,2

1

2

01,2

1

2

1,0

2

2

There will be numerous times that problems must be broken into multiple subproblems to perform solutions as segments … like here for -1<x<0 and 0<x<1.

-2 -1.5 -1 -0.5 0 0.5 1 1.5 2-0.2

0

0.2

0.4

0.6

0.8

1

1.2



Some Common Discrete Random Variables

1. Bernoulli 1,0XS

qpp 10 and pp 1 , for 10 p

else

kp

kq

kpmfkPB

,0

1,

0,

1 kpkqkpmfkPB

k

kq

k

kFB

1,1

10,

0,0

1 kupkuqkpmfkFB

2. Binomial nS X ,,2,1,0

knkk pp

k

np

1 , for nk ,,2,1,0

else

nkppk

n

kpmfkPknk

B

,0

,,1,0,1

kn

nkppj

n

k

kFk

j

jnjB

,1

0,1

0,0

0



4. Geometric

First Version

,2,1,0XS

else

kppkpmfkP

k

B,0

,,1,0,1

kpp

k

kF k

j

jB 0,1

0,0

0

Math Tricks ….

1,1

11

1

00

qfor

q

qpqppp

kk

j

jk

j

j

1

1

0

111

11

k

kk

j

j qp

qppp

Therefore, it is commonly stated as

kq

qp

k

kF kB 0,

1

1

0,01

Second Version

,2,1XS

else

kppkpmfkP

k

B,0

,,2,1,1 1

kq

qp

k

kF kB 1,

1

1

1,0

3. Poisson



,2,1,0XS

ek

pk

k !, for ,2,1,0k

else

nkekkpmfkP

k

B

,0

,,1,0,!

kk

e

k

kF k

j

kB 0,

!

0,0

0



Example 2.4-5 Cell phone received signal power model.

The power can be described as a Rayleigh distribution.

rforrr

rf R

0,2

exp2

2

2

rforr

rFR

0,2

exp12

2

Assume mW1 for the r power radius.

What is the probability that the power W is less than 0.8 mW?

2

2

12

8.0exp18.0RF

or

8.0

02

2

2 12exp

18.0 dr

rrPR

Hint:

2

2

2

2

2 2exp

2exp

2

2

xdx

xx

2

0exp

2

8.0exp

2exp8.0

228.0

0

2rPR

29.032.0exp18.0 RP



2.6 Conditional and Joint Distributions and Densities

Using the Cumulative Distribution Function (CDF), define

BxXBxFX |Pr|

B

BxXBxXBxFX Pr

,Pr|Pr|

, for 0Pr B

where BxX , is the event of all outcomes such that

xX and B

Note: X is the value of the random variable when the experimental outcome is .

The event B that conditions the probability has several possibilities:

1. Every B may be an event that can be expressed in terms of the random variable X. Which means we have a simple conditional probability.

2. Every B may be an event that depends upon some other random variable, which may be either continuous or discrete. A joint probability or independent Prob.

3. Every B may be an event that depends upon both the random variable X and some other random variable. Even more complicated.

For our purposes (#1 above) envision that

mXB or mXB

Then, it can be shown that BxF | is a valid cumulative distribution function with all the expected characteristics:

1. xforBxF ,1|0

2. 0| BF and 1| BF

3. BxF | is non-decreasing as x increases

4. BxFBxFBxXx |||Pr 1221



Example 2.6-1:

Given 10 XB , compute the conditional CDF for values xX .

For any continuous function …

1|10 BxFX

We know that event B has occurred … therefore any resulting CDF for x>10 is the same as 1| BF .

The new function only has non-unity values where 10x . But the CDF must be rescaled by BFX

10,,

10,1

|x

BF

BxF

x

BxF

X

XX

Figure 2.6-1 Conditional and unconditional CDFs of X.

Based on the condition B, is known that 10 XB . Therefore, the conditional CDF

must go to 1.0 for BC or for 10 XBC . Also as a result, the density function in this region becomes 0.0 (the derivative of a constant). It can be seen that …

10,

10,0

|x

BF

xf

x

Bxf

X

XX



For the new “subspace based on the condition B

If we define the density using the derivative,

dx

BxFdBxf X

X

||

In general, envision that

BF

xfBxf

X

XX | , for Bx

and

0| Bxf X , for Bx

Again, the density function is a scaled version of the original density function in the range where the event B exists and is 0 outside of B.

Properties of the new pdf must include

1. xforMxf ,0|

2. 1|

dxMxf

3. duMufMxF

x

||

4. dxMxfxXx

x

x

2

1

|Pr 21



Example p. 100 Cooper and McGillem: Conditional Gaussian density function given that the event, M, is less than or equal to the mean value.

xforx

xf X ,2

exp2

12

2


For the event M,

21 MxF

Then, constructing a conditional cumulative distribution, MXxF | , the density function is defined as

2

2

2exp

2

2

21

|

xxf

XF

xfMxf , for x

and 0| Mxf , for x

The previous and new pdf and CDF can be “sketched” as: (see CondGaussianRandn.m)

-4 -3 -2 -1 0 1 2 3 40

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Cond. gaussian Dist. and Density Fundtions

pdf

PDFc-pdf

C-PDF



Distribution function as a weighted sum of conditional distribution functions.

Define a mutually exclusive, exhaustive set of events: niAi ,,1, .

Note that: 11

n

iiAP

Then total probability requires that

n

iiiXX APAxFxF

1

|

Example 2.6-3: good and defective memory chips with different CDFs

xux

defectxFX

2exp1|

xux

goodxFX

10exp1|

Two different exponential failure rates, one much faster than the other!

Also assume we are given

61 qdefectP and 6

5 pgoodP

Then a total probability for an IC is

goodPgoodxFdefectPdefectxFxF XXX ||

6

5

10exp1

6

1

2exp1

xxxFX

10exp

6

5

2exp

6

11

xxxFX

What is the probability that a chip will fail before 6 months?

534.010

6exp

6

5

2

6exp

6

116

XF



Joint Cumulative Distribution Functions

Cumulative Distribution Function:The probability of the event that the observed random variable X is less than or equal to the allowed value x and that the observed random variable Y is less than or equal to the allowed value y.

yYxXyxFXY ,Pr,

The defined function can be discrete or continuous along the x- and y-axis. Constraints on the cumulative distribution function are:

1. yandxforyxFXY ,1,0

2. 0,,, XYXYXY FxFyF

3. 1, XYF

4. yxFXY , is non-decreasing as either x or y increases

5. xFxF XXY , and yFyF YXY ,

Analogies:

a 2-dimensional probability moving from scalars to vectors (2 or more elements) Calc 3 as compared to Calc 1 & 2?



2-D probability computations: yYyxXxP 121 ,

Think in terms of unions and intersections of 2-D boxes in the x,y plane …

Figure 2.6-4 Point set associated with the event {X ≤ x, Y ≤ y}.

Figure 2.6-5 Point set for the event {x1 < X ≤ x2, y1 < Y ≤ y2}.

Then by inspection we can arrive at …

11211222121 ,,,,, yxFyxFyxFyxFyYyxXxP XYXYXYXY

Read the textbook p. 118-120 and see if you prefer the mathematical way … … math works, geometry provides some intuition …



JointProbabilityDensityFunction(pdf)

The derivative of the cumulative distribution function is the density function

yx

xFyxf X

2

,

Properties of the pdf include

1. yandxforyxf ,0,

2. 1,

dydxyxf

Note: the “volume” of the 2-D density function is one.

3.

y x

dvduvufyxF ,,

4. dyyxfxf X

, and dxyxfyfY

,

5. 2

1

2

1

,,Pr 2121

y

y

x

x

dydxyxfyYyxXx



UniformDensityExample

The uniform density function in two dimensions can be defined as:

else

yyyandxxxforyyxx

yxf YX

,0

,1

, 21211212

,

Determine the density function in y

2

1

,,

x

x

YXY dxyxfyf

2

1

2

1 12121212

1x

x

x

x

Y yyxx

xdx

yyxxyf

21

121212

12 ,1

yyyforyyyyxx

xxyfY

Similarly, the density function in x is

2112

,1

xyxforxx

xf X

Note: this is a characteristic of random variables that are independent …

yfxfyxf YXYX ,,

and

yFxFyxF YXYX ,,

The inverse is also true … if the joint pdf and CDF have this property, then the random variables X and Y are independent!



Exercise 3-1.2 Cooper and McGillem

else

yandxforyxAyxf YX

,0

00,32exp,,

Determine A

0 0

32exp,1 dydxyxAdydxyxf

0 00 0 2

2exp3exp2exp3exp1 dy

xyAdydxxyA

3

1

2

1

3

3exp

2

13exp

2

11

00

Ay

AdyyA

6A

Determine the Distribution Function

y x

dydxyxyxF0 0

32exp6,

y x

dydxxyyxF0 0

2exp3exp6,

y

dyx

yyxF0 2

2exp

2

13exp6,

yxyx

yxF

3exp12exp13

3exp

3

1

2

2exp

2

16,

Then, for

4

1,

2

1Pr yx

4

13exp1

2

12exp1

4

1,

2

1

4

1,

2

1Pr Fyx

4

3exp11exp1

4

1,

2

1

4

1,

2

1Pr Fyx



Conditional Probability (Again, with multiple r.v.)

Using the Cumulative Distribution Function (CDF), define

yF

yxF

M

MxXyYxF

YX

,

Pr

|Pr|

Another way.

12

1221

,,|

yFyF

yxFyxFyYyxF

YYX

Leading to (for the Y interval going to zero),

yf

yxfyYxf

YX

,|

and

xf

yxfxXyf

XY

,|

These are different from the probability of a continuous distribution taking on a single value in X and Y…

0 xXFX or 0 yYFY

An engineering derivation follows:

y

yFyyFy

yxFyyxF

yFyyF

yxFyyxFyYxF

YYyYYyX

,,

lim,,

lim|00

yf

duyuf

yyF

yyxF

yYxFY

x

YX

,,

|

Then taking the partial with respect to x

yf

yxf

yyF

xyyxF

yYxfxyYxF

YYX

X,

,

||

2

yf

yxfyYxf

YX

,|



The corresponding conditional density function is

yf

yxfyYxf

Y

,|

and similarly it can be shown that

xf

yxfxXyf

X

,|

From these equations, it can also be seen that

xfxXyfyfyYxfyxf XY ||,

This provides a way to compute the joint density function based on a conditional density function.

JointDensitytomarginaldensitycomputations.

The joint density total probability concepts can define the x and y marginal densities.

dyyxfxf X

, and dxyxfyfY

,

Then from the conditional density relationship with the joint density


We can replace the joint density functions in the total probability equations to define the pdf densities of x and y based on the conditional densities as

dxxfxXyfdxyxfyf XY

|,

or

dyyfyYxfdyyxfxf YX

|,



To derive the multiple variables Bayes Theorem, we return to


Equating the right two elements result in ….

yf

xfxXyfyYxf

Y

X

||

or

xf

yfyYxfxXyf

X

Y

||

Important Note: the joint probability density function completely specifies:

both marginal density functions and

both conditional density functions.



Independence Random Variables

Property

yfxfyxf YX ,

and

yFxFyxF YX ,

Then

xFyF

yFxF

yF

yxFyYxF X

Y

YX

Y

XYXY

,|

yFxF

yFxF

xF

yxFxXyF Y

X

YX

X

XYXY

,|

and xfyYxf XXY |

yfxXyf YXY |

Independence simplifies required computations, so pay attention to problems statements!

Usefulresultsofindependenceforjointdensityfunctions

yfxfxfxXyfyfyYxfyxf YXXY ||,

Therefore

yfxfxfxXyf YXX |

yfxXyf Y|

and

xfyYxf X|

Note: if they are independent, knowing one does not help with the other!

It only matters if x and y are correlated in some way … and not independent.



Example 2.6-9 Mapping to create a joint density function

The experiment involves rolling one die, having equally likely outcomes from 1 to 6.

6,5,4,3,2,1 and 6,,2,1,61Pr ifori

We define two new random variables,

6,5,4,2,0

1,2

3,4

for

for

for

X

6,5,4,1,0

2,1

3,2

for

for

for

Y

Then, the pmfs become

0,64

2,61

4,61

xfor

xfor

xfor

xpmf

0,64

1,61

2,61

yfor

yfor

yfor

ypmf

2,4,,61

1,4,,0

0,4,,0

2,2,,0

1,2,,0

0,2,,61

2,0,,0

1,0,,61

0,0,,63

,

yxfor

yxfor

yxfor

yxfor

yxfor

yxfor

yxfor

yxfor

yxfor

yxpmf

Note that: ypmfxpmfyxpmf ,

A 2-D plot “describing” the location of the 2D-pmf is shown in the textbook.



Example p. 126 Cooper and McGillem for independence.

1010,15

6, 2

, yandxforyxyxf YX

Computing the marginal densities:

1

0

31

0

2

35

61

5

6

y

xxdxyxyfY

31

5

6 yyfY

and

1

0

22

1

0

2

25

61

5

6

yxydyyxxf X

21

5

6 2xxf X

Note that yfxfyxf YX , , Therefore the variables are not independent!



Exercise 3-3.2 from Cooper and McGillem

Assume X and Y independent

xforxxf X ,1exp5.0

yforyyfY ,1exp5.0

Find 0Pr YX ? That is, the product of the random variables is positive.

0Pr0Pr0Pr0Pr0Pr YXYXYX

0001010Pr YxYx FFFFYX

Find the distribution for X and Y based on the ranges defined for the absolute value

x

X dxxxF 1exp5.0

For 1 xfor and xfor1

1 xfor xfor1

x

X dxxxF 1exp5.0

x

X

xxF

1

1exp5.0

1exp5.0 xxFX

x

X dxxxF1

1exp5.05.0

x

X

xxF

11

1exp5.05.0

xxFX 1exp15.05.0

1839.01exp5.000 YX FF

0338.01839.01839.000 YX FF

6660.01839.011839.010101 Yx FF

6998.00001010Pr YxYx FFFFYX



Gaussian

The Gaussian or Normal probability density function is defined as:

xforx

xf X ,2

exp2

12

2


The Gaussian Cumulative Distribution Function (CDF)

dv

vxF

x

v

X

2

2

2exp

2

1


Normal – Gaussian with zero mean and unit variance.

The Normal probability density function is defined as:

xfor

xxN ,

2exp

2

1 2

The Normal Cumulative Distribution Function (CDF)

dvv

xx

v

N

2exp

2

1 2

Note the relationship between the Gaussian and Gaussian-Normal

x

xF XX

see the MATLAB: GaussianDemo.m

-8 -6 -4 -2 0 2 4 6 80

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1Gaussian PDF and pdf



Joint Gaussian: Independent X and Y

For X and Y independent:

2

2

2

2

2exp

2

1

2exp

2

1,

X

X

XY

Y

Y

XY

xyyxf

2

2

2

2

22exp

2

1,

X

X

Y

Y

XYXY

xyyxf

If both functions have a zero mean and identical variances

2

22

2 2exp

2

1,

xy

yxf XY

see Example 2.6-10

Figure 2.6-10 Graph of the joint Gaussian density.

This has been referred to as a hat function ….



Example 2.6-11 independent Gaussians, zero mean unit variance

Rectangular to circular conversion …

22 xyr and

x

yatan

Note that for an infinitesimal area ddrrdydx

Then, for cumulative distribution function

y x

XY ddyxF 2

exp2

1,

22

We could consider a change to circular area as

r

R ddrF0

2

0

2

2exp

2

1,

r

R ddrF0

2

0

2

2

1

2exp,

r

R drF0

2

2exp

12

exp2

exp2

0

2

rrF

r

R

2exp1

2rrFR

And the probability density function is

2exp

2rrrf R

with

20,2

1f

Also they are they are independent …

randrr

rf R

020,

2exp

2,

2



JointGaussian:NotIndependent

Assume that X and Y are not independent random variables and have equal variances. :

22

22

22 12

2exp

12

1,

xyxyyxf XY

Note: is a correlation coefficient between the two random variables. The description of correlation coefficients is coming in Chap. 4. They are rather important.

Note that if = 0, X and Y are uncorrelated.



Example 2.6-12

Consider the joint pdf:

.

otherwise

yxyxAyxf XY

,0

10,10,,

(i) Compute A.

Performing the double integral

1

0

1

0

1

0

1

0

,1 dydxyxAdydxyxf XY

1

0

21

0

1

0

2

12

1

21 dyyAdyyx

xA

AAy

yA

2

11

2

1

22

11

21

0

2

Therefore

otherwise

yxyxyxf XY

,0

10,10,1,

(ii) Compute the marginal density functions for x and y

10,1,1

0

1

0

ydxyxdxyxfyf XYY

10,2

1

2

1

0

2

yyyx

xyfY

Similarly

10,2

1 xxxf X



(iii) Compute the Joint CDF

y xy x

XYXY dvduvudvduvufyxF0 00 0

1,,

yy x

XY dvvxx

dvvuu

yxF0

2

0 0

2

22,

2222,

22

0

22 yxy

xvxv

xyxF

y

XY

22

2

1, yxyxyxFXY

Now we are computing the “regions” of Figure 2.6-9

Figure 2.6-9 Shaded region in (a) to (e) is the intersection of supp(fXY) with the point set associated with the event {−∞ < X ≤ x,−∞ < Y ≤ y}. In (f), the shaded region is the intersection of supp(fXY) with {X + Y ≤ 1}.



(a) Compute yx 1,1Pr

1,11,1Pr XYFyx

111112

11,1 22 XYF

(b) Compute yx 1,10Pr

1,1,10Pr xFyx XY

xxxxxFXY 222

2

111

2

11,

(c) Compute 10,1Pr yx

yFyx XY ,1,10,1Pr

222

2

111

2

1,1 yyyyyFXY

(d) Compute 10,10Pr yx

22

2

1,10,10Pr yxyxyxFyx XY

(e) Compute 10,0Pr yx

0002

1

,010,0Pr

22

yy

yFyx XY

Similarly Compute 0,10Pr yx

0002

1

0,0,10Pr

22

xx

xFyx XY



(e) Compute 1Pr YX

1

,1Pryx

XY dydxyxfYX

If we let x drive the function,

x goes from 0 to 1

Meanwhile, y goes from 0 to 1-x for each value of x

This allows us to perform the double integral as shown below!

1

0

1

0

1

0

1

0

1,1Pr dxdyyxdxdyyxfYXxx

XY

1

0

21

0

1

0

2

2

11

21Pr dx

xxxdx

yyxYX

x

1

0

21

0

22

22

1

2

211Pr dx

xdx

xxxxYX

3

1

6

2

6

1

2

1

621Pr

31

0

3

xxYX



Failure Rates

Simple approximation for failure rates: Use the exponential density function!

0,exp

0,0

tt

ttfT

0,exp1

0,0

tt

ttFT

In these equations, lambda, , is the failure rate and t is time. The failure rate is assumed to be the same constant for the lifetime of the product!

Homework Problem 2.38. A laser used to scan bar codes in a store is assumed to have a constant failure rate lambda, . What is the maximum value of lambda that will yield a probability of a first breakdown in 100 hours of operation less than or equal to 0.05?

Solution: Using the exponential CDF and pdf, the probability of a failure in time t can be defined using the CDF as …

ttFT exp1 We are given

05.01000 tP with

TTFTtP T exp10 Therefore,

05.0100exp1100 TF

100exp05.01

10005.01ln

100

95.0ln

-4105.1293



Author’s Solution

Backtoamorecompletediscussionoffailurerates

Following the author’s detailed derivations …

Let X denote the time of failure. Then by Bayes’ theorem, the probability that failures occur in the time interval {t, t+dt} given that the object has survived up to time t can be written as

tXP

XtdttXtPtXdttXtP

,

|

The joint probability function contains two events, where the first event is completed contained in the second event. Therefore, the intersection of the two events produces the first event! That is

tXP

dttXtPtXdttXtP

|

If there is a CDF function that described the probability, FX, the right hand functions can be easily defined in terms of the CDF as …

tF

tFdttFtXdttXtP

x

xx

1

|



Using the “engineering derivative” definition

tF

dttf

tF

dtdttFdttF

tXdttXtPx

x

x

xx

11

|

We now want to call the resulting function “the conditional failure rate”, (t).

dtttF

dttftXdttXtP

x

x

1|

Now, the conditional failure rate described the failures expected in a time interval based on the length of time the product has been in service:

This rate can …

be constant be time varying be linear or non-linear be defined any way it has too!

In many cases for failure analysis, there is a common shape for the failure rate.

“The Bathtub curve” – often product failures rates follow a curve that looks like a bathtub … a lot of initial failures (infant mortality),

… very few during the “lifetime” (constant random failures) and

… an increasing number once past the expected lifetime (things eventually wear out).

see: https://en.wikipedia.org/wiki/Bathtub_curve

also see Failure Rate at https://en.wikipedia.org/wiki/Failure_rate



Determining classes of solutions ….

Based on the previous derivation

dtttF

dttftXdttXtP

x

x

1|

We can described the conditional failure rate to CDF as (again “engineering math”

dtttF

tdF

tF

dttf

x

x

x

x

11

If we assume that the CDF is a function, y, … and “integrate”

1

0

1

01

t

t

y

y

dtty

dy

We can integrate using

0

101 lnlnln

1

0y

yyy

y

dyy

y

1

0

1

0

1

0

1ln1

t

t

y

y

y

y

dttyy

dy

1

0

01 1ln1lnt

t

dttyy

1

01

0

1

1ln

t

t

dtty

y

If we relate the function, y, to the CDF and start at time t=0 …

t

X

X dtttF

F

01

01ln

We know that for a failure CDF, 00 XF and 1XF . Therefore,

t

X

dtttF 01

1ln or we have

t

X dtttF0

1ln



Leading to

t

X dtttF0

1ln

t

X dtttF0

exp1

and finally

t

X dtttF0

exp1

Finding the pdf ….

t

XX dttttf

dt

tFd

0

exp

Notice for a constant conditional failure rate

t

tdttFt

X

exp1exp1

0

ttf X exp

The exponential CDF!



Matlab – Examples and Concepts

rand.m

randn.m

Histogram of a probability density function

Generating random numbers from an arbitrary cumulative distribution function.



Uniform Density Example

The uniform density function in two dimensions can be defined as:

else

yandxforyxf YX

,02

1

2

1

2

1

2

1,1

11

1,,

Determine the density in y

2

1

,,

x

x

YXY dxyxfyf

121

211 2

1

21

21

21

xdxyfY

22

1

2

1,1 yforyfY

Similarly

2

1

2

1,1 xforxf X



Distribution

y x

YX dydxyxfyxF ,,,

2

1

2

11,

21

21

, yxdydxyxFy x

YX

Detailed distribution in the entire x,y-plane

yandxfor

yandxfory

yandxforx

yandxforyx

yandxfor

yxF YX

2

1

2

1,1

2

1

2

1

2

1,

2

1

2

1

2

1

2

1,

2

1

2

1

2

1

2

1

2

1,

2

1

2

12

1

2

1,0

,,

chapter 2: introduction to probabilityhomepages.wmich.edu/~bazuinb/ece3800sw/sw_notes02.pdf ·...

Documents