continuous random variables: joint pdfs, conditioning...
TRANSCRIPT
Berlin ChenDepartment of Computer Science & Information Engineering
National Taiwan Normal University
Continuous Random Variables: Joint PDFs, Conditioning, Expectation and Independence
Reference:- D. P. Bertsekas, J. N. Tsitsiklis, Introduction to Probability , Sections 3.4-3.6
Probability-Berlin Chen 2
Multiple Continuous Random Variables (1/2)
• Two continuous random variables and associated with a common experiment are jointly continuous and can be described in terms of a joint PDF satisfying
– is a nonnegative function
– Normalization Probability
• Similarly, can be viewed as the “probability per unit area” in the vicinity of
– Where is a small positive number
X Y
Byx
YX dxdyyxfBYX,
, ,,P
YXf ,
1,, dxdyyxf YX
YXf ,
caf YX ,, ca,
2
,, ,,
,
cafdxdyyxf
cYcaXa
YXaa
cc YX
P
Probability-Berlin Chen 3
Multiple Continuous Random Variables (2/2)
• Marginal Probability
– We have already defined that
• We thus have the marginal PDF
Similarly
AX YX dydxyxf
YAXAX
,
, and
,
PP
AX X dxxfAXP
dyyxfxf YXX ,,
dxyxfyf YXY ,,
Probability-Berlin Chen 4
An Illustrative Example
• Example 3.10. Two-Dimensional Uniform PDF. We are told that the joint PDF of the random variables and is a constant on an area and is zero outside. Find the value of and the marginal PDFs of and .
X YS
cc
X Y
Sx,yx,yf
Sx,yx,yf
S
YX
YX
for 41
otherwise ,0
if ,S area of Size
13.9) Example (cf. be todefined is areaan on PDFjoint uniform ingcorrespond The
,
,
31
31
21
21
21
41
32for 41
41
21for
dx
dxx,yfyf
y
dx
dxx,yfyf
y
X,YY
X,YY
32
32
41
41
41
41
32for 43
41
21
dy
dyx,yfxf
x
dy
dyx,yfxf
xfor
X,YX
X,YX
21
21
41
41
43for
dx
dxx,yfyf
y
X,YY
Probability-Berlin Chen 5
Joint CDFs• If and are two (either continuous or discrete)
random variables associated with the same experiment , their joint cumulative distribution function (Joint CDF) is defined by
– If and further have a joint PDF ( and are continuous random variables) , then
And
If can be differentiated at the point
X Y
yYxXyxF YX ,,, P
X Y YXf ,
yx
yxFyxf YX
YX
,, ,
2
,
x y
YXYX dsdttsfyxF ,, ,,
YXF , yx,
X Y
Probability-Berlin Chen 6
An Illustrative Example
• Example 3.12. Verify that if X and Y are described by a uniform PDF on the unit square, then the joint CDF is given by
10for ,,,, x,yxyyYxXyxF YX P
squareunit in the , allfor ,,1,
,,
2yxyxf
yxyxF
YXYX
X
Y 1,1
0,0 0,1
1,0
Probability-Berlin Chen 7
Expectation of a Function of Random Variables
• If and are jointly continuous random variables, and is some function, then is also a random variable (can be continuous or discrete)– The expectation of can be calculated by
– If is a linear function of and , e.g., , then
• Where and are scalars
X Yg YXgZ ,
Z
dxdyyxfyxgYXgZ YX ,,, ,EE
Z X Y bYaXZ
YbXabYaXZ EEEE
a b
We will see in Section 4.1 methods for computing the PDF of (if it has one).Z
More than Two Random Variables
• The joint PDF of three random variables , and is defined in analogy with the case of two random variables
– The corresponding marginal probabilities
• The expected value rule takes the form
– If is linear (of the form ), then
Probability-Berlin Chen 8
X Y Z
BZYX
ZYX dxdydzzyxfBZYX,,
,, ,,,,P
dydzzyxfxf
dzzyxfyxf
ZYXX
ZYXYX
,,
,,,
,,
,,,
g
dzdxdyzyxfzyxgZYXg ZYX ,,,,,, ,,E
ZcYbXacZbYaX EEEE
cZbYaX
Probability-Berlin Chen 9
Conditioning PDF Given an Event (1/3)
• The conditional PDF of a continuous random variable , given an event– If cannot be described in terms of , the conditional PDF
is defined as a nonnegative function satisfying
• Normalization property
XA
XA
dxxfABX B AXP
xf AX
1 dxxf AX
Probability-Berlin Chen 10
Conditioning PDF Given an Event (2/3)
– If can be described in terms of ( is a subset of the real line with ), the conditional PDF is defined as a nonnegative function satisfying
• The conditional PDF is zero outside the conditioning event
and for any subset
– Normalization Property
A xf AX
X A 0 AXP
otherwise ,0
if , AxAX
xfxf
X
AX P
B
BA AX
BA X
dxxfAX
dxxfAX
AXBXAXBX
,
P
PPP
1 A AXAX dxxfdxxf
remains the same shape as except that it is scaled along
the vertical axis
AXf
Xf
Probability-Berlin Chen 11
Conditioning PDF Given an Event (3/3)
• If are disjoint events with for each , that form a partition of the sample space, then
– Verification of the above total probability theorem
nAAA ,,, 21 0iAPi
n
iAXiX xfAxf
i1P
n
iAXiX
n
i
xAXi
xX
n
iii
xfAxf
x
dttfAdttf
AxXAxX
i
i
1
1
1
respect to with sidesboth of derivative theTaking
P
P
PPP
think of as an event , and use the total probability theorem from Chapter 1
xX B
Probability-Berlin Chen 12
Illustrative Examples (1/2)
• Example 3.13. The exponential random variable is memoryless.– The time T until a new light bulb burns out is exponential
distribution. John turns the light on, leave the room, and when he returns, t time units later, find that the light bulb is still on, which corresponds to the event A={T>t}
– Let X be the additional time until the light bulb burns out. What is the conditional PDF of X given A ?
tetTP
tetf
Tt
T
otherwise ,0
0,lexponentia is
x
t
xt
ee
etTP
xtTPtTP
tTxtTPtTxtTP
xtTxtTPAxXPAX
tTAtTX
and )0 (where by defined is given of CDF lconditiona The
,
.parameter with lexponentia also is event the
given of PDF lconditiona The
A
X
Probability-Berlin Chen 13
Illustrative Examples (2/2)
• Example 3.14. The metro train arrives at the station near your home every quarter hour starting at 6:00 AM. You walk into the station every morning between 7:10 and 7:30 AM, with the time in this interval being a uniform random variable. What is the PDF of the time you have to wait for the first train to arrive?
yPBPyPAPyP BYAYY
BYAY
XBB
XAA
Y
X
on dconditione uniform be Let -on dconditione uniform be Let -
train)30:7 theboard(You 30:715:7 event a be Let -
train)15:7 theboard(You 15:710:7 event a be Let -
time waiting themodel variblerandomLet -30:7 to10:7 interval over the variable
random uniform a is ,by denoted time,arrival The -
1/20
201
151
430
41 ,155For
101
151
43
51
41 ,50For
yPy
yPy
Y
Y
Total Probability theorem:
Probability-Berlin Chen 14
Conditioning one Random Variable on Another
• Two continuous random variables and have a joint PDF. For any with , the conditional PDF of given that is defined by
– Normalization Property
• The marginal, joint and conditional PDFs are related to each other by the following formulas
,,, yxfyfyxf YXYYX
.,, dyyxfxf YXX
X YXy 0yfY
yY
yf
yxfyxf
Y
YXYX
,,
1 dxyxf YX
marginalization
Probability-Berlin Chen 15
Illustrative Examples (1/2)
• Notice that the conditional PDF has the same shape as the joint PDF , because the normalizing factor does not depend on
Figure 3.16: Visualization of the conditional PDF . Let , have a joint PDF which is uniform on the set . For each fixed , we consider the joint PDF along the slice and normalize it so that it integrates to 1
yxf YX yxf YX ,,
yfY x
yxf YX
Xy
SYyY
1
4/14/1
5.35.3,
5.3 , Y
YXYX f
xfxf
1
4/14/1
5.15.1,
5.3 , Y
YXYX f
xfxf
2/1
2/14/1
5.25.2,
5.3 , Y
YXYX f
xfxf
cf. example 3.13
Probability-Berlin Chen 16
Illustrative Examples (2/2)
• Example 3.15. Circular Uniform PDF. Ben throws a dart at a circular target of radius . We assume that he always hits the target, and that all points of impact are equally likely, so that the joint PDF of the random variables and is uniform– What is the marginal PDF
yxf YX ,, yfY
otherwise ,0
,1
otherwise ,0
circle in the is ,if ,circle theof area
1,
2222
,
ryxπr
yxyxf YX
yx yx ,
r
uniform)not is PDF that here (Notice
2
1111
1
222
22
2
22
22222
222
yf
r y, if yrπr
dxπr
dxπr
dxπr
dxx,yfyf
Y
yryrryx
ryxX,YY
22222
222
2
2
1
2
1
ry if x, yr
yrπr
πr
yfx,yf
yxfY
X,YYX
For each value , is uniformy yxf YX
Probability-Berlin Chen 17
Conditional Expectation Given an Event
• The conditional expectation of a continuous random variable , given an event ( ), is defined by
– The conditional expectation of a function also has the form
– Total Expectation Theorem
and
• Where are disjoint events with for each , that form a partition of the sample space
X A
dxxxfAX AXE
Xg
dxxfxgAXg AXE
n
iii AXAX
1EPE
0AP
nAAA ,,, 21 0iAP
i
n
iii AXgAXg
1EPE
Probability-Berlin Chen 18
An Illustrative Example• Example 3.17. Mean and Variance of a Piecewise Constant PDF.
Suppose that the random variable has the piecewise constant PDF
X
otherwise. ,0
,21 if ,3/2,10 if ,3/1
xx
xf X
3/23/2 ,3/13/1
[1,2] interval second in the lies event [0,1] intervalfirst in the lies event Define
212
101
2
1
dxAdxA
XAXA
PP
3 and 2 is ][
intervalan over variablerandom uniform a ofmoment second andmean that theRecall
22 /baba/baa, b
36/116/79/15var
9/153/73/23/13/1 EEE
6/72/33/22/13/1 EEE
2
22
212
12
2211
X
AXAAXAX
AXAAXAX
PP
PP
3/7E,2/3E
3/1E,2/1E
22
2
12
1
AXAX
AXAX
otherwise ,0
2 1 ,1
otherwise ,0
1 0 ,121 21
xAXP
xfxf
xAXP
xfxf
X
AX
X
AX
Probability-Berlin Chen 19
Conditional Expectation Given a Random Variable
• The properties of unconditional expectation carry though, with the obvious modifications, to conditional expectation
dxyxxfyYX YXE
dxyxfxgyYXg YXE
dxyxfyxgyYYXg YX,,E
Probability-Berlin Chen 20
Total Probability/Expectation Theorems
• Total Probability Theorem– For any event and a continuous random variable
• Total Expectation Theorem– For any continuous random variables and
A
dyyfyYAA YPP
YX
Y
dyyfyYXX YEE
dyyfyYXgXg YEE
dyyfyYYXgYXg Y,, EE
Probability-Berlin Chen 21
Independence
• Two continuous random variables and are independent if
– Since that
• We therefore have
• Or
X Y
x,yyfxfyxf YXYX allfor ,,,
0 with all and allfor , yfyxxfyxf YXYX
0 with all and allfor , xfxyyfxyf XYXY
xyfxfyxfyfyxf XYXYXYYX ,,
Probability-Berlin Chen 22
More Factors about Independence (1/2)
• If two continuous random variables and are independent, then– Any two events of the forms and are
independent
– It also implies that
– The converse statement is also true (See the end-of-chapter problem 28)
X Y
AX BY
BYAX
dyyfdxxf
dydxyfxf
dydxyxfBYAX
By YAx X
Ax By YX
Ax By YX
PP
P
,, ,
xFxFyYxXyYxXyxF YXYX PPP ,,,
Probability-Berlin Chen 23
More Factors about Independence (2/2)
• If two continuous random variables and are independent, then–
–
– The random variables and are independent for any functions and
• Therefore,
YXXY EEE
Xg Yhg h
YhXgYhXg EEE
YXYX varvarvar
X Y
Probability-Berlin Chen 24
Recall: the Discrete Bayes’ Rule
• Let be disjoint events that form a partition of the sample space, and assume that , for all . Then, for any event such that we have
nAAA ,,, 21
0iAP iB 0BP
nn
ii
nk kk
ii
iii
ABAABAABA
ABAABA
BABA
BA
PPPPPP
PPPP
PPP
P
11
1
Multiplication ruleTotal probability theorem
Probability-Berlin Chen 25
Inference and the Continuous Bayes’ Rule
• As we have a model of an underlying but unobserved phenomenon, represented by a random variable with PDF , and we make a noisy measurement , which is modeled in terms of a conditional PDF . Once the experimental value of is measured, what information does this provide on the unknown value of ?
XYXf
XYfY
X
Measurement InferenceX
xf X
Y
xyf XY yxf YX
dttyftf
xyfxf
yfyxf
yxfXYX
XYX
Y
YXYX
,,
YXYYXXYX fffff , thatNote
Probability-Berlin Chen 26
Inference and the Continuous Bayes’ Rule (2/2)
• If the unobserved phenomenon is inherently discrete– Let is a discrete random variable of the form that
represents the different discrete probabilities for the unobserved phenomenon of interest, and be the PMF of
N nN
Np N
iNYN
NYN
Y
NYN
iyfip
nyfnpyf
nyfnpyYy
nNyYynN
yYynNyYnN
PPP
PP
Total probability theorem
Inference about a Discrete Random Variable
Probability-Berlin Chen 27
Illustrative Examples (1/2)• Example 3.19. A lightbulb produced by the General Illumination
Company is known to have an exponentially distributed lifetime . However, the company has been experiencing quality control problems. On any given day, the parameter of the PDF of is actually a random variable, uniformly distributed in the interval
. – If we test a lightbulb and record its lifetime ( ), what can
we say about the underlying parameter ?
Y
Y
yY
0,0 , yeyf y
Y
otherwise 0,2/31for ,2
f
231for ,
222/3
12/3
1
/λdtte
edttyftf
yffyf ty
y
Y
YY
2/3 ,1
Conditioned on , has a exponential distribution with parameter
Y
Probability-Berlin Chen 28
Illustrative Examples (2/2)
• Example 3.20. Signal Detection. A binary signal is transmitted, and we are given that and . – The received signal is , where is a normal noise
with zero mean and unit variance , independent of .– What is the probability that , as a function of the observed value
of ?
pS 1PS
pS 11PNSY N
S
y--sesyf sySY and ,1 and 1for ,
21 2/2
Conditioned on , has a normal distribution with mean and unit varianceYsS s
1SYy
yy
y
yyyy
yy
yy
y
SYSSYS
SYS
Y
SYS
eppepe
epepee
pee
epep
ep
yfpyfp
yfp
yf
yfpyYS
11
211
21
21
1111
11111
2/12/1
2/1
2/12/1
2/1
22
2
22
2
P
Inference Based on a Discrete Random Variable
• The earlier formula expressing in terms of can be turned around to yield
Probability-Berlin Chen 29
yYA P yf AY
dttYAtf
yYAyfA
yYAyfyf
Y
Y
YAY
P
PPP
?
)1:propertyion normalizat(
dyyfdyyYAyfA
dyyYAyfdyyfA
yYAyfyfA
AYY
YAY
YAY
PP
PP
PP
Probability-Berlin Chen 30
Recitation
• SECTION 3.4 Joint PDFs of Multiple Random Variables– Problems 15, 16
• SECTION 3.5 Conditioning– Problems 18, 20, 23, 24
• SECTION 3.6 The Continuous Bayes’ Rule– Problems 34, 35