problemsolving - university of illinois at...
TRANSCRIPT
Problem Solving
ECE 313
April 30, 2017
Problem Solving
Correlation and Covariance
ECE 313 | Section B 2/51
Problem Solving
Definition
Let X and Y be random variables with finite second moments.
the correlation: E[XY ]
ECE 313 | Section B 3/51
Problem Solving
Definition
Let X and Y be random variables with finite second moments.
the correlation: E[XY ]the covariance: Cov(X,Y ) = E[(X − E[X])(Y − E[Y ])]
ECE 313 | Section B 4/51
Problem Solving
Definition
Let X and Y be random variables with finite second moments.
the correlation: E[XY ]the covariance: Cov(X,Y ) = E[(X − E[X])(Y − E[Y ])]
the correlation coefficient: ρX,Y = Cov(X,Y )√Var(X)Var(Y )
= Cov(X,Y )σXσY
.
ECE 313 | Section B 5/51
Problem Solving
Covariance generalizes variance
Var(X) = Cov(X,X).
ECE 313 | Section B 6/51
Problem Solving
Covariance generalizes variance
Var(X) = Cov(X,X).
Shortcut for computing variance:
Var(X) = E[X(X − E[X])] = E[X2]− E[X]2.
ECE 313 | Section B 7/51
Problem Solving
Covariance generalizes variance
Var(X) = Cov(X,X).
Shortcut for computing variance:
Var(X) = E[X(X − E[X])] = E[X2]− E[X]2.
Similar shortcuts exist for computing covariances:
Cov(X,Y ) = E[X(Y−E[Y ])] = E[(X−E[X])Y ] = E[XY ]−E[X]E[Y ].
In particular, if either X or Y has mean zero, then E[XY ] = Cov(X,Y ).
ECE 313 | Section B 8/51
Problem Solving
Correlated
If Cov(X,Y ) = 0, X and Y are uncorrelated.
If Cov(X,Y ) > 0, X and Y are positively correlated.
If Cov(X,Y ) < 0, X and Y are negatively correlated.
ρX,Y is a scaled version of Cov(X,Y ).
ECE 313 | Section B 9/51
Problem Solving
Uncorrelated vs. Independent
If Cov(X,Y ) = 0, X and Y are uncorrelated.
Cov(X,Y ) = 0 is equivalent to E[XY ] = E[X]E[Y ].
ECE 313 | Section B 10/51
Problem Solving
Uncorrelated vs. Independent
If Cov(X,Y ) = 0, X and Y are uncorrelated.
Cov(X,Y ) = 0 is equivalent to E[XY ] = E[X]E[Y ].
Does independence of X and Y imply E[XY ] = E[X]E[Y ]?
ECE 313 | Section B 11/51
Problem Solving
Uncorrelated vs. Independent
If Cov(X,Y ) = 0, X and Y are uncorrelated.
Cov(X,Y ) = 0 is equivalent to E[XY ] = E[X]E[Y ].
Does independence of X and Y imply E[XY ] = E[X]E[Y ]?
Yes, independence V uncorrelated.
ECE 313 | Section B 12/51
Problem Solving
Uncorrelated vs. Independent
Does uncorrelated V independence?
ECE 313 | Section B 13/51
Problem Solving
Uncorrelated vs. Independent
Does uncorrelated V independence?
No, independence is a much stronger condition.
Independence requires a larger number of equations to hold, namelyFXY (u, v) = FX(u)FY (v) for every real value of u and v. Uncorrelatedonly requires a single equation to hold.
ECE 313 | Section B 14/51
Problem Solving
Uncorrelated vs. Independent
Correlation is a pairwise property.
A set of random variables being uncorrelated is the same as being pairwiseuncorrelated.
ECE 313 | Section B 15/51
Problem Solving
Uncorrelated vs. Independent
Correlation is a pairwise property.
A set of random variables being uncorrelated is the same as being pairwiseuncorrelated.
In contrast, mutual independence of 3 or more random variables is astronger property than pairwise independence.
ECE 313 | Section B 16/51
Problem Solving
Play with Covariance
ECE 313 | Section B 17/51
Problem Solving
Linearity
Covariance is linear in each of its two arguments:
Cov(X + Y, U + V ) = Cov(X,U) + Cov(X,V ) + Cov(Y,U) + Cov(Y, V )Cov(aX + b, cY + d) = acCov(X,Y ).
for constants a, b, c, d.
Recall thatVar(aX + b) = a2Var(X).
ECE 313 | Section B 18/51
Problem Solving
Variance of sum of r.v.
The variance of the sum of uncorrelated random variables is equal to thesum of the variances of the random variables. For example, if X and Yare uncorrelated,
Var(X + Y ) = Var(X) + Var(Y ),
ECE 313 | Section B 19/51
Problem Solving
Variance of sum of r.v.
The variance of the sum of uncorrelated random variables is equal to thesum of the variances of the random variables. For example, if X and Yare uncorrelated,
Var(X+Y ) = Cov(X+Y,X+Y ) = Cov(X,X)+Cov(Y, Y )+2Cov(X,Y )
= Var(X) + Var(Y ).
ECE 313 | Section B 20/51
Problem Solving
Variance of sum of r.v.
Consider the sum Sn = X1 + · · ·+Xn, such that X1, · · · , Xn are uncor-related (so Cov(Xi, Xj) = 0 if i 6= j) with E[Xi] = µ and Var(Xi) = σ2
for 1 ≤ i ≤ n.
Find E(Sn) and Var(Sn).
ECE 313 | Section B 21/51
Problem Solving
Variance of sum of r.v.
E[Sn] = nµ (1)
ECE 313 | Section B 22/51
Problem Solving
and
Var(Sn) = Cov(Sn, Sn) = Cov
n∑i=1
Xi,n∑j=1
Xj
=
n∑i=1
n∑j=1
Cov(Xi, Xj)
=n∑i=1
Cov(Xi, Xi) +∑i,j:i 6=j
Cov(Xi, Xj)
=n∑i=1
Var(Xi) + 0 = nσ2. (2)
ECE 313 | Section B 23/51
Problem Solving
Practice!
Simplify the following expressions:(a) Cov(8X+3, 5Y −2), (b) Cov(10X−5,−3X+15), (c) Cov(X+2,10X-3Y), (d) ρ10X,Y+4.
ECE 313 | Section B 24/51
Problem Solving
Correlation coefficient
ECE 313 | Section B 25/51
Problem Solving
Correlation coefficient
ρX,Y = Cov(X,Y )√Var(X)Var(Y )
= Cov(X,Y )σXσY
.
ρX,Y is a scaled version of Cov(X,Y ).
The situation is similar to the standardized version of X and Y .
FindCov
(X − E[X]
σX,Y − E[Y ]
σY
).
ECE 313 | Section B 26/51
Problem Solving
Correlation coefficient
Cov(X − E[X]
σX,Y − E[Y ]
σY
)= Cov
(X
σX,Y
σY
)= Cov(X,Y )
σXσY= ρX,Y .
ECE 313 | Section B 27/51
Problem Solving
Correlation coefficient
FindρaX+b,cY+d
ECE 313 | Section B 28/51
Problem Solving
Correlation coefficient
ρaX+b,cY+d = ρX,Y for a, c > 0.
ECE 313 | Section B 29/51
Problem Solving
Correlation coefficient
• |ρX,Y | ≤ 1,
• ρX,Y = 1 if and only if Y = aX + b for some a, b with a > 0, and
• ρX,Y = −1 if and only if Y = aX + b for some a, b with a < 0.
ECE 313 | Section B 30/51
Problem Solving
Unbiased Estimator
Suppose X1, . . . , Xn are independent and identically distributed randomvariables, with mean µ and variance σ2. We estimate µ and σ2 by thesample mean and sample variancedefined as follows:
X = 1n
n∑k=1
Xk σ2 = 1n− 1
n∑k=1
(Xk − X)2.
Note the perhaps unexpected appearance of n− 1 in the sample variance.Of course, we should have n ≥ 2 to estimate the variance (assuming wedon’t know the mean) so it is not surprising that the formula is not definedif n = 1. An estimator is called unbiased if the mean of the estimator isequal to the parameter that is being estimated.
ECE 313 | Section B 31/51
Problem Solving
Q1: Why don’t we use ML parameter estimation for µ and σ2?Q2: Why is σ2 undefined for n = 1?
(a) Is the sample mean an unbiased estimator of µ?(b) Find the mean square error, E[(µ− X)2], for estimation of the meanby the sample mean.(c) Is the sample variance an unbiased estimator of σ2?
ECE 313 | Section B 32/51
Problem Solving
Minimum mean square error estimation
ECE 313 | Section B 33/51
Problem Solving
Constant estimator
Let Y be a random variable with some known distribution. Suppose Y isnot observed but that we wish to estimate Y .
• We use a constant δ to estimate Y .
ECE 313 | Section B 34/51
Problem Solving
Constant estimator
Let Y be a random variable with some known distribution. Suppose Y isnot observed but that we wish to estimate Y .
• We use a constant δ to estimate Y .
• The mean square error (MSE) for estimating Y by δ is defined by
E[(Y − δ)2].
Q. How do we find a δ that minimizes E[(Y − δ)2]? What is the resultingMSE?
ECE 313 | Section B 35/51
Problem Solving
Unconstrained Estimator
We want to estimate Y . We have an observation X. The joint distributionis fX,Y .
• We use the estimator g(X) for some function g.
• The resulting mean square error (MSE) is E[(Y − g(X))2]
Q. What is the function g that minimizes the MSE? What is the resultingMSE?
ECE 313 | Section B 36/51
Problem Solving
Unconstrained Estimator
Suppose you observe X = 10. What do you know about Y ?
ECE 313 | Section B 37/51
Problem Solving
Unconstrained Estimator
Suppose you observe X = 10. What do you know about Y ?
You can derive the conditional pdf of Y given X = 10, denoted byfY |X(v|10).
Which value of Y should you pick?
ECE 313 | Section B 38/51
Problem Solving
Unconstrained EstimatorSuppose you observe X = 10. What do you know about Y ?
You can derive the conditional pdf of Y given X = 10, denoted byfY |X(v|10).
Which value of Y should you pick?
Based on the fact, discussed above, that the minimum MSE constantestimator for a random variable is its mean, it makes sense to estimate Yby the conditional mean:
E[Y |X = 10] =∫ ∞−∞
vfY |X(v|10)dv.
ECE 313 | Section B 39/51
Problem Solving
Unconstrained Estimator
In general, we can show that
g∗(u) = E[Y |X = u] =∫ ∞−∞
vfY |X(v|u)dv.
The minimum MSE is
MSE = E[Y 2]− E[(E[Y |X])2], (3)= Var(Y )− Var(Y |X) (4)
ECE 313 | Section B 40/51
Problem Solving
Linear estimator
• In practice it is not always possible to compute g∗(u).
• The conditional density fY |X(v|u) may not be available or might bedifficult to compute.
• Worse, there might not even be a good way to decide what jointpdf fX,Y to use in the first place.
A reasonable alternative is to consider linear estimators of Y given X.
ECE 313 | Section B 41/51
Problem Solving
Linear estimator
• We use a linear estimator L(X) = aX + b. We only need to find aand b.
• The resulting mean square error (MSE) is E[(Y − (aX + b))2].
Q. What are the a and b that minimize the MSE? What is the resultingMSE?
ECE 313 | Section B 42/51
Problem Solving
Linear estimator
The minimum MSE linear estimator is given by L∗(X) = E[Y |X], where
E[Y |X] = µY +(Cov(Y,X)
Var(X)
)(X − µX)
= µY + σY ρX,Y
(X − µXσX
).
minimum MSE for linear estimation = σ2Y−
(Cov(X,Y ))2
Var(X) = σ2Y (1−ρ2
X,Y ).
ECE 313 | Section B 43/51
Problem Solving
If X and Y are standard, then E[Y |X] = ρX,YX and the MSE is 1−ρ2X,Y .
ECE 313 | Section B 44/51
Problem Solving
Three estimators
• Constant
• Unconstrained
• Linear
Which is the best?
ECE 313 | Section B 45/51
Problem Solving
Three estimators
Constant (estimator) ⊂ Linear ⊂ Unconstrained
E[(Y − g∗(X))2]︸ ︷︷ ︸MSE for g∗(X)=E[Y |X]
≤ σ2Y (1− ρ2
X,Y )︸ ︷︷ ︸MSE for L∗(X)=E[Y |X]
≤ σ2Y︸︷︷︸
MSE for δ∗=E[Y ].
(5)
ECE 313 | Section B 46/51
Problem Solving
Three estimators
All three estimators are linear as functions of the variable to be estimated:
E[aY + bZ + c] = aE[Y ] + bE[Z] + c
E[aY + bZ + c|X] = aE[Y |X] + bE[Z|X] + c
E[aY + bZ + c|X] = aE[Y |X] + bE[Z|X] + c
ECE 313 | Section B 47/51
Problem Solving
Noisy observation
Let X = Y +N , where Y has the exponential distribution with parameterλ, and N is Gaussian with mean 0 and variance σ2
N . Suppose the variablesY and N are independent, and the parameters λ and σ2
N are known andstrictly positive. (Recall that E[Y ] = 1
λ and Var(Y ) = σ2Y = 1
λ2 .)(a) Find E[Y |X], the MSE linear estimator of Y given X, and also findthe resulting MSE.(b) Find an unconstrained estimator of Y yielding a strictly smaller MSEthan E[Y |X] does.
ECE 313 | Section B 48/51
Problem Solving
Uniform distribution
Suppose (X,Y ) is uniformly distributed over the triangular region withvertices at (−1, 0), (0, 1), and (1, 1), shown in Figure 1. (a) Find and sketch
1
u
v
!1 10
Figure 1: Support of fX,Y .
ECE 313 | Section B 49/51
Problem Solving
the minimum MSE estimator of Y given X = u, g∗(u) = E[Y |X = u], forall u such that it is well defined, and find the resulting minimum MSE forusing g∗(X) = E[Y |X] to estimate Y. (b) Find and sketch the functionE[Y |X = u], used for minimum MSE linear estimation of Y from X, andfind the resulting MSE for using E[Y |X] to estimate Y.
ECE 313 | Section B 50/51
Questions?