introduction to estimation theory part
TRANSCRIPT
Copyright 2011 Aaron Lanterman
Introduction to Estimation Theory Part I
Prof. Aaron D. Lanterman School of Electrical & Computer Engineering
Georgia Institute of Technology AL: 404-385-2548
ECE 6279: Spatial Array Processing Spring 2011 Lecture 24
Copyright 2011 Aaron Lanterman
References
• J&D: Section 6.2.4 • Van Trees – “Detection Estimation, and
Modulation Theory: Part I” (many parts out of date, but still the best)
• Vince Poor – “An Introduction to Signal Detection and Estimation” (insanely mathematical)
• Stephen Kay – “Fundamentals of Statistical Signal Processing” – two volumes (a bit sprawling)
Copyright 2011 Aaron Lanterman
Setup
• Model measured data as a realization of a random variable
• Assume has a density that is a function of desired parameter(s)
�
y
�
y
�
y
�
pξ (y)
�
ξ
�
p(y |ξ)
�
p(y;ξ)
Vince Poor: J&D, Van Trees: Aaron’s ECE7251:
(misleading)
Copyright 2011 Aaron Lanterman
Canonical Example: Simple Gaussian
• Example: i.i.d. samples of a real scalar Gaussian
�
p(y;ξ) =12πσ 2
l=0
L−1
∏ exp −[y(l) − µ]2
2σ 2
⎧ ⎨ ⎩
⎫ ⎬ ⎭
�
ξ = (µ,σ 2)
�
y = {y(0),...,y(L −1)}
Copyright 2011 Aaron Lanterman
Big Picture
�
ξ
�
p(y;ξ)
�
y
• Estimator is a function of the data
�
ˆ ξ (y)estimate
�
ˆ ξ estimator
�
ˆ ξ
�
y
�
ˆ ξ (y)
�
ˆ ξ
Copyright 2011 Aaron Lanterman
Maximum Likelihood Estimators
• Makes intuitive sense, but not necessarily magical:
�
ˆ ξ ML (y) = argmaxξ
p(y;ξ)
�
= argmaxξ
ln p(y;ξ)
�
= argmaxξ(ξ)
�
(ξ) (up to an additive constant)
Copyright 2011 Aaron Lanterman
Loglikelihood for Gaussian Example
�
(ξ) = −L2ln2π −
L2lnσ 2 −
[y(l) − µ]2
2σ 2l=0
L−1
∑�
p(y;ξ) =12πσ 2
l=0
L−1
∏ exp −[y(l) − µ]2
2σ 2
⎧ ⎨ ⎩
⎫ ⎬ ⎭
Customary to omit terms that do not contain parameters
Copyright 2011 Aaron Lanterman
ML Estimator for Mean
�
(ξ) = −L2lnσ 2 −
[y(l) − µ]2
2σ 2l=0
L−1
∑• “Usual” trick “usually” works:
�
∂∂µ(ξ) =
y(l) − µσ 2
l=0
L−1
∑
�
= 0
�
y(l)l=0
L−1
∑ = Lµ
�
ˆ µ ML (y) =1L
y(l)l= 0
L−1
∑⎡ ⎣ ⎢
⎤ ⎦ ⎥
Copyright 2011 Aaron Lanterman
ML Estimator for Variance (1)
�
(ξ) = −L2lnσ 2 −
[y(l) − µ]2
2σ 2l=0
L−1
∑• “Usual” trick “usually” works:
�
∂∂σ 2 (ξ) = −
L2σ 2 +
[y(l) − µ]2
2(σ 2)2l=0
L−1
∑
�
= 0
�
Lσ 2 = [y(l) − µ]2l=0
L−1
∑
Copyright 2011 Aaron Lanterman
ML Estimator for Variance (2)
�
σ 2 =1L
[y(l) − µ]2l=0
L−1
∑• Here we lucked out:
�
ˆ σ ML2 (y) =
1L
[y(l) − ˆ µ ML (y)]2
l= 0
L−1
∑
�
ˆ µ ML (y) =1L
y(l)l= 0
L−1
∑⎡ ⎣ ⎢
⎤ ⎦ ⎥
Copyright 2011 Aaron Lanterman
Bias
�
b(ξ) = Eξ [ ˆ ξ (y)]− ξ
�
b(ξ) = 0• If , i.e. , we
�
Eξ [ ˆ ξ (y)] = ξ say estimator is unbiased
Copyright 2011 Aaron Lanterman
Bias of ML Estimate of the Mean
�
b(µ) = Eξ [ ˆ µ (y)]− µ
�
= Eξ1L
y(l)l=0
L−1
∑⎡ ⎣ ⎢
⎤ ⎦ ⎥ − µ
�
=1L
Eξ [y(l)]l=0
L−1
∑ − µ
�
=1LLµ − µ
�
= 0
Copyright 2011 Aaron Lanterman
Bias of ML Estimate of the Variance (1)
�
b(σ 2) = Eξ [ ˆ σ 2(y)]−σ 2
�
Eξ [ ˆ σ 2(y)] = Eξ1L
[y(l) − ˆ µ ML (y)]2
l= 0
L−1
∑⎧ ⎨ ⎩
⎫ ⎬ ⎭
�
= Eξ1L
y(l) − 1L
y(k)k=0
L−1
∑⎡ ⎣ ⎢
⎤ ⎦ ⎥
2
l=0
L−1
∑⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
Copyright 2011 Aaron Lanterman
Bias of ML Estimate of the Variance (2)
�
Eξ [ ˆ σ 2(y)] = Eξ1L
y(l) − 1L
y(k)k= 0
L−1
∑⎡
⎣ ⎢
⎤
⎦ ⎥
2
l= 0
L−1
∑⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
�
= 1LEξ y 2(l) − 2y(l) 1
Ly(k)
k= 0
L−1
∑⎛
⎝ ⎜
⎞
⎠ ⎟ +
1L
y(k)k= 0
L−1
∑⎛
⎝ ⎜
⎞
⎠ ⎟ 2⎡
⎣ ⎢ ⎢
⎤
⎦ ⎥ ⎥ l= 0
L−1
∑⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
Copyright 2011 Aaron Lanterman
First Term
�
Eξ y 2(l)l=0
L−1
∑⎧ ⎨ ⎩
⎫ ⎬ ⎭
�
= Eξ y 2(l){ }l=0
L−1
∑
�
= L(σ 2 + µ 2)
Copyright 2011 Aaron Lanterman
Second Term (1)
�
−2Eξ y(l) 1L
y(k)k=0
L−1
∑⎛ ⎝ ⎜
⎞ ⎠ ⎟
l=0
L−1
∑⎧ ⎨ ⎩
⎫ ⎬ ⎭
�
= −2L
Eξ{y(l)y(k)k=0
L−1
∑l=0
L−1
∑ }
�
= −2L
Eξ{y2(l)} + Eξ{y(l)y(k)}
k≠ l∑⎛
⎝ ⎜ ⎞ ⎠ ⎟
l=0
L−1
∑
Copyright 2011 Aaron Lanterman
Second Term (2)
�
−2L
Eξ{y2(l)} + Eξ{y(l)y(k)}
k≠ l∑⎛
⎝ ⎜ ⎞ ⎠ ⎟
l=0
L−1
∑
�
= −2 [σ 2 + µ 2] + [L −1]µ 2( )
�
= −2(σ 2 + Lµ2)
Copyright 2011 Aaron Lanterman
Third Term (1)
�
Eξ1L
y(k)k=0
L−1
∑⎛ ⎝ ⎜
⎞ ⎠ ⎟ 2
l=0
L−1
∑⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
�
= Eξ1L
y(k)k=0
L−1
∑⎛ ⎝ ⎜
⎞ ⎠ ⎟ 1L
y( j)j=0
L−1
∑⎛
⎝ ⎜ ⎞
⎠ ⎟ l=0
L−1
∑⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
Copyright 2011 Aaron Lanterman
Third Term (2)
�
Eξ1L
y(k)k=0
L−1
∑⎛ ⎝ ⎜
⎞ ⎠ ⎟ 1L
y( j)j=0
L−1
∑⎛
⎝ ⎜ ⎞
⎠ ⎟ l=0
L−1
∑⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
�
=1LEξ y(k)y( j)
j=0
L−1
∑k=0
L−1
∑⎧ ⎨ ⎩
⎫ ⎬ ⎭
�
=1L
Eξ [y2(k)] + Eξ [y(k)y( j)]
j≠k∑
⎛
⎝ ⎜ ⎞
⎠ ⎟ k=0
L−1
∑⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
Copyright 2011 Aaron Lanterman
Third Term (3)
�
1L
Eξ [y2(k)] + Eξ [y(k)y( j)]
j≠k∑
⎛
⎝ ⎜ ⎞
⎠ ⎟ k=0
L−1
∑⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
�
= (σ 2 + µ 2) + (L −1)µ 2
�
= σ 2 + Lµ2
Copyright 2011 Aaron Lanterman
Putting it Back Together
�
1LEξ y 2(l) − 2y(l) 1
Ly(k)
k=0
L−1
∑⎛ ⎝ ⎜
⎞ ⎠ ⎟ +
1L
y(k)k=0
L−1
∑⎛ ⎝ ⎜
⎞ ⎠ ⎟ 2⎡
⎣ ⎢ ⎢
⎤
⎦ ⎥ ⎥ l=0
L−1
∑⎧ ⎨ ⎪
⎩ ⎪
⎫ ⎬ ⎪
⎭ ⎪
�
=1LL(σ 2 + µ 2) − 2(σ 2 + Lµ2) + (σ 2 + Lµ2){ }
�
=L −1L
σ 2
�
Eξ [ ˆ σ 2(y)] =
Copyright 2011 Aaron Lanterman
Almost Forgot What We Were Computing
�
=L −1L
σ 2 −σ 2
�
= −σ 2
L
�
b(σ 2)→ 0• Notice as ;
�
L→∞we say estimator is asymptotically unbiased
�
b(σ 2) = Eξ [ ˆ σ 2(y)]−σ 2
Copyright 2011 Aaron Lanterman
Tweaking the Estimator
• Previous computations suggest an unbiased estimator:
�
ˆ σ UB2 (y) =
1L −1
[y(l) − ˆ µ ML (y)]2
l= 0
L−1
∑