5 cramer-rao lower bound
TRANSCRIPT
1
Cramér – Rao Lower Bound
SOLO HERMELIN
25.09.09http://www.solohermelin.com
2
Cramér-Rao Lower Bound (CRLB)SOLO
Table of Content
The Cramér-Rao Lower Bound on the Variance of the Estimator
The Cramér-Rao Lower Bound on the Variance of the Estimator –Scalar Case
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
GradientMatrix Inversion Relations
Helpfully Relations
Nonrandom Parameters
Random Parameters
Nonrandom and Random Parameters Cramér – Rao Bounds
Discrete Time Nonlinear Estimation
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
Discrete Time Nonlinear Estimation –Special Cases
Probability Density Function of is Gaussian0x
Additive Gaussian Noises
References
Linear System with Zero System Noise
Linear/Gaussian Systems
3
Cramér-Rao Lower Bound (CRLB)
v
( )vxh ,z
x
Estimatorx
SOLO
The Cramér-Rao Lower Bound (CRLB) on the Variance of the Estimator
{ }xE - estimated mean vector
[ ]( ) [ ]( ){ } { } { } { } TTT
x xExExxExExxExE
−=−−=2σ - estimated variance matrix
For a good estimator we want
{ } xxE =- unbiased estimator vector
{ } { } { } TT
x xExExxE
−=2σ - minimum estimation covariance
( ) ( ){ }Tk kzzZ 1::1 = - the observation matrix after k observations
( ) ( ) ( ){ }xkzzLxZL k ,,,1,:1 = - the Likelihood or the joint density function of Z1:k
We have:
( )Tpzzzz ,,, 21 = ( ) Tnxxxx ,,, 21 = ( )Tpvvvv ,,, 21 =
The estimation of , using the measurements of a system corrupted by noise is a random variable with
x x zv
( ) ( ) ( ) ( )∫== dvvpxvZpxZpxZL vkvzkxzk ;||, :1|:1|:1
( ) ( )[ ]{ } ( ) ( )[ ] ( ) ( )[ ] ( ) ( )[ ] [ ] ( )xbxZdxZLZx
kzdzdxkzzLkzzxkzzxE
kkk +==
=
∫∫
:1:1:1 ,ˆ
1,,,1,,1ˆ,,1ˆ
- estimator bias( )xb
therefore:
4
Cramér-Rao Lower Bound (CRLB)
v
( )vxh ,z
x
Estimatorx
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator –Scalar Case
[ ]{ } [ ] [ ] ( )xbxZdxZLZxZxE kkkk +== ∫ :1:1:1:1 ,ˆˆ
We have:
[ ]{ } [ ] [ ] ( )x
xbZd
x
xZLZx
x
ZxEk
kk
k
∂∂+=
∂∂=
∂∂
∫ 1,
ˆˆ
:1:1
:1:1
Since L [Z1:k,x] is a joint density function, we have:
[ ] 1, :1:1 =∫ kk ZdxZL
[ ] [ ] [ ] [ ]0,,0
,:1
:1:1
:1:1
:1 =∂
∂=∂
∂→=∂
∂∫∫∫ k
kk
kk
k Zdx
xZLxZd
x
xZLxZd
x
xZL
[ ]( ) [ ] ( )x
xbZd
x
xZLxZx k
kk ∂
∂+=∂
∂−∫ 1,
ˆ :1:1
:1
Using the fact that: [ ] [ ] [ ]x
xZLxZL
x
xZL kk
k
∂∂=
∂∂ ,ln
,, :1
:1:1
[ ]( ) [ ] [ ] ( )x
xbZd
x
xZLxZLxZx k
kkk ∂
∂+=∂
∂−∫ 1,ln
,ˆ :1:1
:1:1
5
Cramér-Rao Lower Bound (CRLB)SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 1)
[ ]( ) [ ] [ ] ( )x
xbZd
x
xZLxZLxZx k
kkk ∂
∂+=∂
∂−∫ 1,ln
,ˆ :1:1
:1:1
Hermann Amandus Schwarz
1843 - 1921
Let use Schwarz Inequality:
( ) ( ) ( ) ( )∫∫∫ ≤ dttgdttfdttgtf22
2
The equality occurs if and only if f (t) = k g (t)
[ ]( ) [ ] [ ] [ ]xZLx
xZLgxZLxZxf k
kkk ,
,ln:&,ˆ: :1
:1:1:1 ∂
∂=−=choose:
[ ]( ) [ ] [ ]
( ) [ ]( ) [ ]( ) [ ] [ ]
∂
∂−≤
∂
∂+=
∂
∂−
∫∫
∫
kk
kkkk
kk
kk
Zdx
xZLxZLZdxZLxZx
x
xb
Zdx
xZLxZLxZx
:1
2
:1:1:1:1
2:1
2
2
:1:1
:1:1
,ln,,ˆ1
,ln,ˆ
[ ]( ) [ ]( )
[ ] [ ]∫
∫
∂
∂
∂
∂+≥−
kk
k
kkk
Zdx
xZLxZL
xxb
ZdxZLxZx
:1
2
:1:1
2
:1:12
:1,ln
,
1
,ˆ
6
Cramér-Rao Lower Bound (CRLB)SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 2)
[ ]( ) [ ]( )
[ ] [ ]∫
∫
∂
∂
∂
∂+≥−
kk
k
kkk
Zdx
xZLxZL
xxb
ZdxZLxZx
:1
2
:1:1
2
:1:12
:1,ln
,
1
,ˆ
This is the Cramér-Rao bound for a biased estimator
Harald Cramér1893 – 1985
Cayampudi RadhakrishnaRao
1920 -
[ ]{ } ( ) [ ] 1,&ˆ :1:1:1 =+= ∫ kkk ZdxZLxbxZxE
[ ]( ) [ ] [ ] [ ]{ } ( )( ) [ ][ ] [ ]{ }( ) [ ] ( ) [ ] [ ]{ }( ) [ ]
( ) [ ]
1
:1:12
0
:1:1:1:1:1:12
:1:1
:1:12
:1:1:12
:1
,
,ˆˆ2,ˆˆ
,ˆˆ,ˆ
∫
∫∫∫∫
+
−+−=
+−=−
kk
kkkkkkkk
kkkkk
kk
ZdxZLxb
ZdxZLZxEZxxbZdxZLZxEZx
ZdxZLxbZxEZxZdxZLxZx
[ ] [ ]{ }( ) [ ]( )
[ ] [ ]( )xb
Zdx
xZLxZL
xxb
ZdxZLZxEZx
kk
k
kkkkx2
:1
2
:1:1
2
:1:12
:1:12ˆ
,ln,
1
,ˆˆ −
∂
∂
∂
∂+≥−=
∫∫σ
7
Cramér-Rao Lower Bound (CRLB)SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 3)
[ ] [ ]{ }( ) [ ]( )
[ ] [ ]( )xb
Zdx
xZLxZL
xxb
ZdxZLZxEZx
kk
k
kkkkx2
:1
2
:1:1
2
:1:12
:1:12ˆ
,ln,
1
,ˆˆ −
∂
∂
∂
∂+≥−=
∫∫σ
[ ] [ ][ ]
[ ]
[ ] [ ] [ ] 0,,ln
0,
1, :1:1:1
,
,
,ln
:1:1
:1:1
:1
:1
:1
=∂
∂→=∂
∂→= ∫∫∫∂
∂
=∂
∂
kkk
xZL
x
xZL
x
xZL
kk
kk ZdxZLx
xZLZd
x
xZLZdxZL
k
k
k
[ ] [ ] [ ] [ ] [ ][ ]
0,,ln,ln
,,ln
:1
,
:1:1:1
:1:12:1
2
:1
=∂
∂∂
∂+∂
∂→ ∫∫
∂∂
∂∂
k
x
xZL
kkk
kkk
x
ZdxZLx
xZL
x
xZLZdxZL
x
xZL
k
[ ] [ ]0
,ln,ln2
:12:1
2
=
∂
∂+
∂∂→
∂∂
x
xZLE
x
xZLE kk
x
( )
[ ]( )
( )
[ ] ( )xb
xxZL
E
xxb
xb
xxZL
E
xxb
kk
x2
2:1
2
2
2
2
:1
2
2
,ln
1
,ln
1
−
∂∂
∂
∂+−=−
∂
∂
∂
∂+≥σ
8http://www.york.ac.uk/depts/maths/histstat/people/cramer.gif
Cramér-Rao Lower Bound (CRLB)
[ ]( ) [ ]( )
[ ]
( )
[ ]
∂∂
∂
∂+−=
∂
∂
∂
∂+≥−∫
2:1
2
2
2
:1
2
:1:12
:1,ln
1
,ln
1
,
x
xZLE
xxb
x
xZLE
xxb
ZdxZLxZxkk
kkk
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator – Scalar Case (continue – 4)
( )
[ ]( )
( )
[ ] ( )xb
xxZL
E
xxb
xb
xxZL
E
xxb
kk
x2
2:1
2
2
2
2
:1
2
2
,ln
1
,ln
1
−
∂∂
∂
∂+−=−
∂
∂
∂
∂+≥σ
For an unbiased estimator (b (x) = 0), we have:
[ ] [ ]
∂∂
−=
∂
∂≥
2:1
22
:1
2
,ln
1
,ln
1
x
xZLE
x
xZLE
kk
xσ
Return to Table of Content
9
Cramér-Rao Lower Bound (CRLB)SOLO
Gradient
Gradient of a Scalar ( ) ( ) nTnxxxxxL RR ∈=∈ ,,, 21
1
( ) n
nn
x
x
L
x
L
x
L
L
x
x
x
xL R∈
∂∂
∂∂∂∂
=
∂∂
∂∂
∂∂
=∇
2
1
2
1
( ) nxn
nnn
n
n
n
n
Txx
x
L
xx
L
xx
L
xx
L
x
L
xx
L
xx
L
xx
L
x
L
Lxxx
x
x
x
xL R∈
∂∂
∂∂∂
∂∂∂
∂∂∂
∂∂
∂∂∂
∂∂∂
∂∂∂
∂∂
=
∂
∂∂
∂∂∂
∂∂
∂∂
∂∂
=∇∇
2
2
2
2
1
2
2
2
22
2
12
2
1
2
21
2
21
2
21
2
1
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
10
Cramér-Rao Lower Bound (CRLB)SOLO
Gradient
Gradient of a Vector ( ) ( ) ( ) nTn
pTp
p xxxxaaaaxa RRR ∈=∈=∈ ,,,,,, 2121
( ) [ ] nxp
n
p
nn
p
p
p
n
Tx
x
a
x
a
x
a
x
a
x
a
x
a
x
a
x
a
x
a
aaa
x
x
x
xa R∈
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
∂∂
=
∂∂
∂∂
∂∂
=∇
21
22
2
2
1
11
2
1
1
212
1
nTx
Tx
nxnTx vvxxvIx RR ∈=∇=∇∈=∇ &1
( ) ( )[ ] ( )[ ] ( ) ( )[ ] ( ) nTx
Tx
abbaT
x xaxbxbxaxbxaTT
R∈∇+∇=∇=
2
( ) ( ) ( ) ( ) xMMxxMxMxxMx T
Mx
Tx
Tx
Tx
TT
+=∇+∇=∇
3
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Return to Table of Content
11
Cramér-Rao Lower Bound (CRLB)SOLO
Matrix Inversion Relations
∆∆∆−∆+
=
−−−
−−−−−−−
111
1111111
AD
BAADBAA
CD
BA1
2
AofcomplementSchurBADC 1: −−=∆
( ) ( ) 1111111 −−−−−−− +−=+ ADBADCBAADBCA
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
Return to Table of Content
12
Cramér-Rao Lower Bound (CRLB)SOLO
Helpfully Relations
( ) ( ) ( ) ( ) ( )zxfzxfzxfzxf
zxf Txx
Txx
Txx ,ln,ln,
,
1,ln ∇∇−∇∇=∇∇
Proof:
Start with: ( ) ( ) ( )zxfzxf
zxf xx ,,
1,ln ∇=∇
( ) ( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( ) ( )
( ) ( ) ( ) ( )zxfzxfzxfzxf
zxfzxfzxf
zxfzxf
zxfzxfzxf
zxfzxf
zxfzxf
zxfzxf
zxfzxf
zxf
Txx
Txx
Txx
Txx
Txx
Txx
Txx
Txx
Txx
Txx
,ln,ln,,
1
,ln,,
1,
,
1,ln,
,
1,
,
1
,,
1,
,
1,
,
1,ln
∇∇−∇∇=
∇∇−∇∇=∇
∇+∇∇=
∇
∇+∇∇=
∇∇=∇∇
( ) ++ →= RR pnzxf :,Lemma 1: Given a function the following relations holds:
The Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
13
Cramér-Rao Lower Bound (CRLB)SOLO
Helpfully Relations
( ) ( ) ( ) ( ) ( )zxfzxfzxfzxf
zxf Txx
Txx
Txx ,ln,ln,
,
1,ln ∇∇−∇∇=∇∇
Proof:
( ) ++ →= RR pnzxf :,Lemma 1: Given a function the following relations holds:
pz R∈Lemma 2: Let be a random vector with density p (y|x) parameterized by the nonrandom vector , then:
nx R∈( ) ( ){ } ( ){ }xzpExzpxzpE T
xxzT
xxz |ln|ln|ln ∇∇−=∇∇
( ){ } ( ) ( ) ( ) ( ){ }xzpxzpExzpxzp
ExzpE Txxz
Txxz
Txxz |ln|ln|
|
1|ln
0
∇∇−
∇∇=∇∇
( ) ( ) ( ) ( ) ( ) ( ) 0||||
1|
|
1
1
=∇∇=
∇∇=
∇∇ ∫∫
pp
zdxzpzdxzpxzpxzp
xzpxzp
E Txx
Txx
Txxz
RR
Proof:
( ) ( ){ } ( ){ }zxpEzxpzxpE Txxzx
Txxzx ,ln,ln,ln ,, ∇∇−=∇∇
( ){ } ( ) ( ) ( ) ( ){ }zxpzxpEzxpzxp
EzxpE Txxzx
Txxzx
Txxzx ,ln,ln,
,
1,ln ,
0
,, ∇∇−
∇∇=∇∇
Lemma 3: Let be random vectors with joint density p (x,y), then: pn zx RR ∈∈ ,
( ) ( ) ( ) ( ) ( ) ( ) 0,,,,
1,
,
1
1
, =∇∇=
∇∇=
∇∇ ∫∫++
pnpn
zdxdzxpzdxdzxpzxpzxp
zxpzxp
E Txx
Txx
Txxzx
RR
Return to Table of Content
14
Cramér-Rao Lower Bound (CRLB)SOLO
Nonrandom Parameters
nx R∈pz R∈
The Score of the estimation is defined by the logarithm of the likelihood ( )xzpx |ln∇ In Maximum Likelihood Estimation (MLE), this function returns a vector valued Score given by the observations and a candidate parameter vector .Score close to zero are good scores since they indicate that is close to a local optimum of , since
pz R∈ nx R∈x
( )xzp |
( ) ( ) ( )xzpxzp
xzp xx ||
1|ln ∇=∇
Since the measurement vector is stochastic the Expected Value of the Score is given by:
pz R∈
( ){ } ( ) ( )
( ) ( ) ( ) ( ) ( ) 0|||||
1
||ln|ln
1
=∇=∇=∇=
∇=∇
∫∫∫
∫
ppp
p
zdxzpzdxzpzdxzpxzpxzp
zdxzpxzpxzpE
xxx
xxz
RRR
R
v
( )vxh ,z
x
Estimatorx
The parameters are regarded as unknown but fixed. The measurements are
nx R∈ pz R∈
15
Cramér-Rao Lower Bound (CRLB)
( ) ( ) ( ){ } ( ){ }xzpExzpxzpExJ Txxz
Txxz |ln|ln|ln: ∇∇−=∇∇=
SOLO
The Fisher Information Matrix (FIM)
Fisher, Sir Ronald Aylmer 1890 - 1962
The Fisher Information Matrix (FIM) was defined by Ronald AylmerFisher as the Covariance Matrix of the Score
( ){ } ( ) ( ) 0||ln|ln =∇=∇ ∫p
zdxzpxpxzpE xxz
R
The Expected Value of the Score is given by:
The Covariance of the Score is given by:
( ) ( ){ } ( ) ( ) ( )∫ ∇∇=∇∇p
zdxzpxzpxpxzpxzpE Txx
Txxz
R
||ln|ln|ln|ln
Nonrandom ParametersThe Cramér-Rao Lower Bound on the Variance of the Estimator – Multivariable Case
16
Fisher, Sir Ronald Aylmer (1890-1962)
The Fisher information is the amount of information that an observable random variable z carries about an unknown parameter x upon which the likelihood of z, L(x) = f (Z; x), depends. The likelihood function is the joint probability of the data, the Zs, conditional on the value of x, as a function of x. Since the expectation of the score is zero, the variance is simply the second moment of the score, the derivative of the lan of the likelihood function with respect to x. Hence the Fisher information can be written
( ) [ ]( ) [ ]( ){ } [ ]( ){ }x
k
xxx
Tk
x
k
x xZLExZLxZLEx ,ln,ln,ln: ∇∇−=∇∇=J
Cramér-Rao Lower Bound (CRLB)
Return to Table of Content
17
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ( ) rxn
yy
Ty
Trxr
yy
Tyyz ytMyzpEJ RR ∈∇=∈∇∇−=
== **:&|ln:
Nonrandom Parameters
The Likelihood p (z|x) may be over-parameterized so that some of x or combination ofelements of x do not affect p (z|x). In such a case the FIM for the parameters x becomessingular. This leads to problems of computing the Cramér – Rao bounds. Let(r ≤ n) be an alternative parameterization of the Likelihood such that p (z|y) is a welldefined density function for z given and the corresponding FIM is non-singular.We define a possible non-invertible coordinate transformation .
ry R∈
ry R∈
( )ytx =
Theorem 1: Nonrandom Parametric Cramér – Rao Bound Assume that the observation has a well defined probability density function p (z|y)for all , and let denote the parameter that yields the true distribution of .Moreover, let be an Unbiased Estimator of , and let . The estimation error covariance of is bounded for below by
pz R∈ry R∈ *y y
( ) nzx R∈ˆ ( )ytx = ( )** ytx =( )zx
( ) ( ){ } TTz MJMxxxxE 1*ˆ*ˆ −≥−−
where
are matrices that depend on the true unknown parameter vector .*y
18
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ( )**
:&|ln:yy
Ty
T
yy
Tyyz ytMyzpEJ
==∇=∇∇−=
Nonrandom Parameters
Theorem 1: Nonrandom Parametric Cramér – Rao Bound Assume that the observation has a well defined probability density function p (z|y)for all , and let denote the parameter that yields the true distribution of .Moreover, let be an Unbiased Estimator of , and let . The estimation error covariance of is bounded for below by
pz R∈ry R∈ *y y
( ) nzx R∈ˆ ( )ytx = ( )** ytx =( )zx ( ) ( ){ } TT
z MJMxxxxE 1*ˆ*ˆ −≥−−
where
are matrices that depend on the true unknown parameter vector .*y
Proof:
( ) ( )[ ] ( ){ } 0|ˆ =−∇∫p
zdyzpytzx Ty
R
Tacking the gradient w.r.t. on both sides of this relation we obtain:y
( ){ } ( ) ( )[ ] ( ){ } ( ) 0|ˆ| =∇−−∇ ∫∫pp
zdyzpytzdytzxyzp Ty
Ty
RR
( ){ } ( ) ( )[ ] ( ) ( ) ( )
1
||ˆ|ln ∫∫ ∇=−∇pp
zdyzpytzdyzpytzxyzp Ty
Ty
RR
( ){ } ( ) ( )[ ] ( ) ( )ytzdyzpytzxyzp Ty
Ty
p
∇=−∇∫R
|ˆ|ln
Consider the Random Vector:
( )
∇
−yzp
xx
y |ln
ˆwhere: ( )
{ }( ){ }
=
∇
−=
∇
−
0
0
|ln
ˆ
|ln
ˆ
yzpE
xxE
yzp
xxE
yz
z
yz
( )[ ] ( ) ( ) ( )[ ] ( )( )
0|ˆ|ˆˆ
sUnbiasenes
zxof
TT
pp
zdyzpytzxzdxzpxzx =−=− ∫∫RR
Using the Unbiasedness of Estimator:
19
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ( )**
:&|ln:yy
Ty
T
yy
Tyyz ytMyzpEJ
==∇=∇∇−=
Nonrandom Parameters
Theorem 1: Nonrandom Parametric Cramér – Rao Bound Assume that the observation has a well defined probability density function p (z|y)for all , and let denote the parameter that yields the true distribution of .Moreover, let be an Unbiased Estimator of , and let . The estimation error covariance of is bounded for below by
pz R∈ry R∈ *y y
( ) nzx R∈ˆ ( )ytx = ( )** ytx =( )zx ( ) ( ){ } TT
z MJMxxxxE 1*ˆ*ˆ −≥−−
where
are matrices that depend on the true unknown parameter vector .*y
Proof (continue – 1):
Consider the Random Vector: ( )
∇
−yzp
xx
y |ln
ˆ
The Covariance Matrix is Positive Semi-definite by construction:
( ){ }
( ){ }
=
∇
−=
∇
−
0
0
|ln
ˆ
|ln
ˆ
yzpE
xxE
yzp
xxE
yz
z
yz
( ) ( ) 00
0
0
0|ln
ˆ
|ln
ˆ1
11 definiteSemiPositive
T
T
T
T
yyz
IMJ
I
J
MJMC
I
JMI
JM
MC
yzp
xx
yzp
xxE
−
−
−−
≥
−
=
=
∇
−
∇
−
( ) ( ){ }Tz xxxxEC −−= ˆˆ: ( ) ( ){ } ( ){ }yzpEyzpyzpEJ T
yyzT
yyz |ln|ln|ln: ∇∇−=∇∇=
( )( ) ( ){ } ( )ytxxyzpEM Ty
Tyz
T ∇=−∇= ˆ|ln:
( ) ( ){ } TTz
NotationsEquivalent
definiteSemiPositive
T MJMxxxxECMJMC 11 ˆˆ:0 −−
− ≥−−=⇔≥−
( ){ } ( ) ( )[ ] ( ) ( )ytzdyzpytzxyzp Ty
Ty
p
∇=−∇∫R
|ˆ|lnWe found:
q.e.d.
where:
20
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ } ( ) nxn
yy
Ty
Tnxn
yy
Tyyz ybIMyzpEJ RR ∈∇+=∈∇∇−=
== **:&|ln:
Nonrandom Parameters
Corollary 1: Nonrandom Parametric Cramér – Rao Bound (Baiased Estimator) Consider an estimaton problem defined by the likelihood p (y|z), and the fixed unknownparameter . Any estimator with unknown bias has a mean square errorbounded from below by
*y ( )zy ( )yb
( ) ( ){ } ( ) ( )***ˆ*ˆ 1 ybybMJMyyyyE TTTz +≥−− −
where
are matrices that depend on the true unknown parameter vector .*y
Proof:
Theorem 1 yields that:
Introduce the quantity , the estimator is an unbiased estimator of .( )ybyx +=: ( ) ( )zyzx ˆˆ = x
( ) ( ){ } ( )[ ] ( ){ }( ) ( )[ ]ybIyzpEybIxxxxE Ty
Tyyz
TTy
Tz ∇+∇∇−∇+≥−−
−1|lnˆˆ
Using , we obtain:( )ybyx +=:
( ) ( ){ } ( )[ ] ( ){ }( ) ( )[ ] ( ) ( )ybybybIyzpEybIyyyyE TTy
Tyyz
TTy
Tz +∇+∇∇−∇+≥−−
−1|lnˆˆ
after suitably inserting the true parameter .*y
21
Cramér-Rao Lower Bound (CRLB)
[ ]( ) [ ]( ) [ ] [ ]( ) [ ]( ){ }( ) [ ] [ ] ( ) ( ) ( )
( ) [ ] ( ) ( ) ( )xbxbx
xbI
x
xZLE
x
xbI
xbxbx
xbI
x
xZL
x
xZLE
x
xbI
xZxxZxEZdxZLxZxxZx
T
x
kT
T
x
TkkT
x
TkkkkTkk
+
∂
∂+
∂∂
∂
∂+−=
+
∂
∂+
∂
∂
∂
∂
∂
∂+≥
−−=−−
−
−
∫
1
2
2
1
,ln
,ln,ln
,
SOLO
The Cramér-Rao Lower Bound on the Variance of the Estimator
The multivariable form of the Cramér-Rao Lower Bound is:
[ ]( )[ ]
[ ]
−
−=−
n
k
n
k
k
xZx
xZx
xZx
11
[ ]( ) [ ][ ]
[ ]
∂∂
∂∂
=
∂
∂=∇
n
k
k
kk
x
x
xZL
x
xZL
x
xZLxZL
,ln
,ln
,ln,ln
1
Fisher Information Matrix
[ ] [ ] [ ]
∂∂−=
∂
∂
∂
∂=x
k
x
Tkk
x
xZLE
x
xZL
x
xZLE
2
2 ,ln,ln,ln:J
Fisher, Sir Ronald Aylmer 1890 - 1962
Return to Table of Content
22
Cramér-Rao Lower Bound (CRLB)SOLO
Random Parameters
Theorem 2: Random Parameters (Posterior Cramér – Rao Bound)
( ) ( )[ ] ( )∫ −=p
zdyzpytxybR
|ˆ
( ){ } ( ){ } rxnTyz
TrxrTyyyz ytEMyzpEJ RR ∈∇=∈∇∇−= :&,ln: ,
where
then the Mean Square of the Estimate is Bounded from Below
ynrt RR →: x
For Random Parameters there is no true parameter value. Instead, the prior assumptionon the parameter distribution determines the probability of different parameter vectors.Like in the nonrandom parametric case, we assume a possible non-invertible mapping between a parameter vector and the sought parameter . The vectoris assumed to have been chosen such that the joint probability density p (y,z) is a welldefined density.
y
Let be two random vectors with a well defined joint densityp (y,z), and let be an estimate of . If the estimator bias
pr zandy RR ∈∈( ) nzx R∈ˆ ( )ytx =
satisfies ( ) ( ) njandriallforypyb jzi
,,1,,10lim ===±∞→
( ) ( ){ } TTyz MJMxxxxE 1
, ˆˆ −≥−− ( ) ( ){ } 0ˆˆ 1,
definiteSemiPositive
TTyz MJMxxxxE
−− ≥−−−
EquivalentNotations
23
Cramér-Rao Lower Bound (CRLB)SOLO
Random Parameters
Theorem 2: Random Parameters (Posterior Cramér – Rao Bound)
( ) ( ){ } TTyz MJMxxxxE 1
, ˆˆ −≥−− ( ){ } ( ){ }ytEMyzpEJ Tyz
TTyyyz ∇=∇∇−= :&,ln: ,
then the Mean Square of the Estimate is Bounded from Below
Proof:
Let be two random vectors with a well defined joint densityp (y,z), and let be an estimate of . If the estimator bias
pr zandy RR ∈∈( ) nzx R∈ˆ ( )ytx =
( ) ( )[ ] ( )∫ −=p
zdyzpytxybR
|ˆ and ( ) ( ) njandriallforypyb jzi
,,1,,10lim ===±∞→
Compute
( ) ( )[ ] ( ) ( )[ ] ( ) ( )( )
( ) ( )
( )
( )[ ] ( ) ( )[ ]∫∫∫ −∇+−∇=−∇=∇ppp
zdytzxyzpzdyzpytzdypyzpytzxypyb Ty
yp
Ty
yzp
Ty
Ty
RRR
ˆ,,|ˆ,
Integrating both sides w.r.t. over its complete range R r yieldsy( ) ( )[ ] ( ) ( ) ( )[ ] ( ) ( )[ ]∫∫∫
+
−∇+∇−=∇rprr
ydzdytzxyzpydypytydypyb Ty
Ty
Ty
RRR
ˆ,
The (i,j) element of the left hand side matrix is:( ) ( ) ( ) ( ) ( ) ( ) riiiiyjyj
i
j ydydydydydydypybypybydy
ypyb
r iir
111
00
0 +−−−−∞=+∞===
−=
∂∂
∫∫RR
24
Cramér-Rao Lower Bound (CRLB)SOLO
Random Parameters
Theorem 2: Random Parameters (Posterior Cramér – Rao Bound)
( ) ( ){ } TTyz MJMxxxxE 1
, ˆˆ −≥−− ( ){ } ( ){ }ytEMyzpEJ Tyz
TTyyyz ∇=∇∇−= :&,ln: ,
then the Mean Square of the Estimate is Bounded from Below
Let be two random vectors with a well defined joint densityp (y,z), and let be an estimate of . If the estimator bias
pr zandy RR ∈∈( ) nzx R∈ˆ ( )ytx =
( ) ( )[ ] ( )∫ −=p
zdyzpytxybR
|ˆ and ( ) ( ) njandriallforypyb jzi
,,1,,10lim ===±∞→
Proof (continue – 1): We found ( )[ ] ( ) ( )[ ] ( ) ( )∫∫ ∇=−∇+ rrp
ydypytydzdytzxyzp Ty
Ty
RR
ˆ,
( )[ ] ( ) ( )[ ] ( ) ( ) ( ) ( ){ }ytEydypytydzdyzpytzxyzp Tyz
Ty
Ty
rrp
∇=∇=−∇ ∫∫+ RR
,ˆ,ln
Consider the Random Vector: ( )
∇
−yzp
xx
y ,ln
ˆ
The Covariance Matrix is Positive Semi-definite by construction:
( ) ( ) 00
0
0
0,ln
ˆ
,ln
ˆ1
11
,
definiteSemiPositive
T
T
T
T
yyyz
IMJ
I
J
MJMC
I
JMI
JM
MC
yzp
xx
yzp
xxE
−
−
−−
≥
−
=
=
∇
−
∇
−
( )( ) ( ){ } ( ){ }ytExxyzpEM Tyz
Tyyz
T ∇=−∇= ˆ,ln: ,
( ) ( ){ } TTz
NotationsEquivalent
definiteSemiPositive
T MJMxxxxECMJMC 11 ˆˆ:0 −−
− ≥−−=⇔≥− q.e.d.
( ) ( ){ }Tyz xxxxEC −−= ˆˆ: , ( ) ( ){ } ( ){ }yzpEyzpyzpEJ T
yyyzT
yyz ,ln,ln|ln: , ∇∇−=∇∇=where:
( ){ }
( ){ }
=
∇
−=
∇
−
0
0
,ln
ˆ
,ln
ˆ
,
,
, yzpE
xxE
yzp
xxE
yyz
yz
yyz
Return to Table of Content
25
Cramér-Rao Lower Bound (CRLB)SOLO
Nonrandom and Random Parameters Cramér – Rao Bounds
For the Nonrandom Parameters the Cramér – Rao Bound depends on the true unknownparameter vector y , and on the model of the problem defined by p (z|y) and the mapping x = t (y). Hence the bound can only be computed by using simulations, when the true valueof the sought parameter vector y is known.
For the Random Parameters the Cramér – Rao Bound can be computed even in realapplications. Since the parameters are random there is no unknown true parameter value.
Instead, in the posterior Cramér – Rao Bound the matrices J and M are computed bymathematical expectation performed with respect to the prior distribution of the parameters.
Return to Table of Content
26
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
We found that the Cramér – Rao Lower Bound for the Random Parameters is given by:
( ) ( ){ } ( ) ( ){ } ( ){ } 1
:1:1,
1
:1:1:1:1,:1:1|:1:1:1|:1, ,ln,ln,ln:1:1:1:1
−−∇∇−=∇∇≥−− kk
TXXZXkk
TXkkXZX
T
kkkkkkZX XZpEXZpXZpEXXXXEkkkk
( )1−= kk xfxIf we have a deterministic state model, i.e. then we can use the Nonrandom
Parametric Cramér – Rao Lower Bound ( ) ( ){ } ( ) ( ){ } ( ){ } 1
:1:1
1
:1:1:1:1:1:1|:1:1:1|:1 |ln|ln|ln:1:1:1:1
−−∇∇−=∇∇≥−− kk
TXXZkk
TXkkXZ
T
kkkkkkZ XZpEXZpXZpEXXXXEkkkk
After k cycles we have k measurements and k random parameters estimated by an Unbiased Estimator as .
[ ]Tkk zzzZ ,,,: 21:1 =[ ]Tkk xxxxX ,,,,: 210:0 = [ ]Tkkkk xxxX |2|21|1:1|:1 ˆ,,ˆ,ˆ:ˆ =
The CRLB provides a lower bound for second-order (mean-squared) error only. Posteriordensities, which result from Nonlinear Filtering, are in general non-Gaussian. A full statistical characterization of a non-Gaussian density requires higher order moments, inaddition to mean and covariance. Therefore, the CRLB for Nonlinear Filtering does notfully characterize the accuracy of Filtering Algorithms.
27
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
Theorem 3: The Cramér – Rao Lower Bound for the Random Parameters is given by:
Let perform the partitioning [ ] 11:1:1 ,: xnkT
kkk xXX R∈= − [ ] 1|1:1|1:1:1|:1 ˆ,ˆ:ˆ xnkT
kkkkkk xXX R∈= −−
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
After k cycles we have k measurements and k random parameters estimated by an Unbiased Estimator as .
[ ]Tkk zzzZ ,,,: 21:1 =[ ]Tkk xxxxX ,,,,: 210:0 = [ ]Tkkkk xxxX |2|21|1:1|:1 ˆ,,ˆ,ˆ:ˆ =
( ){ } ( ) ( )
( ){ } ( )
( ){ } nxnkk
TxxZXk
nxknkk
TxXZXk
knxknkk
TXXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
R
R
R
∈∇∇−=
∈∇∇−=
∈∇∇−=−
−−
−
−−
:1:1,
1:1:1,
11:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
( ) ( ){ } ( ) nxnkk
Tkkk
TkkkkkkZX BABCJxxxxE R∈−=≥−− −−− 111
||, :ˆˆ( ) ( ){ } ( ) 0ˆˆ11
||,
definiteSemiPositive
kkTkk
TkkkkkkZX BABCxxxxE
−−− ≥−−−−
EquivalentNotations
28
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
The Cramér – Rao Bound for the Random Parameters is given by:
( ) ( ){ } 0,ln,lnˆˆ
1
:1:1,:1:1,,
|
1:11:1|1:1
|
1:11:1|1:1, 1:11:1
definiteSemiPositive
kkT
xXkkxXZX
T
kkk
kkk
kkk
kkkZX XZpXZpE
xx
XX
xx
XXE
kkkk
−−−−−−−− ≥∇∇−
−
−
−
−−−
Proof Theorem 3: Let perform the partitioning [ ] 11:1:1 ,: xnkT
kkk xXX R∈= − [ ] 1|1:1|1:1:1|:1 ˆ,ˆ:ˆ xnkT
kkkkkk xXX R∈= −−
( ){ } ( ){ } ( ){ }( ){ } ( ){ }
1
:1:1,:1:1,
:1:1,:1:1,1
:1:1,,,,ln,ln
,ln,ln,ln
1:1
1:11:11:1
1:11:1
−
−
∇∇−∇∇−
∇∇−∇∇−=∇∇−
−
−−−
−−
kkT
xxZXkkT
XxZX
kkT
xXZXkkT
XXZX
kkT
xXxXZXXZpEXZpE
XZpEXZpEXZpE
kkkk
kkkk
kkkk
nkxnkkk
kkTkk
k
kTkk
Tk
kk
I
BAI
BABC
A
IAB
I
CB
BAR∈
−
=
=
−−
−−
− 11
11
1
00
00:
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
After k cycles we have k measurements and k random parameters estimated by an Unbiased Estimator as .
[ ]Tkk zzzZ ,,,: 21:1 =[ ]Tkk xxxxX ,,,,: 210:0 = [ ]Tkkkk xxxX |2|21|1:1|:1 ˆ,,ˆ,ˆ:ˆ =
29
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation
( ) ( ){ } ( ) ( ){ }( ) ( ){ } ( ) ( ){ }
( ) 00
0
0
0
ˆˆˆ
ˆ
1
111
111
||,1:11:1|1:1|,
|1:11:1|1:1,1:11:1|1:11:11:1|1:1,
definiteSemiPositive
kTkkk
Tkk
kkk
TkkkkkkZX
T
kkkkkkZX
TkkkkkkZX
T
kkkkkkZX
IAB
I
BABC
A
I
BAI
xxxxEXXxxE
xxXXEXXXXE
−−
−−−
−−−
−−−
−−−−−−−−−
≥
−
−
−−−−
−−−−
Proof Theorem 3 (continue – 1): We found
( ) ( ){ } ( ) ( ){ }( ) ( ){ } ( ) ( ){ }
( ) 00
0
0
ˆˆˆ
ˆ
0
11
1
1
||,1:11:1|1:1|,
|1:11:1|1:1,1:11:1|1:11:11:1|1:1,1
definiteSemiPositive
kkTkk
k
kTk
TkkkkkkZX
T
kkkkkkZX
TkkkkkkZX
T
kkkkkkZXkk
BABC
A
IAB
I
xxxxEXXxxE
xxXXEXXXXE
I
BAI
−
−−
−
−
−−−
−−−−−−−−−−
≥
−−
−−−−
−−−−
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
30
Cramér-Rao Lower Bound (CRLB)
( ) ( ){ } ( ) 111||, :ˆˆ
−−− −=≥−− kkTkkk
TkkkkkkZX BABCJxxxxE
SOLO
Discrete Time Nonlinear Estimation
Prof Theorem 3 (continue – 2): We found
( ) ( ){ } ( ) 00
0
ˆˆ*
**11
1
||,
definiteSemiPositive
kkTkk
k
TkkkkkkZX BABC
A
xxxxE
−
−−
−
≥
−−
−−
( ) ( ){ } ( ) 0ˆˆ11
||,
definiteSemiPositive
kkTkk
TkkkkkkZX BABCxxxxE
−−− ≥−−−−
EquivalentNotations
( ){ } ( ) ( )
( ){ } ( )
( ){ } nxnkk
TxxZXk
nxknkk
TxXZXk
knxknkk
TXXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
R
R
R
∈∇∇−=
∈∇∇−=
∈∇∇−=−
−−
−
−−
:1:1,
1:1:1,
11:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
q.e.d.
31
Cramér-Rao Lower Bound (CRLB)
( ){ } ( ) ( )
( ){ } ( )
( ){ } nxnkk
TxxZXk
nxknkk
TxXZXk
knxknkk
TXXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
R
R
R
∈∇∇−=
∈∇∇−=
∈∇∇−=−
−−
−
−−
:1:1,
1:1:1,
11:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
SOLODiscrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
We found
We want to compute Jk recursively, without the need for inverting large matrices as Ak.
( ) ( ){ } ( ) 111||, :ˆˆ
−−− −=≥−− kkTkkk
TkkkkkkZX BABCJxxxxE
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
Theorem 4:The Recursive Cramér–Rao Lower Bound for the Random Parameters is given by:
( ) ( ){ } [ ]( ) nxnkkkkkk
TkkkkkkZX DDJDDJxxxxE R∈+−=≥−−
−−−+−−+−−+
11211121221
111|111|1, :ˆˆ
( ) ( ){ } ( ) nxnkk
Tkkk
TkkkkkkZX BABCJxxxxE R∈−=≥−−
−−− 111||, :ˆˆ
( ){ }( )[ ]{ } [ ]
( ){ } ( ){ } nxnkk
Tkxxzkk
Tkxxxk
nxnT
kkkTkxxxk
nxnkk
Tkxxxk
xzpExxpED
DxxpED
xxpED
kkkkkk
kkk
kkk
R
R
R
∈∇∇−∇∇−=
∈=∇∇−=
∈∇∇−=
+++++
++
+
+++++
+
+
111|11|22
2111|
12
1|11
|ln|ln:
|ln:
|ln:
11111
1
1
( ) ( ){ }000 lnln000
xpxpEJ Txxx ∇∇=The recursions start with the initial
information matrix J0,
32
Cramér-Rao Lower Bound (CRLB)SOLO
( ){ }( ){ }
( ){ }kkT
xxZXk
kkT
xXZXk
kkT
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
:1:1,
:1:1,
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
∇∇−=
∇∇−=
∇∇−=
−
−−
We found
We want to compute Jk recursively, without the need for inverting large matrices as Ak.
( ) ( ){ } ( ) 111||, :ˆˆ
−−− −=≥−− kkTkkk
TkkkkkkZX BABCJxxxxE
Start with:
( ) ( ) ( ) ( )kkkkkkkkkkkkk XxZpXxZzpXxZzpXZp :11:1:11:11:11:111:11:1 ,,,,|,,,, +++++++ ==( )
( )
( )( )
( )kk
xxpMarkov
kkk
xzpMarkov
kkkk XZpXZxpXxZzp
kkkk
:1:1
|
:1:11
|
:11:11 ,,|,,|
111
+++ →
+
→
++=
( ) ( ) ( )1:11:11 ,|| −−−= kkkkkk XZpxxpxzp
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
Proof of Theorem 4:
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
33
Cramér-Rao Lower Bound (CRLB)
( ) 11
1
111
111
111
1
1:11:1,
11|1
|
1:11:1|1:1
11|1
|
1:11:1|1:1
, :,ln
ˆ
ˆ
ˆ
ˆ
1111:11
11:1
11:11:11:11:1
−+
−
+++
+++
+++
−
++
+++
−−−
+++
−−−
=
=
∇∇∇∇∇∇
∇∇∇∇∇∇
∇∇∇∇∇∇
−≥
−
−
−
−
−
−
+++−+
+−
+−−−−
k
kTk
Tk
kkTk
kkk
kk
Txx
Txx
TXx
Txx
Txx
TXx
TxX
TxX
TXX
ZX
T
kkk
kkk
kkk
kkk
kkk
kkk
ZX I
FEL
ECB
LBA
XZpE
xx
xx
XX
xx
xx
XX
E
kkkkkk
kkkkkk
kkkkkk
SOLO
Proof of Theorem 4 (continue – 1):
Compute:
( ) ( ) ( ) ( )kkkkkkkk XZpxxpxzpXZp :1:11111:11:1 ,||, +++++ =
( ){ } ( ) ( ) ( )[ ]{ }( )[ ]{ } kkk
TXXZX
kkkkkkT
XXZXkkT
XXZXk
AXZpE
XZpxxpxzpEXZpEA
kk
kkkk
=∇∇−+=
++∇∇−=∇∇−=
−−
−−−− ++++++
:1:1,
:1:1111,1:11:1,1
,ln00
,ln|ln|ln,ln:
1:11:1
1:11:11:11:1
( ){ } ( ) ( ) ( )[ ]{ }( )[ ]{ } kkk
TxXZX
kkkkkkT
xXZXkkT
xXZXk
BXZpE
XZpxxpxzpEXZpEB
kk
kkkk
=∇∇−+=
++∇∇−=∇∇−=
−
−− ++++++
:1:1,
:1:1111,1:11:1,1
,ln00
,ln|ln|ln,ln:
1:1
1:11:1
( ){ } ( ) ( ) ( )[ ]{ }( )[ ]{ } ( )[ ]{ } 11
:1:1,1|
:1:1111,1:11:1,1
,ln|ln0
,ln|ln|ln,ln:
11
1 kk
C
kkT
xxZX
D
kkT
xxxx
kkkkkkT
xxZXkkT
xxZXk
DCXZpExxpE
XZpxxpxzpEXZpEC
k
kk
k
kkkk
kkkk
+=∇∇−∇∇−=
++∇∇−=∇∇−=
+
++++++
+
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
34
Cramér-Rao Lower Bound (CRLB)
( ) 11
1
111
111
111
1
1:11:1,
11|1
|
1:11:1|1:1
11|1
|
1:11:1|1:1
, :,ln
ˆ
ˆ
ˆ
ˆ
1111:11
11:1
11:11:11:11:1
−+
−
+++
+++
+++
−
++
+++
−−−
+++
−−−
=
=
∇∇∇∇∇∇
∇∇∇∇∇∇
∇∇∇∇∇∇
−≥
−
−
−
−
−
−
+++−+
+−
+−−−−
k
kTk
Tk
kkTk
kkk
kk
Txx
Txx
TXx
Txx
Txx
TXx
TxX
TxX
TXX
ZX
T
kkk
kkk
kkk
kkk
kkk
kkk
ZX I
FEL
ECB
LBA
XZpE
xx
xx
XX
xx
xx
XX
E
kkkkkk
kkkkkk
kkkkkk
SOLO
Proof of Theorem 4 (continue – 2):
Compute:
( ) ( ) ( ) ( )kkkkkkkk XZpxxpxzpXZp :1:11111:11:1 ,||, +++++ =
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ] ( )[ ] ( )[ ] 0,ln|ln|ln
,ln|ln|ln,ln:
0
:1:1,
0
1,
0
11,
:1:1111,1:11:1,1
11:111:111:1
11:111:1
=
∇∇−
∇∇−
∇∇−=
++∇∇−=∇∇−=
+−+−+−
+−+−
+++
++++++
kkT
xXZXkkT
xXZXkkT
xXZX
kkkkkkT
xXZXkkT
xXZXk
XZpExxpExzpE
XZpxxpxzpEXZpEL
kkkkkk
kkkk
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ] ( )[ ]{ } ( )[ ] ( )[ ]{ } 121|
0
:1:1,1,
0
11,
:1:1111,1:11:1,1
:|ln,ln|ln|ln
,ln|ln|ln,ln:
11111
11
kkkT
xxxxkkT
xxZXkkT
xxZXkkT
xxZX
kkkkkkT
xxZXkkT
xxZXk
DxxpEXZpExxpExzpE
XZpxxpxzpEXZpEE
kkkkkkkkkk
kkkk
=∇∇−=
∇∇−∇∇−
∇∇−=
++∇∇−=∇∇−=
++++
++++++
+++++
++
( ){ } ( ) ( ) ( )[ ]{ }
( )[ ]{ } ( )[ ]{ } ( )[ ] 22
0
:1:1,1|11|
:1:1111,1:11:1,1
,ln|ln|ln
,ln|ln|ln,ln:
111111111
1111
kkkT
xxZXkkT
xxxxkkT
xxxz
kkkkkkT
xxZXkkT
xxZXk
DXZpExxpExzpE
XZpxxpxzpEXZpEF
kkkkkkkkkk
kkkk
=
∇∇−∇∇−∇∇−=
++∇∇−=∇∇−=
+++++++++
++++
+++
++++++
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
35
Cramér-Rao Lower Bound (CRLB)
1
2221
1211
1
111
111
111
11
11|1
|
1:11:1|1:1
11|1
|
1:11:1|1:1
,
0
0
:
ˆ
ˆ
ˆ
ˆ
−−
+++
+++
+++
−+
+++
−−−
+++
−−−
+=
=≥
−
−
−
−
−
−
kk
kkkTk
kk
kTk
Tk
kkTk
kkk
k
T
kkk
kkk
kkk
kkk
kkk
kkk
ZX
DD
DDCB
BA
FEL
ECB
LBA
I
xx
xx
XX
xx
xx
XX
E
SOLO
Proof of Theorem 4 (continue – 3):
We found:
[ ] [ ]
+
+
−
+
+
=
−
−−
+
I
DDCB
BAI
DDCB
BADD
DCB
BA
IDCB
BAD
I
Ikkk
Tk
kk
kkkTk
kk
kk
kkTk
kk
kkTk
kk
k
k
0
0
000
0
0
0
12
1
11
12
1
11
2122
11
1
11
211
Therefore: ( ) ( ){ }[ ] [ ] 1211112122
12
1
11
21221
1111|111|1,
00:
ˆˆ
kkkTkkkkk
kkkTk
kk
kkk
kT
kkkkkkZX
DBABDCDDDDCB
BADDJ
JxxxxE
−−
−
+
−+++++++
−+−=
+
−=
≥−−
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
36
Cramér-Rao Lower Bound (CRLB)SOLO
The recursions start with the initial information matrix J0, which can be computed from the initial density p (x0) as follows:
( ) ( ){ } [ ]( ) 11211121221
111|111|1, :ˆˆ−−−
+−−+−−+ +−=≥−− kkkkkkT
kkkkkkZX DDJDDJxxxxE
( ){ }( ){ }
( ){ }kkT
xxZXk
kkT
xXZXk
kkT
XXZXk
XZpEC
XZpEB
XZpEA
kk
kk
kk
:1:1,
:1:1,
:1:1,
,ln:
,ln:
,ln:
1:1
1:11:1
∇∇−=
∇∇−=
∇∇−=
−
−−
Proof of Theorem 4 (continue – 4):
( ) ( ){ } ( ) 111||, :ˆˆ
−−− −=≥−− kkTkkk
TkkkkkkZX BABCJxxxxE
( ){ }( )[ ]{ } [ ]
( ){ } ( ){ }11|1|22
211|
12
1|11
|ln|ln:
|ln:
|ln:
1111111
11
1
+++
+
+
+++++++
++
+
∇∇−∇∇−=
=∇∇−=
∇∇−=
kkTxxxzkk
Txxxxk
T
kkkTxxxxk
kkTxxxxk
xzpExxpED
DxxpED
xxpED
kkkkkkkk
kkkk
kkkk
( ) ( ){ }000 lnln000
xpxpEJ Txxx ∇∇=
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
37
Cramér-Rao Lower Bound (CRLB)SOLO
( ) ( ){ } [ ]( ) 11211121221
111|111|1, :ˆˆ−−−
+−−+−−+ +−=≥−− kkkkkkT
kkkkkkZX DDJDDJxxxxE
Proof of Theorem 4 (continue – 5):
( ){ }( )[ ]{ } [ ]
( ) ( )( ) ( ){ } ( ) ( ){ } nxn
kkTxxxzk
nxnkk
Txxxxk
kkk
nxnT
kkkTxxxxk
nxnkk
Txxxxk
xzpEDxxpED
DDD
DxxpED
xxpED
kkkkkkkk
kkkk
kkkk
RR
R
R
∈∇∇−=∈∇∇−=
+=
∈=∇∇−=
∈∇∇−=
+++
+
+
+++++++
++
+
11|22
1|22
222222
211|
12
1|11
|ln:2|ln:1
21:
|ln:
|ln:
1111111
11
1
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
q.e.d.
( ) [ ] ( )
tMeasuremenUpdated
22
ModelProcessUsingPrediction
1211121221 21: kkkkkkk DDDJDDJ ++−= −
+
Discrete Time Nonlinear Estimation – Recursive Cramér–Rao Lower Bound
38
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation –Special Cases
( ) ( ) ( ) ( )
−−−== −
001
000
0
0000 ˆˆ2
1exp
2
1,ˆ;
0xxPxx
PPxxxp T
xπ
N
( )( ) p
kkk
nkkk
vxhz
wxfx
R
R
∈=
∈= −−
,
, 11 kk vw &1− are system and measurement white-noise sequencesindependent of past and current states and on each other andhaving known P.D.F.s ( ) ( )kk vpwp &1−
( )0xpIn addition the P.D.F. of the initial state , is also given.
Probability Density Function of is Gaussian0x
( ) ( ) ( ) ( )001
0001
0000 ˆˆˆ2
1ln
000xxPxxPxxcxp T
xxx −−=
−−−∇=∇ −−
( ) ( ){ } ( ) ( ) [ ]{ }( ) ( ){ } 1
01
001
01
000001
0
100000
10000
ˆˆ
ˆˆlnln
0
000000
−−−−−
−−
==−−=
−−=∇∇=
PPPPPxxxxEP
PxxxxPExpxpEJT
x
TTxx
Txxxx
Return to Table of Content
39
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation –Special Cases
( ) ( ) ( ) ( )( ) ( )( )
−−−=== +
−++ kkkk
Tkkk
k
kkkwkk xfxQxfxQ
Qwwpxxp 11
11 2
1exp
2
1,0;|
πN
( )( ) p
kkkk
nkkkk
vxhz
wxfx
R
R
∈+=
∈+=
++++
+
1111
1
1& +kk vw are system and measurement Gaussian white-noise sequences, independent of past and current states and on each other with covariances Qk and Rk+1, respectively
( )0xpIn addition the P.D.F. of the initial state , is also given.
Additive Gaussian Noises
( ) ( )( ) ( )( ) ( )[ ] ( )( )kkkkkT
kxkkkkT
kkkxkkx xfxQxfxfxQxfxcxxpkkk
−∇=
−−−∇=∇ +
−+
−++ 1
11
111 2
1|ln
( ) ( ) ( ) ( )( ) ( )( )
−−−=== +++
−++++
++++++ 111
11111
1
11111 2
1exp
2
1,0;| kkkk
Tkkk
k
kkkvkk xhzRxhzR
Rvvpxzpπ
N
( ) ( )( ) ( )( ) ( )[ ] ( )( )1111111111
11111211 111 2
1|ln +++
−++++++
−++++++ −∇=
−−−∇=∇
+++ kkkkkTkxkkkk
Tkkkxkkx xhzRxhxhzRxhzcxzp
kkk
( ) ( )( ) ( )[ ]{ } ( )[ ] 1111 11
|ln −−++ ∇=∇−∇=∇∇
++ kkT
kx
T
kT
kxkT
kkkxkkTxx QxfxfQxfxxxp
kkkkk
( )[ ] ( )[ ]TkTkxk
T
kT
kxk xhHxfFkk 111
:~
&:~
+++∇=∇=
40
Cramér-Rao Lower Bound (CRLB)
SOLO
Discrete Time Nonlinear Estimation –Special Cases
( )( ) p
kkkk
nkkkk
vxhz
wxfx
R
R
∈+=
∈+=
++++
+
1111
1
1& +kk vw are system and measurement Gaussian white-noise sequences, independent of past and current states and on each other with covariances Qk and Rk+1, respectively
( )0xpIn addition the P.D.F. of the initial state , is also given.
Additive Gaussian Noises
( )[ ] ( )( ) ( )( ) ( )[ ]{ } { }1111|111111111
1111|
~~111111 +
−++++
−+++++++
−+++ ++++++
=∇−−∇= kkTkxz
Tk
Tkx
Tk
Tkkkkkkkk
Tkxxz HRHExhRxhzxhzRxhE
kkkkkk
( ){ } ( ){ } { } 1|
1|1|
12 ~|ln
1111
−−+ ++++
−=∇−=∇∇−= kT
kxxkkkTxxxkk
Txxxxk QFEQxfExxpED
kkkkkkkkk
( )[ ] ( )[ ]TkTkxk
T
kT
kxk xhHxfFkk 111
:~
&:~
+++∇=∇=
( ){ } ( ) ( ){ }( )[ ] ( )( ) ( )( ) ( )[ ]{ }
{ }kkT
kxx
T
kT
kxT
kT
kkkkkkkkT
kxxx
kkTxkkxxxkk
Txxxxk
FQFE
xfQxfxxfxQxfE
xxpxxpExxpED
kk
kkkk
kkkkkkkk
~~
|ln|ln|ln:
1|
?
111
|
11|1|11
1
1
11
−
−++
−
+++
+
+
++
=
∇−−∇=
∇∇=∇∇−=
( ) ( ){ } ( ) ( ){ }1111|11|22 |ln|ln|ln:2
11111111 ++++++ ++++++++∇∇=∇∇−= kk
Txkkxxzkk
Txxxzk xzpxzpExzpED
kkkkkkkk
The Jacobians of computed at , respectively.
( ) ( )11& ++ kkkk xhxf
1& +kk xx
( ) ( ){ } ( )( )[ ]{ } 11
1|1|
22
11111|ln:1 −
+−
+ =−∇=∇∇−=+++++ kkkkk
Txxxkk
Txxxxk QxfxQExxpED
kkkkkkk
41
Cramér-Rao Lower Bound (CRLB)
{ }{ }
( )( ) { }1
111|
22
122
1|
12
1|
11
~~2
1
~
~~
11
1
1
+−++
−
−
−
++
+
+
=
=
−=
=
kkTkxzk
kk
kT
kxxk
kkT
kxxk
HRHED
QD
QFED
FQFED
kk
kk
kk
SOLO
Discrete Time Nonlinear Estimation –Special Cases
( )( ) p
kkkk
nkkkk
vxhz
wxfx
R
R
∈+=
∈+=
++++
+
1111
1
1& +kk vw are system and measurement Gaussian white-noise sequences, independent of past and current states and on each other with covariances Qk and Rk+1, respectively
( )0xpIn addition the P.D.F. of the initial state , is also given.
Additive Gaussian Noises
( )[ ] ( )[ ]TkTkxk
T
kT
kxk xhHxfFkk 111
:~
&:~
+++∇=∇= The Jacobians of
computed at , respectively.( ) ( )11& ++ kkkk xhxf
1& +kk xx
( ) [ ] ( )
tMeasuremenUpdated
22
ModelProcessUsingPrediction
1211121221 21: kkkkkkk DDDJDDJ ++−= −
+
We can calculate the expectations using a Monte CarloSimulation. Using we draw ( ) ( ) ( )01 &, xpvpwp kk +( )
( ) ( ) Nivpvwpw
xpx
kikk
ik ,,2,1~&~
~
11
00
=++
We Simulate System States and Measurements( )
( ) Nivxhz
wxfxik
ikk
ik
ik
ikk
ik ,,2,1
1111
1=
+=
+=
++++
+
We then average over realizations to get J0.We average over realization to get next terms and so forth.
0x1x
Return to Table of Content
42
Cramér-Rao Lower Bound (CRLB)
( ) ( ) 1111
22122112111 2&1&& +−
++−−− ==−== kk
Tkkkkk
Tkkkk
Tkk HRHDQDQFDFQFD
SOLO
Discrete Time Nonlinear Estimation –Special Cases
pkkkk
nkkkk
vxHz
wxFx
R
R
∈+=
∈+=
++++
+
1111
1
1& +kk vw are system and measurement Gaussian white-noise sequences, independent of past and current states and on each other with covariances Qk and Rk+1, respectively
( )0xpIn addition the P.D.F. of the initial state , is also given.
Linear/ Gaussian System
( ) ( ) 1111
11
tsMeasuremenUpdated
1111
ModelProcessUsingPrediction
111111 +
−++
−−+
−++
−−−−−+ ++=++−= kk
Tk
Tkkkk
LemmaInverseMatrix
kkTkk
Tkkk
Tkkkkkk HRHFJFQHRHQFFQFJFQQJ
Define ( )Tkkkkkkkkkkkkk FPFQPPJPJ |
1|1
1|
11|11 :&:&: +=== −
+−−
+++
( ) 1111
1|11
111
1
|1
1|1 +−++
−++
−++
−−++ +=++= kk
Tkkkkk
Tk
Tkkkkkkk HRHPHRHFPFQP
The conclusion is that CRLB for the Linear Gaussian Filtering Problem is Equivalent to the Covariance Matrix of the Kalman Filter. Return to Table of Content
43
Cramér-Rao Lower Bound (CRLB)SOLO
Discrete Time Nonlinear Estimation –Special Cases
pkkkk
nkkk
vxHz
xFx
R
R
∈+=
∈=
++++
+
1111
11+kv are measurement Gaussian white-noise sequence,
independent of past and current states with covariance Rk+1.Qk = 0.
( )0xpIn addition the P.D.F. of the initial state , is also given.
Linear System with Zero System Noise
Define ( ) 1
|
01
|11
|1
1|11 :&:&:−=
−+
−−+++ === T
kkkk
Q
kkkkkkkk FPFPPJPJk
( ) 1111
1|11
111
1
|1
1|1 +−++
−++
−++
−−++ +=+= kk
Tkkkkk
Tk
Tkkkkkk HRHPHRHFPFP
Return to Table of Content
44
Cramér-Rao Lower Bound (CRLB)SOLO
References
http://en.wikipedia.org/wiki/Cramer_Rao_bound
Bergman, N., “Recursive Bayesian Estimation - Navigation and Tracking Applications”, PhD Thesis, Linköping University, 1999, Dissertation No. 579, Ch. 4
Van Trees, H., L., “Detection, Estimation and Modulation Theory”, Wiley, New York, 1968, 2001, pp. 146, 66, 72, 79,84,
Tichavský, P., Muravchik, C, Nehorai. A., “Posterior Cramér – Rao bounds for Discrete-Time Nonlinear Filtering”, IEEE Transactions on Signal Processing, 46(5), 1998, pp. 1386 - 1396
Ristic, B., Arulampalam, S., N., Gordon, N., “Beyond the Kalman Filter – Particle Filters for Tracking Applications”, Artech House, 2004, Ch. 4: “Cramér – Rao Bounds for Nonlinear Filtering”
Ristic, B., “Cramér – Rao Bounds for Target Tracking”, Int. Conf. Intelligent Sensors, Sensor Networks and Information Processing, 6 Dec., 2005, http://www.issnip.org/2005/branko_05.pdf
Van Trees, H., L., “Bayesian Bounds”, Keynote Speech, 2005 Adaptive Sensor and Array Processing Workshop, 7 June 2005, http://www.ll.mit.edu/asap/asap_05/pdf/Presentations/01_vantrees.pdf
Return to Table of Content
January 11, 2015 45
SOLO
TechnionIsraeli Institute of Technology
1964 – 1968 BSc EE1968 – 1971 MSc EE
Israeli Air Force1970 – 1974
RAFAELIsraeli Armament Development Authority
1974 – 2013
Stanford University1983 – 1986 PhD AA
46
Harry L. Van Trees
http://teal.gmu.edu/faculty_info/van.html
Harald Cramér1893 – 1985
Cayampudi RadhakrishnaRao
1920 -
Fisher, Sir Ronald Aylmer 1890 - 1962
Branko Ristic
Niclas Bergman
Arye Nehorai
Carlos H. Muravchik
Petr Tichavsky