How to Derive Mean and Mean

Square Error for an Estimator?

Hing Cheung So (蘇慶祥)

http://www.ee.cityu.edu.hk/~hcso

Department of Electronic Engineering City University of Hong Kong

H. C. So Page 1 Nov. 2013

Outline Introduction

Mean and Mean Square Error Formulas for Scalar

Examples for Scalar Estimation

Mean and Mean Square Error Formulas for Vector

Examples for Vector Estimation

List of References H. C. So Page 2 Nov. 2013

Introduction What is Estimation? Estimation refers to accurately finding the values of parameters of interest from the observed data which consist of two components, viz., signal and noise A generic model for estimation of a scalar is:

where is the observation vector, signal is a known function of and noise is an additive random process

The estimation problem is to find given H. C. So Page 3 Nov. 2013

Why Estimation is Needed? Many science and engineering problems can be boiled down to parameter estimation: Radar Radar system transmits an electromagnetic pulse . It is reflected by an aircraft, causing an echo to be received

H. C. So Page 4 Nov. 2013

Time delay is round trip propagation time of radar pulse. If we know , can be obtained as

, is speed of light H. C. So Page 5 Nov. 2013

Mobile Communications

If we know one-way propagation time of the signal traveling between mobile station and base station (BS), then the target position can be obtained using three BSs

H. C. So Page 6 Nov. 2013

Speech Processing

For a voiced speech, it can be modeled as a periodic signal and it is important to estimate its pitch or fundamental frequency for analysis.

0 0.005 0.01 0.015 0.02

-0.6

-0.4

-0.2

0

0.2

0.4

0.6

0.8

time (s)

vow

el o

f "a"

H. C. So Page 7 Nov. 2013

Image Processing Estimation of the position and orientation of an object from a camera image is useful when using a robot to pick it up, e.g., bomb-disposal Biomedical Engineering Estimation the heart rate of a fetus and the difficulty is that the measurements are corrupted by the mother’s heart beat as well Seismology Estimation of the underground distance of an oil deposit based on sound reflection due to the different densities of oil and rock layers Astronomy Estimation of the periods of orbits H. C. So Page 8 Nov. 2013

How to Perform Estimation? Least squares (LS) and maximum likelihood (ML) are two standard estimation approaches Consider the model of The LS estimator does not require the probability density function (PDF) of , and its estimate is obtained by minimizing a sum of squared error:

where

H. C. So Page 9 Nov. 2013

To produce the ML estimator, the PDF of is required Assuming that is a zero-mean Gaussian noise, the PDF of the observed vector , which is parameterized by , is

where

The ML estimate is:

When is white with variance , the PDF reduces to

ML estimate is reduced to LS solution H. C. So Page 10 Nov. 2013

How to Assess Estimators? Two standard performance measures for assessing accuracy of an estimator are bias and mean square error (MSE): and It is desired that or , indicating that the estimator is unbiased, and MSE is as small as possible For an unbiased estimator, MSE is equal to variance:

In general:

H. C. So Page 11 Nov. 2013

Consider a simple problem of estimating a DC level from: where has mean 0 and variance We easily suggest three estimators:

H. C. So Page 12 Nov. 2013

It is easy to show:

;

;

;

is the best among the three because it has zero bias and

minimum variance Is optimum? How to compute bias and MSE for more general cases? H. C. So Page 13 Nov. 2013

Cramér-Rao Lower Bound (CRLB) CRLB is performance bound in terms of minimum achievable variance provided by any unbiased estimators Its derivation requires knowledge of the noise PDF and the PDF must have closed-form Although there are other variance bounds, CRLB is simplest Suppose the PDF of where , is The CRLB for can be obtained in two steps:

Compute the Fisher information matrix CRLB for is the entry of , H. C. So Page 14 Nov. 2013

has the form of:

H. C. So Page 15 Nov. 2013

Consider with zero-mean white Gaussian noise:

That is, is the optimum estimator for

H. C. So Page 16 Nov. 2013

Mean and Mean Square Error Formulas for Scalar Recall the signal model:

Suppose the scalar is estimated by minimizing a differentiable cost function constructed from , : This implies

H. C. So Page 17 Nov. 2013

At small estimation error conditions, is close to . Applying Tayler series expansion yields: If is sufficiently smooth around , then Hence

Similarly,

H. C. So Page 18 Nov. 2013

Note: When is a quadratic function:

and

When , is an unbiased estimate of For unbiased estimator:

H. C. So Page 19 Nov. 2013

Examples for Scalar Estimation For simplicity, we assume that the noise is white Gaussian process with variance DC Level Estimation Recall the model:

Using the LS approach, the cost function to be minimized is

H. C. So Page 20 Nov. 2013

To apply the bias and MSE formulas, we compute:

and

Hence:

and

which align with previous analysis H. C. So Page 21 Nov. 2013

Time-Difference-of-Arrival Estimation The simplest model is to estimate the time-difference-of-arrival between two signals: where , and are independent zero-mean white Gaussian variables with , It is clear that is most similar to

. As a result, can be estimated by maximizing the cross-correlation between and :

H. C. So Page 22 Nov. 2013

However, is generally not an integer and thus is a continuous function of Using the convolution theorem, has the form of

Applying the bias and MSE formulas, we obtain:

and

which is also the CRLB H. C. So Page 23 Nov. 2013

Frequency Estimation of a Complex Sinusoid The signal model is:

where , and

A conventional approach for estimating is to search the periodogram peak:

Applying the bias and MSE formulas, we obtain:

and

which is also the CRLB H. C. So Page 24 Nov. 2013

Frequency Estimation of a Real Sinusoid The signal model is:

where , and According to the linear prediction property:

A LS cost function for estimating is then:

H. C. So Page 25 Nov. 2013

The LS estimate of is:

Hence the frequency estimate is

which is known as the modified covariance method H. C. So Page 26 Nov. 2013

Applying the bias and MSE formulas, we obtain: and

if is sufficiently large Hence

where H. C. So Page 27 Nov. 2013

With tedious calculation, we have

Since

We have:

H. C. So Page 28 Nov. 2013

Mean and Mean Square Error Formulas for Vector For estimation of a vector from minimizing , the formulas are generalized as follows:

and

where is the gradient vector and is the Hessian matrix:

H. C. So Page 29 Nov. 2013

As a result, Similarly, the covariance matrix is:

The MSE of is given by entry of H. C. So Page 30 Nov. 2013

Examples for Vector Estimation Estimation of a Linear Model

The linear data model is:

where is known, is unknown vector, and

Employing , the weighted LS cost function is:

Applying the bias and MSE formulas, we obtain:

and

These align with the best linear unbiased estimator (BLUE) H. C. So Page 31 Nov. 2013

Parameter Estimation of a Real Sinusoid

The signal model is:

where , and , while is a white Gaussian process with variance According to ML or LS, we construct:

H. C. So Page 32 Nov. 2013

Applying the bias and MSE formulas, we obtain:

which is the inverse of the Fisher information matrix That is, the estimator is optimum H. C. So Page 33 Nov. 2013

Localization using Range Measurements Consider positioning of a source at by sensors at known coordinates , If we have the one-way propagation time measurements, they can be easily converted to ranges: where and is white The ML or LS cost function is

H. C. So Page 34 Nov. 2013

To determine the bias and MSE, the steps include:

because Similarly,

H. C. So Page 35 Nov. 2013

As a result,

With tedious calculation, we have

which is the inverse of the Fisher information matrix That is, the estimator is optimum H. C. So Page 36 Nov. 2013

Apart from the nonlinear approach, can be linearized:

where

Hence the signal model is now linear:

where

H. C. So Page 37 Nov. 2013

For sufficiently small noise conditions, we have

H. C. So Page 38 Nov. 2013

The weighted LS cost function to be minimized is

and the estimate is

Applying the bias and MSE formulas, we obtain:

MSEs of and are given by and entries of

H. C. So Page 39 Nov. 2013

To achieve higher accuracy, the information of should be utilized, which results in a constrained optimization problem:

The solution can be derived using the method of Lagrange multipliers To analyze the performance, the constrained problem can be converted to an unconstrained one by putting the relation of into :

H. C. So Page 40 Nov. 2013

Applying the bias and MSE formulas, we obtain:

which is the inverse of the Fisher information matrix That is, the estimator is also optimum H. C. So Page 41 Nov. 2013

List of References [1] H.C. So, Y.T. Chan, K.C. Ho and Y. Chen, “Simple

formulas for bias and mean square error computation,” IEEE Signal Processing Magazine, vol.30, no.4, pp.162-165, Jul. 2013

[2] S.M. Kay, Fundamentals of Statistical Signal Processing: Estimation Theory, Prentice-Hall, NJ: Englewood Cliffs, 1993

[3] V.H. MacDonald and P.M. Schultheiss, “Optimum passive bearing estimation in a spatially incoherent noise environment,'” Journal of the Acoustical Society of America, vol.46, no.1, pt.1, pp.37-43, Jul. 1969

[4] H.C. So, Y.T. Chan and F.K.W. Chan, “Closed-form formulae for optimum time difference of arrival based localization,” IEEE Transactions on Signal Processing,

H. C. So Page 42 Nov. 2013

vol.56, no.6, pp.2614-2620, Jun. 2008 [5] K.W.K. Lui and H.C. So, “Performance of modified

covariance estimator for a single real tone,” IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, vol.E90-A, no.9, pp.2021-2023, Sep. 2007

[6] K.W. Cheung, H.C. So, W.-K. Ma and Y.T. Chan, “Least squares algorithms for time-of-arrival based mobile location,” IEEE Transactions on Signal Processing, vol.52, no.4, pp.1121-1128, Apr. 2004

[7] H.C. So, “Source Localization: Algorithms and Analysis,” Handbook of Position Location: Theory, Practice and Advances, Chapter 2, S.A. Zekavat and M. Buehrer, Eds., Wiley-IEEE Press, 2011

H. C. So Page 43 Nov. 2013