eece 522 notes_02 ch_1 intro to estimation

7/30/2019 EECE 522 Notes_02 Ch_1 Intro to Estimation

http://slidepdf.com/reader/full/eece-522-notes02-ch1-intro-to-estimation 1/15

1/15

Ch. 1 Introduction to Estimation



2/15

An Example Estimation Problem: DSB RxS ( f )

f o

M ( f )

f f –f o

)2cos()(),;( oooo t f t m f t s φ π φ +=

Audio

Amp

BPF

&

Amp

)()()( t wt st x +=X

)ˆˆ2cos( oot f φ π +

Est. Algo.

Electronics Adds

Noise w(t )

(usually “white”)

oo f φ ̂&ˆ

f

)(ˆ f M

Oscillator

w/ oo f φ ̂&ˆ

Goal: Given

Find Estimates(that are optimal in some sense)

)(),;()( t w f t st x oo += φ Describe with

Probability Model:

PDF & Correlation



3/15

Discrete-Time Estimation Problem

These days, almost always work with samples of the observedsignal (signal plus noise): ][],;[][ nw f nsn x oo += φ

Our “Thought” Model: Each time you “observe” x[n] itcontains same s[n] but different “realization” of noise w[n],

so the estimate is different each time.oo f φ ̂&ˆ are RVs

Our Job: Given finite data set x[0], x[1], … x[ N -1]

Find estimator functions that map data into estimates:

)(])1[,],1[],0[(ˆ

)(])1[,],1[],0[(ˆ

22

11

x

x

g N x x xg

g N x x xg f

o

o

=−=

=−=

…

…

φ

These are RVs…

Need to describe w/

probability model



4/15

PDF of Estimate

Because estimates are RVs we describe them with a PDF…Will depend on:

1. structure of s[n]

2. probability model of w[n]

3. form of est. function g(x))ˆ( o f p

o

f ̂o f

Mean measures centroid

Std. Dev. & Variancemeasure spread

Desire:

( ) small}ˆ{ˆ

}ˆ{

22ˆ =

−=

=

oo f

oo

f E f E

f f E

o

σ



5/15

1.2 Mathematical Estimation Problem

General Mathematical Statement of Estimation Problem:For… Measured Data x = [ x[0] x[1] … x[ N -1] ]

Unknown Parameter = [θ 1 θ 2 … θ p ]

is Not Random

x is an N -dimensional random data vector

Q: What captures all the statistical information needed for anestimation problem ?

A: Need the N -dimensional PDF of the data, parameterized by

);( θx pIn practice, not given PDF!!!

Choose a suitable model

• Captures Essence of Reality• Leads to Tractable Answer We’ll use p(x; ) to find )(ˆ xθ g=



6/15

Ex. Estimating a DC Level in Zero Mean AWGN

]0[]0[ w x +=θ Consider a single data point is observedGaussian

zero mean

variance σ2

~ N (θ , σ2)

So… the needed parameterized PDF is:

p( x[0];θ ) which is Gaussian with mean of θ

So… in this case the parameterization changes the data PDF mean:

θ 1

p( x[0];θ 1)

x[0] θ 2

p( x[0];θ 2)

x[0] θ 3

p( x[0];θ 3)

x[0]



7/15

Ex. Modeling Data with Linear Trend

See Fig. 1.6 in Text

Looking at the figure we see what looks like a linear trend

perturbed by some noise…

So the engineer proposes signal and noise models:

[ ] ][][],;[

nw Bn An x B Ans

++= "#"$%

Signal Model: Linear Trend Noise Model: AWGN

w/ zero mean

AWGN = “Additive White Gaussian Noise”

“White” = x[n] and x[m] are uncorrelated for n ≠ m } Iwwww 2))(( σ =−− T E

http://c/Documents%20and%20Settings/mfowler/My%20Documents/EECE%20522%20Est%20Theory/Class%20Notes/Notes_02%20Figure%201-6.pdf

http://c/Documents%20and%20Settings/mfowler/My%20Documents/EECE%20522%20Est%20Theory/Class%20Notes/Notes_02%20Figure%201-6.pdf



8/15

Typical Assumptions for Noise Model

• W and G is always easiest to analyze

– Usually assumed unless you have reason to believe otherwise

– Whiteness is usually first assumption removed

– Gaussian is less often removed due to the validity of Central Limit Thm

• Zero Mean is a nearly universal assumption

– Most practical cases have zero mean

– But if not… µ += ][][ nwnw zm

Non-Zero Mean of µ Zero Mean Now group into signal model

• Variance of noise doesn’t always have to be known to make an

estimate– BUT, must know to assess expected “goodness” of the estimate

– Usually perform “goodness” analysis as a function of noise variance (or

SNR = Signal-to-Noise Ratio)– Noise variance sets the SNR level of the problem



9/15

Classical vs. Bayesian Estimation Approaches

If we view (parameter to estimate) as Non-Random→ Classical Estimation

Provides no way to include a priori information about

If we view (parameter to estimate) as Random

→ Bayesian Estimation

Allows use of some a priori PDF on

The first part of the course: Classical Methods

• Minimum Variance, Maximum Likelihood, Least Squares

Last part of the course: Bayesian Methods

• MMSE, MAP, Wiener filter, Kalman Filter



10/15

1.3 Assessing Estimator PerformanceCan only do this when the value of is known:

• Theoretical Analysis, Simulations, Field Tests, etc.

is a random variableRecall that the estimate )(ˆ xg

Thus it has a PDF of its own… and that PDF completely displays

the quality of the estimate.

Illustrate with 1-D parameter case

θ θ ̂

)ˆ(θ p

Often just capture quality through mean and variance of )(ˆ xg

Desire:

( ) small}ˆ{ˆ

}ˆ{

22ˆ

ˆ

=

−=

==

θ θ σ

θ θ

θ

θ

E E

E m If this is true:

say estimate is

“unbiased



11/15

Equivalent View of Assessing Performance

)ˆ

(ˆ

ee +=−= θ θ θ θ Define estimation error:

RV RV Not RV

Completely describe estimator quality with error PDF: p(e)

p(e)

e

Desire:

( ){ } small}{

0}{

22 =−=

==

e E e E

e E m

e

e

σ

If this is true:

say estimate is

“unbiased



12/15

Example: DC Level in AWGN

Model: x 1,,1,0],[][ −=+= N nnw An…

Gaussian, zero mean, variance σ2

White (uncorrelated sample-to-sample)

PDF of an individual data sample:

−−=

2

2

2 2

)][(exp

2

1])[(

σ πσ

Ai xi x p

Uncorrelated Gaussian RVs are Independent…so joint PDF is the product of the individual PDFs:

−−=

−−= ∑∏

−

=−

=2

1

0

2

2 / 2

1

02

2

2 2

)][(exp

)2(

1

2

)][(exp

2

1)(

σ πσ σ πσ

N

n

N

N

n

An x An x

p x

( property: prod of exp’s gives sum inside exp )



13/15

Each data sample has the same mean ( A), which is the thing we

are trying to estimate… so, we can imagine trying to estimate A by finding the sample mean of the data:

Statistics ∑−

=

=1

0

][1ˆ

N

n

n x N

A

Prob. Theory

Let’s analyze the quality of this estimator…• Is it unbiased?

A A E

i x E N

n x N

E A E n

A

N

n

=⇒

=

=

∑∑ =

−

=

}ˆ{

]}[{1

][1

}ˆ{1

0#$%

Yes! Unbiased!

N A

N

N

N n x

N n x

N A

N

n

N

n

N

n

2

2

21

0

2

2

1

02

1

0

)ˆvar(

1])[var(

1][

1var)ˆvar(

σ

σ σ

=⇒

===

= ∑∑∑

−

=

−

=

−

=

Can make var small by increasing N!!!

Due to Indep.

(white & Gauss.

⇒ Indep.)

• Can we get a small variance?



14/15

Theoretical Analysis vs. Simulations

• Ideally we’d like to be always be able to theoreticallyanalyze the problem to find the bias and variance of the

estimator

– Theoretical results show how performance depends on the problemspecifications

• But sometimes we make use of simulations

– to verify that our theoretical analysis is correct

– sometimes can’t find theoretical results



15/15

Course Goal = Find “Optimal” Estimators

• There are several different definitions or criteria for optimality!

• Most Logical: Minimum MSE (Mean-Square-Error)

– See Sect. 2.4

– To see this result: ( )

)(}ˆvar{

ˆ)ˆ(

2

2

θ θ

θ θ θ

b

E mse

+=

−=

( )

( ) ( )[ ][ ] { }

)(}ˆvar{

)(}ˆ{ˆ)(}ˆ{ˆ

}ˆ

{}ˆ

{ˆ

ˆ)ˆ(

2

2

0

2

2

2

θ θ

θ θ θ θ θ θ

θ θ θ θ

θ θ θ

b

b E E b E E

E E E

E mse

+=

+−+

−=

−+−=

−=

=

" "#" "$%

θ θ θ −= }ˆ{)( E b

Bias

Although MSE makes sense, estimates usually rely on