properties of estimators new update spin

8/2/2019 Properties of Estimators New Update Spin

1/43

PROPERTIES OF ESTIMATORS


2/43

1. Unbiased: E (W) = (centered around truevalue)

2. Consistent: W for large number ofobservations3. Efficient: W1 is better than W2 if

4. Minimum Variance: efficient ANDachieve CRLB

5. Sufficient statistic for

22 )2()1( WEWE


3/43

Let be an estimator for , we saythat is unbiased for if E( )= .If E( )= +b() with b()0, then

is a biased estimator for withbias b().

If one writes of estimation, then anaccurate estimator would be the one resulting insmall estimation errors, so that estimated values willbe near the true value.

error


4/43

If an estimator tn estimates then differencebetween them (tn- ) is called the estimationerror. Bias of the estimator is defined as theexpectation value of this difference:

B=E(tn-)=E(tn)- If the bias is equal to zero then the estimation

is called unbiased. For example sample meanis an unbiased estimator:

0)(1

)(1

)1

()(111

n

i

n

i

i

n

i

i xEn

xEn

xn

ExE


5/43

Here we used the fact that expectation andsummation can change order (Remember thatexpectation is integration for continuous randomvariables and summation for discrete random

variables.) and the expectation of each samplepoint is equal to the population mean.

Knowledge of population distribution was not

necessary for derivation of unbiasedness of thesample mean. This fact is true for the samplestaken from population with any distribution forwhich the first moment exists..


6/43

A biased estimator can be an unbiasedestimator when n is large.

If , then the estimator is

asymptotically unbiased.

lim ( ) 0n

b


7/43

Suppose that X1, X2,, Xn constitute a randomsample from a distribution with mean 1 andstandard deviation 1, and that Y1, Y2,, Yn(independent of Xis) constitute a random sample

from a distribution with mean 2and standarddeviation 2.

Use the rules of expected value to show that X Yis an unbiased estimator of1 -2.

Solution:21

][][][

YEXEYXE


8/43

We have shown that when sampling from

, one finds that the maximum likelihood estimators of andare

Recalling that the distribution of

Thus, we know that the distribution of

Hence,

2

1 2( , )N

2

22

1 2

( 1)

andn S

Xn

2is ( , ), we see that ( ) ;X N E X

n

is unbiased estimator of .X

2 2 2 2

12

1

( 1) 1is where ( )

1

n

n i

i

nS S X X

n

2 2 22 2

2

( 1)( ) ( 1)

1 1

n SE S E n

n n


9/43

Therefore, S2is an unbiased estimator of2

Consequently, since

Therefore,

2 2 2

2

( 1) ( 1) ( 1)( ) [ ] [ ]

n n nE E S E S

n n n

22 2

is a biased estimator of


10/43

Expectation value of the square of the differencesbetween estimator and the expectation of theestimator is called its variance:

Exercise: What is the variance of the sample mean.

As we noted if estimator for is tn then difference

between them is error of the estimation. Expectationvalue of this error is bias. Expectation value of squareof this error is called mean square error (m.s.e.):

2)( ntEM

2))(( nn tEtEV


11/43

It can be expressed by the bias and thevariance of the estimator:

M.S.E is equal to square of the estimators bias plusvariance of the estimator. If the bias is 0 then MSE isequal to the variance.

In estimation it is usually trade of betweenunbiasedness and minimum variance. In ideal worldwe would like to have minimum variance unbiasedestimator. It is not always possible.

)()(

))(())(())()(()()(

2

2222

nn

nnnnnnnn

tBtV

tEtEtEtEtEtEtEtM


12/43


13/43

UNBIASED!!!!


14/43

Let S2 be a sample variance for a random sample from and . Is this

estimator biased or unbiased estimator? Note: meanof chi-square is it df.Solution:

since E(S2)2; therefore S2 is biased estimator and the

biased value is - 2

/n.

1 2, ,..., nX X X

2( , )N

2

2( )

ix x

sn

222 2

1 12

2

2 22

22

2

2

22

( )( 1) ~ ; ~

( )( ) [ ]

( )[ ]

( 1)

i

n n

i

i

x xsn

n

x xE s E

n

x xE

n

nn

n


15/43

If an estimate is such that its biasedconverged to zero as n, then we say that itis asymptotically unbiased.

S2 is asymptotically unbiased when n

22 2

2 2

( )

lim ( ) .n

E s n

E s


16/43

Solution:

2

2

( )If , does this estimator is unbiased

1

estimator of ?

ix x

wn

2

2

2

22

2

2

2

( )

( ) [ ]( 1)

( )[ ]

1

( 1)1

i

i

x x

E w E n

x xE

n

nn

w is unbiased estimator for 2


17/43

THEOREM Let be an unbiased estimator for

based on a random sample of size n,from function , assume we can differentiate

inside the integral whenever needed, then CRLB forthe Var(y) is given by

1 2( , ,..., )ny U x x x

1 2, ,..., nx x x

( ; )f x

2

2

22

2

1

ln ( , )

1

ln ( , )

y

f xnE

f xn E

Cramer RaoLower bound


18/43

Corrollary Suppose y is nor unbiased; say E(y)=+b(),

then CRLB for y is:

2

2

22

2

1 '( )

ln ( , )y

b

f xn E


19/43

Get the CRLB and find the variance for thisestimator

1,0

( ; ) ; ~ ( 1, )

0 ,

x

e xf x y gamma

elsewhere


20/43

Solution:

2

2

2 2 3

2

2 2 3

2 2

2

ln ( ; ) ln

ln ( ; ) 1

ln ( ; ) 1 2

ln ( ; ) 1 2 ( )[ ]

1 2

1

xf x

f x x

f x x

f x E xE E

2

2

2

2

1

1

CRLB:

y

y

n

n

2 2 2

1

( ) 1.

( ) 1.

; ( )

E X

V X

xM x E X

n

x


21/43

Therefore, variance for equal to CRLB.

If an unbiased estimator achive the CRLB;

we say that is efficient for estimating andis called uniformly minimum- variance (orbest) unbiased estimator

2

2

22 2

2 2

( ) ( ) ( )

1

( )

1( )

1 1

i

i

xV V x V

n

V xn

V xn

nn n n


22/43

If we consider all possible unbiased estimator ofsome parameter , the one with the smallestvariance is called the most efficient estimator of .

1

2

3

Sampling distribution of different estimators of

Unbiasedestimators

is the bestestimator of

1


23/43

Let be an estimator that is unbiased for .The relative efficiency for is defined to be:

( ) 1( )

CRLBeffV


24/43

If and are biased estimators of parameter ofa given population, the measure of efficiency ofrelative to is denoted by the ratio

where

1 2

2 2

( ) [( ) ] ( ) ( )MSE E Var b

1

2

2

1 2

1

( ) ( , )( )

MSEeffMSE

Mean square error

CAREFUL HERE


25/43

What it means Conclusion21 2

1

( ) ( , )

( )

MSEeff

MSE

1 2 ( , ) 1eff

1 2 ( , ) 1eff

1 2 ( , ) 1eff

1 2 ( ) ( )MSE MSE

1 2 ( ) ( )MSE MSE

1 2 ( ) ( )MSE MSE

1 2

1 2

1 2

as efficient as

more efficientthan

less efficientthan


26/43

If and are unbiased estimators of parameter of a given population, and , then we saythat is relatively more efficient than .

The measure of efficiency of relative to isdenoted by the ratio

1 2

1 2 ( ) ( )Var Var

1

2

1

2

2

1 21

( ) ( , )

( )

Vareff

Var

CAREFUL HERE


27/43

What it means Conclusion21 2

1

( ) ( , )

( )

Vareff

Var

1 2 ( , ) 1eff

1 2 ( , ) 1eff

1 2 ( , ) 1eff

1 2 ( ) ( )Var Var

1 2 ( ) ( )Var Var

1 2 ( ) ( )Var Var

1 2

1 2

1 2

as efficient as

more efficientthan

less efficientthan


28/43


29/43

Suppose that X1 and X2 are random samplesfrom a normal distribution with mean andstandard deviation . Two estimators formean are as below:

Check whether the two estimators andare unbiased.

Determine the measure of efficiency ofrelative to .

1 21

2

3

X X

1 22

2

X X

1

2

1

2


30/43

Check whether the two estimators and areunbiased.

1 21

1 2

2( )

3

1 2( ) ( )

3 3

1 2

3 3

X XE E

E X E X

1 22

1 2

( )2

1 1( ) ( )

2 2

1 1

2 2

X XE E

E X E X

1

2

Since both and ,then the twoestimators and are unbiased.

1( )E 2( )E

1

2


31/43

Determine the measure of efficiency of relative to

1 21

1 2

2 2

2

2( )

3

1 4( ) ( )9 9

1 4

9 9

59

X XVar Var

Var X Var X

1 2

2

1 2

2 2

2

( )2

1 1( ) ( )4 4

1 1

4 4

12

X XVar Var

Var X Var X

1

2

Since both and ,then the twoestimators and are unbiased.

1( )E 2( )E

1

2


32/43

So, the measure of efficiency of relative to

2

1 2

1

2

2

( , )

2

59

90.9 110

Vareff

Var

1 2

Since that is ,then theestimator is 0.9 times more efficient than .

12

1 2 ( , ) 1eff 1 2

( ) ( )Var Var


33/43

Suppose that X1 , X2 and X3 arerandom samples from a normaldistribution with mean and standard

deviation . Determine the efficiencymeasure of relative to1 2 324

X X X

1 2 3

3

X X X


34/43


35/43


36/43

An unbiased estimator is said to beconsistent if the difference between theestimator and the parameter grows smaller

as the sample size grows larger. E.g. is a consistent estimator of

because: V(X) is

That is, as ngrows larger, the variance of

grows smaller.

X

X


37/43


38/43

Let be a random sample fromfunction f(x;), . Let y1=U( ) be astatistic with pdf g(y1,). We say that y1 is asufficient statistic for the family if and only if

Where for every fix value of thefunc H( ) does not depend on , thusthe conditional pdf of given

is free of

1 2, ,..., nx x x

1 2, ,..., nx x x

( , ),iF f x

1 2 1 21 1

1 2

, ( , )... ( , ) joint pdf for , ,...,

( , ) ( , )

( , ,..., )

n n

n

f x f x f x x x x

g y g y

H x x x

y1=U( )1 2, ,..., nx x x

1 2, ,..., nx x x

1 2, ,..., nx x x 1 1Y y


39/43


40/43

i. find joint pdf for

ii.

Conc: the given func. Is a sufficient statistic.

1 2, ,..., nx x x

1 2

1 1

( , ) ( , )... ( , ) ( , )!

ixn n

n i

i i

ef x f x f x f x

x

1! ( )!!

!

( )! ( )!

ii

i i

xnxn

i ii

x xn ni

i i

eex xx

n xn e n e

x x

Free of


41/43

Note: There is another way to find sufficient statistic by

Neymans Factorization Theorem.

Theorem: Let be a random sample from function

f(x;), . Let y1=U( ) be a statistic of

g(y1,) that y1 is a sufficient statistic for family if and only if there is exist a

function k1(.) if k2(.)where for each fix value of y1=U( ) thefunction k

2( ) does not depend on in

form or in domain.

1 2, ,..., nx x x

1 2, ,..., nx x x

( , ),iF f x 1 1 1 2 2 1 2

1

( ; ) ( ( , ,..., )) ( , ,..., )n

n n

i

f x k U x x x k x x x

1 2, ,..., nx x x

1 2, ,..., nx x x

Is just a statistic


42/43

Let is a r.s from

Solution:

1 2, ,..., nx x x

1

2

(2 )[ (1 )] ,0 1

[ ( )]( ; )

0 ,

Find sufficient statistic for

x x xf x

elsewhere


43/43

properties of estimators new update spin

Documents