Chapter 7Point Estimation

Is there a point to all of this

Chapter 7B

Today in Prob/Stat

This is shaping up to be one terrific

class.

7-3 General Concepts of Point Estimation

7-3.3 Standard Error: Reporting a Point Estimate

Definition

7-3.3 Standard Error: Reporting a Point Estimate

Comment on Notation

Based on review of statistics texts, typical notation does not distinguish between a point estimate definition and its evaluation as a number. Examples:

nSX /ˆ nsX /ˆ Both of these co-exist in texts

1

)(ˆ 1

2

22

n

XXS

n

ii

1

)(ˆ 1

2

22

n

xxs

n

ii

Similarly, you will see both of these in the same text

This is how you will compute it

These are the numbers.

Example 7-5

Example 7-5 (continued)

Mean Square Error (MSE)

Definition:

22 )()ˆ()ˆ()ˆ( biasVEMSE

)ˆ(

)ˆ( efficiency relative

2

1

MSE

MSE

A measure of the worth of an estimator

The MSE assesses the quality of the estimator in terms of its variation and unbiasedness

MSE cont’d

21ˆ ofan that smaller th is ˆ of MSE

Problem 7-14

Which is the better estimate of

n

ii

n

ii X

nXorX

nX

12

2

11

1

2

1

1 2

2 2

1 2

2

1

2

2

2

ˆ12

ˆ 2

E X and E X

V X and V Xn n

MSE nMSE

n

Problem 7-18

1 2 3

21 2 3

ˆ ˆ ˆ( ) ( ) , ( )

ˆ ˆ ˆ( ) 12, ( ) 10, ( ) 6

E E E

V V E

Pick the best of the three estimators of :

7-4 Methods of Point Estimation

Definition

The Method of Moments

1

2 2

1

3 3

1

1

1[ ]

1[ ]

1[ ]

:

1[ ]

n

ii

n

ii

n

ii

np p

ii

X X E Xn

X E Xn

X E Xn

X E Xn

Solve simultaneously for the p unknown parameters

Method of Moments – Normal Probability Distribution

2 2 2

2

2 2 2 21

22 2 22

2 1 1 1

[ ] and [ ]

ˆ ˆ ˆ ˆ; and

ˆ

n

ii

n n n

i i ii i i

E X E X

XX X

n

X X nX X XnX

n n n n

2 2 2 2 2

2 2 2 2 2

using: 2 2

2

i i i i i

i i

x x x x x x x x x nx

x nx nx x nx

Example 7-7 is wrong

A Method of Moments Moment

For the exponential distribution:Since E[X] = 1/

1

1 1ˆ,ˆ n

ii

nX

X X

( ) , 0xf x e x

More Method of Moments MomentsThe Rectangular Distribution

222 2

2 22

22 22

21

; [ ] [ ]2 12

Therefore: [ ]12 2

2 12 2 12

n

ii

b aa bE X E X

b a a bE X

Xb a b aa b a b

X and Xn

Many More Method of Moments Moments

22 22

21

22

22 1

2 12 2 12

12

n

ii

n

ii

Xb a b aa b a b

X and Xn

Xb a

S Xn

22

2 ; 2

12 2

12 2 2

12ˆ 1.7322

12ˆ 2 1.732

2

X a b X b a

S b X b

S X b

b X S X S

a X b X S X S

7-4.2 Method of Maximum Likelihood

Definition

Point Estimation Methods

Max Likelihood Methods – given the observed sample, what is the best set of parameters for the assumed distribution?

),(*...*),(*),()( 21 nxfxfxfL

The max likelihood estimator is the value of that maximizes L(). Maximizing L() is equivalent to maximizing the natural log of L(). Using the log generally gives a simpler function form to maximize.

Maximum Likelihood Estimators

ln L( ,..., ) = 0 ; i = 1,2,...k1 k

i

maximize the log of the likelihood function:

The likelihood function:

n

1 k i 1 ki=1

L( ,..., ) = f( | ,..., )x

Solve k equations for k unknowns

MLE – Geometric Distribution

Let X = a discrete random variable, the number of trials to obtain the first success.

Prob{X=x} = f(x) = (1-p)x-1 p, x = 1, 2, ... , n

= (1- p ) p(1- p ) p (1- p ) p1 2 nx -1 x -1 x -1

= p (1- p )n xii

n

( ) 1

1

( ) 1 2 nL p = f( )f( ) f( )x x x

MLE - Geometric

n

p

x

p

ii

n

( )( )

1

11 01

^

p =

n

n+ ( x -1) =

n

xi=1

n

ii=1

n

i

1

ln ( ) ln ( 1) ln(1 )n

ii

L p n p x p

1( 1)Max

)n

ii

xn L(p) = (1- pp

0 p 1

Example MLE of Geometric

The following data was collected on the number of production runs which resulted in a failure which stopped the production line: 5, 8, 2, 10, 7, 1, 2, 5. Therefore, X = the number of production runs necessary to obtain a failure.

ˆ

i

n

i=1

np =

x

8 = = .240

Prob[X = x] = f(x) = .8 (.2)(x-1)

Mean = 1/p = 40/8 = 5 Pr{X = 3} = .82 (0.2) = 0.128

Exponential MLE

- x jjf( ) = , j = 1, 2,... nx e

j

n- x

j=1

L( ) = e

expn

nj

j=1

= x

ln lnn

j

j=1

L = n - x ln n

jj=1

d L n = - x

d = 0

ˆ 1

j

n

j=1

n =

x

= X

More Exponential MLE

( ) expn

nj

j=1

L = x

Example 7-12

Example 7-12 (continued)

Complications in Using Maximum Likelihood Estimation

• It is not always easy to maximize the likelihood function because the equation(s) obtained may be difficult to solve.

• It may not always be possible to use calculus methods directly to determine the maximum of L().

1( ,..., )0; 1,...,n

j

Lj n

1

1 1

( , ) ( )ixn n

i

i i

xL f t e

1 1

ln , ln ln ( 1) lnn n

ii

i i

xL n n x

1

ln ,0

n

ii

Ln x

1 1

ln ,ln ln ln 0

n ni i

ii i

L x xnn x

Weibull MLE

More Weibull MLE

1/n

ii=1

x=

n

see problem 7-37

1

1

1

lnlnnn

ii ii i=1

n

ii

x x x =

nx

Properties of the MLE

1. MLE’s are invariant: y h y h ( ) ( )

^ ^

then2. MLE’s are Consistent:

as n ,^

3. MLE’s are (best) asymptotically normal:

2 2ˆ

4. Required for certain tests such as the Chi-Square GOF test.5. Has an intuitive appeal.

The Invariance Property

Example 7-13

Now begins a case study in point estimation…

No one is to be admitted once the case study begins You must take your seats – there can be no standing during this part of the presentation Not for the faint-hearted – participate at your own risk No refunds once this part of the presentation begins

MoM and MLE and the Rectangular (uniform) Distribution

Let X1, X2, …,Xn be a random sample from a rectangular distribution where b is unknown.

22Rect(0, ) where and

2 12i

b bX b

method of moments:

2 2 2

ˆset ; 22

ˆ 2 2 22

4ˆ 2 4 412 3

bX b X

bE b E X b

b bVar b Var X Var X

n n n

MoM and MLE and the Rectangular (uniform) Distribution

maximum likelihood estimator:

1 2 ( )1

1( ) ; 0

1 1; max , ,..

nn

n ni

f x x bb

L b b X X X Xb b

( ) 1

1 1

n

nE b E X b

nn b

bias b bn n

2 2

( ) 2 32 1n

nb bVar b Var X

nn n

Compare MoM and MLE for the rectangular

22 )()ˆ()ˆ()ˆ( biasVEMSE

2

2 2

2 2

2 2

ˆ( )3

2 1 1

2

2 1 3

bMSE b

n

nb bMSE b

n n n

b b

n n n

ˆis superior tob b

Let’s Find another Unbiased Estimator

( )

1If ,

1

1let ;n

n nE b b then E b b

n n

nb x E b b

n

2

( )

2 2 2

2 2

1MSE( ) Var

1

2 1 2

n

nb b Var x

n

n nb b

n n n n n

2 2?

?

2compare:

2 1 2

2 1 1 1; 2 1

1

b bMSE b MSE b

n n n n

n

n n n n

What is the best?

The following estimator has about 1/2 the MSE that the MLE has:

( )

2ˆ̂1 n

nb X

n

The overachieving student would use

this one.

A Very Good Summary Statistic is a point estimator of a population parameter.

(S2, s2, s2 ) <=> (Statistic, point estimate, parameter) Bias, variance, mean square error are important properties

of estimators. Relative Efficiency of estimators is ratio of their MSE

Central Limit Theorem – allows the use of normal distr Parameter Estimation

Method of Moments MLE

MLE have some desirable properties. Asymptotically unbiased (examples – uniform

distribution, normal distribution s2) Asymptotic normal distributions. Competitively small variance. Invariance property.

Next Week – Inferential Statistics at its best

Statistical Intervals - confidence- tolerance- prediction