sufficient statistics
DESCRIPTION
Sufficient Statistics. Dayu 11.11. Some Abbreviations. i.i.d. : independent, identically distributed. Content. Estimator, Biased, Mean Square Error (MSE) and Minimum-Variance Unbiased Estimator (MVUE) When MVUE is unique? Lehmann–Scheffé Theorem Biased Complete Sufficient - PowerPoint PPT PresentationTRANSCRIPT
SufficientSufficientStatisticsStatistics
DayuDayu
11.1111.11
Some AbbreviationsSome Abbreviations• i.i.d. : independent, identically
distributed
ContentContent• Estimator, Biased, Mean Square Error
(MSE) and Minimum-Variance Unbiased Estimator (MVUE)When MVUE is unique?
• Lehmann–Scheffé Theorem– Biased– Complete– Sufficient
• the Neyman-Fisher factorization criterion
How to construct MVUE is unique?• Rao-Blackwell theorem
EstimatorEstimator• The probability mass function (or
density) of X is partially unknown, i.e. of the form f(x;θ) where θ is a parameter, varying in the parameter space Θ.
Unbiased• An estimator is said to be unbiased
for a function if it equals in expectation i.e.
• E.g using mean of a sample to estimate mean of the population
is unbiased
t(x)ˆ̂
}ˆ{E
x
n
ii
n
ii
n
ii n
nxE
nxE
nx
nExE
111
1)(
1)(
1)
1()(
Mean Squared Error (MSE)Mean Squared Error (MSE)• MSE of an estimator T of an
unobservable parameter θ isMSE(T)=E[(T- θ)2]
• Since E(Y2)=V(Y)+[E(Y)]2
MSE(T)=var(T)+[bias(T)]2
where bias(T)=E(T- θ)=E(T)- θ• For the unbiased one, MSE=V(T)
since biasd(T)=0
ExamplesExamples
Two estimators for σ2 :
Results from MLE, biased, butsmaller variance
Unbiased, but bigger variance
Minimum-Variance Unbiased Minimum-Variance Unbiased Estimator (MVUE)Estimator (MVUE)
• An unbiased estimator of minimum MSE also has minimum variance.• MVUE is an unbiased estimator of
parameters, whose variance is minimized for all values of the parameters.
• Two theorems– Lehmann-Scheffé theorem can show that MVUE
is unique. – Constructing a MVUE: Rao-Blackwell theorem
Lehmann–Scheffé TheoremLehmann–Scheffé Theorem
• any estimator that is complete, sufficient, and unbiased is the unique best unbiased estimator of its expectation.
• The Lehmann-Scheffé Theorem states that if a complete and sufficient statistic T exists, then the UMVU estimator of g(θ) (if it exists) must be a function of T.
CompletenessCompleteness• Suppose a random variable X has a probability
distribution belonging to a known family of probability distributions, parameterized by θ,
• A function g(X) is an unbiased estimator of zero if the expectation E(g(X)) remains zero regardless of the value of the parameter θ. (by the definition of unbiased)
• Then X is a complete statistic precisely if it admits (up to a set of measure zero) no such unbiased estimator of zero except 0 itself.
Example of CompletenessExample of Completeness• suppose X1, X2 are i.i.d. random variables,
normally distributed with expectation θ and variance 1.
• Not complete: Then X1 — X2 is an unbiased estimator of zero. Therefore the pair (X1, X2) is not a complete statistic.
• Complete: On the other hand, the sum X1 + X2 can be shown to be a complete statistic. That means that there is no non-zero function g such that E(g(X1 + X2 )) remains zero regardless of changes in the value of θ.
Detailed ExplanationsDetailed Explanations
• X1 + X2~(2θ,2)
SufficiencySufficiency• Consider an i.i.d. sample X1, X2,.. Xn
• Two people A and B:– A observe the entire sample X1, X2,.. Xn
– B observes only one number T,T=T(X1, X2,.. Xn)
• Intuitionly, Who has more information?
• Under what condition, B will have as much information about θ as A has?
SufficiencySufficiency• Definition:
– A statistic T(X) is sufficient for θ precisely if the conditional probability distribution of the data X given the statistic T(X) does not depend on θ.
• How to find?: the Neyman-Fisher factorization criterion: If the probability density function of X is f(x;θ), then T satisfies the factorization criterion if and only if functions g and h can be found such that
• h(x): a function that does not depend on θ• g(T(x),θ): a function that depends on data
only throught T(x)• E.g.
• T=x1+x2+.. +xn is a sufficient statistic for p for Bernoulli Distribution B(p)
g(T(x),p)∙1 h(x)=1
Example 2Example 2Test T=x1+x2+.. +xn for Poisson Distribution Π(λ):
g(T(x), λ)h(x): independent of λ
Hence, T=x1+x2+.. +xn is sufficient!
Notes on Sufficient StatisticsNotes on Sufficient Statistics• Note that the sufficient statistic is not
unique. If T(x) is sufficient, so are T(x)/n and log(T(x))
Rao-Blackwell theoremRao-Blackwell theorem• named after
– C.R. Rao (1920- ) is a famous Indian statistician and currently professor emeritus at Penn State University
– David Blackwell (1919-) is Professor Emeritus of Statistics at the UC Berkeley
• describes a technique that can transform an absurdly crude estimator into an estimator that is optimal by the mean-squared-error criterion or any of a variety of similar criteria.
Rao-Blackwell theoremRao-Blackwell theorem• Definition: A Rao–Blackwell estimator δ1(X) of an
unobservable quantity θ is the conditional expected value E(δ(X) | T(X)) of some estimator δ(X) given a sufficient statistic T(X). – δ(X) : the "original estimator"
– δ1(X): the "improved estimator".
• The mean squared error of the Rao–Blackwell estimator does not exceed that of the original estimator.
Conditional ExpectationConditional Expectation
BxBxP
xPBx
BxxP
xfBxxPBxfE
bxfXxB
Bx
)(
)(,0
)|(
)()|()|)((
})(|{
Example IExample I• Phone calls arrive at a switchboard
according to a Poisson process at an average rate of λ per minute.
• λ is not observable• Observe: the numbers of phone calls that
arrived during n successive one-minute periods are observed.
• It is desired to estimate the probability e−λ that the next one-minute period passes with no phone calls.
t=x1+x2+.. +xn is sufficient
Original estimator:
Example IIExample II• To estimate λ for X1 … Xn ~ P(λ)
• Original estimator: X1
We know t= X1 +…+ Xn is sufficient
• Improved estimator by R-B theorem:E[X1| X1 +…+ Xn =t] cannot compute directly
We know Σ[E(Xi| X1 +…+ Xn =t)]
=E(ΣXi| X1 +…+ Xn =t)=t
• Since X1 … Xn are i.i.d. so every term is t/n
In fact, it’s x
Thank you!Thank you!