Chapter Nine: Evaluating Results from Samples

• Review of concepts of testing a null hypothesis.

• Test statistic and its null distribution

• Type I and Type II errors

• Level of significance, probability of a Type II error, and power.

Objective is to develop concepts that are useful for planning

studies.• Choose test statistic (usually this is a sample

mean)

• Balance two types of errors by choice of α (probability of a Type I error) and β (probability of a Type II error)

• Determine sample size so that α and β specifications are met.

• Even arbitrary rules have defined α and β.

Statistical Sampling of Populations

• ASSUME there is a defined population and that we want to estimate such parameters as the mean of the population.

• Observe a small number of randomly selected members from the larger population.

• Apply appropriate statistics to the sample to generate correct conclusions.

Why random samples?

• Other sampling procedures have failed consistently.– 1932 Literary Digest Poll.– 1948 Opinion Polls of Outcome of Presidential

race.

• Random samples, correctly drawn, have known properties.– Value of theorems and mathematical argument.

Procedure of Examining a Test of a Hypothesis

• Start with question.– Is a student a random guesser or knowledgeable

about statistics?

• Specify a null hypothesis– H0: Student is a random guesser.

– Null hypothesis should be plausible.– Probability measures generated by null

hypothesis should be easy to compute with.

Procedure of Examining a Test of a Hypothesis

• Specify a statistic, a sample size, and a rule.– Statistic=S4, number of correct answers in a four

question true-false test, with questions of equal difficulty in random order.

– Sample size is the length of the test, 4 questions (an unreasonably short examination).

– Rule is to reject H0 when S4≥4.

– The value 4 is the “critical value”.

• This is an example of a “binomial test.”

Procedure of Examining a Test of a Hypothesis

• Determine the properties of the procedure.– Specify the null distribution of your test

statistic.– Specify the alternative that you want to be able

to detect with low error rate.• Student who has an 80 percent chance of correctly

answering each question.

Type I error

• Definition: to reject the null hypothesis when it is true.

• This example: Call a random guesser a knowledgeable student.

• Can a Type I error happen? – Yes; anytime a random guesser gets 4 or more

correct answers (that is, exactly 4 correct). – Hopefully, the probability of a Type I error is small.

Level of Significance

• Definition: The level of significance is the probability of a Type I error (reject the null hypothesis when it is true) and is usually denoted by α.

• That is, α=Pr0{Reject H0}

• Example problem: α=Pr0{S4≥4}

– Notice that α is a right sided probability.

• Problem is to find α.

Cumulative Distribution Function

• Use the cumulative distribution function (cdf) to get the answer.

• Definition: The cdf of the random variable X at the argument x (FX(x)) is the probability that the random variable X≤x; that is, FX(x)=Pr{X≤x}.

• Use table look-up on reported cdf to get answer.

CDF of Test Statistic under Null and Alternative Distributions

s F0(s) F1(s)

0 0.0625 0.0016

1 0.3125 0.0272

2 0.6875 0.1808

3 0.9375 0.5904

4 1.0000 1.0000

Finding Level of Significance α

• α=Pr0{Reject H0}

• In this problem, α=Pr0{S4≥4}

• This is a right-sided probability.

• Since cdf’s are left-sided probabilities, use the complement principle.

• Pr0{S4≥4}=1-Pr0{S4≤3}=1-F0(3)=1-0.9375

• The answer is 0.0625.

Type II error

• Definition: to accept the null hypothesis when it is false.

• This example: Call a knowledgeable student a random guesser.

• Can a Type II error happen? – Yes; anytime a knowledgeable student makes a

mistake.– Hopefully, the probability of a Type II error is small.

Probability of a Type II error β

• Definition: The probability of a Type II error (accept the null hypothesis when it is false) is usually denoted by β.

• That is, β=Pr1{Accept H0}.

• Example problem: β=Pr1{S4≤3}=F1(3)=0.5904.

Summary of Findings

• The probability of a Type I error α=0.0625

• The probability of a Type II error β=0.5904 for a student who is able to answer 80 percent of the true-false questions correctly.

• The value of α is within reasonable norms.

• The value of β is unreasonably large.

• The test procedure is not satisfactory.

Further Interpretation of error rates

• The Type I error rate α should be small and roughly equal to β.– Minimax argument assuming cost of a Type I error is

same as cost of a Type II error.

• Increasing the sample size while holding α constant will reduce β.

• There is a tradeoff of α and β.• Changing the design of the measuring process

may lead to reduced error rates.

Power

• Definition: The power of a procedure is 1-β.

• Error rates should be small

• Power should be large.

Additional Definitions

• Statistic: a random variable whose value will be completely specified by the observation of an experimental process.

• Parameter: a property of the population being studied, such as its mean or variance.

• Standard error: standard deviation of a statistic.

Observed significance level.

• Definition of observed significance level: a statistic equal to the probability of observing a result as extreme or more extreme from the null hypothesis as observed in the given data.

• Three types of observed significance level: right, left, and two-sided.

Finding the observed significance level in this example.

• A student takes the four question true-false test and has two answers correct. What is the observed significance level?

• What side? Here, right side.

• Apply definition of observed significance level: osl=Pr0{S4≥2}.

Finding the observed significance level in this example

• Use complement rule to get to a left sided probability.

• Pr0{S4≥2}=1- Pr0{S4≤ 1}=1-F1(1).

• The answer is 1-0.3125=0.6875.• This is a statistic whose value is the

probability that a random guesser does as well or better than the student who took the test.

Summary of Lecture

• We have reviewed the basic concepts of tests of hypotheses.

• We have shown how to find the error rates and observed significance level in a binomial test from a tabulation of the cdfs.

• Interpretation point is that error rates should be small. Increase the sample size, change the design, or change the specifications if error rates are too large.