Analysis of variance (ANOVA)

One Way ANOVA

Analysis of variance (ANOVA) is a hypothesis-testing procedure that is used to evaluate

mean differences between two or more treatments (or populations).

1. There really are no differences between the populations (or treatments). The observed differences between

the sample means are caused by random, unsystematic factors (sampling error) that differentiate one sample

from another.

2. The populations (or treatments) really do have different means, and these population mean differences are

responsible for causing systematic differences between the sample means.

DEFINITION:In ANOVA, the variable (independent or quasi-independent) that designates the groups being

compared is called a factor.

DEFINITION:The individual conditions or values that make up a factor are called the levels of the factor

DEFINITION:The testwise alpha level is the risk of a Type I error, or alpha level, for an individual

hypothesis test.

DEFINITION:When an experiment involves several different hypothesis tests, the experimentwise alpha

level is the total probability of a Type I error that is accumulated from all of the individual tests in the

experiment. Typically, the experimentwise alpha level is substantially greater than the value of alpha used

for any one of the individual tests.

DEFINITION(BETWEEN-TREATMENTS VARIANCE)Remember that calculating variance is simply a

method for measuring how big the differences are for a set of numbers. When you see the term variance,

you can automatically translate it into the term differences. Thus, the between-treatments variance simply

measures how much difference exists between the treatment conditions. There are two possible explanations

for these between-treatment differences:

1. The differences between treatments are not caused by any treatment effect but are simply the naturally

occurring, random, and unsystematic differences that exist between one sample and another. That is, the

differences are the result of sampling error.

2. The differences between treatments have been caused by the treatment effects

DEFINITION(WITHIN-TREATMENTS VARIANCE)Inside each treatment condition, we have a set of

individuals who all receive exactly the same treatment; that is, the researcher does not do anything that

would cause these individuals to have different scores.

DEFINITION: For ANOVA, the denominator of the F-ratio is called the error term. The error term

provides a measure of the variance caused by random, unsystematic differences. When the treatment effect

is zero (H0 is true), the error term measures the same sources of variance as the numerator of the F-ratio, so

the value of the F-ratio is expected to be nearly equal to 1.00.

Exercises

1. Explain the difference between the testwise alpha level and the experimentwise alpha level.

2. The term “analysis” means separating or breaking a whole into parts. What is the basic analysis that

takes place in analysis of variance?

3. If there is no systematic treatment effect, then what value is expected, on average, for the F-ratio in an

ANOVA?

4. What is the implication when an ANOVA produces a very large value for the F-ratio?

Answers

1. When a single research study involves several hypothesis tests, the testwise alpha level is the value

selected for each individual test and the experimentwise alpha level is the total risk of a Type I error that is

accumulated for all of the separate tests.

2. In ANOVA, the total variability for a set of scores is separated into two components: between-treatments

variability and within-treatments variability.

3. When H0 is true, the expected value for the F-ratio is 1.00 because the top and bottom of the ratio are both

measuring the same variance.

4. A large F-ratio indicates the existence of a treatment effect because the differences between treatments

(numerator) are much bigger than the differences that would be expected if there were no effect

(denominator).

ANOVA Notation and Formulas

The Distribution of F-Ratios

In ANOVA, the F-ratio is constructed so that the numerator and denominator of the ratio are measuring

exactly the same variance when the null hypothesis is true. In this situation, we expect the value of F to be

around 1.00. If we obtain an F-ratio that is much greater than 1.00, then it is evidence that a treatment effect

exists and the null hypothesis is false. The problem now is to define precisely which values are “around

1.00” and which are “much greater than 1.00.” To answer this question, we need to look at all of the

possible F values when H0 is true—that is, the distribution of F-ratios.

Before we examine this distribution in detail, you should note two obvious characteristics:

1. Because F-ratios are computed from two variances (the numerator and denominator of the ratio), F values

always are positive numbers. Remember that variance is always positive.

2. When H0 is true, the numerator and denominator of the F-ratio are measuring the same variance. In this

case, the two sample variances should be about the same size, so the ratio should be near 1. In other words,

the distribution of F-ratios should pile up around 1.00.

With these two factors in mind, we can sketch the distribution of F-ratios. The distribution is cut off at zero

(all positive values), piles up around 1.00, and then tapers off to the right . The exact shape of the F

distribution depends on the degrees of freedom for the two variances in the F-ratio. You should recall that

the precision of a sample variance depends on the number of scores or the degrees of freedom. In general,

the variance for a large sample (large df) provides a more accurate estimate of the population variance.

Because the precision of the MS values depends on df, the shape of the F distribution also depends on the df

values for the numerator and denominator of the F-ratio. With very large df values, nearly all of the F-ratios

are clustered very near to 1.00. With the smaller df values, the F distribution is more spread out.

EXAMPLE 1 :

A researcher is interested in the amount of homework required by different academic majors. Students are

recruited from Biology, English, and Psychology to participate in the study. The researcher randomly selects

one course that each student is currently taking and asks the student to record the amount of out-of-class

work required each week for the course. The researcher used all of the volunteer participants, which resulted

in unequal sample sizes. The data are summarized in Table

EXAMPLE 2 :

A human-factors psychologist studied three computer keyboard designs. Three samples of individuals

were given material to type on a particular keyboard, and the number of errors committed by each

participant was recorded. Are these following data sufficient to conclude that there are significant

differences in typing performance among the three keyboard designs?

Post hoc tests

Post hoc tests (or posttests) are additional hypothesis tests that are done after an ANOVA to determine

exactly which mean differences are significant and which are not. As the name implies, post hoc tests are

done after an ANOVA. More specifically, these tests are done after ANOVA when

1. You reject H0 and

2. There are three or more treatments (k >= 3).

TUKEY’S HONESTLY SIGNIFICANT DIFFERENCE (HSD) TEST

The first post hoc test we consider is Tukey’s HSD test. We selected Tukey’s HSD test because it is a

commonly used test in psychological research. Tukey’s test allows you to compute a single value that

determines the minimum difference between treatment means that is necessary for significance. This value,

called the honestly significant difference, or HSD, is then used to compare any two treatment conditions. If

the mean difference exceeds Tukey’s HSD, then you conclude that there is a significant difference between

the treatments. Otherwise, you cannot conclude that the treatments are significantly different.

Tukey's procedure is only applicable for pairwise comparisons.

It assumes independence of the observations being tested, as well as equal variation across

observations (homoscedasticity or homogeinity of variance) .

THE SCHEFFÉ TEST

Because it uses an extremely cautious method for reducing the risk of a Type I error, the Scheffé test has the

distinction of being one of the safest of all possible post hoc tests (smallest risk of a Type I error). The

Scheffé test uses an F-ratio to evaluate the significance of the difference between any two treatment

conditions. The numerator of the F-ratio is an MSbetween that is calculated using only the two treatments

you want to compare. The denominator is the same MSwithin that was used for the overall ANOVA. The

“safety factor” for the Scheffé test comes from the following two considerations:

1. Although you are comparing only two treatments, the Scheffé test uses the value of k from the original

experiment to compute df between treatments. Thus, df for the numerator of the F-ratio is k-1.

2. The critical value for the Scheffé F-ratio is the same as was used to evaluate the

F-ratio from the overall ANOVA. Thus, Scheffé requires that every posttest satisfy the same criterion that

was used for the complete ANOVA.

The Relationship Between ANOVA and t Tests

This relationship can be explained by first looking at the structure of the formulas for F and t. The t statistic

compares distances: the distance between two sample means (numerator) and the distance computed for the

standard error (denominator). The F-ratio, on the other hand, compares variances. You should recall that

variance is a measure of squared distance. Hence, the relationship: F = t2.

Assumptions for the Independent-Measures ANOVA

The independent-measures ANOVA requires the same three assumptions that were

necessary for the independent-measures t hypothesis test:

1. The observations within each sample must be independent .

2. The populations from which the samples are selected must be normal.

3. The populations from which the samples are selected must have equal variances (homogeneity of

variance).

Ordinarily, researchers are not overly concerned with the assumption of normality, especially when large

samples are used, unless there are strong reasons to suspect that the assumption has not been satisfied. The

assumption of homogeneity of variance is an important one. If a researcher suspects that it has been violated,

it can be tested by Hartley’s F-max test and Levene’s test for homogeneity of variance.

EXERCISES

1.Explain why the F-ratio is expected to be near 1.00 when the null hypothesis is true.

2. Several factors influence the size of the F-ratio. For each of the following, indicate whether it influences

the numerator or the denominator of the F-ratio, and indicate whether the size of the F-ratio would increase

or decrease. In each case, assume that all other factors are held constant.

a. An increase in the differences between the sample means.

b. An increase in the size of the sample variances.

3. Why should you use ANOVA instead of several t tests to evaluate mean differences when an experiment

consists of three or more treatment conditions?

4. Posttests are done after an ANOVA.

a. What is the purpose of posttests?

b. Explain why you do not need posttests if the analysis is comparing only two treatments.

c. Explain why you do not need posttests if the decision from the ANOVA is to fail to reject the null

hypothesis.

5.A researcher reports an F-ratio with df = 2, 27 from an independent-measures research study.

a. How many treatment conditions were compared in the study?

b. What was the total number of participants in the study?

6. A research report from an independent-measures study states that there are significant differences

between treatments, F(3, 48) = 2.95, p< .05.

a. How many treatment conditions were compared in the study?

b. What was the total number of participants in the study?

7. The following summary table presents the results from an ANOVA comparing three treatment conditions

with n= 8 participants in each condition. Complete all missing values.

8. A pharmaceutical company has developed a drug that is expected to reduce hunger. To test the drug, two

samples of rats are selected with n = 20 in each sample. The rats in the first sample receive the drug every

day and those in the second sample are given a placebo. The dependent variable is the amount of food eaten

by each rat over a 1-month period. An ANOVA is used to evaluate the difference between the two sample

means and the results are reported in the following summary table. Fill in all missing values in the table.