Experimental Research Methods in Language

Learning

Chapter 13Paired-Samples and

Independent-Samples T-tests

Leading Questions

• If you were convinced that a particular type of instruction can help learners learn more effectively, how would you maintain objectivity while researching the effectiveness of that type of instruction?

• Can you say that a mean score of 25 is significantly higher than a mean score of 22? Why or why not?

• What do you know about t-tests?

T-tests

• T-tests are parametric tests commonly used in experimental research to compare mean score differences.

• The paired-samples t-test is used to compare scores of a pretest and a posttest.

• The independent-samples t-test is used to compare the pretest or posttest scores between two comparison groups.

Paired-samples T-tests

• A paired-samples t-test can be used in a repeated-measures or within-group experimental design in which researchers compare a pretest with a posttest after an experimental treatment.

• Examples of experimental studies: Al-Homoud and Schmitt (2009), Baralt and Gurzynski-Weiss (2011), Rahimi (2013) and Satar and Özdener (2008)

Statistical Assumptions of the Paired-samples T-test

• Type of Scale: The data should be on a continuous scale such as an interval or ratio scale.

• Random Sampling: Theoretically, the participants should be randomly sampled from the population of interest. In reality, researchers may have to work with a small sample of the population of interest, so they may not be able to randomly sample participants.

• Normal Distribution: The pretest and posttest scores should be normally distributed.

Example of a Paired-samples T-test

• Table 13.1.1 reports the descriptive statistics of each test.

Example of a Paired-samples T-test

• Table 13.1.2 reports the correlation coefficient between the two variables (i.e., r = 0.87).

Example of a Paired-samples T-test

• Table 13.1.3 is where we examine whether there is a statistical significance between the two test scores.

What to Do with the Analysis Output

• Examine the t, df and Sig (2-tailed) columns in this output. Sig (2-tailed) will tell us whether there is a statistical significance between the two scores.

• Compute the effect size such as Cohen’s d. Cohen’s d provides further evidence that will allow us to make a claim about the effect of the experimental treatment. In order to compute Cohen’s d effect size for a paired-samples t-test, you can use the following URL by Melody Wiseheart: <http://www.cognitiveflexibility.org/effectsize/>

Independent-samples t-test

• The logic behind the use of an independent-samples t-test is similar to that of the paired-samples t-test.

• Researchers aim to determine whether one mean is significantly different from another.

• Instead of comparing two means from scores from the same participants, researchers compare two means of scores from two different groups of participants (i.e., independent of each other).

Independent-samples t-test

Examples of experimental studies that have employed anindependent-samples t-test:•Al-Homoud and Schmitt (2009)•Henry, Culman, and VanPatten (2009)•Hirata-Edds (2011)•Macaro and Erler (2008)•Rahimi (2013).

Statistical Assumptions of the Independent-samples T-test

• The assumptions for the independent-samples t-test include those for the paired-samples t-test. However, there are two additional assumptions.

• Group independence: This assumption is that participants belong to only one group.

• Homogeneity of variance: This assumption is that the variances for the two groups are equal.

Example of an Independent-samples T-test

• Table 13.2.1 reports on the descriptive statistics (means, SD, and std error mean) between the two groups.

Example of an Independent-samples T-test

• Table 13.2.2 reports on the Levene’s Test for Equality of Variance. This variance must not be statistically significant in order to make sure that both groups are relatively equal.

Example of an Independent-samples T-test

• In Table 13.2.3, examine the t, df and Sig (2-tailed) columns in this output. Sig (2-tailed) will tell us whether there was a statistical significance between the two groups.

Example of an Independent-samples T-test

• We can use Melody Wiseheart’s effect size website to compute a Cohen’s d effect size. Note that for an independent-samples t-test, we do not need to compute a correlation coefficient.

• It was found that there was a statistically significant difference between the two experimental groups (t(38) = 2.49, p < 0.05, d = 0.80, large effect size).

Discussion

• What are similarities between a paired-samples t-test and an independent-samples t-test?

• What are differences between a paired-samples t-test and an independent-samples t-test?

• Why isn’t statistical significance (e.g., p < 0.05) enough to say about the effect of an independent variable on a dependent variable?