SUMMARY

Homoscedasticity

http://blog.minitab.com/blog/statistics-and-quality-data-analysis/dont-be-a-victim-of-statistical-hippopotomonstrosesquipedaliophobia

Tests for homoscedasticity

• F-test of equality of variances (Hartley's test), • F-test is extremely sensitive to the departures from the

normality.• Alternative test is the Levene's test – performed over

absolute values of the deviations from the mean, test statistic distribution: F-distribution

Power of the test• A probability that it correctly rejects the null hypothesis

(H0) when it is false.• Equivalently, it is the probability of correctly accepting the

alternative hypothesis (Ha) when it is true - that is, the ability of a test to detect an effect, if the effect actually exists.

Decision

Reject H0 Retain H0

State of the world

H0 true Type I error

H0 false Type II error

Probability of FN is β

Probability of FP is α

power = 1 - β

What factors affect the power?

To increase the power of your test, you may do any of the following:

1. Increase the effect size (the difference between the null and alternative values) to be detectedThe reasoning is that any test will have trouble rejecting the null hypothesis if the null hypothesis is only 'slightly' wrong. If the effect size is large, then it is easier to detect and the null hypothesis will be soundly rejected.

2. Increase the sample size(s) – power analysis

3. Decrease the variability in the sample(s)

4. Increase the significance level (α) of the testThe shortcoming of setting a higher α is that Type I errors will be more likely. This may not be desirable.

NEW STUFF

Effect size• When a difference is statistically significant, it does not

necessarily mean that it is big, important or helpful in decision-making. It simply means you can be confident that there is a difference.

• For example, you evaluate the effect of sun erruptions on student knowledge (). • The mean score on the pretest was 84 out of 100. The mean score

on the posttest was 83. • Although you find that the difference in scores is statistically

significant (because of a large sample size), the difference is very small suggesting that erruptions do not lead to a meaningful decrease in student knowledge.

Effect size• To know if an observed difference is not only statistically

significant, but also factually important, you have to calculate its effect size.

• The effect size in our case is 84 – 83 = 1.• The effect size is transformed on a common scale by

standardizing (i.e., the difference is divided by a s.d.).

Power analysis• To ensure that your sample size is big enough, you will

need to conduct a power analysis.• For any power calculation, you will need to know:

• What type of test you plan to use (e.g., independent t-test)• The alpha value (usually 0.05)• The expected effect size• The sample size you are planning to use

• Because the effect size can only be calculated after you collect data, you will have to use an estimate for the power analysis.• Cohen suggests that for t-test values of 0.2, 0.5, and 0.8 represent

small, medium and large effect sizes respectively.

Power analysis in R (paired t-test)install.packages("pwr")

library(pwr)

pwr.t.test(d=0.8,power=0.8,sig.level=0.05,type="paired",alternative="two.sided")

Paired t test power calculation

n = 14.30278

d = 0.8

sig.level = 0.05

power = 0.8

alternative = two.sided

NOTE: n is number of *pairs*

Check for normality – histogram

Check for normality – QQ-plotqqnorm(rivers)qqline(rivers)

Check for normality – tests• The graphical methods for checking data normality still

leave much to your own interpretation. If you show any of these plots to ten different statisticians, you can get ten different answers.

• H0: Data follow a normal distribution.

• Shapiro-Wilk test > shapiro.test(rivers) Shapiro-Wilk normality test

data: rivers W = 0.6666, p-value < 2.2e-16

p-value < 2.2e-16

p-value = 3.945e-05

log

Nonparametric statistics• Small samples from considerably non-normal

distributions.• non-parametric tests

• No assumption about the shape of the distribution.• No assumption about the parameters of the distribution (thus they

are called non-parametric).

• Simple to do, however their theory is extremely complicated. Of course, we won't cover it at all.

• However, they are less accurate than their parametric counterparts.• So if your data fullfill the assumptions about normality, use

paramatric tests (t-test, F-test).

Nonparametric tests• If the normality assumption of the t-test is violated, then its

nonparametric alternative should be used.• The nonparametric alternative of t-test is Wilcoxon test.• wilcox.test()• http://stat.ethz.ch/R-manual/R-patched/library/stats/html/wilcox.test.html

ANOVA (ANALÝZA ROZPTYLU)

A problem• You're comparing three brands of beer.

A problem• You buy four bottles of each brand for the following prices.

• What do you think, which of these brands have significantly different prices?• No significant difference between any of these.• Primátor and Kocour• Primátor and Matuška• Kocour and Matuška

Primátor Kocour Matuška

15 39 65

12 45 45

14 48 32

11 60 38

t-test• We can do three t-tests to show if there is a significant

difference between these brands.• How many t-tests would you need to compare four

samples?• 6

• To compare 10 samples, you need 45 t-tests! This is a lot. We don’t want to do a million t-tests.

• But in this lesson you'll learn a simpler method.• Its called Analysis of variance (Analýza rozptylu) –

ANOVA.

Multiple comparisons problem• If you make two comparisons and assuming that both null

hypothesis are true, what is the chance that both comparisons will not be statistically significant ()?

• And what is the chance that one or both comparisons will result in a statistically significant conclusion just by chance?

• For N comparisons, this probability is generally . • So, for example, for 13 independent tests there is about

50:50 chance of obtaining at least one FP.

Multiple comparisons problem

http://www.graphpad.com/guides/prism/6/statistics/index.htm?beware_of_multiple_comparisons.htm

Bennet et al., Journal of Serendipitous and Unexpected Results, 1, 1-5, 2010

Correcting for multiple comparisons• Bonferroni correction – the simplest approach is to

divide the α value by the number of comparisons N. Then define the particular comparison as statistically significant when its p-value is less than .

• For example, for 100 comparisons reject the null in each if its p-value is less than .

• However, this is a bit too conservative, other approaches exist.

> p.adjust()• “There seems no reason to use the unmodified Bonferroni

correction because it is dominated by Holm's method”

Main idea of ANOVA• To compate three or more samples, we can use the same

ideas that underlie t-tests.• In t-test, the general form of t-statistic is

• Similarly, for three or more samples we assess the variability between sample means in numerator and the error (variability within samples) in denominator.

Variability between sample means

Error, variability within samples

Variability between sample means

Variability within samples

ANOVA hypothesis

at least one pair of samples is significantly different

• Follow-up multiple comparison steps – see which means are different from each other.

F ratio

• As between-group variability (variabilita mezi skupinami) increases, F-statistic increases and this leans more in favor of the alternative hypothesis that at least one pair of means is significantly different.

• As within-group variability (variabilita v rámci skupin) increases, F-statistic decreases and this leans more in favor of the null hypothesis that the means are not siginificantly different.

𝐹=between− group variabilitywithin− group variability

Beer brands – a boxplot

𝑥𝑃

13 45 4835

𝑥𝐾𝑥𝑀𝑥𝐺

Primátor Kocour Matuška

15 39 65

12 45 45

14 48 32

11 60 38

Between-group variability

SS – sum of squares, součet čtvercůMS – mean square, průměrný čtverec

SSB – součet čtverců mezi skupinamiMSB – průměrný čtverec mezi skupinami

𝑥𝑃

13 45 4835

𝑥𝐾𝑥𝑀𝑥𝐺

(𝑥𝑃−𝑥𝐺 )2 (𝑥𝑀−𝑥𝐺 )2

(𝑥𝐾−𝑥𝐺 )2

Within-group variability

𝑀𝑆𝑊= 𝑆𝑆𝑊𝑑 𝑓 𝑊

=∑𝑘

(𝑥 𝑖−𝑥𝑘 )2

𝑁−𝑘

SSW – součet čtverců uvnitř skupinMSW – průměrný čtverec uvnitř skupin

• ... value of each data point• ... sample mean• ... total number of data points• ... number of samples• ... number of data points in each sample• ... grand mean

𝑀𝑆𝑊= 𝑆𝑆𝑊𝑑 𝑓 𝑊

=∑𝑘

(𝑥 𝑖−𝑥𝑘 )2

𝑁−𝑘𝑀𝑆𝐵=𝑆𝑆𝐵

𝑑 𝑓 𝐵

=∑𝑘

(𝑥𝑘−𝑥𝐺 )2

𝑘−1

Primátor Kocour Matuška

15 39 65

12 45 45

14 48 32

11 60 38

The summary of variabilities

F-ratio

F-distribution

F distribution

Beer pricesPrimátor Kocour Matuška

15 39 65

12 45 45

14 48 32

11 60 38

13 48 45𝑥𝑘 𝑥𝐺=35.33

𝑆𝑆𝐵=𝑛∑𝑘

( 𝑥𝑘−𝑥𝐺 )2=3011𝑆𝑆𝑊=∑

𝑘(𝑥 𝑖− 𝑥𝑘 )2=862

𝑑 𝑓 𝐵=𝑘−1=2

𝑑 𝑓 𝑊=𝑁−𝑘=9

𝑀𝑆𝐵=𝑆𝑆𝐵𝑑 𝑓 𝐵

=1505.3

𝑀𝑆𝑊=𝑆𝑆𝑊𝑑 𝑓 𝑊

=95.78

𝐹 2,9=𝑀𝑆𝐵𝑀𝑆𝑊

=15.72𝐹 2,9∗ =4.25

F2,9

F9,2

Beer brands – ANOVA