Statistical vs. Practical Significance

Statistical Significance

• Significant differences (i.e., reject the null hypothesis) means that differences in group means are not likely due to sampling error.

• The problem is that statistically significant differences can be found even with very small differences if the sample size is large enough.

Statistical Significance

• In fact, differences between any sample means will be significant if the sample is large enough.

• For example – men and women have different average IQs

Practical Significance

• Practical (or clinical) significance asks the larger question about differences– “Are the differences between samples big

enough to have real meaning.”

• Although men and women undoubtedly have different IQs, is that difference large enough to have some practical implication

Practical Significance

The fifth edition of the APA (2001) Publication Manual states: that it is almost always necessary to include

some index of effect size or strength of relationship in your Results section.… The general principle to be followed … is to provide the reader not only with information about statistical significance but also with enough information to assess the magnitude of the observed effect or relationship. (pp. 25–26)

Practical Significance

• Generally assessed with some measure of effect size

• Effect size can be grouped into two categories:– Difference measures– Variance accounted for measures

Difference effect sizes

• Simple mean difference

• Suppose you design at control group experiment to evaluate the effects of CBT on depression. – Experimental group post test score = 18– Control group post test score = 16– Difference = 18 – 16 = 2

Difference effect sizes

• Problem with simple mean difference– Dependent on the scale of measurement

– Ignores normal variation in scores

– For example, if the following example was based on a scale with a SD of 15 points, a 2 point difference would be small – treatment would only effect depression by .13 SDs.

– If the example was based on a scale with a SD of 1 point, a 2 point difference would be very large – treatment had a 2 SD effect

Difference effect sizes

• We can overcome this problem by standardizing the mean differences

• One measure of this was done by Gene Glass

= (meantx – meancontrol)/ Sdcontrol

• Other SDs may be used such as a pooled (combined) SD from the Tx and Control groups

If variances are equal

2)( 22

12

21

SS

XXd

If variances are unequal

pooled

GG

s

XXd 21

2

11

21

2221

21

nn

nsnsspooled

Difference effect sizes:Interpreting

• Cohen proposed a general method for interpreting these type of effect sizes d = .2 small effect d = .5 medium effect d = .8 large effect

• This is a guideline for interpretation.• You need to interpret effect sizes in the context of

the research

Variance accounted for measures

• When comparing variables, variance accounted for measures tell us how well one variable predicts another or the magnitude of the relation.

• R2 is one such measure from correlational or regression analysis.

• Eta squared (η²) is often used in ANOVA as a measure of shared variance.

• Omega squared (ω2) is also used with ANOVA

Variance accounted for measures:

InterpretingCorrelations can be judged as:R = .1 smallR = .3 moderateR = .5 large

• For measures of variance based on a squared value – take the square root to get a correlation

Confidence Intervals

• Statistics are used to estimate the true population value.

• When providing statistics (estimates of population values) it is useful to provide a range of values that are likely to include the true population value.

• Calculated with the standard error of the statistic

Confidence Intervals for means

Confidence intervals = mean ± z(SEM)Z = 1.96 for a 95% confidence interval

(you can estimate with Z=2 for a 95% confidence interval)

If the mean of a sample = 100 and the SEM = 2

Then a 95% confidence interval would be:

• 100 ± 1.96(2) = 100 ± 3.92• Or 100 ± 2(2) = 100 ± 4 is close enough for govt.

work

Confidence Intervals

• Use confidence intervals when you want to show where some true value is likely to be– Reporting test results