statistical and practical significance
DESCRIPTION
Statistical and Practical Significance. Advanced Statistics Petr Soukup. Outline. Reminder of statistical significance Limits of statistical significance Misuses of statistical significance Alternatives to statistical significance Practical significance Effect sizes. - PowerPoint PPT PresentationTRANSCRIPT
Statistical and Practical Significance
Advanced Statistics
Petr Soukup
L. Rabušic, konference 17.-18. 10. 2002, Brno
Outline
Reminder of statistical significance Limits of statistical significance Misuses of statistical significance Alternatives to statistical significance Practical significance Effect sizes
L. Rabušic, konference 17.-18. 10. 2002, Brno
REMINDER OF STATISTICAL SIGNIFICANCE (NHST)
L. Rabušic, konference 17.-18. 10. 2002, Brno
Hypotheses and tests
Tested hypothesis in experiments (Fisher, 1925)
Null and alternative hypothesis (NHST) (Neyman&Pearson, 1937)
Common tests - t-tests, analysis of variance, analysis of covariance, correlation analysis etc.
L. Rabušic, konference 17.-18. 10. 2002, Brno
Definition of statistical significance
Decision
True status H0 H1
H0 OK (P=1- α) Type I error (P= α)
H1 Type II error (P= β) OK (P= 1-β) Test Power
Definition: Conditional probability, that our sample can be drawn from population in which null hypothesis is valid (α). Statistical significance is P(D/H0) and not P(H0/D)
L. Rabušic, konference 17.-18. 10. 2002, Brno
LIMITS OF NHST
L. Rabušic, konference 17.-18. 10. 2002, Brno
Assumptions for classical NHST
Big big probability samples from infinite or very big finite populations
Three assumptions: Big (infinite) population (at least
100times bigger than the sample) Probability sampling (all units same
probability of selection) Big sample (> 30-50 units)
L. Rabušic, konference 17.-18. 10. 2002, Brno
LIMITS OF NHST
1. data from censuses 2. data from non-probability samples 3. data from small samples 4. data based on sample that are big
proportion of the basic population 5. big data samples from merged
(internationally or by time) files
L. Rabušic, konference 17.-18. 10. 2002, Brno
Beyond the limits of NHST in CSR*
*CSR-Czech sociological review
0% 5% 10% 15%
1-asterisks/testsin aggregated
data
2a-inf. statisticsin quotasampling
2b-weights inquota samples
N=32 articles, Czech sociological review 2000-2006 (selected 29 issues), own research
L. Rabušic, konference 17.-18. 10. 2002, Brno
MISUSES OF NHST
L. Rabušic, konference 17.-18. 10. 2002, Brno
Objections against NHST (Misuses of NHST)a) Insufficient statement about population,
b) null hypotheses are unreal (nill null),
c) mechanical usage of classical 5% statistical significance (asterisks, stepwise methods, best models etc.),
d) statistical significant doesn’t mean important,
e) publishing only statistical significant results (file drawer problem).
L. Rabušic, konference 17.-18. 10. 2002, Brno
Misuses of NHST in CSR*
*CSR-Czech sociological review
0% 10% 20% 30% 40%
C1. asterisks
C2. P=0.000
D. important
E. file drawerproblem
N=32 articles, Czech sociological review 2000-2006 (selected 29 issues), own research
L. Rabušic, konference 17.-18. 10. 2002, Brno
Conference examples (P<0,01)
L. Rabušic, konference 17.-18. 10. 2002, Brno
Conference examples (***)
L. Rabušic, konference 17.-18. 10. 2002, Brno
Conference examples (*** and stepwise)
L. Rabušic, konference 17.-18. 10. 2002, Brno
ALTERNATIVES TO NHST
L. Rabušic, konference 17.-18. 10. 2002, Brno
Some alternatives to statistical significancea) Confidence Intervals (Problems for r,
formulas, regression etc.)b) Test power (quite good in sociology),c) Estimate of minimum sample size & What if
strategy,d) Comparison of models via information
criterias (AIC, BIC)e) Bayesian approach
L. Rabušic, konference 17.-18. 10. 2002, Brno
PRACTICAL SIGNIFICANCE
L. Rabušic, konference 17.-18. 10. 2002, Brno
Practical significance - terminologya) Practical significanceb) Substantive significancec) Logical significanced) Scientific significance sometimes also:e) result importance or f) result meaningfulness
L. Rabušic, konference 17.-18. 10. 2002, Brno
How to measure Practical sig.?
History - Absolute and relative approach
Example: Income differencies
Absolute and relative difference
L. Rabušic, konference 17.-18. 10. 2002, Brno
How to measure Practical sig.?
Effect sizes – measures of practical significance
Some well known:Cohen dHayes ωBut also R2, r, C, Fisher η2 are effect sizes
Problem: Sometimes published but not interpreted
L. Rabušic, konference 17.-18. 10. 2002, Brno
OTHER SIGNIFICANCES
L. Rabušic, konference 17.-18. 10. 2002, Brno
Special significances
Economic significance
Clinical significance
Etc.
L. Rabušic, konference 17.-18. 10. 2002, Brno
CONCLUSION?
Statistical significance is:LIMITEDMISUSEDBUT NOT BAD
Substantive significance is:NOT OFTEN USEDBUT NECESSARY