P Values - part 4The P value and ‘rules’

Robin Beaumont10/03/2012

With much help from

Professor Geoff Cumming

Putting it all together

P Valuesampling

probability

statistic

Rules Alternatives

Summary so far• A P value is a conditional probability which considers a range of

outcomes = ‘area’ in a PDF graph .• The SEM formula allows us to: predict the accuracy of your

estimated mean across a infinite number of samples!• Taking another random sample will give us a different P value• How different? - Does not follow a normal distribution• Dance of the p values – Geoff Cumming • Depends upon if the null hypothesis is actually true in reality!

Remember we have assumed so far that the null hypothesis is true.

Review

Rules

t density: sx = 9.037 n =15

0

12096

-2.656t 2.656

Shaded area=0.0188

Original units:

0

Rule -> If our result has a P value more extreme than our level of acceptability = our critical valueReject the parameter value we based the p value on. = reject the null hypothesis

Given that the sample was obtained from a population with a mean (i.e. parameter value) of 120 a sample with a T(n=15) statistic of -2.656 or 2.656 or one more extreme with occur 1.8% of the time. This is less than one in twenty (i.e. P value < 0.05). Mark the one in twenty value = the critical value for T(n=15) = 2.144= alpha (α) level =0.05)

Therefore we dismiss the possibility that our sample came from a population with a mean of 120

Rules

Say one in twenty 1/20 =Or 1/100

Or 1/1000or . . . .

What value do we now give the parameter value ?

When p value is in the critical region Reject the parameter value we based the p value on is considered untenable = reject the null hypothesis

What is the mean of the population if we have now ruled out one value?

Set a level of acceptability = critical value (CV)

• Allows decision making • Moves thing forward• If the sample did not come from the ‘null distribution’ indicates there is some

effect• Population mean of 120 for substance X indicates they have a propensity to a

range of nasty diseases• Given them miracle drug Y which is believed to reduce the level has this have

any effect in our sample of 15?• If the probability of obtaining our t value (i.e. + associated p value) is below a

certain critical threshold we can say that our sample does not come from the null distribution and there is a effect.

Why do we want to reject the null distribution?

Fisher – only know and only consider the model we have i.e. The parameter we have used in our model – when we reject it we accept that any value but that one can replace it.

Neyman and Pearson + Gossling

Bayesians

Hnull = μ=120 versus alternative Halt = μ≠120 [μ = population mean]Hnull = T= 0 versus alternative Halt = T≠ 0 [T = t statistic]

Hnull = μ=120 versus alternative Halt = μ = XXX [μ = population mean]Hnull = T= 0 versus alternative Halt = T = xxx [T = t statistic]

Fisher – infinitely many alternative ‘red’ distributions

View 1 - The infinite variety of alternative distributions - Fisher

Alternative distributions:

Become flatter further away from null

When coincides with null= same shape

How do we define the alternative hypothesis?Distance measures: Effect size / Non centrality Parameter

Become more asymmetrical as further away from null

Null distribution t(df, ncp=0) Alternative distributions t(df, ncp)

View 2 - The single specified alternative – Neyman + Pearson•Take the distribution around the sample value •Then work our the difference SD’s:

Delta = Δ Non centrality parameter = capital = d x √15 = -.6857 x 3.872 = -2.6557

Population mean of 120 for substance X, SD= 35 units/100 ml. 15 random subjects take miracle drug have a mean = 96

Gpower shows it all!

Population mean of 120 for substance X, SD= 35 units/100 ml. 15 random subjects take miracle drug have a mean = 96

Red line = null distributionRed areas = critical regions = α alpha regionsBlue line = specified alternative distributionBlue shaded area = β beta regions (far right– very small)

α = the reject region

= 120= 96

Correct decisions

incorrect decisions

Two correct and 2 incorrect decisions

Correct decisions

Power = 1 - Beta

Power is good

Insufficient power – unlikely

to get a p value in the critical region

Too much power always p value in

critical region but possibly trivial effect

size

More Power the better up unto a certain point!

Cumming - 2008 Replication and P intervalsRed line = null distributionRed areas = critical regions = α alpha regionsBlue line = specified alternative distributionBlue shaded area = β beta regions (far right– very small)

ImplicationsPower and Cohens effect size measure d reflect one another. When power high p value distribution does provides a measure of evidence for the specific alternative hypothesis.Computer simulations - reality differentReplication is a vitally important research strategy – meta analysis.Power analysis during study design and Confidence intervals along with effect size measures when reporting results – alternative strategy?

Always specify specific alternative hypothesiswww.robin-beaumont.co.uk/virtualclassroom/contents.html

Students bloomers

• The p value did not indicate much statistic significance

• Given that the population comes from one population

• The p value is 0.003 thus rejecting the null hypothesis and there is a statistical significance

• Correlation = 0.25 (p<0.001) indicating that assuming that the data come from a bivariate normal distribution with a correlation of zero you would obtain a correlation of <0.000. There is 95% chance that the relationship among the variables is not due to chance