psychology 290 special topics study course: advanced meta-analysis january 27, 2014
TRANSCRIPT
Psychology 290
Special Topics Study Course: Advanced Meta-analysis
January 27, 2014
Overview
• Discussion of the Hedges / Hanushek exchange.
• Investigation of vote counting as an inferential method.
Education finance exchange
• Larry Hedges was then a professor of educational statistics at the University of Chicago.
• Eric Hanushek was an economist at the University of Rochester.
• Now at Northwestern and Stanford, respectively.
Background of the paper
• Hanushek at the time was extremely active as an expert witness in educational equity lawsuits.
• Paper grew out of a student project in Hedges’ meta-analysis class.
• Discussion.
Vote counting as inference
• To understand vote counting as an inferential method, we need to understand the probability that an individual study will reject the null hypothesis.
• Statisticians have a name for that idea.• Power.
What does power depend on?
• Lots of things: –characteristics of population –choices about how to do inference–characteristics of the sample.
Characteristics of the population
• How strong is the effect? • How much unmodeled variability exists?
Choices about how to do inference
• Alpha level.• One- vs. two-tailed tests.
Characteristics of the sample
• Sample size.
Back to vote counting
• To understand vote counting, we need to understand power.
• We’ve just seen that power is a complex function of lots of factors.
• If we want to understand something that is too complex, what can we do?
• Simplify.
Simplifying
• All of those issues that were characteristics of the population can be simplified by, for the moment, confining our interests to the fixed-effects context.
• In that case, we are assuming that all of the studies are samples from the same population.
Simplifying
• All of the issues that were characteristics of how we do inference are under our control.
• For example, we can simply say that a vote is positive if the null hypothesis is rejected using a two-tailed test with an alpha level of .05.
Simplifying
• The remaining issue that effects power is the sample size of the individual study.
• Obviously, in the real world, different studies will have different N.
• Simplify by assuming that all studies have the same N.
Simplifying
• With these simplifying assumptions, we can treat power (i.e., the probability of a positive vote) as a constant.
Distribution of votes
• We can now think of our studies as a series of independent attempts to vote YES.
• For each attempt, the probability of a YES vote is the same (power).
• We are interested in the total number of YES votes.
• This should sound vaguely familiar.
Distribution of votes
• Suppose that instead of studies and YES votes, I were talking about coin tosses and HEADS outcomes.
• We would be looking at a series of independent coin tosses with a constant probability of success, and would be interested in the probability of a particular number of successes.
Distribution of votes
• With the simplifying assumptions we have made, the number of YES votes follows a binomial distribution.
What should we assume for power?
• Given that Hanushek is arguing that there is no effect, we should be justified in considering it to be “small.”
• Empirical studies of power.
Cohen 1962
• The statistical power of abnormal-social psychological research, Journal of Abnormal and Social Psychology, 65, 145-153.
• Finding: 100% of studies of small effects in that field had power of < .50.
Brewer 1973
• A note on the power of statistical tests in the Journal of Educational Measurement, Journal of Educational Measurement, 10, 71-73.
• Very much the same finding as Cohen.
Understanding vote counting
• What happens with vote counting as the number of studies becomes large?
• (Another digression in R.)• Using the normal approximation to the
binomial distribution.