pk pm jan 2014 power handout

8/13/2019 PK PM Jan 2014 Power Handout

1/14

Power and Sample Size

Sameem Iqbal Siddiqui

Center for Economic Research Pakistan


2/14

Objectives

Introduce the concept

Figure out why we care

Factors affecting power: Understanding theIntuition

Possible ways to calculate power


3/14

Set up

In an experiment in Punjab, we randomly assignself-employed individuals not involved inagriculture to business training. We would like to

see whether this increases their income. From our baseline, we know that mean income for

self-employed individuals is about PKR 11,000.And, we expect the treatment to have an effect of

20% increase on income. Given this setup, what is the sample sizewe would

need to detect the 20% effect?


4/14

What are power calculations?

Power calculations will give you the sample sizerequired to be able to detect the treatment effect. In ourcase, 20%.

Definition of Power: Given a treatment, theprobability of being able to detect an effect.

Studies generally use 80% as a benchmark for power.That is, 80% of the time you will be able to detect the

treatment effect. So, in practice, generally, you will fix the power and see

what sample sizes are needed to achieve that power


5/14

Why do we care?

Being sure ex ante that we are able to detectexpected treatment effects


6/14

Design of Intervention

Main decisions:

1) Level of randomization: Individual or groupo Individual (IND): Assign treatment at the school levelo Group (G): Assign treatment at the village level

2) No of treatments

o Simple T/C oroAlter intensity of treatment oro Combine treatments

3) Compliance100% or lower.

Most important thing to think about is what effects you areinterested in capturing and how the design can help youcapture that while achieving the highest power possible.


7/14

Factors affecting power for IND

Increase in:

Sample size increases power

Size of the effect increases power

Standard deviation decreases power

Significance level decreases power


8/14

Sample Size Calculations for IND

Whats needed Where to get it

Mean of the outcome variable From pilot, baseline or other studies

Standard deviation of the outcome variable From pilot, baseline or other studies

Significance level Usually 5%

Effect size Expected, think about cost-benefitanalysis of program

Power Usually 80%

Level of randomization Depends on study design

Note 1: For group-design, you would need an intracluster coefficient, which tells you the proportion of varianceexplained by the group relative to the overall variance.Note 2: For compliance lower than 100%, we would need to adjust effect size.


9/14

Ways to Calculate Power

Stata commands: sampsi, sampclus

Stata simulations

Stata MDE formula dofile

Excel: MDE formula, set up parameters

Optimal Design software


10/14

Example: Sampsi

Help file: sampsi n1 n2, options

sampsi treatment mean control mean, p(0.8)sd(standard deviation)

This will give you the N needed in each group toachieve 80% power given mean and standarddeviation of outcome variable.


11/14

Sampsi output

Estimated sample size for two-sample comparison of means

Test Ho: m1 = m2, where m1 is the mean in population 1and m2 is the mean in population 2

Assumptions:

alpha = 0.0500 (two-sided)power = 0.8000m1 = 13700.5m2 = 11417.1

sd1 = 15956.8sd2 = 15956.8

n2/n1 = 1.00

Estimated required sample sizes:

n1 = 767n2 = 767

Total Sample: ~1500


12/14

Appendix


13/14

Basic Statistics (1)

What do we mean by detecting an effect?

We are claiming that the mean for the treated isdifferent from the mean of the control.

How do we do this?

A hypothesis test: Test for significance of someclaim

Null Hypothesis: H0: treatment mean =control mean

Alternative Hypothesis: H1: treatment mean !=control mean


14/14

Basic Statistics (2)

Type I Error: Wrongly rejecting the null. That is, the null is correct,but we reject it. Want to keep this as small as possible. P(Type I Error) = Significance level = 5% (traditionally)

Type II Error: Failing to reject a null when it is false

P(Type II Error) = beta, generally unknown Power: 1P(Type II Error) = 1beta

What is the probability that you would be able to reject a false null?I.e. the probability of making the right decision or probability ofnot making a type II error.

Traditionally, we fix this to 80%.

Decision

Reject H0 Do not reject H0

Truth H0 Type I Error Right decision

H1 Right Decision Type II Error

pk pm jan 2014 power handout

Documents