Bigger, better, faster, more! – Sample size calculationSEMINAR SERIES: HOW TO RUIN YOUR CAREFULLY PLANNED STUDY? TIPS FORIMPROVING DATA ANALYSIS – SESSION 6

TOM SMEKENS

TYPE NAME DEPARTMENT IN WINDOW

Would you believe me if I said…

“Patient centered care is not related to health outcomes, based on:

my sample of 8 physicians”

“No! Your sample is not…”

my sample of 5000 orthopedic surgeons”

“No! Your sample is not…”

TYPE NAME DEPARTMENT IN WINDOW

TYPE NAME DEPARTMENT IN WINDOW

4

“Not representative”

Bias Variance

TYPE NAME DEPARTMENT IN WINDOW

“Not representative”

Bias Variance

TYPE NAME DEPARTMENT IN WINDOW

Estimating sampling variance

After the study: standard errors, p-values, confidence intervals…

Before the study: sample size calculation

TYPE NAME DEPARTMENT IN WINDOW

TYPE NAME DEPARTMENT IN WINDOW

Thought process

Hypothesis AnalysisSample

size

TYPE NAME DEPARTMENT IN WINDOW

Sample size goals

Hypothesis testing:Power

Estimation:Precision

TYPE NAME DEPARTMENT IN WINDOW

Sample size goals

Hypothesis testing:Power

Estimation:Precision

TYPE NAME DEPARTMENT IN WINDOW

“A large enough sample tomake the right conclusionin most cases”

Conclusion determined byStatistical significance

“A large enough sample tosufficiently narrow down an estimate”

Precision expressed usingConfidence intervals

Methods of sample size calculation

Derivation

TYPE NAME DEPARTMENT IN WINDOW

Methods of sample size calculation

Derivation Simulation

TYPE NAME DEPARTMENT IN WINDOW

TYPE NAME DEPARTMENT IN WINDOW

TYPE NAME DEPARTMENT IN WINDOW

TYPE NAME DEPARTMENT IN WINDOW

Contents

1. Power

2. Populations

3. Proportions

TYPE NAME DEPARTMENT IN WINDOW

TYPE NAME DEPARTMENT IN WINDOW

1. The preponderance of power

Ingredients (hypothesis testing)

1. Expected value"We want to reduce mean systolic bloodpressure by 5 mmHg"

2. Scale"68% of the population is in a range of 20 mmHgaround the mean"

3. P-value threshold= 5%

4. Power= 80%

TYPE NAME DEPARTMENT IN WINDOW

Ingredients (hypothesis testing)

1. Expected value"We want to reduce mean systolic bloodpressure by 5 mmHg"

2. Scale"68% of the population is in a range of 20 mmHgaround the mean"

3. P-value threshold= 5%

4. Power= 80%

TYPE NAME DEPARTMENT IN WINDOW

If your expectations are true, theprobability of getting a statisticallysignificant result

Ingredients (hypothesis testing)

1. Expected value"We want to reduce mean systolic bloodpressure by 5 mmHg"

2. Scale"68% of the population is in a range of 20 mmHgaround the mean"

3. P-value threshold= 5%

4. Power= 80%

TYPE NAME DEPARTMENT IN WINDOW

If your expectations are true, theprobability of getting a statisticallysignificant result

TYPE NAME DEPARTMENT IN WINDOW

(Bacchetti et al., 2011)

TYPE NAME DEPARTMENT IN WINDOW

(Bacchetti et al., 2011)

80% Power

TYPE NAME DEPARTMENT IN WINDOW

(Bacchetti et al., 2011)

80% Power

Budget

TYPE NAME DEPARTMENT IN WINDOW

(Bacchetti et al., 2011)

80% Power

Budget

Ingredients (hypothesis testing)

1. Expected value"We want to reduce mean systolic blood

pressure by 15 mmHg"

1. Scale"68% of the population is in a range of 20 mmHgaround the mean"

1. P-value threshold= 5%

2. Power= 80%

TYPE NAME DEPARTMENT IN WINDOW

How much power, really???

For a 15 mmHg difference?

80%

For a 5 mmHg difference?

15%

TYPE NAME DEPARTMENT IN WINDOW

Results from an underpowered study

8 mmHg: Not significant...

19 mmHg: Significant!

TYPE NAME DEPARTMENT IN WINDOW

= Filters out plausible, realistic results in favor of fantastical, noisy ones

Ingredients (precision)

1. Scale"68% of the population is in a range of 20 mmHgaround the mean"

1. Confidence level= 95%

2. Margin of error"We will be able to estimate the effect, give or take 2 mmHg"

TYPE NAME DEPARTMENT IN WINDOW

= half the width of the confidence interval

Mentimeter

How often have you based your sample size calculations around precision?

I never calculate any sample sizes...

TYPE NAME DEPARTMENT IN WINDOW

TYPE NAME DEPARTMENT IN WINDOW

TYPE NAME DEPARTMENT IN WINDOW

2. What about population size?

Inference: generalize from a sample to...

A specific group (usually people) in a given place and time

Finite population

A process (usually biomedical) that is repeatedly observed

Infinite population

TYPE NAME DEPARTMENT IN WINDOW

Finite Population Correction

By default: assume infinite population

Optional: account for the population size

-> Smaller p-values, narrower confidence intervals...

Only noticeable if sample is > 1% of the population

TYPE NAME DEPARTMENT IN WINDOW

TYPE NAME DEPARTMENT IN WINDOW

3. Properly prespecify probabilistic properties ofproportion prerogation

Comparing proportions

Instead of mean systolic blood pressure,compare the prevalence of hypertension

1. Expected valueScale

2. P-value threshold = 5%

3. Power = 80%

TYPE NAME DEPARTMENT IN WINDOW

Scale

"Hypertension prevalence is 30%.We want to reduce it to 25%."

=> Test a difference of 5%?

What about 10% to 5%?

TYPE NAME DEPARTMENT IN WINDOW

Comparing proportions: 25% vs 30%

Comparison Value Regression method

Percentage point difference - 0.05 Linear

Risk ratio 0.83 Log binomial

Odds ratio 0.78 Logistic

TYPE NAME DEPARTMENT IN WINDOW

Comparing proportions: 25% vs 30%

Comparison Value Regression method

Percentage point difference - 0.05 Linear

Risk ratio 0.83 Log binomial

Odds ratio 0.78 Logistic

TYPE NAME DEPARTMENT IN WINDOW

Study result: 8% to 11%

Comparison Value Regression method

Percentage point difference - 0.03 Linear

Risk ratio 0.73 Log binomial

Odds ratio 0.70 Logistic

Non-inferiority trials

TYPE NAME DEPARTMENT IN WINDOW

"Not inferior"

(Ellis et al., 2015)

Non-inferiority margin: 4.5 percentage pointsStudy result: 1.7 percentage pointsBut: risk increased by 30%!

TYPE NAME DEPARTMENT IN WINDOW

Sample size calculation with categorical data

Before the study After the study

TYPE NAME DEPARTMENT IN WINDOW

Conclusion

TYPE NAME DEPARTMENT IN WINDOW

Sample size calculation

Essential in planning a quantitative study

But: fraught with bad habits (even from statisticians!)

Future prospects:

Simulation instead of formulas

Precision instead of power

Questions / Comments?

Next seminar is on December 17

Why eating ice cream doesn’t cause summer – Association and causation

Presenter: Jozefien Buyze

These days, it’s commonly known by researchers that an association doesn't necessarily imply causation. Nevertheless, it’s common to still find this mistake even in high impact papers. In this session, we discuss confounding and interaction (also known as effect-modification) and why finding an association may be very valuable in some settings but not sufficient in others.

32

First name Name Tel/Mobile number E-mail address