tests for binary/categorical outcomes

Tests for Binary/Categorical outcomes

Binary or categorical outcomes (proportions)

Outcome Variable

Are the observations correlated? Alternative to the chi-square test if sparse cells:

independent correlated

Binary or categorical(e.g. fracture, yes/no)

Chi-square test: compares proportions between more than two groups

Relative risks: odds ratios or risk ratios

Logistic regression: multivariate technique used when outcome is binary; gives multivariate-adjusted odds ratios

McNemar’s chi-square test: compares binary outcome between correlated groups (e.g., before and after)

Conditional logistic regression: multivariate regression technique for a binary outcome when groups are correlated (e.g., matched data)

GEE modeling: multivariate regression technique for a binary outcome when groups are correlated (e.g., repeated measures)

Fisher’s exact test: compares proportions between independent groups when there are sparse data (some cells <5).

McNemar’s exact test: compares proportions between correlated groups when there are sparse data (some cells <5).


Outcome Variable





Relative risks: odds ratios or risk ratios (for 2x2 tables)







Chi-square test

Probiotics group

Placebo group

p-value

Adjusted OR(95% CI)

p-value

Cumulative incidence at 12 months

12/33 (36.4%)

22/35 (62.9%)

0.029*

0.243(0.075–0.792) 0.019†

*Significant difference between the groups as determined by Pearson's chi-square test. †p value was calculated by multivariable logistic regression analysis adjusted for the antibiotics use, total duration of breastfeeding, and delivery by cesarean section.

Kim et al. Effect of probiotic mix (Bifidobacterium bifidum, Bifidobacterium lactis, Lactobacillus acidophilus) in the primary prevention of eczema: a double-blind, randomized, placebo-controlled trial. Pediatric Allergy and Immunology. Published online October 2009.

Table 3. Cumulative incidence of eczema at 12 months of age

From an RCT of probiotic supplementation during pregnancy to prevent eczema in the infant:

http://www3.interscience.wiley.com/journal/118512964/home

Chi-square testStatistical question: Does the proportion of

infants with eczema differ in the treatment and control groups?

What is the outcome variable? Eczema in the first year of life (yes/no)

What type of variable is it? Binary Are the observations correlated? No Are groups being compared and, if so, how

many? Yes, two groups Are any of the counts smaller than 5? No,

smallest is 12 (probiotics group with eczema) chi-square test or relative risks, or both

Chi-square test of Independence

Chi-square test allows you to compare proportions between 2 or more groups (ANOVA for means; chi-square for proportions).

Example 2

Asch, S.E. (1955). Opinions and social pressure. Scientific American, 193, 31-35.

The Experiment

A Subject volunteers to participate in a “visual perception study.”

Everyone else in the room is actually a conspirator in the study (unbeknownst to the Subject).

The “experimenter” reveals a pair of cards…

The Task Cards

Standard line Comparison lines

A, B, and C

The Experiment Everyone goes around the room and says

which comparison line (A, B, or C) is correct; the true Subject always answers last – after hearing all the others’ answers.

The first few times, the 7 “conspirators” give the correct answer.

Then, they start purposely giving the (obviously) wrong answer.

75% of Subjects tested went along with the group’s consensus at least once.

Further Results

In a further experiment, group size (number of conspirators) was altered from 2-10.

Does the group size alter the proportion of subjects who conform?

The Chi-Square test

Conformed?

Number of group members?

2 4 6 8 10

Yes 20 50 75 60 30

No 80 50 25 40 70

Apparently, conformity less likely when less or more group members…

20 + 50 + 75 + 60 + 30 = 235 conformed

out of 500 experiments.

Overall likelihood of conforming = 235/500 = .47

Expected frequencies if no association between group size and conformity…

Conformed?

Number of group members?

2 4 6 8 10

Yes 47 47 47 47 47

No 53 53 53 53 53

Do observed and expected differ more than expected due to chance?

Chi-Square test

expected

expected) - (observed 22

8553

)5370(

53

)5340(

53

)5325(

53

)5350(

53

)5380(

47

)4730(

47

)4760(

47

)4775(

47

)4750(

47

)4720(

22222

222222

4

Degrees of freedom = (rows-1)*(columns-1)=(2-1)*(5-1)=4

Chi-Square test

expected

expected) - (observed 22

Rule of thumb: if the chi-square statistic is much greater than it’s degrees of freedom, indicates statistical significance. Here 85>>4.

8553

)5370(

53

)5340(

53

)5325(

53

)5350(

53

)5380(

47

)4730(

47

)4760(

47

)4775(

47

)4750(

47

)4720(

22222

222222

4

Degrees of freedom = (rows-1)*(columns-1)=(2-1)*(5-1)=4

Interpretation Group size and conformity are not

independent, for at least some categories of group size

The proportion who conform differs between at least two categories of group size

Global test (like ANOVA) doesn’t tell you which categories of group size differ

Caveat

**When the sample size is very small in any cell (<5), Fisher’s exact test is used as an alternative to the chi-square test.

Review Question 1I divide my study population into smokers, ex-smokers, and never-smokers; I want to compare years of schooling (a normally distributed variable) between the three groups. What test should I use?

a. Repeated-measures ANOVA.b. One-way ANOVA.c. Difference in proportions test.d. Paired ttest.e. Chi-square test.

Review Question 2I divide my study population into smokers, ex-smokers, and never-smokers; I want to compare the proportions of each group that went to graduate school. What test should I use?

a. Repeated-measures ANOVA.b. One-way ANOVA.c. Difference in proportions test.d. Paired ttest.e. Chi-square test.


Outcome Variable












Risk ratios and odds ratios

Probiotics group

Placebo group

p-value

Adjusted OR(95% CI)

p-value

Cumulative incidence at 12 months

12/33 (36.4%) 22/35 (62.9%)

0.029* 0.243(0.075–0.792) 0.019†

*Significant difference between the groups as determined by Pearson's chi-square test. †p value was calculated by multivariable logistic regression analysis adjusted for the antibiotics use, total duration of breastfeeding, and delivery by cesarean section.

Kim et al. Effect of probiotic mix (Bifidobacterium bifidum, Bifidobacterium lactis, Lactobacillus acidophilus) in the primary prevention of eczema: a double-blind, randomized, placebo-controlled trial. Pediatric Allergy and Immunology. Published online October 2009.

Table 3. Cumulative incidence of eczema at 12 months of age

From an RCT of probiotic supplementation during pregnancy to prevent eczema in the infant:

http://www3.interscience.wiley.com/journal/118512964/home

Corresponding 2x2 table

Treatment Placebo

+ 12 22

- 21 13

Treatment Group

Eczema

Risk ratios and odds ratiosStatistical question: Does the proportion of

infants with eczema differ in the treatment and control groups?

What is the outcome variable? Eczema in the first year of life (yes/no)

What type of variable is it? Binary Are the observations correlated? No Are groups being compared and, if so, how

many? Yes, binary Are any of the counts smaller than 5? No,

smallest is 12 (probiotics group with eczema) chi-square test or relative risks, or both

Odds vs. Risk (=probability)

If the risk is… Then the odds are…

½ (50%)

¾ (75%)

1/10 (10%)

1/100 (1%)

Note: An odds is always higher than its corresponding probability, unless the probability is 100%.

1:1

3:1

1:9

1:99

Risk ratios and odds ratios

Absolute risk difference in eczema between treatment and placebo: 36.4%-62.9%=-26.5% (p=.029, chi-square test).

Risk ratio:

Corresponding odds ratio:

58.0%9.62

%4.36

34.0%)9.621/(%9.62

%)4.361/(%4.36

There is a 26.5% decrease in absolute risk, a 42% decrease in relative risk, and a 66% decrease in relative odds.

Why do we ever use an odds ratio?? We cannot calculate a risk ratio from a

case-control study (since we cannot calculate the risk of developing the disease in either exposure group).

The multivariate regression model for binary outcomes (logistic regression) gives odds ratios, not risk ratios.

The odds ratio is a good approximation of the risk ratio when the disease/outcome is rare (~<10% of the control group)

Interpretation of the odds ratio:

The odds ratio will always be bigger than the corresponding risk ratio if RR >1 and smaller if RR <1 (the harmful or protective effect always appears larger)

The magnitude of the inflation depends on the prevalence of the disease.

The rare disease assumption

RROR EDPEDP

EDPEDP

EDPEDP

)~/()/(

)~/(~)~/(

)/(~)/(

1

1

When a disease is rare: P(~D) = 1 - P(D) 1

The odds ratio vs. the risk ratio

1.0 (null)

Odds ratio

Risk ratio Risk ratio

Odds ratio

Odds ratio

Risk ratio Risk ratio

Odds ratio

Rare Outcome

Common Outcome

1.0 (null)

When is the OR is a good approximation of the RR?

General Rule of Thumb:

“OR is a good approximation as long

as the probability of the outcome in the

unexposed is less than 10%”


Outcome Variable



Binary or categorical(e.g. patency, revision)









Recall… Split-face trial:

Researchers assigned 56 subjects to apply SPF 85 sunscreen to one side of their faces and SPF 50 to the other prior to engaging in 5 hours of outdoor sports during mid-day.

Sides of the face were randomly assigned; subjects were blinded to SPF strength.

Outcome: sunburn

Russak JE et al. JAAD 2010; 62: 348-349.

Results:

Table I -- Dermatologist grading of sunburn after an average of 5 hours of skiing/snowboarding (P = .03; Fisher’s exact test)

Sun protection factor Sunburned Not sunburned

85 1 55

50 8 48

The authors use Fisher’s exact test to compare 1/56 versus 8/56. But this counts individuals twice and ignores the correlations in the data!

McNemar’s testStatistical question: Is SPF 85 more effective than

SPF 50 at preventing sunburn? What is the outcome variable? Sunburn on half

a face (yes/no) What type of variable is it? Binary Are the observations correlated? Yes, split-face

trial Are groups being compared and, if so, how

many? Yes, two groups (SPF 85 and SPF 50) Are any of the counts smaller than 5? Yes,

smallest is 0 McNemar’s test exact test (if bigger numbers,

would use McNemar’s chi-square test)

Correct analysis of data…

Table 1. Correct presentation of the data from: Russak JE et al. JAAD 2010; 62: 348-349. (P = .016; McNemar’s test).

SPF-50 side

SPF-85 side Sunburned Not sunburned

Sunburned 1 0

Not sunburned 7 48

Only the 7 discordant pairs provide useful information for the analysis!

McNemar’s exact test… There are 7 discordant pairs; under the

null hypothesis of no difference between sunscreens, the chance that the sunburn appears on the SPF 85 side is 50%.

In other words, we have a binomial distribution with N=7 and p=.5.

What’s the probability of getting X=0 from a binomial of N=7, p=.5?

Probability =

Two-sided probability =

0078.5.5. 077

0

0156.0078.5.5.0078.5.5. 707

7

077

0

McNemar’s chi-square test Basically the same as McNemar’s

exact test but approximates the binomial distribution with a normal distribution (works well as long as sample sizes in each cell >=5)


Outcome Variable



Binary or categorical(e.g. patency, revision)




McNemar’s test: compares binary outcome between correlated groups (e.g., before and after)





Recall: Political party and drinking…

Drinking by political affiliation

Recall: Political party and alcohol…

This association could be analyzed by a ttest or a linear regression or also by logistic regression:

Republican (yes/no) becomes the binary outcome.

Alcohol (continuous) becomes the predictor.

Logistic regression Statistical question: Does alcohol drinking

predict political party? What is the outcome variable? Political

party What type of variable is it? Binary Are the observations correlated? No Are groups being compared? No, our

independent variable is continuous logistic regression

The logistic model…

ln(p/1- p) = + 1*X

Logit function

=log odds of the outcome

The Logit Model (multivariate)

)...()())(1

)(ln( 2211 XβXβ

DP

DP

Logit function (log odds)

Baseline odds

Linear function of risk factors for individual i:

1x1 + 2x2 + 3x3 + 4x4

…

Review question 7 If X=.50, what is the logit (=log odds)

of X?

a. .50b. 0c. 1.0d. 2.0e. -.50

Example: political party and drinking…

Model:

Log odds of being a Republican (outcome)=

Intercept+ Weekly drinks (predictor)

Fit the data in logistic regression using a computer…

Fitted logistic model:

“Log Odds” of being a Republican = -.09 -1.4* (d/wk)

Slope for drinking can be directly translated into an odds ratio:

25.04.1 eInterpretation: every 1 drink more per week decreases your odds of being a Republican by 75% (95% CI is 0.047 to 1.325; p=.10)

To get back to OR’s…

)...()( 2211

)(1

)(disease of odds XβXβe

DP

DP

)...()())(1

)(ln( 2211 XβXβ

DP

DP

“Adjusted” Odds Ratio Interpretation

unexposed for the disease of odds

exposed for the disease of oddsOR

)1()0(

)1()1(

smokingalcohol

smokingalcohol

e

e

)1()0(

)1()1(

smokingalcohol

smokingalcohol

eee

eee

)1(

)1(

1alcohol

alcohol

ee

Adjusted odds ratio, continuous predictor

unexposed for the disease of odds

exposed for the disease of oddsOR

)19()1()1(

)29()1()1(

agesmokingalcohol

agesmokingalcohol

e

e

)19()1()1(

)29()1()1(

agesmokingalcohol

agesmokingalcohol

eeee

eeee

)10(

)19(

)29(age

age

age

ee

e

Practical Interpretation

interest offactor risk

)(ˆrf ORe

x

The odds of disease increase multiplicatively by eß

for for every one-unit increase in the exposure, controlling for other variables in the model.

Multivariate logistic regression

Litvick JR et al. Predictors of Olfactory Dysfunction in Patients With Chronic Rhinosinusitis. The Laryngoscope Dec 2008; 118: pp 2225-2230.

Logistic regressionStatistical question: What factors are associated

with anosmia (and hyposmia)? What are the outcome variables? anosmia vs.

normal olfaction (and hyosmia vs. normal) What type of variable is it? Binary Are the observations correlated? No Are groups being compared? We want to

examine multiple predictors at once, so we need multivariate regression.

multivariate logistic regression

Multivariate logistic regression

Litvick JR et al. Predictors of Olfactory Dysfunction in Patients With Chronic Rhinosinusitis. The Laryngoscope Dec 2008; 118: pp 2225-2230.

Interpretation: being a smoker increases your odds of anosmia by 658% after adjusting for older age, nasal polyposis, asthma, inferior turbinate hypertrophy, and septal deviation.

Logistic regression in cross-sectional and cohort studies… Many cohort and cross-sectional studies report

ORs rather than RRs even though the data necessary to calculate RRs are available. Why?

If you have a binary outcome and want to adjust for confounders, you have to use logistic regression.

Logistic regression gives adjusted odds ratios, not risk ratios.

These odds ratios must be interpreted cautiously (as increased odds, not risk) when the outcome is common.

When the outcome is common, authors should also report unadjusted risk ratios and/or use a simple formula to convert adjusted odds ratios back to adjusted risk ratios.

Example, wrinkle study… A cross-sectional study on risk factors for

wrinkles found that heavy smoking significantly increases the risk of prominent wrinkles. Adjusted OR=3.92 (heavy smokers vs.

nonsmokers) calculated from logistic regression.

Interpretation: heavy smoking increases risk of prominent wrinkles nearly 4-fold??

The prevalence of prominent wrinkles in non-smokers is roughly 45%. So, it’s not possible to have a 4-fold increase in risk (=180%)!

Raduan et al. J Eur Acad Dermatol Venereol. 2008 Jul 3.

Interpreting ORs when the outcome is common… If the outcome has a 10% prevalence in the

unexposed/reference group*, the maximum possible RR=10.0.

For 20% prevalence, the maximum possible RR=5.0

For 30% prevalence, the maximum possible RR=3.3.

For 40% prevalence, maximum possible RR=2.5. For 50% prevalence, maximum possible RR=2.0.

*Authors should report the prevalence/risk of the outcome in the unexposed/reference group, but they often don’t. If this number is not given, you can usually estimate it from other data in the paper (or, if it’s important enough, email the authors).

Interpreting ORs when the outcome is common…

Formula from: Zhang J. What's the Relative Risk? A Method of Correcting the Odds Ratio in Cohort Studies of Common Outcomes JAMA. 1998;280:1690-1691.

)()1( ORPP

ORRR

oo

Where:

OR = odds ratio from logistic regression (e.g., 3.92)

P0 = P(D/~E) = probability/prevalence of the outcome in the unexposed/reference group (e.g. ~45%)

If data are from a cross-sectional or cohort study, then you can convert ORs (from logistic regression) back to RRs with a simple formula:

For wrinkle study…

Zhang J. What's the Relative Risk? A Method of Correcting the Odds Ratio in Cohort Studies of Common Outcomes JAMA. 1998;280:1690-1691.

69.1)92.345(.)45.1(

92.3smokersnon vs.smokers

RR

So, the risk (prevalence) of wrinkles is increased by 69%, not 292%.

Sleep and hypertension study… ORhypertension= 5.12 for chronic insomniacs who

sleep ≤ 5 hours per night vs. the reference (good sleep) group.

ORhypertension = 3.53 for chronic insomiacs who sleep 5-6 hours per night vs. the reference group.

Interpretation: risk of hypertension is increased 500% and 350% in these groups?

No, ~25% of reference group has hypertension. Use formula to find corresponding RRs = 2.5, 2.2

Correct interpretation: Hypertension is increased 150% and 120% in these groups.

-Sainani KL, Schmajuk G, Liu V. A Caution on Interpreting Odds Ratios. SLEEP, Vol. 32, No. 8, 2009 .-Vgontzas AN, Liao D, Bixler EO, Chrousos GP, Vela-Bueno A. Insomnia with objective short sleep duration is associated with a high risk for hypertension. Sleep 2009;32:491-7.

Review problem 8 In a cross-sectional study of heart disease in middle-aged

men and women, 10% of men in the sample had prevalent heart disease compared with only 5% of women. After adjusting for age in multivariate logistic regression, the odds ratio for heart disease comparing males to females was 1.1 (95% confidence interval: 0.80—1.42). What conclusion can you draw?

a. Being male increases your risk of heart disease.b. Age is a confounder of the relationship between gender and heart

disease.c. There is a statistically significant association between gender and

heart disease.d. The study had insufficient power to detect an effect.

Review problem 8 In a cross-sectional study of heart disease in middle-aged

men and women, 10% of men in the sample had prevalent heart disease compared with only 5% of women. After adjusting for age in multivariate logistic regression, the odds ratio for heart disease comparing males to females was 1.1 (95% confidence interval: 0.80—1.42). What conclusion can you draw?

a. Being male increases your risk of heart disease.b. Age is a confounder of the relationship between gender and

heart disease.c. There is a statistically significant association between gender and

heart disease.d. The study had insufficient power to detect an effect.

Review topic: Diagnostic Testing and Screening Tests

Characteristics of a diagnostic test

Sensitivity= Probability that, if you truly have the disease, the diagnostic test will catch it.

Specificity=Probability that, if you truly do not have the disease, the test will register negative.

Calculating sensitivity and specificity from a 2x2 table

+ -

+ a b

- c d

Screening Test

Truly have disease

ba

a

Sensitivity

dc

d

Specificity

Among those with true disease, how many test positive?

Among those without the disease, how many test negative?

a+b

c+d

Hypothetical Example + -

+ 9 1

- 109 881

Mammography

Breast cancer ( on biopsy)

Sensitivity=9/10=.90

10

990

Specificity= 881/990 =.89

1 false negatives out of 10 cases

109 false positives out of 990

Positive predictive value

The probability that if you test positive for the disease, you actually have the disease.

Depends on the characteristics of the test (sensitivity, specificity) and the prevalence of disease.

Calculating PPV and NPV from a 2x2 table

+ -

+ a b

- c d

Screening Test

Truly have disease

ca

a

PPV

db

d

NPV

Among those who test positive, how many truly have the disease?

Among those who test negative, how many truly do not have the disease?

a+c b+d

Hypothetical Example + -

+ 9 1

- 109 881

Mammography


PPV=9/118=7.6%

118 882

Prevalence of disease = 10/1000 =1%

NPV=881/882=99.9%

What if disease was twice as prevalent in the population?

+ -

+ 18 2

- 108 872

Mammography


sensitivity=18/20=.90

20

980

specificity=872/980=.89

Sensitivity and specificity are characteristics of the test, so they don’t change!

What if disease was more prevalent?

PPV=18/126=14.3%

126 874

Prevalence of disease = 20/1000 =2%

NPV=872/874=99.8%

+ -

+ 18 2

- 108 872

Mammography


Conclusions

Positive predictive value increases with increasing prevalence of disease

Or if you change the diagnostic tests to improve their accuracy.

Review question 9In a group of patients presenting to the hospital casualty department with abdominal pain, 30% of patients have acute appendicitis. 70% of patients with appendicitis have a temperature greater than 37.5ºC; 40% of patients without appendicitis have a temperature greater than 37.5ºC.

a. The sensitivity of temperature greater than 37.5ºC as a marker for appendicitis is 21/49.

b. The specificity of temperature greater than 37.5ºC as a marker for appendicitis is 42/70.

c. The positive predictive value of temperature greater than 37.5ºC as a marker for appendicitis is 21/30.

d. The predictive value of the test will be the same in a different population.

e. The specificity of the test will depend upon the prevalence of appendicitis in the population to which it is applied.

tests for binary/categorical outcomes

Documents

eczema chisquare test

group members20

pearsons chisquare test

chisquare statistic

chisquare testapparently

groups anova

cumulative incidence

interpretationgroup