measures of test accuracy and associations dr odife, u.b sr, edm division

MEASURES OF TEST ACCURACY AND ASSOCIATIONS

DR ODIFE, U.BSR, EDM DIVISION

OUTLINE

• INTRODUCTION• SENSITIVITY • SPECIFICITY• PREDICTIVE VALUES• RECEIVER OPERATOR CHARACTERISTICS CURVE• ODDS RATIO• SUMMARY• REFERENCES

INTRODUCTION

• Sensitivity, specificity, predictive values and receiver operator characteristics curves are measures of test accuracy

• Odds ratio is a measure of test associations.

SENSITIVITY The proportion of those people who really have the disease who are correctly identified as such

• Sensitivity = TP/(TP+FN)

TP FP

FN TN

DiseasePresent Absent

Test

Positive

Negative

SPECIFICITY

• The proportion of those people who really do not have the disease who are correctly identified as such

• Specificity = TN/(TN+FP)

TP FP

FN TN


Test

Positive

Negative

SENSITIVITY AND SPECIFICITY

• Sensitivity and specificity are intrinsic characteristics of a test

• Both the sensitivity and specificity of a test need to be known in order to assess its usefulness for a diagnosis.

• A discriminating test would have sensitivity and specificity close to 100%.

• A test with high sensitivity may have low specificity and vice versa.


7

0 5 10 15 20

Quantitative result of the test

TN

Non-affected:

Affected:

TP

Nu

mb

er

of

peop

le t

este

d Threshold forpositive result

Ideal situation

Sensitivity and specificity

8

0 5 10 15 20

TN TP

FN FP

Non-affected:Threshold forpositive result


Nu

mb

er

of

peop

le t

este

d Affected:

Realistic situation


• They are not affected by the prevalence of a disease.

• Both can be altered by changing the threshold or cut-off point for diagnosing a disease.

• Lowering the threshold improves sensitivity but reduces specificity (i.e. more FP)

• Raising the threshold improves specificity but reduces sensitivity (i.e. more FN)


10

TNTP

FN

FP

Non-affected:

Affected:Threshold forpositive result

Nu

mb

er

of

peop

le t

este

d

Quantitative result of the test0 5 10 15 20

Sensitivity and specificity

11

0 5 10 15 20

TN

TP

FN

FP

Non-affected:

Affected:Threshold forpositive result

Nu

mb

er

of

peop

le t

este

d


Positive Predictive Value

• It is the proportion of the people who test positive who truly have the disease

• Positive predictive value (PPV) = TP/(TP+FP)

TP FP

FN TN


Test

Positive

Negative

Negative Predictive Value

• Is the proportion of the people who test negative who truly do not have the disease

• Negative predictive value = TN/(TN+FN)

TP FP

FN TN


Test

Positive

Negative

PPV and NPV

• PPV and NPV give a direct assessment of the usefulness of the test

• They are highly dependent on the prevalence of the disease in the population

• When the prevalence is low, the PPV will be low and NPV will be high.

• When the prevalence is high, the PPV will be high and NPV will be low.

Predictive value positive of a test according to prevalence and specificity

0102030405060708090

100

0 10 20 30 40 50 60 70 80 90 100

Prevalence (%)

PVP % 70%80%90%95%

Specificity

Predictive value negative of a test according to prevalence and sensitivity

Sensitivity

0

10

20

30

40

50

60

70

80

90

100

Prevalence (%)

PVN %

70%80%90%95%

Receiver operating characteristic (ROC) curve

• Developed in the 1950's during World War II for the analysis of radar radio signals .

• It is a by-product of research into making sense of radio signals contaminated by noise.

• Characterize the trade-off between positive hits and false alarms

ROC curve

• Recognized in the 1970's as useful tool for interpreting medical test results.

• Has become very popular in biomedical applications, particularly radiology and imaging

• Can be used to compare overall performance of diagnostic tests/procedures

ROC curve

• ROC curve represents the relationship between the true-positive rate (sensitivity) and the false-positive rate (1-specificity)

• ROC curve plots sensitivity (on the y-axis) against 1-specificity (on the x-axis)

• Each point on the ROC curve represents a sensitivity/specificity pair corresponding to a particular diagnostic threshold.

ROC curve

• The performance of a diagnostic variable can be quantified by calculating the area under the ROC curve (AUROC).

• The ideal test would have an AUROC of 1, whereas a random guess would have an AUROC of 0.5.

• The ability of diagnostic variables to diagnose an outcome can be compared using ROC curves and their AUROCs.

Tru

e P

osi

tive R

ate

(s

en

siti

vit

y)

0%

100%

False Positive Rate (1-specificity)

0%

100%

ROC curve

Ideal classifier

chance

always negative

always positive

ROC Curves

True

Pos

itive

Rat

e

0%

100%

False Positive Rate0%

100%

True

Pos

itive

Rat

e

0%

100%

False Positive Rate0% 100%

A good test: A poor test:

ROC Curves

True

Pos

itive

Rat

e

0%

100%


100%

True

Pos

itive

Rat

e

0%

100%


100%

True

Pos

itive

Rat

e

0%

100%


100%

AUC = 50%

AUC = 90% AUC =

65%

AUC = 100%

True

Pos

itive

Rat

e

0%

100%


100%

ROC Curves

ROC Curve

Uses• Used in communications sectors to examine false alarm

rates• Evaluate the discriminatory ability of a test to distinguish

diseased and normal subjects • Help define optimal cut-off value of a test• Comparing diagnostic efficacy of 2 or more medical tests • Comparing 2 or more observers measuring the same test

ROC Curve

26

00 20 40 60 80 100

20

40

60

80

100

IFA Dilutions

1/101/201/40

1/80

1/160

1/320

1/640

100 - Specificity (%): Proportion of false positives

Sensitivity (%)

ROC Curves

27

0 25 50

75 100

IFAELISA

0

20

40

60

80

100

100 - Specificity (%)

Sensitivity (%)

Area under the ROC curve (AUC)

ROC Curves

Odds

• Odds are ratio of two probabilities • The probability of an event occurring divided

by the probability of the event not occurring.• The odds of an event = probability/(1–

probability).• Odds refer to single entity• It ranges from zero to infinity.

Odds

• A die has 6 sizes, each with a difference number• On one die, the odds of rolling a 1 is…?• 1 side has a 1 on it, 5 sides do NOT• Odds of rolling a 1:

= 1/5, or 20%

Odds ratio

• The odds of the outcome in one group divided by the odds of the outcome in the other group

• It is a ratio of two odds• E.g p1 refers to the probability of the outcome

in group 1, and p2 is the probability of the outcome in group 2.

• Odds ratio (OR)= p1/(1- p1)/ p2 /(1- p2)

Odds ratio

• As a ratio, it ranges from zero to infinity.• Interpretation of OR: –OR = 1: exposure has no association with

disease–OR > 1: exposure may be positively

associated with disease–OR < 1: exposure may be negatively

associated with disease

Odds ratio

Odds of a heart attack in men = 68/32, or 2.125Odds of a heart attack in women = 42/58, or 0.724• Odds ratio = 2.125 / 0.724 = 2.9

Cause of Death in Men and Women

Heart Attack?

Yes No

SexMen 68 32

Women 42 58

Odds Ratio

• The odds of receiving a death sentence if the defendant was Black = 28/45 = 0.6222

• The odds of receiving a death sentence if the defendant was not Black = 22/52 = 0.4231

• The impact of being black on receiving a death penalty is measured by the odds ratio which equals:

= the odds if black ÷ the odds if not black = 0.6222 ÷ 0.4231 = 1.47

Odds ratio

Uses • Appropriate measure of relative effect in case-

control studies.• Commonly used in meta-analysis• ORs are the output of logistic regression

37

Odds ratio

OR in case-control StudyProbability of case being exposed = Pcase

Probability of case being non-exposed =1-Pcase

Odds of case being exposed = Pcase/1- Pcase

Probability of control being exposed = Pcontrol

Probability of control being non-exposed =1-Pcontrol

Odds of control being exposed = Pcontrol/ 1-Pcontrol

Target population

Exposed in past

Not exposed

Exposed

Not Exposed

Odds ratio

Disease

(Cases)

No Disease

(Controls)

SUMMARY

• Sensitivity, specificity, predictive values, ROC curves and odds ratio are measures of test accuracy and associations.

• Sensitivity is the proportion of those people who really have the disease who are correctly identified as such

• Specificity is the proportion of those people who really do not have the disease who are correctly identified as such

• Sensitivity and specificity can be altered by changing the threshold or cut-off point for diagnosing a disease.

SUMMARY

• Sensitivity and specificity are intrinsic characteristics of a test and are not affected by the prevalence of a disease

• Predictive values of a test can be either positive or negative

• PPV is the proportion of the people who test positive who truly have the disease

• NPV is the proportion of the people who test negative who truly do not have the disease

SUMMARY

• ROC curve was developed in the 1950's during World War II for the analysis of radio signals .

• ROC curve is a useful tool for interpreting medical test results.

• ROC curve represents the relationship between sensitivity and specificity for a test

SUMMARY

• Predictive values are affected by the prevalence of a disease

• ROC curve can be used to compare overall performance of diagnostic tests

• The performance of a diagnostic variable can be quantified by calculating the AUROC

• An ideal test would have an AUROC of 1, whereas a random guess would have an AUROC of 0.5.

SUMMARY

• Odd of an event probability of an event occurring divided by the probability of the event not occurring.

• Odd ratio is the odds of the outcome in one group divided by the odds of the outcome in the other group

• The values of odds and OR ranges from ranges from zero to infinity.

SUMMARY

• OR of 1 means exposure has no association with disease, > 1 means exposure may be positively associated with disease and < 1 means exposure may be negatively associated with disease

• Odd ratios are useful in case-control studies, meta-analysis and logistic regression

REFERENCES

• Bland JM, Altman DG. Statistics notes. The odds ratio. BMJ 2000;320:1468.

• Holcomb WL , Chaiworapongsa T, Luke DA, Burgdorf KD. An odd measure of risk: use and misuse of the odds ratio. Obstet Gynecol 2001;98:685–8

• Katz KA. The (relative) risks of using odds ratios. Arch Dermatol 2006;142: 761–4.

REFERENCES

• Davies HT, Crombie IK, Tavakoli M. When can odds ratios mislead? BMJ 1998;316:989–91

• Schechtman E. Odds ratio, relative risk, absolute risk reduction, and the number needed to treat: which of these should we use? Value Health 2002;5:431–6.

measures of test accuracy and associations dr odife, u.b sr, edm division

Documents