introduction to biomedical statistics. signal detection theory what do we actually “detect” when...
Post on 20-Dec-2015
223 views
TRANSCRIPT
Signal Detection Theory
• What do we actually “detect” when we say we’ve detected something?
• We say we’ve “detected” when a criterion value exceeds a threshold
Signal Detection Theory
• examples:– the onset of a light or sound
– the presence of an abnormality on x-ray
Signal Detection Theory
• There are 4 possible situations
Target is:
You
Res
pond
:
Present Absent
Pre
sent
Abs
ent
Signal Detection Theory
• There are 4 possible situations
Hit
Target is:
You
Res
pond
:
Present Absent
Pre
sent
Abs
ent
Signal Detection Theory
• There are 4 possible situations
Hit
Miss
Target is:
You
Res
pond
:
Present Absent
Pre
sent
Abs
ent
Signal Detection Theory
• There are 4 possible situations
Hit False
Alarm
Miss
Target is:
You
Res
pond
:
Present Absent
Pre
sent
Abs
ent
Signal Detection Theory
• There are 4 possible situations
Hit False
Alarm
Miss Correct
Rejection
Target is:
You
Res
pond
:
Present Absent
Pre
sent
Abs
ent
Signal Detection Theory
• There are 4 possible situations
Hit False
Alarm
Miss Correct
Rejection
Target is:
You
Res
pond
:
Present Absent
Pre
sent
Abs
ent
This is the total # of Target Present Trials
Signal Detection Theory
• There are 4 possible situations
Hit False
Alarm
Miss Correct
Rejection
Target is:
You
Res
pond
:
Present Absent
Pre
sent
Abs
ent
This is the total # of Target Present Trials
Signal Detection Theory
• Hit Rate (H) is the proportion of target present trials on which you respond “present”
€
H =# Hits
# Hits+# Misses
Signal Detection Theory
• Notice that H is a proportion, so 1 - H gives you the “miss” rate or…
€
MissRate =# Misses
#Hits+# Misses
Signal Detection Theory
• False-Alarm Rate (FA) is the proportion of target absent trials on which you respond “present”
€
FA =# FalseAlarms
# FalseAlarms+# CorrectRejections
Signal Detection Theory
• Notice that FA is a proportion. 1 minus FA gives you the correct rejections or …
€
CorrectRejectionRate =#CorrectRejections
#FalseAlarms+# CorrectRejections
Signal Detection Theory
• Signal Detection can be modeled as signal + noise with some detection threshold
Signal Detection Theory
Stimulus Intensity
Fre
quen
cy
Noise is normally distributed - Target Absent trials still contain some stimulus
Target Absent
Signal Detection Theory
Stimulus Intensity
Fre
quen
cy
Noise is normally distributed - Target Absent trials still contain some stimulus
Target Present trials contain a little bit extra intensity contributed by the signal
Target Absent
Target Present
Signal Detection Theory
Stimulus Intensity
Fre
quen
cy
Noise is normally distributed - Target Absent trials still contain some stimulus
Target Present trials contain a little bit extra intensity contributed by the signal
Target Absent
This is the signal’s contribution
Target Present
Signal Detection Theory
• We can imagine a static criterion above which we’ll respond “target is present”
Stimulus Intensity
Fre
quen
cy
Target Absent
Target Present
Criterion
Signal Detection Theory
• Notice that H, FA, etc thus have graphical meanings
Stimulus Intensity
Fre
quen
cy
Proportion Hits
Criterion
Signal Detection Theory
• Notice that H, FA, etc thus have graphical meanings
Stimulus Intensity
Fre
quen
cy
Proportion Misses
Criterion
Signal Detection Theory
• Notice that H, FA, etc thus have graphical meanings
Stimulus Intensity
Fre
quen
cy
Proportion False Alarms
Criterion
Signal Detection Theory
• Notice that H, FA, etc thus have graphical meanings
Stimulus Intensity
Fre
quen
cy
Proportion Correct Rejections
Criterion
Signal Detection Theory
• Notice that as H increases, FA also increases
Stimulus Intensity
Fre
quen
cy
H
Criterion
Signal Detection Theory
• Notice that as H increases, FA also increases
Stimulus Intensity
Fre
quen
cy
FA
Criterion
Signal Detection Theory
• d’ (pronounced d prime) is a measure of sensitivity to detect a signal from noise and does not depend on criterion - it is the distance between the peaks of the signal present and signal absent curves
Stimulus Intensity
Fre
quen
cy
• d’ is computed by converting from H and FA proportions into their corresponding Z scores and subtracting Zinv(FA) from Zinv(H)
Some Common Biomedical Statistics
• Sensitivity• Specificity• Positive Predictive Value• Negative Predictive Value• Likelihood Ratio• Relative Risk• Absolute Risk• Number needed to treat/harm
Sensitivity and Specificity
• Consider a test for a condition– e.g. Pregnancy test– e.g. Prostate-specific Antigen (Prostate
Cancer)– e.g. Ultrasound (Breast Cancer)
• These are all signal detection problems
Sensitivity and Specificity
• Four possible situations:
Condition is:
Tes
t R
esul
t:
Present Absent
Pre
sent
Abs
ent
Sensitivity and Specificity
• Four possible situations:
True Positive
Condition is:
Tes
t R
esul
t:
Present Absent
Pre
sent
Abs
ent
Sensitivity and Specificity
• Four possible situations:
True Positive
False Negative
Condition is:
Tes
t R
esul
t:
Present Absent
Pre
sent
Abs
ent
Sensitivity and Specificity
• Four possible situations:
True Positive
False Positive
False Negative
Condition is:
Tes
t R
esul
t:
Present Absent
Pre
sent
Abs
ent
Sensitivity and Specificity
• Four possible situations:
True Positive
False Positive
False Negative
True Negative
Condition is:
Tes
t R
esul
t:
Present Absent
Pre
sent
Abs
ent
Sensitivity and Specificity
• Four possible situations:
True Positive
False Positive
False Negative
True Negative
Condition is:
Tes
t R
esul
t:
Present Absent
Pre
sent
Abs
ent
This is Total # of Condition Present Cases
Sensitivity and Specificity
• Four possible situations:
True Positive
False Positive
False Negative
True Negative
Condition is:
Tes
t R
esul
t:
Present Absent
Pre
sent
Abs
ent
This is Total # of Condition Absent Cases
Sensitivity and Specificity
• Four possible situations:
True Positive
False Positive
False Negative
True Negative
Condition is:
Tes
t R
esul
t:
Present Absent
Pre
sent
Abs
ent
This is Total # of “positive” tests
Sensitivity and Specificity
• Four possible situations:
True Positive
False Positive
False Negative
True Negative
Condition is:
Tes
t R
esul
t:
Present Absent
Pre
sent
Abs
ent
This is Total # of “negative” tests
Sensitivity and Specificity
• Sensitivity is the proportion of condition present cases on which the test returned “positive”
• Analogous to the hit rate (H) in Signal Detection Theory
€
Sensivity =# True Positives
# True Postives + # False Negatives
Sensitivity and Specificity
• Specificity is the proportion of condition absent cases on which the test returned “negative”
• Analogous to the Correct Rejection rate in Signal Detection Theory
€
Specificity =# True Negative
# True Negative + #False Positive
Sensitivity and Specificity
• Notice that 1 minus the Sensitivity is analogous to the FA of Signal Detection Theory
Sensitivity and Specificity
• Notice that 1 minus the Sensitivity is analagous to the FA of Signal Detection Theory
• Recall that in Signal Detection Theory, as criterion were relaxed both H and FA increased and as criterion were more stringent, H and FA decreased
Sensitivity and Specificity
• Notice that 1 minus the Sensitivity is analagous to the FA of Signal Detection Theory
• Recall that in Signal Detection Theory, as criterion were relaxed both H and FA increased and as criterion were more stringent, H and FA decreased
• Sensitivity and Specificity have a similar relationship: as a cut-off value for a test becomes more stringent the sensitivity goes down and the specificity goes up…and vice versa
Sensitivity and Specificity
• “For detecting any prostate cancer, PSA cutoff values of 1.1, 2.1, 3.1, and 4.1 ng/mL yielded sensitivities of 83.4%, 52.6%, 32.2%, and 20.5%, and specificities of 38.9%, 72.5%, 86.7%, and 93.8%, respectively.”
JAMA. 2005 Jul 6;294(1):66-70.
Sensitivity and Specificity
• Likelihood Ratio is the ratio of True Positive rate to False Positive rate
• Loosely corresponds to d’ in that Likelihood ratio is insensitive to changes in criterion €
Likelihood Ratio =Sensitivity
1 - Specificity
Sensitivity and Specificity
• If a test is positive, how likely is it that the condition is present?
• Positive Predictive Value is the proportion of “positive” test results that are correct
€
PPV =# True Positives
# True Postives + #False Positives
Sensitivity and Specificity
• Negative Predictive Value is the proportion of “negative” test results that are correct
€
NPV =# True Negatives
# True Negatives + # False Negatives
Sensitivity and Specificity
• Consider the influence of exposure to some substance or treatment on the presence or absence of a condition
• e.g. smoking and cancer• e.g. aspirin and heart disease
Sensitivity and Specificity
• A similar logic can be applied
A B
C D
Disease:
Exp
osur
e:
Yes No
No
Yes
Sensitivity and Specificity
• A similar logic can be applied
A B
C D
Disease:
Exp
osur
e:
Yes No
This is total # exposure
No
Yes
Sensitivity and Specificity
• A similar logic can be applied
A B
C D
Disease:
Exp
osur
e:
Yes No
This is total non-exposure
No
Yes
Sensitivity and Specificity
• We can think in terms of “Event Rates”
€
Control Event Rate =C
C + D
A B
C D
Disease:
Exp
osu
re:
Yes No
Ye
sN
o
€
Exposure Event Rate =A
A + B
e.g. the proportion of non-smokers who get lung cancer
e.g. the proportion of smokers who get lung cancer
Sensitivity and Specificity
• Relative Risk is the ratio of Exposure Events to Non-Exposure Events
• Often encountered in regard to rate of adverse reactions to drugs
€
Control Event Rate =C
C + D
A B
C D
Disease:
Exp
osu
re:
Yes No
Ye
sN
o
€
Exposure Event Rate =A
A + B
€
Relative Risk =Exposure Event Rate
Control Event Rate=A /(A + B)
C /(C + D)
Sensitivity and Specificity
• Often we are interested in whether the chance of an event changes with exposure
• Relative Risk Reduction is the difference between event rates in the exposure and non-exposure groups, expressed as a fraction of the non-exposure event rate
€
Relative Risk Reduction =Exposure Event Rate - Control Event Rate
Control Event Rate
Sensitivity and Specificity
• Notice that Relative Risk Reduction can be positive or negative: that is, exposure could reduce the risk of some event (e.g. exposure to wine reduces risk of heart disease) or increase the risk (e.g. exposure to cigarette smoke increases risk of heart disease)
€
Relative Risk Reduction =Exposure Event Rate - Control Event Rate
Control Event Rate
Sensitivity and Specificity
• Notice also that these figures do not take into account the absolute numbers
• e.g. control event rate = .264 and exposure event rate = .198• e.g. control event rate = .000000264 and exposure event rate =
.000000198
• Both yield the same relative risk reduction of -25%
Sensitivity and Specificity
• Notice also that these figures do not take into account the absolute numbers
• e.g. control event rate = .264 and exposure event rate = .198• e.g. control event rate = .000000264 and exposure event rate =
.000000198
• Both yield the same relative risk reduction of -25%
• Doesn’t discriminate between large and small effects
Sensitivity and Specificity
• The absolute risk reduction conveys effect size
€
Absolute Risk Reduction = Exposure Rate - Control Rate
Sensitivity and Specificity
• The absolute risk reduction conveys effect size
• An intuitive version is to consider the reciprocal - the “number needed to treat or harm”
• Indicates the number of individuals that would have to be exposed to the treatment in order to cause one to have the outcome of interest
€
Absolute Risk Reduction = Exposure Rate - Control Rate
€
Number Needed to Treat or Harm = 1
Absolute Risk Reduction