judgement and decision making in information systems diagnostic modeling: bayes’ theorem,...
TRANSCRIPT
Judgement and Decision Making in Information Systems
Diagnostic Modeling: Bayes’ Theorem, Influence Diagrams
and Belief Networks
Yuval Shahar, M.D., Ph.D.
Reasoning Under Uncertainty Example: Medical Diagnosis
• Uncertainty is inherent to medical reasoning– relation of diseases to clinical and laboratory findings
is probabilistic– Patient data itself is often uncertain with respect to
value and time – Patient preferences regarding outcomes vary– Cost of interventions and therapy can change
• Principles of diagnosis modeling and computation are identical in engineering, finance, scientific evidence
Test Characteristics
Disease
Test result
Disease present
Disease absent
Total
Positive
True positive (TP)
False positive (FP)
TP+FP
Negative
False negative (FN)
True negative (TN)
FN+TN
TP+FN FP+TN
Test Performance Measures
• The gold standard test: the procedure that defines presence or absence of a disease (often, very costly)
• The index test: The test whose performance is examined• True positive rate (TPR) = Sensitivity:
– P(Test is positive|patient has disease) = P(T+|D+)– Ratio of number of diseased patients with positive tests to total
number of patient: TP/(TP+FN)• True negative rate (TNR) = Specificity
– P(Test is negative|patient has no disease) = P(T-|D-)– Ratio of number of nondiseased patients with negative tests to
total number of patients: TN/(TN+FP)
Test Predictive Values
• Positive predictive value (PV+) = P(D|T+) = TP/(TP+FP)
• Negative predictive value (PV-) = P(D-|T-) = TN/(TN+FN)
Lab Tests: What is “Abnormal”?
The Cut-off Value Trade off
• Sensitivity and specificity depend on the cut off value between what we define as normal and abnormal
• Assume high test values are abnormal; then, moving the cut-off value to a higher one increases FN results and decreases FP results (i.e. more specific) and vice versa
• There is always a trade off in setting the cut-off point
Receiver Operating Characteristic (ROC) Curves: Examples
Receiver Operating Characteristic (ROC) Curves: Interpretation
• ROC curves summarize the trade-off between the TPR (sensitivity) and the false positive rate (FPR) (1-specificity) for a particular test, as we vary the cut-off threshold
• The greater the area under the ROC curve, the better (more sensitive, more specific) the index test we are considering
Application of Bayes’ Theorem to Diagnosis
FPRDPySensitivitDP
DTPDP
DTPDPDTPDP
DTPDP
TP
DTPDP
TDPpositivetestdiseaseP
AP
BAPBPABP
*))(1(*)(
)|()(
)|()(()|()(
)|()(
)(
)|()(
)|():|(
)(
)|()()|(
B),|P(A P(B) B)|P(A)P(A B)&P(A
Odds-Likelihood (Odds Ratio) Form of Bayes’ Theorem
• Odds = P(A)/(1-P(A)); P = Odds/(1+Odds)
• Post-test odds = pretest odds * likelihood ratio
TNR
FNR
DTP
DTPLRratiolikelihoodNegative
FPR
TPR
DTP
DTPLRratioLikelihood
DTP
DTP
DP
DP
TDP
TDP
)|(
)|()(
)|(
)|()(
)|(
)|(*
)(
)(
)|(
)|(
Application of Bayes’ Theorem• Needs reliable pre-test probabilities• Needs reliable post-test likelihood ratios• Assumes one disease only (mutual exclusivity of
diseases)• Can be used in sequence for several tests, but only if
they are conditionally independent given the disease; then we use the post-test probability of Ti as the pre-test probability for Ti+1 (AKA Simple, or Naïve, Bayes)
ni
i
i
i
i
i
LRDP
DP
TDP
TDP
..1)(
)(
)|(
)|(
Relation of Pre-Test and Post-Test Probabilities
Example: Computing Predictive Values
• Assume P(Downs Syndrom):– (A) 0.1% (age 30)– (B) 2% (age 45)
• Assume for amniocentesis: Sensitivity is 99%, Specificity is 99%, for Downs Syndrome
• For both cases, A and B:– PV+ = P(DS|Amnio+) = ??– PV- = P(DS-|Amnio-) = ??
Predictive Values: Down Syndrom
0.99979 -PV
66891.001.0*98.099.0*02.0
99.0*02.0
2%(B)P(DS)
99.999% Amnio-)|P(DS- -PV
0901.000999.000099.0
00099.0
01.0*999.099.0*001.0
99.0*001.0
%1.0)()(
PV
PV
DSPA
Bayesian Diagnostic System Example: de Dombal’s Abdominal-Pain System (1972)
• Domain: Acute abdominal pain (7 possible diagnoses)• Input: Signs and symptoms of patient• Output: Probability distribution of diagnoses• Method: Naïve Bayesian classification• Evaluation: an eight-center study involving 250 physicians and
16,737 patients• Results:
– Diagnostic accuracy rose from 46 to 65%– The negative laparotomy rate fell by almost half– Perforation rate among patients with appendicitis fell by half– Mortality rate fell by 22%
• Results using survey data consistently better than the clinicians’ opinions and even the results using human probability estimates!
Influence Diagrams: An Alternative, Powerful Tool for
Modeling Decisions• A graphical notation for modeling situations
involving multiple decisions, probabilities, and utilities
• Computationally: equivalent to decision trees
• Has several distinct advantages and disadvantages relative to decision trees
Influence Diagrams: Node Conventions
Chance node
Decision node
Utility node
Deterministic node
Link Semantics in Influence Diagrams
Dependence link(possible probabilistic relationship)
Information link
Influence link
“No-forgetting” link
Decision Trees: an HIV Example
Decision node
Chance node
Influence Diagrams: An HIV Example
The Structure of Influence Diagram Links
Belief Networks (Bayesian/Causal Probabilistic/Probabilistic Networks, etc)
Disease
Fever Sinusitis
Runny nose
Headache
Influence diagrams without decision and utility nodes
Gender
Link Semantics in Belief Networks
Dependence
Independence
Conditional independence of B and C, given A
B
CA
Assessment Versus Observation Orders
• Usually it is most convenient to consider relationship between diagnoses and tests in assessment order: P(T+|D+)– This is also the easier order in which experts assess the
probabilities or in which we can learn them from data
• However, in real life we need to compute the probability of diseases given observed findings, that is, in observation order: P(D+|T+)– That is the fashion in which diagnostic problems are
typically presented, although less easy to compute
• Thus, we often need to reverse the dependence arc to solve the influence diagram or belief network (or to draw the corresponding decision tree)
Assessment Order Representation
0.8 T+
0.2 T-
0.2 T+
0.8 T-
0.4 D+
0.6 D-
0.32
0.08
0.12
0.48
D TP(D & T )
Observation order Representation: Reversing the Arcs
0.727 D+
0.273 D-
0.143 D+
0.857 D-
0.44 T+
0.56 T-
0.32
0.08
0.12
0.48
D T P(D & T )
Advantages of Influence Diagrams and Belief Networks:
Modeling Implications• Excellent modeling tool that supports
acquisition from domain experts– Intuitive semantics (e.g., information and
influence links)– Explicit representation of dependencies– Representation in assessment (not observation)
order– very concise representation of large decision
models
Advantages of Influence Diagrams and Belief Networks:
Computational Implications• “Anytime” algorithms available (using
probability theory) to compute the distribution of values at any node given the values of any subset of the nodes (e.g., at any stage of information gathering)
• Explicit support for value of information computations
Disadvantages of Influence Diagrams and Belief Networks
for Modeling• The order of decisions (timing) might be obscured• The precise relationship between decisions and
available information is hidden within the nodes• Highly asymmetric problems might be easier to
represent as decision trees– Influence diagrams require using a lot of 0/1 probabilities
to represent asymmetry (e.g., if the test is not done, the result is sure to be unknown)
Problems in Using Influence Diagrams and Belief Networks
for Computations• Explicit representation of dependencies often
requires acquisition of joint probability distributions (P(A|B,C))
• Computation is in general intractable (NP hard), making even moderate-sized problems hard to solve without specialized algorithms
• Solution of even a relatively simple influence diagram requires the use of a computer and specialized software
Examples of Successful Belief- Network Applications
• In clinical medicine:– Pathological diagnosis at the level of a
subspecialized medical expert (Pathfinder)– Endocrinological diagnosis (NESTOR)
• In bioinformatics:– Recognition of meaningful sites and features in
DNA sequences– Educated guess of tertiary structure of proteins