05 probabilistic graphical models
DESCRIPTION
TRANSCRIPT
Bayesian Networks
Unit 5 Probabilistic Graphical Models (PGM)
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
Wang, Yuan-Kai, 王元凱[email protected]
http://www.ykwang.tw
Department of Electrical Engineering, Fu Jen Univ.輔仁大學電機工程系
2006~2011
Reference this document as: Wang, Yuan-Kai, “Probabilistic Graphical Models,"
Lecture Notes of Wang, Yuan-Kai, Fu Jen University, Taiwan, 2011.
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 2
Goal of This Unit• Learn how to
– Build graphical model (network model) by graph theory
– Inference under uncertainty according to probability theory
• Theory of Bayesian networks– Conditional independence– D-Separation– Basic algorithm:
• Variable Elimination• Introduce some BN models
– MRF, HMM, DBN, Naïve Bayes, …
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p.
Related Units• Background
– Statistical inference– Graph theory
• Next units– Exact inference algorithms– Approximate inference algorithms
3
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 4
References for Self-Study• Chapter 14, Artificial Intelligence-a modern
approach, 2nd, by S. Russel & P. Norvig, Prentice Hall, 2003
• E. Charniak, Bayesian networks without tears, AI Magazine
• T. A. Stephenson, An introduction to Bayesian network theory and usage, IDIAP research report, IDIAP-RR-00-03, 2000
• B. D’Ambrosio, Inference in Bayesian networks, AI Magazine, 1999
• M. I. Jordan & Y. Weiss, Probabilistic Inference in graphical models,
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p.
Contents
1. Representing Uncertain Knowledge .............. 182. Various PGM Models ..................................... 523. Conditional Independence …………………. 664. Inference .......................................................... 885. Applications on Computer Vision ................. 1366. Summary ……………………………………. 1467. References …………………………………… 152
Fu Jen University Department of Electrical Engineering Wang, Yuan-Kai Copyright
5
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 6
Example – Car Diagnosis
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 7
Examples on Computer Vision
Anthropological Measurements
A
LeftHand
Hl
JointsJ
ComponentsC
ObservationsO
ObservationsOij
LeftForearm
Fl
LeftUpper Arm
Ul
HeadH
TorsoT
RightUpper Arm
Ur
RightForearm
Fr
RightHand
Hl
HandSizeSh
HeadSizeShd
TorsoSizeSt
UpperArm Size
Sa
ForearmSize Sf
LeftWrist
Wl
LeftElbow
El
LeftShoulder
Sl
NeckN
RightShoulder
Sr
RightElbow
Er
RightWrist
Wr
HandSizeSh
UpperArm Size
Sa
ForearmSize Sf
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 8
Where do PGMs come from ?• Common problems in real life :
– Complex, Uncertain
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 9
Graph + Probability• Graph has
– Node + Edge• Two kinds of graph
– Directed graph– Undirected graph
• Probability has– Random variable Node– Probability Edge
• Directed graph : conditional probability• Undirected graph: joint probability
X YP(X,Y)
X YP(X|Y)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 10
Probabilistic Modeling of Problems (1/2)
• Usually node has two semantics– Cause– Effect
• Causal relationships between nodes– Probabilistic– Conditional probability P(Y|X): P(Effect|Cause)– X and Y are not independent– Directed graph
Burglary Earthquake
Alarm
John Calls Mary Calls
P(A|B,E)
P(J|A) P(M|A)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 11
Probabilistic Modeling of Problems (2/2)
• If node has no causal semantics• But happens together
(influence each other)– Probabilistic– Joint probability P(X,Y)– X and Y are not independent– Undirected graph
Student X
Student Y
P(X,Y)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 12
Cause/Effect Class/Feature (1/2)• In pattern recognition
/computer vision– Cause class– Effect feature
FaceExpression
EyebrowMotion
MouthMotion
Facial expression image Base image (neutral expression)
P(f2|class)P(f1|class)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 13
Cause/Effect Class/Feature (2/2)• Face detection:
2-class classificationFace
object
Skin Color
Eyepattern
P(f2|class)P(f1|class)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 14
Cause/Effect State/Observation• In video analysis
(Tracking)– Cause State– Effect Observation
( | )t tP z x
Detected position : zt
Real position : xt Predicted positionx-
t+1
Reallocation
Observedlocation
Reallocation
Observedlocation
Reallocationxt
xt+1
zt
xt-1
zt-1
P(xt|xt-1)
P(zt-1|xt-1)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 15
What Are PGMs Good For?
Classification: P(class|feature) Prediction: P(Effect|Cause)=? Diagnosis: P(Cause|Effect)=?
Medicine Bio-informatics
Computer troubleshooting
Stock market
Text Classification
SpeechrecognitionComputer
Vision
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 16
Three Problems in PGM• Representation
– Given a problem– Build its graphical model
(Construction of Bayesian network)• Inference
– Given a set of evidences nodes– Get probabilities of node(s)
• Learning– Learn the CPT of a BN– Learn the graphical structure
of a BN
Reallocation
Observedlocation
Reallocation
Observedlocation
Reallocation
Reallocation
Observedlocation
Reallocation
Observedlocation
Reallocation
xt xt+1
zt
xt-1
zt-1
x z1 32 63 9
P(xt|xt-1)P(zt-1|xt-1)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 17
Structure of Related Lecture Notes
PGM Representation
Inference
Problem
Learning
Data
Unit 5 : Probabilistic GraphicalUnit 9 : Hybrid BNUnits 10~15: Naïve Bayes, MRF,
HMM, DBN,Kalman filter
Unit 6: Exact inferenceUnit 7: Approximate inferenceUnit 8: Temporal inference
Units 16~ : MLE, EM
StructureLearning
ParameterLearning
B E
A
J M
P(A|B,E)P(J|A)P(M|A)
P(B)P(E)
Query
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 18
1. Representing Uncertain Knowledge
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 19
Review (1/3)
)()()|()|(
ePhPhePehP
Posterior
Prior
Probability of Evidence
Likelihood
• Probability of an hypothesis, h, can be updated when evidence, e, has been obtained
• It is usually not necessary to calculate P(e) directly •As it can be obtained by normalizing the posterior probabilities, P(hi | e)
Bayes’ Theorem
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 20
Review (2/3)Marginalization
Hh
hXPXP ),()(
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 21
Review (3/3)• Full joint probability distribution
– Can answer any question P(X|E=e)P(X|E=e) = hP(X, e, h)
– But become intractably large as the number of variables grows
• Independence and conditional independence among random variables – CPTs = FJD– But can greatly reduce the number of
probabilities that need to specified
FJD
CPT
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 22
A Simple Bayesian Network• 1 FJD = 2 CPTs
– P(Cavity, Toothache)= P(Toothache|Cavity)
* P(Cavity)– P(X,Y)=P(X|Y)P(Y)
=P(Y|X)P(X)• Graphical model
can represent– Causal relationship– Joint relationship
Cavity
Toothache
P(C)0.002
T P(T|C)T 0.70F 0.01
CausalRelationship
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 23
A Burglary Network
Burglary Earthquake
Alarm
John Calls Mary Calls
P(B)0.001
P(E)0.002
B E P(A|B,E)T T 0.95T F 0.95F T 0.29F F 0.001
A P(J|A)T 0.90F 0.05
A P(M|A)T 0.70F 0.01
The graph is directedand acyclic
A conditional probability distribution quantifies the effects of the parents on node
(random) variables
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 24
Compact Representation• If all n nodes have k parents• O(2k n) vs. O(2n) parameters
Burglary Earthquake
Alarm
John Calls Mary Calls
P(B)0.001
P(E)0.002
B E P(A|B,E)T T 0.95T F 0.95F T 0.29F F 0.001
A P(J|A)T 0.90F 0.05
A P(M|A)T 0.70F 0.01
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 25
Formal Definition of a BN• Directed Acyclic Graph (DAG)
–Nodes : Random variables–Edges : Direct influence between 2 variables
• CPTs : Quantifies the dependency of two variables P(X|Parent(X))
–Ex : P(C|A,B), P(D|A)• A priori distribution :
for each node with no parents–Ex : P(A) and P(B)
A
E
D C
B
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 26
Conditional Independence in the Directed Acyclic Graph
• Topology of network encodes dependency/independence
• Weather is independentof the other variables
• Cavity has direct influence on Tooth and Catch
• Toothache and Catchare conditionally independent given Cavity
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 27
Conditional Probability Table (CPT)
P(W)0.001
P(C)0.02
C P(T|C)T 0.90F 0.05
C P(Catch|C)T 0.70F 0.01
P(Xi|Parent(Xi)) or P(Xi|Pa(Xi))
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 28
Causality and Bayesian Networks• Not every BN describes causal relationships
between the variables • Consider the dependence between Lung
Cancer, L, and the X-ray test, X. • A BN with causality
• Another BN represents the same distribution and independencies without causality
L XP(l)=0.001 P(x|l)=0.6P(x|l)=0.02
L X P(x1)=0.02058P(l1|x1)=0.02915P(l1|x2)=0.00041
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 29
Example - Construction of BN (1/3)• I have a burglar alarm installed at
home• I am at work• Neighbor John calls to say my
alarm is ringing • But neighbor Mary doesn't call• Sometimes it's set off by minor
earthquakes • Is there a burglar?
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 30
Example - Construction of BN (2/3)• Step 1: Find Random variables
– Burglar, Earthquake, Alarm, JohnCalls, MaryCalls
• Step 2: Represent the causal relationshipsamong random variables
– A burglar can set the alarm off– An earthquake can set the alarm off– The alarm can cause Mary to call– The alarm can cause John to call
• Step 3: Use network topology withprobability
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 31
Example - Construction of BN (3/3)
Burglary Earthquake
Alarm
John Calls Mary Calls
P(B)0.001
P(E)0.002
B E P(A|B,E)T T 0.95T F 0.95F T 0.29F F 0.001
A P(J|A)T 0.90F 0.05
A P(M|A)T 0.70F 0.01
• 5 Boolean random variables + 5 CPTs
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 32
Marginalization in Bayesian Network
mmMHh
MjaebPhjaebPjaebP,
),,,,(),,,,(),,,(
mmM aaA jjJHh
MJAebPhebPebP, , ,
),,,,(),,(),(
Burglary Earthquake
Alarm
John Calls Mary Calls
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 33
Markov Chain, Conditional Probability, Independence, and Directed Edge
• Markov chain
– L and X are dependent, not independent• Markov chain Has conditional prob. Not independent Has directed edge
L XP(X|L)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 34
Common CausesSmoking
Bronchitis Lung Cancer
• Markov condition: I(B, L | S), i.e. P(b | l, s) = P(b | s)
• If SB and SL, and “Joe is a smoker”• “Joe has Bronchitis” v.s. “Joe has Lung Cancer” ?• “Joe has Bronchitis” will not give us any more
information about the probability of “Joe has Lung Cancer”
It is a DAG
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 35
Common Effects
Alarm
Burglary Earthquake
• Markov condition: I(B, E), i.e. P(b | e) = P(b)• Burglary and Earthquake are independent of
each other• However they are conditionally dependent given
Alarm• If the alarm has gone off, news that there had
been an earthquake would ‘explain away’ the idea that a burglary had taken place
It is a DAG
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 36
Markov Assumption• Markov chain v.s.
independence• Random variable X
is independent of its non-descendents, given its parents Pa(X)– Formally,
I (X, NonDesc(X) | Pa(X))Non-descendent
Descendent
Ancestor
Parent
X
Y1 Y2
Non-descendent
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 37
Markov Assumption Example
• In this example:– I ( E, B )– I ( B, {E, R} )– I ( R, {A, B, C} | E )– I ( A, R | B,E )– I ( C, {B, E, R} | A)
Earthquake
Radio
Burglary
Alarm
Call
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 38
Joint Probability Distribution
)|(),|()|()|()(),,,,( acPbeaPerPebPePcarbeP
),,,|(),,|(),|()|()(),,,,( arbecPrbeaPberPebPePcarbeP
• Note that our joint distribution with 5 variables can be represented as:
But due to the Markov condition we have, for example,)|(),,,|( acParbecP
The joint probability distribution can be expressed as
• Ex: the probability that someone has a smoking history, lung cancer but not bronchitis, suffers from fatigue and tests positive in an X-ray test is
000135.06.05.0003.075.02.0),,,,( xflbsP
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 39
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 40
Representing the Joint Distribution
n
iiin xpaxPxxxP
121 ))(|(),...,,(
• For a BN with nodes X1, X2, …, Xn
• An enormous saving can be made regarding the number of values required for the joint distribution
• For n binary variables•2n – 1 values are required for FJD
• For a BN with n binary variables and •Each node has at most k parents •Less than 2kn values are required for CPTs
FJD n CPTs
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 41
Exercise (1/2)S
G
D
U
HE
),,,,,( ChAeugdsP
)(),|(),|()|()()( u|ChPugAePdsuPsgPdPsP
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 42
Exercise (2/2)• P(a, b, c, d, e)
= P(e | a, b, c, d) P(a, b, c, d)by the product rule
= P(e | c) P(a, b, c, d)by cond. indep. assumption
= P(e | c) P(d | a, b, c) P(a, b, c) = P(e | c) P(d | b, c) P(c | a, b) P(a, b)= P(e | c) P(d | b, c) P(c | a) P(b | a) P(a)
a
b c
d e
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 43
Exercises• Facial Expression Recognition• Face Detection• Face Tracking• Body Segmentation
Using GeNIehttp://genie.sis.pitt.edu/
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 44
Another Example : Water-Sprinkler
WetGrass
Cloudy
Sprinkler Rain
P(C)0.5
S R P(W|S,R)T T 0.99T F 0.9F T 0.9F F 0.0
C P(S|C)T 0.1F 0.5
C P(R|C)T 0.8F 0.2
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 45
Inference in Water-Sprinkler (1/2)• If the grass is wet (WetGrass=True)
– Two possible explanations : rain or sprinkler– Which is the more likely?
)Pr(),Pr()|Pr(
TWTWTSTWTS
)Pr(),Pr()|Pr(
TWTWTRTWTR
The grass is more likely to be wet because of the rain
Sprinkler
Rain
430.06471.02781.0
)Pr(
),,,Pr(,
TW
TWTSRCrc
708.06471.04581.0
)Pr(
),,,Pr(,
TW
TWTRSCsc
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 46
Inference in Water-Sprinkler (2/2)
),,|Pr(),|Pr()|Pr()Pr(),,,Pr( SCRWCRSCRCWSRC 2 x 4 x 8 x 16 = 1024
2 x 4 x 4 x 8 = 256
Time needed for calculations
Using conditional independency properties :
Using Bayes chain rule :
),|Pr()|Pr()|Pr()Pr(),,,Pr( SRWCSCRCWSRC
WetGrass
Cloudy
Sprinkler Rain
P(C)0.5
S R P(W|S,R)T T 0.99T F 0.9F T 0.9F F 0.0
C P(S|C)T 0.1F 0.5
C P(R|C)T 0.8F 0.2
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 47
Inference (1/5)
00.10.20.30.40.50.60.70.80.9
1
00.10.20.30.40.50.60.70.80.9
1
Earthquake Burglary
AlarmRadio
Call
P(E=t|C=t)=0.1 P(B=t|C=t) = 0.7
C=t
E B P(A|E,B)e b 0.9 0.1e b 0.2 0.8e b 0.9 0.1e b 0.01 0.99
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 48
Inference (2/5)
00.10.20.30.40.50.60.70.80.9
1
00.10.20.30.40.50.60.70.80.9
1
Earthquake Burglary
AlarmRadio
Call
P(E=t|C=t)=0.1 P(B=t|C=t) = 0.7
C=t
R=t
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 49
Inference (3/5)
00.10.20.30.40.50.60.70.80.9
1
00.10.20.30.40.50.60.70.80.9
1
00.10.20.30.40.50.60.70.80.9
1
Earthquake Burglary
AlarmRadio
Call
P(E=t|C=t)=0.1 P(B=t|C=t) = 0.7
C=t
R=t
00.10.20.30.40.50.60.70.80.9
1
P(E=t|C=t,R=t)=0.97 P(B=t|C=t,R=t) = 0.1
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 50
Inference (4/5)
00.10.20.30.40.50.60.70.80.9
1
00.10.20.30.40.50.60.70.80.9
1
00.10.20.30.40.50.60.70.80.9
1
Earthquake Burglary
AlarmRadio
Call
P(E=t|C=t)=0.1 P(B=t|C=t) = 0.7
C=t
R=t
00.10.20.30.40.50.60.70.80.9
1
P(E=t|C=t,R=t)=0.97 P(B=t|C=t,R=t) = 0.1
Explaining away effect
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 51
Inference (5/5)
00.10.20.30.40.50.60.70.80.9
1
00.10.20.30.40.50.60.70.80.9
1
00.10.20.30.40.50.60.70.80.9
1
Earthquake Burglary
AlarmRadio
Call
P(E=t|C=t)=0.1 P(B=t|C=t) = 0.7
C=t
R=t
00.10.20.30.40.50.60.70.80.9
1
P(E=t|C=t,R=t)=0.97 P(B=t|C=t,R=t) = 0.1
“Probability theory is nothing but common sense reduced to calculation”– Pierre Simon Laplace
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 52
2. Various PGM Models
Factor Graph
NaïveBayes
Taxonomy
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 53
Directional v.s. Undirectional
Directed ( Bayesian networks)
Undirected( Markov networks)
x1 x2
y1 y2
x1 x2
y1 y2
j
jpaji
ipai ppp )|()|()( )()( xyxxyx, a
aZp )(1)( yx,yx,
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 54
Naive Bayes Model• Strong (Naive) assumption of problems
– A single cause directly influences a number of effects
– All effects are conditionally independent, given the cause
i
i
niCauseEffectPCauseP
EffectEffectCauseP)|()(
),,(
2n+1 probabilities O(n)More details on another unit
n
iiin xpaxPxxxP
121 ))(|(),...,,(
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 55
Naïve Bayesian Classifier (NBC)• Use Naïve Bayes for classification
n
ii
n
n
ClassFeaturePClassP
ClassFeatureFeaturePFeatureFeatureClassP
1
1
1
)|()(
),,(),|(
Class
Feature 1 Feature n
FaceExpression
EyebrowMotion
MouthMotion
Faceobject
Skin Color
Eyepattern
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 56
Temporal Causality Represented by Bayesian Networks
• Temporal Causality– In many systems, data arrives sequentially– Dealing causality with time
• Dynamic Bayes nets (DBNs) can be used to model such time-series (sequence) data
• Special cases of DBNs include– State-space models (Kalman filter)– Hidden Markov models (HMMs)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 57
State Space Models (SSM)
n
iiin xpaxPxxxP
121 ))(|(),...,,(
t = 1 2 3
n
iiiii
TTTT
TTTT
XPXXPXYPXXP
XYPXXPXYPXXPXYPXPYXPYYXXP
11011
12212111
:1:111
)()|( where,)|()|(
)|()|()|()|()|()(),(),,,,...,(
Y1 Y3
X1 X2 X3
Y2 YT
XT• Hidden Markov Model• Kalman Filter
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 58
DBN (1/2)
Slice 1(DAG)
Slice 2(DAG)
+
Repeat
t = 1 2 3 4 5
More complex temporal models than HMM & Kalman
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 59
DBN (2/2)t = 1 2 3 4 5
n
iiin xpaxPxxxP
121 ))(|(),...,,(
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 60
Bayesian SSM
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 61
Factorial SSM• Multiple hidden sequences• Avoid exponentially large hidden space
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 62
Example: Markov Random Field• Typical application: image region
labelling
yiyi
xixi
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 63
Example: Conditional Random Field
y
y
y
y
xixi
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 64
Markov Random Fields (1/2)Undirected graph
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 65
MRF (2/2)
Local evidence (compatibility with image)Compatibility with neighbors
y
x
Parametertying
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 66
• A Bayesian network/probabilistic graphical model G, represents a set of Markov Independencies P
• There is a factorization theorem
• This section inspects deeper meanings of conditional independence for– The factorization theorem– Inference algorithms in later units
3. Conditional Independencies
i
iin PaXPXXP )|(),...,( 1
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 67
Conditional Independence• Dependencies
– Two connected nodes influence each other
• Independent– Example: I(B;E)
• Conditional Independent– Example
• I(J;M|A)?• I(B;E|A)?
– d-seperation
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 68
D-Separation• It is a rule describing the influences
between nodes
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 69
Serial (Intermediate Cause)
• Indirect causal effect, no evidence
• Clearly burglary will effect Marry call
• Same situation for indirect evidence effect, because independence is symmetric
• If I(E;M|A) then I(M;E|A)
B
A
M
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 70
Diverging (Common Cause)
A
J M
• Influence can flow from John call to Mary call if we don‘t know whether or not there is alarm.
• But I(J;M|A)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 71
Converging (Common Effect)
E B
A
• Influence can‘t flow from Earthquake to burglary if we don‘t know whether or not there is alarm
• So I(E;B)• Special structure which
cause independence. • V-Structure
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 72
Independence of Two Events
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 73
D-Separation for 3 Nodes
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 74
• Three cases:–Common cause
– Intermediate cause
–Common Effect
Blocked Unblocked
E
R A
E
R A
Path Blockage (1/3)
Blocked Active
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 75
Blocked UnblockedE
C
A
E
C
A
Path Blockage (2/3)
Blocked Active• Three cases:
–Common cause
– Intermediate cause
–Common Effect
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 76
Blocked Unblocked
E B
A
C
E B
A
CE B
A
C
Path Blockage (3/3)
Three cases:– Common cause
– Intermediate cause
– Common Effect
Blocked Active
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 77
General Case
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 78
D-Separation in General• X is d-separated from Y, given Z,
– If all paths from a node in X to a node in Yare blocked, given Z
• Checking d-separation can be done efficiently(linear time in number of edges)– Bottom-up phase:
Mark all nodes whose descendents are in Z– X to Y phase:
Traverse (BFS) all edges on paths from Xto Y and check if they are blocked
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 79
Paths (1/2)• Intuition: dependency must “flow” along
paths in the graph• A path is a sequence of neighboring
variables
Examples:• R E A B• C A E R
Earthquake
Radio
Burglary
Alarm
Call
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 80
Paths (2/2)• For a path between two end nodes X, Y• The path is a
– Active path• If we can find dependency between X & Y
– Blocked path• If we cannot find dependency between X & Y• X & Y are conditional independent• X & Y are D-Separated
• We want to classify situations in which paths are active
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 81
–d-sep(R,B)?
D-Separation Example 1 (1/3)
A
E B
C
R
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 82
D-Separation Example 1 (2/3)
E B
A
C
R
–d-sep(R,B) = yes–d-sep(R,B|A)?
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 83
–d-sep(R,B) = yes–d-sep(R,B|A) = no–d-sep(R,B|E,A)?
D-Separation Example 1 (3/3)
E B
A
C
R
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 84
D-Separation Example 2
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 85
D-Separation Example 3
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 86
d-separation: Car Start Problem• 1. ‘Start’ and ‘Fuel’ are dependent on each other.• 2. ‘Start’ and ‘Clean Spark Plugs’ are dependent on each other.• 3. ‘Fuel’ and ‘Fuel Meter Standing’ are dependent on each other.• 4. ‘Fuel’ and ‘Clean Spark Plugs’ are conditionally dependent on
each other given the value of ‘Start’.• 5. ‘Fuel Meter Standing’ and ‘Start’ are conditionally
independent given the value of ‘Fuel’.
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 87
Exercises
Reallocation
Observedlocation
Reallocation
Observedlocation
Reallocationxt
xt+1
zt
xt-1
zt-1
P(xt|xt-1)
P(zt-1|xt-1)
FaceExpression
EyebrowMotion
MouthMotion
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 88
4. Inference• 4.1 What Is Inference• 4.2 How Inference• 4.3 Inference Methods
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 89
4.1 What Is Inference
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 90
Exercises (1/2)• Face detection Facial Expression Recog.
FaceExpression
EyebrowMotion
MouthMotion
Faceobject
Skin Color
Eyepattern
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 91
Exercises (2/2)• Face tracking
( | )t tP z x
Detected position : zt
Real position : xt Predicted positionx-
t+1
Reallocation
Observedlocation
Reallocation
Observedlocation
Reallocationxt
xt+1
zt
xt-1
zt-1
P(xt|xt-1)
P(zt-1|xt-1)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 92
3 Kinds of Variables in Inference• Remember the general inference
procedure in previous unit (uncertainty inference unit)
• Let P(X|E=e) be the query–X be the query variable–E be the set of evidence variables
• e be the observed values of E–H be the remaining
unobserved variables(Hidden variables)
V S
LT
A B
X D
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 93
The Burglary Example
Burglary Earthquake
Alarm
John Calls Mary Calls
Query : P(Burglary|John Calls=true)
Query variables: XBurglary
Evidence variables: E=eJohn Calls = true
Hidden variables: HEarthquake, Alarm,Marry Calls
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 94
The Asia Example• Query P(L|v,s,d)
–Query variables: L–Evidence variables:
V=true, S=true, D=true–Hidden variables:
T, X, A, B
V S
LT
A B
X D
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 95
arg max P(X|e)
• arg maxx P(X=x|e) will get a decision
P(X=true | e) = 0.8P(X=false | e) = 0.2
• For P(X | e), if X is a Boolean variable • P(X | e) will compute 2 probabilities
P(X=true | e) = 0.8P(X=false | e) = 0.2
Max X = True
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 96
Five Types of Queries in Inference• For a probabilistic graphical model G• Given a set of evidence E=e• Query the PGM with
– P(e) : Likelihood query– arg max P(e) :
Maximum likelihood query– P(X|e) : Posterior belief query– arg maxx P(X=x|e) : (Single query variable)
Maximum a posterior (MAP) query– arg maxx1…xt
P(X1=x1, …, Xt=xt|e) :Most probable explanation (MPE) queryAlso called Viterbi decoding
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 97
Likelihood Query P(e) (1/2)
X1
E1
X2 Xt
E2 Et
…
e1
e2
et
e1:t
Input video
P (E1:t=e1:t)
An HMM for Surprise
Probability of Evidence
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 98
Likelihood Query P(e) (2/2)
tXX
ttt XXeEP
1
),,,( 1:1:1
Hh
hHeEP ),(• Marginalization of all hidden variables
X1
E1
X2 Xt
E2 Et
)()|( where,)|()|( 1011
11
XPXXPXEPXXPtXX
n
iiiii
1 2
),,,( 1:1:1X X X
tttt
XXeEP
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 99
Maximum Likelihood Query arg max P(e)X1
E1
X2 Xt
E2 Et
…
e1
e2
et
e1:t
Input video
P Surprise(e1:t)
An HMM for SurprisePS(Xt|Xt-1),PS(Ei|Xi)
Cry HMM PC(Xt|Xt-1),PC(Ei|Xi)
P Cry(e1:t)Max
X1
E1
X2 Xt
E2 Et
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 100
Maximum Likelihood Query arg max P(e)
• Likelihood query P(E=e))( eEP
Hh
hHeEP ),(
Step 1:Step 2:
Bayes theorem
Marginalizationof all hidden variables
• Query arg max P(E=e)
Hh
hHeEP ),(maxarg
Step 1: Step 2:
Bayes theorem
Marginalizationof all hidden variables
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 101
Posteriori Belief Query P(X|e)• Usually applied on tracking
–Use temporal models of PGM• 4 query types
–Filtering: P(Xt | E1=e1,…, Et=et)=P(Xt |e1:t)–Prediction: P(Xt+1 | e1:t)–Smoothing: P(Xt-k | e1:t)
(Fixed-lag smoothing)X1
E1
X2 Xt
E2 Et
Xt+1
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 102
P(X|e) – Filtering (1/2)• P(Xt | e1:t) X1
E1
X2 Xt
E2 Et
( | )t tP z x
Detected position: ei
Real position: xi Filtered position: x’t
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 103
P(X|e) – Filtering (2/2)• Inference of the query P(Xt|e1:t) is
),()(
),()|(
:1
:1
:1:1
tt
t
tttt
eXPeP
eXPeXP
11
),,( 11:1tXX
ttt XXeXP
)|()|(11 ~1
1 iiXX ti
ii XePXXPt
Step 1:
Step 3:
Step 2:
Bayes theorem
Marginalizationof all hidden variables
Chaining by conditional independence
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 104
P(X|e) – Prediction (1/2)• P(Xt+k | e1:t) for k > 0
X1
E1
X2 Xt
E2 Et
Xt+1For k=1
Detected position : ei
Real position : xi Predicted positionx’
t+1
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 105
P(X|e) – Prediction (2/2)• Inference of the query P(Xt+1|e1:t) is
),()(
),()|(
:11
:1
:11:11
tt
t
tttt
eXPeP
eXPeXP
tXX
ttt XXeXP
1
),,( 1:11
)|()|()|(1 ~1
11 iiXX ti
iitt XePXXPXXPt
Step 1:
Step 3:
Step 2:
Bayes theorem
Marginalizationof all hidden variables
Chaining by conditional independence
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 106
P(X|e) – Smoothing (1/3)• P(Xk | e1:t) for 1 k < t
X1
E1
X2 Xt
E2 Et
Xk
Ek
Detected position: zt
Real position: xt Smoothed position: xt
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 107
P(X|e) – Smoothing (2/3)• Inference of the query P(Xk|e1:t) is
),()(
),()|(
:1
:1
:1:1
tk
t
tktk
eXPeP
eXPeXP
tKk XXXX
tt XXeP,,,
1:1111
),(
)|()|(,,, ~1
1111
iiXXXX ti
ii XePXXPtKk
Step 1:
Step 3:
Step 2:
Bayes theorem
Marginalizationof all hidden variables
Chaining by conditional independence
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 108
P(X|e) – Smoothing (3/3)• Fixed-lag smoothing
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 109
MAP Query (1/2)• arg maxx P(Xi=x|e)• Usually applied on Classification
– Find most likely class X=x, given the evidence e (feature)
FacialExpression
EyebrowMotion
MouthMotion
X={Surprise, Smile, …}
P(X=Surprise|e)
P(X=Smile|e)
If P(X=Smile|e) is the max probabilitySmile = arg maxx P(Xi=x|e)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 110
MAP Query (2/2)• MAP query arg maxx P(X=x|E=e)
Hhx
hHexXP ),,(maxarg
Step 1:
Step 2:
Bayes theorem
Marginalizationof all hidden variables
),(maxarg)(
),(maxarg
)|(maxarg
exXPeP
exXP
exXP
x
x
x
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 111
MPE Query• Also called Viterbi decoding• arg maxx P(X1=x1,…, Xt=xt|e1:t) • = arg maxx1:t
P(X1:t|e1:t)• = Smoothing for X1:t-1 + Filtering
X1
E1
X2 Xt
E2 Et
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 112
Exercises• Face Detection• Facial Expression Recognition• Face Tracking• Body Segmentation
FacialExpression
EyebrowMotion
MouthMotion
X={Surprise, Smile, …}
Reallocation
Observedlocation
Reallocation
Observedlocation
Reallocationxt
xt+1
zt
xt-1
zt-1
P(xt|xt-1)
P(zt-1|xt-1)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 113
4.2 How Inference• Inference of the query P(X|E=e) is
),()(
),()|(
eEXPeEP
eEXPeEXP
Hh
hHeEXP ),,(
Hh ni
ii XPaXP~1
))(|(
Step 1:
Step 3:
Step 2:
Bayes theorem
Marginalizationof all hidden variables
Chaining by conditional independence
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 114
The 4th Step of Inference
• Step 4: Compute the sum product?– Need an efficient algorithm– First, we will explain the computation of
the sum-product by an enumeration algorithm• Easy but not efficient
– Then, more efficient methods will be explained in next two units
Hh ni
ii XPaXPeEXP~1
))(|()|( Steps 1 - 3
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 115
The Burglary Example (1/3)• A posterior query on the burglary
network–P(B|j, m)–= P(B, j, m) / P(j, m)–= P(B, j, m)–= e a P(B, e, a, j, m)
This will use the full joint distribution table
E and A are hidden variables
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 116
The Burglary Example (2/3)• Rewrite the full joint entries using
product of CPT entries–P(B|j,m)–= E A P(B, E, A, j, m)–= E A P(j, m, A, B , E)–= E A P(j|m,A,B,E)P(m|A,B,E)
P(A|B,E)P(B|E)P(E) (Chain rule)–= eaP(B)P(e)P(a|B,e)P(j|a)P(m|a)
(Conditional Independence)(All probabilities are CPT entries)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 117
The Burglary Example (3/3)• P(B|j,m) = < P(b|j,m), P(b|j,m)>
–= EAP(B)P(E)P(A|B,E)P(j|A)P(m|A)– =< EAP(b)P(E)P(A|b,E)P(j|A)P(m|A),EAP(b)P(E)P(A|b,E)P(j|A)P(m|A)>
–= <0.00059224, 0.0014919>– <0.284, 0.716>
• The chance of a burglary, given calls from both neighbors, is about 28%
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 118
Inference by Enumeration• A query P(X|e)
–= h Xi P(Xi | Pa(Xi))• Enumerate all P(Xi | Pa(Xi))• Multiply all P(Xi | Pa(Xi))• Sum all produts
Please refer the Unit - Uncertainty Inference
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 119
Expression Tree of Sum Product
• a + bc a1b1+a2b2
2
1iiiba
+
* acb
+
*
b1a1
*
b2a2
?1
n
iiiba
Hh ni
ii XPaXPeEXP~1
))(|()|(
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 120
Problem of Sum Product :Repeated Computation
• a1b + a2b • a1b1+a1b2+a1b3+a2b1+a2b2+a2b3
2
1
3
1i jjiba
2
1iiba
+
*
ba1
*
ba2
*
+
a2a1
b
2
1
3
1i jji ba
+ +
*
a2a1
b1 b2
b3+
= (a1+a2) *(b1+b2+b3)
b3
+
*
b1a1
*
b2a2
*
b3a3
+*
b1a1
*
b2a2
*
a3
++ +
2
1iiab
= (a1+a2)b
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 121
The Burglary Example (1/3)• Rewrite the full joint entries using
product of CPT entries–P(B|j,m)–= EAP(B)P(E)P(A|B,E)P(j|A)P(m|A)–= P(B) EP(E) AP(A|B,e)P(j|A)P(m|A)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 122
The Burglary Example (2/3)• P(b|j,m)= EAP(b)P(E)P(A|b,E)P(j|A)P(m|A)
+
**
P(j|a)P(m|a)
*
P(e)P(a|b,e)
P(b)
*A=a
**
P(j|a)P(m|a)
*
P(e)P(a|b,e)
P(b)
*A= a
E=e+
P(b)
**
P(j|a)P(m|a)
*
P(e)P(a|b,e)
*A=a
**
P(j|a)P(m|a)
*
P(e)P(a|b,e)
P(b)
*A= a
E= e+
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 123
The Burglary Example (3/3)• P(b|j,m)= P(b)EP(E)AP(A|b,E)P(j|A)P(m|A)
**
P(j|a)P(m|a)
P(a|b,e)
E=eA=a
P(b)
**
P(j|a)P(m|a)
P(a|b,e)
E=eA= a *
*
P(j|a)P(m|a)
P(a|b,e)
E=eA=a *
*
P(j|a)P(m|a)
P(a|b,e)
E=eA= a
P(e)+
*
P(e)+
*
+*
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 124
Enumeration as Depth-First Search• Recursive depth-first enumeration
–Suppose n Boolean variables–O(n) space–O(2n) time
• Enumeration is inefficient–Repeated computation–e.g., computes P(j|a)P(m|a) for each
value of e
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 125
Enumeration Algorithm (1/2)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 126
Enumeration Algorithm (2/2)
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 127
Exercise: Enumeration
smart study
prepared fair
pass
p(smart)=.8 p(study)=.6
p(fair)=.9
p(prep|…) smart smartstudy .9 .7study .5 .1
p(pass|…)smart smart
prep prep prep prepfair .9 .7 .7 .2fair .1 .1 .1 .1
Query: What is the probability that a student studied, given that they pass the exam?
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 128
Complexity of InferenceTheorem:
Computing queries in a Bayesian network is NP-hard
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 129
Hardness• Hardness does not mean we cannot
solve inference– It implies that we cannot find a general
procedure that works efficiently for all networks
– For particular families of networks, we can have provably efficient procedures
– We will characterize such families in the next two classes
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 130
4.3 Inference Methods• Steps of inference• Step 1: Bayesian theorem• Step 2: Marginalization• Step 3: Conditional independence• Step 4: Sum product computation
–Enumeration–Exact inference–Approximate inference
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 131
Exact v.s. Approximate Inference• Exact inference
– Get exact probability of the query• Approximate inference
– Get approximate probability of the query– Avoid exponential complexity of exact
inference in discrete loopy graphs– Because cannot compute messages in
closed form (even for trees) in the non-linear/non-Gaussian case
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 132
Exact Inference• Enumeration• Variable Elimination• Belief Propagation
– Message Passing• Junction Tree
– Clustering, Join tree
Unit –Exact Inference
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 133
Approximate Inference• Stochastic Simulation
– Monte Carlo method– Sampling method– Include: direct sampling (logic sampling),
likelihood weighting sampling• Markov Chain Monte Carlo Sampling
(MCMC)• Loopy Belief Propagation
Unit – Approximate Inference
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 134
Software Implemented Inference Methods
PNL GeNIe/SmileEnumeration v (Naïve)Variable Elimination
Junction Tree v v (Clustering)
Belief Propagation v (Pearl) v (Polytree)Direct Sampling v (Logic)
Likelihood Sampling v(LWSampling) v(Likelihood sampling)MCMC Sampling v(Gibbswithanneal) (Other 5 sampleings)
Loopy Belief Propagation
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 135
Exercise• Use GeNIe (http://genie.sis.pitt.edu/)
to – Generate a PGM– Inference the PGM
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 136
5. Applications on CV• Face Recognition• Human Body Tracking• Super-resolution
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 137
Face Modeling and Recognition Using Bayesian NetworksGang Song*, Tao Wang, Yimin Zhang, Wei Hu, Guangyou Xu*, Gary Bradski
Face feature finder (separate)
Learn Gabor filter “jet” at each point
System:
Add Pose switching variable
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 138
Face Modeling and Recognition Using Bayesian NetworksGang Song*, Tao Wang, Yimin Zhang, Wei Hu, Guangyou Xu*, Gary Bradski
Results:
Results:
BNPFR – Bayesnet with PoseBNFR – Bayesnet w/o PoseEHMM – Embedded HMMEGM – Gabor jets
Pose
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 139
Looking for all possible joint configuration J is computationally impractical. Therefore, segmentation takes place in two stages. First, we segment the head and torso, and determine the position of the neck. Then, we jointly segment the upper arms, forearms and hands, and determine the position of the remaining joints.
The Segmentation Problem
AHT ji
Aijijji
HTijijF qPqPPQQQJ,QJ,
JOJOQJO,,
),|(),|(maxarg),|(maxarg
HTA QQ , state assignments for the arm and head&torso regionsHTA JJ , joints for the arms and head&torso components.
Step I Step II
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 140
Upper Body Model
LeftHand
Hl
JointsJ
ComponentsC
ObservationsOObservations
Oij
F A J CF
F AAJAJAJOji
ijijq
ijijijji
ij uPPqPqOPuOPPij,,
})()|(),|(),,|({})({)(
Anthropological Measurements
A
LeftForearm
Fl
LeftUpper Arm
Ul
HeadH
TorsoT
RightUpper Arm
Ur
RightForearm
Fr
RightHand
Hl
HandSizeSh
HeadSizeShd
TorsoSizeSt
UpperArm Size
Sa
ForearmSize Sf
LeftWrist
Wl
LeftElbow
El
LeftShoulder
Sl
NeckN
RightShoulder
Sr
RightElbow
Er
RightWrist
Wr
HandSizeSh
UpperArm Size
Sa
ForearmSize Sf
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 141
Body Tracking Results
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 142
MRF (1/3)Undirected graph
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 143
MRF (2/3)
Local evidence (compatibility with image)Compatibility with neighbors
y
x
Parametertying
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 144
MRF (3/3)Image patches
(xi, yi)
(xi, xj)
image
scene
Scenepatches
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 145
MRFs for Super-Resolution
ActualCubic SplineInput Bayesian Net
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 146
6. Summary• What we know the representation
problem–What is a Bayesian network–A little about “how to inference”
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 147
Probabilistic Graphical Models• Given a PGM = given a joint
probability function• We can immediately write down
– Its joint probability, and– Its equivalent conditional probability
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 148
Directed Graph
),,( 1 ClassFeatureFeatureP n Class
Feature 1 Feature n
n
ii ClassFeaturePClassP
1
)|()(
=
X1
E1
X2 Xt
E2 Et
),,,,( 11 tt EEXXP
t
itttt XEPXXPXEPXP
21111 )|()|()|()(
=
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 149
Undirected Graph
),,,,(),(
11 NN YYXXPYXP
Ni
iiiNeighborj
ji YXXXP~1)(
),(),(
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 150
All Queries have the same Form
n
ii
n
n
ClassFeaturePClassP
ClassFeatureFeaturePFeatureFeatureClassP
1
1
1
)|()(
),,(),|(
Class
Feature 1 Feature n
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 151
We Still Need to Know• Inference
–Details of inference algorithms• Learning
–How to learn CPTs–How to build or automatically learn
the structure of a Bayesian network by given a set of learning data
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 152
7. References• A Brief Introduction to Graphical Models and
Bayesian Networks (Kevin Murph, 1998)– http://www.cs.ubc.ca/~murphyk/Bayes/bnintro.html
• Artificial Intelligence I (Dr. Dennis Bahler)– http://www.csc.ncsu.edu/faculty/bahler/courses/csc520f02/bayes1.html
• Nir Friedman– http://www.cs.huji.ac.il/~nir/
• Judea Pearl, Causality (on-line book)– http://bayes.cs.ucla.edu/BOOK-2K/index.html
• Introduction to Bayesian Networks– A tutorial for the 66th MORS symposium– Dennis M. Buede, Joseph A. Tatmam, Terry A.
Bresnick
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 153
References• An introduction to Bayesian network theory
and usage, T. A. Stephenson, IDIAP Research report IDIAP-RR 00-03, Feb. 2000. [Available: http://www.rpi.edu/~liaow/file/Intro_BN.pdf]
• Bayesian network without tears, E. Charniak, AI Magazine, 1991. [Available: http://www.rpi.edu/~liaow/file/BNtears.pdf]
Fu Jen University Department of Electrical Engineering Yuan-Kai Wang Copyright
Bayesian Networks Unit : Probabilistic Graphical Models p. 154
References• Inference in belief Networks : A procedural
guide (Cecile Huang)• Tutorial on graphical models and BNT
– presented to the Mathworks, May 2003 • Java Applets : Prof R.D. Boyle
([email protected])• HMMs – Summary (L R Rabiner and B H Juang )
– http://www.comp.leeds.ac.uk/roger/HiddenMarkovModels/html_dev/summary/s1_pg2.html