bayesian belief networks - linköping...
TRANSCRIPT
Introduction Bayesian Belief Networks Summary
Bayesian Belief Networks
Philipp Engel, Kristopher Gustavsson, Jorg Schad
Linkopings Universitet,Dept. of Science and Technology, ITN
27.04.2008
1 / 16
Introduction Bayesian Belief Networks Summary
Motivation
Clippy
Partly implemented usinga Bayesian Belief Network(BBN)
Predicts User Intention
One example of manyBBN applications
BBN are fast growingtechnique for Reasoning
2 / 16
Introduction Bayesian Belief Networks Summary
Motivation
Clippy
Partly implemented usinga Bayesian Belief Network(BBN)
Predicts User Intention
One example of manyBBN applications
BBN are fast growingtechnique for Reasoning
2 / 16
Introduction Bayesian Belief Networks Summary
Motivation
Clippy
Partly implemented usinga Bayesian Belief Network(BBN)
Predicts User Intention
One example of manyBBN applications
BBN are fast growingtechnique for Reasoning
2 / 16
Introduction Bayesian Belief Networks Summary
Motivation
Clippy
Partly implemented usinga Bayesian Belief Network(BBN)
Predicts User Intention
One example of manyBBN applications
BBN are fast growingtechnique for Reasoning
2 / 16
Introduction Bayesian Belief Networks Summary
Motivation
Clippy
Partly implemented usinga Bayesian Belief Network(BBN)
Predicts User Intention
One example of manyBBN applications
BBN are fast growingtechnique for Reasoning
2 / 16
Introduction Bayesian Belief Networks Summary
BBN History
1763 Bayes Theorem
1913 Wigmore charts,
1985 Pearl created term ”Bayesian networks”
Since growing Number of Applications
3 / 16
Introduction Bayesian Belief Networks Summary
BBN History
1763 Bayes Theorem
1913 Wigmore charts,
1985 Pearl created term ”Bayesian networks”
Since growing Number of Applications
3 / 16
Introduction Bayesian Belief Networks Summary
BBN History
1763 Bayes Theorem
1913 Wigmore charts,
1985 Pearl created term ”Bayesian networks”
Since growing Number of Applications
3 / 16
Introduction Bayesian Belief Networks Summary
BBN History
1763 Bayes Theorem
1913 Wigmore charts,
1985 Pearl created term ”Bayesian networks”
Since growing Number of Applications
3 / 16
Introduction Bayesian Belief Networks Summary
Bayesian Classification
Bayesian Classifiers
Probalistic reasoning
P(Class = c |E ) = x
Naive Bayes Classifier
Strong Independence Assumption
Not always applicableconsider duplicated variables (perfect correlation)
Bayesian Belief Networks
Modelled Dependencies
Indenpendence given Markov Blanket of Node(details follow)
4 / 16
Introduction Bayesian Belief Networks Summary
Bayesian Classification
Bayesian Classifiers
Probalistic reasoning
P(Class = c |E ) = x
Naive Bayes Classifier
Strong Independence Assumption
Not always applicableconsider duplicated variables (perfect correlation)
Bayesian Belief Networks
Modelled Dependencies
Indenpendence given Markov Blanket of Node(details follow)
4 / 16
Introduction Bayesian Belief Networks Summary
Bayesian Classification
Bayesian Classifiers
Probalistic reasoning
P(Class = c |E ) = x
Naive Bayes Classifier
Strong Independence Assumption
Not always applicableconsider duplicated variables (perfect correlation)
Bayesian Belief Networks
Modelled Dependencies
Indenpendence given Markov Blanket of Node(details follow)
4 / 16
Introduction Bayesian Belief Networks Summary
Bayesian Classification
Bayesian Classifiers
Probalistic reasoning
P(Class = c |E ) = x
Naive Bayes Classifier
Strong Independence Assumption
Not always applicableconsider duplicated variables (perfect correlation)
Bayesian Belief Networks
Modelled Dependencies
Indenpendence given Markov Blanket of Node(details follow)
4 / 16
Introduction Bayesian Belief Networks Summary
Bayesian Classification
Bayesian Classifiers
Probalistic reasoning
P(Class = c |E ) = x
Naive Bayes Classifier
Strong Independence Assumption
Not always applicableconsider duplicated variables (perfect correlation)
Bayesian Belief Networks
Modelled Dependencies
Indenpendence given Markov Blanket of Node(details follow)
4 / 16
Introduction Bayesian Belief Networks Summary
Bayesian Classification
Bayesian Classifiers
Probalistic reasoning
P(Class = c |E ) = x
Naive Bayes Classifier
Strong Independence Assumption
Not always applicableconsider duplicated variables (perfect correlation)
Bayesian Belief Networks
Modelled Dependencies
Indenpendence given Markov Blanket of Node(details follow)
4 / 16
Introduction Bayesian Belief Networks Summary
Overview
Bayesian Belief Network is a directed, acyclic graph,
nodes representing thevariables, each with anassociated cpt, conditonalpropability table
and arcs modelling thedependencies betweenvariables
If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.
5 / 16
Introduction Bayesian Belief Networks Summary
Overview
Bayesian Belief Network is a directed, acyclic graph,
nodes representing thevariables, each with anassociated cpt, conditonalpropability table
and arcs modelling thedependencies betweenvariables
If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.
5 / 16
Introduction Bayesian Belief Networks Summary
Overview
Bayesian Belief Network is a directed, acyclic graph,
nodes representing thevariables, each with anassociated cpt, conditonalpropability table
and arcs modelling thedependencies betweenvariables
If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.
5 / 16
Introduction Bayesian Belief Networks Summary
Overview
Bayesian Belief Network is a directed, acyclic graph,
nodes representing thevariables, each with anassociated cpt, conditonalpropability table
and arcs modelling thedependencies betweenvariables
If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.
5 / 16
Introduction Bayesian Belief Networks Summary
Overview
Bayesian Belief Network is a directed, acyclic graph,
nodes representing thevariables, each with anassociated cpt, conditonalpropability table
and arcs modelling thedependencies betweenvariables
If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.
5 / 16
Introduction Bayesian Belief Networks Summary
Overview
Bayesian Belief Network is a directed, acyclic graph,
nodes representing thevariables, each with anassociated cpt, conditonalpropability table
and arcs modelling thedependencies betweenvariables
If there is an arc from node a to b, a is called the parent of b. If anode has no parents, the probability distribution is unconditional,otherwise it is conditional.
5 / 16
Introduction Bayesian Belief Networks Summary
Conditional Independence
A variable (node) is conditionally independentof its non-descendants given its parents.
Lung tumor is dependent of Cancer
Age is independent of Cancer
Smoking is a parent of Cancer
Examples:
Age is cond. independent of Gender
Lung tumor is cond. dependent ofCancer
Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking
6 / 16
Introduction Bayesian Belief Networks Summary
Conditional Independence
A variable (node) is conditionally independentof its non-descendants given its parents.
Lung tumor is dependent of Cancer
Age is independent of Cancer
Smoking is a parent of Cancer
Examples:
Age is cond. independent of Gender
Lung tumor is cond. dependent ofCancer
Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking
6 / 16
Introduction Bayesian Belief Networks Summary
Conditional Independence
A variable (node) is conditionally independentof its non-descendants given its parents.
Lung tumor is dependent of Cancer
Age is independent of Cancer
Smoking is a parent of Cancer
Examples:
Age is cond. independent of Gender
Lung tumor is cond. dependent ofCancer
Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking
6 / 16
Introduction Bayesian Belief Networks Summary
Conditional Independence
A variable (node) is conditionally independentof its non-descendants given its parents.
Lung tumor is dependent of Cancer
Age is independent of Cancer
Smoking is a parent of Cancer
Examples:
Age is cond. independent of Gender
Lung tumor is cond. dependent ofCancer
Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking
6 / 16
Introduction Bayesian Belief Networks Summary
Conditional Independence
A variable (node) is conditionally independentof its non-descendants given its parents.
Lung tumor is dependent of Cancer
Age is independent of Cancer
Smoking is a parent of Cancer
Examples:
Age is cond. independent of Gender
Lung tumor is cond. dependent ofCancer
Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking
6 / 16
Introduction Bayesian Belief Networks Summary
Conditional Independence
A variable (node) is conditionally independentof its non-descendants given its parents.
Lung tumor is dependent of Cancer
Age is independent of Cancer
Smoking is a parent of Cancer
Examples:
Age is cond. independent of Gender
Lung tumor is cond. dependent ofCancer
Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking
6 / 16
Introduction Bayesian Belief Networks Summary
Conditional Independence
A variable (node) is conditionally independentof its non-descendants given its parents.
Lung tumor is dependent of Cancer
Age is independent of Cancer
Smoking is a parent of Cancer
Examples:
Age is cond. independent of Gender
Lung tumor is cond. dependent ofCancer
Cancer is cond. independent ofAge and Gender given Exposure to Toxicsand Smoking
6 / 16
Introduction Bayesian Belief Networks Summary
Inference
Some useful inference rules. Q is the query variable, E is Evidence.
Conditional Probability: P(Q = q|E ) = P(q,E)P(E)
X1, ...,Xn be unknown network variables that q depends on
Joint probability:
P(Q = q,E = e) =∑
X1,...,Xn
P(q|e, x1, ...xn)P(e, x1, ...xn)
General Product rule for Bayesian Networks:
P(X1, ...,Xn) =n∏
i=1
P(Xi |parents(Xi ))
7 / 16
Introduction Bayesian Belief Networks Summary
Example
P(C = T |LT = T ,S = T ) = P(LT=T ,S=T ,C=T )P(LT=T ,S=T )
=∑
β P(LT=T ,S=T ,C=T ,ET=β)∑β,γ P(S=T ,LT=T ,C=β,ET=γ)
=P(LT=T |C=T )P(S=T )
∑β P(C=T |S=T ,ET=β)P(ET=β)
P(S=T )∑
β,γ P(LT=T |C=β)P(C=β|S=T ,ET=γ)P(ET=γ)
= 0.6∗0.3(0.25∗0.95+0.38∗0.05)0.3∗(0.6∗0.38∗0.05+0.6∗0.25∗0.95+0.02∗0.62∗0.05+0.02∗0.75∗0.95) ' 87.26%
8 / 16
Introduction Bayesian Belief Networks Summary
Learning
Step 1: Create the Network Structure
Specified by domain experts
Search Algorithm (Hill Climbing, K2...)
Independence Test for Variables
Step 2: Create Probability Table for each Node
Pearl’s Bi-directional Belief Updating Algorithm
Maximum Likelyhood
9 / 16
Introduction Bayesian Belief Networks Summary
Applications for BBN
Meteorology - Weather forecast
Medical diagnosis - Indicating possible Diagnosis
Troubleshooting and online help - Reducing Cost and Time forSupport
Astronomy - Classification of data from deep-space networkwithout previously established categories
Agriculture - Identifying cattle parentship
10 / 16
Introduction Bayesian Belief Networks Summary
Limitations of BBNs
No Derivation of human readable Rules
Problems when certain Event has not occured in the TrainingSet
Can be computational difficulty
Not expressive enough for many real-world Applications
11 / 16
Introduction Bayesian Belief Networks Summary
BBN Inference Demo
http://www.aispace.org/bayes/index.shtml
”Load From File”’ for opening XML file
Switch to Solve Screen
Make some Observations (example Smoking = True)
Query other Nodes (example Propability of Cancer)
12 / 16
Introduction Bayesian Belief Networks Summary
Weka -BNN Structure Learning Demo
Assumptions for Weka Bays Net
Discrete Finite Variables
No Missing Values
Preprocessing!
Various Algoritms to learn BNN Structure (K2, hill climbing,simulated annealing..)SimpleEstimator or BMAEstimator for Distibution LearningWeka BN Editor for Viewing and Modifying Networksjava weka.classifers.bayes.net.GUI file
13 / 16
Introduction Bayesian Belief Networks Summary
Weka -BNN Structure Learning Demo
Assumptions for Weka Bays Net
Discrete Finite Variables
No Missing Values
Preprocessing!
Various Algoritms to learn BNN Structure (K2, hill climbing,simulated annealing..)
SimpleEstimator or BMAEstimator for Distibution LearningWeka BN Editor for Viewing and Modifying Networksjava weka.classifers.bayes.net.GUI file
13 / 16
Introduction Bayesian Belief Networks Summary
Weka -BNN Structure Learning Demo
Assumptions for Weka Bays Net
Discrete Finite Variables
No Missing Values
Preprocessing!
Various Algoritms to learn BNN Structure (K2, hill climbing,simulated annealing..)SimpleEstimator or BMAEstimator for Distibution Learning
Weka BN Editor for Viewing and Modifying Networksjava weka.classifers.bayes.net.GUI file
13 / 16
Introduction Bayesian Belief Networks Summary
Weka -BNN Structure Learning Demo
Assumptions for Weka Bays Net
Discrete Finite Variables
No Missing Values
Preprocessing!
Various Algoritms to learn BNN Structure (K2, hill climbing,simulated annealing..)SimpleEstimator or BMAEstimator for Distibution LearningWeka BN Editor for Viewing and Modifying Networksjava weka.classifers.bayes.net.GUI file
13 / 16
Introduction Bayesian Belief Networks Summary
Summary
Bayesian Belief Networks are
Powerful and Growing Reasoning Technique
Reduced Representation of the Full Joint Distribution
BUT
Some Limitations
Provide no intutive rules
14 / 16
Introduction Bayesian Belief Networks Summary
15 / 16
Introduction Bayesian Belief Networks Summary
Bibliography
http://en.wikipedia.org/wiki/Bayesian network
Bouckaert, R. 2007 Bayesian Network Classifiers in Weka
Costa, P. et al. Bayesian Networks
Heckermann, K 2004 Bayesian Networks for Datamining
Kahney,L. 2001 MS Office Helper not Dead yet
Niedermayer,D 1998 An Introduction to Bayesian Networksand their Contemporary Applications
Pang-Ning et al 2006 Introduction to Datamining
Introduction of BBN,http://www.murrayc.com/learning/AI/bbn.shtml
Vomlelhttp://staff.utia.cas.cz/vomlel/slides/presentace-karny.pdf
16 / 16