Introduction to Belief Propagation and its Generalizations.
Max Welling
Donald Bren Schoolof Information and Computer and Science
University of California Irvine
Graphical Models
A ‘marriage’ between probability theory and graph theory
Why probabilities? • Reasoning with uncertainties, confidence levels• Many processes are inherently ‘noisy’ robustness issues
Why graphs?• Provide necessary structure in large models: - Designing new probabilistic models. - Reading out (conditional) independencies.
• Inference & optimization: - Dynamical programming - Belief Propagation
Types of Graphical Model
Undirected graph (Markov random field)
Directed graph(Bayesian network)
i ij
jiijii xxxZ
xP)(
)( ),()(1
)(
i
j
)( ii x),()( jiij xx
)|()( )(parentsi
ii xxPxP
i
Parents(i)
factor graphs
interactions
variables
Example 1: Undirected Graph
neighborhoodinformation
high informationregions
low information
regions
air or water ?
?
?
Undirected Graphs (cont’ed)Nodes encode hidden information (patch-identity).
They receive local information from the image (brightness, color).
Information is propagated though the graph over its edges.
Edges encode ‘compatibility’ between nodes.
Example 2: Directed Graphs
war animals computersTOPICS …
Iraqi the Matlab
Why do we need it?• Answer queries : -Given past purchases, in what genre books is a client interested? -Given a noisy image, what was the original image?
• Learning probabilistic models from examples
(expectation maximization, iterative scaling ) •Optimization problems: min-cut, max-flow, Viterbi, …
Inference in Graphical Models
Example: P( = sea | image) ?
Inference: • Answer queries about unobserved random variables, given values of observed random variables.
• More general: compute their joint posterior distribution: ( | ) { ( | )}iP u o or P u o
learning
inference
Approximate Inference
Inference is computationally intractable for large graphs (with cycles).
Approximate methods:
• Markov Chain Monte Carlo sampling. • Mean field and more structured variational techniques.• Belief Propagation algorithms.
Belief Propagation on trees
ik
k
k
k
ij k
k
k
Mki
k
iikx
iijiijjji xMxxxxMi
)()(),()(
Compatibilities (interactions)
external evidence
k
kkiiii xMxxb )()()(
message
belief (approximate marginal probability)
Belief Propagation on loopy graphs
ik
k
k
k
ij k
k
k
Mki
k
iikx
iijiijjji xMxxxxMi
)()(),()(
Compatibilities (interactions)
external evidence
k
kkiiii xMxxb )()()(
message
belief (approximate marginal probability)
Some facts about BP
• BP is exact on trees.
• If BP converges it has reached a local minimum of an objective function (the Bethe free energy Yedidia et.al ‘00 , Heskes ’02)often good approximation
• If it converges, convergence is fast near the fixed point.
• Many exciting applications: - error correcting decoding (MacKay, Yedidia, McEliece, Frey) - vision (Freeman, Weiss) - bioinformatics (Weiss) - constraint satisfaction problems (Dechter) - game theory (Kearns) - …
BP Related Algorithms
• Convergent alternatives (Welling,Teh’02, Yuille’02, Heskes’03)
• Expectation Propagation (Minka’01)
• Convex alternatives (Wainwright’02, Wiegerinck,Heskes’02)
• Linear Response Propagation (Welling,Teh’02)
• Generalized Belief Propagation (Yedidia,Freeman,Weiss’01)
• Survey Propagation (Braunstein,Mezard,Weigt,Zecchina’03)
Generalized Belief Propagation
kiik
xiijiij
jji
xMxxx
xM
i
)()(),(
)(
Idea: To guess the distribution of one of your neighbors, you ask your other neighbors to guess your distribution. Opinions get combined multiplicatively.
kiik
xiijiij
jji
xMxxx
xM
i
)()(),(
)(
BP GBP
Marginal Consistency
( )A AP x ( )B BP x
( )A B A BP x
\ \
( ) ( ) ( )A A B B A B
A A A B A B B Bx x x x
P x P x P x
Solve inference problem separately on each “patch”,then stitch them togetherusing “marginal consistency”.
Region Graphs (Yedidia, Freeman, Weiss ’02)
C=1C=1 C=1
C=… C=… C=…
C=…
C=…
C=… C=… C=… C=…
C=1
Region: collection of interactions & variables.
)(
1
Anc
cc
Stitching together solutions on local clusters by enforcing “marginal consistency” on their intersections.
