1

Markov Random Fields (MRF)Presenter Kuang-Jui HsuDate 2011/5/23(Tues.)Outline Introduction

Conditional Independence Properties

Factorization Properties

Illustration: Image De-noising

Relation to Directed Graphs

IntroductionBased on a undirected graph

The MRF model has a simple form and is easy to use

Based on conditional independence properties

Conditional Independence Properties In an undirected graph, there are three sets of nodes, denoted A, B, C, and A is conditionally independent of B given C

Shorthand notation: p(A|B, C) = p(A|C)Conditional independence propertyTesting Methods in a Graph

Testing Methods in a Graph

Testing Methods in a Graph

Testing Methods in a Graph

Testing Methods in a Graph

Testing Methods in a Graph

Testing Methods in a Graph

Simple Form

A node will be conditionally independent of all other nodes conditioned only on neighbouring nodesFactorization Properties In a directed graph

Generalized form:In an Undirected Graph

Consider two nodes and that are not connected.

Must be conditionally independent

So, the conditional independence property can be expressed as The set x of all variables with and removed

Factorization PropertyCliqueThis leads us to consider a graphical concept: Clique

Clique:

Maximal Clique:

Define the factors in the potential function by using the clique

Generally, consider the maximal cliques, because other cliques must be the subsets of maximal cliquesPotential FunctionPotential FunctionPotential function over the maximal cliques of the graph

CliqueThe set of variables in that clique

The joint distribution: Partition function: a normalization constant

Equal to zero or positivePartition FunctionThe normalization constant is the major limitationsA model with M discrete nodes each having K states, then the evaluation involves summing over states

Exponential growth Needed for parameter learningBecause it will be a function of any parameters that govern the potential functions

Connection between Conditional Independence And FactorizationDefine :

For any node , the following conditional property holds

All nodes expect

The neighborhood of

Define :

A distribution can be expressed as

The Hammerley-Clifford theorem states that the sets and identical.

Potential Function ExpressionRestrict the potential function to be positive It is convenient to express them as exponentials

Energy functionBoltzmann distribution

The total energy is obtained by adding the energies of each of the maximal energy

Illustration: Image De-noising

Noisy imageDescribed by an array of binary pixel values , where the index i = 1, . . ., D runs over all pixels.

Illustration: Image De-noisingNoise-free imageDescribed by an array of binary pixel values , and randomly flipping the sign of pixels with some small probability

Create the MRF ModelA strong correlation between and

A strong correlation between the neighbouring pixles

MRF model:

The graph has two types of cliques, each of which contain two variables.

The clique form , uses the form of the energy function

The clique form , uses the form of the energy function

The parameters and are positive, and are neighbour

The Energy Function The complete energy function:

The joint distribution

1. postitve2. negativeSolve by ICMFor the purpose of image restoration, find an image x having a high probabilityUse a simple iterative technique called iterated condition mode ( ICM)Simply an application of coordinate-wise gradient ascent The steps of ICM

Evaluate the total energy for -1 and 1choose the lower energy, and update Stop until convergence

Result

Use ICMUse graph-cutRelation to Directed Graphs

Solve the problem of taking a model that is specified using a directed graph and trying to convert it to undirected graph

Directed graph

Undirected graph

Relation to Directed Gaphs

This is easily done by identifying

Relation to Directed Graphs

Consider how to generalize this constructionThis can be achieved if the clique potentials of the undirected graph are given by the conditional distributions of the directed graph.

Ensure that the set of variables that appears in each of conditional distributions is a member of at least one clique of the undirected graph Generalize This Construction

For nodes having one parent

For nodes having more than one parent

Involving the four variables, so they must belong to a single clique if this conditional distribution is to be absorbed in a clique potential

The process has become known as moralizationMoral graphConvert the Directed Graph to the Undirected Graph Discard some conditional independence propertiesConvert the Directed Graph to the Undirected Graph In fact,we can simply using a fully connected undirected graph However, this would discard all conditional propertiesThe moralization adds the fewest extra links and so retain the maximum number of independence properties Special Graph

There are two type of graph that can express different conditional independence properties Type 1: dependence map(D-map)Type 2: Independence map(I-map)Dependence Map(D-Map)

Every conditional independence statement satisfied by the distribution is reflected in the graphA completely disconnected graphIndependence Map(I-Map)

Every conditional independence statement implied by a graph is satisfied by a specific distributionA full connected graphA perfect map: both I-map and D-mapPerfect Map

The set of all distributions P over a given set of variablesDirectedgraphUndirectedgraph