the factor graph approach to model-based signal processing

The Factor Graph Approach to Model-Based Signal Processing

Hans-Andrea Loeliger

2

Outline

Introduction

Factor graphs

Gaussian message passing in linear models

Beyond Gaussians

Conclusion

3

Outline

Introduction

Factor graphs


Beyond Gaussians

Conclusion

4

Introduction

Engineers like graphical notation

It allow to compose a wealth of nontrivial algorithms from tabulated “local” computational primitive

5

Outline

Introduction

Factor graphs


Beyond Gaussians

Conclusion

6

Factor Graphs

A factor graph represents the factorization of a function of several variables

Using Forney-style factor graphs

7

Factor Graphs cont’d

Example:

1 2 3 4( , , , , ) ( ) ( , , ) ( , , ) ( )f u w x y z f u f u w x f x y z f z

8


1 2 3 4 5( , , , , ) ( ) ( ) ( | , ) ( | ) ( | )f u w x y z f u f w f x u w f y x f z x

(a) Forney-style factor graph (FFG); (b) factor graph as in [3]; (c) Bayesian network; (d) Markov random field (MRF)

9


Advantages of FFGs:

suited for hierarchical modeling

compatible with standard block diagram

simplest formulation of the summary-product message update rule

natural setting for Forney’s result on FT and duality

10

Auxiliary Variables

Let Y1 and Y2 be two independent observations of X:

1 2 1 2( , , ) ( ) ( | ) ( | )f x y y f x f y x f y x

1 2 1 2( , ', ", , ) ( ) ( | ') ( | ") ( , ', ")f x x x y y f x f y x f y x f x x x

( , ', ") ( ') ( ")f x x x x x x x 1 2 1 2

' "

( , , ) ( , ', ", , ) ' "x x

f x y y f x x x y y dx dx

11

Modularity and Special Symbols

Let and with Z1, Z2 and X independent

The “+”-nodes represent the factors and

1 1Y X Z 2 2Y X Z

1 1( )x z y 2 2( )x z y

12

Outline

Introduction

Factor graphs


Beyond Gaussians

Conclusion

13

Computing Marginals

Assume we wish to compute

For example, assume that can be written as

1

1,...,

( ) ( ,..., )n

k

k k nx xexcept x

f x f x x

1 7( ,..., )f x x

1 7 1 1 2 2 3 1 2 3 4 4

5 3 4 5 6 5 6 7 7 7

( ,..., ) ( ) ( ) ( , , ) ( )

( , , ) ( , , ) ( )

f x x f x f x f x x x f x

f x x x f x x x f x

14

Computing Marginals cont’d

3 3 3 3( ) ( ) ( )C Df x x x

1 2

3 1 1 2 2 3 1 2 3,

( ) ( ) ( ) ( , , )Cx x

x f x f x f x x x

4 5

3 4 4 5 3 4 5 5,

( ) ( ) ( , , ) ( )D Fx x

x f x f x x x x

6 7

5 6 5 6 7 7 7,

( ) ( , , ) ( )Fx x

x f x x x f x

1 2

3 1 2 3 1 2 3,

( ) ( ) ( ) ( , , )C A Bx x

x x x f x x x

4 5

3 5 3 4 5 4 5,

( ) ( , , ) ( ) ( )D E Fx x

x f x x x x x

6 7

5 6 5 6 7 7,

( ) ( , , ) ( )F Gx x

x f x x x x

15

Message Passing View cont’d

1 2

3

4 5 6 7

5

3

3 3 1 2 3 1 2 3,

5 3 4 5 4 6 5 6 7 7, ,

( ) ( ) ( ) ( , , )

( , , ) ( ) ( , , ) ( )

X

X

X

A Bx x

E Gx x x x

f x x x f x x x

f x x x x f x x x x

��

�

�

16

Sum-Product Rule

The message out of some node/factor along some edge is formed as the product of and all incoming messages along all edges except , summed over all involved variables except

kX

lf

kX

kX

3 33 3 3 3( ) ( ) ( ) ��

X Xf x x x

lf

17

denotes the message in the direction of the arrow

denotes the message in the opposite direction

Arrows and Notation for Messages

��

X

�X

18

Marginals and Output Edges

'' "

'

( ) ( ') ( ") ( ") ( ') ' "

( ) ( )

X X Xx x

X X

x x x x x x x dx dx

x x

��

��

19

Max-Product Rule

The message out of some node/factor along some edge is formed as the product of and all incoming messages along all edges except , maximized over all involved variables except

kX

lf

kX

kX

11

,...,

( ) max ( ,..., )n

k

k k nx xexcept x

f x f x x

lf

20

Message of the form:

Arrow notation:

/ is parameterized by mean / and variance /

2 2( ) / 2( ) x mx e

Scalar Gaussian Message

��

X �X

��Xm

�Xm

2��

X2

�X

21

Scalar Gaussian Computation Rules

22

Vector Gaussian Messages

Message of the form:

Message is parameterized

either by mean vector m and covariance matrix V=W-1

or by W and Wm

( ) expH

x x m W x m

23

Vector Gaussian Messages cont’d

Arrow notation:

is parameterized by and or by and

Marginal:

is the Gaussian with mean and covariance matrix

��

X Xm��

XV��

XW��

X XW m��

( ) ( )X Xx x ��

Xm1

X XV W

k kk

k k k kk k

X XX

X X X XX X

W W W

W m W m W m

��

��

24

Single Edge Quantities

25

Elementary Nodes

26

Matrix Multiplication Node

27

Composite Blocks

28

Reversing a Matrix Multiplication

29

Combinations

30

General Linear State Space Model

1K K K K K

K K K

X A X B U

Y C X

31


If is nonsingular

and － forward

and － backward

If is singular

and － forward

and － backward

Cont’d

1kX

��1kY

�

kX��

kY�

'kX�

1'kX

�

kA

kA

1kX

��1kY

�

kX��

kU��

kX�

1kX

�

32


By combining the forward version with backward version, we can get

Cont’d

k kk

k k k kk k

X XX

X X X XX X

W W W

W m W m W m

��

��

33

Gaussian to Binary

( ) ( ) ( ) ( )X Y Y

y

x y x y dy x ��

2

( 1) 2ln

( 1)

YXX

X Y

mL

��

��

21

Y X

Y X X

m m

m

��

��

34

Outline

Introduction

Factor graphs


Beyond Gaussians

Conclusion

35

Message Types

A key issue with all message passing algorithms is the representation of messages for continuous variables

The following message types are widely applicable

Quantization of continuous variables

Function value and gradient

List of samples

36

Message Types cont’d

All these message types, and many different message computation rules, can coexist in large system models

SD and EM are two example of message computation rules beyond the sum-product and max-product rules

37

LSSM with Unknown Vector C

38

Steep Descent as Message Passing

Suppose we wish to find

( ) ( ) ( )A Bf f f

max arg max ( )f

(ln ( )) (ln ( )) (ln ( ))A B

d d df f f

d d d

39


Steepest descent:

where s is a positive step-size parameter

Cont’d

ln ( ) |old

new oldd

s fd

40


Gradient messages:

Cont’d

( ) ln ( )d

d

��

41

Steep Descent as Message PassingCont’d

( ) ( ) ( ) ( )

( ) ( )

X Yx y

X Yx

x x y x dxdy

x x dx

��

��

42

Outline

Introduction

Factor graphs


Beyond Gaussians

Conclusion

43

Conclusion

The factor graph approach to signal processing involves the following steps:

1) Choose a factor graph to represent the system model

2) Choose the message types and suitable message computation rules

3) Choose a message update schedules

44

Reference

[1] H.-A. Loeliger, et al., “The factor graph approach to model-based signal processing”

[2] H.-A. Loeliger, “An introduction to factor graphs,” IEEE Signal Proc. Mag., Jan. 2004, pp.28-41

[3] F.R. Kschischang, B.J. Fery, and H.-A. Loeliger, “Factor graphs and the sum-product algorithm,” IEEE Trans. Inform. Theory, vol. 47, pp.498-519

the factor graph approach to model-based signal processing

Documents