solving markov random fields using second order cone programming relaxations m. pawan kumar philip...

Solving Markov Random Fields using

Second Order Cone Programming Relaxations

M. Pawan Kumar

Philip Torr

Andrew Zisserman

Aim• Accurate MAP estimation of pairwise Markov random fields

2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Labelling m = {1, 0, 0, 1}

Random Variables V = {V1,..,V4}

Label Set L = {0,1}


2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Cost(m) = 2


2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Cost(m) = 2 + 1


2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Cost(m) = 2 + 1 + 2


2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1


2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1 + 3


2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1 + 3 + 1


2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3


2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Cost(m) = 2 + 1 + 2 + 1 + 3 + 1 + 3 = 13

Minimum Cost Labelling = MAP estimate

Pr(m) exp(-Cost(m))


2

5

4

2

6

3

3

7

0

1 1

0

0

2 3

1

1

4 1

0

V1 V2 V3 V4

Label ‘0’

Label ‘1’

Objectives• Applicable for all neighbourhood relationships• Applicable for all forms of pairwise costs• Guaranteed to converge

MotivationSubgraph Matching - Torr - 2003, Schellewald et al - 2005

G1

G2

Unary costs are uniform

V2 V3

V1

MRF

ABCD A

BCD

ABCD

MotivationSubgraph Matching - Torr - 2003, Schellewald et al - 2005

G1

G2

| d(mi,mj) - d(Vi,Vj) | <

12

YES NO

Potts Model

Pairwise Costs

Motivation

V2 V3

V1

MRF

ABCD A

BCD

ABCD

Subgraph Matching - Torr - 2003, Schellewald et al - 2005

MotivationMatching Pictorial Structures - Felzenszwalb et al - 2001

Part likelihood Spatial Prior

Outline

Texture

Image

P1 P3

P2

(x,y,,)

MRF

Motivation

Image

P1 P3

P2

(x,y,,)

MRF

• Unary potentials are negative log likelihoods

Valid pairwise configuration

Potts Model

Matching Pictorial Structures - Felzenszwalb et al - 2001

12

YES NO

Motivation

P1 P3

P2

(x,y,,)

Pr(Cow)Image

• Unary potentials are negative log likelihoodsMatching Pictorial Structures - Felzenszwalb et al - 2001

Valid pairwise configuration

Potts Model

12

YES NO

Outline• Integer Programming Formulation

• Previous Work

• Our Approach– Second Order Cone Programming (SOCP)– SOCP Relaxation– Robust Truncated Model

• Applications– Subgraph Matching– Pictorial Structures

Integer Programming Formulation2

5

4

2

0

1 3

0

V1 V2

Label ‘0’

Label ‘1’Unary Cost

Unary Cost Vector u = [ 5

Cost of V1 = 0

2

Cost of V1 = 1

; 2 4 ]

Labelling m = {1 , 0}


5

4

2

0

1 3

0

V1 V2

Label ‘0’


Unary Cost Vector u = [ 5 2 ; 2 4 ]T


Label vector x = [ -1

V1 0

1

V1 = 1

; 1 -1 ]T

Recall that the aim is to find the optimal x


5

4

2

0

1 3

0

V1 V2

Label ‘0’


Unary Cost Vector u = [ 5 2 ; 2 4 ]T


Label vector x = [ -1 1 ; 1 -1 ]T

Sum of Unary Costs = 12

∑i ui (1 + xi)


5

4

2

0

1 3

0

V1 V2

Label ‘0’

Label ‘1’Pairwise Cost


0Cost of V1 = 0 and V1 = 0

0

00

0Cost of V1 = 0 and V2 = 0

3

Cost of V1 = 0 and V2 = 11 0

00

0 0

10

3 0

Pairwise Cost Matrix P


5

4

2

0

1 3

0

V1 V2

Label ‘0’




0 0

00

0 3

1 0

00

0 0

10

3 0

Sum of Pairwise Costs14

∑ij Pij (1 + xi)(1+xj)


5

4

2

0

1 3

0

V1 V2

Label ‘0’




0 0

00

0 3

1 0

00

0 0

10

3 0

Sum of Pairwise Costs14

∑ij Pij (1 + xi +xj + xixj)

14

∑ij Pij (1 + xi + xj + Xij)=

X = x xT Xij = xi xj

Integer Programming FormulationConstraints

• Each variable should be assigned a unique label

∑ xi = 2 - |L|i Va

• Marginalization constraint

∑ Xij = (2 - |L|) xij Vb

Integer Programming FormulationChekuri et al. , SODA 2001

x* = argmin 12

∑ ui (1 + xi) + 14

∑ Pij (1 + xi + xj + Xij)

∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi {-1,1}

X = x xT

ConvexNon-Convex


• Previous Work



Linear Programming Formulation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi {-1,1}

X = x xT

Chekuri et al. , SODA 2001Retain Convex Part

Relax Non-convex Constraint


x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

X = x xT




x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]


Feasible Region (IP) x {-1,1}, X = x2


Feasible Region (IP)Feasible Region (Relaxation 1)

x {-1,1}, X = x2

x [-1,1], X = x2


Feasible Region (IP)Feasible Region (Relaxation 1)Feasible Region (Relaxation 2)

x {-1,1}, X = x2

x [-1,1], X = x2

x [-1,1]



• Bounded algorithms proposed by Chekuri et al, SODA 2001

• -expansion - Komodakis and Tziritas, ICCV 2005

• TRW - Wainwright et al., NIPS 2002

• TRW-S - Kolmogorov, AISTATS 2005

• Efficient because it uses Linear Programming

• Not accurate

Semidefinite Programming Formulation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi {-1,1}

X = x xT

Lovasz and Schrijver, SIAM Optimization, 1990Retain Convex Part


x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

X = x xT


Retain Convex Part


Lovasz and Schrijver, SIAM Optimization, 1990


x1

x2

xn

1

...

1 x1 x2... xn

1 xT

x X

=

Rank = 1

Xii = 1

Positive SemidefiniteConvex

Non-Convex


x1

x2

xn

1

...

1 x1 x2... xn

1 xT

x X

=

Xii = 1

Positive SemidefiniteConvex

Schur’s Complement

A B

BT C

=I 0

BTA-1 I

A 0

0 C - BTA-1B

I A-1B

0 I

0

A 0 C -BTA-1B 0


X - xxT 0

1 xT

x X

=1 0

x I

1 0

0 X - xxT

I xT

0 1


x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

X = x xT



Retain Convex PartLovasz and Schrijver, SIAM Optimization, 1990

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]


Xii = 1 X - xxT 0

Retain Convex PartLovasz and Schrijver, SIAM Optimization, 1990

Feasible Region (IP) x {-1,1}, X = x2


Feasible Region (IP)Feasible Region (Relaxation 1)

x {-1,1}, X = x2

x [-1,1], X = x2



x {-1,1}, X = x2

x [-1,1], X = x2

x [-1,1], X x2



• Formulated by Lovasz and Schrijver, 1990

• Finds a full X matrix

• Max-cut - Goemans and Williamson, JACM 1995

• Max-k-cut - de Klerk et al, 2000

• Accurate

• Not efficient because of Semidefinite Programming

Previous Work - Overview

LP SDP

ExamplesTRW-S,

-expansion

Max-k-Cut

Accuracy Low High

Efficiency High Low

Is there a Middle Path ???


• Previous Work



Second Order Cone Programming

Second Order Cone || v || t OR || v ||2 st

x2 + y2 z2

Minimize fTx

Subject to || Ai x+ bi || <= ciT x + di

i = 1, … , L

Linear Objective Function

Affine mapping of Second Order Cone (SOC)

Constraints are SOC of ni dimensions

Feasible regions are intersections of conic regions



|| v || t tI v

vT t0

LP SOCP SDP

=1 0

vT I

tI 0

0 t2 - vTv

I v

0 1



• Previous Work



Matrix Dot Product

A B = ∑ij Aij Bij

A11 A12

A21 A22

B11 B12

B21 B22

= A11 B11 + A12 B12 + A21 B21 + A22 B22

SDP Relaxation

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

Xii = 1 X - xxT 0

Derive SOCP relaxation from the SDP relaxation

FurtherRelaxation

1-D ExampleX - xxT 0

X - x2 ≥ 0

For two semidefinite matrices, the dot product is non-negative

A A 0

x2 X

SOC of the form || v ||2 st


x {-1,1}, X = x2

x [-1,1], X = x2

x [-1,1], X x2

SOCP Relaxation

Same as the SDP formulation

2-D Example

X11 X12

X21 X22

1 X12

X12 1

=X =

x1x1 x1x2

x2x1 x2x2

xxT =x1

2 x1x2

x1x2

=x2

2

2-D Example(X - xxT)

1 - x12 X12-x1x2. 0

1 0

0 0 X12-x1x2 1 - x22

x12 1

-1 x1 1

C1. 0 C1 0


1 - x12 X12-x1x2

C2. 0

. 00 0

0 1 X12-x1x2 1 - x22

x22 1

LP Relaxation-1 x2 1

C2 0


1 - x12 X12-x1x2

C3. 0

. 01 1

1 1 X12-x1x2 1 - x22

(x1 + x2)2 2 + 2X12


C3 0


1 - x12 X12-x1x2

C4. 0

. 01 -1

-1 1 X12-x1x2 1 - x22

(x1 - x2)2 2 - 2X12


C4 0

SOCP Relaxation

Consider a matrix C1 = UUT 0

(X - xxT)

||UTx ||2 X . C1

C1 . 0

Continue for C2, C3, … , Cn


Kim and Kojima, 2000

SOCP Relaxation

How many constraints for SOCP = SDP ?

Infinite. For all C 0

We specify constraints similar to the 2-D example

SOCP RelaxationMuramatsu and Suzuki, 2001

1 0

0 0

0 0

0 1

1 1

1 1

1 -1

-1 1

Constraints hold for the above semidefinite matrices


1 0

0 0

0 0

0 1

1 1

1 1

1 -1

-1 1

a + b

+ c + d

a 0

b 0

c 0

d 0

Constraints hold for the linear combination


a+c+d c-d

c-d b+c+d

a 0

b 0

c 0

d 0Includes all semidefinite matrices where

Diagonal elements Off-diagonal elements

SOCP Relaxation - A

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

Xii = 1 X - xxT 0

SOCP Relaxation - A

x* = argmin 12

∑ ui (1 + xi) + 14


∑ xi = 2 - |L|i Va

∑ Xij = (2 - |L|) xij Vb

xi [-1,1]

(xi + xj)2 2 + 2Xij (xi - xj)2 2 - 2Xij

Specified only when Pij 0

Triangular Inequality

• At least two of xi, xj and xk have the same sign

• At least one of Xij, Xjk, Xik is equal to one

Xij + Xjk + Xik -1Xij - Xjk - Xik -1-Xij - Xjk + Xik -1-Xij + Xjk - Xik -1

• SOCP-B = SOCP-A + Triangular Inequalities


• Previous Work



Robust Truncated ModelPairwise cost of incompatible labels is truncated

Potts ModelTruncated Linear Model

Truncated Quadratic Model

• Robust to noise

• Widely used in Computer Vision - Segmentation, Stereo

Robust Truncated ModelPairwise Cost Matrix can be made sparse

P = [0.5 0.5 0.3 0.3 0.5]

Q = [0 0 -0.2 -0.2 0]

Reparameterization

Sparse Q matrix Fewer constraints

Compatibility Constraint

Q(ma, mb) < 0 for variables Va and Vb

Relaxation ∑ Qij (1 + xi + xj + Xij) < 0

SOCP-C = SOCP-B + Compatibility Constraints

SOCP Relaxation

• More accurate than LP

• More efficient than SDP

• Time complexity - O( |V|3 |L|3)

• Same as LP

• Approximate algorithms exist for LP relaxation

• We use |V| 10 and |L| 200


• Previous Work



Subgraph MatchingSubgraph Matching - Torr - 2003, Schellewald et al - 2005

G1

G2

Unary costs are uniform

V2 V3

V1

MRF

ABCD A

BCD

ABCD

Pairwise costs form a Potts model

Subgraph Matching

• 1000 pairs of graphs G1 and G2

• #vertices in G2 - between 20 and 30

• #vertices in G1 - 0.25 * #vertices in G2

• 5% noise to the position of vertices

• NP-hard problem

Subgraph Matching

Method Time (sec) Accuracy (%)

LP 0.85 6.64

SDP-A 35.0 93.11

SOCP-A 3.0 92.01

SOCP-B 4.5 94.79

SOCP-C 4.8 96.18


• Previous Work



Pictorial Structures

Image

P1 P3

P2

(x,y,,)

MRF

Matching Pictorial Structures - Felzenszwalb et al - 2001

Outline

Texture


Image

P1 P3

P2

(x,y,,)

MRF

Unary costs are negative log likelihoods

Pairwise costs form a Potts model

| V | = 10 | L | = 200


LBP

GBP

SOCP


LBP

GBP

SOCP

LBP and GBP do not converge

Pictorial StructuresROC Curves for 450 +ve and 2400 -ve images

Conclusions• We presented an SOCP relaxation to solve MRF

• More efficient than SDP

• More accurate than LP, LBP, GBP

• #variables can be reduced for Robust Truncated Model

• Provides excellent results for subgraph matching and pictorial structures

Future Work

• Quality of solution– Additive bounds exist– Multiplicative bounds for special cases ??

• Message passing algorithm ??– Similar to TRW-S or -expansion– To handle image sized MRF

solving markov random fields using second order cone programming relaxations m. pawan kumar philip...

Documents

x ij x

label vector x

x x t x ij

x j slide

unique label x

v1v1 v2v2 label

optimal x slide

v1v1 v2v2 v3v3 v4v4