oxford reading group
DESCRIPTION
Oxford Reading Group. Phil Torr (Oxford Brookes). Outline of Lecture. Motivation: What problems may be solved by Semi Definite Programming SDP : segmentation, matching, classification. What is SDP? How can it be implemented. Linear Programming. Includes many problems Graph cut - PowerPoint PPT PresentationTRANSCRIPT
Oxford Reading GroupOxford Reading Group
Phil Torr (Oxford Brookes)
Outline of LectureOutline of Lecture
Motivation: What problems may be solved by Semi Definite Programming SDP: segmentation, matching, classification.
What is SDP?
How can it be implemented.
Linear ProgrammingLinear Programming
Includes many problemsGraph cut
Segmentationstereo
Shortest pathTrackingObject recognitionStereo
Some L1 regressions
SDPSDP
Has been described as the most exciting mathematical programming development in the 1990’s (Robert M Freund; MIT).
The simplex method etc from LP can be generalized to SDP.
Classification by EllipsoidsClassification by Ellipsoids
Rosen [95].Constrain A matrices to be PSD.
Another Application: Another Application:
Recall Min Cut ProblemRecall Min Cut Problem
s-t graph cut
A graph with two terminals S and T
“source”
S T
“sink”
• Cut cost is a sum of severed edge weights • Minimum cost s-t cut can be found in polynomial time
Goal: divide the graph into two parts separating red and blue nodes
Segmentation - ModelSegmentation - Model
• Input: Image consisting of pixels
• Output: Segmentation of pixels
• Color model
• Coherence or Edge model
Input Output
5 Hard Degmentation – Probablistic Framework GRABGRABCUTCUT
Color ModelColor Model
Assignment variable:
Which mixture component does a pixel belong to?
7 Hard Degmentation – Probablistic Framework GRABGRABCUTCUT
GraphCut for InferenceGraphCut for Inference
Cut: A collection of edges which separates the Source from the Sink
MinCut: The cut with minimum weight (sum of edge weights)
Solution: Global optimum (MinCut) in polynomial time
Image
Sink
Source
Foreground
Background
Cut
21 Hard Degmentation – Probablistic Framework GRABGRABCUTCUT
22 Hard Degmentation – Probablistic Framework GRABGRABCUTCUT
GraphCut for InfernceGraphCut for Infernce
MinCut minimizes the energy of the MRF:
Image
Sink
Source
Foreground
Background
Cut
constant
MinCutMinCut
Edge weights must all be positive,
Then soluble in polynomial time, by max flow algorithm.
Weights are capacities therefore negative capacity does not make sense.
MaxCutMaxCut
Edge weights may be negative,
Note MaxCut and MinCut are same problem, however term MaxCut used when weights can be negative and positive.
MaxCut NP complete.
Negative weights, MaxCutNegative weights, MaxCut
Edge weights may be negative,
Problem is NP-hard
With SDP, an approximation ratio of 0.878 can be obtained! (Goemans-Williamson ’95), i.e. within 13% of the global energy minimum.
Why negative weightsWhy negative weights
In example above the MinCut produces an odd segmentation, negative weights encode the idea of repulsive force that might yield a better segmentation.
Pop outPop out
Maybe need attractive and repulsive forces to get Gestalt effects:
MaxCut Integer ProgramMaxCut Integer Program
GraphCut
LaplacianLaplacian
∑ wij (1 - xi xj ) = x┬ (diag(W 1) - W) x
Laplacian of graph
MaxCut Integer ProgramMaxCut Integer Program
Laplacian (semi positive definite)
Min/Max Cut minimizes this integer program (cf Hopfield network):
Solving via relaxationSolving via relaxation
Problem above NP complete
Some NP complete problems can be approximated using a “relaxation”, e.g. from binary to continuous variables etc.
Next semi definite programming is explained and then it is shown what relaxation can be used to help solve MaxCut.
Solving via relaxationSolving via relaxation
Keuchel et al 2002; suggest adding a constraint
Where e = (1, … 1) to favour partitions with equal numbers of nodes.
Solving via Solving via relaxationrelaxation
The vector e corresponds to no cut so is an eigenvector of L with eigenvalue 0, a natural relaxation of the problem is to drop the integer constraint and solve for the second smallest eigenvector of L (the Fiedler vector, c.f. Shi & Malik)
Eigenvector may or may not be binary.
Linear Programming (Linear Programming (LPLP))
Canonical form, inequalities removed by introduction of surplus variables.
LP ExampleLP Example
Semidefinite Programming Semidefinite Programming
If X and Ai are diagonal this is a linear program,<X,Y> also used for inner produce of two matrices.
Positive Semidefinite MatricesPositive Semidefinite Matrices
W is Gram matrix, not weightW is Gram matrix, not weight
Semi Definite ProgrammingSemi Definite Programming
Semi Definite ProgrammingSemi Definite Programming
If X diagonal then reduces to LP
The feasible solution space of SDP is convex.
Polynomial time exact solution.
Note most non trivial applications of SDP are equivalent to minimizing the sum of the first few eigenvalues of X with respect to some linear constraints on X.
Recall:Recall:
Min/Max Cut minimizes this integer program:
SDP relaxation of MaxCutSDP relaxation of MaxCut
The relaxation is to relax the rank one constraint for X; allowing X to be real valued. Bad news, many more variables.
Binary variables Binary variables
}1,1{ s.t.2
1Max
i
jiij
x
xxw
Graph Vertices: Unit VectorsGraph Vertices: Unit Vectors
1||||, s.t.2
1Max
in
i
jiij
vRv
vvw
AlgorithmAlgorithm
X is a continuous matrix, recovered by SDP.Recover Gram matrix V, X = VVT by Cholesky decomposition matrix, each vertex represented by a vector vi on unit sphere.Choose random hyperplane to bisect unit sphere and define cut.
RecallRecall
ji vv ji |v| cos (a)|v |
Vertices on same side of cut nearby on unit sphere yields large dot product, vertices far apart yield small (negative) dot product.
An SDP Relaxation of An SDP Relaxation of MAX CUT – MAX CUT –
Geometric intuitionGeometric intuition
Embed the vertices of the graph on the unit sphere such that vertices that are joined by edges are far apart.
Random separationRandom separation
vi
vj
Algorithm AnalysisAlgorithm Analysis
The probability that two vectors are separated by a The probability that two vectors are separated by a random hyperplane:random hyperplane:
Algorithm AnalysisAlgorithm Analysis
Calculate expected value of cut.
Note value of relaxation exceeds cost of MaxCut (as it lives in a less constrained space).
Note the following identity:
11
1
ratio min2
1 = 0.8785Cos ( ) 6..
x x
x
1
1
1
1
exp
sdp
ratio min
1
2
2
1
cos
0.8785
( )
cos (.
)6. .
ii
jj
ijj
x
i
w
v v
x
v
w
v
x
Expected Value of CutExpected Value of Cut
Classification by EllipsoidsClassification by Ellipsoids
Rosen [95].Constrain A matrices to be PSD.
Another Application: Another Application:
Set of matches m, which can be on or off, so that m = 0,1
Change of variable k = 2m -1 changes this to the max cut problem.