difference of convex (dc) decomposition of nonconvex polynomials with algebraic techniques georgina...
TRANSCRIPT
1
Difference of Convex (DC) Decomposition of
Nonconvex Polynomials with Algebraic Techniques
Georgina HallPrinceton, ORFE
Joint work with Amir Ali AhmadiPrinceton, ORFE
7/13/2015 MOPTA 2015
2
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Difference of Convex (DC) programming
• Problems of the form
where:
• , ,• , are convex.
3
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Concave-Convex Computational Procedure (CCCP)
• Heuristic for minimizing DC programming problems.• Has been used extensively in: • machine learning (sparse support vector machines (SVM), transductive SVMs,
sparse principal component analysis)• statistical physics (minimizing Bethe and Kikuchi free energies).
• Idea:Input
x initial point
Convexify by linearizing x
convex affineconvex
Solve convex subproblem
Take to be the solution of
𝑘≔𝑘+1𝒇 𝒊𝒌 (𝒙 )
𝒇 𝒊(𝒙)
4
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Concave-Convex Computational Procedure (CCCP)
• Toy example: , where
Initial point:
Convexify to obtain
Minimize and obtain
Reiterate
𝑥0𝑥0𝑥1𝑥2𝑥3𝑥4𝑥∞
5
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
CCCP for nonconvex polynomial optimization problems (1/2)
CCCP relies on input functions being given as a difference of convex functions.
We will consider polynomials in variables and of degree
• Any polynomial can be written as a difference of convex polynomials. • Proof by Wang, Schwing and Urtasun• Alternative proof given later in this presentation, as corollary of stronger
theorem
What if we don’t have access to such a decomposition?
6
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
CCCP for nonconvex polynomial optimization problems (2/2)
• In fact, for any polynomial, an infinite number of decompositions.Example
xPossible decompositions
Which one would be a natural choice for CCCP?
7
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Picking the “best” decomposition (1/2)Algorithm
Linearize around a point to obtain convexified version of
Idea
Pick such that it is as close as possible to affine
Mathematical translation
Minimize curvature of ( is the hessian of
At a point
s.t. convex
Over a region
s.t. convex
8
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Picking the “best” decomposition (2/2)Theorem: Finding the “best” decomposition of a degree-4 polynomial over a box is NP-hard.
Proof idea: Reduction via testing convexity of quartic polynomials is hard (Ahmadi, Olshevsky, Parrilo, Tsitsiklis).
The same is likely to hold for the point version, but we have been unable to prove it.
How can we efficiently find such a decomposition?
9
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex relaxations for DC decompositions (1/6)
SOS, DSOS, SDSOS polynomials (Ahmadi, Majumdar)
• Families of nonnegative polynomials.
Type Characterization Testing membership
Sum of squares (sos) , polynomials, s.t. SDP
Scaled diagonally dominant sum of squares (sdsos)
p monomials, SOCP
Diagonally dominantsum of squares (dsos)
p LP
⇓
⇓
10
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex relaxations for DC decompositions (2/6)
DSOS-convex, SDSOS-convex, SOS-convex polynomials
Definitions:• is dsos-convex if is dsos.• is sdsos-convex if is sdsos.• is sos-convex if is sos.
convex⇔𝐻𝑝 (𝑥 )≽ 0 ,∀ 𝑥⇔ ⇐ sos/sdsos/dsos
LP
SOCP
SDP
11
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex relaxations for DC decompositions (3/6)Comparison of these sets on a parametric family of polynomials:
𝑐=−0.5
𝑎 𝑎
𝑏 𝑏
dsos-convex sdsos-convex sos-convex=convex
𝑎
𝑏
𝑐=0 𝑐=1
12
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
is sdd diagonal, s.t. dd.
is diagonally dominant (dd)
Convex relaxations for DC decompositions (4/6)
Original problem
s.t. convex
s.t.
convex
⇔
How to use these concepts to do DC decomposition at a point ?
Relaxation 1: sos-convex
s.t.
sos-convex
SDP
Relaxation 2: sdsos-convex
s.t.
sdsos-convex
SOCP + “small” SDP
Relaxation 3: dsos-convex
s.t.
dsos-convex
LP + “small” SDP
Relaxation 4: sdsos-convex+sdd
s.t. sdd (**) sdsos-convex
SOCP
Relaxation 5: dsos-convex + dd
s.t. dd (*)
dsos-convexLP
13
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex relaxations for DC decompositions (5/6)
Can any polynomial be written as the difference of two dsos/sdsos/sos convex polynomials?Lemma about cones: Let a full dimensional cone ( any vector space). Thenany can be written as .Proof sketch:
KE
𝒌𝒌 ′
such that
⇔𝑣=1
1−𝛼𝑘′−
𝛼1−𝛼
𝑘
𝑘1∈𝐾𝑘2∈𝐾𝒗
¿ :𝑘 ′
14
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Convex relaxations for DC decompositions (6/6)
Theorem: Any polynomial can be written as the difference of two dsos-convex polynomials.Corollary: Same holds for sdsos-convex, sos-convex and convex.Proof idea:• Need to show that dsos-convex polynomials is full-dimensional cone.• “Obvious” choices (i.e., ) do not work.
Induction on : for take
𝑎0>2 (𝑑−2 )𝑑 (𝑑−1)
+ 𝑑4 (𝑑−1)
𝑎𝑑4
𝑎1=1 𝑎𝑘+1=( 𝑑−2𝑘2𝑘+2 )𝑎𝑘 ,𝑘=1 ,…, 𝑑4−1
15
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
s.t. psd/sdd/dd,
s/d/sos-convex
Comparing the different relaxations (1/4)• Impact of relaxations on solving
for random ().
Type of relaxationTime (s) Opt value Time (s) Opt Value Time (s) Opt value
dsos-convex + dd 1.05 17578.54 2.79 21191.55 20.80 168327.89
dsos-convex + psd 1.19 15855.77 3.19 19426.13 25.36 146847.73
sdsos-convex + sdd 1.21 1089.41 5.17 1962.64 34.66 7936.57
sdsos-convex + psd 1.21 1069.79 5.29 1957.03 39.43 7935.72
sos-convex + psd MOSEK 2.02 193.07 93.74 317.63 ------------------
sos-convex + psd SEDUMI 11.48 193.06 10324.12 317.63 ------------------
Computer: 8Gb RAM, 2.40GHz
processor
16
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Comparing the different relaxations (2/4)• Iterative decomposition algorithm implemented for unconstrained
Decompose using one of the relaxations at
point
Minimize convexified using an SDP subroutine [Lasserre; de Klerk and
Laurent]
DSOS DD DSOS PSD SDSOS SDD SDSOS PSD SOS PSD
-250000
-200000
-150000
-100000
-50000
0
• Value of the objective after 3 mins. • Algorithm given above.• 5 different relaxations used• random with , • Average over 25 iterations• Solver: Mosek
17
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Comparing the different relaxations (3/4)• Constrained case: where
Minimize convexified
Relaxation:s.t.
sdsos convex
Decompose at a point
Iterative decomposition
Minimize convexified
What relaxation to use?Decompose over B
Original problem: s.t.
convex
Minimize convexified
Relaxation:s.t.
sdsos convex
Decompose once at
Single decomposition One min-max decomp.vs vs
Equivalent formulation:
convex
First relaxation:
sdsos-convex
Second relaxation:
sos sdsos-convex
18
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Comparing the different relaxations (4/4)• Constrained case: single decomposition vs. iterative decomposition
vs. min-max decomposition
• Value of the objective after 3 mins. • Algorithms described above.• random with , • Radius random integer between
100 and 400.• Average over 200 iterations
Single decomp
Iter decomp Min max
-16000
-14000
-12000
-10000
-8000
-6000
-4000
-2000
0
2000
4000
19
DC Decomposition of Nonconvex Polynomials with Algebraic Techniques
Main messages• To apply CCCP to polynomial optimization, a DC decomposition is
needed. Choice of decomposition impacts convergence speed.
• Not computationally tractable to find “best” decomposition.
• Efficient convex relaxations based on the concepts of dsos-convex (LP), sdsos-convex (SOCP), and sos-convex (SDP) polynomials.
• Dsos-convex and sdsos-convex scale to a larger number of variables.
20
Thank you for listeningQuestions?