Solving LP and MIP Models with Piecewise Linear Objective Functions
Zonghao Gu Gurobi Optimization Inc. Columbus, July 23, 2014
Overview } Introduction } Piecewise linear (PWL) function ◦ Convex and convex relaxation
} Modeling ◦ Variables for pieces ◦ SOS2, binary formulations for non-convexity ◦ Direct handing
} Convex PWL objective ◦ How to extend primal and dual simplex
} Non-convex PWL objective ◦ How to extend branch-and-bound algorithm
} Possible future work
2
Introduction
3
Definition A linear program with separable PWL objec4ve func4on is an op4miza4on problem of the form
4
njlinearpiecewiseareWhere
nj
mitoSubject
Minimize
xcuxl
bxa
xc
jj
jjj
ij
n
jij
j
n
jj
,,1,)(
,,1,
,,1,
)(
1
1
…
…
…
=
=≤≤
==∑
∑
=
=
Types
} Convex cj(xj) ◦ Treated as LP
} Non-convex cj(xj) ◦ Treated as MIP
5
Motivations } Demands ◦ Models with true piecewise linear structures ◦ Approximation of nonlinear functions ◦ A lot of different applications ◦ Customer models and requests
} Traditional approaches ◦ One variable for each piece ◦ SOS2 or binary variables for non-convex function
} New approach ◦ Can we handle it directly to improve performance?
6
Previous Work quick incomprehensive survey
} Convex case ◦ Fourer and Marsten, Solving Piecewise-Linear
Programs: Experiments with a Simplex Approach, 1992 § Extend primal simplex to handle variables with piecewise
objective function directly § No piece variables § Use XMP subroutine library
} Non-convex case ◦ SOS2 formulation, Beale and Tomlin, 1970 ◦ Branch-and-cut without binary variables, Keha, de
Farias and Nemhauser, 2006 ◦ Any work without adding piece variables?
7
Piecewise Linear Function
8
Piecewise Linear Function } Definition c(x) = akx +bk, pk ≤ x ≤ pk+1, k = 1,…,t-1 where ak, bk, pk are constants for k = 1,…,t and p1 < p2 < … < pt
} Convex c((x+y)/2) ≤ (c(x) + c(y))/2
9
Convex PWL Function } Continuous } Slopes are non-decreasing
10
Non-convex PWL Function } There can be jumps at breaking points
11
Convex Relaxation } Convex relaxation r(x) is convex and r(x) ≤ c(x) for all x } Strongest convex relaxation ◦ A convex relaxation ◦ For all x, there exist x1 and x2 such that r(x) = α c(x1) + (1-α) c(x2) with x = αx1+(1- α) x2, 0 ≤ α ≤ 1 ◦ Hereafter, relaxation always means strongest one
} Relaxation of a PWL function ◦ is a convex PWL function ◦ May contain fewer pieces
12
Convex Relaxation
13
Finding Convex Relaxation } Algorithm ◦ Step 1: initialize a set of ordered points S = {(p1, c(p1)), (p2, c(p2))} ◦ Step 2: Loop k from 3 to t § Find max j, such that the slope between (pk, c(pk)) and point j
in S is larger than the slope between points j and j-1 in S § If no such j, set j = 1 § Remove the points after j from S and add (pk, c(pk)) to S
} Results ◦ S defines PWL convex relaxation ◦ Complexity O(t) § Linear, because each breaking point can be only removed
from S once § Quite similar to Graham scan algorithm for convex hull of a
finite points, but no sorting needed 14
Modeling
15
Direct PWL Formulation
16
njlinearpiecewiseareWhere
nj
mitoSubject
Minimize
xcuxl
bxa
xc
jj
jjj
ij
n
jij
j
n
jj
,,1,)(
,,1,
,,1,
)(
1
1
…
…
…
=
=≤≤
==∑
∑
=
=
Commonly Used Approachλ Formulation
} One variable for each piece of PWL function ◦ Suppose variable x have a PWL function c(x)
defined by t points, (p1, c1), (p2, c2), …, (pt, ct) ◦ We introduce variables λ1,λ2,…, λt for the points,
such that
17
)4(,...,1,0
)3(1
)2()(
)1(
1
1
1
tjfor
cxc
px
j
t
jj
j
t
jj
j
t
jj
=≥
=∑
∑=
∑=
=
=
=
λ
λ
λ
λ
Convex PWL Functions } Translation ◦ Substitute c(x) in the direct formulation by using
equations (2) ◦ Add equations (1) and (3), and inequalities (4)
} Pure LP ◦ Size can be much bigger, if PWL functions have a lot
of pieces ◦ Direct handling of PWL by extending simplex may
have a big advantage
18
Non-Convex PWL Functions } MIP formulations (SOS2 and binary) ◦ Substitute c(x) in the direct formulation by using equations (2) ◦ Add equations (1) and (3), and inequalities (4) ◦ Add either SOS2 constraints or binary variables
§ SOS2 formulation: add SOS2 constraints on λ variables § Binary formulation: add binaries y1, y2,…,yt-1 with following constraints λ1 ≤ y1 λj ≤ yj-1 + yj, for j = 2, …, t-1 λt ≤ yt-1 Σ yj = 1
} Relaxations ◦ Direct formulation: replace c(x) with r(x) ◦ Relaxations of the three formulations have the same objective
value ◦ Model size
§ Binary: biggest; SOS2: smaller; Direct: smallest (could be much smaller)
19
Simplex For convex PWL Formulation
20
Primal Simplex for LP } Important aspects ◦ Crash basis and phase I ◦ Pricing to find enter variable ◦ Ratio test to find leaving variable ◦ Linear algebra to compute and update basis
factorization and to solve equations (ftran, btran)
21
Primal Simplex for PWL LP } Important aspects ◦ Crash basis and phase I § Pretty much the same ◦ Pricing to find enter variable
§ Need to consider both directions for a nonbasic variable at a breaking point of PWL function
◦ Ratio test to find leaving variable § Different § Longer step ◦ Linear algebra to compute and update basis
factorization and to solve equations (ftran, btran) § Pretty much the same
22
Ratio Test of PWL Primal Simplex
} Example, consider the dictionary for ratio test x1 = 2.5 + 0.5 x3 + ∑ aj1 xj x2 = 10 + x3 + ∑ aj2 xj
x1: basic, PWL with points (0, 0), (1, 1), (2, 3), (3, 6), (4, 10) x2 : basic, no PWL, lb = 0, ub = inf x3 : nonbasic at 0, entering with reduced cost -1.7, PWL with points (0, 0), (2, 1), (4, 4), (6, 10) } Ratio test with a shorter step
x1 : (2.5-2)/0.5 = 1 x2: (10-0)/1 = 10 ◦ x3 enters the basis with step length 1, x1 leaves. It can be
mapped to the λ formulation ◦ No need to consider objective or reduced costs
23
Ratio Test of PWL Primal Simplex
} Ratio test with a longer step x1 : (2.5-2)/0.5 = 1, (2.5 – 1)/0.5 =3, (2.5-0)/0.5 = 5 x2: (10-0)/1 = 10 (> 5, eliminated) x3 : 2, 4, 6 (> 5 eliminated) Possible steps 1, 2, 3, 4, 5 with corresponding unit
objective changes -1.7, -1.2 (-1.7+0.5), -0.2(-1.2+1), 0.3(-0.2+0.5), 1.8(0.3+1.5) ◦ x3 enters the basis with step length 3, x1 leaves to 1. This is
equivalent to 3 iterations for the λ formulation ◦ We can use the median algorithm to do the ratio test. Its
complexity is O(m log(t)), which is usually much cheaper than solving (ftran, btran)
24
Dual Simplex for LP } Important aspects ◦ Crash basis and phase I § Bounded variables don’t matter much, cheaper ◦ Pricing to find leaving variable ◦ Ratio test to find entering variable § Basically solve an LP with a single constraint on
variables with possible lower and upper bounds ◦ Linear algebra to compute and update basis
factorization and to solve equations (ftran, btran)
25
Dual Simplex for PWL LP } Important aspects ◦ Crash basis and phase I § Different, not important, just do something simple ◦ Pricing to find leaving variable § Pretty much the same ◦ Ratio test to find leaving variable § Solve a PWL LP with a single constraint on variables with
possible lower and upper bounds § Median algorithm ◦ Linear algebra to compute and update basis factorization
and to solve equations (ftran, btran) § Pretty much the same ◦ Values for basic variables may change from one piece to
another for each iteration § It needs to address
26
Preliminary Computational Test } Model set ◦ 100 easy models from our LP set, many from netlib ◦ Replace objective of every variable with a PWL function with
10 pieces } Method ◦ Primal simplex ◦ No presolve
§ To avoid removing pieces } Comparison between direct PWL and λ formulations ◦ All 100 models:
§ 5.91X fewer iterations, 3.28X faster (too fast to be reliable) ◦ 21 models with > 1s runtime
§ 5.89X fewer iterations, 7.13X faster ◦ Still some issues in the code
§ Possible further fewer iterations and better runtime performance
27
Advantages of PWL Simplex } Faster iteration ◦ Because of smaller model size § Especially for dual simplex and for primal devex or
steepest edge pricing
} Fewer iterations ◦ One iteration often equals several iterations on the λ formulation
28
Branch-and-bound For non-convex PWL
Formulation
29
MIP Bound-and-Bound Solver } Important aspects ◦ Presolve § Stronger primal reductions like bound strengthening § Harder for dual reductions ◦ Solve relaxation ◦ Select variable to branch ◦ Cutting planes ◦ Heuristics § Any relaxation solution is MIP feasible
30
Solving Relaxation } Root relaxation ◦ Replace c(x) with convex relaxation r(x) and solve the
relaxation, the objective value is the same as that for the λ formulation before adding cuts
} Node relaxations ◦ Use changed bounds to update r(x) ◦ It is primal feasible for one branch ◦ Warm start with primal or dual depending on the
situation
} Comment ◦ The advantages of PWL simplex carry over
31
Selecting Variable } Variable Candidates ◦ Every variable x with r(x) < c(x) is a candidate ◦ Basic (not at a breaking point) vs nonbasic (at a breaking
point)
} Selecting ◦ Pick x with max c(x) – r(x)? ◦ Extend current pseudo, strong and reliability branching?
} Where to split ◦ Not good at non-breaking point ◦ Easy for nonbasic variable ◦ Two choices for basic variable, left or right breaking point
32
Choosing Split (1) } Pick right breaking point to x = a ◦ c(x) – r(x) doesn’t change at x = a
33
Choosing Split (2) } Pick left breaking point to x = a ◦ c(x) – r(x) becomes zero at x = a
34
Cutting Planes } Primal feasible regions ◦ Same for the relaxation and the original models ◦ Impossible to have primal cuts
} Cutting planes with dual arguments ◦ Possible and how?
35
Possible Future Work
36
Separable PWL Coefficients } Much more useful for approximating
nonlinear programs } The λ formulation ◦ Model size may be too big
} Extend simplex ◦ To handle the convex case directly to have faster
iterations and fewer iterations ◦ Harder
} Extend branch-and-bound algorithm ◦ To handle the non-convex case directly
37
Non-separable PWL Function } Separating ◦ We can separate Q by factorization and
introduction of new variables
} Mixed-integer models for nonseparable piecewise linear optimization: Unifying framework and extensions, Vielma, Ahmed and Nemhauser, 2009
38
Computational Tests } Translation the λ models back } Approximate convex QP and non-convex QP } Ask you and our customers to try and to send us
models } Ask you for ideas
39
Thank You
