2007 : solving linear problems with mosek (seattle 2007)

http://www.mosek.com

Solving Linear OptimizationProblems with MOSEK.

Bo Jensen ∗

MOSEK ApS,Fruebjergvej 3, Box 16, 2100 Copenhagen,

Denmark.Email: [email protected]

INFORMS Annual Meeting Seattle Nov. 7, 2007

∗Erling D. Andersen

Introduction

2 / 26

Topics

Introduction

Topics

The linear optimizer

The simplexoptimizers

Computationalresults

Conclusions

3 / 26

■ The problem:(P ) min cT x

st Ax = b,

x ≥ 0.

■ The linear optimizers.

◆ Interior-point optimizer (Not main focus in this talk).◆ Simplex optimizer.

■ What is the recent improvements?■ What is the (relative) performance?


Introduction

Topics




Conclusions

4 / 26

The general flow :

■ Presolve.■ Form the reduced primal or dual.■ Scale (optimizer specific).■ Optimize (interior-point or simplex).■ Basis identification (interior-point only).■ Undo scaling and dualizing.■ Postsolve.

The simplex optimizers

5 / 26

What makes a good simplex optimizer ?

Introduction


What makes a goodsimplex optimizer ?

MOSEKsimplex-overview

Exploiting sparsityaggressively

Primal (dual)Degeneracy

Dual bound flippingidea used moreaggressively

Numerical stability

Network optimizer


Conclusions

6 / 26

■ Exploit sparsity (i.e. LU and FTRAN and BTRANroutines).

■ Exploit problem dependent structure.■ Choose right path (i.e. good pricing strategy).■ Long steps (i.e. avoid degeneracy).■ Numerical stability (i.e. reliable and consistent results).■ Fast hotstarts (i.e. MIP and other hotstart applications).■ Other tricks.

MOSEK simplex-overview

Introduction







Numerical stability

Network optimizer


Conclusions

7 / 26

■ Primal and dual simplex optimizer.

◆ Efficient cold start and warm start.◆ Crashes an initial basis.◆ Multiple pricing options:

■ Full (Dantzig).■ Partial.■ Approximate/exact steepest edge.■ Hybrid.

◆ Degeneration handling.

■ Revised simplex algorithm + many enhancements.■ Many enhancements still possible!.

Exploiting sparsity aggressively

Introduction







Numerical stability

Network optimizer


Conclusions

8 / 26

■ Simplex algs. require solution of the linear equationsystems

Bf = A:j and BT g = ei.

in each iteration.


Introduction







Numerical stability

Network optimizer


Conclusions

8 / 26



in each iteration.■ Assume a sparse LU factorization of the basis

B = LU.


Introduction







Numerical stability

Network optimizer


Conclusions

8 / 26




B = LU.

■ f can be computed as follow. Solve

Lf̄ = A:j

and thenUf = f̄ .


Introduction







Numerical stability

Network optimizer


Conclusions

8 / 26




B = LU.

■ f can be computed as follow. Solve

Lf̄ = A:j

and thenUf = f̄ .

■ Simple implementation requires O(nz(L) + nz(U)) flops.

Exploiting sparsity aggressively (continued)

Introduction







Numerical stability

Network optimizer


Conclusions

9 / 26

■ Consider the simple example:

10 1x 0 1

f̄1

f̄2

f̄3

=

0x

0


Introduction







Numerical stability

Network optimizer


Conclusions

9 / 26


10 1x 0 1

f̄1

f̄2

f̄3

=

0x

0

■ Clearly sparsity in the RHS can be exploited! (doneextensively in MOSEK).


Introduction







Numerical stability

Network optimizer


Conclusions

9 / 26


10 1x 0 1

f̄1

f̄2

f̄3

=

0x

0


■ Gilbert and Peierls [GIL:88] demonstrate how to solve thetriangular system in O(minimal number of flops).


Introduction







Numerical stability

Network optimizer


Conclusions

9 / 26


10 1x 0 1

f̄1

f̄2

f̄3

=

0x

0



■ Aim: Solves with L and U and updates to the LU shouldrun in O(minimal number of flops) and not in O(m) forinstance.


Introduction







Numerical stability

Network optimizer


Conclusions

9 / 26


10 1x 0 1

f̄1

f̄2

f̄3

=

0x

0



■ Aim: Solves with L and U and updates to the LU shouldrun in O(minimal number of flops) and not in O(m) forinstance.

■ Drawback: Both L and U must be stored row and columnwise because solves with LT and UT are required too.

Primal (dual) Degeneracy

Introduction







Numerical stability

Network optimizer


Conclusions

10 / 26

The simplex optimizer may take very small or zero step sizes,why ?


Introduction







Numerical stability

Network optimizer


Conclusions

10 / 26


■ Primal step size δp:lB ≤ xB − δpB

−1aq ≤ uB


Introduction







Numerical stability

Network optimizer


Conclusions

10 / 26



−1aq ≤ uB

■ Basic variables on a bound may imply a zero primal step.


Introduction







Numerical stability

Network optimizer


Conclusions

10 / 26



−1aq ≤ uB

■ Basic variables on a bound may imply a zero primal step.■ Dual step size δd:

cj − yT Aj − (+)δd(eiB−1N)j ≥ 0 ∀j ∈ NL

cj − yT Aj − (+)δd(eiB−1N)j ≤ 0 ∀j ∈ NU


Introduction







Numerical stability

Network optimizer


Conclusions

10 / 26



−1aq ≤ uB




■ Non basic variables with zero reduced cost may imply azero dual step.

Degeneration posses both a theoretical and a practicalproblem for the simplex optimizer !


Introduction







Numerical stability

Network optimizer


Conclusions

10 / 26



−1aq ≤ uB






What is our options ?


Introduction







Numerical stability

Network optimizer


Conclusions

10 / 26



−1aq ≤ uB






What is our options ?

One approach is to perturb lj and uj (cj).

Primal (dual) Degeneracy (continued)

Introduction







Numerical stability

Network optimizer


Conclusions

11 / 26

MOSEK 5 has been improved on degenerated problems:

■ Better and more aggressive perturbation scheme.


Introduction







Numerical stability

Network optimizer


Conclusions

11 / 26


■ Better and more aggressive perturbation scheme.■ Sparsity issues important (very tricky).


Introduction







Numerical stability

Network optimizer


Conclusions

11 / 26


■ Better and more aggressive perturbation scheme.■ Sparsity issues important (very tricky).■ Clean up perturbations with dual (primal) simplex.


Introduction







Numerical stability

Network optimizer


Conclusions

11 / 26


■ Better and more aggressive perturbation scheme.■ Sparsity issues important (very tricky).■ Clean up perturbations with dual (primal) simplex.■ Many examples where ”tailed” solves are substantial

reduced.


Introduction







Numerical stability

Network optimizer


Conclusions

11 / 26


■ Better and more aggressive perturbation scheme.■ Sparsity issues important (very tricky).■ Clean up perturbations with dual (primal) simplex.■ Many examples where ”tailed” solves are substantial

reduced.■ Still room for improvement.

Dual bound flipping idea used more aggressively

Introduction







Numerical stability

Network optimizer


Conclusions

12 / 26

Dual step size δd :cj − yT Aj − (+)δd(eiB

−1N)j ≥ 0 ∀j ∈ NL


A ranged variable i.e. −∞ < lj < xj < uj < ∞ may not bebinding in dual min-ratio if profitable.


Introduction







Numerical stability

Network optimizer


Conclusions

12 / 26


−1N)j ≥ 0 ∀j ∈ NL



■ This involves flipping nonbasic variables to oppositebound to remain dual feasible and cost one extra solve.


Introduction







Numerical stability

Network optimizer


Conclusions

12 / 26


−1N)j ≥ 0 ∀j ∈ NL




■ Longer dual steplengths.


Introduction







Numerical stability

Network optimizer


Conclusions

12 / 26


−1N)j ≥ 0 ∀j ∈ NL




■ Longer dual steplengths.■ Reduces degeneracy.


Introduction







Numerical stability

Network optimizer


Conclusions

12 / 26


−1N)j ≥ 0 ∀j ∈ NL




■ Longer dual steplengths.■ Reduces degeneracy.■ Less iterations.


Introduction







Numerical stability

Network optimizer


Conclusions

12 / 26


−1N)j ≥ 0 ∀j ∈ NL




■ Longer dual steplengths.■ Reduces degeneracy.■ Less iterations.■ More flexibility in pivot choice (i.e. potentially more

stable).


Introduction







Numerical stability

Network optimizer


Conclusions

12 / 26


−1N)j ≥ 0 ∀j ∈ NL




■ Longer dual steplengths.■ Reduces degeneracy.■ Less iterations.■ More flexibility in pivot choice (i.e. potentially more

stable).■ Improves sparsity of the basis when degenerated! (i.e. if

xBi becomes feasible no basis exchange is needed).

Dual bound flipping idea used more aggressively(continued)

Introduction







Numerical stability

Network optimizer


Conclusions

13 / 26

Bound flipping examples:

Iter Time

Problem Rows Cols NB WB NB WB

osa-60 10280 232966 6938 5111 58.12 8.84

world 34506 32734 54566 32606 218.81 50.03

pds-40 66844 212859 34274 26599 96.51 18.48

ken-18 105127 154699 151203 51452 258.18 13.92

client 27216 20567 80555 63660 208.40 84.09

WB = MOSEK 5 Dual simplex with bound flips

NB = MOSEK 5 Dual simplex with no bound flips

Numerical stability

Introduction







Numerical stability

Network optimizer


Conclusions

14 / 26

■ Improving numerical stability.

◆ Moved LU update before updating solution.

Numerical stability

Introduction







Numerical stability

Network optimizer


Conclusions

14 / 26



■ Saves one solve with L in eiT B−1 [GOL:77].

■ More stable approach.

Numerical stability

Introduction







Numerical stability

Network optimizer


Conclusions

14 / 26





◆ Better handling of singularities (sing. variables aretemporary fixed).

Numerical stability

Introduction







Numerical stability

Network optimizer


Conclusions

14 / 26





◆ Better handling of singularities (sing. variables aretemporary fixed).

◆ Switch to safe mode if deemed unstable.

Network optimizer

Introduction







Numerical stability

Network optimizer


Conclusions

15 / 26

MOSEK 5 features a network simplex optimizer.

■ Solves pure network flow problems (i.e. LP’s with twonon-zeros in each column either 1 or -1).

■ Can extract embedded network structure in a model (i.e.network with side constraints).

■ Using standard interface, only one parameter has to beset.

■ Huge problems can be solved in limited time, for instancea problem with 8 million variables can be solved in lessthan 200 seconds.

Computational results

16 / 26

Test setup

Introduction



Test setup

Network Vs.Standard simplex

Primal Simplex

Dual Simplex

Numerical difficultproblems-primalsimplex

Numerical difficultproblems-dualsimplex

Conclusions

17 / 26

■ 577 problems (mixed size).■ A Dual Core server with 4GB RAM running Windows

2003 (Intel CPU).■ A Quad Core server with 8GB RAM running Windows

2003 (Intel CPU).■ See [HM:07] for a benchmark comparing Mosek with

other solvers.

All results presented in one table is obtained using one of thetwo computers only.

Network Vs. Standard simplex

Introduction



Test setup


Primal Simplex

Dual Simplex



Conclusions

18 / 26

small mediumnetw psim dsim netw psim dsim

Num. 30 30 30 43 43 43Firsts 30 0 1 43 0 0Total time 13.7 114.8 27.8 589.9 10676.6 3015.2G. avg. 0.39 2.42 0.70 6.30 91.74 19.70

largenetw psim dsim

Num. 2 2 2Firsts 2 0 0Total time 366.3 2905.8 968.9G. avg. 182.98 1115.71 468.76

Table 1: Performance of the network flow, primal simplex anddual simplex optimizer on pure network problems.

Primal Simplex

Introduction



Test setup


Primal Simplex

Dual Simplex



Conclusions

19 / 26

small medium large5 4 5 4 5 4


Table 2: Performance of the version 4 and version 5 primalsimplex optimizer

Dual Simplex

Introduction



Test setup


Primal Simplex

Dual Simplex



Conclusions

20 / 26



Table 3: Performance of the version 4 and version 5 dual sim-plex optimizer

Numerical difficult problems-primal simplex

Introduction



Test setup


Primal Simplex

Dual Simplex



Conclusions

21 / 26


Num. 9 9 19 19 2 2Firsts 5 5 13 6 2 0Total time 2.7 2.8 235.9 319.6 1297.7 1503.3G. avg. 0.19 0.18 7.19 9.54 413.26 464.04Fails 0 0 0 3 0 3

Table 4: Performance of the version 4 and 5 of the primal sim-plex optimizer on numerical difficult problems.

Numerical difficult problems-dual simplex

Introduction



Test setup


Primal Simplex

Dual Simplex



Conclusions

22 / 26


Num. 11 11 19 19 4 4Firsts 7 6 13 6 4 0Total time 3.9 6.6 3198.3 345.9 4736.3 12820.5G. avg. 0.24 0.31 8.44 9.67 802.24 2525.35Fails 0 0 0 1 0 1

Table 5: Performance of the version 4 and 5 dual simplex opti-mizer on numerical difficult problems.

Conclusions

23 / 26

Conclusions

Introduction



Conclusions

ConclusionsA number openissues exists

References

24 / 26

■ Simplex:

◆ MOSEK 5 substantial faster than MOSEK 4.◆ MOSEK 5 more stable than MOSEK 4.◆ Dual simplex faster than primal.

A number open issues exists

Introduction



Conclusions


References

25 / 26

■ Simplex:

◆ Degeneracy (non-perturbation method might beneeded in extreme cases).

◆ Improve primal pricing.◆ Better crashing on special problems.◆ Choose more sparse path.

References

Introduction



Conclusions


References

26 / 26

[HM:07] H.Mittelmann http://plato.la.asu.edu/bench.html

[GIL:88] J. R. Gilbert and T. Peierls, ”Sparse partial pivoting in timeproportional to arithmetic operations”, SIAM J. Sci. Statist.Comput., 9, 1988, pp. 862–874.

[GOL:77] D. Goldfarb, ”On the Bartels-Golub decomposition forlinear programming bases,” Mathematical. Programming, 13,1977, pp 272-279

[KOS:02] E. Kostina, ”The Long Step Rule in the Bounded-VariableDual Simplex Method: Numerical Experiments”,Mathematical Methods of Operations Research, 55 2002, I. 3.

[MAR:03] Maros I, ”A Generalized Dual Phase-2 SimplexAlgorithm”, European Journal of Operational Research, 149,2003, pp. 1–16

2007 : solving linear problems with mosek (seattle 2007)

Documents