1

Linear & Integer Programming : Introduction

Engg System Design Optimization

M S Prasad

General Optimization Program

• Standard form:

• where,

• Too general to solve, must specify properties of X, f,g and h more precisely.

Complexity Analysis

• (P) – Deterministic Polynomial time algorithm• (NP) – Non-deterministic Polynomial time

algorithm, – Feasibility can be determined in polynomial time

• (NP-complete) – NP and at least as hard as any known NP problem

• (NP-hard) – not provably NP and at least as hard as any NP problem,– Optimization over an NP-complete feasibility

problem

Optimization Problem Types – Real Variables

• Linear Program (LP)– (P) Easy, fast to solve, convex

• Non-Linear Program (NLP)– (P) Convex problems easy to solve– Non-convex problems harder, not guaranteed to find global

optimum

Let: X1, X2, X3, ………, Xn = decision variables

Z = Objective function or linear function

Requirement: Maximization of the linear function Z.Z = c1X1 + c2X2 + c3X3 + ………+ cnXn

…..Eq (1)

subject to the following constraints:

Problems

6

Linear programming (LP) model

• LP:

• Matrix form:

)n,,1j(0xbxa...xaxa

bxa...xaxabxa...xaxa

:tosubjectxc...xcxcmin

j

mnmn22m11m

2nn2222121

1nn1212111

nn2211

0xbAxxcmin T

objective function

constraints

variablerestrictions

where:x, c: n-vectorA: m,n-matrixb: m-vector

7

Linear programming example

)2,1j(0x12x4x3

4xx2:tosubjectxxmax

j

21

21

21

00

xx

124

xx

4312

:tosubjectxx

11max

2

1

2

1

2

1T

or:

8

Linear programming example: graphical

solution (2D)

1 2 3 4 5 6x1

1

2

3

4

56x2

)objective(6xx 21

4xx2 21

12x4x3 21

)2,1j(0x12x4x3

4xx2:tosubjectxxmax

j

21

21

21

0solutionspace

9

Linear programming (cont.)

• Decision variables can take the fractional values. ( May not suitable in some areas such as Resource mgmt , allocation etc)

• Solution techniques:– (dual) simplex method– interior point methods (e.g. Karmarkar

algorithm)

10

Integer programming (IP) models

• Integer variable restriction– IP: integer variables only– MIP: part integer, part non-integer variables– BIP: binary (0-1) variables

• General IP-formulation:

• Complex solution space)Zx(integer x

bAxxcmin T

11

Integer programming example: graphical solution

(2D)

1 2 3 4 5 6x1

1

2

3

4

56x2

)objective(6xx 21

)2,1j(Zx12x4x3

4xx2:tosubjectxxmax

j

21

21

21

02 optimal solutions!

12

Total unimodularity property for integer programming

modelsSuppose that all coefficients are integer in the model:

i.e.

Example: transportation problem

if A has the total unimodularity property(i.e. every square submatrix has determinant 0,1,-1)there is an optimal integer solution x*& the simplex method will find such a solution

0xbAxxcmin T

j,i,b,a iij

13

Integer programming tricks

PROBLEM: x = 0 or x k

use binary indicator variable y=

restrictions:

kfor x,0for x ,

10

0,1yykx

x)on upperbound an is (M yMx

14

PROBLEM: fixed costs: if xi>0 then costs C(xi)

use indicator variable yi=

restrictions :

Integer programming tricks (2)

0xbAx

)x(C minimize

0.for x0,for x xck

0)x(C

:where

i

i

iiii

0for x,0for x ,

10

i

i

0,1yxcyk)x(C

yMx

i

iiiii

ii

)i(

15

• Hard vs. soft restrictions– hard restriction: must hold, otherwise

unfeasibilityfor example:

– soft restriction: may be violated, with a penaltyfor example:

(Integer) programming tricks (3)

0Y,0x

Y5xx100Yxc minimize

21

T

5xx 21

16

• Absolute values:

solution:

(Integer) programming tricks (4)

ttt

ttt

yyyyyy

free y,0x

ybxa

y min

tt,j

ttt,jj

j

jt

0y,0y,0x

yybxa

yy min

ttt,j

tttt,jj

j

ttt

goal

variation

17

• Conjunctive/disjunctive programming - conjunctive set of constraints: must all be satisfied- disjunctive set of constraints: at least one must be satisfied

• example (Appendix A.4):

Integer programming tricks (5)

jkj

kjk

jjj

pxxor

pxx

xw min

}1,0{y

)y1(MpxxyMpxx

xw min

2jkj

1kjk

jjj

18

IP examplenonpreemptive single machine, total weighted completion time (App. A.3)

function) (objective xtw imizemin

j job of time completionxt

j job of time completionxt

n

1j

1-Cmax

0tjtj

1-Cmax

0tjt

jt

1 xif jt

otherwise 0 and t, timeat j job if ,1xjt completes

objective function: minimize weighted completion time:

model definition:

19

IP example (cont.)Restriction: all jobs must be completed once:

1x1-Cmax

0tjt

Restriction: only one job per time t:

t) timeper job oneexactly :on(restricti 1x

t) during process in is j job (if 1xn

1j

pt

tsjs

pt

tsjs

j

j

if job j is in process during t, it must be completed somewhere during [t,t+pj]

20

IP example (cont.)Complete IP-model:

1)-Cmax,0,t n,,1,j(for 1,0x

1)-Cmax,0,t(for 1x

n),1,j(for 1x

:tosubject

xtw inimizem

jt

n

1j

pt

tsjs

1-Cmax

0tjt

n

1j

1-Cmax

0tjtj

j

nCmaxinteger

variables

21

IP example (cont.)Additional restriction: precedence constraints

Model definition: SUCC(j) = successors of job j

job j must be completed before all jobs in SUCC(j):

n),1,j SUCC(j),k(for txtx

k job of timestart ptx

j job of time completiontx

1maxC

0tkt

1maxC

0tjt

k

1maxC

0tkt

1maxC

0tjt

22

Integer programmingsolution techniques

• Heuristic vs. explicit approach: – trade-off between solution quality and computation time– trade-off between implementation effort/costs and yield

(i.e. profits gained from solution quality improvement)

• Heuristic methods; for example:– local search (e.g. simulated annealing, tabu search, k-opt)– (composite) dispatching rules (e.g. EDD, SPT, MS)– adaptive search– rounding fractional solutions– beam search

23

• Explicit methods; 3 categories:1. dynamic programming2. cutting plane (polyhedral) methods3. branch and bound

or: hybrid methods (combination of the above)

• Commercial IP solvers usually use a combination of heuristics and 2, 3

Integer programmingsolution techniques (cont.)

24

Dynamic programming• Problem divided into stages• Each stage can have various states• A recursive objective function is used to

iterate through all states and all stages (forwards or backwards)

))}x,i(i(F)x,i(c{min)i(F tt1t1ttttxttt

T),0,(t xt ti

(constant) c)i(F 000

25

Cutting plane methodsSTEP 0: Create a relaxation of the problem

byomitting restrictions(e.g. the integrality restrictions)

STEP 1: Solve the current problemSTEP 2: If solution is infeasible then generate

a restriction that cuts of the solution,and add it to the problem STEP 1Otherwise: DONE

26

Branch and bound• Enumeration in a search tree

• each node is a partial solution, i.e. a part of the solution space

...

...

root node

child nodes

child nodes

Level 0

Level 1

Level 2

27

Branch and bound example 1

Disjunctive programming (appendix A.4):disjunctive set of constraints: at least one must be satisfied

xj = completion time of job jrestriction:

)Ik,j( pxxor pxxeither jkjkjk

solve LP without disjunctive restrictions(= LP relaxation)

if disjunct. restr. violated for j & k

Level 0

Level 1

...kjk pxx jkj pxx

28

Branch and bound (cont.)• Upper bound: e.g. a feasible solution• Lower bound:

e.g. a solution to an “easier” problem

• Node elimination (fathom/discard nodes):

when lower bound >= upper bound

29

Branch and bound (cont.)

• Branching strategy:how to partition solution space

• Node selection strategy:– sequence of exploring nodes:

• depth first (tries to obtain a solution fast)• breadth/best bound first (tries to find the best solution)

– which nodes to explore (filter and beam width)• filter width: #nodes selected for thorough evaluation• beam width: #nodes that are branched on ( filter

width) Beam search

30

Branch and bound example 2• Single machine, maximum lateness,

release and due dates

lower bound: EDD + preemption

Jobs 1 2 3 4p(j) 4 2 6 5r(j) 0 1 3 5d(j) 8 12 11 10

(?,?,?,?)

(1,?,?,?) (4,?,?,?)(2,?,?,?)

Level 0

Level 1(3,?,?,?)

31

Branch and bound example 2• Lower bound for: (1,?,?,?)

Lower bound: Lmax = max(0,17-12,15-11,0)=5

Jobs 1 2 3 4p(j) 4 2 6 5r(j) 0 1 3 5d(j) 8 12 11 10

t

r(2) r(3) r(4)

d(4)<d(3)

d(3)<d(2)

0 1 2 3 4 5 6 7 8 9 10 111213141516173 3 21 4

32

(?,?,?,?)

(1,?,?,?) (4,?,?,?)(2,?,?,?)

Level 0

Level 1(3,?,?,?)

Branch and bound example 2 (cont.)

LB=5 LB=7*=UB

(1,2,?,?) (1,3,?,?) Jobs 1 2 3 4p(j) 4 2 6 5r(j) 0 1 3 5d(j) 8 12 11 10

infeasible:

(1,3,4,3,2)LB=6*

=UB(1,2,4,3)

LB=5*=UB(1,3,4,2)

DONE

(1,2,4,3) (1,3,4,2)

33

Branch and bound example 3

1 2 3 4 5 6x1

1

2

3

4

56x2

)objective(6xx 21

0

4.2x,8.0x

2

1

0x1 1x1

3x,0x

2

1

2x,1x

2

1

2x2 3x2

2x,1x

2

1

3x,0x

2

1

obj: 3 obj: 3

obj: 3 obj: 3

LP solution:

34

Beam search example 1single-machine, total weighted

tardinessUpper bound: ATC rule (apparent tardiness cost):• schedule 1 job at a time• every time a machine comes available, determine ranking of jobs:

pK)0,tpdmax(

j

jj

jj

epw)t(I

MS rule

WSPTrule

look-ahead parameter:K = 4.5 + R(R 0.5)K = 6 - 2R (R 0.5)

= due date range factor

maxminmax C/ddR

35

Beam search example 1 (cont.)

single-machine, total weighted tardiness(?,?,?,?)

(1,?,?,?) (4,?,?,?)(2,?,?,?) (3,?,?,?)

1,2,3)(j pwep

w)t(Ij

jpK)0,tpdmax(

j

jj

jj

Upper bound by ATC rule: )3,2,1j( 0)0,tpdmax( jj

Jobs 1 2 3 4p(j) 10 10 13 4d(j) 4 2 1 12w(j) 14 12 1 12

w(j)/p(j) 1.4 1.2 0.1 3

36

Beam search example 1 (cont.)

single-machine, total weighted tardiness(?,?,?,?)

(1,?,?,?) (4,?,?,?)(2,?,?,?) (3,?,?,?)

Jobs 1 2 3 4p(j) 10 10 13 4d(j) 4 2 1 12w(j) 14 12 1 12

w(j)/p(j) 1.4 1.2 0.1 3

Jobs C(j) d(j) T(j) w(j)*T(j)1 10 4 6 842 24 2 22 2643 37 1 36 364 14 12 2 24

Upper boundby ATC rule:

Total = 408

37

Beam search example 1 (cont.)

single-machine, total weighted tardiness(?,?,?,?)

(1,?,?,?) (4,?,?,?)(2,?,?,?) (3,?,?,?)

UB=408UB=436UB=814UB=440

discardedexploredfurther

(beam width = 2)

4 nodesanalyzed(filter width=4)

38

Beam search example 1 (cont.)

(?,?,?,?)

(1,?,?,?) (4,?,?,?)(2,?,?,?) (3,?,?,?)

UB=408 436 814 440

(1,2,?,?) (1,3,?,?) (1,4,?,?)

UB=480 706 408

(1,4,2,3) (1,4,3,2)UB=408 554

best solution

(2,1,?,?) (2,3,?,?) (2,4,?,?)

436

(2,4,1,3) (2,4,3,1)436 608