mathematical programming problem: min/max subject to , , ,

Linear Programming 2012 1

Mathematical Programming Problem:min/max

subject to , ,,

If linear (affine) function linear programming problemIf (or part of them) nonlinear function nonlinear programming prob-lemIf solution set (or some of the variables) restricted to be integer points integer programming problem

Chapter 1 Introduction

Linear Programming 2012 2

Linear programming: problem of optimizing (maximize or minimize) a linear (objective) function subject to linear inequality (and equality) con-straints.

General form:{max, min} subject to

, (There may exist variables unrestricted in sign)

inner product of two column vectors :

If , , then are said to be orthogonal. In 3-D, the angle between the two vectors is 90 degrees.( vectors are column vectors unless specified otherwise)

Linear Programming 2012 3

Big difference from systems of linear equations is the existence of ob-jective function and linear inequalities (instead of equalities)

Much deeper theoretical results and applicability than systems of lin-ear equations.

: (decision) variables : right-hand-side { , , } : i th constraint { , } 0 : nonnegativity (nonpositivity) constraint : objective function

Other terminology:feasible solution, feasible set (region), free (unrestricted) variable, op-timal (feasible) solution, optimal cost, unbounded

Linear Programming 2012 4

Important submatrix multiplications Interpretation of constraints: see as submatrix multiplication.

A: matrix

1

1

' | |

' | |n

m

aA A A

a

denote constraints as { , , }

nj

mi iijj xeaxAAx 1 1 ' , where is i-th unit vector

mi

nj jjii eAyayAy 1 1 ''''

Linear Programming 2012 5

Any LP can be expressed as min

max min () and take negative of the optimal cost nonnegativity (nonpositivity) are special cases of inequalities which will be handled separately in the algorithms.

Feasible solution set of LP can always be expressed as (or ) (called polyhedron, a set which can be described as a solution set of finitely many linear inequalities)

We may sometimes use max , form (especially, when we study poly-hedron)

Linear Programming 2012 6

Standard form problems

Standard form : min , ,

Two view points:Find optimal weights (nonnegative) from possible nonnegative linear combi-

nations of columns of A to obtain b vectorFind optimal solution that satisfies linear equations and nonnegativity

Reduction to standard formFree (unrestricted) variable , , (slack variable) , (surplus variable)

nj

mi iijj xeaxAAx 1 1 '

Linear Programming 2012 7

Any (practical) algorithm can solve the LP problem in equality form only (except nonnegativity)

Modified form of the simplex method can solve the problem with free variables directly (without using difference of two variables).It gives more sensible interpretation of the behavior of the algorithm.

Linear Programming 2012 8

1.2 Formulation examples See other examples in the text. Minimum cost network flow problem

Directed network , ( )arc capacity , unit flow cost : net supply at node i ( > 0: supply node, < 0: demand node), (We may assume = 0)Find minimum cost transportation plan that satisfies supply, demand at each node and arc capacities.

minimize subject to i = 1, …, n

(out flow - in flow = net flow at node i)(some people use, in flow – out flow = net flow) , ,

Linear Programming 2012 9

Choosing paths in a communication network ( (fractional) multi-commodity flow problem)

Multicommodity flow problem: Several commodities share the network. For each commodity, it is min cost network flow problem. But the commodities must share the capacities of the arcs. Gen-eralization of min cost network flow problem. Many applications in communication, distribution / transportation systems

Several commodities caseActually one commodity. But there are multiple origin and destination pairs

of nodes (telecom, logistics, ..). Each origin-destination pair represent a commodity.

Given telecommunication network (directed) with arc set A, arc capacity bits/sec, , unit flow cost /bit , , demand bits/sec for traffic from node k to node l.Data can be sent using more than one path.Find paths to direct demands with min cost.

Linear Programming 2012 10

Decision variables: : amount of data with origin k and destination l that traverses link

Let = if if 0 otherwise

Formulation (flow based formulation)minimize subject to ,

(out flow - in flow = net flow at node i for commodity from node k to node l) (The sum of all commodities should not exceed the capacity of link (i, j) )

Linear Programming 2012 11

Alternative formulation (path based formulation)Let K: set of origin-destination pairs (commodities)

: demand of commodity P(k): set of all possible paths for sending commodity kKP(k;e): set of paths in P(k) that traverses arc eAE(p): set of links contained in path p

Decision variables: : fraction of commodity k sent on path p

minimize subject to for all

, for all for all ,

where If , it is a single path routing problem (path selection problem, integer

multicommodity flow problem).

Linear Programming 2012 12

path based formulation has smaller number of constraints, but enor-mous number of variables.can be solved easily by column generation technique (later). Integer version is more difficult to solve.

Extensions: Network design - also determine the number and type of facilities to be installed on the links (and/or nodes) together with routing of traffic.

Variations: Integer flow. Bifurcation of traffic may not be allowed. De-termine capacities and routing considering rerouting of traffic in case of network failure, Robust network design (data uncertainty), ...

Linear Programming 2012 13

Pattern classification (Linear classifier)Given m objects with feature vector .Objects belong to one of two classes. We know the class to which each sample object belongs.We want to design a criterion to determine the class of a new object using the feature vector.

Want to find a vector with such that, if , then , and if , then . (if it is possible)

Linear Programming 2012 14

Find a feasible solution that satisfies

, , for all sample objects i

Is this a linear programming problem? ( no objective function, strict inequality in constraints)

Linear Programming 2012 15

Is strict inequality allowed in LP?consider min x, x > 0 no minimum point. only infimum of objec-tive value exists

If the system has a feasible solution , we can make the difference of the values in the right hand side and in the left hand side large by us-ing solution for M > 0 and large. Hence there exists a solution that makes the difference at least 1 if the system has a solution. Remedy: Use ,

,

Important problem in data mining with applications in target marketing, bankruptcy prediction, medical diagnosis, process monitoring, …

VariationsWhat if there are many choices of hyperplanes? any reasonable criteria?What if there is no hyperplane separating the two classes?Do we have to use only one hyperplane?Use of nonlinear function possible? How to solve them?

• SVM (support vector machine), convex optimizationMore than two classes?

Linear Programming 2012 16

Linear Programming 2012 17

1.3 Piecewise linear convex objective functions

Some problems involving nonlinear functions can be modeled as LP.

Def: Function is called a convex function if for all and all [0, 1].

( the domain may be restricted)f called concave if is convex

(picture: the line segment joining and in is not below the locus of )

Linear Programming 2012 18

Def: , 1, 2 0, 1+ 2 = 1Then 1x + 2y is said to be a convex combination of x, y. Generally, , where and is a convex combination of the points .

Def: A set is convex if for any , we have for any , .Picture: ,

, (line segment joining lies in )

x (1 = 1)

y (1 = 0)

(𝑥− 𝑦 )

(𝑥− 𝑦 )

If we have , (without ), it is called an affine combination of x and y. Picture: ,

, (1 is arbitrary) (line passing through points )

Linear Programming 2012 19

Linear Programming 2012 20

Picture of convex function

x y(1 )x y

( (1 ) )f x y

( ) (1 ) ( ) f x f y

)(xf

nx R

1( , ( )) nx f x R ( , ( ))y f y

))()1()(,)1(( yfxfyx

Linear Programming 2012 21

relation between convex function and convex set

Def: . Define epigraph of as epi = . Then previous definition of convex function is equivalent to epi being a

convex set. When dealing with convex functions, we frequently con-sider epi to exploit the properties of convex sets.

Consider operations on functions that preserve convexity and opera-tions on sets that preserve convexity.

Linear Programming 2012 22

Example:Consider , (maximum of affine functions, called a piecewise linear convex function.)

nx R

)(xf

𝑐1′ 𝑥+𝑑1

𝑐2′ 𝑥+𝑑2

𝑐3′ 𝑥+𝑑3

𝑥

Linear Programming 2012 23

Thm: Let be convex functions. Then is also convex.

pf) =

Linear Programming 2012 24

Min of piecewise linear convex functions

Minimize Subject to

Minimize Subject to ,

Linear Programming 2012 25

Q: What can we do about finding maximum of a piecewise linear convex function?

maximum of a piecewise linear concave function (can be obtained as minimum of affine functions)?

Minimum of a piecewise linear concave function?

Linear Programming 2012 26

Convex function has a nice property such that a local minimum point is a global minimum point. (when domain is or convex set) (HW later)

Hence finding the minimum of a convex function defined over a con-vex set is usually easy. But finding the maximum of a convex function is difficult to solve. Basically, we need to examine all local maximum points.

Similarly, finding the maximum of a concave function is easy, but find-ing the minimum of a concave function is difficult.

Linear Programming 2012 27

Suppose we have in constraints, where is a piecewise linear convex function

,

Q: What about constraints ? Can it be modeled as LP?

Def: , is a convex function, The set is called the level set of .

level set of a convex function is a convex set. (HW later)solution set of LP is convex (easy) non-convex solution set can’t be modeled as LP.

Linear Programming 2012 28

Problems involving absolute values Minimize

subject to (assume )

More direct formulations than piecewise linear convex function is possi-ble.

(1)Min subject to , ,

(2)Min subject to

(want if , if and , i.e., at most one of is positive in an optimal solution. guarantees that.)

Linear Programming 2012 29

Data Fitting

Regression analysis using absolute value functionGiven m data points , .Want to find that predicts results given with function .Want that minimizes prediction error for all .

minimize subject to ,

,

Linear Programming 2012 30

Alternative criterionminimize

minimize subject to ,

,

Quadratic error function can't be modeled as LP, but need calculus method (closed form solution)

Linear Programming 2012 31

Special case of piecewise linear objective function : separable piecewise linear objective function.function , is called separable if

𝑥𝑖

𝑓 𝑖(𝑥 𝑖)

𝑎3𝑎2𝑎10

𝑐3

𝑐2𝑐1

𝑐4

slope:

𝑥1 𝑖 𝑥2𝑖 𝑥3 𝑖 𝑥4 𝑖

𝑐1<𝑐2<𝑐3<𝑐4

Approximation ofnonlinear function.

Linear Programming 2012 32

Express variable in the constraints as , where, , ,

In the objective function, use : min

Since we solve min problem, it is guaranteed that we get in an optimal solution implies have values at their upper bounds.

Linear Programming 2012 33

1.4 Graphical representation and solution

Let .Geometric intuition for the solution sets of

Linear Programming 2012 34

{𝑥 :𝑎′ 𝑥≥0 }

{𝑥 :𝑎’𝑥 0 }

Geometry in 2-D

𝑎

{𝑥 :𝑎’ 𝑥=0 }

0

Linear Programming 2012 35

Let be a (any) point satisfying . Then

Hence , where is any solution to , or .Similarly, for , .

{𝑥 :𝑎′ 𝑥≥𝑏}

{𝑥 :𝑎′ 𝑥≤𝑏}

𝑎

{𝑥 :𝑎′ 𝑥=0 }

0 {𝑥 :𝑎′ 𝑥=𝑏}

𝑧

Linear Programming 2012 36

min s.t. , ,

𝑐=(−1 ,−1)

𝑐=(1 ,1)

𝑐=(1 ,0)

𝑐=(0 ,1)

{𝑥 :𝑥 1+𝑥2=0 } {𝑥 :𝑥 1+𝑥2=𝑧 }𝑥1

𝑥2

Linear Programming 2012 37

Representing complex solution set in 2-D( variables, equations (coefficient vectors are linearly indepen-dent), nonnegativity, and )

𝑥1

𝑥3

𝑥2 𝑥1=0

𝑥2=0 𝑥3=0

See text sec. 1.5, 1.6 for more backgrounds