11 od nonlinear programming a-2008-2

1NONLINEAR

PROGRAMMING

Nonlinear Programming

Linear programming has a fundamental role in OR.

In linear programming all its functions (objective function and constraint functions) are linear.

This assumption frequently does not hold, and nonlinear programming problems are formulated:

Find x = (x1, x2,..., xn) to

Maximize f (x)

subject to

gi(x) bi , for i = 1, 2, ..., m

and x 0

Joo Miguel da Costa Sousa / Alexandra Moutinho 351

Nonlinear Programming

There are many types of nonlinear programming

problems, depending on f(x) and gi(x) assumed differentiable or piecewise linear functions.

Different algorithms are used for different types.

Some problems can be solved very efficiently, whilst

others, even small, can be very difficult.

Nonlinear programming is a particularly large subject.

Only some important types will be dealt with here. Some applications are give in the following.


Application: product-mix problem

In product-mix problems (as Wyndor Glass Co.) the goal is to determine optimal mix of production levels.

Sometimes price elasticity is present: the amount of sold product has an inverse relation to price charged:


Price elasticity

p(x) is the price required to sell x units.

c is the unit cost for producing and distributing product.

Profit from producing and selling x is:

P(x) = xp(x) cx


Product-mix problem

If each product has a similar profit function, overall

objective function is

Other nonlinearity: marginal cost varies with

production level.

It may decrease when production level is increased due to the learning-curve effect.

It may increase due to overtime or more expensive production facilities when production increases.

=

= 1

( ) ( )n

j j

j

f x P x


2Application: transportation problem

Determine optimal plan for shipping goods from

various sources to various destinations (see P&T Company problem).

Cost per unit shipped may not be fixed. Volume discounts are sometimes available for large shipments.

Marginal cost can have a pattern like in the figure.

Cost of shipping x units is a piecewise linear function C(x), with slope equal to the marginal cost.


Volume discounts on shipping costs

Marginal cost Cost of shipping


Transportation problem

If each combination of source and destination has a

similar shipping cost function, so that

cost of shipping xij units from source i (i = 1, 2, ..., m) to destination j (j = 1, 2, ..., n) is given by a nonlinear

function Cij(xij),

the overall objective function is

= =

= 1 1

Minimize ( ) ( )m n

ij ij

i j

f C xx


Graphical illustration

Example: Wyndor Glass Co. problem with NL constraint


Graphical illustration

Example: Wyndor Glass Co. with NL objective function


Example: Wyndor Glass Co. (3)


3Global and local optimum

Example: f(x) with three local maxima (x = 0, 2, 4), and three local minima (x = 1, 3, 5). Global?


Guaranteed local maximum

Global maximum when:

Function always curving downward is a concave function (concave downward).

Function always curving upward is a convex function (concave upward).

2

2

( )0, for all

f xx

x


Guaranteed local optimum

Nonlinear programming with no constraints and

concave objective function, a local maximum is the global maximum.

Nonlinear programming with no constraints and convex objective function, a local minimum is the global minimum.

With constraints, these guarantees still hold if the feasible region is a convex set.

The feasible region for a NP problem is a convex set if

all gi(x) are convex functions.


Ex: Wyndor Glass with one concave gi(x)


Types of NP problems

Unconstrained Optimization: no constraints

necessary condition for a solution x* = x to be optimal:

when f (x) is a concave function this condition is sufficient.

when xj has a constraint xj 0, sufficient condition changes to:

Maximize ( )f x

= = =

*( ) 0 at , for 1,2, ,j

fj n

x

xx x

= =

= = >

* *

* *

0 at , if 0( )

0 at , if 0

j

jj

xf

xx

x xx

x x


Example: nonnegative constraint


4Types of NP problems

Linearly Constrained Optimization

All constraints are linear and objective function is nonlinear.

Special case: Quadratic Programming

Objective function is quadratic.

Many applications, e.g. portfolio selection, predictive control.

Convex Programming assumptions for maximization:

1. f (x) is a concave function.

2. Each gi(x) is a convex function.

For a minimization problem, f (x) must be a convex function.



Separable Programming is a special case of convex

programming with additional assumption

3. All f (x) and gi(x) are separable functions.

A separable function is a function where each term involves only a single variable (satisfies assumption of

additivity but not proportionality):

Nonconvex Programming: local optimum is not assured to be a global optimum.

=

= 1

( ) ( )n

j j

j

f f xx



Geometric Programming is applied to engineering

design as well as economics and statistics problems

Objective function and constraint functions are of the form:

ci and aij are typically physical constraints.

When all ci are strictly positive, functions are generalized positive polynomials (posynomials). If the objective function is to be minimized, a convex programming algorithm can be applied.

=

= = 1 21 21

( ) ( ), where ( ) i i inN

a a a

i i i n

i

g c P P x x xx x x



Fractional Programming

when f (x) has the linear fractional programming form:

problem can be transformed into a linear programming problem.

= 12

( )Maximize ( )

( )

ff

f

xx

x

+=

+0

0

( )c

fd

cxx

dx


One-variable unconstrained optimization

Methods for solving unconstrained optimization with

only one variable x (n = 1), where the differentiable

function f (x) is concave.

Necessary and sufficient condition for optimum:

= =

*( ) 0 at .

f xx x

x


Solving the optimization problem

If f (x) is not simple, problem cannot be solved analytically.

If not, search procedures can solve the problem

numerically.

We will describe two common search procedures:

Bisection method

Newtons method


5Bisection method

Since f (x) is concave, we know that:

Can hold if 2nd derivative 0 for some (not all) values of x.

If derivative of x is positive, x is a lower bound of x*.

If derivative of x is negative, x is an upper bound of x*.

>

*

*

*

( )0 if ,

( )0 if ,

( )0 if .

df xx x

dx

df xx x

dx

df xx x

dx


Bisection method

Notation:

In the bisection method, new trial solution is the

midpoint between the two current bounds.

*

*

*

= current trial solution,

= current lower bound on ,

= current upper bound on ,

= error tolerance for .

x

x x

x x

x


Algorithm of the Bisection Method

Initialization: Select . Find initial upper and lower bounds. Select initial trial as:

Iteration:

1. Evaluate

2. If

3. If

4. Select a new

Stopping rule: If stop. Otherwise, go to 1.

+ =2

x xx

=( )

at ',df x

x xdx

+ =

2

x xx

=( )

0, reset ,df x

x xdx

=( )

0, reset ,df x

x xdx

2x xJoo Miguel da Costa Sousa / Alexandra Moutinho 376

Example

Maximize = 4 6 ( ) 12 3 2f x x x x


Solution

First two derivatives: =

= +

3 5

22 4

2

( ) 12(1 )

( ) 12(3 5 )

df xx x

dx

d f xx x

dx

Iteration df (x)/dx x x New x f (x)

0 0 2 1 7.0000

1 12 0 1 0.5 5.7812

2 +10.12 0.5 1 0.75 7.6948

3 +4.09 0.75 1 0.875 7.8439

4 2.19 0.75 0.875 0.8125 7.8672

5 +1.31 0.8125 0.875 0.84375 7.8829

6 0.34 0.8125 0.84375 0.828125 7.8815

7 +0.51 0.828125 0.84375 0.8359375 7.8839


= 0.01

Solution

x* 0.836

0.828125 < x* < 0.84375

Bisection method converges relatively slowly.

Only information about first derivative is being used.

Additional information can be obtained by using

second derivative, as in Newtons method.


6Newtons method

This method approximates f (x) within neighborhood of current trial solution by a quadratic function

This quadratic approximation is Taylor series

truncated after second derivative term:

Maximized by setting f (xi+1) equal to zero (xi, f (xi), f (xi)and f (xi) are constants):

+ + +

+ + 21 1 1

( )( ) ( ) ( )( ) ( )

2

ii i i i i i i

f xf x f x f x x x x x

+ +

+

+ =

=

1 1

1

( ) ( ) ( )( ) 0

( )

( )

i i i i i

ii i

i

f x f x f x x x

f xx x

f x


Algorithm of Newtons Method

Initialization: Select . Find initial trial solution xi by inspection. Set i = 1.

Iteration i:

1. Calculate f (xi) and f (xi).

2. Set

Stopping rule: If |xi+1 - xi | , stop; xi+1 is optimal. Otherwise, i = i + 1 (another iteration).

+

=

1( )

.( )

ii i

i

f xx x

f x


Example

Maximize again

New solution is given by:

Selecting x1 = 1, and = 0.00001:

= 4 6 ( ) 12 3 2f x x x x

+

= = = +

+ +

3 5 3 5

1 2 4 2 4

( ) 12(1 ) 1

( ) 12(3 5 ) 3 5

i i i i ii i i i

i i i i i

f x x x x xx x x x

f x x x x x

Iteration i xi f (xi) f (xi) f (xi) xi+1

1 1 7 12 96 0.875

2 0.875 7.8439 2.1940 62.733 0.84003

3 0.84003 7.8838 0.1325 55.279 0.83763

4 0.83763 7.8839 0.0006 54.790 0.83762


Multivariable unconstrained optimization

Problem: maximizing a concave function f (x) of multiple variables x = (x1, x2,..., xn) with no constraints.

Necessary and sufficient condition for optimality:

partial derivatives equal to zero.

No analytical solution numerical search procedure must be used.

One of these is the gradient search procedure:

It identifies and uses the direction of movement from the current trial solution that maximizes the rate at which f (x)is increased.


Gradient search procedure

Use values of partial derivatives to select the specific

direction to move, using the gradient.

Gradient of a point x = x is the vector with partial derivatives evaluated at x = x:

Moves in the direction of this gradient until f (x) stops increasing. Each iteration changes the trial solution

x:

= =

1 2

( ) , , , at n

f f ff

x x xx x x

= + *Reset ( )t fx x x


Gradient search procedure

where t* is value of t 0 that maximizes f (x + t f(x)):

The function f (x + t f(x)) is simply f (x) where:

Iterations continue until f (x) = 0 within tolerance:=

= + = , for 1,2, ,j j

j

fx x t j n

xx x

+ = + *

0 ( ( )) max ( ( ))

tf t f f t fx x x x

=

, for 1,2, , .j

fj n

x


7Summary of gradient search procedure

Initialization: Select and any initial trial solution x. Go to stopping rule.

Iteration:

1. Express f (x + tf (x)) as a function of t by setting

and substitute these expressions into f (x).

=

= + = , for 1,2, ,j j

j

fx x t j n

xx x


Summary of gradient search procedure

Iteration (concl.):

2. Use search procedure to find t = t* that maximizesf (x + t f(x)) over t 0.

3. Reset x = x + t* f (x). Go to stopping rule.

Stopping rule: Evaluate f(x) at x = x. Check if:

If so, stop with current x as the approximation of x*. Otherwise, perform another iteration.

=

, for 1,2, , .j

fj n

x


Example

Maximize

Thus,

Verify that f (x) is concave (see Appendix 2 of Hilliers book).

Suppose that x = (0, 0) is initial trial solution. Thus,

= + 2 21 2 2 1 2( ) 2 2 2 .f x x x x xx

=

= +

2 1

1

1 2

2

2 2

2 2 4

fx x

x

fx x

x

=(0,0) (0,2)f


Example (2)

Iteration 1 sets

By substituting these expressions into f (x):

Because

= + =

= + =1

2

0 (0) 0

0 (2) 2

x t

x t t

+ =

= +

=

2 2

2

( ( )) (0,2 )

2(0)(2 ) 2(2 ) 0 2(2 )

4 8

f t f f t

t t t

t t

x x

= = * 2

0 0 (0,2 ) max (0,2 ) max {4 8 }

t tf t f t t t


Example (3)

and

it follows that

so

This completes first iteration. For new trial, gradient is:

( ) = =24 8 4 16 0d t t tdt

=*1

4t

= + =

1 1Reset (0,0) (0,2) 0,

4 2x

=

10, (1,0)

2f


Example (4)

As < 1, Iteration 2:

so

= + =

1 10, (1,0) ,

2 2t tx

+ = + + =

= +

= +

2

2

2

1 1 ( ( )) 0 , 0 ,

2 2

1 1 1(2 ) 2 2

2 2 2

1

2

f t f f t t f t

t t

t t

x x

= = +

2

0 0

1 1 1 *, max , max

2 2 2t tf t f t t t


8Example (5)

Because

and

then

so

This completes second iteration. See figure.

= = +

* 2

0 0

1 1 1 , max , max

2 2 2t tf t f t t t

+ = =

2 1 1 2 0

2

dt t t

dt

=*1

2t

= + =

1 1 1 1Reset 0, (1,0) ,

2 2 2 2x


Illustration of example

Optimal solution is (1, 1), as f (1, 1) = (0, 0)


Newtons method

It is a quadratic approximation of objective function

f (x).

When objective function is concave and x and its gradient f (x) are written as column vectors,

The solution x that maximizes the approximating quadratic function is:

where 2f (x) is the n n Hessian matrix.

= 2 1[ ( )] ( ),f fx x x x


Newtons method

The inverse of the Hessian matrix is commonly

approximated in various ways.

Approximations of Newtons methods are referred to

as quasi-Newton methods (or variable metric methods).

Recall that this topic was mentioned in Intelligent Systems, e.g. in neural network leaning.


11 od nonlinear programming a-2008-2

Documents