1 OF 21

3. OPTIMIZATION

3.1 BACKGROUND

Root location and optimization are related in the sense that both involve guessing and searching for a

point on a function. The fundamental difference between the two types of problems is that root location

involves searching for zeros of a function or functions; optimization involves searching for either the

minimum or maximum. The optimum is the point where the curve is flat. This corresponds to the x

value where the derivative f ' x is equal to zero. Additionally, the second derivative, f ' ' x,

indicates whether the optimum is a minimum or a maximum: if f ' ' x0, the point is a maximum; if f ' ' x0, the point is a minimum.

That is you can differentiate the the function and locate the root (that is, the zero) of the new function.

In fact, some optimization methods seek to find an optima by solving the root problem: f ' x =0. It

should be noted that such searches are often complicated because f ' x is not available analytically.

Thus, one must some times use finite-difference approximations to estimate the derivative.

Beyond viewing optimization as a roots problem, it should be noted that the task of locating optima is

aided by some extra mathematical structure that is not part of simple root finding. This tends to make

optimization a more tractable task, particularly for multidimensional cases.

3.1.1 OPTIMIZATION AND ENGINEERING PRACTICE

Engineers must continuously design devices and products that perform tasks in an efficient fashion. In

doing so, they are constrained by the limitations of the physical world. Further, they must keep costs

down. Thus, they are always confronting optimization problems that balance performance and

limitations. Some common instances are listed below.

• Design aircraft for minimum weight and maximum strength.

• Optimal trajectories of space vehicles.

2 OF 21

• Design civil engineering structures for minimum cost.

• Design water-resource projects to mitigate flood damage while yield maximum hydropower.

• Predict structural behaviour by minimizing potential energy.

• Material-cutting strategy for minimum cost.

• Design pump and heat transfer equipment for maximum efficiency.

• Maximize power output of electrical networks and machinery while minimizing heat generation.

• Optimal planning and scheduling.

• Statistical analysis and models with minimum error.

• Optimal pipeline networks.

• Inventory control.

• Maintenance planning to minimize cost.

• Minimize waiting and idling times.

• Design waste treatment systems to meet water-quality standards at least cost.

3.1.2 MATHEMATICAL BACKGROUND

An optimization or mathematical programming problem generally can be stated as:

Minimize f x

Subject to g i x ≤0 i=1,2,... ,m

hix =0 i=1,2,... , p

where x is an n-dimensional design vector, f x is the objective function, g i x ≤0 are inequality

constraints, hix =0 are equality constraints.

The process of finding a maximum versus finding a minimum is essentially identical because the same

value, x*, both minimizes f(x) and maximize -f(x). This equivalence is illustrated graphically for a one-

dimensional function below.

3 OF 21

3.2 ONE-DIMENSIONAL UNCONSTRAINED OPTIMIZATION

In the root location, several roots can occur for a single function. Similarly, both local and global optima

can occur in optimization. In almost all instances, we will be interested in finding the absolute highest

or lowest value of a function. Thus, we must take care that we do not mistake a local result for the

global optimum.

Distinguishing a global from a local extremum can be very difficult problem for the general case. There

are there useful ways to approach this problem. First, insight into the behaviour of low-one dimensional

functions can some times be obtained graphically. Second, finding optima based on widely varying and

perhaps randomly generated starting guesses, and then selecting the largest of these as global.

Finally, perturbing the starting point associated with a local optimum and seeing if the routine returns a

better point or always returns to the same point. Although all these approaches can be utility, the fact

is that in some problems (usually the large ones), there may be no practical way to ensure that you

have located a global optimum. However, although you should always be sensitive to the issue, it is

fortunate that there are numerous engineering problems where you can locate the global optimum in an

unambiguous fashion.

3.2.1 GOLDEN-SECTION SEARCH

In solving for the root of a single nonlinear equation, the goal was to find the value of variable x that

yields a zero of the function f x . Single-variable optimization has the goal of finding the value of x

that yields an extremum, either a maximum or minimum of f x .

The golden-section search is simple and similar to the bisection approach for locating roots. For

simplicity, we will focus on the problem of finding a maximum.

4 OF 21

L0=L1L2 or L1L0=L2L1

L1L1L2

=L2L1

1

1L2L1

=L2L1

Given R=L2L1

11R

=R

R2R−1=0

Solve for R

R=−11−4 1−1

2=5−12

ALGORITHM

Step 1: Select the two initial guesses, x L and xU, that bracket one local extremum of f x .

Step 2: Two interior points x1 and x 2 are chosen according to the golden ratio.

d=5−12

xU−x L

x1= xLd

x 2=xU−d

Step 3: Evaluate the function at these two interior points x1 and x 2.

If f x1 f x2, then x L=x2 ==> [ x2, xU ]

If f x2 f x1, then xU=x1 ==> [ x L , x1]

5 OF 21

As the iterations are repeated, the interval containing the extremum is reduced rapidly. In fact, each

round the interval is reduced by a factor of the golden ratio. This is not quite as good as the reduction

achieved with bisection, but this is a harder problem.

Ea=1−R∣xu−xLxopt ∣100

EXAMPLE 1 Use the golden-section search to find the maximum of

f x =2sin x−x2

10

within the interval x L=0 and xU=4.

-6.0 -4.0 -2.0 0.0 2.0 4.0 6.0 8.0 10.0

-8.00

-6.00

-4.00

-2.00

0.00

2.00

4.00

f(x)

Iter xL xU f(xL) f(xU) d x1 x2 f(x1) f(x2)0 0 4 0 -3.1136 2.4721 2.4721 1.5279 0.6300 1.76471 0 2.4721 0 0.6300 1.5279 1.5279 0.9443 1.7647 1.53102 0.9443 2.4721 1.5310 0.6300 0.9443 1.8885 1.5279 1.5432 1.76473 0.9443 1.8885 1.5310 1.5432 0.5836 1.5279 1.3050 1.7647 1.75954 1.3050 1.8885 1.7595 1.5432 0.3607 1.6656 1.5279 1.7136 1.76475 1.3050 1.6656 1.7595 1.7136 0.2229 1.5279 1.4427 1.7647 1.77556 1.3050 1.5279 1.7595 1.7647 0.1378 1.4427 1.3901 1.7755 1.77427 1.3901 1.5279 1.7742 1.7647 0.0851 1.4752 1.4427 1.7732 1.77558 1.3901 1.4752 1.7742 1.7732 0.0526 1.4427 1.4226 1.7755 1.77579 1.3901 1.4427 1.7742 1.7755 0.0325 1.4226 1.4102 1.7757 1.7754

10 1.4102 1.4427 1.7754 1.7755 0.0201 1.4303 1.4226 1.7757 1.7757

6 OF 21

-6.0 -4.0 -2.0 0.0 2.0 4.0 6.0 8.0 10.0

-8.00

-6.00

-4.00

-2.00

0.00

2.00

4.00

f(x)

3.2.2 PARABOLIC INTERPOLATION

Parabolic interpolation takes advantage of the fact that a second-order polynomial often provides a

good approximation to the shape of f x near an optimum.

Just as there is only one straight line connecting two points, there is only one quadratic polynomial or

parabola connecting three points. Thus, if we have three points that jointly bracket an optimum, we can

find a parabola to the points. Then we can differentiate it, set the result equal to zero, and solve for an

estimate of the optimal x. It can be shown through some algebraic manipulations that the result is

x3=f x0x1

2−x2

2 f x1 x2

2−x0

2 f x2x0

2−x1

2

2 f x0 x1− x22 f x1x2− x02 f x2x0−x1

where x0, x1, and x 2 are the initial guesses, and x3 is the value of x that corresponds to the maximum

value of the parabolic fit to the guesses.

7 OF 21

After generating the new point, x3, which is similar to the secant method, is to merely assign the new

points sequentially.

EXAMPLE 2 Use parabolic interpolation to approximate the maximum of

f x =2sin x−x2

10

with initial guess of x0=0, x1=1, and x 2=4.

Iter x0 fx0 x1 fx1 x2 fx2 x3 fx31 0 0 1 1.582942 4 -3.113605 1.505535 1.7690792 1 1.582942 1.505535 1.769079 4 -3.113605 1.490253 1.7714313 1 1.582942 1.490253 1.771431 1.505535 1.769079 1.425636 1.7757224 1 1.582942 1.425636 1.775722 1.490253 1.771431 1.426602 1.7757255 1.425636 1.775722 1.426602 1.775725 1.490253 1.771431 1.427548 1.7757266 1.426602 1.775725 1.427548 1.775726 1.490253 1.771431 1.427551 1.7757267 1.427548 1.775726 1.427551 1.775726 1.490253 1.771431 1.427552 1.7757268 1.427551 1.775726 1.427552 1.775726 1.490253 1.771431 1.427552 1.775726

0 1 2 3 4 5 6 7 8 9

1.380000

1.400000

1.420000

1.440000

1.460000

1.480000

1.500000

1.520000

x3

3.2.3 NEWTON'S METHOD

Recall that the Newton-Raphson method is an open method that finds the root x of a function such that f x =0. The method is summarized as:

x i1=x i−f x i

f ' x i

A similar open approach can be used to fine an optimum of f x by defining a new function,

g x = f ' x. Thus, because the same optimal value x * satisfies both f ' x*=g x*=0, we can use

the following method as a technique to find the minimum or maximum of f x .

x i1=x i−f ' x i

f ' ' xi

8 OF 21

EXAMPLE 3 Use Newton's method to find the maximum of

f x =2sin x−x2

10

with initial guess of x0=2.5.

f ' x =2cos x−x5

f ' ' x=−2sin x−15

Iter x fx f'(x) f''(x)0 2.5 1 -2.10229 -1.396941 0.995082 2 0.889853 -1.877612 1.469011 2 -0.09058 -2.189653 1.427642 2 -0.00020 -2.179544 1.427552 2 -0.00000 -2.179525 1.427552 2 0 -2.17952

0 1 2 3 4 5 6

0

0.5

1

1.5

2

2.5

3

x

Although Newton's method works well in some cases, it is impractical for cases where the derivatives

cannot be conveniently evaluated. For these cases, other approaches that do not involve derivative

evaluation are available. For example, a secant-like Newton's method can be developed by using finite

difference approximations for the derivative evaluation.

9 OF 21

3.3 MULTIDIMENSIONAL UNCONSTRAINED OPTIMIZATION

Techniques for multidimensional unconstrained optimization can be classified in a number of ways. The

approaches that do not require derivative evaluation are called non-gradient or direct methods. Those

that require derivatives are called gradient methods.

3.3.1 DIRECT METHODS

These methods vary from simple brute force approaches to more elegant techniques that attempt to

exploit the nature of the function.

3.3.1.1 RANDOM SEARCH

As the name implies, this method repeatedly evaluates the function at randomly selected values of the

independent variables. If a sufficient number of samples are conducted, the optimum will eventually be

located.

Random number generators typically generate values between 0 and 1. If we designate such a number

as r x, the following formula can be used to generate x values randomly within a range between x L and xU.

x=x LxU−x L r x

Similarly for y, a formula can be used to generate y values randomly within a range between y L and yU

as follows.y= y L yU− y L r y

10 OF 21

EXAMPLE 4 Use a random number generator to locate the maximum of

f x , y= y− x−2 x2−2 x y− y2

in the domain bounded by x = –2 to 2 and y = 1 to 3.

Iter rx xL xU x ry yL yU y f(x,y) Max f(x,y)0 0.457177 -2 2 -0.17129 0.724690 1 3 2.449380 -2.59836 -2.598361 0.580900 -2 2 0.323598 0.644624 1 3 2.289247 -4.96603 -2.598362 0.676533 -2 2 0.706134 0.103299 1 3 1.206599 -3.65670 -2.598363 0.067229 -2 2 -1.73109 0.012434 1 3 1.024868 -0.73945 -0.739454 0.073551 -2 2 -1.70579 0.330744 1 3 1.661488 0.455584 0.4555845 0.761418 -2 2 1.045673 0.576060 1 3 2.152120 -10.2129 0.455584

397 0.720023 -2 2 0.880092 0.791837 1 3 2.583673 -11.0687 1.249151398 0.054367 -2 2 -1.78253 0.519305 1 3 2.038610 0.578147 1.249151399 0.269250 -2 2 -0.92300 0.559340 1 3 2.118679 0.760100 1.249151400 0.274284 -2 2 -0.90287 0.330417 1 3 1.660833 1.174018 1.249151

0 50 100 150 200 250 300 350 400 450

-12

-10

-8

-6

-4

-2

0

2

f(x,y)

This simple brute force approach works even for discontinuous and non-differentiable functions.

Furthermore, it always finds the global optimum rather than a local optimum. It major shortcoming is

that as the number of independent variables grows, the implementation effort required can become

onerous. In addition, it is not efficient because it takes no account of the behaviour into account as

well as the results of previous trials to improve the speed of convergence.

11 OF 21

3.3.2 GRADIENT METHODS

As the name implies, gradient methods explicitly use derivative information to generate efficient

algorithm to locate optima.

The first derivative of a one-dimensional function provides a slope or tangent to the function being

differentiated. For the standpoint of optimization, this is useful information. For example, if the slope is

positive, it tells us that increasing the independent variable will lead to a higher value of the function

we are exploring. The first derivative, thus, may tell us when we have reached an optimal value since

this is the point that the derivative goes to zero. Furthermore, the sign of the second derivative can tell

us whether we have reached a minimum (positive second derivative) or a maximum (negative second

derivative).

To fully understand multidimensional searches, we must first understand how the first and second

derivatives are expressed in a multidimensional context.

THE GRADIENT If we are interested in gaining evaluation as quickly as possible, the gradient tells us

what direction to move locally and how much we will gain by taking it.

f x = {∂ f x∂ x1

∂ f x∂ x2⋮

∂ f x∂ xn

}THE HESSIAN For one-dimensional problems, both the first and second derivatives provide valuable

information for searching out optima. The first derivative provides a steepest trajectory of the function

and tells us that we have reached an optimum. Once at an optimum, the second derivative tells us

whether we are a maximum (negative f ' ' x) or a minimum (positive f ' ' x).

You might expect that if the partial second-order derivatives with respect to both x and y are both

negative, then you reached a maximum. In fact, whether a maximum or a minimum occurs involves not

only the partials with respect to x and y but also the second partial with respect to x and y. Assuming

that the partial derivatives are continuous at near the point being evaluated, the following quantity can

be computed:

∣H∣=∂2 f

∂2 x2

∂2 f∂ y2

– ∂2 f

∂ x ∂ y2

12 OF 21

Three cases can occur

• If ∣H∣0 and ∂2 f /∂ x20, then f x , y has a local minimum.

• If ∣H∣0 and ∂2 f /∂ x20, then f x , y has a local maximum.

• If ∣H∣0, then f x , y has a saddle point.

The quantity ∣H∣ is equal to the determinant of a matrix made up of the second derivatives.

H = [∂2 f

∂ x2∂2 f∂ x ∂ y

∂2 f∂ y ∂ x

∂2 f∂ y2

]where this matrix is formally referred to as the Hessian of f.

FINITE-DIFFERENCE APPROXIMATIONS It should be mentioned that, for cases where they are

difficulty or inconvenient to compute analytically, both the gradient and Hessian can be evaluated

numerically. That is, the independent variables can be perturbed slightly to generate the required

partial derivatives.

∂ f∂ x

=f x x , y − f x− x , y

2 x

∂ f∂ y

=f x , y y − f x , y− y

2 y

∂2 f

∂ x2=f x x , y−2f x , y f x− x , y

2 x2

∂2 f

∂ y2=f x , y y −2f x , y f x , y− y

2 y2

∂2 f∂ x∂ y

=f x x , y y− f x x , y− y − f x− x , y y f x− x , y− y

4 x y

where is some small fractional value.

13 OF 21

3.3.2.1 STEEPEST ASCENT/DESCENT METHOD

An obvious strategy for climbing a hill would be to determine the maximum slope that your stating

position and then start walking in that direction. However, unless you are lucky and started on a ride

that pointed directly to the summit, as soon as you moved, your path would diverge from the steepest

ascent direction.

Starting at x0 and y0, the coordinates of any point in the gradient direction can be expressed as

x=x0∂ f∂ x

h

y= y0∂ f∂ yh

where h is distance along the h axis.

Then we can convert a two-dimensional function of x and y into one-dimensional function in h. The

optimum h* is the value of the step that maximizes f h and hence f x , y in the gradient direction.

This problem is equivalent to finding the maximum of a function of a single variable h. This method is

called steepest ascent where an arbitrary step size h is used.

EXAMPLE 5 Use the steepest ascent to locate the maximum of

f x , y=2 x y2 x− x2−2 y2

using initial guesses, x=−1 and y=1.

f={2 y2−2 x2 x−4 y }

At x=−1 and y=1x=−16hy=1−6h

Thus

f h=−180 h272 h−7

Use optimum step size h*=0.2.

14 OF 21

Iter x y dfdx dfdy f(x,y) h*0 -1.0000 1.0000 6.0000 -6.0000 -7.0000 0.21 0.2000 -0.2000 1.2000 1.2000 0.2000 0.22 0.4400 0.0400 1.2000 0.7200 0.7184 0.23 0.6800 0.1840 1.0080 0.6240 1.0801 0.24 0.8816 0.3088 0.8544 0.5280 1.3397 0.25 1.0525 0.4144 0.7238 0.4474 1.5261 0.2

63 1.9999 1.0000 0.0000 0.0000 2.0000 0.264 1.9999 1.0000 0.0000 0.0000 2.0000 0.265 2.0000 1.0000 0.0000 0.0000 2.0000 0.266 2.0000 1.0000 0.0000 0.0000 2.0000 0.2

0 5 10 15 20 25 30 35 40 45

-8.0000

-6.0000

-4.0000

-2.0000

0.0000

2.0000

4.0000

f(x,y)

The second derivative can be used to evaluate the optimum as shown.

H = [−2 22 −4]

∣H∣=40

Therefore f 2,1=2 is a maximum.

15 OF 21

3.3.2.1 NEWTON'S METHOD

Newton's method for a single variable can be extended to multi-variable cases. Write a second-order

Taylor series for f x near x=x i,

f x= f x i f T x−x i12x−x i

T H i x−x i

At the minimum,∂ f x∂ x j

=0 for j=1,2,. .. , n

Thus f= f x iH ix−x i=0

If H is nonsingular,

x i1=x i−H i−1 f

which can be shown to converge quadratically near the optimum. This method again performs better

than the steepest ascent method. However, note that the method requires both the computation of

second derivatives and matrix inversion at each iteration. Thus, the method is not very useful in

practice for functions with large numbers of variables. Furthermore, Newton's method may not

converge if the starting point is not close to the optimum.

EXAMPLE 6 Use the Newton's method to locate the maximum of

f x , y=2 x y2 x− x2−2 y2

using initial guesses, x=−1 and y=1.

f={2 y2−2 x2 x−4 y }H = [−2 2

2 −4]Iter x,y f(x,y) G H inv(H) inv(H)*G0 -1.00 -7.00 6.00 -2 2 -1.00 -0.50 -3.00

1.00 -6.00 2 -4 -0.50 -0.50 0.001 2.00 2.00 0.00 -2 2 -1.00 -0.50 0.00

1.00 0.00 2 -4 -0.50 -0.50 0.002 2.00 2.00 0.00 -2 2 -1.00 -0.50 0.00

1.00 0.00 2 -4 -0.50 -0.50 0.003 2.00 2.00 0.00 -2 2 -1.00 -0.50 0.00

1.00 0.00 2 -4 -0.50 -0.50 0.00

16 OF 21

0 0.5 1 1.5 2 2.5 3 3.5

-8.00

-6.00

-4.00

-2.00

0.00

2.00

4.00

f(x,y)

3.4 CONSTRAINED OPTIMIZATION

Linear programming (LP) is an optimization approach that deals with meeting a desired objective

function such as maximizing profit or minimizing cost in the presence of constraints such as limited

resources. The term linear connotes that the mathematical functions representing both the objective

and constraints are linear. The term programming does not mean computer programming but rather

connotes scheduling or setting and agenda.

There are a number of approaches for handling nonlinear optimization problems in presence of

constraints. These can generally be divided into indirect and direct approaches. A typical indirect

approach uses so-called penalty functions. These involve placing additional expression to make the

objective function less optimal as the solution approaches a constraint. Thus, the solution will be

discouraged from violating constraints. Although such methods can be useful in some problems, they

can become arduous when the problem involves many constraints.

3.4.1 PENALTY FUNCTION METHOD

We will focus on understanding the necessary conditions for optimality for the constrained problem:

Minimize f x

Subject to g i x ≤0 for i=1,... ,m

h j x=0 for j=1,. .. , l

3.4.1.1 EXTERIOR PENALTY FUNCTIONS

For the general problem, the quadratic penalty function P x is developed as

P x=∑i=1

m

{max [0, g i x]}2∑

j=1

l

[h jx ]2.

Now let us define a composite function.

T x = f x r P x

17 OF 21

where r0 is called a penalty parameter. The parameter r controls the degree of penalty for violating the

constraint. The unconstrained optimization T(x), for increasing values of r, results in a sequence of points that

converges to the minimum of x* = 5 from the outside of the feasible region. Since convergence is from the

outside of the feasible region, we call these exterior point methods. The idea is that if x strays too far from the

feasible region, the penalty term r i P x becomes large when ri is large. As r i∞ the tendency will be to draw

the unconstrained minima towards the feasible region so as to reduce the value of the penalty term. That is, a

large penalty is attached to being infeasible.

PROCEDURE

1. For some r10, find an unconstrained local minimum of T x , r 1. Denote the solution by x r1.

2. Choose r2r1 and using x r1 as a starting guess, minimize T x , r 2. Denote the solution as x r 2.

3. Proceed in this fashion, minimizingT x , r i for a strictly monotonically increasing sequence r i.

To illustrate the concept, consider a simple problem involving only one variable, x.Minimize f x

Subject to g x ≤0

where f x is a monotonically decreasing function of x. We will now define a penalty function.

EXAMPLE 7

Minimize f x=100 / x

Subject to g x = x−5≤0

T x , r = 100 / xr {max [0, x−5]}2

Choose: r=10

T x , r = 100 / x10 {max [0, x−5]}2

= 100 / x10 {x−5}2

Choose: r=5

T x , r = 100 / x5 {max [0, x−5]}2

= 100 / x5 {x−5}2

Choose: r=2

T x , r = 100 / x2 {max [0,x−5]}2

= 100 / x2 {x−5}2

Choose: r=1

T x , r = 100 / x1{max [0, x−5]}2

= 100 / x1{x−5}2

The minimum of T for various values of r leads to point x r as given in the following table.

18 OF 21

r x f x* f* g*1 1 100 6.2713 15.9457 1.271310 6.2713 15.9457 5.1859 19.283 0.1859100 5.1859 19.283 5.0198 19.9209 0.01981000 5.0198 19.9209 5.0020 19.9920 0.0020

10000 5.0020 19.9920 5.0020 19.9992 0.0020100000 5.0020 19.9992 5.0000 19.9999 0.00001000000 5.0000 19.9999 5.0000 20.0000 0.0000

3.5 4 4.5 5 5.5 6 6.5

0

5

10

15

20

25

30

35

40

r=10r=5r=2r=1

3.4.1.2 INTERIOR PENALTY FUNCTIONS

In interior point of barrier methods, we approach the optimum from the interior of the feasible region. Only

inequality constraints are permitted in this class of methods. Thus, we consider the feasible region to be defined

by ={x : g i x ≤0, i=1,. .. , m}. Interior penalty functions, B x , are defined with properties: (I) B is

continuous, (ii) B x ≥0, (iii) B x ∞ as bold x approaches the boundary of . Two popular choices are:

Inverse Barrier Function: B x =−∑i=1

m1

g ix

Log Barrier Function: B x =−∑i=1

m

log [−g ix ]

We should define log −g i=0 when g i x −1 to ensure that B≥0. This is not a problem when constraints

are expressed in normalized from in which case g−1 implies a highly inactive constraint which will not play a

role in determining the optimum.

For instance, we should express the constraint x≤1000 as g=x /1000 –1≤0. The T-function is defined as:

T x , r = f x 1rB x

19 OF 21

Starting from a point x0∈ and r0, the minimum of T will in general provide us with a new point which is in

, since the value of T becomes infinite on the boundary. As r is increased, the weighting of the penalty term is

decreased. This allows further reduction in f x while maintaining feasibility.

EXAMPLE 8

Minimize f x =2x−62

Subject to g x = x−5≤0

Choose x0=3, r=1

T x , r = 2x−621r

−1x−5

= 2x−62−11

1x−5

1/r x f x* f* g*100 1 50 2.3742 26.2926 -2.625810 2.3742 26.2926 3.9071 8.7606 -1.09291 3.9071 8.7606 4.5803 4.0309 -0.4197

0.5 4.5803 4.0309 4.6910 3.4271 -0.30900.3 4.6910 3.4271 4.7546 3.1020 -0.24540.2 4.7546 3.1020 4.7962 2.8982 -0.20380.1 4.7962 2.8982 5.0000 2.0000 0.0000

3.5 4 4.5 5 5.5 6 6.5

-25

-20

-15

-10

-5

0

5

10

15

20

25

1/r = 1001/r = 101/r = 1

20 OF 21

3.5 GLOBAL OPTIMUM

3.5.1 PARAMETERIZATION TECHNIQUE

Minimize P x

Let V =P x , thus

V =J x x .

x =J + x V

Then x* can be obtained from x=∫ x d or numerical method such as the Runge-Kutta method.

The value V can be view as a slope of a searching path as shown.

V =Pd x −P x 0

f−0

where Pd x f is the desired value of the objective function P x and can be estimated byPd x f = P x 0

21 OF 21

Example 9 Use the Newton's method and the parameterization technique to find the minimum off x =x−1x2x−3x5

starting at x0=4.

f x =x 43 x3−15 x2−19 x30

f ' x =4 x39 x2−30 x−19

f ' ' x=12 x218 x−30

-8 -6 -4 -2 0 2 4 6

-100

-50

0

50

100

150

200

250

300

350

f(x)

NEWTON'S METHOD

Iter x f(x) f'(x) f''(x)0 4 162 261 2341 2.884615 -8.37494 65.36220 121.77512 2.347870 -28.0815 11.94685 78.411613 2.195510 -29.0349 0.848812 67.362334 2.182909 -29.0403 0.005604 66.473475 2.182825 -29.0403 0.000000 66.46753