By Liyun ZhangAug.9th,2012

*Optimization for Nonlinear

Programming

*Topic*Method for Single Variable Unconstraint Optimization :

1. Quadratic Interpolation method 2. Golden Section method

*Method for Multivariable Constraint Optimization Method

1. Lagrange Multiplier Method 2. Exterior-point Penalty Method

*Quadratic Interpolation

*Solve for: unconstraint optimization problems with single variable and unimodal function

*Advantage and Disadvantage: converges fast but UNRILIABLE (you will see it in a later example)

*`

target function: f(x)select three points x0, x1, x2 on f(x) such that:x0<x1<x2,f2=min(f0, f1, f2)

there must exist a quadratic function: q(x)=a0+a1x+a2x^2 which connects these points such that:

q(x0)=a0+a1x0+a2x0^2=f(x0)q(x1)=a0+a1x1+a2x1^2=f(x1) q(x2)=a0+a1x2+a2x2^2=f(x2)

*Mathematical Background:

*Mathematical Background(cont.):

suppose (x3,f3) is the minimum point of q(x), then q’(x)=0, or x3= -we can solve that a1=

a2= - so x3=

( x 1^2− x2^2) f 0+( x 2^2− x0^2) f 1+( x 0^2− x1^2) f 2( x 0−x 1)( x1− x2)( x2−x 0)

Initialization:Determine x0, x1 and x2 which is known to contain the minimum of the function f(x).Step 1If x3-x1<ε (a sufficiently small number), then the minimum occurs at xo=x3 and stop iteration, else go to Step 2.Step 2 Evaluate x3 and x1If x3<x1, go to Step 3, else go to Step 4. Step 3Evaluate f(x3) and f(x1)If f(x3)<=f(x1), then determine new x0, x1, x2 as shown following, and go to Step5 x0=x0 x2=x1 x1=x3If f(x3)>f(x1), then determine the new x1, x2, x3 as shown following, and go to Step5 x0=x3 x1=x1 x2=x2

*Algorithm:

*Algorithm(cont.):Step 4Evaluate f(x3) and f(x1)If f(x3)<=f(x1), then determine new x0, x1, x2 as shown following, and go to Step5 x0=x1 x1=x3 x2=x2If f(x3)>f(x1), then determine the new x1, x2, x3 as shown following, and go to Step5 x0=x0 x1=x1 x2=x3Step 5Finish the shrinking of the old interval, start a new round of iteration, and compute the new x3. code

illustration

*Application and Comparison:Example 1.

Find the minimum point and value of the function f(x)=(x^2-2)^2/2-1Solution:f1001=inline('(x.*x-2).^2/2-1','x');a=0;b=5;TolX=1e-5;TolFun=1e-8;MaxIter=100;[xoq,foq]=Opt_Quadratic(f1001,[a,b],TolX,TolFun,MaxIter)xoq = 1.414187140007763foq = -0.999999997207487

Fminbnd method attempts to solve the optimum of one variable function within a fixed interval. The most common syntaxes are:[x, f]=fminbnd (fun, x1, x2)[x, f]=fminbnd (fun, x1, x2, options)In Ex. 1, we havex1=0x2=5and the result is:x = 1.414214286263162f = -0.999999999997904

*Application and Comparison(cont.):

*Golden Section Method*Solve for: unconstraint optimization problems with single variable and unimodal function

*Advantages: 1. more reliable than quadratic interpolation method 2. still converges fast since: a. rely on interval approximation instead of derivative calculations. b. the rate of the convergence is fixed to avoid the wider interval being used many times which may slow down the rate of convergence. c. interior points are selected to reduce the bounds as quickly as possible.

*Mathematical Background:

Select the intermediate points x3 and x4:c/a=a/bc/(b-c)=a/bsok=a/b=(1+ )/2x3=x1+1/(k+1)*(x2-x1)x4=x1+k/(k+1)*(x2-x1)

*Algorithm:Initialization:Determine x1 and x2 which is known to contain the minimum of the function f(x).Step 1Determine two intermediate points x3 and x4 such that x3=x1+(1-r)*h x4=x1+r*hwhere r=k/(k+1)= ( -1)/2 h=x2-x1Step 2If x4-x3<ε (a sufficiently small number), then the minimum occurs at xo=x3 (f(x3) < f(x4)) or xo=x4 (f(x3) > f(x4)) and stop iterating, else go to Step3.Step3 Evaluate f(x3) and f(x4)If f(x3)<f(x4), then determine new x1 and x2 as shown following, and go to Step4. x1=x1 x2=x4If f(x3)>f(x4), then determine the new x1 and x2 as shown following, and go to Step4. x1=x3 x2=x2Step 4Finish the shrinking of the old interval, start a new round of iteration, and compute the new x3, x4.

code

illustration

Example 2. Find the minimum point and value of the function x-(x^2-2)^3/2, x∈[0,4]Solution:f1002=inline('x-(x.*x-2).^3/2','x'); a=0; b=4; TolX=1e-4; TolFun=1e-4; MaxIter=100;   [xog,fog]=Opt_Golden(f1002,a,b,TolX,TolFun,MaxIter)xog = 3.999999954235401fog = -1.367999892407431e+003 [xoq,foq]=Opt_Quadratic(f1002,[a,b],TolX,TolFun,MaxIter)xoq = 1.938216986786059foq = -0.772297597277559

*Application and Comparison

*Application and Comparison(cont.):

* Apply primarily on: optimization problems with multivariable functions and equality constraints* Mathematical Backgrounds: Consider the function z=f(x,y) which is subject to ϕ(x,y)=0 or y=g(x), suppose x0 is the minimum point of z=f(x,g(x)), then x0 is the solution of the following equation system: f’x+ λ ϕ’x=0 f’y+ λ ϕ’y=0 ϕ(x,y)=0 More generally, we can introuduce a transformed function L(x1,x2…xm; λ1,λ2…λn)=f(x1,x2…xm)+ ∑λn ϕn(x1,x2…xn) and find all x satisfying the following equation system

f’xi+ ∑ λn ϕn’xi=0 (i=1,2…m)ϕk(x1,x2…xm) =0 (k=1,2…n)

substitute {x} back into f(x1,x2…xm) and then compare their value

*Lagrange Multiplier Method

*Algorithm and comparisonExample 3. Find the minimum point and value of the function f(x1,x2)=x1+x2, s.t x1^2+x2^2=2 Solution: code>> [xo,yo,fo]=Lag()xo = -1 1yo = -1 1fo = -2 2so minimum is -2 at (-1,-1)

*Exterior-point Penalty Method* Apply primarily on:

optimization problems with multivariable functions and equality or inequality constraints, and the initial guess point is out of the feasible domain.

* Advantage: work well when the initial guess point is out of the domain so that the computation is subjected to less restrictions compared with the interior- point penalty method.

Consider the function f(x) which is subject to a series of equality or inequality constraints Ci(x)>=0(i=1,2…m),to find the minimum point of f(x), we combine these conditions into a single function:F(x)=f(x)+c*∑g(Ci(x)). penalty coefficients: ci=c1*p^(i-1) where p>=2 penalty function: g(Ci(x))=min(0, Ci(x))^2Note that the optimum point of the unconstrained function will eventually converge to the original constrained one when c*∑g(Ci(x))<=ɛ.One way to search for the optimum point of the unconstrained function is the Newton method, which resorts to the derivative method.

*Mathematical Background

*Algorithm: InitializationSet up the initial optimum guess point x0, c1,p(p>=2), accuracy ɛ>0, and let k=1. Step 1Combine the given conditions into a new function F(x)=f(x)+ci*(∑hp(x)^2+∑gs(x)^2), hp(x)=0 (p=1,2…m) gs(x)>=0 (s=1,2…n) Step 2 Search for the minimum of F(x) from the point xk-1 with unconstraint optimization methods.Step 3Let S(xk)= ck* (∑hp(x)^2+∑gs(x)^2), if S(xk)<= ɛ, stop iteration, else let k=k+1 and go to step 2

code

* Application and ComparisonExample 4.

Find the minimum point and value of the function f(x1,x2)=x1^2-x1*x2+x2-x1+1, s.t 2x1+3x2-9=0x2^2+x1^2-6>=0x1>=0 x2>=0Solution:let c1=0.05, p=2, (x1,x2)=(2,2)

>> syms x1 x2;

>> f=x1^2-x1*x2+x2-x1+1;

>> g=[x1^2+x2^2-6;x1;x2]; h=[2*x1+3*x2-9];

>> [x,minf]=minEPF(f,[2 2],g,h,0.05,2,[x1 x2])

>> x= 1.4000

2.0668

minf= 0.7333

Fmincon attempts to solve constraint optimization problems of the form:min f(x) subject to: A*x<=b, Aeq*x=beq (linear constraints) x subject to: c(x)<=0, ceq(x)=0 (nonlinear constraints) lb<=x<=ub (bounds)The common syntax for fmincon is [x,fval]=fmincon(fun,x0,A,b,Aeq,beq,lb,ub,nonlcon)where x0 is the initial guess. In Ex. 4, we have>> A=[-1 0; 0 -1];>> b=[0; 0];>> Aeq=[2 3];>> beq=[9];and the result is x = 1.399999998247254 2.066666667835164fval= 0.733333333333334