nonlinear programming i li xiaolei. introductory concepts a general nonlinear programming problem...
TRANSCRIPT
Nonlinear ProgrammingI
Li Xiaolei
Introductory concepts
A general nonlinear programming problem (NLP)
can be expressed as follows:
mnm
n
n
n
borxxxg
borxxxg
borxxxgts
xxxfzor
) ,,( ),,,(
(1) ) ,,( ),,,(
) ,,( ),,,( ..
),,,( min)( max
21
2212
1211
21
objective function
constraints
Introductory concepts
DEFINATION
the feasible region for NLP(1) is the set of points
(x1,x2,…,xn) that satisfy the m constraints in (1). A
point in the feasible region is a feasible point, and
a point that is not in the feasible region is an
infeasible point.
Introductory concepts
DEFINATION
For a maximization problem, any point x* in the
feasible region for which f(x*) ≥f(x) holds for all
points x in the feasible region is an optimal
solution to the NLP. For a minimization problem,
x is the optimal solution if f(x*) ≤ f(x) for all
feasible x.
Examples of NLPs EXAMPLE 1
It costs a company c dollars per unit to
manufacture a product. If the company
charges p dollars per unit for the product,
customers demand D(p) units. To maximize
profits, what price should the firm charge?
Solution The firm’s decision variable is p.
Since the firm’s profit is (p-c)D(p), the firm
wants to solve the following unconstrained
maximization problem: max (p-c)D(p).
Examples of NLPs
EXAMPLE 2
If K units of capital and L units of labor are used,
a company can produce KL units of a
manufactured good. Capital can be purchased at
$4/unit and labor can be purchased at $1/unit. A
total of $8 is available to purchase capital and
labor. How can the firm maximized the quantity of
the good that can be manufactured?
Solution
Let K=units of capital purchased and L=units of
labor purchased. Then K and L must satisfy
4K+L≤8,K≥0, and L ≥ 0. Thus, the firm wants to
solve the following constrained maximization
problem:
0,
84 ..
max
LK
LKts
KLz
Differences Between NLPs and LPs
The feasible region for any LP is a convex set. If
an LP has an optimal solution, there is an
extreme point of the feasible region that is
optimal.
Even if the feasible region for an NLP is a convex
set, the optimal solution need not be an extreme
point of the NLP’s feasible region.
Figure to EXAMPLE 2
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
1
2
3
4
5
6
7
8
9
k0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5
0
1
2
3
4
5
6
7
8
9
k
l
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
1
2
3
4
5
6
7
8
9
k
lk l=2 = 0
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
1
2
3
4
5
6
7
8
9
k
lk l=4 = 0
KL=1
KL=2
KL=4
4K+L=8
D
Thus, the optimal solution to the example is
z=4,k=1,l=4(point D).
Point D is not an extreme point of the NLP’s
feasible region.
In fact, the optimal solution for an NLP may not
be on the boundary of the feasible region.
Local Extremum
DEFINITION 1
For any NLP (maximization), a feasible point
x=(x1,x2,…,xn) is a local maximum if for sufficiently
small ε, any feasible point x’=(x1’,x2’,…,xn’) having
|xi-xi’|<ε (i=1,2,…,n) satisfies f(x)≥f(x’). Analogously,
for a minimization problem, a point x is a local
minimum if f(x)≤f(x’) holds for all feasible x’ that
are close to x.
For a NLP, local maximum maybe not an optimal solution.
z
x 0
A
B
C
z=f(x)
Convex and Concave Function
Let f(x1,x2,…,xn) be a function that is defined for all points (x1,x2,…,xn) in a convex set S.
DEFINITION 2 A function f(x1,x2,…,xn) is a convex function on a
convex set S if for any x’∈S and x’’ ∈S f(cx’+(1-c)x’’) ≤cf(x’)+(1-c)f(x’’) holds for 0≤c≤1.
A=(x’,f(x’)) D=(x”,f(x”))C=(cx’+(1-c)x”,cf(x’)+(1-c)f(x”))B=(cx’+(1-c)x”,f(cx’+(1-c)x”))From figure: f(cx’+(1-c)x”) ≤cf(x’)+(1-c)f(x”)
x
y
A
DC
x’ x”cx’+(1-c)x”
y=f(x)
B
DEFINITION 3
A function f(x1,x2,…,xn) is a concave function on a
convex set S if for any x’∈S and x’’ ∈S
f(cx’+(1-c)x’’) ≥cf(x’)+(1-c)f(x’’)
holds for 0≤c≤1.
From definition 2 and 3, we see that f(x1,x2,…,xn) is
a convex function if and only if - f(x1,x2,…,xn) is a
concave function, and conversely.
A=(x’,f(x’)) D=(x”,f(x”))C=(cx’+(1-c)x”,cf(x’)+(1-c)f(x”))B=(cx’+(1-c)x”,f(cx’+(1-c)x”))From figure: f(cx’+(1-c)x”) ≥cf(x’)+(1-c)f(x”)
x
y
A
DC
x’ x”cx’+(1-c)x”
y=f(x)
B
x
y
AC
x’ x”cx’+(1-c)x”
y=f(x)
B
f(x) is not a convex or a concave function.
A linear function of the form f(x)=ax+b is both a
convex and a concave function.
f(cx’+(1-c)x”)=a[cx’+(1-c)x”]+b
=c(ax’+b)+(1-c)(ax”+b)
=cf(x’)+(1-c)f(x”)
Theorem 1 Consider NLP(1) and assume it is a maximization
problem. Suppose the feasible region S for NLP (1) is a convex set. If f(x) is concave on S, then any local maximum for NLP(1) is an optimal solution to this NLP.
Theorem 1’ Consider NLP(1) and assume it is a minimization
problem. Suppose the feasible region S for NLP(1) is a convex set. If f(x) is convex on S, then any local minimum for NLP(1) is an optimal solution to this NLP.
Theorems 1 and 1’ demonstrate that if we are
maximizing a concave function (or minimizing a
convex function )over a convex feasible region S,
then any local maximum (or local minimum) will
solve NLP(1), we will repeatedly apply theorem 1
and 1’.
We now explain how to determine if a function
f(x) of a single variable is convex or concave.
Theorem 2
Suppose f’’(x) exists for all x in a convex set S.
Then f(x) is a convex function on S if and only if
f’’(x) ≥0 for all x in S.
Since f(x) is convex if and only if - f(x) is
concave, Theorem 2’ must also be true.
Theorem 2’
Suppose f’’(x) exists for all x in a convex set S.
Then f(x) is a concave function on S if and only if
f’’(x) ≤0 for all x in S.
Example 1. show that f(x) = is a convex function on
S = R1.
2. show that f(x) = is a concave function on
S =(0,∞).
Solution
1. f’’(x)=2≥0, so f(x) is convex on S.
2. f’’(x)=- /4≤0,so f(x) is a concave function
on S.
2x
x21
23x
How can we determine whether a function f(x1,x2,
…,xn) of n variables is convex or concave on a set
S R∈ n?
We assume that f(x1,x2,…,xn) has continuous
second-order partial derivatives.
We require three definitions.
Definition
The Hessian of f(x1,x2,…,xn) is the n×n matrix
whose ijth entry is
We let H(x1,x2,…,xn) denote the value of the
Hessian at (x1,x2,…,xn) . For example ,if
f(x1,x2) = , then
ji xx
f
2
2221
31 2 xxxx
22
26 1xH(x1,x2)=
Definition
An ith principal minor of an n×n matrix is the determinant of any i×i matrix obtained by deleting n–i rows and the corresponding n–i columns of the matrix.
Thus ,for the matrix ,the first principal minors are -2 and -4, and the second principal minor is -2(-4)-(-1)(-1)=7. for any matrix, the first principal minors are just the diagonal entries of the matrix.
41
12
Definition
The kth leading principal minor of an n×n matrix is
the determinant of the k×k matrix obtained by
deleting the last n-k rows and columns of the
matrix.
We let Hk(x1,x2,…,xn) be the kth leading principal
minor of the H matrix evaluated at the point (x1,x2,
…,xn) . Thus, if f(x1,x2) =
,then H1(x1,x2)=6, and H2(x1,x2)= 6x1(2)-2(2)=12x1-4
2221
31 2 xxxx
Theorem 3
Suppose has continuous
second-order partial derivatives for each
point = S. Then
is a convex function on S iff for each S,
all principal minors of H are nonnegative.
Example : Show that =
is a convex function on S= .
),...,,( 21 nxxxf
x ),...,,( 21 nxxx ),...,,( 21 nxxxf
x
),( 21 xxf 2221
21 2 xxxx
2R
Solution
The first principal minors are the diagonal entries
(both equal 2≥0). The second principal minor is
2(2)-2(2)=0 ≥0. Since for all point, all principal
minors of H are nonnegative, theorem 3 shows
that f(x1,x2) is a convex function on R2.
22
22),( 21 xxH
Theorem 3’
Suppose has continuous
second-order partial derivatives for each
point S. Then
is a concave function on S iff for each
and =1,2,…n, all nonzero principal
minors have the same sign as .
),...,,( 21 nxxxf
),...,,( 21 nxxxx ),...,,( 21 nxxxf
sxk
k)1(
Solving NLPs with one variable In this section, we explain how to solve the
NLP :
max (or min )
s.t. (4)
(If b=∞,the feasible region for NLP (4) is ,
and if a=-∞, the feasible region for (4) is .)
)(xf
bax ,
ax bx
Solving NLPs with one variable To find the optimal solution to (4), we find
all local maxima (or minima). A point that
is a local maximum or minimum for (4) is
called a local extremum. Then the optimal
solution to (4) is the local maximum
(minimum) having the largest (smallest)
value of . Of cause if a=-∞ or b=∞, (4)
may have no optimal solution .
)(xf
NLPs with no solution
y=f(x)
y
x b Max f(x)
s.t. x∈ (-∞ ,b ]
y=f(x)
y
x a
Min f(x)
s.t. x∈[a,∞)
Extremum candidates
Case 1 Points where a<x<b, and f’(x)=0
Case 2 Points where f’(x) does not exist.
Case 3 Endpoints a and b of the interval [a,b].
Case 1. points where and =0 Theorem 4: If =0, and <0, then is a
local maximum. If =0 and >0,then
is a local minimum.
Theorem 5:If =0, and
1 If the first non-vanishing (nonzero) derivative at
is an odd-order derivative( and so
on), then is not a local maximum or a local
minimum.
bxa )(' xf
)('' 0xf 0x)(' 0xf)(' 0xf )('' 0xf 0x
)(' 0xf
0x)()( 0
)5(0
3 xfxf ,)(0x
2 If the first non-vanishing derivative
at is positive and is an even-order
derivative, then is a local minimum.
3 If the first non-vanishing derivative at
is negative and is an even-order
derivative , then is a local maximum.
0x0x
0x
0x
Figure 14
a b
x1 x0 x2
y
x
y=f(x)
f ’(x0)>0
f(x1)<f(x0)
f(x2)>f(x0)
x0 not a local extremum
a b
x1 x0 x2
y
x
y=f(x)
f ’(x0)<0
f(x1)>f(x0)
f(x2)<f(x0)
x0 not a local extremum
Figure 14
a b
x0
y
x
y=f(x)
f ’(x0)=0
for x<x0,f ’(x)>0
for x>x0 f ’(x)<0
x0 is a local maximum
a b x0
y
x
y=f(x)
f ’(x0)=0
for x<x0,f ’(x)<0
for x>x0 f ’(x)>0
x0 is a local minimum
Figure 14
a b
x0
y
x
y=f(x)
f ’(x0)=0 but x0 is neither
a local maximum nor a
local minimum
Case 2. Point Where does not exist
If does not have a derivative at ,
may be a local maximum, a local minimum,
or neither (see Figure 15). In this case, we
determine whether is a local maximum
or a local minimum by checking values of
at points < and > near
The four possible cases that can occur that
are summarized in Table 7.
)(' xf
)(xf 0x 0x
0x
)(xf 1x 0x 2x 0x0x
Figure 15
x1 x2 x0
y
x
y=f(x)
x0 is not a local extremum
x1 x2 x0
y
x
y=f(x)
x0 is not a local extremum
(a) (b)
Figure 15
x1 x2 x0
y
x
y=f(x)
x0 is a local maximum
x1 x2 x0
y
x
y=f(x)
x0 is a local minimum
(c) (d)
Table 7 If does not exist
Relationship x0 Figure
Not local extremum 15a
Not localextremum 15b
local maximum 15c
local minimum 15d
)()();()( 2010 xfxfxfxf
)()();()( 2010 xfxfxfxf
)()();()( 2010 xfxfxfxf
)()();()( 2010 xfxfxfxf
)(' xf
Case 3.Endpoints a and b of [a,b]
figure 16
a b
y
x
y=f(x)
f ’(a)>0
a is a local minimum
a b
y
x
y=f(x)
f ’(a)<0
a is a local maximum
Case 3.Endpoints a and b of [a,b]
figure 16
a b
y
x
y=f(x)
f ’(b)>0
b is a local maximum
a b
y
x
y=f(x)
f ’(b)<0
b is a local minimum
Example 15
It costs a monopolist $5/unit to produce a
product. If he produces x units of the product,
each can be sold for 10-x dollars(0≤x≤10). To
maximize profit, how much should the
monopolist produce?
Solution
Let P(x) be the monopolist’s profit if he produces x
unites. Then
P(x)=x(10-x)-5x=5x-x2 (0≤x≤10)
Thus,the monopolist wants to solve the following
NLP:
max P(x)
s.t. 0≤x≤10
Classify all extremum candidates: Case 1 P’(x)=5-2x, so P’(2.5)=0. since P’’(x)=-2,
x=2.5 is a local maximum yielding a profit of P(2.5)=6.25.
Case 2 P’(x) exists for all points in [0,10],so there are no Case 2 candidates.
Case 3 a=0 has P’(0)=5>0,so a=0 is a local minimum; b=10 has P’(10)=-15<0, so b=10 is a local minimum.
Thus, x=2.5 is the only local maximum.
Example 16
Let
Find
63 )4(3)(
30 )1(2)(2
2
xforxxf
xforxxf
60 ..
)(max
xts
xf
Solution
Case 1 For 0≤x<3,f’(x)=-2(x-1) and f’’(x)=-2. For 3<x ≤6, f’(x)=2(x-4) and f’’(x)=2. Thus, f’(1)=f’(4)=0. Since f’’(1)<0, x=1 is a local maximum. Since f’’(4)>0, x=4 is a local minimum.
Case 2 We see that f(x) has no derivative at x=3(for x slightly less than 3, f’(x)is near -4, and for x slightly bigger than 3, f’(x)is near -2). Since f(2.9)=-1.61, f(3)=-2,and f(3.1)=-2.19, x=3 is not a local extremum.
Case 3 Since f’(0)=2>0, x=0 is a local minimum. Since f’(6)=4>0, x=6 is a maximum.
Thus, on [0,6], f(x) has a local maximum for x=1 and x=6. Since f(1)=2 and f(6)=1, we find that the optimal solution to the NLP occurs for x=1.