(iii) lagrange multipliers and kuhn-tucker conditions d nagesh kumar, iisc introduction to...

(iii) Lagrange Multipliers and Kuhn-tucker Conditions

D Nagesh Kumar, IISc

Introduction to Optimization

Water Resources Systems Planning and Management: M2L3

Objectives

D Nagesh Kumar, IISc2

To study the optimization with multiple decision variables

and equality constraint : Lagrange Multipliers.

To study the optimization with multiple decision variables

and inequality constraint : Kuhn-Tucker (KT) conditions


Constrained optimization with equality constraints


A function of multiple variables, f(x), is to be optimized subject to one or more

equality constraints of many variables. These equality constraints, gj(x), may or may

not be linear. The problem statement is as follows:

Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m

where

(1)

1

2

n

x

x

x

X


Constrained optimization…


With the condition that ; or else if m > n then the problem

becomes an over defined one and there will be no solution. Of the

many available methods, the method of constrained variation and the

method of using Lagrange multipliers are discussed.

m n


Solution by method of Lagrange multipliers


Continuing with the same specific case of the optimization problem with

n = 2 and m = 1 we define a quantity λ, called the Lagrange multiplier as

(2)

Using this in the constrained variation of f [ given in the previous lecture]

And (2) written as (3)

(4)

* *1 2

2

2 (x , x )

/

/

f x

g x

* *1 2

1 1 (x , x )

0f g

x x

* *1 2

2 2 (x , x )

0f g

x x

1 2

11

1 2 2 (x *, x *)

/0

/

g xf fdf dx

x g x x


Solution by method of Lagrange multipliers…


Also, the constraint equation has to be satisfied at the extreme point

(5)

Hence equations (2) to (5) represent the necessary conditions for the point

[x1*, x2*] to be an extreme point.

λ could be expressed in terms of as well has to be non-

zero.

These necessary conditions require that at least one of the partial

derivatives of g(x1 , x2) be non-zero at an extreme point.

* *1 2

1 2 ( , )( , ) 0

x xg x x

1/g x 1/g x



The conditions given by equations (2) to (5) can also be generated by

constructing a functions L, known as the Lagrangian function, as

(6)

Alternatively, treating L as a function of x1,x2 and , the necessary

conditions for its extremum are given by

(7)

1 2 1 2 1 2( , , ) ( , ) ( , )L x x f x x g x x

1 2 1 2 1 21 1 1

1 2 1 2 1 22 2 2

1 2 1 2

( , , ) ( , ) ( , ) 0

( , , ) ( , ) ( , ) 0

( , , ) ( , ) 0

L f gx x x x x x

x x x

L f gx x x x x x

x x x

Lx x g x x

Solution by method of Lagrange multipliers…


Necessary conditions for a general problem


For a general problem with n variables and m equality constraints the

problem is defined as shown earlier

Maximize (or minimize) f(X), subject to gj(X) = 0, j = 1, 2, … , m

where

In this case the Lagrange function, L, will have one Lagrange multiplier j

for each constraint as

(8)1 2 , 1 2 1 1 2 2( , ,..., , ,..., ) ( ) ( ) ( ) ... ( )n m m mL x x x f g g g X X X X

1

2

n

x

x

x

X



L is now a function of n + m unknowns, , and the

necessary conditions for the problem defined above are given by

(9)

which represent n + m equations in terms of the n + m unknowns, xi and j.

The solution to this set of equations gives us

and (10)

1 2 , 1 2, ,..., , ,...,n mx x x

1

( ) ( ) 0, 1, 2,..., ; 1, 2,...,

( ) 0, 1, 2,...,

mj

jji i i

jj

gL fi n j m

x x x

Lg j m

X X

X

*1

*2

*n

x

x

x

X

*1

** 2

*m

Necessary conditions for a general problem…


Sufficient conditions for a general problem


A sufficient condition for f(X) to have a relative minimum at X* is that each

root of the polynomial in Є, defined by the following determinant equation

be positive.

(11)

11 12 1 11 21 1

21 22 2 12 22 2

1 2 1 2

11 12 1

21 22 2

1 2

0

0 0

0 0

n m

n m

n n nn n n mn

n

n

m m mn

L L L g g g

L L L g g g

L L L g g g

g g g

g g g

g g g



where

(12)

Similarly, a sufficient condition for f(X) to have a relative maximum at X*

is that each root of the polynomial in Є, defined by equation (11) be

negative.

If equation (11), on solving yields roots, some of which are positive and

others negative, then the point X* is neither a maximum nor a minimum.

2* *

*

( , ), for 1,2,..., 1,2,...,

( ), where 1,2,..., and 1,2,...,

iji j

ppq

q

LL i n and j m

x x

gg p m q n

x

X

X

Sufficient conditions for a general problem…


Example


Minimize ,

Subject to

Solution

with n = 2 and m = 1

L =

or

2 21 1 2 2 1 2( ) 3 6 5 7 5f x x x x x x X

1 2 5x x

1 1 2( ) 5 0g x x X

1 2 , 1 2 1 1 2 2( , ,..., , ,..., ) ( ) ( ) ( ) ... ( )n m m mx x x f g g g L X X X X

2 21 1 2 2 1 2 1 1 23 6 5 7 5 ( 5)x x x x x x x x

1 2 11

1 2 1

1

6 6 7 0

1(7 )

61

5 (7 )6

x xx

x x

L

1 23


Example…


1 2 12

1 2 1

1 2 2 1

6 10 5 0

13 5 (5 )

21

3( ) 2 (5 )2

x xx

x x

x x x

L

2

1

2x

1

11

2x

11 1* , ; * 23

2 2

X λ

11 12 11

21 22 21

11 12

0

0

L L g

L L g

g g

Hence,


Example…


2

11 21 ( )

6Lx

X*,λ*

L 2

12 211 2 ( )

6L Lx x

X*,λ*

L 2

22 22 ( )

10Lx

X*,λ*

L

111

1 ( )

112 21

2 ( )

1

1

gg

x

gg g

x

X*,λ*

X*,λ*

6 6 1

6 10 1 0

1 1 0

( 6 )[ 1] ( 6)[ 1] 1[ 6 10 ] 0

2

The determinant becomes

or

*λSince is negative, X*, correspond to a maximum


Kuhn – Tucker Conditions


KT condition: Both necessary and sufficient if the objective function

is concave and each constraint is linear or each constraint function is

concave, i.e., the problems belongs to a class called the convex

programming problems.


Kuhn-Tucker Conditions: Optimization Model

Consider the following optimization problem

Minimize f(X)

subject to

gj(X) ≤ 0 for j=1,2,…,p

where the decision variable vector

X=[x1,x2,…,xn]

D Nagesh Kumar, IIScWater Resources Systems Planning and Management: M2L316

Kuhn-Tucker Conditions

1

0 1, 2,...,

0 1, 2,...,

0 1, 2,...,

0 1, 2,...,

m

jji i

j j

j

j

f gi n

x x

g j m

g j m

j m

Kuhn-Tucker conditions for X* = [x1* , x2

* , . . . xn*] to be a local minimum are


Kuhn Tucker Conditions …

In case of minimization problems, if the constraints are of

the form gj(X) ≥ 0, then λj have to be non-positive

If the problem is one of maximization with the constraints

in the form gj(X) ≥ 0, then λj have to be nonnegative.


Example 1


Minimize

subject to

2 2 21 2 32 3f x x x

1 1 2 3

2 1 2 3

2 12

2 3 8

g x x x

g x x x



1 21 2 0

i i i

g gf

x x x

0 j jg

0 jg

0j


Example 1…

1 1 2

2 1 2

3 1 2

2 0 (14)

4 2 0 (15)

6 2 3 0

x

x

x

(16)

1 1 2 3

2 1 2 3

( 2 12) 0 (17)

( 2 3 8) 0 (18)

x x x

x x x

1 2 3

1 2 3

2 12 0 (19)

2 3 8 0 (20)

x x x

x x x

1

2

0 (21)

0 (22)


From (17) either = 0 or ,

Case 1: = 0

From (14), (15) and (16) we have x1 = x2 = and x3 =

Using these in (18) we get

From (22), , therefore, =0,

Therefore, X* = [ 0, 0, 0 ]

This solution set satisfies all of (18) to (21)

1 1 2 32 12 0x x x

2 / 2 2 / 222 2 28 0, 0 8or

2 0 2

Example 1…

1

D Nagesh Kumar, IISc21 Water Resources Systems Planning and Management: M2L3

Case 2:

Using (14), (15) and (16), we have

or

But conditions (21) and (22) give us and

simultaneously, which cannot be

possible with

Hence the solution set for this optimization problem is

X* = [ 0 0 0 ]

1 2 32 12 0x x x

1 2 1 2 1 22 2 312 0

2 4 3

1 217 12 144

1 0 2 0

1 217 12 144

Example 1…


Minimize

subject to

2 21 2 160f x x x

1 1

2 1 2

80 0

120 0

g x

g x x

Example 2


1 21 2 0

i i i

g gf

x x x

0 j jg

0 jg

0j


Example 2…

1 1 2

2 2

2 60 0 (23)

2 0 (24)

x

x

1 1

2 1 2

( 80) 0 (25)

( 120) 0 (26)

x

x x

1

1 2

80 0 (27)

120 0 (28)

x

x x

1

2

0 (29)

0 (30)


From (25) either = 0 or ,

Case 1

From (23) and (24) we have and

Using these in (26) we get

Considering , X* = [ 30, 0]. But this solution set violates (27)

and (28)

For , X* = [ 45, 75]. But this solution set violates (27)

1 1( 80) 0x

21 302x 2

2 2x

2 2 150 0

2 0 150or

2 0

2 150

Example 2…


Case 2:

Using in (23) and (24), we have

Substitute (31) in (26), we have

For this to be true, either

1( 80) 0x

1 80x

2 22 40 0x x

2 20 40 0x or x

Example 2…

2 2

1 2

2

2 220 (31)

x

x


For ,

This solution set violates (27) and (28)

For ,

This solution set is satisfying all equations from (27) to (31) and hence

the desired

Thus, the solution set for this optimization problem is

X* = [ 80, 40 ]

2 0x 1 220

2 40 0x 1 2140 80and

Example 2…


BIBLIOGRAPHY / FURTHER READING

1. Rao S.S., Engineering Optimization – Theory and Practice, Fourth

Edition, John Wiley and Sons, 2009.

2. Ravindran A., D.T. Phillips and J.J. Solberg, Operations Research –

Principles and Practice, John Wiley & Sons, New York, 2001.

3. Taha H.A., Operations Research – An Introduction, 8th edition, Pearson

Education India, 2008.

4. Vedula S., and P.P. Mujumdar, Water Resources Systems: Modelling

Techniques and Analysis, Tata McGraw Hill, New Delhi, 2005.


Thank You


(iii) lagrange multipliers and kuhn-tucker conditions d nagesh kumar, iisc introduction to...

Documents

m2l3optimization methods

lagrange function

m2l4d nagesh kumar

m2l4objectivesd nagesh

mwhere1water resources

optimization problem

function of n

general problemd nagesh