karush kuhn tucker slides

8/12/2019 Karush Kuhn Tucker Slides

1/45

Lagrange Multipliers and theKarush-Kuhn-Tucker conditions

March 20, 2012


2/45

Optimization

Goal:Want to nd the maximum or minimum of a function subject tosome constraints.

Formal Statement of Problem:Given functions f , g1, . . . , g m and h1, . . . , h l dened on somedomain R n the optimization problem has the form

minx f (x ) subject to gi (x )

0

i and h j (x ) = 0

j


3/45


4/45

Unconstrained Optimization


5/45

Unconstrained Minimization

Assume:Let f : R be a continuously differentiable function.

Necessary and sufficient conditions for a local minimum:x is a local minimum of f (x ) if and only if

1 f has zero gradient at x:

x f (x ) = 0

2 and the Hessian of f at w is positive semi-denite:

v t ( 2 f (x )) v 0 , v R n

where

2 f (x ) =

2 f ( x )x 21

2 f ( x )

x 1 x n

.... . .

... 2 f ( x )

x n x 1 2 f ( x )

x 2n


6/45

Unconstrained Maximization

Assume:Let f : R be a continuously differentiable function.

Necessary and sufficient conditions for local maximum:x is a local maximum of f (x ) if and only if

1 f has zero gradient at x:

f (x ) = 0

2 and the Hessian of f at x is negative semi-denite:

v t ( 2 f (x )) v 0 , v R n

where

2 f (x ) =

2 f ( x )x 21

2 f ( x )

x 1 x n

.... . .

... 2 f ( x )

x n x 1 2 f ( x )

x 2n


7/45


8/45

Tutorial Example

Problem:This is the constrained optimization problem we want to solve

minx R 2 f (x ) subject to h(x ) = 0

where

f (x ) = x1 + x

2 and h(x ) = x2

1 + x2

2 2


9/45

Tutorial example - Cost function

x 1

x2

x 1 + x 2 = 2

x 1 + x 2 = 1

x 1 + x 2 = 0

x 1 + x 2 = 1

x 1 + x 2 = 2

iso-contours of f (x )

f (x ) = x1 + x2


10/45

Tutorial example - Feasible region

x 1

x2

x 1 + x 2 = 2

x 1 + x 2 = 1

x 1 + x 2 = 0

x 1 + x 2 = 1

x 1 + x 2 = 2

iso-contours of f ( x )

feasible region: h(x ) = 0

h (x ) = x21 + x22 2


11/45

Given a point x F on the constraint surface

x 1

x2feasible point x F


12/45

Given a point x F on the constraint surface

x 1

x2

x

Find x s.t. h(x F + x ) = 0 and f (x F + x ) < f (x F)?


13/45

Condition to decrease the cost function

x 1

x2 x f (x F )

At any point x the direction of steepest descent of the costfunction f (x ) is given by x f (x ).


14/45

Condition to decrease the cost function

x 1

x2 x

Here f ( x F + x ) < f ( x F )

To move x from x such that f (x + x ) < f (x ) must have

x ( x f (x )) > 0


15/45

Condition to remain on the constraint surface

x 1

x2

x h(x F )

Normals to the constraint surface are given by x h(x )


16/45


x 1

x2

x h(x F )

Note the direction of the normal is arbitrary as the constraint beimposed as either h(x ) = 0 or h (x ) = 0


17/45


x 1

x2

Direction orthogonal to x h ( x F )

To move a small x from x and remain on the constraint surfacewe have to move in a direction orthogonal to x h(x ).


18/45

To summarize...

If x F lies on the constraint surface: setting x orthogonal to x h(x F ) ensures h(x F + x ) = 0 . And f (x F + x ) < f (x F ) only if

x ( x f (x F )) > 0

d f l l


19/45

Condition for a local optimum

Consider the case when

x f (x F ) = x h (x F )

where is a scalar.

When this occurs If x is orthogonal to x h (x F ) then

x ( x F f (x )) = x x h (x F ) = 0

Cannot move from x F to remain on the constraint surface anddecrease (or increase) the cost function .

This case corresponds to a constrained local optimum!

C di i f l l i


20/45


Consider the case when

x f (x F ) = x h (x F )

where is a scalar.

When this occurs If x is orthogonal to x h (x F ) then

x ( x F f (x )) = x x h (x F ) = 0

Cannot move from x F to remain on the constraint surface anddecrease (or increase) the cost function .

This case corresponds to a constrained local optimum!

C di i f l l i


21/45


x 1

x2

critical point

critical point

A constrained local optimum occurs at x when x f (x ) and x h(x ) are parallel that is

x f (x ) = x h(x )


22/45

F thi f t L g g M lti li k


23/45

From this fact Lagrange Multipliers make sense

Remember our constrained optimization problem is

minx R 2

f (x ) subject to h(x ) = 0

Dene the Lagrangian as note L (x , ) = f (x )

L (x , ) = f (x ) + h (x )

Then x a local minimum there exists a unique s.t.

1 x L (x

,

) = 0 encodes x f (x

) =

x h (x

)2 L (x , ) = 0 encodes the equality constraint h(x ) = 0

3 y t ( 2xx L (x , ))y 0 y s.t. x h (x ) t y = 0

Positive denite Hessian tells us we have a local minimum

The case of multiple equality constraints


24/45

The case of multiple equality constraints

The constrained optimization problem is

minx R 2

f (x ) subject to h i (x ) = 0 for i = 1 , . . . , l

Construct the Lagrangian (introduce a multiplier for each constraint )

L (x , ) = f (x ) + li=1 i h i (x ) = f (x ) + t h (x )


1 x L (x , ) = 02 L (x , ) = 0

3 y t ( 2xx L (x , ))y 0 y s.t. x h (x ) t y = 0


25/45

Constrained Optimization:

Inequality Constraints

Tutorial Example Case 1


26/45

Tutorial Example - Case 1

Problem:Consider this constrained optimization problem

minx R 2

f (x ) subject to g(x ) 0

where

f (x ) = x21 + x22 and g(x ) = x21 + x22 1

Tutorial example Cost function


27/45


x 1

x2


minimum of f ( x )

f (x ) = x21 + x22



28/45


x 1

x2

feasible region: g(x ) 0


g(x ) = x21 + x22 1

How do we recognize if x is at a local optimum?


29/45

How do we recognize if x F is at a local optimum?

x 1

x2How can we recognize x Fis at a local minimum?

Remember x F denotes a feasible point.

Easy in this case


30/45

Easy in this case

x 1

x2How can we recognize x F

is at a local minimum?Unconstrained minimum

of f (x ) lies withinthe feasible region.

Necessary and sufficient conditions for a constrained localminimum are the same as for an unconstrained local minimum.

x f (x F ) = 0 and xx f (x F ) is positive denite

This Tutorial Example has an inactive constraint


31/45

This Tutorial Example has an inactive constraint

Problem:Our constrained optimization problem

minx R 2


where

f (x ) = x21 + x22 and g(x ) = x21 + x22 1

Constraint is not active at the local minimum ( g(x ) < 0):Therefore the local minimum is identied by the same conditionsas in the unconstrained case.

Tutorial Example - Case 2


32/45

Tutorial Example Case 2

Problem:This is the constrained optimization problem we want to solve

minxR 2


where

f (x ) = ( x 1 1.1)2 + ( x 2 1.1)2 and g(x ) = x21 + x22 1



33/45

p

x 1

x2


minimum of f ( x )

f (x ) = ( x1 1.1)2 + ( x2 1.1)2



34/45

p g

x 1

x2


feasible region: g(x ) 0

g(x ) = x21 + x22 1



35/45

g p

x1

x2

Is x F at a local minimum?

Remember x F denotes a feasible point.



36/45

g p

x1

x2

How can we tell if x Fis at a local minimum?

Unconstrained localminimum of f (x )lies outside of the

feasible region.

the constrained local minimum occurs on the surface of theconstraint surface.



37/45

x1

x2

How can we tell if x Fis at a local minimum?

Unconstrained localminimum of f (x )lies outside of the

feasible region.

Effectively have an optimization problem with an equalityconstraint : g(x ) = 0 .

Given an equality constraint


38/45

x1

x2

A local optimum occurs when x f (x ) and x g(x ) are parallel:

- x f (x ) = x g(x )


39/45

Want a constrained local minimum...


40/45

x1

x2 Is a constrained

local minimum as x f (x F ) points

away from thefeasible region

Constrained local minimum occurs when x f (x ) and x g(x )point in the same direction:

x f (x ) = x g(x ) and > 0

Summary of optimization with one inequality constraint


41/45

Given

minx R 2


If x corresponds to a constrained local minimum then

Case 1 :Unconstrained local minimumoccurs in the feasible region.

1 g(x ) < 0

2 x f (x ) = 0

3 xx f (x ) is a positivesemi-denite matrix.

Case 2 :Unconstrained local minimumlies outside the feasible region.

1 g(x ) = 0

2 x f (x

) = x g(x

)with > 0

3 y t xx L(x ) y 0 for ally orthogonal to x g(x ).

Karush-Kuhn-Tucker conditions encode these conditions


42/45

Given the optimization problem

minx R 2


Dene the Lagrangian as

L (x , ) = f (x ) + g (x )


1 x L (x , ) = 0

2

03 g(x ) = 0

4 g(x ) 0

5 Plus positive denite constraints on xx L (x , ).

These are the KKT conditions .

Lets check what the KKT conditions imply


43/45

Case 1 - Inactive constraint: When = 0 then have L (x , ) = f (x ). Condition KKT 1 = x f (x ) = 0 . Condition KKT 4 = x is a feasible point.

Case 2 - Active constraint:

When > 0 then have L (x , ) = f (x ) + g(x ). Condition KKT 1 = x f (x ) = x g(x ).

Condition KKT 3 = g(x ) = 0 . Condition KKT 3 also = L (x , ) = f (x ).

KKT conditions for multiple inequality constraints


44/45

Given the optimization problem

minx R 2 f (x ) subject to g j (x )

0 for j = 1 , . . . , m

Dene the Lagrangian as

L (x , ) = f (x ) + m j =1

j g j (x ) = f (x ) + t g (x )


1 x L (x , ) = 0

2 j 0 for j = 1 , . . . , m

3 j g(x) = 0 for j = 1 , . . . , m

4 gj (x ) 0 for j = 1 , . . . , m


KKT for multiple equality & inequality constraints


45/45

Given the constrained optimization problem

minx R 2

f (x )

subject to

h i (x ) = 0 for i = 1 , . . . , l and gj (x ) 0 for j = 1 , . . . , m

Dene the Lagrangian asL (x , , ) = f (x ) + t h (x ) + t g (x )


1 x L (x , , ) = 02 j 0 for j = 1 , . . . , m3 j gj (x ) = 0 for j = 1 , . . . , m4 gj (x ) 0 for j = 1 , . . . , m5 h (x ) = 0


karush kuhn tucker slides

Documents