Lecture 7: Lagrangian Relaxation and Duality Theory

(3 units)

Outline

I Lagrangian dual for linear IP

I Lagrangian dual for general IP

I Dual Search

I Lagrangian decomposition

1 / 23

Joseph Louis Lagrange

Joseph Louis Lagrange (1736-1813)

Lagrange was one of the creators of the calculus of variations,deriving the Euler-Lagrange equations for extrema of functionals.He also extended the method to take into account possibleconstraints, arriving at the method of Lagrange multipliers.

2 / 23

Lagrangian Dual for Linear Integer Programming

I IP with “nice” and complicated constraints:

max cT x

s.t. Ax ≤ b (nice constraints)Dx ≤ d (complicated constraintsx ∈ Zn

+.

Observation: Dropping Dx ≤ d results in an easy problem.

3 / 23

I More general form:

(IP) max cT x

s.t. Dx ≤ d

x ∈ X ,

where Dx ≤ d is m hard constraints.I Largrangian relaxation:

(LP(u)) z(u) = max cT x + uT (d − Dx)

s.t. x ∈ X .

Property: Problem (LP(u)) is a relaxation of (IP) for allu ≥ 0.

I Lagrnagian dual of (IP):

(LD) minu≥0

z(u).

For equality constraints: Dx = d . The Lagrangain dual is

(LD) minu∈Rm

z(u).

4 / 23

Example 1

I Uncapacitated facility location problem:The uncapacitatedfacility location problem (UFLP) involves locating an undeter-mined number of facilities to minimize the sum of the(annualized) fixed setup costs and the variable costs of servingthe market demand from these facilities.

I Formulation:

(IP) max∑

i∈M

∑

j∈N

cijxij −∑

j∈N

fjyj

s.t.∑

j∈N

xij = 1, i ∈ M

xij − yj ≤ 0, i ∈ M, j ∈ N

x ∈ R|M|×‖N|, y ∈ {0, 1}|N|.

Locations: N = {1, . . . , n}; Clients: M = {1, . . . ,m}, cij :profit

5 / 23

I Lagrangian relaxation (IP(u)):

z(u) = max∑

i∈M

∑

j∈N

(cij − ui )xij −∑

j∈N

fjyj +∑

i∈M

ui

s.t. xij − yj ≤ 0, i ∈ M, j ∈ N

x ∈ R|M|×‖N|, y ∈ {0, 1}|N|.I Decomposition: z(u) = zj(u) +

∑i∈M ui , where

zj(u) = max∑

i∈M

(cij − ui )xij − fjyj

s.t. xij − yj ≤ 0, i ∈ M

xij ≥ 0, i ∈ M, yj ∈ {0, 1}.

It is a subproblem for location j . We have

zj(u) = max{0,∑

i∈M

max(cij − ui , 0)− fj}.

6 / 23

Example 2

I Generalized assignment problem (GAP). The generalizedassignment problem is an excellent example for illustratingLagrangian relaxation because it is rich with readily apparentstructure.

I Formulation:

Z = minm∑

i=1

n∑

j=1

cijxij (1)

s.t.m∑

i=1

xij = 1, j = 1, . . . , n (2)

n∑

j=1

aijxij ≤ bi , i = 1, . . . ,m (3)

xij ∈ {0, 1}, for all i , j . (4)

7 / 23

I Relaxations 1:

ZD1(u) = minm∑

i=1

n∑

j=1

cijxij +n∑

j=1

uj(m∑

i=1

xij − 1)

s.t. (3), (4)

= minm∑

i=1

n∑

j=1

(cij + uj)xij −n∑

j=1

uj

s.t. (3), (4).

This problem reduces to m 0-1 knapsack problems and canthus be solved in time proportional to n

∑mi=1 bi .

8 / 23

I Relaxation 2

ZD2(v) = minm∑

i=1

n∑

j=1

cijxij +m∑

i=1

vi (n∑

j=1

aijxij − bi )

s.t. (2), (4)

= minn∑

j=1

m∑

i=1

(cij + viaij)xij −m∑

i=1

vibi

s.t. (2), (4).

Since constraints (2) are generalized upper bound (GUB)constraints, it is 0-1 GUB problem. Such a problem is easilysolved in time proportional to nm by determiningmini (cij + viaij) for each j and setting the associated xij = 1.Remaining xij are set to zero.

I Classical paper: Marshall L. Fisher, The Lagrangian RelaxationMethod for Solving Integer Programming Problems, Manage.Sci., Vol. 50, No. 12 Supplement, December 2004, pp.1861–1871. (Original, MS, Vol. 27(1), 1-18, 1981).

9 / 23

How Good is Lagrangian Dual?

I Consider linear program:

min{cT x | Ax ≤ b, Cx ≤ d , x ∈ X},where X is a finite integer set, e.g., X = {0, 1}n.

I Let {x1, . . . , xN} be the finite integer points in{x ∈ X | Cx ≤ d}.

I Dual problem:

v(D) = maxλ≥0

d(λ)

= maxλ≥0

minx∈X

[cT x + λT (Ax − d)]

= maxλ≥0

mini=1,...,N

[cT x i + λT (Ax i − b)]

= max η

s.t. cT x i + λT (Ax i − b) ≤ η, i = 1, . . . ,N,

λ ≥ 0, η ∈ R.

10 / 23

I The latter problem is a linear problem. Taking its dual gives

v(D) = minN∑

i=1

µi (cT x i )

s.t.N∑

i=1

µi (Ax i − d) ≤ 0,

N∑

i=1

µi = 1, µi ≥ 0.

Setting x =∑N

i=1 µixi , we imply that (D) is equivalent to the

following LP:

(PR) min cT x

s.t. Ax ≤ b,

x ∈ Co({x ∈ X | Cx ≤ d}).

11 / 23

I Theorem:

v(D) = v(PR) = min{cT x | Ax ≤ b, x ∈ Co({x ∈ X | Cx ≤ d})}.

I v(D) ≥ v(LP) = min{cT x | Ax ≤ b, Cx ≤ d}).I If Co({x ∈ X | Cx ≤ b}) = {x ∈ X | Cx ≤ b}, then

v(D) = v(LP). Lagrangian bound is equal to the LPrelaxation. Example: {x | Zn

+ | l ≤ x ≤ u}.

12 / 23

Figure: Lagrangian bound for linear integer programming.

13 / 23

Key issues in Lagrangian dual

I How to solve the Lagrangian relaxation?

I How to solve the dual problem?

I When Lagrangian relaxation useful?

14 / 23

Solving Relaxation Problem

I Linear integer program;

I Many combinatorial optimization problems

I separable integer programming (knapsack problem)

I 0-1 unconstrained quadratic problem

I 0-1 quadratic knapsack program

15 / 23

Dual Search Methods

I Problem: How to solve the dual problem (D)?

(D) maxλ≥0

d(λ).

I Three Basic Methods:I Subgradient methodI Outer approximation methodI Bundle method

16 / 23

Subgradient methodI Subgradient method

λk+1 = P+(λk + skξk/‖ξk‖),where sk > 0 is the stepsize.

I Stepsize rules for sk :I Rule 1 for stepsize (constant):

sk = ε,

where ε > 0 is a constant.I Rule 2 for stepsize:

+∞∑

k=1

s2k < +∞ and

+∞∑

k=1

sk = +∞.

I Rule 3 for stepsize:

sk → 0, k → +∞, and+∞∑

k=1

sk = +∞.

17 / 23

I A sophisticated stepsize rule:

sk = ρwk − d(λk)

‖ξk‖ , 0 < ρ < 2,

where wk is an approximation of the optimal value v(D),wk ≥ d(λk) and ξk 6= 0.

18 / 23

Outer approximation methodI Rewrite dual problem (D) as

max µ

s.t. µ ≤ f (x) + λT (g(x)− b),∀ x ∈ X ,

λ ≥ 0.

I Approximate the LP by:

max µ

s.t. µ ≤ f (x j) + λT (g(x j)− b), ∀x j ∈ T k ,

λ ≥ 0.

I Finite convergenceI Singly constrained case of (P):

(Ps) min f (x)

s.t. g(x) ≤ b

x ∈ X .

19 / 23

0 0.5 1 1.5 2 2.5 3 3.5 4−4

−2

0

2

4

6

8

d(λ)

λ1 =5/3 λ2=2

l1

l2

l3

λ

y

Figure: Dual function.

20 / 23

Lagrangian decomposition

I Purpose:I Induce decomposition of the problem into independent

subproblems;I Capture different structural characteristics of the problem;I Obtain stronger bounds than by standard Lagrangean

relaxation schemes

I How?I Identify the parts of the problem that should be split;I Replace variables in each part by copies or substitute new

expressions;I Dualize the copy or substitution expression.

21 / 23

I Convex program with nice and hard constraints:

(CP) min f (x) + g(x)

s.t. x ∈ X (nice constraints)x ∈ Y (hard constraints)

I Lagrangian decomposition via copying constraints

(CP) min f (x) + g(z)

s.t. x ∈ X (nice constraints)x = z , (link constraints)z ∈ Y (hard constraints)

22 / 23

I Associating a multiplier to each link constraint:

(CP) min f (x) + g(z)

s.t. x ∈ X

λ ⇒ x = z ,

z ∈ Y

I Decomposition:

d(λ) = min f (x) + g(z) + λT (x − z)

s.t. x ∈ X

z ∈ Y

= min f (x) + λT x + min g(z)− λT z

s.t. x ∈ X , s.t. z ∈ Y

= d2(λ) + d2(λ) ⇒ two easier subproblems!

23 / 23