UMass Lowell Computer Science 91.503 Analysis of Algorithms

Prof. Giampiero Pecelli Fall, 2010

Paradigms for Optimization ProblemsParadigms for Optimization ProblemsDynamic Programming Dynamic Programming

& & Greedy AlgorithmsGreedy Algorithms

Optimization

This, generally, refers to classes of problems that possess This, generally, refers to classes of problems that possess multiple solutions at one level, and where we have a real-multiple solutions at one level, and where we have a real-valued function defined on the solutions.valued function defined on the solutions.

ProblemProblem: find a solution that minimizes or maximizes the : find a solution that minimizes or maximizes the value of this function.value of this function.

NoteNote: there is no guarantee that such a solution will be : there is no guarantee that such a solution will be unique and, moreover, there is no guarantee that you will unique and, moreover, there is no guarantee that you will find it (local maxima, anyone?) unless the search is over a find it (local maxima, anyone?) unless the search is over a small enough search space or the function is restricted small enough search space or the function is restricted enough.enough.

Optimization

QuestionQuestion: are there classes of problems for which you can : are there classes of problems for which you can guarantee an optimizing solution can be found?guarantee an optimizing solution can be found?

AnswerAnswer: yes. BUT you also need to find such a solution in a : yes. BUT you also need to find such a solution in a "reasonable" amount of time."reasonable" amount of time.

We are going to look at two classes of problems, and the We are going to look at two classes of problems, and the techniques that will succeed in constructing their solutions in techniques that will succeed in constructing their solutions in a "reasonable" (i.e., low degree polynomial in the size of the a "reasonable" (i.e., low degree polynomial in the size of the initial data) amount of time.initial data) amount of time.

Optimization

We begin with a rough comparison that contrasts a method We begin with a rough comparison that contrasts a method you are familiar with (divide and conquer) and the method you are familiar with (divide and conquer) and the method (still unspecified) of Dynamic Programming (developed by (still unspecified) of Dynamic Programming (developed by Richard Bellman in the late 1940's and early 1950's).Richard Bellman in the late 1940's and early 1950's).

For some history and other ideas, see:For some history and other ideas, see:http://en.wikipedia.org/wiki/Dynamic_programming

Two Algorithmic Models:

Divide &Conquer

DynamicProgramming

View problem as collection ofsubproblems“Recursive” nature

Independent subproblems

Number of subproblems depends onpartitioning

factors

typically small

PreprocessingCharacteristic running time typically log

function of ndepends on numberand difficulty ofsubproblems

Primarily for optimizationproblemsOptimal substructure:optimal solution to problemcontains within it optimalsolutions to subproblems

Overlapping subproblems

Dynamic ProgrammingDynamic Programming

Example: Rod Cutting (text)

You are given a rod of length You are given a rod of length nn ≥ 0 ( ≥ 0 (nn in inches) in inches)

A rod of length A rod of length ii inches will be sold for inches will be sold for ppii dollars dollars

Cutting is free (simplifying assumption)Cutting is free (simplifying assumption)

ProblemProblem: given a table of prices : given a table of prices ppii determine the maximum determine the maximum

revenue revenue rrnn obtainable by cutting up the rod and selling the obtainable by cutting up the rod and selling the

pieces.pieces.Length i

Price pi

1 2 3 4 5 6 7 8 9 101 5 8 9 10 17 17 20 24 30

Example: Rod Cutting

We can see immediately (from the values in the table) that We can see immediately (from the values in the table) that nn ≤ ≤ ppnn ≤ 3 ≤ 3nn. .

This is not very useful because:This is not very useful because: The range of potential revenue is very largeThe range of potential revenue is very large Our finding quick upper and lower bounds depends on Our finding quick upper and lower bounds depends on

finding quickly the minimum and maximum finding quickly the minimum and maximum ppii//ii ratios (one ratios (one

pass through the table), but then we are back to the point pass through the table), but then we are back to the point above…. above….

Example: Rod Cutting Step 1: Characterizing an Optimal Solution

QuestionQuestion: in how many different ways can we cut a rod of length : in how many different ways can we cut a rod of length nn??For a rod of length 4:For a rod of length 4:

224 - 14 - 1 = 2 = 233 = 8 = 8

For a rod of length For a rod of length nn: 2: 2nn-1-1. . ExponentialExponential: we cannot try all possibilities for : we cannot try all possibilities for nn "large". The obvious exhaustive approach won't work."large". The obvious exhaustive approach won't work.

Example: Rod Cutting Step 1: Characterizing an Optimal Solution

QuestionQuestion: in how many different ways can we cut a rod of length : in how many different ways can we cut a rod of length nn??

Proof DetailsProof Details: a rod of length : a rod of length nn can have exactly can have exactly nn-1 possible cut positions – -1 possible cut positions – choose 0 ≤ choose 0 ≤ kk ≤ ≤ nn-1 actual cuts. We can choose the -1 actual cuts. We can choose the kk cuts (without repetition) cuts (without repetition) anywhere we want, so that for each such anywhere we want, so that for each such kk the number of different choices is the number of different choices is

When we sum up over all possibilities (When we sum up over all possibilities (kk = 0 to = 0 to kk = = nn-1):-1):

For a rod of length For a rod of length nn: 2: 2nn-1-1. .

€

n −1k

⎛ ⎝ ⎜

⎞ ⎠ ⎟k= 0

n−1∑ = (n −1)!k!(n −1− k)!k= 0

n−1∑ = 1+1( )n−1 = 2n−1.€

n −1k

⎛ ⎝ ⎜

⎞ ⎠ ⎟

Example: Rod Cutting Characterizing an Optimal Solution

Let us find a way to solve the problem recursively (we might be able to modify the Let us find a way to solve the problem recursively (we might be able to modify the solution so that the maximum can be actually computed): assume we have cut a solution so that the maximum can be actually computed): assume we have cut a rod of length rod of length nn into 0 ≤ into 0 ≤ k k ≤ ≤ nn pieces of length pieces of length ii11, …, , …, iikk,,

nn = = ii11 +…+ +…+ iikk, , with revenuewith revenuerrnn = = ppii11 + … + + … + ppikik

Assume further that this solution is optimal.Assume further that this solution is optimal.

How can we construct it?How can we construct it?

AdviceAdvice: when you don’t know what to do next, start with a simple example and : when you don’t know what to do next, start with a simple example and hope something will occur to you…hope something will occur to you…

Example: Rod Cutting Characterizing an Optimal Solution

We begin by constructing (by hand) the optimal solutions for We begin by constructing (by hand) the optimal solutions for ii = 1, …, 10: = 1, …, 10:rr11 = 1 = 1 from sln. 1 = 1 (no cuts)from sln. 1 = 1 (no cuts)rr22 = 5 = 5 from sln. 2 = 2 (no cuts)from sln. 2 = 2 (no cuts)rr33 = 8 = 8 from sln. 3 = 3 (no cuts)from sln. 3 = 3 (no cuts)rr44 = 10 = 10 from sln. 4 = 2 + 2from sln. 4 = 2 + 2rr55 = 13 = 13 from sln. 5 = 2 + 3from sln. 5 = 2 + 3rr66 = 17 = 17 from sln. 6 = 6 (no cuts)from sln. 6 = 6 (no cuts)rr77 = 18 = 18 from sln. 7 = 1 + 6 or 7 = 2 + 2 + 3from sln. 7 = 1 + 6 or 7 = 2 + 2 + 3rr88 = 22 = 22 from sln. 8 = 2 + 6from sln. 8 = 2 + 6rr99 = 25 = 25 from sln. 9 = 3 + 6from sln. 9 = 3 + 6rr1010 = 30 = 30 from sln. 10 = 10 (no cuts)from sln. 10 = 10 (no cuts)

Length i

Price pi

1 2 3 4 5 6 7 8 9 101 5 8 9 10 17 17 20 24 30

Example: Rod Cutting Characterizing an Optimal Solution

Notice that in some cases Notice that in some cases rrnn = = ppnn, while in other cases the optimal revenue , while in other cases the optimal revenue rrnn is is obtained by cutting the rod into smaller pieces. obtained by cutting the rod into smaller pieces. In ALL cases we have the recursionIn ALL cases we have the recursion

rrnn = max( = max(ppnn, , rr11 + + rrnn-1-1, , rr22 + + rrnn-2-2, …, , …, rrnn-1-1 + + rr11))exhibiting exhibiting optimal substructure optimal substructure (meaning?)(meaning?)A slightly different way of stating the same recursion, which avoids repeating A slightly different way of stating the same recursion, which avoids repeating some computations, issome computations, is

rrnn = max = max1≤1≤ii≤≤nn((ppii + + rrnn--ii))And this latter relation can be implemented as a simple top-down recursive And this latter relation can be implemented as a simple top-down recursive procedure:procedure:

Example: Rod Cutting Characterizing an Optimal Solution

Time Out:Time Out: How to justify the step from: How to justify the step from:rrnn = max( = max(ppnn, , rr11 + + rrnn-1-1, , rr22 + + rrnn-2-2, …, , …, rrnn-1-1 + + rr11))

totorrnn = max = max1≤1≤ii≤≤nn((ppii + + rrnn--ii))

NoteNote: every optimal partitioning of a rod of length : every optimal partitioning of a rod of length nn has a first cut – a segment of, has a first cut – a segment of, say, length say, length ii. The optimal revenue, . The optimal revenue, rrnn, must satisfy , must satisfy rrnn = = ppii + r + rn-in-i, where , where rrn-in-i is the is the optimal revenue for a rod of length optimal revenue for a rod of length nn – – ii. If the latter were not the case, there would . If the latter were not the case, there would be a better partitioning for a rod of length be a better partitioning for a rod of length nn – – ii, giving a revenue , giving a revenue r’r’n–in–i > > rrn-in-i and a and a total revenue total revenue r’r’nn = = ppnn + + r’r’n-in-i > > ppii + r + rn-i n-i == r rnn. . Since we do not know which one of the leftmost cut positions provides the largest Since we do not know which one of the leftmost cut positions provides the largest revenue, we just maximize over all the possible first cut positions.revenue, we just maximize over all the possible first cut positions.

Example: Rod Cutting Characterizing an Optimal Solution

We can also notice that all the items we choose the maximum of are optimal in We can also notice that all the items we choose the maximum of are optimal in their own right: each substructure (max revenue for rods of lengths 1, …, their own right: each substructure (max revenue for rods of lengths 1, …, nn-1) is -1) is also optimal (again, also optimal (again, optimal substructure propertyoptimal substructure property).).

Nevertheless, we are still in trouble: computing the recursion leads to recomputing Nevertheless, we are still in trouble: computing the recursion leads to recomputing a number (= overlapping subproblems) of values – how many?a number (= overlapping subproblems) of values – how many?

Example: Rod Cutting Characterizing an Optimal Solution

Let’s call Cut-Rod(p, 4), to see the effects on a simple case:Let’s call Cut-Rod(p, 4), to see the effects on a simple case:

The number of nodes for a tree corresponding to a rod of size The number of nodes for a tree corresponding to a rod of size nn is: is:

€

T 0( ) =1, T(n) =1+ T( j)j= 0

n−1∑ = 2n, n ≥1.

Example: Rod Cutting Beyond Naïve Time Complexity

We have a problem: “reasonable size” problems are not solvable in “reasonable We have a problem: “reasonable size” problems are not solvable in “reasonable time” (but, in this case, they are solvable in “reasonable space”). time” (but, in this case, they are solvable in “reasonable space”).

SpecificallySpecifically::• Note that navigating the whole tree requires 2Note that navigating the whole tree requires 2nn stack-frame activations. stack-frame activations.• Note also that no more than Note also that no more than nn + 1 stack-frames are active at any one time and that + 1 stack-frames are active at any one time and that no more than no more than nn + 1 different values need to be computed or used. + 1 different values need to be computed or used.

Can we exploit these observationsCan we exploit these observations??A standard solution method involves saving the values associated with each A standard solution method involves saving the values associated with each TT((jj), ), so that we compute each value only once (called “so that we compute each value only once (called “memoizingmemoizing” = writing yourself a ” = writing yourself a memo).memo).

Example: Rod Cutting Naïve Caching

We introduce two procedures:We introduce two procedures:

Example: Rod Cutting More Sophisticated Caching

We now remove some unnecessary complications:We now remove some unnecessary complications:

Example: Rod Cutting Time Complexity

Whether we solve the problem in a top-down or bottom-up manner the asymptotic Whether we solve the problem in a top-down or bottom-up manner the asymptotic time is Θ(time is Θ(nn22), the major difference being recursive calls as compared to loop ), the major difference being recursive calls as compared to loop iterations.iterations.Why??Why??

Example: Longest Common Subsequence (LCS): Motivation

Strand of DNA: string over finite set {A,C,G,T}Strand of DNA: string over finite set {A,C,G,T} each element of set is a base: each element of set is a base: adenine, guanine, cytosine or thymineadenine, guanine, cytosine or thymine

Compare DNA similaritiesCompare DNA similarities SS1 1 = = ACCACCGGGGTCGTCGAGAGTTGGCGCGCCGGGGAAGCCGGCCGAAAAGCCGGCCGAA SS2 2 = = GTCGTGTCGTTTCGGAACGGAATTGCCGGCCGTTTTGCGCTTCCTTGGTTAAAAAA One measure of similarity:One measure of similarity:

find the longest string Sfind the longest string S33 containing bases that also appear (not containing bases that also appear (not necessarily necessarily consecutivelyconsecutively) in S) in S11 and S and S22

SS3 3 = = GTCGTCGGAAGCCGGCCGAAGTCGTCGGAAGCCGGCCGAAsource: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.

Example: LCS Definitions

The sequenceThe sequence is a is a subsequencesubsequence of of if if (strictly increasing indices (strictly increasing indices of of XX) such that) such that example: example: is a subsequence of is a subsequence of

with index sequence with index sequence ZZ is is common subsequencecommon subsequence of of XX and and YY if if ZZ is subsequence is subsequence

of both of both X X and and YY example:example:

common subsequence but not common subsequence but not longestlongest common subsequence. Longest?common subsequence. Longest?

Longest Common Subsequence ProblemLongest Common Subsequence Problem: Given 2 sequences : Given 2 sequences X,X, YY, find maximum-length common subsequence , find maximum-length common subsequence ZZ..

source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.

€

Z = z1,z2,K ,zk

€

X = x1, x2,K xm

€

∃ i1,i2,K ik

€

∀ j =1,2,K k,

€

x i j= z j

€

Z = B,C,D,B

€

X = A,B,C,B,D, A,B

€

2,3,5,7

€

X = A,B,C,B,D, A,B

€

Y = B,D,C, A,B, A

€

B,C, A

€

B,C,B,A

Example: LCS Step 1: Characterize an LCS

THM 15.1: Optimal LCS Substructure THM 15.1: Optimal LCS Substructure Given sequences:Given sequences:For any LCSFor any LCS of of XX and and YY: :

1 1 if if xxmm = = yynn then then zzkk = = xxmm = = yynn and and ZZkk-1-1 is an LCS of is an LCS of XXmm-1-1 and and YYnn-1-1

2 2 if if xxmm ≠ ≠ yynn then then zzkk ≠ ≠ xxmm ZZ is an LCS of is an LCS of XXmm-1-1 and and YY

3 if 3 if xxmm ≠ ≠ yynn then then zzkk ≠ ≠ yynn Z Z is an LCS of is an LCS of XX and and YYnn-1-1

PROOF: based on producing PROOF: based on producing contradictionscontradictions1 a) 1 a) Suppose Suppose zzkk ≠ ≠ xxmm. Appending . Appending xxmm = = yynn to to Z Z contradicts contradicts longest longest nature ofnature of Z Z..

b) To establish b) To establish longest longest nature of nature of ZZkk-1-1, suppose common subsequence , suppose common subsequence W W of of XXmm-1-1 and and YYnn-1-1 has has length length > > kk-1-1. Appending . Appending xxmm to to WW yields common subsequence of length yields common subsequence of length > > kk = contradiction.= contradiction.

2 2 Common subsequence Common subsequence WW of of XXmm-1-1 and and YY of length of length > > kk would also be common subsequence of would also be common subsequence of XXmm, , YY, contradicting , contradicting longest longest nature of nature of ZZ.. 3 3 Similar to proof of (2)Similar to proof of (2)

source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.

€

X = Xm = x1,x2,K , xm

€

Y = Yn = y1,y2,K ,yn

€

Z = Zk = z1,z2,K ,zk

Example: LCS Step 2: A Recursive Solution

Implications of Thm 15.1:Implications of Thm 15.1:

??yesyes nono

Find LCS(Xm-1, Yn-1) Find LCS(Xm-1, Y) Find LCS(X, Yn-1)

LCS(X, Y) = LCS(Xm-1, Yn-1) + xm LCS(X, Y) = max(LCS(Xm-1, Y), LCS(X, Yn-1))

LCS(X, Y)

Example: LCS Step 2: A Recursive Solution (continued)

Overlapping subproblem structure:Overlapping subproblem structure:

Recurrence for length of optimal solution:Recurrence for length of optimal solution:

Conditions of problem can exclude some subproblems!

),(),( 111 YXLCSYXLCS mnm −−− ⊆),( 1−⊆ nYXLCS

),( YXLCS⊆

c[i,j]= c[i-1,j-1]+1 if i,j > 0 and xi=yj

max(c[i,j-1], c[i-1,j]) if i,j > 0 and xi=yj

0 if i=0 or j=0

(mn) distinct (mn) distinct subproblemssubproblems

source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.

Example: LCS Step 3: Compute Length of an LCS

c tablec table

(represent b table)(represent b table)source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.

Example: LCS Step 4: Construct an LCS

source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.

Example: LCS Improve the Code

Can eliminate Can eliminate bb table table cc[[ii,,jj]] depends only on 3 other depends only on 3 other cc table entries: table entries:

cc[[ii-1,-1,jj-1], -1], cc[[ii-1,-1,jj], ], cc[[ii,j,j--1]1] given value of given value of cc[[ii,,jj]], can pick the one in O(1) time, can pick the one in O(1) time reconstruct LCS in O(reconstruct LCS in O(mm++nn) time similar to PRINT-LCS) time similar to PRINT-LCS same same ((mnmn) space, but ) space, but ((mnmn) was needed anyway...) was needed anyway...

Asymptotic space reductionAsymptotic space reduction leverage: need only 2 rows of leverage: need only 2 rows of cc table at a time table at a time

row being computedrow being computed previous rowprevious row

can also do it with ~ space for 1 row of can also do it with ~ space for 1 row of cc table table but does not preserve LCS reconstruction databut does not preserve LCS reconstruction data

source: 91.503 textbook Cormen, et source: 91.503 textbook Cormen, et al.al.

Algorithmic Paradigm Context

Divide &Conquer

DynamicProgramming

GreedyAlgorithm

View problem as collection ofsubproblems“Recursive” natureIndependent subproblems overlapping typically

sequentialdependence

Number of subproblems depends onpartitioning

factors

typically small

Preprocessing typically sortCharacteristic running time typically log

function of ndepends on numberand difficulty ofsubproblems

often dominatedby nlogn sort

Primarily for optimizationproblemsOptimal substructure:optimal solution to problemcontains within it optimalsolutions to subproblemsGreedy choice property:locally optimal producesglobally optimalHeuristic version useful forbounding optimal value