UMass Lowell Computer Science 91.503

Analysis of Algorithms Prof. Karen Daniels

UMass Lowell Computer Science 91.503

Analysis of Algorithms Prof. Karen Daniels

Design Patterns for Optimization Problems

Dynamic ProgrammingMatrix Parenthesizing

Longest Common SubsequenceActivity Selection

Algorithmic Paradigm ContextAlgorithmic Paradigm Context

Divide & Conquer

Dynamic Programming

Greedy Algorithm

View problem as collection of subproblems

“Recursive” nature Independent subproblems overlapping typically

sequential dependence

Number of subproblems depends on partitioning

factors

typically small

Preprocessing typically sort Characteristic running time typically log

function of n depends on number and difficulty of subproblems

often dominated by nlogn sort

Primarily for optimization problems (find an optimal solution)

Optimal substructure: optimal solution to problem contains within it optimal solutions to subproblems

Greedy choice property: locally optimal produces globally optimal

Heuristic version useful for bounding optimal value

Subproblem solution orderMake choice, then solve subproblem(s)

Solve subproblem(s), then make choice

Dynamic Programming Approach to Optimization ProblemsDynamic Programming Approach to Optimization Problems

1. Characterize structure of an optimal solution.

2. Recursively define value of an optimal solution.

3. Compute value of an optimal solution, typically in bottom-up fashion.

4. Construct an optimal solution from computed information.

(separate slides for rod cutting)

source: 91.503 textbook Cormen, et al.

Dynamic Programming

Matrix Parenthesization

Example: Matrix Parenthesization Definitions

Example: Matrix Parenthesization Definitions

Given “chain” of n matrices: <A1, A2, … An, >

Compute product A1A2… An efficiently

Multiplication order matters!

Matrix multiplication is associative

Minimize “cost” = number of scalar

multiplications

source: 91.503 textbook Cormen, et al.

Example: Matrix Parenthesization Step 1: Characterizing an Optimal SolutionExample: Matrix Parenthesization Step 1: Characterizing an Optimal Solution

Observation: Any parenthesization of AiAi+1… Aj must split it between Ak and Ak+1 for some k.

THM: Optimal Matrix Parenthesization:If an optimal parenthesization of AiAi+1… Aj splits at k, then parenthesization of prefix AiAi+1… Ak must be an optimal parenthesization.

Why? If existed less costly way to parenthesize prefix, then substituting that parenthesization would yield less costly way to parenthesize AiAi+1… Aj , contradicting optimality of that parenthesization.

source: 91.503 textbook Cormen, et al.

common DP proof technique: “cut-and-paste” proof by contradiction

Example: Matrix Parenthesization Step 2: A Recursive SolutionExample: Matrix Parenthesization Step 2: A Recursive Solution

Recursive definition of minimum parenthesization cost:

m[i,j]= min{m[i,k] + m[k+1,j] + pi-1pkpj} if i < j

0 if i = j

How many distinct subproblems?

i <= k < j

each matrix Ai has dimensions pi-1 x pi

source: 91.503 textbook Cormen, et al.

Example: Matrix Parenthesization Step 3: Computing Optimal CostsExample: Matrix Parenthesization Step 3: Computing Optimal Costs

0

2,625

2,500

1,000

s: value of k that achieves optimal cost in computing m[i, j]

source: 91.503 textbook Cormen, et al.

Example: Matrix Parenthesization Step 4: Constructing an Optimal SolutionExample: Matrix Parenthesization Step 4: Constructing an Optimal Solution

PRINT-OPTIMAL-PARENS(s, i, j)if i == j

print “A”i

else print “(“ PRINT-OPTIMAL-PARENS(s, i, s[i, j]) PRINT-OPTIMAL-PARENS(s, s[i, j]+1, j) print “)“

source: 91.503 textbook Cormen, et al.

Example: Matrix Parenthesization Memoization

Example: Matrix Parenthesization Memoization

• Provide Dynamic Programming efficiency

• But with top-down strategy• Use recursion• Fill in m table “on demand”

• (can modify to fill in s table)

source: 91.503 textbook Cormen, et al.

MEMOIZED-MATRIX-CHAIN(p)

1 n = p.length – 1

2 let m[1…n,1…n] be a new table.

3 for i = 1 to n

4 for j = i to n

5 m[i,j] =

6 return LOOKUP-CHAIN(m, p,1,n)

LOOKUP-CHAIN(m,p,i,j) 1 if m[i,j] <

2 return m[i,j]

3 if i==j

4 m[i,j] = 0

5 else for k = i to j-1

6 q = LOOKUP-CHAIN(m,p,i,k)

+ LOOKUP-CHAIN(m,p,k+1,j)

+ pi-1 pk pj

7 if q < m[i,j]

8 m[i,j] = q

9 return m[i,j]

Dynamic Programming

Longest Common Subsequence

Example: Longest Common Subsequence (LCS): MotivationExample: Longest Common Subsequence (LCS): Motivation

• Strand of DNA: string over finite set {A,C,G,T}• each element of set is a base: adenine, guanine, cytosine or thymine

• Compare DNA similarities• S1 = ACCGGTCGAGTGCGCGGAAGCCGGCCGAA

• S2 = GTCGTTCGGAATGCCGTTGCTCTGTAAA

• One measure of similarity:• find the longest string S3 containing bases that also appear (not

necessarily consecutively) in S1 and S2

• S3 = GTCGTCGGAAGCCGGCCGAA

source: 91.503 textbook Cormen, et al.

Example: LCS Definitions

Example: LCS Definitions

• Sequence is a subsequence of if (strictly increasing

indices of X) such that• example: is subsequence of

with index sequence

• Z is common subsequence of X and Y if Z is subsequence of both X and Y• example:

• common subsequence but not longest• common subsequence. Longest?

kzzzZ ,,, 21 mxxxX ,,, 21 kiii ,,, 21

ji zxkjj ,,2,1

BDCBZ ,,, BADBCBAX ,,,,,,

ABACDBY ,,,,,

7,5,3,2

ACB ,, ABCB ,,,

BADBCBAX ,,,,,,

Longest Common Subsequence Problem: Given 2 sequences X, Y, find maximum-length common subsequence Z.

source: 91.503 textbook Cormen, et al.

Example: LCS Step 1: Characterize an LCS

Example: LCS Step 1: Characterize an LCS

THM 15.1: Optimal LCS Substructure Given sequences:For any LCS of X and Y:

1 if then and Zk-1 is an LCS of Xm-1 and Yn-1

2 if then Z is an LCS of Xm-1 and Y

3 if then Z is an LCS of X and Yn-1

mxxxX ,,, 21 nyyyY ,,, 21 kzzzZ ,,, 21

nm yx nmk yxz

nm yx

nm yx mk xz

nk yz

PROOF: based on producing contradictions

1 a) Suppose . Appending to Z contradicts longest nature of Z.

b) To establish longest nature of Zk-1, suppose common subsequence W of Xm-1 and Yn-1 has length > k-1. Appending to W yields common subsequence of length > k = contradiction.

2 To establish optimality (longest nature), common subsequence W of Xm-1 and Y of length > k would also be common subsequence of Xm, Y, contradicting longest nature of Z.

3 Similar to proof of (2)

mk xz mx

mx

source: 91.503 textbook Cormen, et al.

(using prefix notation)

Example: LCS Step 2: A Recursive Solution

Example: LCS Step 2: A Recursive Solution

Implications of Theorem 15.1:

nm yx ?yes no

Find LCS(Xm-1, Yn-1) Find LCS(Xm-1, Y) Find LCS(X, Yn-1)

LCS1(X, Y) = LCS(Xm-1, Yn-1) + xm LCS2(X, Y) = max(LCS(Xm-1, Y), LCS(X, Yn-1))

An LCS of 2 sequences contains, as a prefix, an LCS of prefixes of the sequences.

Example: LCS Step 2: A Recursive Solution (continued)

Example: LCS Step 2: A Recursive Solution (continued)

• Overlapping subproblem structure:

• Recurrence for length of optimal solution:

Conditions of problem can exclude some subproblems!

),(),( 111 YXLCSYXLCS mnm

),(),( 111 nnm YXLCSYXLCS),( YXLCS

c[i,j]= c[i-1,j-1]+1 if i,j > 0 and xi=yj

max(c[i,j-1], c[i-1,j]) if i,j > 0 and xi=yj

0 if i=0 or j=0

Q(mn) distinct subproblems

source: 91.503 textbook Cormen, et al.

Example: LCS Step 3: Compute Length of an LCSExample: LCS Step 3: Compute Length of an LCS

source: 91.503 textbook Cormen, et al.

c table

(represent b table)

0

1

2

3

4

What is the asymptotic worst-

case time complexity?

Example: LCS Step 4: Construct an LCSExample: LCS Step 4: Construct an LCS

source: 91.503 textbook Cormen, et al.

8

Dynamic Programming…leading to a Greedy Algorithm…

Activity Selection

Activity Selection Optimization Problem Activity Selection Optimization Problem

• Problem Instance:• Set S = {a1,a2,...,an} of n activities

• Each activity i has:• start time: si

• finish time: fi

• Activities require exclusive use of a common resource.• Activities i, j are compatible iff non-overlapping:

• Objective:• select a maximum-sized set of mutually compatible activities

ii fs

)[ ii fs )[ jj fs

source: 91.404 textbook Cormen, et al.

Activity SelectionActivity Selection

1 2 3 4 5 6 7 8 9 10 1211 13 14 15 16

1

2

3

4

8

7

6

5

Activity Time Duration

Activity Number

What is an answer in this case?

Activity SelectionActivity Selection

}:{ jkkikij sfsfSaS Solution to Sij including ak produces 2 subproblems:1) Sik (start after ai finishes; finish before ak starts)2) Skj (start after ak finishes; finish before aj starts)

source: 91.404 textbook Cormen, et al.

0 if}1],[],[{max

0 if0],[

; ijSajki

ij

Sjkckic

Sjic

ijk

c[i,j]=size of maximum-size subset of mutually compatible activities in Sij.