Download - Design and Analysis of Algorithms CSE 350
![Page 1: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/1.jpg)
Design and Analysis of Algorithms
CSE 350
Unit 3
![Page 2: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/2.jpg)
Syllabus
Unit 3 Dynamic Programming
A Overview, Difference between dynamic programming and divide and
conquer, All pair shortest path problems: Floyd-Warshall Algorithm
B Applications and analysis: Matrix Chain Multiplication, 0/1 Knapsack
Problem
C Applications and analysis: Longest Common sub-sequence, Optimal
Binary Search tree
![Page 3: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/3.jpg)
Dynamic Programming
Dynamic programming (usually referred to as DP ) is a very powerful technique to solve a
particular class of problems.
The idea is very simple, If you have solved a problem with the given input, then save the result for
future reference, so as to avoid solving the same problem again.. shortly 'Remember your Past' :) .
If the given problem can be broken up in to smaller sub-problems and these smaller subproblems
are in turn divided in to still-smaller ones, and in this process, if you observe some over-lapping
subproblems, then its a big hint for dynamic programming.
Also, the optimal solutions to the subproblems contribute to the optimal solution of the given
problem.
![Page 4: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/4.jpg)
Dynamic Programming is an algorithmic paradigm that solves a given complex problem by
breaking it into subproblems and stores the results of subproblems to avoid computing the same
results again.
Following are the two main properties of a problem that suggests that the given problem can be
solved using Dynamic programming.
1) Overlapping Subproblems
2) Optimal Substructure
![Page 5: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/5.jpg)
1. Overlapping subproblems
Like Divide and Conquer, Dynamic Programming combines solutions to sub-problems.
Dynamic Programming is mainly used when solutions of same subproblems are needed again and again.
In dynamic programming, computed solutions to subproblems are stored in a table so that these don’t
have to be recomputed.
So Dynamic Programming is not useful when there are no common (overlapping) subproblems because
there is no point storing the solutions if they are not needed again.
![Page 6: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/6.jpg)
For example, Binary Search doesn’t have common subproblems.
If we take an example of following recursive program for Fibonacci Numbers, there are many subproblems
which are solved again and again.
![Page 7: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/7.jpg)
![Page 8: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/8.jpg)
We can see that the function fib(3) is being called 2 times. If we would have stored the value of
fib(3), then instead of computing it again, we could have reused the old stored value.
There are following two different ways to store the values so that these values can be reused:
a) Memoization (Top Down)
b) Tabulation (Bottom Up)
![Page 9: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/9.jpg)
a) Memoization (Top Down):
The memoized program for a problem is similar to the recursive version with a small modification that
it looks into a lookup table before computing solutions.
We initialize a lookup array with all initial values as NIL.
Whenever we need the solution to a subproblem, we first look into the lookup table.
If the precomputed value is there then we return that value, otherwise, we calculate the value and put
the result in the lookup table so that it can be reused later.
![Page 10: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/10.jpg)
This is a variation of dynamic programming that often offers the efficiency of the usual
dynamic-programming approach while maintaining a top-down strategy.
The idea is to memoize the natural, but inefficient, recursive algorithm.
As in ordinary dynamic programming, we maintain a table with subproblem solutions, but the
control structure for filling in the table is more like the recursive algorithm.
![Page 11: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/11.jpg)
A memoized recursive algorithm maintains an entry in a table for the solution to each subproblem.
Each table entry initially contains a special value to indicate that the entry has yet to be filled in.
When the subproblem is first encountered during the execution of the recursive algorithm, its solution is
computed and then stored in the table.
Each subsequent time that the subproblem is encountered, the value stored in the table is simply looked
up and returned.
![Page 12: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/12.jpg)
![Page 13: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/13.jpg)
MEMOIZED-MATRIX-CHAIN, like MATRIX-CHAIN-ORDER, maintains a table m[1 . . n, 1 . .
n] of computed values of m[i, j].
Each table entry initially contains the value ∞ to indicate that the entry has yet to be filled in.
When the call LOOKUP-CHAIN(p,i, j) is executed, if m[i, j] < ∞ in line 1, the procedure simply
returns the previously computed cost m[i, j] (line 2).
Otherwise, the cost is computed as in RECURSIVE-MATRIX-CHAIN, stored in m[i, j], and
returned.
LOOKUP-CHAIN(p,i, j) always returns the value of m[i, j], but it computes it only if this is the
first time that LOOKUP-CHAIN has been called with the parameters i and j.
![Page 14: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/14.jpg)
b) Tabulation (Bottom Up):
The tabulated program for a given problem builds a table in bottom up fashion and returns
the last entry from table.
For example, for the same Fibonacci number, we first calculate fib(0) then fib(1) then
fib(2) then fib(3) and so on. So literally, we are building the solutions of subproblems
bottom-up.
![Page 15: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/15.jpg)
Both Tabulated and Memoized store the solutions of subproblems.
In Memoized version, table is filled on demand while in Tabulated version, starting from the
first entry, all entries are filled one by one.
Unlike the Tabulated version, all entries of the lookup table are not necessarily filled in
Memoized version.
![Page 16: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/16.jpg)
2. Optimal Substructure
A given problems has Optimal Substructure Property if optimal solution of the given problem can be
obtained by using optimal solutions of its subproblems.
For example, the Shortest Path problem has following optimal substructure property:
If a node x lies in the shortest path from a source node u to destination node v then the shortest path
from u to v is combination of shortest path from u to x and shortest path from x to v.
The standard All Pair Shortest Path algorithm like Floyd–Warshall and Single Source Shortest path
algorithm for negative weight edges like Bellman–Ford are typical examples of Dynamic
Programming.
![Page 17: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/17.jpg)
On the other hand, the Longest Path problem doesn’t have the
Optimal Substructure property.
Here by Longest Path we mean longest simple path (path without
cycle) between two nodes.
Consider the following unweighted graph.
There are two longest paths from q to t: q→r→t and q→s→t.
Unlike shortest paths, these longest paths do not have the optimal
substructure property.
For example, the longest path q→r→t is not a combination of
longest path from q to r and longest path from r to t, because the
longest path from q to r is q→s→t→r and the longest path from r
to t is r→q→s→t.
![Page 18: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/18.jpg)
The development of a dynamic-programming algorithm can be broken into a sequence of four steps.
1. Characterize the structure of an optimal solution.
2. Recursively define the value of an optimal solution.
3. Compute the value of an optimal solution in a bottom-up fashion.
4. Construct an optimal solution from computed information.
Steps 1–3 form the basis of a dynamic-programming solution to a problem.
Step 4 can be omitted if only the value of an optimal solution is required.
![Page 19: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/19.jpg)
Divide and Conquer Method vs Dynamic Programming
Divide and Conquer Method Dynamic Programming
1.It deals (involves) three steps at each level of recursion:
Divide the problem into a number of subproblems.
Conquer the subproblems by solving them recursively.
Combine the solution to the subproblems into the solution for
original subproblems.
•1.It involves the sequence of four steps:Characterize the structure
of optimal solutions.
•Recursively defines the values of optimal solutions.
•Compute the value of optimal solutions in a Bottom-up minimum.
•Construct an Optimal Solution from computed information.
2. It is Recursive. 2. It is non Recursive.
3. It does more work on subproblems and hence has more time
consumption.
3. It solves subproblems only once and then stores in the table.
4. It is a top-down approach. 4. It is a Bottom-up approach.
5. In this subproblems are independent of each other. 5. In this subproblems are interdependent.
6. For example: Merge Sort & Binary Search etc. 6. For example: Matrix Multiplication.
![Page 20: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/20.jpg)
All pair shortest path problems: Floyd-Warshall Algorithm
The Floyd-Warshall algorithm is a shortest path algorithm for graphs.
Like the Bellman-Ford algorithm or the Dijkstra's algorithm, it computes the shortest path in a
graph.
However, Bellman-Ford and Dijkstra are both single-source, shortest-path algorithms. This means
they only compute the shortest path from a single source.
Floyd-Warshall, on the other hand, computes the shortest distances between every pair of vertices in
the input graph.
![Page 21: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/21.jpg)
Imagine that you have 5 friends: Billy, Jenna, Cassie, Alyssa, and Harry.
You know a few roads that connect some of their houses, and you know the lengths of those roads.
But, Floyd-Warshall can take what you know and give you the optimal route given that information.
For example, look at the graph below, it shows paths from one friend to another with corresponding
distances.
![Page 22: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/22.jpg)
Floyd-Warshall will tell the optimal distance between each pair of friends.
It will clearly tell you that the quickest path from Alyssa's house to Harry's house is the connecting
edge that has a weight of 1.
But, it will also tell you that the quickest way to get from Billy's house to Jenna's house is to first go
through Cassie's, then Alyssa's, then Harry's house before ending at Jenna's. This is the power of
Floyd-Warshall; no matter what house you're currently in, it will tell the fastest way to get to every
other house.
![Page 23: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/23.jpg)
The Floyd-Warshall algorithm is an example of dynamic programming.
It breaks the problem down into smaller subproblems, then combines the answers to those
subproblems to solve the big, initial problem.
The idea is this: either the quickest path from A to C is the quickest path found so far from A to C, or
it's the quickest path from A to B plus the quickest path from B to C.
Floyd-Warshall is extremely useful in networking, similar to solutions to the shortest path problem.
However, it is more effective at managing multiple stops on the route because it can calculate the
shortest paths between all relevant nodes.
In fact, one run of Floyd-Warshall can give you all the information you need to know about a static
network to optimize most types of paths. It is also useful in computing matrix inversions.
![Page 24: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/24.jpg)
We initialize the solution matrix same as the input graph matrix as a first step.
Then we update the solution matrix by considering all vertices as an intermediate vertex.
The idea is to one by one pick all vertices and updates all shortest paths which include the picked
vertex as an intermediate vertex in the shortest path.
When we pick vertex number k as an intermediate vertex, we already have considered vertices {0, 1,
2, .. k-1} as intermediate vertices.
![Page 25: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/25.jpg)
For every pair (i, j) of the source and destination vertices respectively, there are two possible cases.
1) k is not an intermediate vertex in shortest path from i to j. We keep the value of dist[i][j] as it is.
2) k is an intermediate vertex in shortest path from i to j. We update the value of dist[i][j] as
dist[i][k] + dist[k][j] if dist[i][j] > dist[i][k] + dist[k][j]
The following figure shows the above optimal substructure property in the all-pairs shortest path
problem.
![Page 26: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/26.jpg)
![Page 27: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/27.jpg)
Numerical and Gate Questions (PDF 5)
![Page 28: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/28.jpg)
Matrix-chain Multiplication
Suppose we have a sequence or chain A1, A2, …, An of n matrices to be multiplied
That is, we want to compute the product A1 A2 … An
There are many possible ways (parenthesizations) to compute the product
![Page 29: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/29.jpg)
Example: Consider the chain A1, A2, A3, A4 of 4 matrices
Let us compute the product A1A2A3A4
There are 5 possible ways:
1. (A1(A2(A3A4)))
2. (A1((A2A3)A4))
3. ((A1A2)(A3A4))
4. ((A1(A2A3))A4)
5. (((A1A2)A3)A4)
![Page 30: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/30.jpg)
To compute the number of scalar multiplications necessary, we must know:
Algorithm to multiply two matrices
Matrix dimensions
![Page 31: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/31.jpg)
Algorithm to Multiply 2 Matrices
Input: Matrices Ap×q and Bq×r (with dimensions p×q and q×r)
Result: Matrix Cp×r resulting from the product A·B
MATRIX-MULTIPLY(Ap×q , Bq×r)
1. for i ← 1 to p
2. for j ← 1 to r
3. C[i, j] ← 0
4. for k ← 1 to q
5. C[i, j] ← C[i, j] + A[i, k] · B[k, j]
6. return C
Scalar multiplication in line 5 dominates time to compute Number of scalar multiplications = pqr
![Page 32: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/32.jpg)
Example: Consider three matrices A10100, B1005, and C550
There are 2 ways to parenthesize
((AB)C) = D105 · C550
AB 10·100·5=5,000 scalar multiplications
DC 10·5·50 =2,500 scalar multiplications
(A(BC)) = A10100 · E10050
BC 100·5·50=25,000 scalar multiplications
AE 10·100·50 =50,000 scalar multiplications
Total:
7,500
Total:
75,000
![Page 33: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/33.jpg)
Matrix-chain multiplication problem:
Given a chain A1, A2, …, An of n matrices, where for i=1, 2, …, n,
matrix Ai has dimension pi-1pi
Parenthesize the product A1A2…An such that the total number of
scalar multiplications is minimized
Brute force method of exhaustive search takes time exponential in n
![Page 34: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/34.jpg)
Dynamic Programming Approach:
Let us use the notation Ai..j for the matrix that results from the product Ai Ai+1 … Aj
An optimal parenthesization of the product A1A2…An splits the product between Ak and
Ak+1 for some integer k where1 ≤ k < n
First compute matrices A1..k and Ak+1..n ; then multiply them to get the final matrix A1..n
![Page 35: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/35.jpg)
Recursive definition of the value of an optimal solution
Let m[i, j] be the minimum number of scalar multiplications necessary to compute Ai..j
Minimum cost to compute A1..n is m[1, n]
Suppose the optimal parenthesization of Ai..j splits the product between Ak and Ak+1 for some
integer k where i ≤ k < j
![Page 36: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/36.jpg)
Ai..j = (Ai Ai+1…Ak)·(Ak+1Ak+2…Aj)= Ai..k · Ak+1..j
Cost of computing Ai..j = cost of computing Ai..k + cost of computing Ak+1..j + cost of
multiplying Ai..k and Ak+1..j
Cost of multiplying Ai..k and Ak+1..j is pi-1pk pj
m[i, j ] = m[i, k] + m[k+1, j ] + pi-1pk pj for i ≤ k < j
m[i, i ] = 0 for i=1,2,…,n
![Page 37: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/37.jpg)
But… optimal parenthesization occurs at one value of k among all
possible i ≤ k < j
Check all these and select the best one
m[i, j ] =0 if i=j
min {m[i, k] + m[k+1, j ] + pi-1pk pj } ifi<j
i ≤ k< j
![Page 38: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/38.jpg)
To keep track of how to construct an optimal solution, we use a table s
s[i, j ] = value of k at which Ai Ai+1 … Aj is split for optimal parenthesization
Algorithm: next slide
First computes costs for chains of length l=1
Then for chains of length l=2,3, … and so on
Computes the optimal cost bottom-up
![Page 39: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/39.jpg)
Algorithm to Compute Optimal Cost:
Input: Array p[0…n] containing matrix dimensions and n
Result: Minimum-cost table m and split table s
MATRIX-CHAIN-ORDER(p[ ], n)
for i ← 1 to n
m[i, i] ← 0
for l ← 2 to n
for i ← 1 to n-l+1
j ← i+l-1
m[i, j] ←
for k ← i to j-1
q ← m[i, k] + m[k+1, j] + p[i-1] p[k] p[j]
if q < m[i, j]
m[i, j] ← q
s[i, j] ← k
return m and s
Takes O(n3) time
Requires O(n2) space
![Page 40: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/40.jpg)
Our algorithm computes the minimum-cost table m and the split table s
The optimal solution can be constructed from the split table s
Each entry s[i, j ]=k shows where to split the product Ai Ai+1 … Aj for the minimum cost
![Page 41: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/41.jpg)
Numerical and Gate Questions (PDF 5)
![Page 42: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/42.jpg)
0-1 Knapsack problem
Given a knapsack with maximum capacity W, and a set S consisting of n items
Each item i has some weight wi and benefit value bi (all wi , bi and W are integer values)
Problem:
How to pack the knapsack to achieve maximum total value of packed items?
![Page 43: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/43.jpg)
W = 20
w
i
bi
109
85
54
43
32
Weight Benefit value
This is a knapsack
Max weight: W = 20
Items
![Page 44: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/44.jpg)
The problem is called a “0-1” problem, because each item must be entirely accepted
or rejected.
Just another version of this problem is the “Fractional Knapsack Problem”, where
we can take fractions of items.
![Page 45: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/45.jpg)
0-1 Knapsack problem: brute-force approach
Let’s first solve this problem with a straightforward algorithm
Since there are n items, there are 2n possible combinations of items. (From Set Theory)
We go through all combinations and find the one with the most total value and with total
weight less or equal to W
Running time will be O(2n)
Can we do better?
Yes, with an algorithm based on dynamic programming
![Page 46: Design and Analysis of Algorithms CSE 350](https://reader034.vdocuments.net/reader034/viewer/2022042307/625b56a876753158db189977/html5/thumbnails/46.jpg)
Numerical and Gate Questions (PDF 5)