COSC 3101A - Design and Analysis of Algorithms

9

Knapsack Problem

Huffman Codes

Introduction to Graphs

Many of these slides are taken from Monica Nicolescu, Univ. of Nevada, Reno, monica@cs.unr.edu

6/29/2004 Lecture 9 COSC3101A 2

The Knapsack Problem

• The 0-1 knapsack problem

– A thief rubbing a store finds n items: the i-th item is

worth vi dollars and weights wi pounds (vi, wi

integers)

– The thief can only carry W pounds in his knapsack

– Items must be taken entirely or left behind

– Which items should the thief take to maximize the

value of his load?

• The fractional knapsack problem

– Similar to above

– The thief can take fractions of items

6/29/2004 Lecture 9 COSC3101A 3

Fractional Knapsack Problem

• Knapsack capacity: W

• There are n items: the i-th item has value vi and

weight wi

• Goal:

– find xi such that for all 0 xi 1, i = 1, 2, .., n

wixi W and

xivi is maximum

6/29/2004 Lecture 9 COSC3101A 4

Fractional Knapsack Problem

• Greedy strategy 1:– Pick the item with the maximum value

• E.g.:– W = 1

– w1 = 100, v1 = 2

– w2 = 1, v2 = 1

– Taking from the item with the maximum value:

Total value taken = v1/w1 = 2/100

– Smaller than what the thief can take if choosing the other item

Total value (choose item 2) = v2/w2 = 1

6/29/2004 Lecture 9 COSC3101A 5

Fractional Knapsack Problem

Greedy strategy 2:

• Pick the item with the maximum value per pound vi/wi

• If the supply of that element is exhausted and the thief can

carry more: take as much as possible from the item with the

next greatest value per pound

• It is good to order items based on their value per pound

n

n

w

v

w

v

w

v ...

2

2

1

1

6/29/2004 Lecture 9 COSC3101A 6

Fractional Knapsack Problem

Alg.: Fractional-Knapsack (W, v[n], w[n])

1. While w > 0 and as long as there are items remaining

2. pick item with maximum vi/wi

3. xi min (1, w/wi)

4. remove item i from list

5. w w – xiwi

• w – the amount of space remaining in the knapsack (w = W)

• Running time: (n) if items already ordered; else (nlgn)

6/29/2004 Lecture 9 COSC3101A 7

50

Fractional Knapsack - Example

• E.g.:

1020

30

50

Item 1

Item 2

Item 3

$60 $100 $120

10

20

$60

$100

+

$240

$6/pound $5/pound $4/pound

20---30

$80

+

6/29/2004 Lecture 9 COSC3101A 8

Greedy Choice

Items: 1 2 3 … j … nOptimal solution: x1 x2 x3 xj xn

Greedy solution: x1’ x2’ x3’ xj’ xn’• We know that: x1’ x1

– greedy choice takes as much as possible from item 1

• Modify the optimal solution to take x1’ of item 1– We have to decrease the quantity taken from some item j: the new

xj is decreased by: (x1’ - x1) w1/wj

• Increase in profit:• Decrease in profit:

jj111111 /w v)wx - ’(x v)x - ’(x

j

j11 w

v w v

j

j

1

1

w

v

w

v True, since x1 had the

best value/pound ratio

v)x - ’(x 111

jj111 /w v)wx - ’(x

6/29/2004 Lecture 9 COSC3101A 9

Optimal Substructure

• Consider the most valuable load that weights at

most W pounds

• If we remove a weight w of item j from the

optimal load

The remaining load must be the most valuable

load weighing at most W – w that can be taken

from the remaining n – 1 items plus wj – w

pounds of item j

6/29/2004 Lecture 9 COSC3101A 10

The 0-1 Knapsack Problem

• Thief has a knapsack of capacity W

• There are n items: for i-th item value vi and

weight wi

• Goal:

– find xi such that for all xi = {0, 1}, i = 1, 2, .., n

wixi W and

xivi is maximum

6/29/2004 Lecture 9 COSC3101A 11

The 0-1 Knapsack Problem

• Thief has a knapsack of capacity W

• There are n items: for i-th item value vi and

weight wi

• Goal:

– find xi such that for all xi = {0, 1}, i = 1, 2, .., n

wixi W and

xivi is maximum

6/29/2004 Lecture 9 COSC3101A 12

50

0-1 Knapsack - Greedy Strategy

• E.g.:

1020

30

50

Item 1

Item 2

Item 3

$60 $100 $120

10

20

$60

$100

+

$160

50

20 $100

$120

+

$220

30

$6/pound $5/pound $4/pound

• None of the solutions involving the greedy choice (item 1) leads to an optimal solution– The greedy choice property does not hold

6/29/2004 Lecture 9 COSC3101A 13

0-1 Knapsack - Dynamic Programming

• P(i, w) – the maximum profit that can be

obtained from items 1 to i, if the

knapsack has size w

• Case 1: thief takes item i

P(i, w) =

• Case 2: thief does not take item i

P(i, w) =

vi + P(i - 1, w-wi)

P(i - 1, w)

6/29/2004 Lecture 9 COSC3101A 14

0-1 Knapsack - Dynamic Programming

0 0 0 0 0 0 0 0 0 0 0

0

0

0

0

0

0

0:

n

1 w - wi W

i-1

0

first

P(i, w) = max {vi + P(i - 1, w-wi), P(i - 1, w) }

Item i was taken Item i was not taken

i

w

second

6/29/2004 Lecture 9 COSC3101A 15

P(i, w) = max {vi + P(i - 1, w-wi), P(i - 1, w) }

0 0 0 0 0 0

0

0

0

0

Item Weight Value

1 2 12

2 1 10

3 3 20

4 2 150 1 2 3 4 5

1

2

3

4

W = 5

0

12 12 12 12

10 12 22 22 22

10 12 22 30 32

10 15 25 30 37

P(1, 1) =

P(1, 2) =

P(1, 3) =

P(1, 4) =

P(1, 5) =

P(2, 1)=

P(2, 2)=

P(2, 3)=

P(2, 4)=

P(2, 5)=

P(3, 1)=

P(3, 2)=

P(3, 3)=

P(3, 4)=

P(4, 5)=

P(4, 1)=

P(4, 2)=

P(4, 3)=

P(4, 4)=

P(4, 5)=

max{12+0, 0} = 12

max{12+0, 0} = 12

max{12+0, 0} = 12

max{12+0, 0} = 12

max{10+0, 0} = 10

max{10+0, 12} = 12

max{10+12, 12} = 22

max{10+12, 12} = 22

max{10+12, 12} = 22

P(2,1) = 10

P(2,2) = 12

max{20+0, 22}=22

max{20+10,22}=30

max{20+12,22}=32

P(3,1) = 10

max{15+0, 12} = 15

max{15+10, 22}=25

max{15+12, 30}=30

max{15+22, 32}=37

0

P(0, 1) = 0

Example:

6/29/2004 Lecture 9 COSC3101A 16

Reconstructing the Optimal Solution

0 0 0 0 0 0

0

0

0

0

0 1 2 3 4 5

1

2

3

4

0

12 12 12 12

10 12 22 22 22

10 12 22 30 32

10 15 25 30 37

0

• Start at P(n, W)• When you go left-up item i has been taken• When you go straight up item i has not been

taken

• Item 4

• Item 2

• Item 1

6/29/2004 Lecture 9 COSC3101A 17

Optimal Substructure

• Consider the most valuable load that weights at

most W pounds

• If we remove item j from this load

The remaining load must be the most valuable

load weighing at most W – wj that can be taken

from the remaining n – 1 items

6/29/2004 Lecture 9 COSC3101A 18

Overlapping Subproblems

0 0 0 0 0 0 0 0 0 0 0

0

0

0

0

0

0

0:

n

1 W

i-1

0

P(i, w) = max {vi + P(i - 1, w-wi), P(i - 1, w) }

i

w

E.g.: all the subproblems shown in grey may depend on P(i-1, w)

6/29/2004 Lecture 9 COSC3101A 19

Huffman Codes

• Widely used technique for data compression

• Assume the data to be a sequence of characters

• Looking for an effective way of storing the data

6/29/2004 Lecture 9 COSC3101A 20

Huffman Codes

• Idea:

– Use the frequencies of occurrence of characters to

build a optimal way of representing each character

• Binary character code– Uniquely represents a character by a binary string

a b c d e f

Frequency (thousands) 45 13 12 16 9 5

6/29/2004 Lecture 9 COSC3101A 21

Fixed-Length Codes

E.g.: Data file containing 100,000 characters

• 3 bits needed

• a = 000, b = 001, c = 010, d = 011, e = 100, f = 101

• Requires: 100,000 3 = 300,000 bits

a b c d e f

Frequency (thousands) 45 13 12 16 9 5

6/29/2004 Lecture 9 COSC3101A 22

Variable-Length Codes

E.g.: Data file containing 100,000 characters

• Assign short codewords to frequent characters and long codewords to infrequent characters

• a = 0, b = 101, c = 100, d = 111, e = 1101, f = 1100• (45 1 + 13 3 + 12 3 + 16 3 + 9 4 + 5 4)

1,000

= 224,000 bits

a b c d e f

Frequency (thousands) 45 13 12 16 9 5

6/29/2004 Lecture 9 COSC3101A 23

Prefix Codes

• Prefix codes:

– Codes for which no codeword is also a prefix of some

other codeword

– Better name would be “prefix-free codes”

• We can achieve optimal data compression using

prefix codes

– We will restrict our attention to prefix codes

6/29/2004 Lecture 9 COSC3101A 24

Encoding with Binary Character Codes

• Encoding

– Concatenate the codewords representing each

character in the file

• E.g.:

– a = 0, b = 101, c = 100, d = 111, e = 1101, f = 1100

– abc = 0 101 100 = 0101100

6/29/2004 Lecture 9 COSC3101A 25

Decoding with Binary Character Codes

• Prefix codes simplify decoding– No codeword is a prefix of another the codeword

that begins an encoded file is unambiguous

• Approach– Identify the initial codeword– Translate it back to the original character– Repeat the process on the remainder of the file

• E.g.:

– a = 0, b = 101, c = 100, d = 111, e = 1101, f = 1100– 001011101 = 0 0 101 1101 = aabe

6/29/2004 Lecture 9 COSC3101A 26

Prefix Code Representation• Binary tree whose leaves are the given characters• Binary codeword

– the path from the root to the character, where 0 means “go to the left child” and 1 means “go to the right child”

• Length of the codeword – Length of the path from root to the character leaf (depth of node)

100

86 14

58 28 14

a: 45 b: 13 c: 12 d: 16 e: 9 f: 5

0

0

0

1

1 1

1

1

0

0 0

100

a: 45

0

55

1

25 30

0 1

c: 12 b: 13

10

14

f: 5 e: 9

10

d: 16

10

6/29/2004 Lecture 9 COSC3101A 27

Optimal Codes

• An optimal code is always represented by a full binary tree– Every non-leaf has two children– Fixed-length code is not optimal, variable-length is

• How many bits are required to encode a file?– Let C be the alphabet of characters– Let f(c) be the frequency of character c

– Let dT(c) be the depth of c’s leaf in the tree T corresponding to a prefix code

Cc

T cdcfTB )()()( the cost of tree T

6/29/2004 Lecture 9 COSC3101A 28

Constructing a Huffman Code

• A greedy algorithm that constructs an optimal prefix code

called a Huffman code

• Assume that:– C is a set of n characters

– Each character has a frequency f(c)

– The tree T is built in a bottom up manner

• Idea:– Start with a set of |C| leaves

– At each step, merge the two least frequent objects: the frequency of

the new node = sum of two frequencies

– Use a min-priority queue Q, keyed on f to identify the two least

frequent objects

a: 45c: 12 b: 13f: 5 e: 9 d: 16

6/29/2004 Lecture 9 COSC3101A 29

Example

a: 45c: 12 b: 13f: 5 e: 9 d: 16 a: 45c: 12 b: 13 d: 1614

f: 5 e: 9

0 1

d: 16

c: 12 b: 13

25 a: 45

f: 5 e: 9

140 01 1

f: 5 e: 9

14c: 12 b: 13

25

d: 16

30 a: 450 0

0

1 1

1

a: 45

f: 5 e: 9

14c: 12 b: 13

25

d: 16

30

550

0 0

0

1

11

1

f: 5 e: 9

14c: 12 b: 13

25

d: 16

30

55a: 45

1000

0

0 0

0

1

1

11

1

6/29/2004 Lecture 9 COSC3101A 30

Building a Huffman Code

Alg.: HUFFMAN(C)

1. n C 2. Q C3. for i 1 to n – 1

4. do allocate a new node z5. left[z] x EXTRACT-MIN(Q)

6. right[z] y EXTRACT-MIN(Q)

7. f[z] f[x] + f[y]8. INSERT (Q, z)

9. return EXTRACT-MIN(Q)

O(n)

O(nlgn)

Running time: O(nlgn)

6/29/2004 Lecture 9 COSC3101A 31

Greedy Choice Property

Lemma: Let C be an alphabet in which each

character c C has frequency f[c]. Let x and y

be two characters in C having the lowest

frequencies.

Then, there exists an optimal prefix code for C in

which the codewords for x and y have the same

length and differ only in the last bit.

6/29/2004 Lecture 9 COSC3101A 32

Proof of the Greedy Choice

• Idea:

– Consider a tree T representing an arbitrary optimal

prefix code

– Modify T to make a tree representing another optimal

prefix code in which x and y will appear as sibling

leaves of maximum depth

The codes of x and y will have the same length and

differ only in the last bit

6/29/2004 Lecture 9 COSC3101A 33

Proof of the Greedy Choice (cont.)

• a, b – two characters, sibling leaves of maximum depth in T

• Assume: f[a] f[b] and f[x] f[y]

• f[x] and f[y] are the two lowest leaf frequencies, in order

f[x] f[a] and f[y] f[b]

• Exchange the positions of a and x (T’) and of b and y (T’’)

x

y

ba

a

y

bx

a

b

yx

T T’ T’’

6/29/2004 Lecture 9 COSC3101A 34

Proof of the Greedy Choice (cont.)

B(T) – B(T’) =

= f[x]dT(x) + f[a]dT(a) – f[x]dT’(x) – f[a]dT’(a)

= f[x]dT(x) + f[a]dT(a) – f[x]dT(a) – f[a]dT (x)

= (f[a] - f[x]) (dT(a) - dT(x))

0

x

y

ba

a

y

bx

a

b

yx

T T’ T’’

Cc Cc

TT cdcfcdcf )()()()( '

≥ 0a is a leaf of maximum depth

≥ 0x is a minimum frequency leaf

6/29/2004 Lecture 9 COSC3101A 35

Proof of the Greedy Choice (cont.)

B(T) – B(T’) 0

Similarly, exchanging y and b does not increase the cost

B(T’) – B(T’’) 0

B(T’’) B(T) and since T is optimal B(T) B(T’’)

B(T) = B(T’’) T’’ is an optimal tree, in which x and y are

sibling leaves of maximum depth

x

y

ba

a

y

bx

a

b

yx

T T’ T’’

6/29/2004 Lecture 9 COSC3101A 36

Discussion

• Greedy choice property:

– Building an optimal tree by mergers can begin with

the greedy choice: merging the two characters with

the lowest frequencies

– The cost of each merger is the sum of frequencies of

the two items being merged

– Of all possible mergers, HUFFMAN chooses the one

that incurs the least cost

6/29/2004 Lecture 9 COSC3101A 37

Graphs

• Applications that involve not only a set of items, but also the connections between them

• Maps• Hypertexts• Circuits• Schedules• Transactions• Matching• Computer Networks

6/29/2004 Lecture 9 COSC3101A 38

Graphs - Background

Graphs = a set of nodes (vertices) with edges (links) between them.

Notations:• G = (V, E) - graph• V = set of vertices V = n• E = set of edges E = m

1 2

3 4

1 2

3 4

Directedgraph

Undirectedgraph

1 2

3 4

Acyclicgraph

6/29/2004 Lecture 9 COSC3101A 39

Other Types of Graphs

• A graph is connected if there is

a path between every two

vertices

• A bipartite graph is an

undirected graph G = (V, E) in

which V = V1 + V2 and there are

edges only between vertices in

V1 and V2

1 2

3 4

Connected

1 2

3 4

Not connected

1 2

3

4

49

76

8

6/29/2004 Lecture 9 COSC3101A 40

Graph Representation

• Adjacency list representation of G = (V, E)– An array of V lists, one for each vertex in V– Each list Adj[u] contains all the vertices v such that

there is an edge between u and v• Adj[u] contains the vertices adjacent to u (in arbitrary order)

– Can be used for both directed and undirected graphs

1 2

5 4

3

2 5 /

1 5 3 4 /

1

2

3

4

5

2 4

2 5 3 /

4 1 2

Undirected graph

6/29/2004 Lecture 9 COSC3101A 41

Properties of Adjacency-List Representation

• Sum of the lengths of all the

adjacency lists

– Directed graph:

• Edge (u, v) appears only once in u’s list

– Undirected graph:

• u and v appear in each other’s adjacency

lists: edge (u, v) appears twice

1 2

5 4

3

Undirected graph

1 2

3 4

Directed graphE

2 E

6/29/2004 Lecture 9 COSC3101A 42

Properties of Adjacency-List Representation

• Memory required

(V + E)

• Preferred when

– the graph is sparse: E << V 2

• Disadvantage

– no quick way to determine whether there

is an edge between node u and v

• Time to list all vertices adjacent to u:

(degree(u))

• Time to determine if (u, v) E:

– O(degree(u))

1 2

5 4

3

Undirected graph

1 2

3 4

Directed graph

6/29/2004 Lecture 9 COSC3101A 43

Graph Representation

• Adjacency matrix representation of G = (V, E)– Assume vertices are numbered 1, 2, … V – The representation consists of a matrix A V x V :

– aij = 1 if (i, j) E

0 otherwise

1 2

5 4

3

Undirected graph

1

2

3

4

5

1 2 3 4 5

0 1 10 0

1 1 1 10

1 10 0 0

1 1 10 0

1 1 10 0

Matrix A is

symmetric:

aij = aji

A = AT

6/29/2004 Lecture 9 COSC3101A 44

Properties of Adjacency Matrix Representation

• Memory required (V2), independent on the number of edges in G

• Preferred when– The graph is dense E is close to V 2

– We need to quickly determine if there is an edge between two vertices

• Time to list all vertices adjacent to u: (V)

• Time to determine if (u, v) E: (1)

6/29/2004 Lecture 9 COSC3101A 45

Weighted Graphs

• Weighted graphs = graphs for which each edge

has an associated weight w(u, v)

w: E R, weight function

• Storing the weights of a graph

– Adjacency list:

• Store w(u,v) along with vertex v in u’s adjacency list

– Adjacency matrix:

• Store w(u, v) at location (u, v) in the matrix

6/29/2004 Lecture 9 COSC3101A 46

Searching in a Graph

• Graph searching = systematically follow the edges of the graph so as to visit the vertices of the graph

• Two basic graph searching algorithms:– Breadth-first search

– Depth-first search

– The difference between them is in the order in which they explore the unvisited edges of the graph

• Graph algorithms are typically elaborations of the basic graph-searching algorithms

6/29/2004 Lecture 9 COSC3101A 47

Breadth-First Search (BFS)

• Input:– A graph G = (V, E) (directed or undirected)– A source vertex s V

• Goal:– Explore the edges of G to “discover” every vertex

reachable from s, taking the ones closest to s first

• Output:– d[v] = distance (smallest # of edges) from s to v, for

all v V– A “breadth-first tree” rooted at s that contains all

reachable vertices

6/29/2004 Lecture 9 COSC3101A 48

Breadth-First Search (cont.)

• Discover vertices in increasing order of distance from the source s – search in breadth not depth– Find all vertices at 1 edge from s, then all vertices at 2

edges from s, and so on

1 2

5 4

3

7

7

9

11

12

6

6/29/2004 Lecture 9 COSC3101A 49

Breadth-First Search (cont.)

• Keeping track of progress:

– Color each vertex in either white,

gray or black

– Initially, all vertices are white

– When being discovered a vertex

becomes gray

– After discovering all its adjacent

vertices the node becomes black

– Use FIFO queue Q to maintain the

set of gray vertices

1 2

5 4

3

1 2

5 4

3

source

1 2

5 4

3

6/29/2004 Lecture 9 COSC3101A 50

Breadth-First Tree

• BFS constructs a breadth-first tree

– Initially contains the root (source vertex s)

– When vertex v is discovered while scanning

the adjacency list of a vertex u vertex v

and edge (u, v) are added to the tree

– u is the predecessor (parent) of v in the

breadth-first tree

– A vertex is discovered only once it has at

most one parent

1 2

5 4

3

source

6/29/2004 Lecture 9 COSC3101A 51

BFS Additional Data Structures

• G = (V, E) represented using adjacency lists

• color[u] – the color of the vertex for all u

V

[u] – predecessor of u

– If u = s (root) or node u has not yet been

discovered [u] = NIL

• d[u] – the distance from the source s to

vertex u

• Use a FIFO queue Q to maintain the set of

gray vertices

1 2

5 4

3

d=1=1

d=1=1

d=2=5

d=2=2

source

6/29/2004 Lecture 9 COSC3101A 52

BFS(G, s)

1. for each u V[G] - {s}

2. do color[u] WHITE

3. d[u] ←

4. [u] = NIL

5. color[s] GRAY

6. d[s] ← 0

7. [s] = NIL

8. Q

9. Q ← ENQUEUE(Q, s) Q: s

0

r s t u

v w x y

r s t u

v w x y

r s t u

v w x y

6/29/2004 Lecture 9 COSC3101A 53

BFS(V, E, s)

10. while Q

11. do u ← DEQUEUE(Q)

12. for each v Adj[u]

13. do if color[v] = WHITE

14. then color[v] ← GRAY

15. d[v] ← d[u] + 1

16. [v] = u

17. ENQUEUE(Q, v)

18. color[u] BLACK

0

1

r s t u

v w x y

Q: w

Q: s 0

r s t u

v w x y

1 0

1

r s t u

v w x y

Q: w, r

6/29/2004 Lecture 9 COSC3101A 54

Example

1 0

1

r s t u

v w x yQ: s

0

r s t u

v w x yQ: w, r

v w x y

1 0 2

1 2

r s t u

Q: r, t, x

1 0 2

2 1 2

r s t u

v w x yQ: t, x, v

1 0 2 3

2 1 2

r s t u

v w x yQ: x, v, u

1 0 2 3

2 1 2 3

r s t u

v w x yQ: v, u, y

1 0 2 3

2 1 2 3

r s t u

v w x yQ: u, y

1 0 2 3

2 1 2 3

r s t u

v w x yQ: y

r s t u1 0 2 3

2 1 2 3

v w x yQ:

6/29/2004 Lecture 9 COSC3101A 55

Analysis of BFS

1. for each u V - {s}

2. do color[u] WHITE

3. d[u] ←

4. [u] = NIL

5. color[s] GRAY

6. d[s] ← 0

7. [s] = NIL

8. Q

9. Q ← ENQUEUE(Q, s)

O(V)

(1)

6/29/2004 Lecture 9 COSC3101A 56

Analysis of BFS

10. while Q

11. do u ← DEQUEUE(Q)

12. for each v Adj[u]

13. do if color[v] = WHITE

14. then color[v] = GRAY

15. d[v] ← d[u] + 1

16. [v] = u

17. ENQUEUE(Q, v)

18. color[u] BLACK

(1)

(1)

Scan Adj[u] for all vertices in the graph• Each vertex is scanned only once, when the vertex is dequeued

• Sum of lengths of all adjacency lists = (E)• Scanning operations: O(E)

• Total running time for BFS = O(V + E)

6/29/2004 Lecture 9 COSC3101A 57

Shortest Paths Property

• BFS finds the shortest-path distance from the source vertex s V to each node in the graph

• Shortest-path distance = (s, u)– Minimum number of edges in any path from s to u

r s t u

1 0 2 3

2 1 2 3

v w x y

source

6/29/2004 Lecture 9 COSC3101A 58

Readings

• Chapter 16• Chapter 22