9/10/2015

1

4.4 The recursion-tree method

• Let us see how a recursion tree wouldprovide a good guess for the recurrence

= 3 4 )• Start by nding an upper bound• Floors and ceilings usually do not matter

when solving recurrences• Create a recursion tree for the recurrence

= 3 4 having written out theimplied constant coef cient > 0

10-Sep-15MAT-72006 AADS, Fall 2015 128

10-Sep-15MAT-72006 AADS, Fall 2015 129

9/10/2015

2

10-Sep-15MAT-72006 AADS, Fall 2015 130

• Subproblem sizes decrease by a factor of 4each time we go down one level =>– we eventually must reach a boundary

condition• Subproblem size for a node at depth is /4

– It hits = 1 when /4 = 1 or, equivalently,when = log

• Thus, the tree has log + 1 levels (at depths0, 1, 2, … , log )

10-Sep-15MAT-72006 AADS, Fall 2015 131

9/10/2015

3

• Each level has 3 × the nodes of level above– the number of nodes at depth is 3

• Subproblem sizes reduce by a factor of 4 foreach level we go down, each node at depth ,

= 0,1, 2, … , log 1, has cost of 4 )• Multiplying, we see that the total cost over all

nodes at depth is 3 4 ) = (3 16)• The bottom level, at depth log , has 3 =

nodes, each contributing cost (1), for atotal cost of (1), which is ), since

(1) is a constant

10-Sep-15MAT-72006 AADS, Fall 2015 132

• Add up the costs over all levels to determinethe cost for the entire tree:

=3

16 )

=(3 16) 1

(3 16) 1 )

By the value of a geometric (or exponential)series

=1

110-Sep-15MAT-72006 AADS, Fall 2015 133

9/10/2015

4

• We can use an in nite decreasing geometricseries as an upper bound

=3

16 )

316

=1

(3 16)

=1613 )

10-Sep-15MAT-72006 AADS, Fall 2015 134

• We have derived a guess of forour recurrence = 3 4 )

• Root of the tree contributes to the totalcost – a constant fraction of the total cost– The cost of the root dominates the total cost

• If is indeed an upper bound for therecurrence, then it must be a tight bound

• First recursive call contributes cost of ),must be a lower bound for the

recurrence10-Sep-15MAT-72006 AADS, Fall 2015 135

9/10/2015

5

• Use the substitution method to verify that theguess was correct, i.e., is anupper bound for the recurrence

• We want to show that for someconstant > 0

= 3 444)

=3

16

as long as (16 13)10-Sep-15MAT-72006 AADS, Fall 2015 136

• A more intricate example is the recurrence) = /3) + (2 /3) + )

• represents the constant in the term• Adding the values across the levels of the

recursion tree yields value for every level• The longest simple path from the root to a

leaf is (2 3 (2 3) 1• (2 3) = 1 when = log• The height of the tree is log

10-Sep-15MAT-72006 AADS, Fall 2015 137

9/10/2015

6

10-Sep-15MAT-72006 AADS, Fall 2015 138

• We expect the solution to the recurrence tobe number of levels × cost of each level:

log ) = lg )• Not every level contributes a cost of• If this recursion tree were a complete binary

tree of height log , there would be2 leaves

• The cost of each leaf is a constant• The total cost of all leaves would then be

) which, since log 2 > 1, is( lg )

10-Sep-15MAT-72006 AADS, Fall 2015 139

9/10/2015

7

• This recursion tree is not a complete binary tree,however, and has fewer than leaves

• As we go down from the root, more and moreinternal nodes are absent

• Consequently, levels toward the bottomcontribute less than to the total cost

• we can use the substitution method to verify thatlg is an upper bound for the solution to

the recurrence

10-Sep-15MAT-72006 AADS, Fall 2015 140

4.5 The master method

Theorem 4.1 (Master theorem)

Let and > 1, and ) a function, )is de ned on nonneg. integers by recurrence

) + ,where we means either or .

) has the following asymptotic bounds:

10-Sep-15MAT-72006 AADS, Fall 2015 141

9/10/2015

8

1. If = ( ) for some constant > 0,then ) =

2. If ) = ,then ) = lg

3. If ) = for some constant > 0,and if ) ) for some constant < 1and all suf ciently large ,

then ) = )

10-Sep-15MAT-72006 AADS, Fall 2015 142

• In each of the three cases, we compare thefunction ) with the function

• The larger of the two determines the solutionto the recurrence– In case 1, the function is the larger,

then the solution is ) =– In case 3, the function ) is the larger, then

the solution is ) =– In case 2, the two functions are the same

size, we multiply by a logarithmic factor, andthe solution is ) = lg

10-Sep-15MAT-72006 AADS, Fall 2015 143

9/10/2015

9

• Be aware of some technicalities:• In case 1, not only must ) be smaller than

, it must be polynomially smaller• ) must be asymptotically smaller than

by a factor of for some > 0• In case 3, not only must ) be larger than

, it also must be polynomially largerand satisfy the “regularity” condition that

( ) ( )10-Sep-15MAT-72006 AADS, Fall 2015 144

Using the master method

) = 9 3)

• We have = 9, = 3, ) = , and thus)

• Since ) = ), where = 1, wecan apply case 1 of the master theorem andconclude that the solution is )

10-Sep-15MAT-72006 AADS, Fall 2015 145

9/10/2015

10

) = (2 3) + 1

• Here = 1, = 3 2, ) = 1, and= 1

• Case 2 applies, since =(1), and thus the solution to the recurrence

is (lg )

10-Sep-15MAT-72006 AADS, Fall 2015 146

( ) = ( 4) + lg

• We have = 3, = 4, lg , and)

• Since ) = ), where 0.2,case 3 applies provided that the regularitycondition holds for )

• For suf ciently large , we have that) = 3( 4) lg( 4)

(3 4) lg = ) for = 3 4• By case 3, the solution is lg )

10-Sep-15MAT-72006 AADS, Fall 2015 147

9/10/2015

11

• The master method does not apply to) = 2 2) lg

even if it seems to have the proper form:= 2, = 2, lg , and

• One might think that case 3 applies, sinceis asymptotically larger than

• It is, however, not polynomially larger: ratio= lg is asymptotically less than

for any positive constant• Consequently, the recurrence falls into the

gap between case 2 and case 310-Sep-15MAT-72006 AADS, Fall 2015 148

• The recurrence of the running time of thesimple divide-and-conquer algorithm formatrix multiplication

) = 8 2) )

• Now = 8, = 2, and and so

• Since is polynomially larger than– i.e., for = 1case 1 applies, and ( ) =

10-Sep-15MAT-72006 AADS, Fall 2015 149

9/10/2015

12

• The recurrence of the running time ofStrassen’s algorithm

) = 7 2) )

• Here we have = 7, = 2, andand so

• Recalling that 2.80 < lg 7 < 2.81, we see that= O( ) for = 0.8

• Again, case 1 applies, and we have thesolution ( ) =

10-Sep-15MAT-72006 AADS, Fall 2015 150

5 Probabilistic Analysis andRandomized Algorithms

• We need to hire a new of ce assistant with thehelp of an employment agency that sends us acandidate each day

• We interview that person and then decide eitherto hire her or not

• We pay the agency a fee for each interview• To actually hire an applicant is more costly

– We must re the current of ce assistant and– pay a substantial hiring fee to the agency

10-Sep-15MAT-72006 AADS, Fall 2015 151

9/10/2015

13

• We are committed to having, at all times, thebest possible person for the job

• After interviewing an applicant, if she is betterquali ed than the current assistant, we will

re the current of ce assistant and hire thenew applicant

• We are willing to pay the price of this strategy,but wish to estimate what that price will be

10-Sep-15MAT-72006 AADS, Fall 2015 152

5.1 The hiring problem

HIRE-ASSISTANT )

1. best 0 // candidate 0 is a least-quali ed dummy candidate

2. for 1 to3. interview candidate4. if candidate is better than best5. best6. hire candidate

10-Sep-15MAT-72006 AADS, Fall 2015 153

9/10/2015

14

• Interviewing has a low cost, , whereas hiring isexpensive, costing

• Letting be the number of people hired, thetotal cost of the algorithm is )

• No matter how many people we hire, we alwaysinterview candidates and thus always incur thecost

• Let us concentrate on analyzing , the hiringcost

• This quantity varies with each run of thealgorithm

10-Sep-15MAT-72006 AADS, Fall 2015 154

Worst-case analysis

• In the worst case, we hire every candidate thatwe interview– candidates come in increasing order of quality– we hire times, for a total hiring cost of

• We have no idea about the order in which thecandidates arrive, nor do we have any controlover this order

• It is natural to ask what we expect to happen in atypical or average case

10-Sep-15MAT-72006 AADS, Fall 2015 155

9/10/2015

15

Probabilistic analysis

• We can assume that the applicants come in arandom order

• Assume that we can compare any twocandidates and decide the better quali ed

there is a total order on the candidates• Rank each candidate with a unique number

1, … ,• Use ) to denote the rank of applicant

10-Sep-15MAT-72006 AADS, Fall 2015 156

• A higher rank corresponds to a better quali edapplicant

• Ordered list 1 , 2 , … , ) is apermutation of the list 1,2, … ,

• “The applicants come in a random order”– “this list of ranks is equally likely to be any

one of the ! permutations of the numbers 1through ”

• Alternatively, the ranks form a uniform randompermutation; i.e., each of the possible !permutations appears with equal probability

10-Sep-15MAT-72006 AADS, Fall 2015 157

9/10/2015

16

Randomized algorithms

• We call an algorithm randomized if its behavioris determined also by values produced by arandom-number generator RANDOM

• A call to RANDOM returns an integerbetween and , inclusive, with each suchinteger being equally likely

• E.g., RANDOM(0,1) produces 0 with probability1 2, and 1 with probability 1 2

10-Sep-15MAT-72006 AADS, Fall 2015 158

• A call to RANDOM 3,7 returns either 3,4,5,6,7,each with probability 1 5

• Each integer returned by RANDOM isindependent of the integers returned on previouscalls

• In analyzing the running time of a randomizedalgorithm, we take the expectation of the runningtime over the distribution of values returned bythe RANDOM

• We refer to the running time of a randomizedalgorithm as an expected running time

10-Sep-15MAT-72006 AADS, Fall 2015 159

9/10/2015

17

5.2 Indicator random variables

• Indicator random variables provide a convenientmethod for converting between probabilities andexpectations

• Suppose we are given a sample space and anevent

• Then the indicator random variable }associated with is de ned as

1 if occurs0 if does not occur

10-Sep-15MAT-72006 AADS, Fall 2015 160

• Let us determine the expected number of headsthat we obtain when ipping a fair coin

• Our sample space is = { }, withPr = Pr = 1 2

• We de ne an indicator random variable ,associated with the coin coming up heads ( )

• This variable counts the number of headsobtained in this ip

= 1 if H occurs0 if T occurs

10-Sep-15MAT-72006 AADS, Fall 2015 161

9/10/2015

18

• Recall that E = Pr }• The expected number of heads obtained in one

ip of the coin is simply the expected value ofour indicator variable

E = E= 1 Pr + 0 Pr= 1 (1 2) + 0 (1 2)= 1 2

• Thus the expected number of heads obtained byone ip of a fair coin is 1 2

10-Sep-15MAT-72006 AADS, Fall 2015 162

• The expected value of an indicator randomvariable associated with an event is equal tothe probability that occurs

Lemma 5.1 Given a sample space and anevent in , let }. Then E = Pr{ }

Proof By the de nition of an indicator and thede nition of expected value, we have =

= 1 Pr + 0 Pr = Pr , wheredenotes , the complement of

10-Sep-15MAT-72006 AADS, Fall 2015 163

9/10/2015

19

• Compute the number of heads in coin ips• Let be the indicator associated with the event

in which the th ip comes up heads:{the th flip results in the event }

• Let denote the total number of heads in thecoin ips, so that

=

• To compute the expected number of heads, takethe expectation of both sides to obtain

E = E

10-Sep-15MAT-72006 AADS, Fall 2015 164

• Linearity of expectationE = E + E

makes the use of indicator random variables apowerful analytical technique– it applies even when there is dependence

among the variables• We now can easily compute the expected

number of heads:

E = E = E

1 2 = 2

10-Sep-15MAT-72006 AADS, Fall 2015 165

9/10/2015

20

Analysis of the hiring problem

• Let be the random variable whose valuenumber of times we hire a new assistant

• let be the indicator random variableassociated with the event in which the thcandidate is hired

candidate is hired= 1 if candidate is hired

0 if candidate is not hired

•

10-Sep-15MAT-72006 AADS, Fall 2015 166

• By Lemma 5.1E = Pr candidate is hired

and we must compute the probability that lines5–6 of HIRE-ASSISTANT are executed

• Candidate is hired (line 6) exactly when she isbetter than each of candidates 1, … ,

• We assume that the candidates arrive in arandom order: the rst candidates haveappeared in a random order

• Any one of these rst candidates is equallylikely to be the best-quali ed so far

10-Sep-15MAT-72006 AADS, Fall 2015 167

9/10/2015

21

• Candidate has a probability of 1/ of beingbetter quali ed than candidates 1, … , 1 andthus a probability of 1/ of being hired.

• By Lemma 5.1, we conclude that E = 1/• Now we can compute E :

E = E = E

= = ln (1)

by harmonic series

= = ln (1)

10-Sep-15MAT-72006 AADS, Fall 2015 168

• We interview people, but actually hire onlyapprox. ln of them, on average

• The following lemma summarize this

Lemma 5.2 Assuming that candidates come inrandom order, algorithm HIRE-ASSISTANT has anaverage-case total hiring cost of ln )

Proof Follows immediately from our de nition ofthe hiring cost and the equation, which shows thatthe expected number of hires is approximately ln

10-Sep-15MAT-72006 AADS, Fall 2015 169

9/10/2015

22

5.3 Randomized algorithms

• The algorithm above is deterministic• The number of times we hire a new of ce

assistant differs for different inputs• Given the rank list = 1,2,3,4,5,6,7,8,9,10 , a

new assistant is always hired 10 times• For the list of ranks = 10,9,8,7,6,5,4,3,2,1 , a

new assistant is hired only once• = 5,2,1,8,4,7,10,9,3,6 , a new assistant is

hired three times

10-Sep-15MAT-72006 AADS, Fall 2015 170

• Consider an algorithm that rst permutes theinput before determining the best candidate

• Now we randomize in the algorithm, not in theinput distribution

• For input we cannot say how many times thebest is updated – it differs with each run

• Not even an enemy can produce a bad input,random permutation yields input order irrelevant

• The algorithm performs badly only if the random-number generator produces an “unlucky”permutation

10-Sep-15MAT-72006 AADS, Fall 2015 171

9/10/2015

23

RANDOMIZED-HIRE-ASSISTANT )

1. randomly permute the list of candidates2. best 0 // candidate 0 is a least-quali ed

// dummy candidate3. for 1 to4. interview candidate5. if candidate is better than candidate best6. best7. hire candidate

10-Sep-15MAT-72006 AADS, Fall 2015 172

Randomly permuting arrays

• We assume that we are given an array which,w.l.o.g., contains the elements 1, 2, … ,

• Produce a random permutation of the array• Assign element ] a random priority ], and

sort the elements according to priorities• E.g., if our initial array is = 1,2,3,4 and we

choose random priorities = 36,3,62,19 , wewould produce array = 2,4,1,3

• We call this procedure PERMUTE-BY-SORTING

10-Sep-15MAT-72006 AADS, Fall 2015 173

9/10/2015

24

PERMUTE-BY-SORTING )

1. . length2. let [1. . ] be a new array3. for 1 to4. RANDOM(1, )5. sort , using as sort keys• We use a range of 1 to for random

numbers to make it likely that all the prioritiesin are unique

10-Sep-15MAT-72006 AADS, Fall 2015 174

• It remains to prove that the procedure producesa uniform random permutation,– It is equally likely to produce every

permutation of the numbers 1 through

Lemma 5.4 PERMUTE-BY-SORTING produces auniform random permutation of the input,assuming that all priorities are distinct

Proof See the book.

10-Sep-15MAT-72006 AADS, Fall 2015 175

9/10/2015

25

• It is better to permute the given array in place• RANDOMIZE-IN-PLACE does so in ) time• In its th iteration, it chooses the element ]

randomly from among elements through]

• After the th iteration, is never altered

RANDOMIZE-IN-PLACE )12 for 1 to3 swap with [RANDOM( , )]

10-Sep-15MAT-72006 AADS, Fall 2015 176

• A -permutation on a set of elements is asequence containing of the elements, withno repetitions

• There are ! ! -permutations

loop invariant:• Just prior to the th iteration of the for loop of

lines 2–3, for each possible 1)-permutationof the elements, the subarray [1. . 1]contains this 1)-permutation with probability

+ 1 ! !

10-Sep-15MAT-72006 AADS, Fall 2015 177

9/10/2015

26

Initialization: loop invariant trivially holds

Maintenance:• Consider a particular -permutation, and denote

the elements in it by , … ,• This permutation consists of an 1)-

permutation , … , followed by thevalue that the algorithm places in ]

• Let denote the event in which the rst 1)iterations have created the particular 1)-permutation , … , in [1. . 1]

10-Sep-15MAT-72006 AADS, Fall 2015 178

• By the loop invariant, Pr = + 1 ! !• Let be the event that th iteration puts in

position ]• The -permutation , … , appears in

1. . precisely when both and occurPr Pr Pr{

• Pr = 1 + 1 because in line 3 thealgorithm chooses randomly from the

+ 1 values in positions . .

Pr =1

+ 1+ 1 !!

=!

!10-Sep-15MAT-72006 AADS, Fall 2015 179

9/10/2015

27

Termination:• At termination, + 1, and we have that the

subarray 1. . is a given -permutation withprobability

+ 1 + 1 ! ! = 0! ! = 1 !

• Thus, RANDOMIZE-IN-PLACE produces a uniformrandom permutation

10-Sep-15MAT-72006 AADS, Fall 2015 180