amortized and master method.pdf

7/24/2019 Amortized and Master method.pdf

1/63

Amortized Analysis and Master Method


2/63

Deliverables

Copyright @ gdeepak.com 6/5/2012 7:15 PM

Amortized Analysis

Aggregate Analysis

Accounting Method

Potential Analysis

Solving Recurrence Relations

Master Method


3/63

Amortized Analysis

Analyze a sequence of operations on a data structure.

Goal is to show that although some individual operations mbe expensive, on average the cost per operation is small.

Such analysis is not immediately obvious and needs deeknowledge of problem parameters and complexity bounds.

No probability is involved. It is average cost in the worst c

Aggregate analysis, Accounting method, and Potential meth



4/63

Aggregate Analysis

It requires knowledge of which series of operations possible. It is common case with data structures, whhave state that persists between operations.

Idea is that a worst case operation can alter the state iway that worst case cannot occur for a long time, thamortizing its cost. E.g. doubling an array will be operation for a heavy cost.

Aggregate analysis determines the upper bound T(n) on total cost of a sequence of n operations, then calcula

average cost to be T(n)/n



5/63

Aggregate Analysis-Example

k-bit binary counter A[0 . . k 1] of bits, where A[0] is thleast significant bit and A[k 1] is the most significant biCounts upward from 0.

Value of counter is

Initially, counter value is 0, so A[0 . . k 1] = 0.

Copyright @ gdeepak.com

1

[ ].2k

i

i o

A i

6/5/2012 7:15 PM


6/63

Binary counter algorithm

Increment(A, k)i0

while (i < k and A[i ] = 1)

do A[i ]0

ii + 1if i < k

then A[i ]1



7/63

Binary Counter example k = 3


0

1

2

34

5

6

7

0 0 0

0 0 1

0 1 0

0 1 11 0 0

1 0 1

1 1 0

1 1 1

Counter Value

0

1

3

47

8

10

11

cost

6/5/2012 7:15 PM


8/63

Binary counter- analysis

Cost of Increment = # of bits flippedEach call could flip k bits, so n Increments takes O(nk) time.

Observation is that not every bit flips every time.

bit flips how often times in n Increments

0 every time n

1 1/2 the time n/22 1/4 the time n/4

...

i 1/2i the time n/2i

i k never 0



9/63

Binary counter-Aggregate analysis

Therefore, total # of flips =< n =

= 2n .

Average cost per operation = O(1)


1

2ii o

1

2

n

i

i o

n

11

2

n

6/5/2012 7:15 PM


10/63

Stack Operations example

Stack operations Push(S, x): O(1) O(n) for any sequence of n operatio

Pop(S): O(1) O(n) for any sequence of n operations

Multipop(S, k)

whileS is not empty and k > 0

do Pop(S)

kk 1



11/63

Stack Complexity for Multipop

Running time of Multipop is linear in #of Poperations

# of iterations of while loop is min(s, k), where s =of objects on stack therefore, total cost = min(s, k)

Worst-case cost of Multipop is O(n)

For Sequence of n Push, Pop, Multipop operatio

worst-case cost of sequence is O(n2

).Copyright @ gdeepak.com 6/5/2012 7:15 PM


12/63

Aggregate Analysis for stacks

Each object can be popped only once per time thais pushed.

Have n Pushes n Pops, including those Multipop.

Therefore, total cost = O(n) and average overoperations O(1) per operation

No probability is involved and shows worst-ca

O(n) cost for sequenceCopyright @ gdeepak.com 6/5/2012 7:15 PM


13/63

Accounting method

A set of elementary operations are chosen to be used in

algorithm and their cost is set to 1.

Actually, cost differs because a prepayment may necessary to complete an operation with some payment over to be used later from savings.

Assign different charges to different operations. Some charged more than actual cost and some are charged less

Amortized cost = amount we charge



14/63

Accounting Method


When amortized cost > actual cost, store the differencespecific objects in the data structure as credit.

Use credit later to pay for operations whose actual cosamortized cost.

Differs from aggregate analysis because differoperations can have different costs while in aggreganalysis, all operations have same cost.

Need credit to never go negative

6/5/2012 7:15 PM


15/63

Stack

Intuition: When pushing an object, pay `2. ` 1 pays for the PUSH.

` 1 is prepayment for it being popped by either POP or Multipop.

Each object has ` 1, which is credit, credit can never go negative

Total amortized cost, = O(n), is an upper bound on total actual co


operation Actual cost Amortized Cost

PUSH 1 2

Pop 1 0

Multipop min(k, s) 0

6/5/2012 7:15 PM


16/63

Accounting Method

Let ci= actual cost of ith

operation,= amortized cost of ithoperation.

Then require

for all sequences of n operations.Total credit stored = - 0


ic

1

n

i

i

c

1

n

i

i

c

1

n

i

i

c

1

n

i

i

c

6/5/2012 7:15 PM


17/63

Binary Counter-Example

Charge `2 to set a bit to 1. `1 pays for setting a bit to 1. `1 is prepayment for flipping it back to 0. Have ` 1 of credit for every 1 in the counter. Therefore, credit 0. Amortized cost of increment: Cost of resetting bits to

paid by credit. At most 1 bit is set to 1. Therefore, amortized cost

For n operations, amortized cost = O(n).



18/63

Potential Method

Like the accounting method, but think of creditpotential stored with entire data structure. Accounting method stores credit with specific objects. Potential method stores potential in the data structure

a whole. Can release potential to pay for future operations. Most flexible of the amortized analysis methods. Let Di= data structure after ith operation ,



19/63

Potential Method

D0= initial data structure ci= actual cost of ith operation ,

= amortized cost of ith operation .

Potential function : DiR

(Di) is the potential associated with data structure D

= ci+ (Di) (Di1)

Increase in potential due to ith operation


ic

ic

6/5/2012 7:15 PM


20/63

Potential Method

Total amortized cost

= + DiDi-1

telescoping: every term except D0and Dn is added and subtrac

= + DnD0

If we require that (Di) (D0) for all i , then amortized co

an upper bound on actual cost (D0) = 0, (Di) 0


1

n

i

ic

1

n

i

i

c

1

n

i

i

c

1

n

i

i

c

1

n

i

i

c

6/5/2012 7:15 PM


21/63

Stack Example

= # of objects in stack

D0= empty stack (D0) = 0.

Since # of objects in stack is always 0, (Di) 0 = (D0) for al

Therefore, amortized cost of a sequence of n operations = O(n).


Operation Actual cost s=#of objects initially

AmortizedCost

Push 1 (s + 1) s =1 1+ 1 = 2

Pop 1 (s 1) s = 1 1 1 = 0

Multipop k=min(k, s) (sk) s = k k k = 0

6/5/2012 7:15 PM


22/63

Binary Counter example k = 3


0

1

2

34

5

6

7

0 0 0

0 0 1

0 1 0

0 1 11 0 0

1 0 1

1 1 0

1 1 1

Counter Value0

1

3

4

7

8

10

11

cost

6/5/2012 7:15 PM


23/63

Potential Method



24/63

Potential Method


ic

6/5/2012 7:15 PM

S l i R E i f


25/63

Solving Recurrence Equations forComplexity 3 methods

Guess the solution and prove

Using Recursion Tree Method

Master Method



26/63

Guess the solution and prove

1. Guess the form of the solution2. Verify by induction and show that the solution works f

a set of constants

T(n)=2T(n/2) + n Which means that now the problem h

been divided into two sub problems and the size of the suproblem is n/2



27/63

Guess and Substitute by Example

We guess that it works for T(n) = O(n lg n)

To prove that T(n) cn lgn for an appropriate c.

Assuming that the guess works we will have

T(n) = 2T(n/2)+ n

= n lg(n/2) +n + n

= n lg n n lg 2 + n +n

= n lg n + n cnlg n

this is true for a specific value of c


2( lg )2 2 2

n n nn

6/5/2012 7:15 PM


28/63

Guess by reasoning and experience

Looks similar to binary search, try O(logn)

Assume T(k) = O(log2k) for all k < n

Show that T(n) = O(log2n)

We need to prove T(n/2) c log2(n/2)


dnTnT )2/()(

6/5/2012 7:15 PM


29/63

Guess by reasoning and experience


dnTnT )2/()(dnc )2/(log2

dcnc 2loglog 22

nc 2log

if c d

Residualdcnc 2log

6/5/2012 7:15 PM


30/63

Recursion Tree

Guessing may be difficult for recurrences which does notresemble any common recurrence

Need to check the cost of tree at each level of recursion asum costs of all the levels. For this information of depth the tree, no of leaves of the tree is required

In the end answer can be verified by substitution using th

above as a guess.Copyright @ gdeepak.com

2)4/(3)( nnTnT

cnnTnTnT )3/2(2)3/()(

6/5/2012 7:15 PM


31/63

Recursion Tree (Level 1)


2)4/(3)( nnTnT cn2

T(n/4 ) T(n/4 )T(n/4 )

6/5/2012 7:15 PM


32/63

Recursion Tree (Level 2)


2)4/(3)( nnTnT

cn2

2

4

nc

2

4

nc

2

4

nc

16

nT

16

nT

16

nT

16

nT

16

nT

16

nT

16

nT

16

nT

16

nT

3/

6/5/2012 7:15 PM

2)4/(3)( TT


33/63


2)4/(3)( nnTnT

cn2

2

4

nc

2

4

nc

2

4

nc

2

16

nc

3/

2

16

nc

2

16

nc

2

16

nc

2

16

nc

2

16

nc

2

16

nc

2

16

nc

2

16

nc

(3/1

What is the cost at each level?2

16

3

cn

d

6/5/2012 7:15 PM


34/63



35/63

Important to grasp

Data is divided into 4 parts at each level


14

d

n

0

4

log

d

n

04loglog dn

nd log4log

nd4

log

6/5/2012 7:15 PM


36/63

Important to grasp

No of leaves in the tree with three children of everynode at each level


nd 4log33

6/5/2012 7:15 PM

12 d


37/63


)3(16

3...

16

3

16

3)( 4

log2

1

2

2

22 n

d

cncncncnnT

)3(16

34

4log

1log

0

2 nn

i

i

cn

xx

k

k

1

10

let x = 3/16

)3(16

34log

0

2 n

i

i

cn

)3()16/3(1

14log2 n

cn

)3(13

164log2 ncn )(

13

16)( 3log2 4ncnnT

)()( 2nOnT 6/5/2012 7:15 PM


38/63

Proving after Guess through recursion tre

Now we have reasonable guess with the help of recursiotree , let us prove it by guessing

Assume T(k) = O(k2) for all k < n Show that T(n) = O(n

Given that T(n/4) = O((n/4)2), then T(n/4) c(n/4)2


2

)4/(3)( nnTnT 22)4/(3 nnc 22 16/3 ncn

2cn13

16c

6/5/2012 7:15 PM

i h


39/63

Recurrence equations-Master Metho

Where a 1 and b>1 are constants and f(n) is asymptotically positive function

We are dividing a problem of size n into a sub problems, eof size n/b, where a and b are positive constants. a problems are solved recursively each in time T(n/b) wheand b are positive constants. Cost of dividing the problem combining the results of the sub problems is described byfunction f(n) which can be written as D(n)+C(n).


dnnfbnaTdncnT

if)()/(if)(

6/5/2012 7:15 PM

h f h d


40/63

Three cases of Master Method

Compare f (n) with nlogba:

1. f (n) = O(nlogba ) for some constant > 0.

f (n) grows polynomially slower than nlogba (by an nfacto

Solution:T(n) = (nlogba) .

2. f (n) = (nlogbalgkn) for some constant k 0.f (n) and nlogbagrow at similar rates.

Solution: T(n) = (nlogbalgk+1n)


Th C f M M h d


41/63

Three Cases of Master Method

Compare f (n) with nlogba:

3. f (n) = (nlogba+ ) for some constant > 0.

f (n) grows polynomially faster than nlogba(by an nfacto

and f (n) satisfies the regularity condition that

a f (n/b) c f (n) for some constant c < 1.Solution: T(n) = ( f (n))


M Th


42/63

Master Theorem

There is a gap between cases 1 and 2 when f (n) is smaller tbut not polynomially smaller. Similarly, there is a gap betwcases 2 and 3 whenf (n) is larger but not polynomially largethe functionf (n) falls into these gaps, or if regularity conditin case 3 fails to hold, master method cannot be used.


.1somefor)()/(provided

)),((is)(then),(is)(if3.

)log(is)(then),log(is)(if2.

)(is)(then),(is)(if1.

log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

dnnfbnaT

dncnT

if)()/(

if)(

6/5/2012 7:15 PM

E W


43/63

Easy Way


( ) ( ) kn

T n aT nb

blog ak

k k

k k

if a > b then T(n) = O(n ) (Bottom Heavy)

if a b then T(n) = O(n log(n)) (Steady State)

if a < b then T(n) = O(n ) (Top Heavy)

6/5/2012 7:15 PM

T( ) T( / )


44/63

T(n)=4T(n/2)+n

a=4 = n2b=2

f(n)=n

We are in case 1 so T(n) is (n2)


abn

log





log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

2

2

2

is f( ) ( )?

is f( ) ( log )?

is f( ) ( )?

k

n O n

n n n

n n

6/5/2012 7:15 PM

T( ) T( / ) l


45/63

T(n)=2T(n/2)+nlogn

a=2 = nb=2

f(n)=nlogn

We are in case 2 and k=1 so T(n) is (n log2 n)


abn

log





log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

1

1

1

is f( ) ( )?is f( ) ( log

is f( ) ( )?

k

n O n

n n

n n

6/5/2012 7:15 PM

T( ) 4T( /2)+ 3


46/63

T(n)=4T(n/2)+n3

a=4 = 2

b=2

f(n)=nlogn

We are in case 3 says T(n) is (n3).For n=1/2 4*(1/2)3/2 (1/2)3 (1/8) for = 1/






log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

ab

n

log 2

2

2

is f( ) ( )?

is f( ) ( log

is f( ) ( )?

k

n O n

n n

n n

6/5/2012 7:15 PM

T(n) 8T(n/2)+n2


47/63

T(n)=8T(n/2)+n2

Solution: logba=3, so case 1 says T(n) is (n3).


.1somefor)()/(provided)),((is)(then),(is)(if3.



log

1loglog

loglog

nfbnafnfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

dnnfbnaT

dnc

nT if)()/(

if

)(

6/5/2012 7:15 PM

T(n) 9T(n/3)+n3


48/63

T(n)=9T(n/3)+n3

Solution: logba=2, so case 3 says T(n) is (n3)


.1somefor)()/(provided)),((is)(then),(is)(if3.



log

1loglog

loglog

nfbnafnfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

dnnfbnaT

dnc

nT if)()/(

if

)(

6/5/2012 7:15 PM

Master Theorem


49/63

Master Theorem

T(n)=T(n/2)+1 (binary search)

Solution: logba=0, so case 2 says T(n) is (log n).






log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnf

nnTnOnf

a

kaka

aa

b

bb

bb

6/5/2012 7:15 PM

T(n) = 4T(n/2) + n2/lgn


50/63

T(n) = 4T(n/2) + n2/lgn

a = 4, b = 2 nlogba= n2;f (n) = n2/lgn.

Master method does not apply




)log(is)(then),log(is)(if2.)(is)(then),(is)(if1.

log

1loglog

loglog

nfbnaf

nfnTnnf

nnnTnnnfnnTnOnf

a

kaka

aa

b

bb

bb

6/5/2012 7:15 PM


51/63


)()/()( nfbnaTnT

)(nf

a

)/( bnf

)/( bnf )/( bnf

)/( 2bnf

a

)/( 2bnf )/( 2bnf )/(2

bnf

a

)/( 2bnf )/( 2bnf )/( 2bnf

a

)/( 2bnf )/(2bnf

f

(naf

(2 nfa

6/5/2012 7:15 PM

Depth of the tree


52/63

Depth of the tree

Data is divided by b at each level of the tree


1db

n

0log

db

n

log log 0dn b

nbd loglog

nd blog

6/5/2012 7:15 PM


53/63


Proof-Master Method


54/63

Proof-Master MethodCase 1: The Weight decreases

Geometrically from the root tothe leaves. The root hold aconstant fraction of the totalweight. First term is dominating.Case 2: Weight is approximatelysame on all levels. It ispolynomial or slowly decreasingfunction.Case 3: The Weight increasesGeometrically from the root tothe leaves. The leaves hold aconstant fraction of the totalweight. Last term is dominating.


1)(log

0

log

1)(log

0

log

2233

22

2

)/()1(

)/()1(

...

/()/()/(

)()/()/(

))/())/(()()/()(

n

i

iia

n

i

iin

b

b

b

b

bnfaTn

bnfaTa

bnafbnfabnTa

nfbnafbnTa

bnbnfbnaTanfbnaTnT

6/5/2012 7:15 PM

( ) 4 ( / 2)T n T n n


55/63


cn3

cn

2

cn

4 c

n

4 c

n

4 c

n

4

cn

2

cn

4 c

n

4 c

n

4 c

n

4

cn

2

cn

4 c

n

4 c

n

4 c

n

4

cn

2

cn

4 c

n

4 c

n

4

cn

cn

2 c

n

2 c

n

2 c

n

2

cn Total cn

Total 4c

Total 16c

6/5/2012 7:15 PM

3( ) 4 ( / 2)T n T n n


56/63


cn3

cn

2

3

cn

8

3

cn

8

3

cn

8

3

cn

8

3

cn

2

3

cn

8

3

cn

8

3

cn

8

3

cn

8

3

cn

2

3

cn

8

3

cn

8

3

cn

8

3

cn

8

3

cn

2

3

cn

8

3

cn

8

3

cn

8

cn3

cn

2

3

cn

2

3

cn

2

3

cn

2

3

cn3 Total cn3

4c

3

=1

16c

3

= 1

6/5/2012 7:15 PM


57/63



58/63


cn

cn

cn

2 c

n

2

cn

cn

2

cn

4 c

n

4

cn

2

cn

4 c

n

4

Total cn

Total cn

Total cn

T(n)=2T(n/2) + n

6/5/2012 7:15 PM

T(n)= T(n/3)+T(2n/3)+n


59/63


Questions, Comments and Suggestion


60/63

Questions, Comments and Suggestion


Question 1


61/63

Question 1

For recurrence T(n) = T(n-1) + n what will be the big oh

complexity.

A) O(n)

B) O(nlogn)

C) O(n2)

D) O(2n)


Question 2


62/63

Question 2

The recurrence relation that arises in relation to

complexity of binary search is

(A) T(n) = T(n/2) + k, k is a constant

(B) T(n) = 2T(n/2) + k, k is a constant

(C) T(n) = T(n/2) + log n

(D) T(n) = T(n/2) + n


Question 3


63/63

Q 3

Which of the following sorting algorithms has the low

worst-case complexity?

(A) Merge sort

(B) Bubble sort

(C) Quick sort

(D) Selection sort


amortized and master method.pdf

Documents