Analysis of Algorithm

Why Analysis?

• We need to know the “behavior” of algorithms– How much resource (time/space) does it use

• So that we know when to use which algorithm– So that two algorithm can be compared whether

which one is better, with respect to the situtation

Can we?

• Sure!• If we have the algorithm– We can implement it– We can test it with input

• But… – Is that what we really want?– If you wish to know whether falling from floor 20 of

Eng. 4 building would kill you• Will you try?

Prediction

• We wish to know the “Behavior” of the algorithm– Without actually trying it

• Back to the suicidal example– Can you guess whether you survive jumping from

20th floor of Eng. 4 building?– What about 15th floor?– What about 10th floor?– What about 5th floor?– What about 2nd floor?

– Why?

Modeling

• If floor_number > 3 then– Die

• Else– Survive (maybe?)

Describe behavior using some kind of model, rule, etc.

Generalization

• What about jumping from Central World’s 20th floor?

• What about jumping from Empire State’s 20th floor?

• What about jumping from Bai Yok’s 20th floor?

• Can our knowledge (our analysis of the situation) be applicable on the above questions?

Generalization

• Knowledge from some particular instances might be applicable to another instance

Analysis

• We need something that can tell us the behavior of the algorithm that is…– Useful (give us knowledge without actually doing it)– Applicable (give us knowledge for similar kind of

situation)

GeneralizationModeling

Analysis (Measurement)

• What we really care?

RESOURCE

Time(CPU power)

Space(amount of RAM)

Model

• How to describe performance of an algorithm?

Usage of Resource how well

does an algo use

resource?

Model

• Resource Function

Time Function of algorithm A

SpaceFunction of algorithm A

Input ?

Input ?

Time used

Space used

Size of input

Example

• Inserting a value into a sorted array• Input: – a sorted array A[1..N]– A number X

• Output– A sorted array A[1..N+1] which includes X

Algorithm

• Element Insertion

• Assume that X = 20– What if A = [1,2,3]? How much time?– What if A = [101,102,103]?

idx = N;while (idx >= 1 && A[idx] > X) { A[idx + 1] = A[idx]; idx--;} A[idx] = X;

Usually, resource varies according to size of input

Using the Model

Time

Size of Input

best

worst

average

Resource Function

• Give us resource by “size of input”

– Why?• Easy to compute• Applicable (usually give meaningful, fairly accurate

result without much requirement)

Conclusion

• Measurement for algorithm– By modeling and generalization– For prediction of behavior

• Measurement is functions on the size of input– With some simplification– Best, avg, worst case

ASYMPTOTIC NOTATION

Comparing two algorithms

• We have established that a “resource function” is a good choice of measurement

• The next step, answering which function is “better”

What is “better” in our sense?

• Takes less resource• Consider this which one is better?

f(x)

g(x)

Slice

f(x)

g(x)

What is “better” in our sense?

• which one is better?– Performance is now a function, not a single value– Which slice to use?

• Can we say “better” based on only one slice?• Use the slice where it’s really matter– i.e., when N is large– What is large N?

• Infinity?

– Implication?

Comparison by infinite N

• There is some problem

– Usually,

– The larger the problem, the more resource used

)(lim xfn

Separation between Abstraction and Implementation

• Rate of Growth– by changing the size of input, how does the

TIME and SPACE requirement change– Compare by how f(x) grows when x increase,

w.r.t. g(x)

)(/)( xgxf)(

)(lim

xg

xfn

Compare by RoG

)(

)(lim

xg

xfn

0 : f(x) grows “slowzer” than g(x)

∞ : f(x) grows “faster” than g(x)

else : f(x) grows “similar” to g(x)

Growth Rate Comparison

0)(/)(lim

ngnfn

• 0.5n • 1 • log n • log6 n • n0.5 • n3 • 2n • n!

Sometime it is simple

Some time it is not

l’Hôpital’s Rule

• Limit of ratio of two functions equal to limit of ratio of their derivative.

– Under specific condition

l’Hôpital’s Rule

• If

• then

The problem of this approach

• What if f(x) cannot be differentiated?

• Too complex to find derivative

Compare by Classing

• Coarse grain comparison– Another simplification

• Work (mostly) well in practice• Classing

Classing

• Simplification by classification• Grading Analogy

Score Grade

>= 80 A

70 <= x < 80 B

60 <= x < 70 C

50 <= x < 60 D

< 50 F

Compare by Classification

algo

Compare by Classification

algo

Group A

Group C

Group B

Group F

Group Dgrouping

Compare by Classification

algo

Group A

Group C

Group B

Group F

Group DDescribe “simplified” property

>= 8070 <= x < 80

60 <= x < 70

50 <= x < 60

x < 50

Compare by Classification• Group by some similar property– Select a representative of the group– Use the representative for comparison

• If we have the comparison of the representative– The rest is to do the classification

Complexity Class

• We define a set of complexity class– using rate of growth

• Here comes the so-called Asymptotic Notation– Q, O, W, o, w

• Classify by asymptotic bound

Asymptote

• Something that bounds curves

Asymptote

Curve

Asymptote

• Remember hyperbola?

O-notation

O(g(n)) = {f(n) : positive constants c and n0, such that n n0,

we have 0 f(n) cg(n) }

For function g(n), we define O(g(n)), big-O of n, as the set:

g(n) is an asymptotic upper bound for f(n).

Intuitively: Set of all functions whose rate of growth is the same as or lower than that of g(n).

n0

cg(x)

f(x)

f(x) O(g(x))

-notation

g(n) is an asymptotic lower bound for f(n).

Intuitively: Set of all functions whose rate of growth is the same as or higher than that of g(n).

(g(n)) = {f(n) : positive constants c and n0, such that n n0,

we have 0 cg(n) f(n)}

For function g(n), we define (g(n)), big-Omega of n, as the set:

n0

cg(x)

f(x)

f(x) (g(x))

-notation

(g(n)) = {f(n) : positive constants c1, c2, and n0, such that n n0,

we have 0 c1g(n) f(n) c2g(n)

}

For function g(n), we define (g(n)), big-Theta of n, as the set:

g(n) is an asymptotically tight bound for f(n).

Intuitively: Set of all functions thathave the same rate of growth as g(n).

n0

c1g(x)

f(x)

c2g(x)

f(x) (g(x))

Example

• F(n) = 300n + 10 – is a member of (30n)– why?• let c1 = 9

• let c2 = 11

• let n = 1

Another Example

• F(n) = 300n2 + 10n– is a member of (10n2)– why?• let c1 = 29

• let c2 = 31

• let n = 11

How to Compute?

• Remove any constant– F(n) = n3+2n2 + 4n + 10• is a member of (n3+n2 + n)

• Remove any lower degrees– F(n) = n3+2n2 + 4n + 10• is a member of (n3)

Relations Between Q, W, O

• I.e., (g(n)) = O(g(n)) Ç W(g(n))

• In practice, asymptotically tight bounds are obtained from asymptotic upper and lower bounds.

For any two functions g(n) and f(n), f(n) = (g(n)) if and only if

f(n) = O(g(n)) and f(n) = (g(n)).

Practical Usage

• We say that the program has a worst case running time of O(g(n))

• We say that the program has a best case running time of W(g(n))

• We say that the program has a tight-bound running time of Q(g(n))

Example

• Insertion sort takes O(n2) in the worst case

– Meaning: at worst, insertion sort, takes time that grows not more than quadratic of the size of the input

• Insertion sort takes W(n) in the best case

– Meaning: at best, insertion sort, takes time that grows not less than linear to the size of the input

o-notation

o(g(n)) = {f(n): c > 0, n0 > 0 such that n n0, we have 0 f(n) < cg(n)}.

For a given function g(n), the set little-oh:

ω(g(n)) = {f(n): c > 0, n0 > 0 such that n n0, we have 0 cg(n) < f(n)}.

w -notation

For a given function g(n), the set little-omega:

Remark on Notation

• An asymptotic group is a set– Hence f(n) is a member of an asymptotic group– E.g., f(n) O( n )

• Strictly speaking, f(n) = O( n ) is syntactically wrong• But we will see this a lot• It’s traditions

Comparison of Functions

f (n) O(g(n)) f (n) g(n)f (n) (g(n)) f (n) g(n)f (n) (g(n)) f (n) = g(n)f (n) o(g(n)) f (n) < g(n)

f (n) w (g(n)) f (n) > g(n)

Where < , > , = means grows slower, faster, equally

Lost of Trichotomy

• Trichotomy– given two numbers a,b– Only one of the following must be true• a < b, a > b, a=b

• For our asymptotic notation– given f(n) and g(n)– it is possible that• f(n) != O(g(n)) and f(n) != (g(n)) • e.g., n, n1+sin n

Properties

• Transitivityf(n) = (g(n)) & g(n) = (h(n)) f(n) = (h(n))

f(n) = O(g(n)) & g(n) = O(h(n)) f(n) = O(h(n))f(n) = (g(n)) & g(n) = (h(n)) f(n) = (h(n))

f(n) = o (g(n)) & g(n) = o (h(n)) f(n) = o (h(n))f(n) = ω(g(n)) & g(n) = ω(h(n)) f(n) = ω(h(n))

• Reflexivityf(n) = (f(n))

f(n) = O(f(n)) f(n) = (f(n))

Properties

• Symmetryf(n) = (g(n)) g(n) = (f(n))

• Complementarityf(n) = O(g(n)) g(n) = (f(n)) f(n) = o(g(n)) g(n) = ω((f(n))

Complexity Graphs

log(n)

n

Complexity Graphs

log(n)

n

n

n log(n)

Complexity Graphs

n10

n log(n)

n3

n2

Complexity Graphs (log scale)

n10

n20nn

1.1n

2n

3n

Eventually, 1.1n will overcome n10

Common Class of Growth Rate

• constant : ( 1 )Θ• logarithmic : ( Θ log n )• polylogarithmic : ( Θ logc n ) , c ≥ 1• sublinear : ( nΘ a ) , 0 < a < 1• linear : ( Θ n )• quadratic : ( Θ n2 )• polynomial : ( nΘ c ) , c ≥ 1• exponential : ( Θ cn ) , c > 1

Logarithm

• Base of log is irrelevant– log b n = ( log c n ) / ( log c b )

– log 10 n = ( log 2 n ) / ( log 2 10 ) = ( log n )Θ

• any polynomial function of n does not matter– log n30 = 30 log n = ( log n )Θ

Conclusion

• Compare which one is better– By comparing their ratio when n approaches

infinity– By comparing their asymptotic notation of the

resource function

• Asymptotic notation– What is– Property