cs 201 data structures and algorithms chapter 2: algorithm analysis - i text: read weiss, §2.1 –...

CS 201Data Structures and

Algorithms

Chapter 2: Algorithm Analysis - I

Text: Read Weiss, §2.1 – 2.4.2

1Izmir University of Economics

Definition

• An Algorithm is a clearly specified set of simple instructions to be followed to solve a problem.

• Once an algorithm is given and decided somehow to be correct, an important step is to determine how much in the way of resources, such as time or space, the algorithm will require.

Izmir University of Economics 2

Mathematical Background• Definition 2.1. T(N) = O(f(N)) if there are

c,n0 ≥0 such that T(N)≤cf(N) when N ≥ n0.

• Definition 2.2. T(N) = Ω(g(N)) if there are c,n0 ≥0 such that T(N)≥cg(N) when N ≥ n0.

• Definition 2.3. T(N) = Ɵ(h(N)) iff T(N)=O(h(N)) and T(N)=Ω(h(N)).

• Definition 2.4. T(N) = o(p(N)) if for all c, there exists an n0 such that T(N)<cp(N) when N > n0. (T(N) = o(p(N)) if T(N)=O(p(N)) and T(N) <> Ɵ(p(N))


O (Big-Oh) Notation• Definitions establish a relative order among

functions. We compare their relative rates of growth.

• Example: For small values of N, T(N)=1000N is larger than f(N)=N2. For N≥n0=1000 and c= 1, T(N) ≤ cf(N). Therefore, 1000N = O(N2) (Big-Oh notation).

• Big-Oh notation says that the growth rate of T(N) is less than or equal to that of f(N). T(N)=O(f(N)) means f(N) is an upper bound on T(N).


Ω and Ɵ notations

• T(N) = Ω(g(N)) (pronounced “omega”) says that the growth rate of T(N) is greater than or equal to that of g(N). g(N) is a lower bound on T(N).

• T(N) = Ɵ(h(N)) (pronounced “theta”) says that the growth rate of T(N) equals the growth rate h(N).

• T(N) = o(p(N)) (pronounced “little-oh”) says that the growth rate of T(N) is less than that of p(N).

• Example: N2=O(N3), N3=Ω(N2)


Rules for Asymptotic Analysis-I

• Rule 1: If T1(N)=O(f(N)) and T2(N)=O(g(N)), then

a)T1(N)+T2(N)=O(f(N)+g(N))

(intuitevely, max(O(f(N)), O(g(N)))

b) T1(N)*T2(N)=O(f(N)*g(N))

• Rule 2: If T(N) is a polynomial

of degree k, then T(N)=Ɵ(Nk).• Rule 3: logkN=O(N) for any

constant k.


Rules for Asymptotic Analysis-II• It is very bad style to include constants or low-order

terms inside a Big-Oh. Do not say T(N)=O(2N2) or T(N)=O(N2+N). The correct form is T(N)=O(N2).

• The relative growth rates of f(N) and g(N) can always be determined by limN(f(N)/g(N)): (using L’Hỏpital’s rule if necessary)

1. The limit is 0: f(N)=o(g(N))

2. The limit is c≠0: f(N)=Ɵ(g(N))

3. The limit is : g(N)=o(f(N))

4. The limit oscillates: no relation


)()(lim

)()(lim

)(lim,)(lim

NgNf

NNgNf

N

NgN

NfN

Rules for Asymptotic Analysis-III• Sometimes simple algebra is just sufficient.• Example: f(N)=NlogN, g(N)=N1.5 are given.

Decide which of the two functions grows faster!• This amounts to comparing logN and N0.5. This,

in turn, is equivalent to testing log2N and N. But we already know that N grows faster than any power of a log.

• It is bad to say f(N)≤O(g(N)), because the inequality is implied by the definition. f(N)≥O(g(N)) is incorrect since it does not make sense.


Model of Computation• Our model is a normal computer.

• Instructions are executed sequentially.

• It has a repertoire of simple instructions (addition, multiplication, comparison, assignment).

• It takes one (1) time unit to execute these simple instructions (assume our model has fixed size integers and no fancy operations like matrix inversion or sorting).

• It has infinite memory.

• Weaknesses: disk read vs addition, page faults when memory is not infinite


What to Analyze• The most important resource to analyze is generally

the running time of a program.

• Compiler and computer used affect it but are not taken into consideration.

• The algorithm used and the input to it will be considered.

• Tavg(N) (average running time: typical behavior) and Tworst(N) (worst case: it is generally the required quantity: guarantee for performance: bound for all input) running times on input size N.

• The details of the programming language do not affect a Big-Oh answer. It is the algorithm that is analyzed not the program (implementation).


Maximum Subsequence Sum Problem - I

• Definition: Given (possibly negative) integers

A1, A2, ..., AN, find the maximum value of .

(For convenience, the maximum subsequence

sum is 0 if all integers are negative)• Example: For input -2, 11, -4, 13, -5, -2, the

answer is 20 (A2 through A4).

• There are many algorithms to solve this problem. We will discuss 4 of these.


j

ikkA

Maximum Subsequence Sum Problem - II

• For a small amount of input, they all run in a blink of the eye.

• Algorithms should not form bottlenecks.Times do not include the time to read.


Maximum Subsequence Sum Problem - III

-O(NlogN) Algorithm is not linear. Verify it by a straight-edge.

-Relative growth rates are evident.


Maximum Subsequence Sum Problem - IV

• Illustrates how useless inefficient algorithms are for even moderately large amounts of input.


Running Time Calculations• Several ways to estimate the running time of a program (empirical vs analytical)• Big-Oh running times. Here is a simple program fragment to

calculate- The declarations count for no time.- Lines 1 and 4 count for 1 unit each- Line 3 counts for 4 units per time

executed (2 multiplications, 1 addition,

1 assignment) and is executed N

times for a total of 4N units.- Line 2 costs 2N+2 units (1 unit for initial

assignment, N+1 units for comparison tests

N units for all increments).- Total time T(N) is 6N+4 which is O(N).


Nii

13

unsigned int sum( int n ) { unsigned int i, partial_sum; /*1*/ partial_sum = 0; /*2*/ for( i=1; i<=n; i++ ) /*3*/ partial_sum += i*i*i; /*4*/ return( partial_sum ); }

Some shortcuts could be taken without affecting the final answer (Line 3 is an O(1) statement, Line 1 is insignificant compared to for loop.

General Rules - I• RULE 1-FOR LOOPS:The running time of a for loop is at

most the running time of the statements inside the for loop (including tests) times the number of iterations.

• RULE 2-NESTED FOR LOOPS: Analyze these inside out. The total running time of a statement inside a group of nested for loops is the running time of the statement multiplied by the product of the sizes of all the for loops.

Example: the following program fragment is O(n2):


for( i = 0; i < n; i++ ) for( j=0; j < n; j++ ) k++;

General Rules - II• RULE 3-CONSECUTIVE STATEMENTS: These just add

(which means that the maximum is the one that counts – Rule 1(a) on page 6).

Example: the following program fragment, which has O(n) work followed by O (n2) work, is also O (n2):


for( i = 0; i < n; i++) a[i] = 0; for( i = 0; i < n; i++ ) for( j = 0; j < n; j++ ) a[i] += a[j] + i + j;

General Rules - III• RULE 4-lF/ELSE: For the fragment

• the running time of an if/else statement is never more than the running time of the test plus the larger of the running times of S1 and S2.

• Clearly, this can be an over-estimate in some cases, but it is never an under-estimate.


if( condition ) S1 else S2

General Rules - IV• Other rules are obvious, but a basic strategy of analyzing

from the inside (or deepest part) out works. If there are function calls, obviously these must be analyzed first.

• If there are recursive procedures, there are several options. If it is really just a thinly veiled for loop, the analysis is usually trivial. Example: The following function is really just a simple loop and is obviously O (n):


unsigned int factorial( unsigned int n ) { if( n <= 1 ) return 1; else return( n * factorial(n-1) ); }

General Rules - V• When recursion is properly used, it is difficult to convert the

recursion into a simple loop structure. In this case, the analysis will involve a recurrence relation that needs to be solved.

• Example: Consider the following program:- If the program is run for values of naround 40, it becomes terribly inefficient.Let T(n) be the running time for theFunction fib(n). T(0)=T(1)=1 (time to dothe test at Line 1 and return). For n≥2,the total time required is thenT(n ) = T(n - 1) + T(n - 2) + 2(where the 2 accounts for the work at Line 1 plus the addition atLine 3).

T(n) ≥ fib(n)=fib(n-1)+fib(n-2) ≥ (3/2)n (for n > 4, by induction)

• Huge amount of redundant work (violates 4th rule of recursion-compound interest rule. fib(n-1) has already computed fib(n-2)


unsigned int fib( unsigned int n ) {/*1*/ if( n <= 1 )/*2*/ return 1; else /*3*/ return( fib(n-1) + fib(n-2) ); }

cs 201 data structures and algorithms chapter 2: algorithm analysis - i text: read weiss, §2.1 –...

Documents