1

Analysis of Algorithms

Chapter - 01Introduction

2

This Chapter Contains the following Topics:

1.Syllabus & Marks Distribution2.Study of Algorithms

i. Algorithmii. Study of Algorithmsiii. Pseudocode Conventionsiv. Recursive Algorithmsv. Sorting Problem

3.Growth of Functionsi. Objectiveii. Example of Function Growthiii. Complexity of Algorithmiv. Asymptotic Notations

3

Syllabus &

Marks Distribution

4

Introduction, Asymptotic Notations Recurrences Sorting Algorithms Divide & Conquer Binary search tree Dynamic Programming Greedy Graph Algorithms Minimum spanning tree Data Compression Binomial heap Fibonacci heap

Text Books1. Introduction to Algorithms, 2nd edition by

Corman, Rivest,Leiserson, and Stein.2. Computer Algorithms: Introduction to Design

and Analysis By Sara Baase, 3rd Edition, Addison-Wesley Publishing Company

Tentative Syllabus & Text Book

5

Grading

Note: This is only a guide, percentages and rules may be changed during the semester as needed.

Percentages: The final grade will be composed of the following Four components:

⋆ 50% : Final exam. ⋆ 20% : First exam. ⋆ 20% : Second exam. ⋆ 10% : Assignments and Quizzes.

6

Study of

Algorithms

7

The word ‘algorithm’ comes from the name of a Persian author, Abu Ja’far Mohammad Ibn Musa Al Khowarizmi (825 AD).

Definition: An algorithm is a finite set of instructions that, if followed, carries out a particular task. In addition, all algorithms must

satisfy the following criteria:1. Input: Zero or more quantities are externally

supplied.2. Output: At least one quantity is produced.3. Definiteness: Each instruction is clear and

unambiguous.4. Finiteness: If we trace out the instructions

of an algorithm, then for all cases, the algorithm terminates after a finite number of steps.

5. Effectiveness: Every instruction must be very basic so that it can be carried out, in principle, by a person using only pen and paper. It must be feasible.

Algorithm

8

Study of Algorithms

The study of algorithms includes many important and active areas of research.

Given a problem, how do we find an efficient algorithm for its solution?

Once we have found an algorithm, how can we compare this algorithm with other algorithms that solve the same problem?

How should we judge the goodness of an algorithm?

About the study of algorithms, most researchers agree that there are four distinct

areas of study one can identify. 1. How to devise algorithms? 2. How to validate algorithms?3. How to test a program?4. How to analyze algorithms?

9

Study of Algorithms (Contd.)

How to devise algorithms? - Creating an algorithm is an art which may never be fully automated.

How to validate algorithms?- Once an algorithm is devised, it is necessary to show that it computes the correct answer for all possible legal inputs.- A program can be written and then be verified.

How to test a program?- Debugging is the process of executing programs on sample data sets to determine whether the faulty results occur and, if so, to correct them.- Profiling is the process of executing a correct program on data sets and measuring the time and space it takes to compute the results.

10

Study of Algorithms (Contd.)

How to analyze algorithms?- Analysis of algorithms refers to the task of determining how much computing time and storage an algorithm requires.- It allows you to make quantitative judgments about the value of one algorithm over another.- It allows you to predict whether the software will meet any efficiency constraints that exist.- Questions such as how well does an algorithm perform in the best, worst and average cases.

Theoretical analysis:−All possible inputs.−Independent of hardware/ software implementation Experimental Study:−Some typical inputs.−Depends on hardware/ software implementation.−A real program.

11

Pseudocode Conventions

Comments begin with // and continue until the end of line.Blocks are indicated with matching braces: {and}.An Identifier begins with a letter: max. Assignment of values to variables is done using assignment statement : Variable:=expression. Logical operators: and, or and not are provided. Relational operators: <, ≤, =, ≠, ≥ and > provided. Elements of arrays are accessed using: [ and ]. While loop:

while (condition) do{

(statement 1)………………(statement n)

}For loop: for variable:=value1 to value2 step step do

{(statement 1)………………(statement n)

}

12

Pseudocode Conventions (Contd.)

Repeat-until loop: repeat

{(statement 1)…………(statement n)

} until (condition)Conditional statement:

if (condition) then (statement) if (condition) then (statement 1)

else (statement 2) Case statement:

case{ : (condition 1): (statement 1)

…………………. : (condition n): (statement n) : else: (statement n+1)}

Input and output are done using: read and write. There is only one type of procedure: Algorithm. An algorithm consists of a heading and a body. The heading of an algorithm takes the form

Algorithm Name ((parameter list))

13

Recursive Algorithms

A recursive function is a function that is defined in terms of itself. Similarly, an algorithm is said to recursive if the same algorithm is invoked in the body. An algorithm that calls itself is direct recursive. Algorithm A is said to be indirect recursive if it calls another algorithm which in turn calls A. Algorithm TowersOfHanoi (n, x, y, z)// Move the top n disks from tower x to tower y.{ if (n ≥ 1) then { TowersOfHanoi (n-1, x, z, y);

write (“Move top disk from tower”, x,“to top of tower”, y);TowersOfHanoi (n-1, z, y, x);

}}

Illustrate the above algorithm with an example.

14

Sorting Problem

Input: A sequence of n numbers (a1, a2,…, an). Output: A permutation of n numbers (reordering) (a'1, a'2,…, a'n) of the input sequence such that a'1≤

a'2 ≤… ≤ a'n. Solve the problem using Insertion sort. Insert an element to a sorted array such that the order of the resultant array be not changed.

Algorithm InsertionSort (A, n){ for i:=2 to n do { key:=A[i]; // Insert A[i] into the sorted sequence A[1…i-1]. j:=i-1; while ( (j>0) and (A[j]>key) ) do {

A[j+1]:=A[j];j:=j-1;

} A[j+1]:=key; }} Illustrate the above algorithm with an example.

15

At the start of each iteration of the ‘for loop’, the sub-array A[1…i-1] consists of the elements originally in A[1…i-1] but in sorted order.We use Loop invariants to help us understand why an algorithm is correct. We must show three things about a loop invariants:

Initialization: It is true prior to the first iteration of the loop. Maintenance: if it is true before an iteration of the loop, it remains true before the next iteration. Termination: When the loop terminates, the invariant gives us a useful property that helps show that the algorithm is correct.

Let us see how these properties hold for insertion sort:

Initialization: Before 1st loop iteration, when i=2, the subarray A[1…i-1] consists of single element A[1], which is already sorted. Maintenance: The body of outer ‘for loop’ works by moving A[i-1], A[i-2], A[i-3], and so on by one position to the right until the proper position for A[i] is found. Termination: When i=n+1, the subarray A[1…n] consists of the elements originally in A[1…n], but in sorted order.

Loop Invariant and Correctness

16

T(n) = c1n + c2(n-1) + c3(n-1) + c4Σti + c5 Σ(ti -1) + c6 Σ(ti -1) + c7(n-1)

Best Case: ti=1, for i=2,3,…..n

T(n) = c1n + c2(n-1) + c3(n-1) + c4(n-1) + c7(n-1) = (c1+c2+c3+c4+c7)n – (c2+c3+c4+c7)

So, T(n)=an+b

Worst Case: ti = i, for i=2,3,……,n.Σi = n(n+1)/2 – 1

Σ(i-1) = n(n-1)/2

T(n)= c1n + c2(n-1) + c3(n-1) + c4(n(n+1)/2 – 1) + c5 (n(n-1)/2) + c6 (n(n-1)/2) + c7(n-1) = (c4/2 + c5/2 + c6/2)n2 + (c1+c2+c3+c4/2-c5/2-c6/2+c7)n - (c2+c3+c4+c7)

So, T(n)=an2+bn+c

Analysis of Insertion Sort

17

Growth of

Functions

18

Objective

Objective: A language to express that Algorithm A is better or worse or equivalent to Algorithm B.

Need to define “≤” between functions measuring the growth of functions.

Independence of the hardware/ software environment: Turing machine, RAM machine, classroom model, today computers, and future super-computers.

Ignore constants that can be affected by changing the environment.

19

Examples of Function Growth

Running Time 1 second 1 minute 1 hour

400n 2,500 150,000 9,000,000

20n log n⌈ ⌉ 4,096 166,666 7,826,087

2n2 707 5,477 42,426

n4 31 88 244

2n 19 25 31

Maximum size of a problem that can be solved in one second, one minute, and one hour, given below:

Increase in the maximum size of a problem by using a computer that is 256 times faster than the previous one. Each entry is given as a function of m, the previous maximum problem size.

Running Time New Maximum Size

400n 256m

20n log n⌈ ⌉ ≈256(logm)/(7+logm))m

2n2 16m

n4 4m

2n m+8

20

Complexity of Algorithm

How much of resources the algorithm requires?

− Usually: time and space (memory). Complexity: as a function of the input length n.

− Usually an integer n > 0.− Usually a monotonic non-decreasing function T(n).

The limiting behavior of the complexity as size increases is called the asymptotic complexity of the algorithm.

Worst Case and Average Case Complexity

T(n) is a worst case complexity:− If for all inputs of length n the complexity is T(n).

T(n) is an average case complexity:− If the average complexity over all length n inputs is T(n).− Averaging following some distribution of the inputs.− Usually the uniform distribution.

21

Big-Oh: f(n) = O(g(n)) if f(n) asymptotically less than or equal to g(n).

Big-Omega: f(n) = ῼ(g(n)) if f(n) asymptotically greater than or equal to g(n).

Big-Theta: f(n) = ⊖(g(n)) if f(n) asymptotically equal to g(n).

Little-oh: f(n) = o(g(n)) if f(n) asymptotically strictly less than g(n).

Little-omega: f(n) = ω(g(n)) if f(n) asymptotically strictly greater than g(n).

Asymptotic Notations

22

Big-Oh:: f(n) = O(g(n)), if− There exists a real constant c > 0 and an integer constant n0 > 0 such that f(n) ≤ cg(n) for every integer n ≥ n0.

Big-Oh, Big-Omega, Big-Theta

f(n) = O(g(n)) g(n) = O(f(n))

g(n) grows faster YES NO

f(n) grows faster NO YES

Same growth YES YES

Big-Omega:: f(n) = ῼ(g(n)), if− There exists a real constant c > 0 and an integer constant n0 > 0 such that f(n) ≥ cg(n) for every integer n ≥ n0.

Big-Theta:: f(n) = ⊖(g(n)), if

− There exist two real constants c′, c′′ > 0 and an integer constant n0 > 0 such that c′′g(n) ≤ f(n) ≤ c′g(n) for every integer n ≥ n0.

23

Big-Oh

24

Big-Omega

25

Big-Theta

26

Examples

3n + 2 = O(n)As 3n + 2 ≤ 4n, for all n≥2.

100n + 6 = O(n)As 100n + 6 ≤ 101n, for all n≥ 6.

10n2 + 4n + 2 = O(n2)As 10n2 + 4n + 2 ≤ 11n2, for all n≥ 5.

1000n2 + 100n - 6 = O(n2)As 1000n2 + 100n - 6 ≤ 1001n2, for all n≥100.

6*2n + n2 =O(2n)As 6*2n + n2 ≤ 7*2n, for all n≥4.

3n + 2 = O(n2)3n + 2 ≤3n2, for all n≥2.

But 3n + 2 ≠ O(1) and 10n2 + 4n + 2 ≠ O(n).3n + 2 = Ω(n)

As 3n + 2 ≥ 3n, for all n≥1.10n2 + 4n + 2 = Ω (n2)

As 10n2 + 4n + 2 ≥ n2, for all n ≥ 1.6*2n + n2 = Ω(2n)

As 6*2n + n2 ≥ 2n, for all n≥1.3n + 2 = Θ(n)

3n ≤ 3n + 2 ≤ 4n, for all n ≥2.10n2 + 4n + 2 = Θ(n2)6*2n + n2 = Θ(2n)

27

f(n) = o(g(n)) if limn→∞f(n)/g(n) = 0:− For any constant c > 0 there exists an

integer constant n0 > 0 such that f(n) < cg(n) for every integer n ≥ n0.

f(n) = ω(g(n)) if limn→∞f(n)/g(n) = ∞:− For any constant c > 0 there exists an

integer constant n0 > 0 such that f(n) > cg(n) for every integer n ≥ n0.

Little-oh and Little-omega

28

constant 1

log log log log n

logarithmic log n

poly-logarithmic logk n constant integer k>1

sub-linear nε Constant 0<ε<1

linear n

above-linear n log n

quadratic n2

cubic n3

polynomial nk constant integer k>1

super-polynomial nlog n

exponential 2n

factorial n!

super-exponential 22…2 n powers

Hierarchy of Functions

Simplifying the AnalysisSimplifying the Analysis

Order NotationOrder Notation

Algorithm Analysis: Example

• Alg.: MIN (a[1], …, a[n]) m ← a[1];

for i ← 2 to n

if a[i] < m

then m ← a[i];

• Running time: – the number of primitive operations

(steps) executed before termination

T(n) =1 [first step] + (n) [for loop] + (n-1) [if condition] +

(n-1) [the assignment in then] = 3n - 1• Order (rate) of growth:

– The leading term of the formula– Expresses the asymptotic behavior of

the algorithm

Extra Examples

– 2n2 = O(n3):

– n2 = O(n2):

n = O(n2):

2n2 ≤ cn3 2 ≤ cn c = 1 and n0= 2

n2 ≤ cn2 c ≥ 1 c = 1 and n0= 1

n ≤ cn2 cn ≥ 1 c = 1 and n0= 1

Examples

– 100n + 5 ≠ (n2) For the above question, we had a solution

proposed as follows to prove 100n + 5 = (n2) :-

cn2 < 100n + 5

Let no = 100 . That gives us,

c x 10000 < 100 x 100 + 5

c < 10005 / 10000

So the contant c = 1.0005

But does this work when c = 1.0005 and n = 200 ( remember, it should satisfy for all n > no . Here n is 200 and no is 100)

No, it doesn’t work. Hence you cannot prove that

100n + 5 = (n2)

Examples•Prove that 100n + 5 = O(n2)

–100n + 5 ≤ 100n + n = 101n ≤

101n2

for all n ≥ 5

n0 = 5 and c = 101 is a solution

–100n + 5 ≤ 100n + 5n = 105n ≤

105n2

for all n ≥ 1

n0 = 1 and c = 105 is also a solution

Must find SOME constants c and n0 that satisfy the

asymptotic notation relation

2n

2

1)n(n

1)n(n

n

1ii

n

1i

n

ij

n

1i

n

1i1)(n1)i(n1

Nested Dependent Loop

for i =1 to n do for j = i to n do sum=sum+1