introduction to daa

1

Analysis and Design of algorithms

2

Books Fundamentals of Computer algorithms

Horowitz, Sahani and Rajasekaran

Introduction to Algorithms Coremen, Leiserson The Design and Analysis of Computer

Algorithms Aho, Hopcroft and Ullman

3

ALGORITHM

A finite set of instructions which if followed accomplish a particular task.

In addition every algorithm must satisfy following criteria:

4

1. Input: zero or more quantities externally supplied

2. Output: at least one quantity is produced

3. Definiteness: Each instruction must be clear and unambiguous.

4. Finiteness: In all cases algorithm must terminate after finite number of steps.

5. Effectiveness: each instruction must be sufficiently basic.

5

Efficiency of Algorithms

• Two algorithms on two systems

• Algorithm A1 50 n lg n

• Algorithm A2 2 n2

Super computer108 ins/sec

A2

P.C106 ins /sec

A1

6

For n = 106

Time taken by Super Computer = 2.(106)2/ 108 = 20,000

sec

Time taken by P .C.

= 50 . 106 lg 106 / 106 = 1,000 sec

7

Thus by using a fast algorithm , the personal computer gives results

20 times faster than the result given by super computer using a slow algorithm.

Thus a good algorithm is like a sharp knife, it does exactly what it is supposed to do

with a minimum amount of effort.

8

• Pseudocode is an English language like representation of the code required for an algorithm.

• It is partly English, partly structured code.

• The English part provides a relaxed syntax that is easy to read.

• The code part consists of an extended version of the basic algorithmic constructs-sequence, selection and iteration.

Pseudocode

9

Sequence, selection, loop

• A sequence is a series of statements that do not alter the execution path within an algorithm.

• Statements such as assign and add are sequence statements.

• A call to another algorithm is also considered a sequence statement.

• Selection statements evaluate one or more alternatives. Paths are followed based on its result.

10

• The typical selection statement is the two way selection

• if (condition) action 1 else action 2.

• The part of the loop are identified by indentation.

• Loop iterates a block of code. It closely resembles the while loop. It is a pretest loop.

11

1 i= 02 Loop(all data are read)

1 i = i + 12 read numbers into array[i]3 sum = sum + number

3 Average = sum / I4 Print (average)5 J = 06 Loop (j < i)

1 j = j+ 12 dev = array[j] – average3 print (array [ j] . Dev)

7 Return8 End deviation

Linear loop1 l=1

2 Loop(I <= 1000)

1 application code

2 I = I + 1

The body of the loop is repeated 1000 times.

1 I = 1

2 Loop (I <= n)

1 Application code

2 I = I + 2

For this the code time is proportional to n

Logarithm Loops

Multiply loops Divide loops

1 I = 1 1 I = n

2 Loop (I < n) 2 loop( l>= 1)1 application code 1 application

code 2 I = l*2 2I=I/2

F(n) = [log n] F(n) = [log n]

Nested loop- linear logarithmic

• 1 I = 1

• 2 loop(I <= n)1 j = 1

2 loop( j < = n)

1 application code

2 j = j *2

3 I = I + 1

F(n ) = [n log n]

Dependent Quadratic

1 I = 1

2 loop ( I < = n)

1 j = 1

2 loop( j < = l)

1 application code

2 j = j + 1

3 I = I + 1

no of iterations in the body of the inner loop is

1 + 2 + 3 + 4 +… + 10 = 55 I.e. n(n +1)/2

On an average = ( n+1/)2

thus total no of iterations = n (n+1)/2

Quadratic

1 l=1

2 Loop (I < = n)1 j = 1

2 Loop( j < = l)

1 application code

2 j = j+1

3 I = j+1

F(n) = n2

Algorithm Analysis

18

ComplexitySome questions to answer: How fast can we solve a

problem? There may be many algorithms for a given problem. Which algorithm to use? What are the classical algorithm design techniques ? Are there problems inherently difficult to solve?

19

How do we express the complexity of algorithm?

Resources : Time and Space

Complexity lower bounds for problems.

Complexity classes P, NP etc.

Analysis vs. Design

• Analysis: predict the cost of an algorithm in terms of resources and performance

• Design: design algorithms which minimize the cost

Machine model: Generic Random Access Machine (RAM)

• Executes operations sequentially• Set of primitive operations:

• Arithmetic. Logical, Comparisons, Function calls

• Simplifying assumption: all ops cost 1 unit• Eliminates dependence on the speed of our computer,

otherwise impossible to verify and to compare

L1.22

Review of Complexity

Most of the primary sorting algorithms run on different space and time complexity.

Time Complexity is defined to be the time the computer takes to run a program (or algorithm in our case).

Space complexity is defined to be the amount of memory the computer needs to run a program.

(Note:-What is Running Time and it is expressed as function of the size of its input.)

Time ComplexityReal Time:

To analyze the real time complexity of a program we need to determine two numbers for each statement in it:

• amount of time a single statement will take.

• No. of times it is executed.

• Product of these two, will be the total time taken by the statement.

First no. depends upon the machine and compiler used , hence the real time complexity is machine dependent.

Frequency count• To make analysis machine independent

it is assumed that every instruction takes the same constant amount of time for execution.

• Hence the determination of time complexity of a program is the matter of summing the frequency counts of all the statements.

L1.26

Kinds of analysis

Worst-case: (usually)• T(n) = maximum time of algorithm on any

input of size n.Average-case: (sometimes)

• T(n) = expected time of algorithm over all inputs of size n.

• Need assumption of statistical distribution of inputs.

Best-case: (NEVER)• Cheat with a slow algorithm that works fast

on some input.

L1.27

INSERTION SORT

• Insertion sort keeps making the left side of the array sorted until the whole array is sorted.

• Real life example: – An example of an insertion sort occurs in

everyday life while playing cards. To sort the cards in your hand you extract a card, shift the remaining cards, and then insert the extracted card in the correct place.

L1.28

Insertion sortINSERTION-SORT (A, n) ⊳ A[1 . . n]

for j ← 2 to ndo key ← A[ j]

i ← j – 1while i > 0 and A[i] > key

do A[i+1] ← A[i]i ← i – 1

A[i+1] = key

“pseudocode”

i j

keysorted

A:1 n

L1.29

Example of insertion sort8 2 4 9 3 6

L1.30

Example of insertion sort8 2 4 9 3 6

L1.31

Example of insertion sort8 2 4 9 3 6

2 8 4 9 3 6

L1.32

Example of insertion sort8 2 4 9 3 6

2 8 4 9 3 6

L1.33

Example of insertion sort8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

L1.34

Example of insertion sort8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

L1.35

Example of insertion sort8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

L1.36

Example of insertion sort8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

L1.37

Example of insertion sort8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

2 3 4 8 9 6

L1.38

Example of insertion sort8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

2 3 4 8 9 6

L1.39

Example of insertion sort8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

2 3 4 8 9 6

2 3 4 6 8 9 done

L1.40

Selection sort• How does it work:

– first find the smallest in the array and exchange it with the element in the first position, then find the second smallest element and exchange it with the element in the second position, and continue in this way until the entire array is sorted.

• How does it sort the list in a non increasing order?

• Selection sort is:– The simplest sorting techniques. – a good algorithm to sort a small number of elements– an incremental algorithm – induction method

• Selection sort is Inefficient for large lists.

Incremental algorithms process the input elements one-by-one and maintain the solution for the elements processed so far.

L1.41

Selection Sort Algorithm

Input: An array A[1..n] of n elements.Output: A[1..n] sorted in nondecreasing order.

1. for i 1 to n - 12. k i3. for j i +1 to n {Find the i th smallest element.}

4. if A [ j ] < A[ k] then k j5. end for6. if k i then

7. temp A[ i ] (interchange A[ i ] and A[ k ] )

8. A[ i] A[ k] 9. A[k ] temp 10. end for

Selection Sort in Action

L1.43

Analysis of Algorithms

• Algorithm analysis: quantify the performance of the algorithm, i.e., the amount of time and space taken in terms of n.

• T(n) is the total number of accesses made from the beginning of selection_sort until the end.

• selection_sort itself simply calls swap and find_min_index as i goes from 1 to n-1

1

1

min)(n

i

swapelementfindnT

= n-1 + n-2 + n-3 + … + 1 = n(n-1)/2

Or = ∑ (n - i) = n (n - 1) / 2 - O(n2)

Binary search

• sorted sequence : (search 9)

14 5 7 9 10 12 15

step 1 step 2 step 3 • best case: 1 step = O(1)

• worst case: (log2 n+1) steps = O(log n)

• average case: O(log n) steps

Growth Rate of an Algorithm (Order of Growth)

• We often want to compare the performance of algorithms

• When doing so we generally want to know how they perform when the problem size (n) is large

• Since cost functions are complex, and may be difficult to compute, we approximate them using Asymptotic notation

Example of a Cost Function

• Cost Function: tA(n) = n2 + 20n + 100– Which term dominates?

• It depends on the size of n– n = 2, tA(n) = 4 + 40 + 100

• The constant, 100, is the dominating term

– n = 10, tA(n) = 100 + 200 + 100• 20n is the dominating term

– n = 100, tA(n) = 10,000 + 2,000 + 100• n2 is the dominating term

– n = 1000, tA(n) = 1,000,000 + 20,000 + 100• n2 is the dominating term

Asymptotic notation Introduction

• Exact counting of operations is often difficult (and tedious), even for simple algorithms

• Often, exact counts are not useful due to other factors, e.g. the language/machine used, or the implementation of the algorithm (different types of operations do not take the same time anyway)

• Asymptotic-notation is a mathematical language for evaluating the running-time (and memory usage) of algorithms.

Big O Notation

• O notation approximates the cost function of an algorithm– The approximation is usually good enough, especially

when considering the efficiency of algorithm as n gets very large

– Allows us to estimate rate of function growth

• Instead of computing the entire cost function we only need to count the number of times that an algorithm executes its barometer instruction(s)– The instruction that is executed the most number of

times in an algorithm (the highest order term)

The general idea is …

• when using Big-O notation, rather than giving a precise figure of the cost function using a specific data size n

• express the behaviour of the algorithm as its data size n grows very large

• so ignore– lower order terms and– constants

O Notation Examples

• All these expressions are O(n):– n, 3n, 61n + 5, 22n – 5, …

• All these expressions are O(n2):– n2, 9 n2, 18 n2+ 4n – 53, …

• All these expressions are O(n log n):– n(log n), 5n(log 99n), 18 + (4n – 2)(log (5n + 3)), …

Asymptotic Notation (cont.)

• Note: Even though it is correct to say “7n - 3 is O(n3)”, a better statement is “7n - 3 is O(n)”, that is, one should make the approximation as tight as possible

• Simple Rule: Drop lower order terms and constant factors

7n-3 is O(n)

8n2log n + 5n2 + n is O(n2log n)

Worst/ best/average cases

• Worst case is the longest running time for any input of size n

• O-notation represents upper bound I.e. an upper bound for worst case.

• Best case is time taken for some input data set that results in best possible performance.You cannot do better. This is lower bound.

• Average case is the average performance

•The “Big-Oh” Notation:

•given functions f(n) and g(n), we say that f(n) is O(g(n) ) if and only if there are positive constants c and n0 such that f(n)≤ c g(n) for

n ≥ n0

Example

g(n) n

c g(n) 4n

n

f(n) = 2n + 6For functions f(n) and g(n) (to the right) there are positive constants c and n0 such that:

f(n) ≤ c g(n) for n ≥ n0

conclusion:

2n+6 is O(n).

Asymptotic Notation (terminology)• Special classes of algorithms:

constant O(1)

logarithmic: O(log n)

linear: O(n)

quadratic: O(n2)

polynomial: O(nk), k ≥ 1

exponential: O(an), a > 1

EXAMPLE

Consider 1/3 n2 – 5n

The dominating term is n2

Therefore it should be of O(n2)

•Given a positive constant c , a positive integer n0 to be found such that

1/3 n2 - 5 n c n2

Asymptotic Analysis of The Running Time

• Use the Big-Oh notation to express the number of primitive operations executed as a function of the input size.

• Comparing the asymptotic running time-an algorithm that runs in O(n) time is better

than one that runs in O(n2) time-similarly, O(log n) is better than O(n)-hierarchy of functions: log n << n << n2 <<

n3 << 2n

Categories of algorithm efficiency

Efficiency

Constant

Big O

O(1)

Logarithmic O(log n)

Linear O(n)

Linear logarithmic O(n log n)

Quadratic O(n2)

Polynomial O(nk)

Exponential O(cn)

Factorial O(n!)

-notation

(g(n)) = {f(n): +ve constants c1, c2, and n0 such that 0 c1g(n) f(n) c2g(n),

n n0 }

For function g(n), (g(n)) is given by:

g(n) is an asymptotically tight bound for f(n).

Intuitively: Set of all functions thathave the same rate of growth as g(n).

O -notation

O(g(n)) = {f(n): +ve constants c and n0 such that 0 f(n) cg(n), n n0 }

For function g(n), O(g(n)) is given by:

g(n) is an asymptotic upper bound for f(n).

Intuitively: Set of all functions whose rate of growth is the same as or lower than that of g(n).

f(n) = (g(n)) f(n) = O(g(n)).(g(n)) O(g(n)).

-notation

(g(n)) = {f(n): +ve constants c and n0 such that 0 cg(n) f(n), n n0 }

For function g(n), (g(n)) is given by:

g(n) is an asymptotic lower bound for f(n).

Intuitively: Set of all functions whose rate of growth is the same as or higher than that of g(n).

f(n) = (g(n)) f(n) = (g(n)).(g(n)) (g(n)).

Relations Between , O,

Practical Complexity

0

250

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

f(n) = n

f(n) = log(n)

f(n) = n log(n)

f(n) = n^2

f(n) = n^3

f(n) = 2^n