IT 301: Algorithm Analysis

Lecture-00

Jesmin Akhter Lecturer,IIT

Jahangirnagar University, Savar, DhakaBangladesh

04/21/23

Course Overview Algorithmic Basics:

Introduction(L00) Algorithm analysis(L01) Techniques for analysis of Algorithm(L02+L03).

Methodes for the design of efficient algorithms: 3 sets Divide and Conquer (L04) Greedy Algorithms (L05) Dynamic Programming (L06)

Back tracking,Branch and bound : (L07) Binary search and Traversal Techniques: (L8) Sorting and Selection :

Heapsort : L9 3 sets Quicksort : L10 Bucket Sort, Radix Sort: (L11+L12)

Spanning Trees, Shortest paths: (L13) Lower Bound Theory: (L14) NP-Completeness :

NP-Completeness: (L15) 2 sets NP-hard: (L16) NP-Complete problems: (L17)

Tutorial: (L18+L19+L20)

Course Policy

No late homework is acceptable. Cheating directly affects the reputation of

Department and the University and lowers the morale of other students.

Cheating in homework and exam will not be tolerated.

An automatic grade of 0 will be assigned for any student caught cheating.

Presenting another person’s work as your own constitutes cheating.

Primary Focus

Develop thinking ability.

problem solving skills.

(algorithm design and application) formal thinking.

(proof techniques & analysis)

What Will We Do? Examine interesting problems. Devise algorithms to solve them. Prove these algorithms correct. Analyze their resource usage (time, memory). Understand data structures & core algorithms. Learn problem-solving techniques. See applications in real-world situations.

Goals

Intro 6

Be very familiar with a collection of core algorithms. Be fluent in algorithm design paradigms: divide & conquer,

greedy algorithms, dynamic programming. Be able to analyze the correctness and runtime performance

of a given algorithm. Be familiar with the inherent complexity (lower bounds &

intractability) of some problems. Be intimately familiar with basic data structures. Be able to apply techniques in practical problems.

Textbook & References Introduction to Algorithms, 2nd Ed. by Cormen, Leiserson, Rivest,

& Stein (CLRS), McGraw Hill, 2002.

OTHER REFERENCES: Algorithmics: The Spirit of Computing, Harel How to Solve It, Polya. The Design and Analysis of Computer Algorithms,

Aho, Hopcroft and Ullman. Algorithms, Sedgewick. Algorithmics: Theory & Practice, Brassard & Bratley. Writing Efficient Programs & Programming Pearls, Bentley. The Science of Programming, by Gries.

The Craft of Programming, by Reynolds.

How to Succeed in this Course

Start early on all assignments. DON'T procrastinate.

Complete all reading before class. Participate in class. Think in class. Review after each class. Be formal and precise on all problem sets

and in-class exams.

IT 301: Algorithm Analysis

Lecture-01

Jesmin Akhter Lecturer,IIT

Jahangirnagar University, Savar, DhakaBangladesh

04/21/23

What is an Algorithm? (And how do we analyze one?)

What is an Algorithm

An algorithm is a finite sequence of unambiguous instructions for solving a well-specified computational problem.

All algorithm must satisfy the following criteria: Input. Output. Definiteness. Effectiveness. Finiteness.

A technique for solving a well-specified computational problem.

Example: sortinginput: A sequence of numbers.output: An ordered permutation of the input.

DefinitenessEffectivenessFiniteness

Input Output

What is an Algorithm

Definiteness: Each instruction is clear and unambiguous.

Effectiveness: Each instruction is executable; in other words, feasibility.

Finiteness: The algorithm terminates after a finite number of steps.

What is an Algorithm

Algorithm Analysis Determining performance characteristics.

(Predicting the resource requirements.) Time, memory, communication bandwidth etc.

• Computation time (running time) is of primary concern.

Why analyze algorithms? Choose the most efficient of several possible

algorithms for the same problem. Is the best possible running time for a problem

reasonably finite for practical purposes? Is the algorithm optimal (best in some sense)? –

Is something better possible?

Running Time

Run time expression should be machine-independent. Use a model of computation or “hypothetical”

computer. Our choice – RAM model (most commonly-used).

Model should be Simple. Applicable.

RAM Model Generic single-processor model. Supports simple constant-time instructions found in

real computers. Arithmetic (+, –, *, /, %, floor, ceiling). Data Movement (load, store, copy). Control (conditional and unconditional branch, subroutine

call). Data types: Integer and floating point. Run time (cost) is uniform (1 time unit) for all simple

instructions. Memory is unlimited. Flat memory model – no hierarchy. Access to a word of memory takes 1 time unit. Sequential execution – no concurrent operations.

Running Time – Definition

Call each simple instruction and access to a word of memory a “primitive operation” or “step.”

Running time of an algorithm for a given input is The number of steps executed by the algorithm

on that input.

Often referred to as the complexity of the algorithm.

Complexity and Input

Complexity of an algorithm generally depends on Size of input.

• Input size depends on the problem.– Examples: No. of items to be sorted.– No. of vertices and edges in a graph.

Other characteristics of the input data.• Are the items already sorted? • Are there cycles in the graph?

Worst, Average, and Best-case Complexity

Worst-case Complexity Maximum steps the algorithm takes for any possible

input. Most tractable measure.

Average-case Complexity Average of the running times of all possible inputs. Demands a definition of probability of each input, which is

usually difficult to provide and to analyze. Best-case Complexity

Minimum number of steps for any possible input. Not a useful measure. Why?

A Simple Example – Linear Search INPUT: a sequence of n numbers, key to search for.OUTPUT: true if key occurs in the sequence, false otherwise.

n

i 21

LinearSearch(A, key) cost times1 i 1 c1 12 while i ≤ n and A[i] != key c2 x3 do i++ c3 x-14 if i n c4 15 then return true c5 16 else return false c6 1x ranges between 1 and n+1.So, the running time ranges between c1+ c2+ c4 + c5 – best caseand c1+ c2(n+1)+ c3n + c4 + c6 – worst case

A Simple Example – Linear Search INPUT: a sequence of n numbers, key to search for.OUTPUT: true if key occurs in the sequence, false otherwise.

n

i 21

Assign a cost of 1 to all statement executions.Now, the running time ranges between 1+ 1+ 1 + 1 = 4 – best caseand 1+ (n+1)+ n + 1 + 1 = 2n+4 – worst case

LinearSearch(A, key) cost times1 i 1 1 12 while i ≤ n and A[i] != key 1 x3 do i++ 1 x-14 if i n 1 15 then return true 1 16 else return false 1 1

A Simple Example – Linear Search INPUT: a sequence of n numbers, key to search for.OUTPUT: true if key occurs in the sequence, false otherwise.

n

i 21

If we assume that we search for a random item in the list,on an average, Statements 2 and 3 will be executed n/2 times.Running times of other statements are independent of input.Hence, average-case complexity is 1+ n/2+ n/2 + 1 + 1 = n+3

LinearSearch(A, key) cost times1 i 1 1 12 while i ≤ n and A[i] != key 1 x3 do i++ 1 x-14 if i n 1 15 then return true 1 16 else return false 1 1

Order of growth

Principal interest is to determine how running time grows with input size – Order of growth. the running time for large inputs – Asymptotic complexity.

In determining the above, Lower-order terms and coefficient of the highest-order term are

insignificant. Ex: In insertion sort the worst case running time : an2+bn+c,

which term dominates the running time for very large n? Complexity of an algorithm is denoted by the highest-order

term in the expression for running time. Ex: Θ(n2) Constant complexity when running time is independent of the input

size – denoted Ο(1). Linear Search: Best case Θ(1), Worst and Average cases: Θ(n).

Order of growth One algorithm to be more efficient than other

if its worst case running time has a lower order of growth. Due to constant factors and lower order terms

an algorithm whose running time has a higher order of growth might take less time for small inputs than an algorithm whose running time has a lower order of growth.

But for large enough inputs, a Θ(n2) algorithm will run more quickly than a Θ(n3) algorithm .

Comparison of Algorithms

Complexity function can be used to compare the performance of algorithms.

Algorithm A is more efficient than Algorithm B for solving a problem, if the complexity function of A is of lower order than that of B.

Examples: Linear Search – (n) vs. Binary Search – (lg

n) Insertion Sort – (n2) vs. Quick Sort – (n lg n)

Comparisons of Algorithms Sorting For 106 numbers,

insertion sort: (n2): it took 5.56 hrs on a supercomputer using machine language

merge sort: (n lg n): It took 16.67 min on a PC using C/C++.

Efficiency of Algorithms

Computer speed is not infinite and memory is not free so we need efficient algorithms.

Algorithms efficiency can be determined by time and space it need.

For example, insertion sort takes time equal to C1n2 to sort n items while merge sort takes C2nlogn.

Which one would work faster depends on size of n.

Example of Efficiency

Computer A is 1000 times faster than B. A takes 2n2

time to sort n items using insertion sort

B takes 50logn time to sort n items using merge sort

)5.5seconds(020,00ns/secondinstructio10

nsinstructio)2.(1010

27

hours

mins) 20second(1163ns/secondinstructio10

nsinstructiolog1050.107

77

Correctness Proofs

Proving (beyond “any” doubt) that an algorithm is correct. Prove that the algorithm produces correct output

when it terminates. Partial Correctness. Prove that the algorithm will necessarily

terminate. Total Correctness. Techniques

Proof by Construction. Proof by Induction. Proof by Contradiction.

Loop Invariant

Logical expression with the following properties. Holds true before the first iteration of the loop –

Initialization. If it is true before an iteration of the loop, it remains true

before the next iteration – Maintenance. When the loop terminates, the invariant gives a useful

property that helps show that the algorithm is correct – Termination.

Similar to mathematical induction.

Correctness Proof of Linear Search

Use Loop Invariant for the while loop: At the start of each iteration of the while loop, the

search key is not in the subarray A[1..i-1].

LinearSearch(A, key)1 i 12 while i ≤ n and A[i] != key3 do i++4 if i n5 then return true6 else return false

If the algm. terminates, then it produces correct result.

Initialization.Maintenance.Termination.

Argue that it terminates.

Go through correctness proof of insertion sort in the text.(2.1)(18)

The problem of sorting

Input: sequence a1, a2, …, an of numbers.

Example:

Input: 8 2 4 9 3 6

Output: 2 3 4 6 8 9

Output: permutation a'1, a'2, …, a'n such

that a'1 a'2 … a'n .

Insertion sort

INSERTION-SORT (A, n) ⊳ A[1 . . n]for j ← 2 to n

do key ← A[ j]i ← j – 1while i > 0 and A[i] > key

do A[i+1] ← A[i]i ← i – 1

A[i+1] = key

“pseudocode”

i j

keysorted

A:1 n

Example of insertion sort

8 2 4 9 3 6

Example of insertion sort

8 2 4 9 3 6

Example of insertion sort

8 2 4 9 3 6

2 8 4 9 3 6

Example of insertion sort

8 2 4 9 3 6

2 8 4 9 3 6

Example of insertion sort

8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

Example of insertion sort

8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

Example of insertion sort

8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

Example of insertion sort

8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

Example of insertion sort

8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

2 3 4 8 9 6

Example of insertion sort

8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

2 3 4 8 9 6

Example of insertion sort

8 2 4 9 3 6

2 8 4 9 3 6

2 4 8 9 3 6

2 4 8 9 3 6

2 3 4 8 9 6

2 3 4 6 8 9 done