Download - Lec 02 Intro Algorithms(2)

Advanced Data Structures and Algorithms

�CS 361 - Fall 2014 �

Tamer Nadeem �Dept. of Computer Science�

Lec. #02: Introduction to Algorithm Analysis

CS 361 - Advanced Data Structures and Algorithms Fall 2014

Class Objective/Overview

• Main Objective: Start thinking about designing and analyzing algorithms.

• Doing a Timing Analysis

• Understand running-time complexity/analysis of an algorithm using Big-O notation and Big-Oh Operations

• Apply Big-O analysis on simple sorting/searching algorithms

• Worst-Case and Average Case Analysis

• Understand simple sorting and searching techniques that will be used as examples.

• Understand and use of recursion.


Introduction

• Algorithm •  The basic method; it determines the data-items computed. •  Also, the order in which those data-items are computed (and hence the order

of read/write data-access operations).

• An algorithm can be analyzed in terms of time efficiency and/or space utilization.

• An algorithm’s running time is influenced by several factors: •  Speed of the machine running the program •  Language in which the program was written. Programs in assembly language

run faster than those in C or C++, which in turn run faster than those in Java. •  Efficiency of the compiler that created the program •  The size of the input: processing 1000 records will take more time than

processing 10 records. •  Organization of the input: item at the top of the list will take less time to find

Program = Algorithm + Data Structure


Algorithm Complexity

•  A code of an algorithm is judged by its correctness, its ease of use, and its efficiency.

•  This course focus on the computational complexity (time efficiency) of algorithms that apply to container objects (data structures) that hold a large collection of data.

•  We will learn how to develop measures of efficiency (criteria) that depends on n, the number of data items in the container.

•  The criteria of computational complexity often include the number of comparison tests and the number of assignment statements used by the algorithm.


Running Time

•  The term running time is often used to refer to the computational complexity/time efficiency (how fast the algorithm).

•  Approaches: •  Theoretical analysis • Empirical Analysis

•  Empirical analysis: •  Count the number of times specific operations are performed by

executing an instrumented version of the program. •  Measure directly the actual program-execution time in a run.

Example: Instrumented code: countComparisons++; //initialized elsewhere

if (x < y) small = x; else small = y;


Running Time

•  The running time depends on the input (e.g., an already sorted sequence is easier to sort). •  Parameterize the running time by the size of the input n •  Seek upper bounds on the running time T(n) for the input

size n, because everybody likes a guarantee. •  T(n) function counts the frequency of the basic operations

in terms of n. •  Number of times the instructions in the algorithm are

executed.


Theoretical analysis of running time

• Running time T(n) is analyzed by determining the number of repetitions of the basic operation as a function of input size (n) •  Basic operation: the operation that contributes most towards the

running time of the algorithm •  Parameterize the running time by the size of the input n •  Seek upper bounds on the running time T(n) for the input size n,

because everybody likes a guarantee.

running time execution time

for basic operation Number of times basic operation is executed

input size

T(n) ≈ copC(n)


Example on Analyzing Running Time

1. n = read input from user 2. sum = 0 3. i = 0 4. while i < n 5. number = read input from user 6. sum = sum + number 7. i = i + 1 8. mean = sum / n

•  Pseudo code algorithm illustrates the calculation of the mean (average) of a set of n numbers:

•  The running time/computing time/time complexity/time efficiency of this algorithm in terms of input size n is:

T(n) = 4n + 5.

Statement Number of times executed 1 1 2 1 3 1 4 n+1 5 n 6 n 7 n 8 1


Another Example of Runtime Analysis

The total running time, T(N), for this algorithm is: T(N) = (2N2 + 3N + 1) * tasst + (N2 + 2N + 1) * tcomp + (4N2 + 2N) * tadd + (3N2 + N) * tmult

= c1N2 + c2N + c3

1: for ( i = 0; i < N; ++ i ) { 2: a [ i ] = 0; 3: for ( j = 0; j < N; ++ j ) 4: a [ i ] = a [ i ] + i * j ; 5: }

c1 = 2tasst + tcomp + 4tadd + 3tmult c2 = 3tasst + 2tcomp + 2tadd + tmult c3 = tasst + tcomp


Example: Sequential Search

Statement Number of times executed 1 1 2 ? 3 ? 4 1

1. 2. 3. 4.


Best-case, Average-case, Worst-case

For some algorithms efficiency depends on form of input:

• Worst case: Tworst(n) – maximum running time over inputs of size n •  An upper bound on the running time for any input •  guarantee that the algorithm will never take longer •  The worst case can occur fairly often

• Best case: Tbest(n) – minimum over inputs of size n

• Average case: Tavg(n) – “average” over inputs of size n •  Number of times the basic operation will be executed on typical input •  NOT the average of worst and best case •  Expected number of basic operations considered as a random variable

under some assumption about the probability distribution of all possible inputs

•  May be difficult to calculate/define what “average” means


Back to Sequential Search Example

• Worst case: Tworst(n) =

• Best case: Tbest(n) =

• Average case: Tavg(n) =


Order of growth

• Example: T(n) = cn2

•  How much faster will algorithm run on computer that is twice as fast?

•  How much longer does it take to solve problem of double input size?

• Main Idea: •  Ignore machine-dependent constants.

•  Look at growth of T(n) as n à

“Asymptotic Analysis”

∞


Asymptotic Notation: Big-Oh

•  The basic idea of big-Oh notation is: •  Suppose f and g are both real-valued functions

of a real variable x.

•  If, for large values of n > n0, the graph of f lies below the graph of some multiple (c) of g such that: f(n) ≤ c g(n) when n ≥ n0 à f is of order g, i.e., f(n) = O(g(n)). à The growth rate of f(n) is less than or equal to the growth rate of g(n) à g(n) represents an upper bound on f(n).

•  Also called asymptotic upper bound.

• Big Oh notation is used to capture the most dominant term in a function, and to represent the growth rate.

•  Ignores constant factors and small input sizes


• Ex1: f(n) = 5n, g(n) = n, is f = O(g)? •  we have to show the existence of a constant c as given in Definition 1.

Clearly 5 is such a constant so f(n) = 5 * g(n).

•  We could choose a larger C such as 6, because the definition states that f(n) must be less than or equal to C * g(n), but we usually try and find the smallest one.

• Ex2: f(n) = 4n+5, g(n) = n, is f = O(g)? •  we need to produce a constant c such that: f(n) <= c * n for large n .

•  If we try c = 4, this doesn't work because 4n + 5 is not less than 4n. We can set c to at least 9 to for all values of n. If n = 1, c has to be 9, but c can be smaller for greater values of n (if n = 100, C can be 5).

• Ex3: f(n) = n2, g(n) = n, is f = O(g)?

Big Oh Examples


• Ex4: f(n) = 4n2, g(n) = n2, is f = O(g)? •  Similar to previous example, clearly we can choose c to be 5 so f(n) = 5 *

g(n).

• Ex5: f(n) = n2 + 3n - 1, g(n) = n2, is f = O(g)? •  f(n) = n2 + 3n – 1

< n2 + 3n (subtraction makes things smaller so drop it) <= n2 + 3n2 (since n <= n2 for all integers n) <= 4n2

•  Therefore, if C = 4, we have shown that f(n) = O(n2).

•  O(n2): reads “order n-squared” or “Big-Oh n-squared”

•  Notice that all we are doing is finding a simple function that is an upper bound on the original function. Because of this, we could also say that: f(n) = O(n3) since (n3) is an upper bound on n2

Big Oh Examples


• Ex6: f(n) = 2n2. Then •  f(n) = O(n4) •  f(n) = O(n3) •  f(n) = O(n2) (best answer, asymptotically tight)

• Rule of thump: Drop low-order terms; ignore leading constants.

• Examples:

•  100n3 + 30000n è O(n3)

•  3n3 + 90n2 – 5n + 6046 è O(n3)

•  100n3 + 2n5+ 30000n è O(n5)

Big Oh Examples


Omega (Ω): •  We say that f(n)= Ω(g(n)) iff there exists

positive constants c, and n such that c g(n) ≤ f(n) for all n ≥ n0

•  Example: 3n3 + 90n2 – 5n + 6046 = Ω(n3)

Ω-notation & Θ-notation

Theta (Θ): •  We say that f(n)= Θ(g(n)) iff there exist

positive constants c1, c2, and n0 such that c1 g(n) ≤ f (n) ≤ c2 g(n) for all n ≥ n0

•  In other words f(n)= Θ(g(n)) iff f(n) = O(g(n)) and f(n) = Ω(g(n))

•  Example: 3n3 + 90n2 – 5n + 6046 = Θ(n3)


Functions in order of increasing growth rate

Function Name C Constant LogN Logarithmic Log2N Log-squared N Linear NlogN NlogN N2 Quaratic N3 Cubic 2n Exponential


Functions in order of increasing growth rate


Big Oh Examples

•  N2 / 2 – 3N = O(N2) •  1 + 4N = O(N) •  7N2 + 10N + 3 = O(N2) = O(N3) •  log10 N = log2 N / log2 10 = O(log2 N) = O(log N) •  sin N = O(1); 10 = O(1), 1010 = O(1) • 

•  •  log N + N = O(N) •  logk N = O(N) for any constant k •  N = O(2N), but 2N is not O(N) •  210N is not O(2N)

)( 21

NONNiN

i=⋅≤∑ =

)( 3212 NONNiN

i=⋅≤∑ =


Constant Time Algorithms: An algorithm is O(1) when its running time is independent of the number of data items. The algorithm runs in constant time.

•  e.g., direct insert at rear of array involves a simple assignment statement and thus has efficiency O(1)

Linear Time Algorithms: An algorithm is O(n) when its running time is proportional to the size of the list.

•  e.g., sequential search. When the number of elements doubles, the number of operations doubles.

Special Cases

front rear

Direct Insert at Rear

Sequential Search for the Minimum Element in an Array

32 46 8 12 3

minimum element found in the list after n comparisons

n = 51 2 3 4 5


nn2

log2n

Polynomial Algorithms: • Algorithms with running time O(n2) are quadratic.

•  practical only for relatively small values of n. •  Whenever n doubles, the running time of the algorithm increases by

a factor of 4. • Algorithms with running time O(n3) are cubic.

•  efficiency is generally poor; doubling the size of n increases the running time eight-fold.

Logarithmic Time Algorithms: The logarithm of n, base 2, is commonly used when analyzing computer algorithms

•  Ex. log2(2) = 1, log2(75) = 6.2288 •  When compared to the functions

n and n2, the function log2 n grows very slowly

Special Cases


Comparison of different O()

n log2n n log2n n2 n3 2n 2 1 2 4 8 4 4 2 8 16 64 16 8 3 24 64 512 256 16 4 64 256 4096 65536 32 5 160 1024 32768 4294967296 128 7 896 16384 2097152 3.4 x 1038 1024 10 10240 1048576 1073741824 1.8 x 10308 65536 16 1048576 4294967296 2.8 x 1014 Forget it!


Algebraic Rule 1: f(n) + g(n) ∈ O(f(n) + g(n))

Algebraic Rule 2: f(n) * g(n) ∈ O(f(n) * g(n))

Algebraic Rule 3: O(c * f(n)) = O(f(n))

The Algebra of Big-O


Proof of Algebraic Rule 3: O(c * f(n)) = O(f(n))

1.  Given running time T(n) = O(c * f(n))

2.  By O() definition, ∃c1,n0|n>n0 à T(n) <= c1(c * f(n))

3.  à ∃c1,n0|n>n0 à T(n) <= (c1 * c) f(n))

4.  Let c2 = c1 * c à ∃c1,n0|n>n0 à T(n) <= c2 * f(n))

5.  à T(n) = O(f(n))

6.  à O(c * f(n)) = O(f(n))

Similary, O(c1 * f(n) + c2 * g(n)) = O(f(n) + g(n))



Algebraic Rule 4: Larger Terms Dominate a Sum

if∃n0|∀n>n0, f(n)>=g(n),

then O(f(n) + g(n)) = O(f(n))



Proof of Algebraic Rule 4: ∀n>n0, f(n)>=g(n),

then O(f(n) + g(n)) = O(f(n))

1.  Given running time T(n) = O(f(n) + g(n))

2.  By O() definition, ∃c,n1|n>n1 à T(n) <= c(f(n) + g(n))

3.  Assume ∀n>n0, f(n)>=g(n) then n > max(n0,n1) à T(n) <= c (f(n) + g(n)) … (1) n > max(n0,n1) à f(n) > g(n) … (2)

4.  Using (2) to replacing g(n) by f(n) in (1) then n > max(n0,n1) à T(n) <= c (f(n) + f(n)) … (3) n > max(n0,n1) à T(n) <= 2c * f(n) … (4)

5.  à T(n) = O(f(n))



Algebraic Rule 5: Logarithms are Fast

∀k ≥ 0, O(logk(n)) ⊂ O(n)



Analysis Rules for Program

Statements


1.  Expressions and Assignments

•  The complexity of an expression is the sum of the complexity of all of the operations within it.

•  Assignment statements have a complexity equal to the sum of the complexities of the expressions to either side of the “=” operator, plus the complexity of the actual copy.

2.  Function Calls

•  Function/Procedure calls are counted as the complexity of their bodies.

3.  Loops

•  Compute the run time by adding up the time required for all the iterations.

Analysis Rules for Program Statements

Individual Statements


3.  Loops

•  Compute the run time by adding up the time required for all the iterations.

•  The running time of a loop is at most:

where tinit is the time required to do the loop initialization, tcondition is the time to evaluate the loop condition, with tfinal condition being the time to evaluate the loop condition the final time (when we exit the loop), and tbody is the time required to do the loop body.


Example: A simple loop for ( int j = 0; j < n; ++ j ) a [ i +10* j ] = i + j ;

tloop = O(cinit + Σ (ccondition + cincrement + cbody) + ccondition) = O(c0 + n * c1) = O(n)



3.  Conditional Statement

•  The worst case time for the if is the slower of the two possibilities.

•  The running time of an if-then-else is at most:

where tcondition is the time to evaluate the if condition, with tthen is the time to do the ‘then’ body, and telse is the time required to do the ‘else’ body.

•  A missing else clause (or, for that matter, any “empty” statement list) is O (1).




1.  Sequences of Statements

•  The time for consecutive (straight-line) statements is the sum of the individual statements.

2.  Nested Statements

•  In general, nested statements are evaluated by applying the other rules from the inside out.


Example: A simple nested loop for ( int i = 0; i < n; ++ i ) { for ( int j = i ; j < n; ++ j ) { a [ i ] [ j ] = a [ j ] [ i ] ; } }

Combining Statements


2.  Nested Statements (contd’) •  In general, nested statements are evaluated by applying the other

rules from the inside out.


Example: A simple nested loop for ( int i = 0; i < n; ++ i ) { for ( int j = i ; j < n; ++ j ) { a [ i ] [ j ] = a [ j ] [ i ] ; } }

Combining Statements


Analysis of Sorting/Searching

Algorithms


Another Example: Insertion Sort

Consider A[j]=9. Not in the correct place. Need to make room for 9. We shift all elements right, starting from 10.


Example of Insertion Sort


Algorithm of Insertion Sort


Example of Insertion Sort

outer loop

inner loop


Code of Insertion Sort

void insertionSort (int *a , int size) { int i , j , n = size; int key; // place a[i] into the sublist // a[0] . . . a[i-1], 1 <= i < n, // so it is in the correct position for (int j=1; j<n; ++j) // outer loop { key = a[j]; i=j-1; while ((i > 0) && (key < a[i] )) // inner loop { a[i+1] = a[i]; i = i - 1; } a[i+1] = key; } }


Comments on Insertion Sort

•  If the key value is greater than all the sorted elements in the sublist, the algorithm does 0 iterations of the inner loop.

•  If the initial array is sorted: •  Then each new key will be greater than all the ones already

inserted into the array. •  For each outer loop iteration, the algorithm does 0 iterations of

the inner loop.

•  This special case is very common. Many practical problems require the construction of a sorted sequence of elements from a set of data that is already sorted or nearly so, with only a few items out of place.

•  Note the "work from the back” is very efficient for such inputs.


Algorithm of Insertion Sort


Insertion sort analysis

•  T(N) counts number of comparisons •  T(N) = 1 + 2 + 3 + … + (n-1)

= (n-1)n/2 = n2/2 – n/2

•  T(N) = O(n2)

Worst-case Analysis

Best Case Analysis •  T(N) = 1 + 1 + 1 + … + 1

= n T(N) = O(n)

Average Case Analysis •  T(N) = O(n2)


Sequential Search

int seqSearch (const int arr[ ], int first, int last, int target) // Look for target within a sorted array arr [first . . last -1]. // Return the position where found, or last if not found . { int i = first; while (i < last && arr[i] != target ) i++; return i ; } 6 4 2 9 5 10

index = seqSearch(arr, 0, 8, 3);

7Index 0 1 2 3 4 5 6 7

3

target = 38

match at index = 5return index 5

• Search algorithm start with a target value and employ sequential visit to the elements looking for a match.

•  If target is found, the index of the matching element becomes the return value.

0 1 2 6 7 8

first last

Array arr

543


Sequential Search for a Sorted Array

int seqSearch (const int arr[ ], int first, int last, int target) // Look for target within a sorted array arr [first . . last -1]. // Return the position where found, or last if not found . { int i = first; while (i < last && arr[i] < target ) i++; return i ; }

• Previous code for any general array (i.e., non sorted).

• When arrays have been sorted, they could be searched using a slightly faster variant of the sequential sort.

•  If array is sorted, then encountering a value greater than the target would imply that the target value is not in the array.

•  Replacing the != operator by a < causes the algorithm to exit the loop as soon as we see a value that is equal to or greater than the target .


Binary Search

•  An alternative method of searching sorted arrays is the binary search

Example: Search for number 85 in the 63-entry list

Six probes are needed with a 63-entry list, in the worst case

1 2 5 6 8

12 16 17 18 21 23 24 30 32 33 35 38 40 44 45 47

49 54 57 58 59 64 66 67 69 70 71 74 75 77 80 81 83 85 87 88 90

91 93 99

100 101 102 105 107 110 111 116 117 118 120 122 125 126 128 130 131 133

1

2

3

4

5

First Last Middle Entry Outcome 1 63 32 71 > 33 63 48 102 < 33 47 40 87 < 33 39 36 80 > 37 39 38 83 > 37 37 37 85 =


Binary Search Tree

1 2 5 6 8

12 16 17 18 21 23 24 30 32 33 35 38 40 44 45 47

49 54 57 58 59 64 66 67 69 70 71 74 75 77 80 81 83 85 87 88 90

91 93 99

100 101 102 105 107 110 111 116 117 118 120 122 125 126 128 130 131 133 1

5

8

16

18

23

30

33

38

44

47

54

58

64

67

70

74

77

81

85

88

91

99

101 105

110 116

118 122

126 130

133

2 12 21 32 40 49 59 69 75 83 90 100 107 117 125 131

6 24 45 66 80 93 111 128

17 57 87 120

35

71

102

Example 1: Find 47

< 71

> 35

< 57

> 45

< 49

Found 63-item list

Example 2: Find 112

> 71

> 102

< 120

> 111

< 117

< 116 Not found


Binary Search analysis

•  T(N) counts number of comparisons •  T(N) = O(log n)

Worst-case Analysis

Best Case Analysis •  T(N) = O(1)

Average Case Analysis •  T(N) = O(log n)


Code of Binary Search int binSearch (const int arr[ ], int first, int last, int target) // Look for target within a sorted array arr [first . . last -1]. // Return the position where found, or , or last if not found { int mid; // index of the midpoint int midValue ; // object that is assigned arr[mid] int origLast = last ; // save original value of last // repeatedly reduce the area of search // until it is just one element while (first < last) { // test for nonempty sublist mid = (first + last) / 2; midValue = arr[mid] ; if (target == midValue) return mid; else if (target < midValue) last = mid; // search lower sublist , reset last else first = mid+1; // search upper sublist , reset first } return origLast; }


Recursion


The Concept of Recursion

•  Let’s consider the problem of evaluating the power xn where x is real number and n is a nonnegative integer.

Double power(double x, int n){

double product = 1; int i;

for (i = 1; i <= n; i++) product *= x; //xn = x*…*x (n times)

return product; }

Iterative Approach

•  Another approach, since xn = xm * x(n-m), we can split the problem to smaller problems. For example, 215 could be computed as 25 * 210 = 32 * 1024.

•  Recursion, in simple, is solving a problem by solving smaller problems of “same” form and using their results to compute the final result.

•  Recursion is the process a function goes through when one of the steps of the function involves invoking the function itself. A procedure that goes through recursion is said to be 'recursive'.


The Concept of Recursion

Double power(double x, int n){

if (n == 0) return 1; else return x * power(x, n-1); }

Recursive Approach

•  A function exhibits recursive behavior when it can be defined by two properties:

•  A simple base case (or cases) (stopping condition) •  A set of rules that reduce all other cases toward the base

case

•  Example: Computing xn

•  Base case: x0 = 1 •  Rule: xn = xn-1 * x

Double factorial(int x){

if (x == 0) return 1; else return x * factorial(x-1); }

Recursive Approach

•  Example: Computing x! •  Base case: 0! = 1 •  Rule: x! = x * (x-1)!


Fibonacci Numbers

int fib(int n){

if (n <= 1) return n; else return fib(n-1) + fib(n-2); }

Recursive Approach

•  Fibonacci numbers are the sequence of integers 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, …

•  Example: Computing fib(n) //Fibonacci element with index n

•  Base case: fib(0) = 0 fib(1) = 1

•  Rule: fib(n) = fib(n-1) + fib(n-2) int fibiter(int n){

int oneback = 1, twoback = 0, current, i;

if (n <= 1) return n; else for (i=2; i <= n; i++){ current = oneback + twoback; twoback = oneback; oneback = current; } return current; }

Iterative Approach


Evaluating Fibonacci Numbers #include <iostream> #include “d_timer.h”

int fibiter(int n){ //from previous slide . . . . . }

int fib(int n){ //from previous slide . . . . . }

int main(){ timer t1, t2;

t1.start(); fibiter(45); t1.stop(); t2.start(); fib(45); t2.stop();

cout << Time required by Iterative version is “ << t1.time() << “ sec” << endl; cout << Time required by Recursive version is “ << t2.time() << “ sec” << endl;

return 0; }


Tower of Hanoi

Needle A

. . . . . . . .

Needle CNeedle B Needle C

. . . . . . . .

Needle BNeedle A


Tower of Hanoi

Needle A Needle B Needle C

1

Needle B Needle CNeedle A

2

3

Needle A Needle B Needle C Needle A Needle B Needle C


4


56


7


Questions?

Fall 2014 CS 361 - Advanced Data Structures and Algorithms

•  Due Sun Sep 14, 11:59pm

•  Written Assignment •  Weiss, Chapter #2:

Assignment #2: Algorithm Analysis

Question # Page # 1 71 7 71

8.(a, b, & e) 72 12 73 25 74

26.(a-d) 75

•  Programming Assignment •  Weiss, Chapter #2:

You program should read “inarray.txt” file that contains the array elements in format identical to one shown in the question. Then, it outputs the majority element to the standard output.

Question # Page # 26.e 75

Fall 2014 CS 361 - Advanced Data Structures and Algorithms

•  Due Sun Sep 14, 11:59pm

Assignment #2: Algorithm Analysis

•  Submission Format:

•  Written Assignment •  Create single PDF file with name: cs361_assignment_2 •  Have a cover page with your name and your email •  Submit through Blackboard.

•  Programming Assignment •  Make sure your program compiles and executes using g++ on Dept’s Linux

machines. •  Creat a “Readme.txt” file that list how to compile and execute your program.

Include your name and your email. •  Your main file should be named “main.cpp”. •  You should have a Makefile. •  Zip all files (.h, .c, Makefile, etc.) and name it: progam1.zip •  Submit through Blackboard.

Download - Lec 02 Intro Algorithms(2)

Top Related