Chapter 2.Chapter 2.

Getting StartedGetting Started

OutlineOutline

Familiarize you with the Familiarize you with the framework to think to think about the design and analysis of about the design and analysis of algorithmsalgorithms

Introduce two sorting algorithms: Introduce two sorting algorithms: insertion sort and and merge sort

Start to understand how to analyze the Start to understand how to analyze the efficiency of algorithms of algorithms

We mainly concern about running time, or speed. Other issues could also affect efficiency, e.g. memory, storage.

Example: Sorting ProblemExample: Sorting Problem

Input: A sequence of n numbersInput: A sequence of n numbers

Output: A Output: A permutationpermutation (reordering) of the (reordering) of the input sequence such that input sequence such that

1 2, ,..., na a a

' ' '1 2 ... na a a

PseudocodePseudocode

Describe algorithms as programs written Describe algorithms as programs written in a in a pseudo codepseudo code

Employ whatever expressive method is Employ whatever expressive method is most most clear and conciseclear and concise to specify a to specify a given algorithmgiven algorithm

Not concerned with issues of Not concerned with issues of software software engineeringengineering, such as data abstraction, , such as data abstraction, modularity and error handling modularity and error handling

Insertion SortInsertion Sort

It is an efficient algorithm for sorting It is an efficient algorithm for sorting a a small number of elementssmall number of elements

It works the way many people sort a hand It works the way many people sort a hand of playing cardsof playing cards

In insertion sort, the input numbers are In insertion sort, the input numbers are sorted in placesorted in place: the number are : the number are rearranged rearranged within the arraywithin the array

Insertion Sort

Find an appropriate position to insert the new card into sorted

cards in hands

Comparing and exchange in reverse

order

Prove the CorrectnessProve the Correctness

Design, Prove and AnalyzeDesign, Prove and Analyze

Often Use a Often Use a loop invariantloop invariant

How to define How to define loop invariant loop invariant is important

E.g. for insertion sort:E.g. for insertion sort:

Loop invariantLoop invariant: : At the start of each iteration of At the start of each iteration of the “outer” the “outer” for for loop (line1-8)loop (line1-8)

-- the loop indexed by -- the loop indexed by j -- j --

the sub-array the sub-array AA[1 [1 . . j-. . j-1] consists of the elements 1] consists of the elements originally in originally in AA[1 [1 .. j-.. j-1] but in sorted order.1] but in sorted order.

Loop InvariantLoop Invariant

To use a loop invariant to prove correctness, To use a loop invariant to prove correctness, show three things about it:show three things about it:Initialization: Initialization:

It is trueIt is true prior to the first iterationprior to the first iteration of the loop. of the loop.Maintenance: Maintenance:

If it is true before an iteration of the loop, it If it is true before an iteration of the loop, it remains trueremains true before the next iteration. before the next iteration.Termination: Termination: When the When the loop terminatesloop terminates, the invariant—usually , the invariant—usually along with the reason that the loop terminated—gives along with the reason that the loop terminated—gives us a useful property that helps show that the algorithm us a useful property that helps show that the algorithm is correct.is correct.

Analyzing algorithmsAnalyzing algorithmsRandom-access machine(RAM) modelRandom-access machine(RAM) model

How do we analyze an algorithm’s running How do we analyze an algorithm’s running time?time?– The time taken by an algorithm depends on the The time taken by an algorithm depends on the

input itself (input itself (e.g. already sortede.g. already sorted))– Input size: depends on the problem being studied. Input size: depends on the problem being studied.

((parameterize in input sizeparameterize in input size))– Want upper bound (Want upper bound (guarantee to usersguarantee to users))– Running time: Running time: on a particular input, it is the number on a particular input, it is the number

of of primitiveprimitive operations(stepsoperations(steps) executed.) executed.

RAM MODELRAM MODEL

Do not model the memory hierarchy

Kinds of AnalysisKinds of Analysis

Worst-case (Usually)Worst-case (Usually) T(n)=T(n)=maxmax time on any input of size n time on any input of size n

Average-case (Sometimes)Average-case (Sometimes)

T(n)=T(n)=expected timeexpected time over all input of size n over all input of size n

How do we know the probability of every particular input is?How do we know the probability of every particular input is?

I do not know. Make assumption of statistical distribution of inputs (what I do not know. Make assumption of statistical distribution of inputs (what is the common assumption? )is the common assumption? )

Best-Case (bogus)Best-Case (bogus)

Some slow algorithms work well on some input , cheating.Some slow algorithms work well on some input , cheating.

Detailed Analysis of Detailed Analysis of AlgorithmAlgorithm

n : the number of inputs

t j : the # of times the while loop test is executed.

The loop test is executed one time more than the loop body.

Thus, the running time of the above Insertion-Sort algorithm is:

T(n) = c1n + c2(n-1) + c4(n-1) + c5 j=2 .. n tj+ c6 j=2 .. n(tj -1) + c7 j=2 .. n(tj -1) + c8(n-1)

= (c5/2 + c6/2 + c7/2) n2 + (c1 + c2 + c4 + c5/2 - c6/2 - c7/2 + c8) n – (c2+c4+c5+c8).

tt j : the # of times the while loop test is executed.: the # of times the while loop test is executed.

The loop test is executed one time more than the loop body.The loop test is executed one time more than the loop body.Thus, the running time of the above Insertion-Sort algorithm is:Thus, the running time of the above Insertion-Sort algorithm is:

T(n) = cT(n) = c1n + cn + c2(n-1) + c(n-1) + c4(n-1) + c(n-1) + c5 j=2 .. n t tj+ c+ c6 j=2 .. n(t(tj -1) + c -1) + c7 j=2 .. n(t(tj -1) -1) + c+ c8(n-1)(n-1)

Detailed Analysis of Running Detailed Analysis of Running timetime

This worst-case running time can be expressed as an2+bn+c , it is thus a quadratic functionquadratic function

AnalysisAnalysis of of Insertion SortInsertion Sort– The running time of the algorithm is:The running time of the algorithm is:

(cost of statement) x ( # of times statement is (cost of statement) x ( # of times statement is executed) executed) all statementsall statements

– ttj j = # of times that while loop test is executed for that value of= # of times that while loop test is executed for that value of jj..

– BestBest case: case: the array is already sorted (all the array is already sorted (all ttjj = 1 = 1))– WorstWorst case: case:

the array is in reverse order (the array is in reverse order (ttj j = j= j).).The worst case running time gives a guaranteed upper bound The worst case running time gives a guaranteed upper bound on the running time for any input.on the running time for any input.

– AverageAverage case: case: On average, the key in On average, the key in A[ j ]A[ j ] is less than half the elements in is less than half the elements in A[1 .. j-1]A[1 .. j-1] and and it’s greater than the other half. (it’s greater than the other half. (ttjj = j /2 = j /2).).

n(n-1)/2

Order of GrowthOrder of Growth

The The abstractionabstraction to ease analysis and focus on the to ease analysis and focus on the important featuresimportant features..

Look only at the Look only at the leading termleading term of the formula for of the formula for running time.running time.

Drop lower-order terms.Drop lower-order terms.Ignore the constant coefficient in the leading term.Ignore the constant coefficient in the leading term.

Example: Example: an² + bn + c = (n²)Drop lower-order terms Drop lower-order terms an²an²Ignore constant coefficient Ignore constant coefficient n²n²The worst case running time The worst case running time T(n)T(n) grows like grows like n²n²; it does ; it does not equalnot equal n²n²..The running time is The running time is (n²) (n²) to capture the notion that to capture the notion that the order of the order of growth is n².growth is n².

Order of growth (2)Order of growth (2)

We usually consider one algorithm to be We usually consider one algorithm to be more efficient than another if its worst-more efficient than another if its worst-case running time has a lower case running time has a lower order of order of growthgrowth

Due to Due to constant factorsconstant factors and lower-order and lower-order terms, this evaluation may be error for terms, this evaluation may be error for small inputs but for small inputs but for large enough inputslarge enough inputs, it , it is true.is true.

Designing algorithmsDesigning algorithms

Divide and ConquerDivide and Conquer

DivideDivide the problem into a number of the problem into a number of subproblemssubproblems..

ConquerConquer the the subproblemssubproblems by solving them by solving them recursively..Base case: Base case:

If the subproblems are small enough, If the subproblems are small enough, just solve themjust solve them..

CombineCombine the the subproblemsubproblem solutionssolutions to give to give a solution to the original problem.a solution to the original problem.

Cf.) Cf.) Incremental methodIncremental method – insertion sort. – insertion sort.

Merge SortMerge SortA sorting algorithm based on divide and conquer.A sorting algorithm based on divide and conquer. The The worst-caseworst-case running time: running time:

TTmerge_sort < T < Tinsertion_sort in its order of growth in its order of growth

To sort To sort AA[[p . . rp . . r]:]:

DivideDivide by by splitting splitting into two into two subarrays subarrays AA[[p .. qp .. q]] and and AA[[qq+1 +1 .. r.. r],], where where q q is the halfway point of is the halfway point of AA[[p .. rp .. r].].ConquerConquer by by recursively sortingrecursively sorting the two subarrays the two subarrays AA[[p .. qp .. q] and ] and AA[[qq+1 +1 .. r.. r].].Combine Combine by by mergingmerging the two sorted subarrays the two sorted subarrays AA[[p .. qp .. q] and ] and AA[[qq+1 +1 .. r.. r] to produce a single sorted subarray ] to produce a single sorted subarray AA[[p .. rp .. r]. ]. To accomplish this step, we’ll define a procedure To accomplish this step, we’ll define a procedure

MERGEMERGE(A, p, q, r)(A, p, q, r)..

MERGE-SORT(A, 1, n)

The two largest elements in two

arrays are sentinels

Execute r-p+1 times

Merging: Merging: MERGE(A, p, q, MERGE(A, p, q, r)r)

Input: Array Input: Array AA and indices and indices p, q, rp, q, r such that such that– p p q q rr– Subarray Subarray A[p .. q]A[p .. q] is sorted and subarray is sorted and subarray A[q+1 .. r]A[q+1 .. r] is sorted. is sorted.

By the By the restrictions on p, q, rrestrictions on p, q, r, neither subarray is empty., neither subarray is empty.

Output: The two subarrays are merge into a Output: The two subarrays are merge into a single sorted single sorted subarraysubarray in in A[p .. r].A[p .. r].

T(T(nn) = ) = ((nn)), , where where n=r-p+1n=r-p+1 = the # of elements being merged. = the # of elements being merged.

What is What is n ??– The size of the original problem => the size of a subproblem.The size of the original problem => the size of a subproblem.– Use this technique when we analyze Use this technique when we analyze recursive algorithmrecursive algorithm

Line 1-3 and 8-11 takes constant time and for loop

takes (n1+n2(n1+n2) time) time

Loop Invariant of MERGELoop Invariant of MERGE

Loop InvariantLoop Invariant At the At the startstart of each iteration of the for loop of lines 12-17, of each iteration of the for loop of lines 12-17,

the subarray A[p…the subarray A[p…k-1k-1] contains k-p elements of ] contains k-p elements of L[1..nL[1..n11+1] and R[1.. n+1] and R[1.. n22+1],+1],in sorted order. Moreover, . Moreover, L[i]L[i] and and R[j]R[j] are the are the smallest elementssmallest elements of their arrays of their arrays that that have not been copied backhave not been copied back into A. into A.

Show correctness using loop invariantShow correctness using loop invariantWe show this loop invariant holds We show this loop invariant holds priori topriori to the first iteration the first iteration

of the for loop and each iteration of the for loop and each iteration maintainsmaintains the the invariant and the invariant show correctness when the invariant and the invariant show correctness when the loop loop terminatesterminates..

Loop InvariantLoop InvariantInitializationInitialization

Priori to the first iteration of the loop, we have k=p. A[p..k-1] Priori to the first iteration of the loop, we have k=p. A[p..k-1] is empty. i=j=1is empty. i=j=1

MaintenanceMaintenanceWe first suppose L[i]<=R[j]. Because A[p..k-1] contains k-p We first suppose L[i]<=R[j]. Because A[p..k-1] contains k-p

smallest elements, after line 14 copies L[i] into A[k], A[p..k] smallest elements, after line 14 copies L[i] into A[k], A[p..k] will contain the k-p+1 smallest elements. Increment k, i for will contain the k-p+1 smallest elements. Increment k, i for next iteration.next iteration.

TerminationTerminationAt termination, k=r+1. By the loop invariant, the subarray At termination, k=r+1. By the loop invariant, the subarray

A[p..k-1], which is A[p..k-1], which is A[p, r]A[p, r] contains the k-p=r-p+1 smallest contains the k-p=r-p+1 smallest elements in sorted order. elements in sorted order.

Analyzing Divide-and-Analyzing Divide-and-Conquer AlgorithmsConquer Algorithms

Use a Use a recurrencerecurrence (equation)(equation) to describe the to describe the running time of a divide-and-conquer algorithm.running time of a divide-and-conquer algorithm.

Let Let T(n)T(n) = = running time on a problem of a size running time on a problem of a size nn..– If the problem size is small enough(say, If the problem size is small enough(say, nn c for some c for some

constant c), constant c),

we have a we have a base casebase case – – c(=c(=ΘΘ(1)). (1)).

– Otherwise, suppose that we Otherwise, suppose that we dividedivide into into aa sub-problemssub-problems, ,

each each 1/ b1/ b the size of the original the size of the original..

(In merge sort, (In merge sort, a=b=2a=b=2.).)

– Let Let D(nD(n)) be the time to be the time to dividedivide a size- a size-nn problem. problem.

Continue…Continue…

– There are a sub-problems to solve, each of size There are a sub-problems to solve, each of size n/ n/ b (the division may not be equally)b (the division may not be equally)

each sub-problem takes each sub-problem takes T(T(n/ bn/ b)) time to time to solvesolve

we spend we spend aaT(T(n/ bn/ b)) time solving time solving subproblems.subproblems.

– Let Let C(n)C(n) be the time to be the time to combinecombine solutions. solutions.– We get the recurrence:We get the recurrence:

T(n)T(n) = = (1)(1) ifif nn c c

aaT(T(n/ bn/ b) + ) + D(n)D(n) ++ C(n)C(n) otherwise.otherwise.

Analyzing Merge Sort – Analyzing Merge Sort – Use a Recurrence.Use a Recurrence.

For simplicity, assume that For simplicity, assume that n=2n=2k

The base case: when The base case: when n =1n =1, , T(n)T(n)==(1). (1).

When When n n 2, 2, time for merge sort steps:time for merge sort steps: – DivideDivide:: Just compute q as the average of p and r Just compute q as the average of p and r D(n)D(n)==(1).(1).

– ConquerConquer:: Recursively solve 2 subproblems, each of size Recursively solve 2 subproblems, each of size n/ 2n/ 2

22T(T(n/2n/2))

– CombineCombine:: MERGE on an MERGE on an n element subarray takes n element subarray takes (n) (n) time time C(n)C(n)==(n).(n).

– Since Since D(n)+C(n)D(n)+C(n)==(1) (1) + + (n)(n) = = (n)(n),, the recurrence for merge the recurrence for merge sort running time is:sort running time is:

T(n)T(n) = = (1)(1) ifif n=1n=1

22T(T(n/ 2n/ 2) + ) + (n)(n) n>1n>1..

Continue…Continue…

SolvingSolving the merge-sort the merge-sort recurrencerecurrence: : T(n)T(n) = = (n log(n log22 n) n)– Let c be a constant for Let c be a constant for T(n)T(n) of the base case and of the of the base case and of the

time per array element for the divide and conquer steps.time per array element for the divide and conquer steps.– Re-wirte the recurrence asRe-wirte the recurrence as

T(n) = c ifif n=1n=1

2T(n/ 2) + cn n>1n>1..

Draw a recursion tree, which shows successive Draw a recursion tree, which shows successive expansions of the recurrenceexpansions of the recurrence. . (n is exact power of 2)(n is exact power of 2)

20

21

22

2 lg ,

# 1 lg 1

k n k n

level k n

Merge SortMerge Sort

The total cost is cn(lgn+1), which is The total cost is cn(lgn+1), which is (n lg n)(n lg n)

Since the logarithm function grows more slowly Since the logarithm function grows more slowly than any linear function, for large enough input, than any linear function, for large enough input, merge sort with its merge sort with its (n lg n) (n lg n) running time running time outperforms insertion soft, whose running time outperforms insertion soft, whose running time is is (n(n22)), in the worst case., in the worst case.

HomeworkHomework

2.1-3, 2.2-42.1-3, 2.2-4

2.3-2, 2.3-42.3-2, 2.3-4