instructor neelima gupta [email protected]. introduction to some tools to designing algorithms...

InstructorNeelima Gupta

[email protected]

Introduction to some tools to designing algorithms through Sorting

• Iterative• Divide and Conquer

Iterative Algorithms: Insertion Sort – an example

x1,x2,........., xi-1,xi,.......…,xn

For I = 2 to nInsert the ith element xi in the partially sorted list x1,x2,........., xi-1.

(at rth position)

An Example: Insertion Sort

15 8 7 101 2 3 4

12 5 5 6

InsertionSort(A, n) {for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key) {

A[j+1] = A[j]j = j - 1

}

At Iteration 1: key = 8

Thanks Brijesh Kumar (08) : MCA -12


8 15 7 101 2 3 4

12 55 6



A[j+1] = A[j]j = j - 1

}




7 8 15 101 2 3 4

12 55 6




A[j+1] = A[j]j = j - 1

}Thanks Brijesh Kumar (08) : MCA -12


7 8 10 151 2 3 4

12 55 6



A[j+1] = A[j]j = j - 1

}




7 8 10 121 2 3 4

15 55 6




A[j+1] = A[j]j = j - 1



5 7 8 101 2 3 4

12 155 6

Final Output



A[j+1] = A[j]j = j - 1


Analysis: Insertion Sort

Thanks : MCA 2012 Dharam Deo Prasad

InsertionSort(A, n) {

for i = 2 to n {

key = A[i]j = i - 1;while (j > 0) and (A[j] > key)

{A[j+1] = A[j]j = j - 1

}A[j+1] = key

} }

Statement C N

InsertionSort(A, n) {for i = 2 to n { c1 n

key = A[i] c2 (n-1)j = i - 1; c3 (n-1)while (j > 0) and (A[j] > key) c4 Σ(Ti + 1) {

A[j+1] = A[j] c5 Σ Ti j = j - 1 c6 Σ Ti

}A[j+1] = key c7 (n-1)

}} where Ti is number of while expression evaluations for the ith for loop iterationCi is the constant time required for 1 execution of the statementN is the number of times the statement is executed

Running Time Analysis


Total time


• T(n) = (c1 + c2 + c3 + c7 )n – (c2 + c3 + c7) + [(c4 + c5 + c6) Ti + c4 ]

n

i=2

∑

Worst Case


Worst case: Ti = i – 1

i.e. Ti = (i – 1)

= n(n-1)/2hence, T(n) = (c1 + c2 + c3 + c4 + c7 )n – (c2 +c3 + c4 + c7) +

(c4 + c5 + c6)n(n-1/2)= an2 + bn + c

where,a = 1/2 (c4 + c5 + c6)

b = -1/2 (c4 + c5 + c6) + (c1+

c2 + c3 + c4 + c7 )

c = -(c2 + c3 + c4 + c7)

n

i=2

∑n

i=2

∑

Best Case


Best case: Ti = 1

i.e. Ti = 1

=(n – 1)hence, T(n) = (c1 + c2 + c3 + c4 + c7 )n – (c2 +c3 + c4 + c7) +

(c4 + c5 + c6)(n-1)= an + b

where,a = c1 + c2 + c3 + 2c4

+ c5 + c6 + c7

b = -(c2 + c3 + 2c4 + c5

+ c6 + c7 )

n

i=2

∑n

i=2

∑

Analysis of AlgorithmsBefore we move ahead, let us define the

notion of analysis of algorithms more formally

Input SizeTime and space complexity

This is generally a function of the input sizeHow we characterize input size depends:

Sorting: number of input items Multiplication: total number of bits Graph algorithms: number of nodes & edges Etc

Lower BoundsPlease understand the following

statements carefully.Any algorithm that sorts by removing at

most one inversion per comparison does at least n(n-1)/2 comparisons in the worst case.

Hence Insertion Sort is optimal in this category of algorithms.

Optimal?What do you mean by the term “Optimal”?

Answer: If an algorithm runs as fast in the worst case as it is possible (in the best case), we say that the algorithm is optimal. i.e if the worst case performance of an algorithm matches the lower bound, the algorithm is said to be “optimal”

Inversion :-

Example : 4, 2, 3

No. of pairs = 3C2 , out of which (4,2) is out of order i.e. inversion(2,3) is in order(4,3) inversion

Thanks to:Dileep Jaiswal (11) :MCA 2012

In ‘n’ elements there will be n(n-1)/2 inversions in worst case.

Thus, if an algorithm sorts by removing at most one inversion per comparison then it must do at least n(n-1)/2 comparisons in the worst case.


Insertion Sort

x1,x2,…...,xk-1, xk,...........…..,xi-1,xi

Let xi is inserted after xk-1

No. of comparisons = (i-1) – (k – 1) +1 = i – k + 1 No. of inversions removed = (i-1) – (k – 1) = i – k No. of inversions removed/comparison

= (1-k+1)/(i-k)

< = 1


Thus Insertion sort falls in the category of comparison algorithms that remove at most one inversion per comparison.

Insertion sort is optimal in this category of algorithms .


SELECTION SORTThe algorithm works as follows:Find the maximum value in the array.Swap it with the value in the last positionRepeat the steps above for the remainder of

the array.

Thanks to: MCA 2012 Chhaya Nathani(9)

Selection Sort : An Example For i = n to 2

a) Select the maximum in a[1].......a[i]. b) Swap it with a[i].


Selection Sort

x1,x2,…, xk,...........……, xn

Let the maximum is located at xk

Swap it with xn

ContinueThanks to:Dileep Jaiswal (11) :MCA 2012

Selection Sort

x1,x2,…, xk,...........……, xn-i+1,……xn

Suppose we are at the ith iteration

Let the maximum is located at xk

Swap it with xn-i+1


5 20 6 4 15 3 10 2


An Example: Selection Sort

5 2 6 4 15 3 10 20



5 2 6 4 10 3 15 20




5 2 6 4 3 10 15 20


Thanks to: MCA 2012 Chhaya Nathani(9)Thanks to: MCA 2012 Chhaya

Nathani(9)

5 2 3 4 6 10 15 20



Nathani(9)Thanks to: MCA 2012 Chhaya

Nathani(9)

4 2 3 5 6 10 15 20





Nathani(9)

3 2 4 5 6 10 15 20





Nathani(9)

2 3 4 5 6 10 15 20

DONE!!!!


SelectionSort(A, n) { for i = n to 2 {

max=i for j=i-1 to 1

{If a[j]>a[max]Max=j }

If(max!=i){Swap a[i]……a[max]}


ANALYSIS: SELECTION SORTSelecting the largest element requires

scanning all n elements (this takes n − 1 comparisons)

and then swapping it into the last position. Finding the next largest element requires scanning the remaining n − 1 elements and so on...

(n − 1) + (n − 2) + ... + 2 + 1 = n(n − 1) / 2 i.e Θ(n2) comparisonsT(n)= Θ(n2)


Selection Sort

x1,x2,…, xk,...........……, xn-i+1,……xn

Suppose we are at the ith iteration

Let the maximum is located at xk No. of comparisons = (n – i + 1) - 1 = n - i No. of inversions removed = (n – i + 1) – 1 – (k-1) = n – i

– k + 1No. of inversions removed/comparison

= ( n – i – k +1)/ (n – i) < = 1 (since k >= 1)


Thus Selection sort also falls in category of comparison algorithms that remove at most one inversion per comparison.

Selection sort is also optimal in this category of algorithms .


Merge Sort(Divide and Conquer

Technique)

Divide the list into nearly equal halves Sort each half recursively Merge these lists

Thanks to:Gaurav Gulzar (MCA -11)

18 9 15 13 20 10 7 5

18 9 15 13 20 10 7 5

18 9

18

15 13

9 15 13 20 10 7 5

20 10 7 5

Divide into 2

Subsequence

Thanks to Himanshu (MCA 2012, Roll No 14)

Next Sort the 2 subsequences and merge them.


5 7 9 10 13 15 18 20

9 13 15 18 5 7 10 20

9 18

18

13 15

9 15 13 20 10 7 5

10 20 5 7

Sort & merge 2 Subsequence

Initial Subsequence

Sorted Sequence


MergingLet we have two sorted lists :

A:

B:

Compare a1 with b1 and put smaller one in new array C and increase the index of array having smaller element.


a1 a2 a3 ……………. an

b1 b2 b3 ……………. bm

Let a1 < b1

A:

B:

Sorted Array C

C:


a1 a2 a3 ……………. an

b1 b2 b3 ……………. bm

a1

Example 1:

A:

B:

Sorted Array C

C:

20

40

10

30

35

3845

10

20

50

52


60

80

Example 1:

A:

B:

Sorted Array C

C:

20

40

10

30

35

3845

10

20

30

35

50

52


60

80

Example 1:

A:

B:

Sorted Array C

C:

20

40

10

30

35

3845

10

20

30

35

50

52

38


60

80

Example 1:

A:

B:

Sorted Array C

C:

20

40

10

30

35

3845

10

20

30

35

50

52

38 40 45


60

80

Example 1:

A:

B:

Sorted Array C

C:

20

40

10

30

35

3845

10

20

30

35

50

52

38 40 45 50


60

80

Example 1:

A:

B:

Sorted Array C

C:

20

40

10

30

35

3845

10

20

30

35

50

52

38 40 45 50 52


60

80

Example 1:

A:

B:

Sorted Array C

C:

20

40

10

30

35

3845

10

20

30

35

50

52

38 40 45 50 52


60

80

60 80

Worst Case Analysis

If no. of elements in first list is ‘n’ & in second list is ‘m’ then:

Total No. of comparisons = n-1+m

i.e. O(m + n) in worst case and


What is the best case?

Number of Comparisons in the best case:

min(n , m), i.e. Ω(min(n , m))


Example 2:

A:

B:

Sorted Array C

C:

5 8

10

30

35

3845

5 8

50

52


Example 2:

A:

B:

Sorted Array C

C:

5 8

10

30

35

3845

5 810

30

50

52


Example 2:

A:

B:

Sorted Array C

C:

5 8

10

30

35

3845

5 810

30

50

52

35


Example 2:

A:

B:

Sorted Array C

C:

5 8

10

30

35

3845

5 810

30

50

52

35

38


Example 2:

A:

B:

Sorted Array C

C:

5 8

10

30

35

3845

5 810

30

50

52

35

38

45


Example 2:

A:

B:

Sorted Array C

C:

5 8

10

30

35

3845

5 810

30

50

52

35

38

45

50


Example 2:

A:

B:

Sorted Array C

C:

5 8

10

30

35

3845

5 810

30

50

52

35

38

45

50

52


What is the best case?Arrays

Total number of steps: Ω(m+n)…copying the rest of the elements of the bigger array.

Linked List :Total number of steps:

min(n , m), i.e. Ω(min(n , m)) in best case.

Analysis of Merge Sort Since size of both the lists to be merged is n/2, in either case (arrays or linked list), time to merge the two lists is Θ(n).

If T(n) = no. of comparisons performed on an Input of size ‘n’ then :

T(n) = T(n/2) + T(n/2) + Θ(n) = 2T(n/2) +cn ∀ n>= n0

∴ T(n) = O(nlogn)


If T(n) = no. of comparisons performed on an Input of size ‘n’ then :

T(n) = T(n/2) + T(n/2) + Θ(n) = 2T(n/2) +cn ∀ n>= n0

∴ T(n) = O(nlogn)


Final PointsMerging is Θ(m + n) in case of an ArrayIf we use link list then it is Ω(min(m , n))Time for merging is O(n) and Ω(n/2) i.e.

Θ(n)


Conclusion Merge Sort = Θ(nlogn)Have we beaten the lower bound?

No,It just means that merge sort does not fall in the previous category of algorithms. It removes > 1 inversions per comparisons.


What is the Worst case

Best Case

for Merge Sort?

Merge Sort Vs Insertion Sort

What is the advantage of merge sort?

What is the advantage of insertion sort?

Merge Sort Vs Insertion Sort contd..Merge Sort is faster but does not take

advantage if the list is already partially sorted.

Insertion Sort takes advantage if the list is already partially sorted.

Lower BoundAny algorithm that sorts by comparison only

does at least (n lg n) comparisons in the worst case.

Decision Trees• Provides an abstraction of comparison sorts. In a decision tree, each node represents a comparison. Insertion sort applied on x1, x2, x3

x1:x2

x2:x3

x1:x3

x3:x1

x1<x2<x3

x3>x1>x2

x2:x3

x2>x1>x3

x2>x3>x1

x1>x2>x3

x1>x3>x2

x2>x

3

x1<x2

x3>x2

x3<x

1

x3>x1

x1>x2

x3>x

1

x1>x3

x2>x3

x3>x2

•What is the minimum number of leaves in a decision tree?

•Longest path of the tree gives us the height of the tree, and actually represents the worst case scenario for the algorithm.

•height of a decision tree = Ω (n log n)i.e. any comparison sort will perform at least (n logn) comparisons in the worst case.

Decision Trees

Proof : •Let h denotes the height of the tree. •What’s the maximum # of leaves of a binary tree of height h? : 2h

•2h >= number of leaves >= n! (where n = no. of elements and n! is a lower bound on the no. of leaves in the decision tree)

=> h>= log(n!) > n log n ( By Stirling’s Approximation)

( SA: n!= √ (2.π.n) .(n/e)n

> (n/e)n

Thus, log(n!) > n log n – n log e > n logn. )

Merge Sort is OptimalThus the time to comparison sort n elements is

(n lg n)Corollary: Merge-sort is asymptotically optimal

comparison sorts.Later we’ll see another sorting algorithm in

this category namely heap-sort that is also optimal.

We’ll also see some algorithms which beat this bound. Needless to say those algorithms are not purely based on comparisons. They do something extra.

Quick Sort(Divide and Conquer

Technique) Pick a pivot element x Partition the array into two subarrays around

the pivot x such that elements in left subarray is less than equal to x and element in right subarray is greater than x

Recursively sort left subarray and right subarray

Thanks to:Krishn Kant Kundan (MCA -19)

x ≤ x >x

Quick Sort (Algorithm)

QUICKSORT(A, p, q)if p < q

k=PARTITION(A, p, q) QUICKSORT(A, p, k-1) QUICKSORT(A, k+1, q)


Quick Sort (Example)Let we have following elements in our array :-

i j

i j

i j


27 14 9 22 8 41 56 31 15 53 99 11 30 24

14 27 9 22 8 41 56 31 15 53 99 11 30 24

14 9 27 22 8 41 56 31 15 53 99 11 30 24

Quick Sort (Example)

i j

i j

i j


14 9 22 27 8 41 56 31 15 53 99 11 30 24

14 9 22 8 27 41 56 31 15 53 99 11 30 24

14 9 22 8 27 24 56 31 15 53 99 11 30 41


i j

i j

i j


14 9 22 8 24 27 56 31 15 53 99 11 30 41

14 9 22 8 24 27 30 31 15 53 99 11 56 41

14 9 22 8 24 27 11 31 15 53 99 30 56 41


i j

i j

i j


14 9 22 8 24 11 27 31 15 53 99 30 56 41

14 9 22 8 24 11 27 99 15 53 31 30 56 41

14 9 22 8 24 11 27 53 15 99 31 30 56 41


i j

i j (stop)

(recursive call on left array) (recursive call on right array)


14 9 22 8 24 11 27 15 53 99 31 30 56 41

14 9 22 8 24 11 15 27 53 99 31 30 56 41

14 9 22 8 24 11 15 27 53 99 31 30 56 41

i j i j

i j i j

i j i j

i j i j

i j i j

i j i j (stop)

i j (st op)

14 9 22 8 24 11 15 53 99 31 30 56 4127

9 14 22 8 24 11 15 27 53 41 31 30 56 99

9 14 15 8 24 11 22 27 41 53 31 30 56 99

9 14 11 8 24 15 22 27 41 31 53 30 56 99

9 11 14 8 24 15 22 27 41 31 30 53 56 99

9 11 8 14 24 15 22 27 41 31 30 53 56 99

9 11 8 14 24 15 22 27 41 31 30 53 56 99

9 11 8 14 24 15 22 27 41 31 30 53 56 99

i j i j i j i j

9 8 11 14 15 24 22 27 31 41 30 53 56 99

i j i j i j i j (stop)

8 9 11 14 15 22 24 27 31 30 41 53 56 99

(stop) i j i j (stop) (stop) i j i j (stop)

8 9 11 14 15 22 24 27 31 30 41 53 56 99

Sorted Array


Analyzing QuicksortWorst Case?

Partition is always unbalancedWorst Case input?

Already-sorted input, if the first element is always picked as the pivot

Best case?Partition is perfectly balanced

Best Case input??

Worst Case and Best Case input when the middle element is always picked as the pivot?

Worst Case of QuicksortIn the worst case:

T(1) = (1)T(n) = T(n - 1) + (n)

Does the recursion look familiar?T(n) = (n2)

Best Case of QuicksortIn the best case:

T(n) = 2T(n/2) + (n)

Does the recursion familiar?T(n) = (n lg n)

Why does Qsort works well in practice?

Suppose that partition() always produces a 9-to-1 split. This looks quite unbalanced!

The recurrence is:T(n) = T(9n/10) + T(n/10) + n

T(n) = θ (n log n)Such an imbalanced partition and θ(n log n)

time?

Why does Qsort works well in practice?Intuitively, a real-life run of quicksort will

produce a mix of “bad” and “good” splitsPretend for intuition that they alternate

between best-case (n/2 : n/2) and worst-case (n-1 : 1)

What happens if we bad-split root node, then good-split the resulting size (n-1) node?

Why does Qsort works well in practice?Intuitively, a real-life run of quicksort will

produce a mix of “bad” and “good” splitsPretend for intuition that they alternate

between best-case (n/2 : n/2) and worst-case (n-1 : 1)

What happens if we bad-split root node, then good-split the resulting size (n-1) node? We end up with three subarrays, size 1, (n-1)/2, (n-

1)/2 Combined cost of splits = n + n -1 = 2n -1 = O(n) No worse than if we had good-split the root node!

Why does Qsort works well in practice?

Intuitively, the O(n) cost of a bad split (or 2 or 3 bad splits) can be absorbed into the O(n) cost of each good split

Thus running time of alternating bad and good splits is still O(n lg n), with slightly higher constants

How can we be more rigorous? : we’ll do average analysis of Qsort later while doing randomized algorithms.

Quicksort Vs Merge Sort

Merge Sort takes O(n lg n) in the worst caseQuick Sort takes O(n2) in the worst caseSo why would anybody use Qsort instead of

merge sort?Because in practice, Qsort is quick as the worst

case doesn’t happen often.

Up Next

Linear-Time Sorting Algorithms

The End

instructor neelima gupta [email protected]. introduction to some tools to designing algorithms...

Documents

thanks brijesh kumar

j 1c6 ti

ajc5 ti j

aj keyc4 ti

c2 c3 c4 c7ni

dharam deo prasadtn

dharam deo prasadthanks

c4 c5 c6b