cs 4407 algorithms lecture 2: iterative and divide and conquer...
TRANSCRIPT
-
1
CS 4407
Algorithms
Lecture 2:
Iterative and Divide and Conquer
Algorithms
Prof. Gregory Provan
Department of Computer Science
University College Cork
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Lecture Outline
Growth Functions Mathematical specification of growth functions
Iterative Algorithms Divide-and-Conquer Algorithms
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Today’s Learning Objectives
Describe mathematical principles for specifying the
growth of run-time of an algorithm
– Classify the growth functions
• , O, , o,
Iterative Algorithm Analysis
Divide-and-Conquer Algorithm Analysis
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Analyzing Algorithms
Assumptions – generic one-processor, random access machine
– running time (others: memory, communication, etc)
Worst Case Running Time: the longest time for any input
of size n – upper bound on the running time for any input
– in some cases like searching, this is close
Average Case Behavior: the expected performance
averaged over all possible inputs – it is generally better than worst case behavior
– sometimes it’s roughly as bad as worst case
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Notation: Growth Functions
Theta f(n) = θ(g(n)) f(n) ≈ c g(n)
BigOh f(n) = O(g(n)) f(n) ≤ c g(n)
Omega f(n) = Ω(g(n)) f(n) ≥ c g(n)
Little Oh f(n) = o(g(n)) f(n) > c g(n)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
-notation
(g(n)) = {f(n): there exist
positive constants c1, c2 and
n0 such that
0 c1g(n) f(n) c2g(n),
for all n n0 }
For a given function g(n),
we denote by (g(n)) the
set of functions
We say g(n) is an asymptotically tight bound for f(n)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
O-notation
For a given function g(n), we
denote by O(g(n)) the set of
functions
O(g(n)) = {f(n): there exist
positive constants c and n0
such that
0 f(n) cg(n),
for all n n0 }
We say g(n) is an asymptotic upper bound for f(n)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
-notation
For a given function g(n),
we denote by (g(n))
the set of functions
(g(n)) = {f(n): there
exist positive constants
c and n0 such that
0 cg(n) f(n)
for all n n0 }
We say g(n) is an asymptotic lower bound for f(n)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Relations Between , , O
For any two functions g(n) and f(n),
f(n) = (g(n)) if and only if f(n) = O(g(n)) and
f(n) = (g(n)).
i.e., (g(n)) = O(g(n)) (g(n)).
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Complexity of “Simple” Algorithms
Simple Interation
Dealing with Loops
– Loop invariant method
Divide and Conquer Algorithms
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Iterative Algorithm Running Time
1. n = read input from user
2. sum = 0
3. i = 0
4. while i < n
5. number = read input from user
6. sum = sum + number
7. i = i + 1
8. mean = sum / n
T(n), or the running time of a particular algorithm on input of size n, is taken to be the number of
times the instructions in the algorithm are executed. Pseudo code algorithm illustrates the
calculation of the mean (average) of a set of n numbers:
Statement Number of times executed 1 1
2 1
3 1
4 n+1
5 n
6 n
7 n
8 1
The computing time for this algorithm in terms on input size n is: T(n) = 4n + 5.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Analysis of Simple Programs (no recursion)
Sum the “costs” of the lines of the program
Compute the bounding function
– e.g., O(*)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Analysing Loops
Use Loop Invariants
Intuitive notion and example
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Designing an Algorithm Define Problem Define Loop
Invariants
Define Measure of
Progress
Define Step Define Exit Condition Maintain Loop Inv
Make Progress Initial Conditions Ending
km
79 km
to school
Exit
Exit
79 km 75 km
Exit
Exit
0 km Exit
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Typical Loop Invariant
If the input consists of an array of objects
I have a solution for the first i objects.
Extend the solution into a solution for the first
i+1.
i objects
Exit
79 km 75 km Exit i to i+1
Done when
solution for n
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Typical Loop Invariant
If the output consists of an array of objects
I have an output produced for the first i objects.
Produce the i+1-st output object.
i objects
Exit 79 km 75 km
Exit i to i+1 Done when
output n objects.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example of Approach
Binary search
– Standard algorithm
Input
– Sorted list, and key
Goal: find key in list
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Define Problem: Binary Search
PreConditions
– Key 25
– Sorted List
PostConditions
– Find key in list (if there).
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Define Loop Invariant
Maintain a sublist
Such that
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Define Loop Invariant
Maintain a sublist.
If the key is contained in the original list, then
the key is contained in the sublist.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Define Step
Make Progress
Maintain Loop Invariant
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Define Step
Cut sublist in half.
Determine which half key would be in.
Keep that half.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Define Step
Cut sublist in half.
Determine which half the key would be in.
Keep that half.
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key ≤ mid,
then key is in
left half.
If key > mid,
then key is in
right half.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Define Step
It is faster not to check if the middle element
is the key.
Simply continue. key 43
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key ≤ mid,
then key is in
left half.
If key > mid,
then key is in
right half.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Make Progress
The size of the list becomes smaller.
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
79 km
75 km
79 km 75 km
Exit
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Initial Conditions km
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If the key is contained in
the original list,
then the key is contained
in the sublist.
• The sublist is the entire original list.
n km
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Ending Algorithm
If the key is contained in the original list,
then the key is contained in the sublist.
Sublist contains one element.
Exit
Exit
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
0 km
• If the key is contained in the original list,
then the key is at this location.
key 25
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
If key not in original list
If the key is contained in
the original list,
then the key is contained
in the sublist.
• Loop invariant true, even if the key is not in the list.
• If the key is contained in the original list,
then the key is at this location.
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
key 24
• Conclusion still solves the problem.
Simply check this one location for the key.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Running Time
The sublist is of size n, n/2, n/4,
n/8,…,1
Each step (1) time.
Total = (log n)
key 25
3 5 6 13 18 21 21 25 36 43 49 51 53 60 72 74 83 88 91 95
If key ≤ mid,
then key is in
left half.
If key > mid,
then key is in
right half.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Code
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Divide and Conquer Algorithms
Well-known class of algorithm
Requires special approach for complexity
analysis
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Divide-and-Conquer Analysis
• The analysis of divide and conquer algorithms require us to solve a recurrence.
• Recurrences are a major tool for analysis of algorithms
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
MergeSort
T(n/2) T(n/2)
cn
A L G O R I T H M S
divide A L G O R I T H M S
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
MergeSort
T(n/2) T(n/2)
cn
A L G O R I T H M S
Divide #1 A L G O R I T H M S
Divide #2 A L G O R I T H M S
T(n/4) T(n/4) T(n/4) T(n/4)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
MergeSort
12
2
1
)(
ncnn
T
nc
nT
(n/2) +c (n/2) +c
cn
Solve T(n) = T(n/2) + T(n/2) + cn
T(n/4) T(n/4) T(n/4) T(n/4)
Recurrence
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Recursion-tree method
• A recursion tree models the costs (time) of a recursive execution of an algorithm.
• The recursion tree method is good for generating guesses for the substitution method.
• The recursion-tree method can be unreliable, just like any method that uses ellipses (…).
• The recursion-tree method promotes intuition, however.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example of recursion tree
T(n)
Solve T(n) = T(n/4) + T(n/2) + n2:
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example of recursion tree
T(n/4) T(n/2)
n2
Solve T(n) = T(n/4) + T(n/2) + n2:
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
(n/4)2 (n/2)2
T(n/16) T(n/8) T(n/8) T(n/4)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example of recursion tree
(n/16)2 (n/8)2 (n/8)2 (n/4)2
(n/4)2 (n/2)2
(1)
Solve T(n) = T(n/4) + T(n/2) + n2:
n2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
(n/16)2 (n/8)2 (n/8)2 (n/4)2
(n/4)2 (n/2)2
(1)
2nn2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
(n/16)2 (n/8)2 (n/8)2 (n/4)2
(n/4)2 (n/2)2
(1)
2
165 n
2nn2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
(n/16)2 (n/8)2 (n/8)2 (n/4)2
(n/4)2
(1)
2
165 n
2n
2
25625 n
n2
(n/2)2
…
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example of recursion tree
Solve T(n) = T(n/4) + T(n/2) + n2:
(n/16)2 (n/8)2 (n/8)2 (n/4)2
(n/4)2
(1)
2
165 n
2n
2
25625 n
13
1652
165
1652 n
…
Total =
= (n2)
n2
(n/2)2
geometric series
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Solution: geometric series
1
11 2
xxx for |x| < 1
1
11
12
x
xxxx
nn for x 1
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
The master method
The master method applies to recurrences of the form
T(n) = a T(n/b) + f (n) ,
where a 1, b > 1, and f is asymptotically positive.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Idea of master theorem
f (n/b) f (n/b) f (n/b)
(1)
Recursion tree:
…
f (n) a
f (n/b2) f (n/b2) f (n/b2) …
a h = logbn
f (n)
a f (n/b)
a2 f (n/b2)
…
#leaves = ah
= alogbn
= nlogba nlogba (1)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Three common cases
Compare f (n) with nlogba:
1. f (n) = O(nlogba – ) for some constant > 0.
• f (n) grows polynomially slower than nlogba (by an n factor).
Solution: T(n) = (nlogba) .
# leaves in
recursion tree
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Idea of master theorem
f (n/b) f (n/b) f (n/b)
(1)
Recursion tree:
…
f (n) a
f (n/b2) f (n/b2) f (n/b2) …
a h = logbn
f (n)
a f (n/b)
a2 f (n/b2)
…
nlogba (1) CASE 1: The weight increases geometrically from the root to the leaves. The leaves hold a constant fraction of the total weight. (nlogba)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Three common cases
Compare f (n) with nlogba:
2. f (n) = (nlogba lgkn) for some constant k 0.
• f (n) and nlogba grow at similar rates.
Solution: T(n) = (nlogba lgk+1n) .
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Idea of master theorem
f (n/b) f (n/b) f (n/b)
(1)
Recursion tree:
…
f (n) a
f (n/b2) f (n/b2) f (n/b2) …
a h = logbn
f (n)
a f (n/b)
a2 f (n/b2)
…
nlogba (1) CASE 2: (k = 0) The weight is approximately the same on each of the logbn levels. (nlogbalg n)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Three common cases (cont.)
Compare f (n) with nlogba:
3. f (n) = (nlogba + ) for some constant > 0.
• f (n) grows polynomially faster than nlogba (by an n factor),
and f (n) satisfies the regularity condition that a f (n/b) c f (n) for some constant c < 1.
Solution: T(n) = ( f (n) ) .
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Idea of master theorem
f (n/b) f (n/b) f (n/b)
(1)
Recursion tree:
…
f (n) a
f (n/b2) f (n/b2) f (n/b2) …
a h = logbn
f (n)
a f (n/b)
a2 f (n/b2)
…
nlogba (1) CASE 3: The weight decreases geometrically from the root to the leaves. The root holds a constant fraction of the total weight. ( f (n))
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Examples
Ex. T(n) = 4T(n/2) + n a = 4, b = 2 nlogba = n2; f (n) = n. CASE 1: f (n) = O(n2 – ) for = 1. T(n) = (n2).
Ex. T(n) = 4T(n/2) + n2 a = 4, b = 2 nlogba = n2; f (n) = n2. CASE 2: f (n) = (n2lg0n), that is, k = 0. T(n) = (n2lg n).
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Examples
Ex. T(n) = 4T(n/2) + n3 a = 4, b = 2 nlogba = n2; f (n) = n3. CASE 3: f (n) = (n2 + ) for = 1 and 4(cn/2)3 cn3 (reg. cond.) for c = 1/2. T(n) = (n3).
Ex. T(n) = 4T(n/2) + n2/lg n a = 4, b = 2 nlogba = n2; f (n) = n2/lg n. Master method does not apply. In particular, for every
constant > 0, we have n (lg n).
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Growth Function Summary
Defined the mathematical principles for specifying the run-
time of an algorithm
Classified the growth functions
– , O, , o,
Defined several relationships among the growth functions
Theta f(n) = θ(g(n)) f(n) ≈ c g(n)
BigOh f(n) = O(g(n)) f(n) ≤ c g(n)
Omega f(n) = Ω(g(n)) f(n) ≥ c g(n)
Little Oh f(n) = o(g(n)) f(n) > c g(n)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Appendix: Review of Growth Functions
Detailed review of different growth
functions
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
A Simple Example
INPUT: a sequence of n numbers
OUTPUT: the smallest number among them
1. x T[1]
2. for i 2 to n do
3. if T[i] < x then x T[i]
* Performance of this algorithm is a function of n.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Runtime Analysis
Elementary operation: an operation whose execution time can be bounded above by a constant depending on implementation used.
Assume all elementary operations can be executed at unit cost. This is not true, but they are within some constant factor of each other. We are mainly concerned about the order of growth.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Order of Growth
For very large input size, it is the asymptotic rate of
growth (or order of growth) that matters.
We can ignore the lower-order terms, since they are
relatively insignificant for very large n. We can also
ignore leading term’s constant coefficients, since they
are not as important for the rate of growth in
computational efficiency for very large n.
Higher order functions of n are normally considered less
efficient.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Comparisons of Algorithms
Multiplication – classical technique: O(nm)
– divide-and-conquer: O(nmln1.5) ~ O(nm0.59)
For operands of size 1000, takes 40 & 15 seconds
respectively on a Cyber 835.
Sorting – insertion sort: (n2)
– merge sort: (n lg n)
For 106 numbers, it took 5.56 hrs on a supercomputer
using machine language and 16.67 min on a PC using
C/C++.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Why Order of Growth Matters
Computer speeds double every two years, so
why worry about algorithm speed?
When speed doubles, what happens to the
amount of work you can do?
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Effect of Faster Machines
• Higher gain with faster hardware for more efficient algorithm.
• Results are more dramatic for higher speeds: for the n lg n
algorithm, doubling the speed almost doubles the number of items
that can be sorted.
No. of items sorted
Ο(n2)
H/W Speed
Comp. of Alg. 1 M* 2 M Gain
1000 1414 1.414
O(n lgn) 62700 118600 1.9
* Million operations
per second.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Classifying Functions
Functions
Con
stant
Lo
garith
mic
Po
ly L
ogarith
mic
Poly
no
mial
Exp
on
ential
Exp
Dou
ble E
xp
(log n)5 n5 25n 5 log n 5 2 n5 2
5n
2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Ordering Functions
Functions
Con
stant
Lo
garith
mic
Po
ly L
ogarith
mic
Poly
no
mial
Exp
on
ential
Exp
Dou
ble E
xp
(log n)5 n5 25n 5 log n 5 2 n5 2
5n
2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Classifying Functions
Functions
Con
stant
Lo
garith
mic
Po
ly L
ogarith
mic
Poly
no
mial
Exp
on
ential
Ex
p
Dou
ble E
xp
(log n)θ(1) nθ(1) 2θ(n) θ(log n) θ(1) 2 nθ(1) 2
θ(n)
2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Asymptotic Notation
, O, , o,
Used to describe the running times of algorithms
Instead of exact running time, say (n2)
Defined for functions whose domain is the set of
natural numbers, N
Determine sets of functions, in practice used to
compare two functions
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example
3n2 + 7n + 8= (n2)
What constants for n0, c1, and c2 will work?
Make c1 a little smaller than leading coefficient,
and c2 a little bigger
To compare orders of growth, look at the
leading term
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Definition of Theta
f(n) is sandwiched between c1g(n) and c2g(n)
for some sufficiently small c1 (= 0.0001)
for some sufficiently large c2 (= 1000)
c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )
f(n) = θ(g(n))
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Definition of Theta
For all sufficiently large n
For some definition of “sufficiently large”
c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )
f(n) = θ(g(n))
n0
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Definition of Theta
c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )
3n2 + 7n + 8 = θ(n2)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Definition of Theta
c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )
3n2 + 7n + 8 = θ(n2)
c1·n2 3n2 + 7n + 8 c2·n
2 ? ?
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Definition of Theta
c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )
3n2 + 7n + 8 = θ(n2)
3·n2 3n2 + 7n + 8 4·n2 3 4
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Definition of Theta
c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )
3n2 + 7n + 8 = θ(n2)
3·12 3·12 + 7·1 + 8 4·12 1
False
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Definition of Theta
c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )
3n2 + 7n + 8 = θ(n2)
3·72 3·72 + 7·7 + 8 4·72 7
False
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Definition of Theta
c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )
3n2 + 7n + 8 = θ(n2)
3·82 3·82 + 7·8 + 8 4·82 8
True
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Definition of Theta
c c n n n c g n f n c g n1 2 0 0 1 2, , , , ( ) ( ) ( )
3n2 + 7n + 8 = θ(n2)
3·n2 3n2 + 7n + 8 4·n2 n 8 8
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Running Times
“The running time is O(f(n))”
Worst case is O(f(n))
“Running time is (f(n))” Best case is (f(n))
Can still say “Worst case running time is (f(n))”
– Means worst case running time is given by some
unknown function g(n) (f(n))
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Example
Insertion sort takes (n2) worst case time, so
sorting (as a problem) is O(n2)
Any sort algorithm must look at each item, so
sorting is (n)
In fact, using (e.g.) merge sort, sorting is (n lg n)
in the worst case
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Asymptotic Notation in Equations
Asymptotic notation is used to replace expressions containing lower-order terms.
For example,
4n3 + 3n2 + 2n + 1 = 4n3 + 3n2 + (n)
= 4n3 + (n2) = (n3)
In equations, (f(n)) always stands for an anonymous function g(n) (f(n)).
– In the example above, (n2) stands for 3n2 + 2n + 1.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
o-notation
For a given function g(n), we denote by o(g(n)) the set of functions:
o(g(n)) = {f(n): for any positive constant c > 0, there exists a constant n0 > 0 such that 0 f(n) < cg(n) for all n n0 }
f(n) becomes insignificant relative to g(n) as n approaches infinity: lim [f(n) / g(n)] = 0
n
We say g(n) is an upper bound for f(n) that is not
asymptotically tight.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
O(*) versus o(*)
O(g(n)) = {f(n): there exist positive constants c and n0 such that 0 f(n) cg(n), for all n n0 }.
o(g(n)) = {f(n): for any positive constant c > 0, there exists a constant n0 > 0 such that 0 f(n) < cg(n) for all n n0 }.
Thus o(f(n)) is a weakened O(f(n)).
For example: n2 = O(n2)
n2 o(n2)
n2 = O(n3)
n2 = o(n3)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
-notation
For a given function g(n), we denote by (g(n)) the set of
functions
(g(n)) = {f(n): for any positive constant c > 0, there
exists a constant n0 > 0 such that 0 cg(n) < f(n) for
all n n0 }
f(n) becomes arbitrarily large relative to g(n) as n
approaches infinity: lim [f(n) / g(n)] =
n
We say g(n) is a lower bound for f(n) that is not
asymptotically tight.
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Limits
lim [f(n) / g(n)] = 0 f(n) (g(n))
n
lim [f(n) / g(n)] < f(n) (g(n))
n
0 < lim [f(n) / g(n)] < f(n) (g(n))
n
0 < lim [f(n) / g(n)] f(n) (g(n))
n
lim [f(n) / g(n)] = f(n) (g(n))
n
lim [f(n) / g(n)] undefined can’t say
n
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Additional Useful Relations
The following slides provide many useful
relations for orders of growth
These relations may prove helpful in
homeworks and exams
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Comparison of Functions
f g a b
f (n) = O(g(n)) a b
f (n) = (g(n)) a b
f (n) = (g(n)) a = b
f (n) = o(g(n)) a < b
f (n) = (g(n)) a > b
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Properties
Transitivity f(n) = (g(n)) & g(n) = (h(n)) f(n) = (h(n))
f(n) = O(g(n)) & g(n) = O(h(n)) f(n) = O(h(n))
f(n) = (g(n)) & g(n) = (h(n)) f(n) = (h(n))
f(n) = o (g(n)) & g(n) = o (h(n)) f(n) = o (h(n))
f(n) = (g(n)) & g(n) = (h(n)) f(n) = (h(n))
Symmetry
f(n) = (g(n)) if and only if g(n) = (f(n))
Transpose Symmetry f(n) = O(g(n)) if and only if g(n) = (f(n))
f(n) = o(g(n)) if and only if g(n) = ((f(n))
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Useful Facts
For a 0, b > 0, lim n ( lga n / nb ) = 0, so lga n =
o(nb), and nb = (lga n )
– Prove using L’Hopital’s rule and induction
lg(n!) = (n lg n)
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Examples
A B
5n2 + 100n 3n2 + 2
log3(n2) log2(n
3)
nlg4 3lg n
lg2n n1/2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Examples
A B
5n2 + 100n 3n2 + 2 A (B) A (n2), n2 (B) A (B)
log3(n2) log2(n
3)
nlg4 3lg n
lg2n n1/2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Examples
A B
5n2 + 100n 3n2 + 2 A (B) A (n2), n2 (B) A (B)
log3(n2) log2(n
3) A (B) logba = logca / logcb; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3)
nlg4 3lg n
lg2n n1/2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Examples
A B
5n2 + 100n 3n2 + 2 A (B) A (n2), n2 (B) A (B)
log3(n2) log2(n
3) A (B) logba = logca / logcb; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3)
nlg4 3lg n A (B) alog b = blog a; B =3lg n=nlg 3; A/B =nlg(4/3) as n
lg2n n1/2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Examples
A B
5n2 + 100n 3n2 + 2 A (B) A (n2), n2 (B) A (B)
log3(n2) log2(n
3) A (B) logba = logca / logcb; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3)
nlg4 3lg n A (B) alog b = blog a; B =3lg n=nlg 3; A/B =nlg(4/3) as n
lg2n n1/2 A (B) lim ( lga n / nb ) = 0 (here a = 2 and b = 1/2) A (B)
n
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Review on Summation
Why do we need summation formulas?
We need them for computing the running times of some
algorithms. (CLRS/Chapter 3)
Example: Maximum Subvector
Given an array a[1…n] of numeric values (can be positive,
zero and negative) determine the subvector a[i…j] (1 i
j n) whose sum of elements is maximum over all
subvectors.
1 -2 2 2
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Reviews on Summation
MaxSubvector(a, n)
maxsum 0;
for i 1 to n
for j = i to n
sum 0;
for k i to j
sum += a[k]
maxsum max(sum, maxsum);
return maxsum;
n n j T(n) = 1 i=1 j=i k=i
NOTE: This is not a simplified solution. What is the final answer?
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Reviews on Summation
Constant Series: For n 0,
b
1 = b - a + 1 i=a
Linear Series: For n 0,
n
i = 1 + 2 + … + n = n(n+1) / 2 i=1
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Reviews on Summation
Quadratic Series: For n 0,
n
i2 = 12 + 22 + … + n2 = (2n3 + 3n2 + n) / 6 i=1
Linear-Geometric Series: For n 0,
n
ici = c + 2c2 + … + ncn = [(n-1)c(n+1) - ncn + c] / (c-1)2 i=1
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Divide-and-Conquer Examples
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Integer Multiplication
Let X = A B and Y = C D where A,B,C and D
are n/2 bit integers
Simple Method: XY = (2n/2A+B)(2n/2C+D)
Running Time Recurrence
T(n) < 4T(n/2) + 100n
How do we solve it?
-
CS 4407, Algorithms University College Cork,
Gregory M. Provan
Substitution method
1. Guess the form of the solution.
2. Verify by induction.
3. Solve for constants.
The most general method:
Example: T(n) = 4T(n/2) + 100n
• [Assume that T(1) = (1).]
• Guess O(n3) . (Prove O and separately.)
• Assume that T(k) ck3 for k < n .
• Prove T(n) cn3 by induction.