chapter 01tadavis/cs303/f16/ch02m.pdfgeneral rules (cont.) 40 even worse! โextremely inefficient,...
TRANSCRIPT
9/8/2016
1
Chapter 2
Algorithm Analysis
Introduction
2
โalgorithm
โset of simple instructions to solve a problem
โanalyzed in terms, such as time and memory, required
โtoo long (minutes, hours, years) โ no good
โtoo much memory (terabytes) โ no good
โwe can estimate resource usage using a formal
mathematical framework to count basic operations
โbig-O ๐(๐(๐))
โbig-omega ฮฉ (๐(๐))
โbig-theta ฮ (๐ ๐ )
Asymptotic Notation
3
โ formal definitions
๐ ๐ = ๐(๐(๐)) ๐ ๐ = ฮฉ (๐(๐)) ๐ ๐ = ฮ(๐ ๐ )
๐ ๐ and ๐(๐) are positive for ๐ > 1
when we say that ๐ is true when ๐ is sufficiently large, we
mean there exists ๐ such that ๐ is true for all ๐ > ๐
we are comparing relative rates of growth
Asymptotic Notation: Big-๐
4
๐ ๐ = ๐(๐(๐)) if there exist constants ๐ and ๐ such that
๐ ๐ โค ๐ ยท ๐(๐) for all ๐ > ๐
loosely speaking, ๐ ๐ = ๐ ๐ ๐ is an analog of ๐ โค ๐
example: if ๐ ๐ = 54๐2 + 42๐ and ๐ ๐ = ๐2 then ๐ = ๐(๐) since
๐ ๐ โค 55๐(๐)
for all ๐ > 42
Asymptotic Notation: ฮฉ
5
๐ ๐ = ฮฉ(๐(๐)) if there exist constants ๐ and ๐ such that
๐ ๐ โฅ ๐ ยท ๐(๐) for all ๐ > ๐
loosely speaking, ๐ ๐ = ฮฉ ๐ ๐ is an analog of ๐ โฅ ๐
Asymptotic Notation: ฮ
6
๐ ๐ = ฮ(๐(๐)) if there exist constants ๐1, ๐2, and ๐ such
that
๐1 ยท ๐ ๐ < ๐ ๐ < ๐2 ยท ๐(๐) for all ๐ > ๐
loosely speaking, ๐ ๐ = ฮ ๐ ๐ is an analog of ๐ โ ๐
9/8/2016
2
Relationship of ๐,ฮฉ, and ฮ
7
โnote that
๐ ๐ = ๐ ๐ ๐ โ ๐ ๐ = ฮฉ(๐ ๐ )
since
๐ ๐ < ๐๐ ๐ โ ๐ ๐ >1
๐๐ ๐
โalso
๐ ๐ = ฮ ๐ ๐ โ ๐ ๐ = ๐ ๐ ๐ and ๐ ๐ = ฮฉ ๐ ๐
since ๐ ๐ = ฮ(๐(๐)) means that there exist ๐1, ๐2, and ๐
such that
๐1 ยท ๐ ๐ < ๐ ๐ < ๐2 ยท ๐(๐) for all ๐ > ๐
Asymptotic Notation: First Observation
8
โ for sufficiently large ๐,
๐ ๐ = ๐(๐(๐)) gives an upper bound on ๐ in terms of ๐
๐ ๐ = ฮฉ (๐(๐)) gives a lower bound on ๐ in terms of ๐
๐ ๐ = ฮ(๐ ๐ ) gives both an upper and lower bound on ๐
in terms of ๐
Asymptotic Notation: Second Observation
9
โ it is the existence of constants that matters for the purposes of asymptotic complexity analysis โ not the exact values of the constants
โexample: all of the following imply ๐2 + ๐ = ๐(๐2):
๐2 + ๐ < 4๐2 for all ๐ > 1
๐2 + ๐ < 2๐2 for all ๐ > 1
๐2 + ๐ < 1.01๐2 for all ๐ > 100
๐2 + ๐ < 1.000001๐2 for all ๐ > 1000
Asymptotic Notation: Third Observation
10
โwe typically ignore constant factors and lower-order terms, since all we want to capture is growth trends
โexample:
๐ ๐ = 1,000๐2 + 10,000๐ + 42
โ ๐ ๐ = ๐(๐2)
but the following are discouraged:
๐ ๐ = ๐(1,000๐2) true, but bad form
๐ ๐ = ๐(๐2 + ๐) true, but bad form
๐ ๐ < ๐(๐2) < built into ๐
Asymptotic Notation: Fourth Observation
11
๐(๐(๐)) assures that you canโt do any worse than ๐(๐), but
it can unduly pessimistic
ฮฉ (๐(๐)) assures that you canโt do any better than ๐(๐), but
it can unduly optimistic
ฮ(๐ ๐ ) is the strongest statement: you canโt do any better
than ๐(๐), but you canโt do any worse
A Shortcoming of ๐
12
an ๐ ๐ = ๐(๐(๐)) bound can be misleading
example: ๐ = ๐(๐2), since ๐ < ๐2 for all ๐ > 1
however, for large ๐, ๐ โช ๐2 โ the functions ๐ ๐ = ๐ and
๐ ๐ = ๐2 are nothing alike for large ๐
we donโt write something like ๐ ๐ = ๐(๐2) if we know, say,
that ๐ ๐ = ๐(๐)
upper bounds should be as small as possible!
9/8/2016
3
A Shortcoming of ฮฉ
13
an ๐ ๐ = ฮฉ(๐(๐)) bound can be misleading
example: ๐2 = ฮฉ(๐), since ๐2 > ๐ for all ๐ > 1
however, for large ๐, ๐2 โซ ๐ โ the functions ๐ ๐ = ๐ and
๐ ๐ = ๐2 are nothing alike for large ๐
we donโt write something like ๐ ๐ = ฮฉ(๐) if we know, say,
that ๐ ๐ = ฮฉ(๐2)
lower bounds should be as large as possible!
ฮ is the Most Informative
14
a ๐ ๐ = ฮ(๐(๐)) relationship is the most informative:
๐1 ยท ๐ ๐ < ๐ ๐ < ๐2 ยท ๐(๐) for all ๐ > ๐
example: 2๐2 + ๐ = ฮ(๐2)
2๐2 < 2๐2 + ๐ < 3๐2
for all ๐ > 1
for large values of ๐, the ratio (2๐2 + ๐)/๐2 tends to 2;
in this sense,
2๐2 + ๐ โ 2๐2
Typical Growth Rates
15
https://expcode.wordpress.com/2015/07/19/big-o-big-omega-and-big-theta-notation/
Additional Rules
16
Rule 1
if ๐1 ๐ = ๐(๐(๐)) and ๐2 ๐ = ๐(๐(๐)) then
(a) ๐1 ๐ + ๐2 ๐ = ๐(๐ ๐ + ๐(๐)) (or just
๐(max (๐ ๐ , ๐(๐)))
(b) ๐1 ๐ โ ๐2 ๐ = ๐(๐ ๐ โ ๐(๐))
Rule 2
if ๐(๐) is a polynomial of degree ๐, then ๐ ๐ = ฮ(๐๐)
Rule 3
logk๐ = ๐ ๐ for any constant ๐ (logarithms grow very slowly)
Typical Growth Rates
17
Function Name
๐ constant
log ๐ logarithmic
๐ linear
๐ log ๐
๐2 quadratic
๐3 cubic
2๐ exponential
Typical Growth Rates
18
http://www.cosc.canterbury.ac.nz/cs4hs/2015/files/complexity-tractability-julie-mcmahon.pdf
9/8/2016
4
Complexity and Tractability
19
assume computer speed of 1 billion ops/sec
T(n)
n n n log n n2
n3
n4
n10
2n
10 .01ms .03ms .1ms 1ms 10ms 10s 1ms
20 .02ms .09ms .4ms 8ms 160ms 2.84h 1ms
30 .03ms .15ms .9ms 27ms 810ms 6.83d 1s
40 .04ms .21ms 1.6ms 64ms 2.56ms 121d 18m
50 .05ms .28ms 2.5ms 125ms 6.25ms 3.1y 13d
100 .1ms .66ms 10ms 1ms 100ms 3171y 4ยด1013
y
103
1ms 9.96ms 1ms 1s 16.67m 3.17ยด1013
y 32ยด10283
y
104
10ms 130ms 100ms 16.67m 115.7d 3.17ยด1023
y
105
100ms 1.66ms 10s 11.57d 3171y 3.17ยด1033
y
106
1ms 19.92ms 16.67m 31.71y 3.17ยด107y 3.17ยด10
43y
What to Analyze
20
in order to analyze algorithms, we need a model of
computation
โstandard computer with sequential instructions
โeach operation (+, -, *, /, =, etc.) takes one time unit, even
memory accesses
โ infinite memory
โobviously unrealistic
Complex Arithmetic Operations
21
โexamples: ๐ฅ๐ log2 ๐ฅ ๐ฅ
โcount the number of basic operations required to execute
the complex arithmetic operations
โ for instance, we could compute ๐ฅ๐ as
๐ฅ๐ โ ๐ฅ โ ๐ฅ โ โฏโ ๐ฅ โ ๐ฅ
there are ๐ โ 1 multiplications, so there are ๐ โ 1 basic
operations to compute ๐ฅ๐ plus one more basic operation
to assign or return the result
What to Analyze
22
different types of performance are analyzed
โbest case
โmay not represent typical behavior
โaverage case
โoften reflects typical behavior
โdifficult to determine what is average
โdifficult to compute
โworst case
โguarantee on performance for any input
โtypically considered most important
โshould be implementation-independent
Maximum Subsequence Sum Problem
23
example: Maximum Subsequence Sum Problem
โgiven (possibly negative) integers, A1, A2, โฆ, AN, find the
maximum value of
โe.g, for input -2, 11, -4, 13, -5, -2, the answer is 20 (A2
through A4)
j
ik
kA
Maximum Subsequence Sum Problem
24
many algorithms to solve this problem
โwe will focus on four with run time in the table below
9/8/2016
5
Maximum Subsequence Sum Problem
25
notes
โcomputer actually run on not important since weโre looking
to compare algorithms
โ for small inputs, all algorithms perform well
โ times do not include read time
โalgorithm 4: time to read would surpass time to compute
โreading data is often the bottleneck
โ for algorithm 4, the run time increases by a factor of 10 as
the problem size increases by a factor of 10 (linear)
โalgorithm 2 is quadratic; therefore an increase of input by
a factor of 10 yields a run time factor of 100
โalgorithm 1 (cubic): run time factor of 1000
Maximum Subsequence Sum Problem
26
Maximum Subsequence Sum Problem
27
Running Time Calculations
28
at least two ways to estimate the running time of a program
โempirically
โas in previous results
โrealistic measurement
โanalysis
โhelps to eliminate bad algorithms early when several
algorithms are being considered
โanalysis itself provides insight into designing efficient
algorithms
โpinpoints bottlenecks, which need special coding
A Simple Example
29
calculate ๐3๐๐=1
A Simple Example
30
analysis
โdeclarations count for no time; ignore call and return costs
โ lines 1 and 4 count for 1 time unit each
โ line 3 counts for 4 time units
โ2 multiplications
โ1 addition
โ1 assignment
โ line 2 has hidden costs
โ initializing i (1 time unit)
โtesting i โค n (๐ + 1)
โ incrementing i (๐)
โ total: 6๐ + 4 or ๐(๐)
9/8/2016
6
A Simple Example
31
analysis
โ too much work to analyze all programs like this
โsince weโre dealing with ๐, we can use shortcuts to help
โ line 3 counts for 4 time units, but is ๐(1)
โ line 1 is insignificant compared to the for loop
General Rules
32
concentrate on loops and recursive calls
โRule 1 โ FOR loops
โrunning time of the statements inside (including tests)
times the number of iterations
โRule 2 โ Nested loops
โanalyze from the inside out
โrunning time of the statements times the product of the
sizes of all the loops
โwatch out for loops that
โcontribute only a constant amount of computation
โcontain break statements
Single Loops
33
for (int i = 0; i < n; i++)
a[i]+= i;
for (int i = 5; i < n; i++)
a[i]+= i;
for (int i = 0; i < n; i++)
{
if (i >= 7) break;
a[i]+= i;
}
Nested Loops
34
for (int i = 0; i < n; ++i)
for (int j = 0; j < n; ++j)
++k;
โconstant amount of computation inside nested loop
โmust multiply number of times both loops execute
โ๐(๐2)
for (int i = 0; i < n; ++i)
for (int j = i; j < n; ++j)
++k;
โeven though second loop is not executed as often, still
๐(๐2)
Nested Loops
35
for (int i = 0; i < 7; ++i)
for (int j = 0; j < n; ++j)
++k;
โouter loop is executed a fixed number of times
โ๐(๐)
General Rules (cont.)
36
concentrate on loops and recursive calls
โRule 3 โ Consecutive statements
โsimply add the running times
for (i = 0; i < n; ++i)
a[i] = 0
for (i = 0; i < n; ++i)
for (j = 0; j < n; ++j)
a[i] += a[j] + i + j
โ๐(๐) work followed by ๐(๐2) work, so ๐(๐2)
9/8/2016
7
General Rules (cont.)
37
concentrate on loops and recursive calls
โRule 4 โ if/else
if (test)
S1
else
S2
โtotal running time includes the test plus the larger of the
running times of S1 and S2
โcould be overestimate in some cases, but never an
underestimate
General Rules (cont.)
38
concentrate on loops and recursive calls
โ function calls analyzed first
โrecursion
โ if really just a for loop, not difficult (the following is ๐(๐))
long factorial (int n)
{
if (n <= 1)
return 1;
else
return n * factorial (n โ 1);
}
General Rules (cont.)
39
previous example not a good use of recursion
โ if recursion used properly, difficult to convert to a simple
loop structure
what about this example?
long fib (int n)
{
if (n <= 1)
return 1;
else
return fib(n โ 1) + fib (n โ 2);
}
General Rules (cont.)
40
even worse!
โextremely inefficient, especially for values > 40
โanalysis
โfor ๐ = 0 or ๐ = 1, ๐(๐) is constant: ๐(0) = ๐(1) = 1
โsince the line with recursive calls is not a simple
operation, it must be handled differently
๐ ๐ = ๐ ๐ โ 1 + ๐ ๐ โ 2 + 2 for ๐ โฅ 2
โwe have seen that this algorithm <5
3
๐
โsimilarly, we could show it is >3
2
๐
โexponential!
General Rules (cont.)
41
with current recursion
โ lots of redundant work
โviolating compound interest rule
โ intermediate results thrown away
running time can be reduced substantially with simple for loop
Maximum Subsequence Sum Problem Revisited
42
example: Maximum Subsequence Sum Problem
โgiven (possibly negative) integers, A1, A2, โฆ, AN, find the
maximum value of
โe.g, for input -2, 11, -4, 13, -5, -2, the answer is 20 (A2
through A4)
โpreviously, we reviewed running time results for four
algorithms
j
ik
kA
9/8/2016
8
Maximum Subsequence Sum Problem Revisited
43
many algorithms to solve this problem
โwe will focus on four with run time in the table below
Maximum Subsequence Sum Problem Revisited
44
Algorithm 1
45
Algorithm 1
46
exhaustively tries all possibilities
โ first loop iterates ๐ times
โsecond loop iterates ๐ โ ๐ times, which could be small,
but we must assume the worst
โ third loop iterates ๐ โ ๐ + 1 times, which we must also
assume to be size ๐
โ total work on line 14 is ๐(1), but itโs inside these nested
loops
โ total running time is therefore ๐ 1 โ ๐ โ ๐ โ ๐ = ๐(๐3)
โwhat about lines 16-17?
Algorithm 1
47
more precise calculation
โuse better bounds on loops to compute how many times
line 14 is calculated
โcompute from the inside out
โ then
Algorithm 1
48
more precise calculation (cont.)
โ to complete the calculation
9/8/2016
9
Algorithm 2
49
to speed up the algorithm, we can remove a for loop
โunnecessary calculation removed
โnote that
โnew algorithm is ๐(๐2)
Algorithm 2
50
Algorithm 3
51
recursive algorithm runs even faster
โdivide and conquer strategy
โdivide: split problem into two roughly equal subproblems
โconquer: merge solutions to subproblems
โmaximum subsequence can be in one of three places
โentirely in left half of input
โentirely in right half of input
โ in both halves, crossing the middle
โ first two cases solved recursively
โ last case solved by finding the largest sum in the first half that includes the last element of the first half, plus the largest sum in the second half that includes the first element in the second half
Algorithm 3
52
consider the following example
โmaximum subsequence for the first half: 6
โmaximum subsequence for the second half: 8
โmaximum subsequence that crosses the middle
โmaximum subsequence in the first half that includes the
last element of the first half: 4
โmaximum subsequence in the second half that includes
the first element in the second half: 7
โtotal: 11
Algorithm 3
53
Algorithm 3 (cont.)
54
9/8/2016
10
Algorithm 3
55
notes
โdriver function needed
โgets solution started by calling function with initial
parameters that delimit entire array
โ if left == right
โbase case
โrecursive calls to divide the list
โworking toward base case
โ lines 18-24 and 26-32 calculate max sums that touch the
center
โmax3 returns the largest of three values
Algorithm 3
56
analysis
โ let ๐(๐) be the time it takes to solve a maximum
subsequence problem of size ๐
โ if ๐ = 1, program takes some constant time to execute
lines 8-12; thus ๐ 1 = 1
โotherwise, the program must perform two recursive calls
โ the two for loops access each element in the subarray,
with constant work; thus ๐(๐)
โall other non-recursive work in the function is constant
and can be ignored
โrecursive calls divide the list into two ๐/2 subproblems if
๐ is even (and more strongly, a power of 2)
โtime: 2๐(๐/2)
Algorithm 3
57
total time for the algorithm is therefore
๐ ๐ = 2๐๐
2+๐ where ๐ 1 = 1
โnote that
๐ ๐ = ๐ ๐ โ 1 + ๐ is ๐ ๐ = ๐๐ or just ๐(๐)
and
๐ ๐ = ๐ ๐ โ 1 + ๐๐ is ๐ ๐ = ๐๐(๐ + 1)/2
or just ๐(๐2)
Algorithm 3
58
we can solve the recurrence relation directly (later)
๐ ๐ = 2๐๐
2+ ๐ where ๐ 1 = 1
โ for now, note that
๐ 1 = 1
๐ 2 = 4 = 2 โ 2 ๐ 4 = 12 = 4 โ 3
๐ 8 = 32 = 8 โ 4 ๐ 16 = 80 = 16 โ 5
thus, if ๐ = 2๐ ,
๐ ๐ = ๐ โ ๐ + 1 = ๐ log๐ +๐ = ๐(๐ log๐)
Algorithm 4
59
linear algorithm to solve the maximum subsequence sum
โobservation: we never want a negative sum
โas before, remember the best sum weโve encountered so
far, but add a running sum
โ if the running sum becomes negative, reset the starting
point to the first positive element
Algorithm 4
60
9/8/2016
11
Algorithm 4
61
notes
โ if a[i] is negative, it cannot possibly be the first element of
a maximum sum, since any such subsequence would be
improved by beginning with a[i+1]
โsimilarly, any negative subsequence cannot possible be a
prefix for an optimal subsequence
โthis allows us to advance through the original sequence
rapidly
โcorrectness may be difficult to determine
โcan be tested on small input by running against more
inefficient brute-force programs
โadditional advantage: data need only be read once (online
algorithm) โ extremely advantageous
Logarithms in the Running Time
62
logarithms will show up regularly in analyzing algorithms
โwe have already seen them in divide and conquer
strategies
โgeneral rule: an algorithm is ๐(๐ log๐) if it takes constant
๐(1) time to cut the problem size by a fraction (typically 1
2)
โ if constant time is required to merely reduce the problem
by a constant amount (say, smaller by 1), then the
algorithm is ๐(๐)
โonly certain problems fall into the ๐(log๐) category
since it would take ฮฉ(๐) just to read in the input
โfor such problems, we assume the input is pre-read
Binary Search
63
binary search
โgiven an integer X and integers A0, A1, โฆ, AN-1, which are
presorted and already in memory, find I such that Ai = X, or
return i = -1 if X is not in the input
โobvious solution: scan through the list from left to right to
find X
โruns in linear time
โdoes not take into account that elements are presorted
โbetter solution: check if X is in the middle element
โ if yes, done
โ if smaller, apply same strategy to sorted left subarray
โ if greater, use right subarray
Binary Search
64
Binary Search
65
analysis
โall the work done inside the loop takes ๐(1) time per
iteration
โ first iteration of the loop is for ๐ โ 1
โsubsequent iterations of the loop halve this amount
โtotal running time is therefore ๐(log๐)
โanother example using sorted data for fast lookup
โperiodic table of elements
โ118 elements
โat most 8 accesses would be required
Euclidโs Algorithm
66
Euclidโs algorithm
โcomputes gcd (greatest common divisor) of two integers
โ largest integer that divides both
9/8/2016
12
Euclidโs Algorithm
67
Euclidโs Algorithm
68
notes
โalgorithm computes gcd (M, N) assuming ๐ โฅ ๐
โ if ๐ > ๐, the values are swapped
โalgorithm works by continually computing remainders until
0 is reached
โ the last non-0 remainder is the answer
โ for example, if ๐ = 1,989 and ๐ = 1,590, the sequence of
remainders is 399, 393, 6, 3, 0
โtherefore gcd = 3
โgood, fast algorithm
Euclidโs Algorithm
69
analysis
โneed to determine how many remainders will be
computed
โlog๐ is a good guess, but as can be seen from the
example, the values do not decrease in a uniform way
โwe can prove that the remainder is at most half of its
original value after two iterations
โthis would show that the number of iterations is at most
2log๐ = ๐(log๐)
Euclidโs Algorithm
70
analysis (cont.)
โShow: if ๐ > ๐, then ๐ < ๐/2
โProof: 2 cases
โ if ๐ โค ๐/2, then since the remainder is smaller than ๐,
the theorem is true
โ if ๐ > ๐/2, then ๐ goes into ๐ once with a remainder
of ๐โ๐ < ๐/2, and the theorem is true
โ in our example, 2log๐ is about 20, but we needed only 7
operations
โthe constant can be refined to 1.44log๐
โthe average case (complicated!) is
(12 ln2 ln๐)/๐2 +1.47
Exponentiation
71
exponentiation
โraising an integer to an integer power
โresults are often large, so machine must be able to
support such values
โnumber of multiplications is the measurement of work
โobvious algorithm: for ๐๐ uses ๐ โ 1 multiplications
โrecursive algorithm: better
โ๐ โค 1 base case
โotherwise, if ๐ is even, ๐๐ = ๐๐/2 โ ๐๐/2
โ if ๐ is odd, ๐๐ = ๐(๐โ1)/2 โ ๐(๐โ1)/2 โ ๐
โexample: ๐62 = ๐31 2
๐3 = ๐2 ๐,๐7 = ๐3 2๐, ๐15 = ๐7 2๐, ๐31 = ๐15 2๐
Exponentiation
72
9/8/2016
13
Exponentiation
73
analysis
โnumber of multiplications: 2log๐
โat most, 2 multiplications are required to halve the
problem (if ๐ is odd)
โsome additional modifications can be made, but care must
be taken to avoid errors and inefficiencies
Limitations of Worst-Case Analysis
74
analysis can be shown empirically to be an overestimate
โanalysis can be tightened
โaverage running time may be significantly less than worst-
case running time
โoften, the worst-case is achievable by very bad input,
but still a large overestimate in the average case
โunfortunately, average-case analysis is extremely
complex (and in many cases currently unsolved), so
worst-case is the best that weโve got