On Time Versus Input Size

Great Theoretical Ideas In Computer Science

S. RudichV. Adamchik

CS 15-251 Spring 2006

Lecture 17 March 21, 2006 Carnegie Mellon University

# of bits

time

Welcome to the 9th week

1. Time complexity of algorithms2. Space complexity of songs

How to add 2 n-bit numbers.

**

*

**

**

* **

* **

*

* **

*

* **

*

**

**

**

**

+

How to add 2 n-bit numbers.

**

*

* **

*

* **

*

* **

*

* **

*

* **

*

* **

*

* **

*

* **

*

* **

*

***

*

+*

*

“Grade school addition”

Time complexity of grade school addition

+T(n) = amount of time grade school

addition uses to add two n-bit numbers

What do you mean by “time”?

* * * * * * * * * ** * * * * * * * * ** * * * * * * * * *

* * * * * * * * * * *

Our Goal

We want to define “time” taken by the method of grade school addition without

depending on the implementation details.

Hold on!

But you agree that T(n) does depend on the implementation!

A given algorithm will take different amounts of time on the same inputs depending on such factors as:

Processor speedInstruction setDisk speedBrand of compiler

These objections are serious, but they are not undefeatable.

There is a very nice sense in which we can analyze grade

school addition without having to worry about implementation

details.

Here is how it works . . .

Grade school addition

On any reasonable computer M, adding 3 bits and writing down 2 bits (for short ) can be done in constant time c.

Total time to add two n-bit numbers using grade school addition: c*n[c time for each of n columns]

Grade school addition

On another computer M’, the time to perform may be c’.

Total time to add two n-bit numbers using grade school addition: c’*n[c’ time for each of n columns]

Different machines result in different slopes, but time grows linearly as input size increases.

# of bits in the numbers

time

Machine M

: cn

Machine M’: c’n

You measure “time” asthe number of elementary “steps” defined in any other way, provided each such “step” takes constant timein a reasonable implementation.

Constant: independent of the length n of the input.

Thus we arrive at an implementation independent

insight:

Grade School Addition is a linear time algorithm.

AbstractionAbstraction: : Abstract away the inessential Abstract away the inessential

features of a problem or solutionfeatures of a problem or solution

=

“Keep it simple, stupid!”

The process of abstracting away details and determining the rate of resource usagein terms of the problem size is one of the fundamental ideas in computer science.

How to multiply 2 n-bit numbers.

X* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

n2

How to multiply 2 nHow to multiply 2 n--bit numbers.bit numbers.

X* * * * * * * * * * * * * * * *

* * * * * * * ** * * * * * * *

* * * * * * * ** * * * * * * *

* * * * * * * ** * * * * * * *

* * * * * * * ** * * * * * * *

* * * * * * * * * * * * * * * *

n2

The total time is bounded by c n2 (abstracting away

the implementation details).

Grade School Addition: Linear timeGrade School Multiplication: Quadratic

time

No matter how dramatic the difference in the constants, the quadratic curve will eventually

dominate the linear curve

# of bits in the numbers

time

Time vs. Input Size

For any algorithm, define n – the input size

and T(n) - the amount of time used

on inputs of size n

We will ask:

What is the growth rate of T(n) ?

Useful notation to discuss growth rates

For any monotonic functions f, g from the positive integers to the positive integers, we say

f(n) = O( g(n) )if

g(n) eventually dominates f(n)

[Formally: there exists a constant c such that for all sufficiently large n: f(n) ≤ c*g(n) ]

More useful notation: Ω

For any monotonic functions f, g from the positive integers to the positive integers, we say

f(n) = Ω( g(n) )if:

f(n) eventually dominates g(n)

[Formally: there exists a constant c such that for all sufficiently large n: f(n) ≥ c*g(n) ]

Yet more useful notation: Θ

For any monotonic functions f, g from the positive integers to the positive integers, we say

f(n) = Θ( g(n) )if:

f(n) = O( g(n) ) and f = Ω( g(n) )

# of bits in numbers

time

f = Θ(n) means that f can be sandwiched between two lines

from some point on.

Quickies

• n = O(n2) ?– YES

• n = O(√n) ?– NO

• n2 = Ω(n log n) ?– YES

• 3n2 + 4n + = Ω(n2) ?– YES

• 3n2 + 4n + = Θ(n2) ?– YES

• n2 log n = Θ(n2) ?– NO

n2 log n = Θ(n2)

n2 log n = O(n2) n2 log n = Ω(n2)

Two statements in one!

NO YES

Names For Some Growth Rates

Linear Time: T(n) = O(n)Quadratic Time: T(n) = O(n2)Cubic Time: T(n) = O(n3)

Polynomial Time: for some constant k, T(n) = O(nk).

Example: T(n) = 13 n5

Large Growth Rates

Exponential Time: for some constant k, T(n) = O(kn)

Example: T(n) = n*2n

Logarithmic Time: T(n) = O(logn) Example: T(n) = 15 log2(n)

Now, something different…Now, something different…

Complexity of Songs

Suppose we want to sing a song of length n.

Since n can be large, we want to memorize songs which require only a small amount of brain space.

Let S(n) be the space complexity of a song.

Complexity of Songs

The amount of space S(n) can be measured in either characters of words.

What is asymptotic relation between # of characters and the # of words?

Complexity of Songs

How long is the longest English word?

not the longest one but the funniest…

"Supercalifragilisticexpialidocious"

(34 letters)

Complexity of Songs

The amount of space S(n) can be measured in either characters of words.

What is asymptotic relation between # of characters and the # of words?

# of characters = Θ (# of words)

Complexity of Songs

S(n) is the space complexity of a song.

What is the upper and lower bounds for S(n)?

Complexity of Songs

S(n) is the space complexity of a song.

What is the upper and lower bounds for S(n)?

S(n) = O(n) S(n) = Ω(1)

Complexity of Songs with a refrain

Let S be a song with m verses of length V and a refrain of length R.The refrain is first, last and between.

What is the space complexity of S?

Complexity of Songs with a refrain

Let S be a song with m verses of length V and a refrain of length R.

The song length: n = R + (V+R)*mThe space: s = R + V*mEliminate m to get s = V*n/(V+R) +constTherefore, S(n) = O(n)

100 Bottles of Beer

n bottles of beer on the wall, n bottles of beer. You take one down and pass it around.n-1 bottles of beer on the wall.

What is the space complexity of S?

100 Bottles of Beer

n bottles of beer on the wall, n bottles of beer. You take one down and pass it around.n-1 bottles of beer on the wall.

What is the space complexity of S?

We have to remember one verse (template) and the current value of n. S = Θ(1) + O(log n)

The k Days of Christmas

On the first day of Christmas, my true love sent to me

A partridge in a pear tree.

The k Days of Christmas

On the k day of Christmas, my true love sent to me

gift1 ,…, giftk

What is the space complexity of this song?

The k Days of Christmas

What is the space complexity of this song?

The song length: n = template + k*gift1+(k-1)*gift2+…+ giftk

Let all gifts be the same length G.n = Θ(1) + G*(1+2+…+k) = O(k2)

The space: S = Θ(1) + k*G = O(k) = O(n)

Does there exist arbitrarily long songs of space complexity O(1) ?

•The worst time complexity •The best time complexity •The average time complexity•The amortized time complexity •The randomized time complexity

Time complexities

In sorting and searching (array) algorithms, the input size is the number of items.

In graph algorithm the input size is presented by V+E

In number algorithms, the input size is the number of bits.

The input size

Primality

boolean prime(int n) int k = 2; while( k++ <= Math.sqrt(n) ) if(n%k == 0) return false; return true;

What is the complexity of this algorithm?

Primality

The number of passes through the loop is n.Therefore, T(n) = O(n).

But what is n?

For a given algorithm, the input size is defined as the number of characters it takes to write (or encode) the input.

The input size

Primality

The number of passes through the loop is n.

If we use base 10 encoding, it takes O( log10 n ) = input_sizecharacters (digits) to encode n.

Therefore, T(n) = 10^(input_size/2)The time complexity is not polynomial!

Fibonacci Numbers

int fib( int n ) int [] f = new int[n+2]; f[0]=0; f[1]=1; for( int k = 2; k<=n; k++)

f[k] = f[k-1] +f[k-2]; return f[n];

What is the complexity of this algorithm?

Fibonacci Numbers

public static int fib( int n )int [] f = new int[n+2];f[0]=0; f[1]=1;for( int k = 2; k<=n; k++)

f[k] = f[k-1] +f[k-2];return f[n];

input_size = log n, then n = 2^(input_size)The algorithm is exponential in terms of size.

0-1 Knapsack problem

A thief breaks into a jewelry store carrying a knapsack. Each item in the store has a value and a weight. The knapsack will break if the total weight exceeds W. The thief’s dilemma is to maximize the total value

What is the time complexity using the dynamic programming approach?

0-1 Knapsack problem

We create a table of size n*W,where n is the number of items andW is the maximum weight allowed

T = Θ(n*W) But the input size of W is log W, therefore

T = Θ( n*2^(input size of W) ) If you change W by one bit, the amount of work will double.

Ok, so…

How much time does it take to square

the number n using grade school

multiplication?

Grade School Multiplication:Quadratic time

c(log n)2 time to square the number n

# of bits in numbers

time

Some Big Ones

Doubly Exponential Time means that for some constant k

Triply Exponential

And so forth.

T(n) = 22k n

T(n) = 222k n

Exponential Times

2STACK(0) = 1

2STACK(n) = 22STACK(n-1)

2STACK(1) = 22STACK(2) = 42STACK(3) = 162STACK(4) = 65536

2STACK(5) ¸ 1080 = atoms in universe

2222: : :2

“tower of n 2’s”

2STACK(n) =

And the inverse of 2STACK: log*

2STACK(0) = 1 log*(1) = 0

2STACK(n) = 22STACK(n-1)

2STACK(1) = 2 log*(2) = 12STACK(2) = 4 log*(4) = 22STACK(3) = 16 log*(16) = 32STACK(4) = 65536 log*(65536) = 4

2STACK(5) ¸ 1080 log*(atoms) = 5 = atoms in universe

log*(n) = # of times you have to apply the log function to n to make it ≤ 1

So an algorithm that can be shown to run

in O(n log*n) Time

isLinear Time for all

practical purposes!!

Ackermann’s Function

A(0, n) = n + 1 for n ≥ 0 A(m, 0) = A(m - 1, 1) for m ≥ 1 A(m, n) = A(m - 1, A(m, n - 1)) for m, n ≥ 1

A(4,2) > # of particles in universeA(5,2) can’t be written out in this universe

Inverse Ackermann function

A(0, n) = n + 1 for n ≥ 0 A(m, 0) = A(m - 1, 1) for m ≥ 1 A(m, n) = A(m - 1, A(m, n - 1)) for m, n ≥ 1

Define: A’(k) = A(k,k) Inverse Ackerman α(n) is the inverse of A’

Practically speaking: n × α(n) ≤ 4n

Inverse Ackermann function

It grows slower than log* n

The amortized time of Union-Find takes

O(a(V, E))Finding a MST O(E a(V, E))

Time complexity of grade school addition

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

*

**

*

+*

*

*

**

*

T(n) = The amount of time grade school addition uses to add two n-bit numbers

We saw that T(n) was linear.

T(n) = Θ(n)

Time complexity of grade school multiplication

T(n) = The amount of time grade school multiplication uses

to add two n-bit numbers

We saw that T(n) was quadratic.

T(n) = Θ(n2)

X* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

* * * * * * * * * * * * * * * *

n2

Next Class

Will Bonzo be able to add two numbers faster than Θ(n)?

Can Odette ever multiply two numbers faster than Θ(n2)?

Tune in on Thursday, same time, same place…