Linked lists vs. Arrays

Linked lists have several advantages over arrays. Insertion of an element at a specific point of a list is a

constant-time operation, whereas insertion in an array may require moving half of the elements, or more.

While one can "delete" an element from an array in constant time by somehow marking its slot as "vacant",

an algorithm that iterates over the elements may have to skip a large number of vacant slots.

Moreover, arbitrarily many elements may be inserted into a linked list, limited only by the total memory

available; while an array will eventually fill up, and then have to be resized — an expensive operation, that

may not even be possible if memory is fragmented. Similarly, an array from which many elements are

removed may have to be resized in order to avoid wasting too much space.

On the other hand, arrays allow random access, while linked lists allow only sequential access to elements.

Singly-linked lists, in fact, can only be traversed in one direction. This makes linked lists unsuitable for

applications where it's useful to look up an element by its index quickly, such as heapsort. Sequential access

on arrays is also faster than on linked lists on many machines, because they have greater locality of

reference and thus profit more from processor caching.

Another disadvantage of linked lists is the extra storage needed for references, which often makes them

impractical for lists of small data items such as characters or boolean values. It can also be slow, and with a

naïve allocator, wasteful, to allocate memory separately for each new element, a problem generally solved

using memory pools.

Some hybrid solutions try to combine the advantages of the two representations. Unrolled linked lists store

several elements in each list node, increasing cache performance while decreasing memory overhead for

references. CDR coding does both these as well, by replacing references with the actual data referenced,

which extends off the end of the referencing record.

A good example that highlights the pros and cons of using arrays vs. linked lists is by implementing a

program that resolves the Josephus problem. The Josephus problem is an election method that works by

having a group of people stand in a circle. Starting at a predetermined person, you count around the circle n

times. Once you reach the nth person, take them out of the circle and have the members close the circle.

Then count around the circle the same n times and repeat the process, until only one person is left. That

person wins the election. This shows the strengths and weaknesses of a linked list vs. an array, because if

you view the people as connected nodes in a circular linked list then it shows how easily the linked list is

able to delete nodes (as it only has to rearrange the links to the different nodes). However, the linked list will

be poor at finding the next person to remove and will need to recurse through the list until it finds that

person. An array, on the other hand, will be poor at deleting nodes (or elements) as it cannot remove one

node without individually shifting all the elements up the list by one. However, it is exceptionally easy to

find the nth person in the circle by directly referencing them by their position in the array.

The list ranking problem concerns the efficient conversion of a linked list representation into an array.

Although trivial for a conventional computer, solving this problem by a parallel algorithm is complicated

and has been the subject of much research.

Arrays Strengths

1. Easy to use

2. No memory management needed

3. Can access any element by index

4. Fairly quick to loop

Weaknesses

1. Static size (can’t increase the size)

2. Most likely not enough or too much memory (you never know how many elements are needed)

Linked Lists Strengths

1. Dynamic size (can increase or decrease the list)

2. No memory is wasted

Weaknesses

1. Lots of overhead code (lots of malloc calls and assigning pointers)

2. Must traverse entire list to go to the nth node.

Recursive Data Structures public class Main {

public static final int NOT_FOUND = -1;

public static int binarySearch(Comparable[] a, Comparable x) {

return binarySearch(a, x, 0, a.length - 1);

}

private static int binarySearch(Comparable[] a, Comparable x, int lo

w, int high) {

if (low > high)

return NOT_FOUND;

int mid = (low + high) / 2;

if (a[mid].compareTo(x) < 0)

return binarySearch(a, x, mid + 1, high);

else if (a[mid].compareTo(x) > 0)

return binarySearch(a, x, low, mid - 1);

else

return mid;

}

public static void main(String[] args) {

int SIZE = 8;

Comparable[] a = new Integer[SIZE];

for (int i = 0; i < SIZE; i++)

a[i] = new Integer(i * 2);

for (int i = 0; i < SIZE * 2; i++)

System.out.println("Found " + i + " at " + binarySearch(a, new I

nteger(i)));

}

}

Recursion (computer science) Recursion in computer science is a way of thinking about and solving many types of problems. In fact,

recursion is one of the central ideas of computer science.[1]

Solving a problem using recursion means the

solution depends on solutions to smaller instances of the same problem.[2]

"The power of recursion evidently lies in the possibility of defining an infinite set of objects by a finite

statement. In the same manner, an infinite number of computations can be described by a finite recursive

program, even if this program contains no explicit repetitions." [3]

Most high-level computer programming languages support recursion by allowing a function to call itself

within the program text. Imperative languages define looping constructs like “while” and “for” loops that are

used to perform repetitive actions. Some functional programming languages do not define any looping

constructs but rely solely on recursion to repeatedly call code. Computability theory has proven that these

recursive only languages are mathematically equivalent to the imperative languages, meaning they can solve

the same kinds of problems even without the typical control structures like “while” and “for”.

Contents [hide] 1 Recursive algorithms 2 Recursive programming

o 2.1 Recursive procedures (generative recursion) 2.1.1 Factorial 2.1.2 Fibonacci 2.1.3 Greatest common divisor 2.1.4 Towers of Hanoi 2.1.5 Binary search

o 2.2 Recursive data structures (structural recursion) 2.2.1 Linked lists 2.2.2 Binary trees

o 2.3 Recursion versus iteration 3 Tail-recursive functions 4 Order of function calling

o 4.1 Function 1 o 4.2 Function 2 with swapped lines

5 Direct and indirect recursion 6 See also 7 Notes and References 8 External links

[edit] Recursive algorithms A common method of simplification is to divide a problem into sub-problems of the same type. This is

known as dialecting. As a computer programming technique, this is called divide and conquer, and it is key

to the design of many important algorithms, as well as a fundamental part of dynamic programming.

Virtually all programming languages in use today allow the direct specification of recursive functions and

procedures. When such a function is called, the computer (for most languages on most stack-based

architectures) or the language implementation keeps track of the various instances of the function (on many

architectures, by using a call stack, although other methods may be used). Conversely, every recursive

function can be transformed into an iterative function by using a stack.

Most (but not all) functions and procedures that can be evaluated by a computer can be expressed in terms of

a recursive function (without having to use pure iteration),[citation needed]

in continuation-passing style;

conversely any recursive function can be expressed in terms of (pure) iteration, since recursion in itself is

iterative too.[citation needed]

In order to evaluate a function by means of recursion, it has to be defined as a

function of itself, along with a base case to finish recursion. For example, the factorial function can be

defined recursively as n! = n * (n - 1)! , where 0! is defined as 1, the base case. Clearly thus, not all function

evaluations lend themselves to a recursive approach. In general, all non-infinite functions can be described

recursively directly; infinite functions (e.g. the series for e = 1+1/1!+1/2!+1/3!...) need an extra 'stopping

criterion', e.g. the number of iterations, or the number of significant digits, because otherwise recursive

iteration would result in an endless loop.

[edit] Recursive programming Creating a recursive procedure essentially requires defining a "base case", and then defining rules to break

down more complex cases into the base case. Key to a recursive procedure is that with each recursive call,

the problem domain must be reduced in such a way that eventually the base case is arrived at.

Some authors classify recursion as either "generative" or "structural". The distinction is made based on

where the procedure gets the data that it works on. If the data comes from a data structure like a list, then the

procedure is "structurally recursive"; otherwise, it is "generatively recursive".[4]

Many well-known recursive algorithms generate an entirely new piece of data from the given data and recur

on it. HTDP (How To Design Programs) refers to this kind as generative recursion. Examples of generative

recursion include: gcd, quicksort, binary search, mergesort, Newton's method, fractals, and adaptive

integration.[5]

[edit] Recursive procedures (generative recursion)

[edit] Factorial A classic example of a recursive procedure is the function used to calculate the factorial of an integer.

Function definition:

Pseudocode (recursive):

function factorial is:

input: integer n such that n >= 0

output: [n × (n-1) × (n-2) × … × 1]

1. if n is 0, return 1

2. otherwise, return [ n × factorial(n-1) ]

end factorial

A recurrence relation is an equation that relates later terms in the sequence to earlier terms[6]

.

Recurrence relation for factorial:

bn = nbn − 1 b0 = 1

Computing the recurrence relation for n = 4:

b4 = 4 * b3

= 4 * 3 * b2

= 4 * 3 * 2 * b1

= 4 * 3 * 2 * 1 * b0

= 4 * 3 * 2 * 1 * 1

= 4 * 3 * 2 * 1

= 4 * 3 * 2

= 4 * 6

= 24

This factorial function can also be described without using recursion by making use of the typical looping

constructs found in imperative programming languages:

Pseudocode (iterative):

function factorial is:

input: integer n such that n >= 0

output: [n × (n-1) × (n-2) × … × 1]

1. create new variable called running_total with a value of 1

2. begin loop

1. if n is 0, exit loop

2. set running_total to (running_total × n)

3. decrement n

4. repeat loop

3. return running_total

end factorial

Scheme, however, is a functional programming language and does not define any looping constructs. It

relies solely upon recursion to perform all looping. Because Scheme is tail-recursive, a recursive procedure

can be defined that implements the factorial procedure as an iterative process — meaning that it uses

constant space but linear time.

[edit] Fibonacci Another well known recursive sequence is the Fibonacci numbers. The first few elements of this sequence

are: 0, 1, 1, 2, 3, 5, 8, 13, 21...

Function definition:

Pseudocode

function fib is:

input: integer n such that n >= 0

1. if n is 0, return 0

2. if n is 1, return 1

3. otherwise, return [ fib(n-1) + fib(n-2) ]

end fib

Java language implementation: /**

* Recursively calculate the kth Fibonacci number.

*

* @param k indicates which Fibonacci number to compute.

* @return the kth Fibonacci number.

*/

private static int fib(int k) {

// Base Case:

// If k <= 2 then fib(k) = 1.

if (k <= 2) {

return 1;

}

// Recursive Case:

// If k > 2 then fib(k) = fib(k-1) + fib(k-2).

else {

return fib(k-1) + fib(k-2);

}

}

Recurrence relation for Fibonacci:

bn = bn-1 + bn-2

b1 = 1, b0 = 0

Computing the recurrence relation for n = 4:

b4 = b3 + b2

= b2 + b1 + b1 + b0

= b1 + b0 + 1 + 1 + 0

= 1 + 0 + 1 + 1 + 0

= 3

This Fibonacci algorithm is especially bad because each time the function is executed, it will make two

function calls to itself each of which in turn makes two more function calls and so on until they "bottom out"

at 1 or 0. This is an example of "tree recursion", and grows exponentially in time and linearly in space

requirements.[7]

. The author of course failed to consider an alternative approach.

Pseudocode

function fib is:

input: integer Times such that Times >= 0, relative to TwoBack and

OneBack

long TwoBack such that TwoBack = fib(x)

long OneBack such that OneBack = fib(x)

1. if n is 0, return TwoBack

2. if n is 1, return OneBack

3. if n is 2, return TwoBack + OneBack

4. otherwise, return [ fib(Times-1, OneBack, TwoBack + OneBack) ]

end fib

To obtain the tenth number in the Fib. sequence, one must perform Fib(10,0,1). Where 0 is considered

TwoNumbers back and 1 is considered OneNumber back. As can be seen in this approach, no trees are being

created, therefore the efficiency is much greater, being a linear recursion. The recursion in condition 4,

shows that OneNumber back becomes TwoNumbers back, and the new OneNumber back is calculated,

simply decrementing the Times on each recursion.

Implemented in the Java or the C# programming language: public static long fibonacciOf(int times, long twoNumbersBack, long

oneNumberBack) {

if (times == 0) { // Used only for

fibonacciOf(0, 0, 1)

return twoNumbersBack;

} else if (times == 1) { // Used only for

fibonacciOf(1, 0, 1)

return oneNumberBack;

} else if (times == 2) { // When the 0 and 1

clauses are included,

return oneNumberBack + twoNumbersBack; // this clause merely

stops one additional

} else { // recursion from

occurring

return fibonacciOf(times - 1, oneNumberBack, oneNumberBack +

twoNumbersBack);

}

}

[edit] Greatest common divisor Another famous recursive function is the Euclidean algorithm, used to compute the greatest common divisor

of two integers. Function definition:

Pseudocode (recursive):

function gcd is:

input: integer x, integer y such that x >= y and y > 0

1. if y is 0, return x

2. otherwise, return [ gcd( y, (remainder of x/y) ) ]

end gcd

Recurrence relation for greatest common divisor, where x%y expresses the remainder of x / y:

gcd(x,y) = gcd(y,x%y) gcd(x,0) = x

Computing the recurrence relation for x = 27 and y = 9:

gcd(27, 9) = gcd(9, 27 % 9)

= gcd(9, 0)

= 9

Computing the recurrence relation for x = 259 and y = 111:

gcd(259, 111) = gcd(111, 259 % 111)

= gcd(111, 37)

= gcd(37, 0)

= 37

Notice that the "recursive" algorithm above is in fact merely tail-recursive, which means it is equivalent to

an iterative algorithm. Below is the same algorithm using explicit iteration. It does not accumulate a chain of

deferred operations; rather, its state is maintained entirely in the variables x and y. Its "number of steps

grows the as the logarithm of the numbers involved."[8]

Pseudocode (iterative):

function gcd is:

input: integer x, integer y such that x >= y and y > 0

1. create new variable called remainder

2. begin loop

1. if y is zero, exit loop

2. set remainder to the remainder of x/y

3. set x to y

4. set y to remainder

5. repeat loop

3. return x

end gcd

The iterative algorithm requires a temporary variable, and even given knowledge of the Euclidean algorithm

it is more difficult to understand the process by simple inspection, although the two algorithms are very

similar in their steps.

[edit] Towers of Hanoi Main article: Towers of Hanoi

For a full discussion of this problem's description, history and solution see the main article or one of the

many references.[9][10]

Simply put the problem is this: given three pegs, one with a set of N disks of

increasing size, determine the minimum (optimal) number of steps it takes to move all the disks from their

initial position to another peg without placing a larger disk on top of a smaller one.

Function definition:

Recurrence relation for hanoi:

hn = 2hn − 1 + 1 h1 = 1

Computing the recurrence relation for n = 4:

hanoi(4) = 2*hanoi(3) + 1

= 2*(2*hanoi(2) + 1) + 1

= 2*(2*(2*hanoi(1) + 1) + 1) + 1

= 2*(2*(2*1 + 1) + 1) + 1

= 2*(2*(3) + 1) + 1

= 2*(7) + 1

= 15

Example Implementations:

Pseudocode (recursive):

function hanoi is:

input: integer n, such that n >= 1

1. if n is 1 then return 1

2. return [2 * [call hanoi(n-1)] + 1]

end hanoi

Although not all recursive functions have an explicit solution, the Tower of Hanoi sequence can be reduced

to an explicit formula.[11]

An explicit formula for Towers of Hanoi:

h1 = 1 = 21 - 1

h2 = 3 = 22 - 1

h3 = 7 = 23 - 1

h4 = 15 = 24 - 1

h5 = 31 = 25 - 1

h6 = 63 = 26 - 1

h7 = 127 = 27 - 1

In general:

hn = 2n - 1, for all n >= 1

Binary search The binary search algorithm is a method of searching an ordered array for a single element by cutting the

array in half with each pass. The trick is to pick a midpoint near the center of the array, compare the data at

that point with the data being searched and then responding to one of three possible conditions: the data is

found, the data at the midpoint is greater than the data being searched for, or the data at the midpoint is less

than the data being searched for.

Recursion is used in this algorithm because with each pass a new array is created by cutting the old one in

half. The binary search procedure is then called recursively, this time on the new (and smaller) array.

Typically the array's size is adjusted by manipulating a beginning and ending index. The algorithm exhibits

a logarithmic order of growth because it essentially divides the problem domain in half with each pass.

Example Implementation of Binary Search: /*

Call binary_search with proper initial conditions.

INPUT:

data is a array of integers SORTED in ASCENDING order,

toFind is the integer to search for,

count is the total number of elements in the array

OUTPUT:

result of binary_search

*/

int search(int *data, int toFind, int count)

{

// Start = 0 (beginning index)

// End = count - 1 (top index)

return binary_search(data, toFind, 0, count-1);

}

/*

Binary Search Algorithm.

INPUT:

data is a array of integers SORTED in ASCENDING order,

toFind is the integer to search for,

start is the minimum array index,

end is the maximum array index

OUTPUT:

position of the integer toFind within array data,

-1 if not found

*/

int binary_search(int *data, int toFind, int start, int end)

{

//Get the midpoint.

int mid = start + (end - start)/2; //Integer division

//Stop condition.

if (start > end)

return -1;

else if (data[mid] == toFind) //Found?

return mid;

else if (data[mid] > toFind) //Data is greater than toFind,

search lower half

return binary_search(data, toFind, start, mid-1);

else //Data is less than toFind,

search upper half

return binary_search(data, toFind, mid+1, end);

}

Recursive data structures (structural recursion)

Main article: recursive data type An important application of recursion in computer science is in defining dynamic data structures such as

Lists and Trees. Recursive data structures can dynamically grow to a theoretically infinite size in response to

runtime requirements; in contrast, a static array's size requirements must be set at compile time.

"Recursive algorithms are particularly appropriate when the underlying problem or the data to be treated are

defined in recursive terms." [12]

The examples in this section illustrate what is known as "structural recursion". This term refers to the fact

that the recursive procedures are acting on data that is defined recursively.

As long as a programmer derives the template from a data definition, functions employ structural recursion.

That is, the recursions in a function's body consume some immediate piece of a given compound value.[5]

Linked lists Below is a simple definition of a linked list node. Notice especially how the node is defined in terms of

itself. The "next" element of struct node is a pointer to a struct node. struct node

{

int n; // some data

struct node *next; // pointer to another struct node

};

// LIST is simply a synonym for struct node * (aka syntactic sugar).

typedef struct node *LIST;

Procedures that operate on the LIST data structure can be implemented naturally as a recursive procedure

because the data structure it operates on (LIST) is defined recursively. The printList procedure defined

below walks down the list until the list is empty (NULL), for each node it prints the data element (an

integer). In the C implementation, the list remains unchanged by the printList procedure. void printList(LIST lst)

{

if (!isEmpty(lst)) // base case

{

printf("%d ", lst->n); // print integer followed by a space

printList(lst->next); // recursive call

}

}

Binary trees Below is a simple definition for a binary tree node. Like the node for Linked Lists, it is defined in terms of

itself (recursively). There are two self-referential pointers - left (pointing to the left sub-tree) and right

(pointing to the right sub-tree). struct node

{

int n; // some data

struct node *left; // pointer to the left subtree

struct node *right; // point to the right subtree

};

// TREE is simply a synonym for struct node * (aka syntactic sugar).

typedef struct node *TREE;

Operations on the tree can be implemented using recursion. Note that because there are two self-referencing

pointers (left and right), that tree operations will require two recursive calls. For a similar example see the

Fibonacci function and explanation above. void printTree(TREE t) {

if (!isEmpty(t)) { // base case

printTree(t->left); // go left

printf("%d ", t->n); // print the integer followed by a

space

printTree(t->right); // go right

}

}

The above example illustrates an in-order traversal of the binary tree. A Binary search tree is a special case

of the binary tree where the data elements of each node are in order.

Recursion versus iteration

In the "factorial" example the iterative implementation is likely to be slightly faster in practice than the

recursive one. This is almost definite for the Euclidean Algorithm implementation. This result is typical,

because iterative functions do not pay the "function-call overhead" as many times as recursive functions, and

that overhead is relatively high in many languages. (Note that an even faster implementation for the factorial

function on small integers is to use a lookup table.)

There are other types of problems whose solutions are inherently recursive, because they need to keep track

of prior state. One example is tree traversal; others include the Ackermann function and divide-and-conquer

algorithms such as Quicksort. All of these algorithms can be implemented iteratively with the help of a

stack, but the need for the stack arguably nullifies the advantages of the iterative solution.

Another possible reason for choosing an iterative rather than a recursive algorithm is that in today's

programming languages, the stack space available to a thread is often much less than the space available in

the heap, and recursive algorithms tend to require more stack space than iterative algorithms. However, see

the caveat below regarding the special case of tail recursion.

Tail-recursive functions Main article: Tail recursion

Tail-recursive functions are functions ending in a recursive call that does not build-up any deferred

operations. For example, the gcd function (re-shown below) is tail-recursive; however, the factorial function

(also re-shown below) is "augmenting recursive" because it builds up deferred operations that must be

performed even after the final recursive call completes. With a compiler that automatically optimizes tail-

recursive calls, a tail-recursive function such as gcd will execute using constant space. Thus the process it

generates is iterative and equivalent to using imperative language control structures like the "for" and

"while" loops.

Tail recursion: Augmenting recursion:

//INPUT: Integers x, y such that x >=

y and y > 0

//INPUT: n is an Integer such

that n >= 1

int gcd(int x, int y)

{

if (y == 0)

return x;

else

return gcd(y, x % y);

}

int fact(int n)

{

if (n == 1)

return 1;

else

return n * fact(n - 1);

}

The significance of tail recursion is that when making a tail-recursive call, the caller's return position need

not be saved on the call stack; when the recursive call returns, it will branch directly on the previously saved

return position. Therefore, on compilers which support tail-recursion optimization, tail recursion saves both

space and time.

Order of function calling The order of calling a function may change the execution of a function, see this example in C language:

Function 1 void recursiveFunction(int num) {

if (num < 5) {

printf("%d\n", num);

recursiveFunction(num + 1);

}

}

[edit] Function 2 with swapped lines void recursiveFunction(int num) {

if (num < 5) {

recursiveFunction(num + 1);

printf("%d\n", num);

}

}

[edit] Direct and indirect recursion

Direct recursion is when a function calls itself. Indirect recursion is when (for example) function A calls

function B, function B calls function C, and then function C calls function A. Long chains and branches are

possible, see Recursive descent parser.

Complexity and Big-O Notation

Contents Introduction

o Test Yourself #1 o Test Yourself #2

Big-O Notation How to Determine Complexities

o Test Yourself #3 o Test Yourself #4

Best-case and Average-case Complexity When do Constants Matter? Introduction

An important question is: How efficient is an algorithm or piece of code? Efficiency covers lots of

resources, including:

CPU (time) usage memory usage disk usage network usage

All are important but we will mostly talk about CPU time in 367. Other classes will discuss other resources

(e.g., disk usage may be an important topic in a database class).

Be careful to differentiate between:

1. Performance: how much time/memory/disk/... is actually used when a program is run. This depends on the machine, compiler, etc. as well as the code.

2. Complexity: how do the resource requirements of a program or algorithm scale, i.e., what happens as the size of the problem being solved gets larger.

Complexity affects performance but not the other way around. The time required by a method is proportional to the number of "basic operations" that it performs. Here are

some examples of basic operations:

one arithmetic operation (e.g., +, *). one assignment one test (e.g., x == 0) one read one write (of a primitive type)

Some methods perform the same number of operations every time they are called. For example, the size

method of the List class always performs just one operation: return numItems; the number of

operations is independent of the size of the list. We say that methods like this (that always perform a fixed

number of basic operations) require constant time.

Other methods may perform different numbers of operations, depending on the value of a parameter or a

field. For example, for the array implementation of the List class, the remove method has to move over all of

the items that were to the right of the item that was removed (to fill in the gap). The number of moves

depends both on the position of the removed item and the number of items in the list. We call the important

factors (the parameters and/or fields whose values affect the number of operations performed) the problem

size or the input size.

When we consider the complexity of a method, we don't really care about the exact number of operations

that are performed; instead, we care about how the number of operations relates to the problem size. If the

problem size doubles, does the number of operations stay the same? double? increase in some other way?

For constant-time methods like the size method, doubling the problem size does not affect the number of

operations (which stays the same).

Furthermore, we are usually interested in the worst case: what is the most operations that might be

performed for a given problem size (other cases -- best case and average case -- are discussed below). For

example, as discussed above, the remove method has to move all of the items that come after the removed

item one place to the left in the array. In the worst case, all of the items in the array must be moved.

Therefore, in the worst case, the time for remove is proportional to the number of items in the list, and we

say that the worst-case time for remove is linear in the number of items in the list. For a linear-time method,

if the problem size doubles, the number of operations also doubles.

TEST YOURSELF #1

Assume that lists are implemented using an array. For each of the following List methods, say whether (in

the worst case) the number of operations is independent of the size of the list (is a constant-time method), or

is proportional to the size of the list (is a linear-time method):

the constructor add (to the end of the list) add (at a given position in the list) isEmpty contains get

solution

Constant and linear times are not the only possibilities. For example, consider method createList:

List createList( int N ) {

List L = new List();

for (int k=1; k<=N; k++) L.add(0, new Integer(k));

return L;

}

Note that, for a given N, the for-loop above is equivalent to: L.add(0, new Integer(1) );

L.add(0, new Integer(2) );

L.add(0, new Integer(3) );

...

L.add(0, new Integer(N) );

If we assume that the initial array is large enough to hold N items, then the number of operations for each call to add is proportional to the number of items in the list when add is called (because it has to move every item already in the array one place to the right to make room for the new item at position 0). For the N calls shown above, the list lengths are: 0, 1, 2, ..., N-1. So what is the total time for all N calls? It is proportional to 0 + 1 + 2 + ... + N-1.

Recall that we don't care about the exact time, just how the time depends on the problem size. For method

createList, the "problem size" is the value of N (because the number of operations will be different for

different values of N). It is clear that the time for the N calls (and therefore the time for method createList) is

not independent of N (so createList is not a constant-time method). Is it proportional to N (linear in N)?

That would mean that doubling N would double the number of operations performed by createList. Here's a

table showing the value of 0+1+2+...+(N-1) for some different values of N:

N 0+1+2+...+(N-1)

4 6

8 28

16 120

Clearly, the value of the sum does more than double when the value of N doubles, so createList is not linear

in N. In the following graph, the bars represent the lengths of the list (0, 1, 2, ..., N-1) for each of the N calls.

The value of the sum (0+1+2+...+(N-1)) is the sum of the areas of the individual bars. You can see that the

bars fill about half of the square. The whole square is an N-by-N square, so its area is N2; therefore, the sum

of the areas of the bars is about N2/2. In other words, the time for method createList is proportional to the

square of the problem size; if the problem size doubles, the number of operations will quadruple. We say

that the worst-case time for createList is quadratic in the problem size.

TEST YOURSELF #2

Consider the following three algorithms for determining whether anyone in the room has the same birthday

as you.

Algorithm 1: You say your birthday, and ask whether anyone in the room has the same birthday. If anyone

does have the same birthday, they answer yes.

Algorithm 2: You tell the first person your birthday, and ask if they have the same birthday; if they say no,

you tell the second person your birthday and ask whether they have the same birthday; etc, for each person

in the room.

Algorithm 3: You only ask questions of person 1, who only asks questions of person 2, who only asks

questions of person 3, etc. You tell person 1 your birthday, and ask if they have the same birthday; if they

say no, you ask them to find out about person 2. Person 1 asks person 2 and tells you the answer. If it is no,

you ask person 1 to find out about person 3. Person 1 asks person 2 to find out about person 3, etc.

Question 1: For each algorithm, what is the factor that can affect the number of questions asked (the

"problem size")?

Question 2: In the worst case, how many questions will be asked for each of the three algorithms?

Question 3: For each algorithm, say whether it is constant, linear, or quadratic in the problem size in the

worst case.

solution

Big-O Notation We express complexity using big-O notation. For a problem of size N: a constant-time method is "order 1": O(1) a linear-time method is "order N": O(N) a quadratic-time method is "order N squared": O(N2)

Note that the big-O expressions do not have constants or low-order terms. This is because, when N gets large enough, constants and low-order terms don't matter (a constant-time method will be faster than a linear-time method, which will be faster than a quadratic-time method). See below for an example. Formal definition:

A function T(N) is O(F(N)) if for some constant c and for all values of N greater than some value n0: T(N) <= c * F(N)

The idea is that T(N) is the exact complexity of a method or algorithm as a function of the problem size N, and that F(N) is an upper-bound on that complexity (i.e., the actual time/space or whatever for a problem

of size N will be no worse than F(N)). In practice, we want the smallest F(N) -- the least upper bound on the actual complexity. For example, consider T(N) = 3 * N

2 + 5. We can show that T(N) is O(N

2) by choosing c = 4 and n0 = 2.

This is because for all values of N greater than 2:

3 * N2 + 5 <= 4 * N2 T(N) is not O(N), because whatever constant c and value n0 you choose, I can always find a value of N

greater than n0 so that 3 * N2 + 5 is greater than c * N.

How to Determine Complexities In general, how can you determine the running time of a piece of code? The answer is that it depends on what kinds of statements are used. 1. Sequence of statements

2. statement 1; 3. statement 2; 4. ... 5. statement k;

(Note: this is code that really is exactly k statements; this is not an unrolled loop like the N calls to add shown above.) The total time is found by adding the times for all statements: total time = time(statement 1) + time(statement 2) + ... + time(statement k)

If each statement is "simple" (only involves basic operations) then the time for each statement is constant and the total time is also constant: O(1). In the following examples, assume the statements are simple unless noted otherwise.

6. if-then-else statements 7. if (condition) { 8. sequence of statements 1 9. } 10. else {

11. sequence of statements 2

12. }

Here, either sequence 1 will execute, or sequence 2 will execute. Therefore, the worst-case time is the slowest of the two possibilities: max(time(sequence 1), time(sequence 2)). For example, if sequence 1 is O(N) and sequence 2 is O(1) the worst-case time for the whole if-then-else statement would be O(N).

13. for loops 14. for (i = 0; i < N; i++) {

15. sequence of statements

16. }

The loop executes N times, so the sequence of statements also executes N times. Since we assume the statements are O(1), the total time for the for loop is N * O(1), which is O(N) overall.

17. Nested loops First we'll consider loops where the number of iterations of the inner loop is independent of the value

of the outer loop's index. For example: for (i = 0; i < N; i++) {

for (j = 0; j < M; j++) {

sequence of statements

}

}

The outer loop executes N times. Every time the outer loop executes, the inner loop executes M times. As a result, the statements in the inner loop execute a total of N * M times. Thus, the complexity is O(N * M). In a common special case where the stopping condition of the inner loop is j < N instead of j < M (i.e., the inner loop also executes N times), the total complexity for the two loops is O(N2). Now let's consider nested loops where the number of iterations of the inner loop depends on the

value of the outer loop's index. For example: for (i = 0; i < N; i++) {

for (j = i+1; j < N; j++) {

sequence of statements

}

}

Now we can't just multiply the number of iterations of the outer loop times the number of iterations of the inner loop, because the inner loop has a different number of iterations each time. So let's think about how many iterations that inner loop has. That information is given in the following table:

Value of i Number of iterations of inner loop

0 N

1 N-1

2 N-2

... ...

N-2 2

N-1 1

So we can see that the total number of times the sequence of statements executes is: N + N-1 + N-2 + ... + 3 + 2 + 1. We've seen that formula before: the total is O(N2).

TEST YOURSELF #3

What is the worst-case complexity of the each of the following code fragments?

1. Two loops in a row: 2. for (i = 0; i < N; i++) { 3. sequence of statements 4. } 5. for (j = 0; j < M; j++) { 6. sequence of statements 7. }

How would the complexity change if the second loop went to N instead of M? 8. A nested loop followed by a non-nested loop:

9. for (i = 0; i < N; i++) { 10. for (j = 0; j < N; j++) {

11. sequence of statements

12. }

13. }

14. for (k = 0; k < N; k++) {

15. sequence of statements

16. }

17. A nested loop in which the number of times the inner loop executes depends on the value of the outer loop index:

18. for (i = 0; i < N; i++) {

19. for (j = N; j > i; j--) {

20. sequence of statements

21. }

22. }

solution

Statements with method calls:

When a statement involves a method call, the complexity of the statement includes the complexity of

the method call. Assume that you know that method f takes constant time, and that method g takes

time proportional to (linear in) the value of its parameter k. Then the statements below have the time

complexities indicated. f(k); // O(1)

g(k); // O(k)

When a loop is involved, the same rule applies. For example: for (j = 0; j < N; j++) g(N);

has complexity (N2). The loop executes N times and each method call g(N) is complexity O(N).

TEST YOURSELF #4

For each of the following loops with a method call, determine the overall complexity. As above,

assume that method f takes constant time, and that method g takes time linear in the value of its

parameter. 1. for (j = 0; j < N; j++) f(j);

2. for (j = 0; j < N; j++) g(j);

3. for (j = 0; j < N; j++) g(k);

solution

Best-case and Average-case Complexity

Some methods may require different amounts of time on different calls, even when the problem size is the

same for both calls. For example, consider the add method that adds an item to the end of the list. In the

worst case (the array is full), that method requires time proportional to the number of items in the list

(because it has to copy all of them into the new, larger array). However, when the array is not full, add will

only have to copy one value into the array, so in that case its time is independent of the length of the list; i.e.,

constant time.

In general, we may want to consider the best and average time requirements of a method as well as its

worst-case time requirements. Which is considered the most important will depend on several factors. For

example, if a method is part of a time-critical system like one that controls an airplane, the worst-case times

are probably the most important (if the plane is flying towards a mountain and the controlling program can't

make the next course correction until it has performed a computation, then the best-case and average-case

times for that computation are not relevant -- the computation needs to be guaranteed to be fast enough to

finish before the plane hits the mountain).

On the other hand, if occasionally waiting a long time for an answer is merely inconvenient (as opposed to

life-threatening), it may be better to use an algorithm with a slow worst-case time and a fast average-case

time, rather than one with so-so times in both the average and worst cases.

Note that calculating the average-case time for a method can be tricky. You need to consider all possible

values for the important factors, and whether they will be distributed evenly.

When do Constants Matter? Recall that when we use big-O notation, we drop constants and low-order terms. This is because when the

problem size gets sufficiently large, those terms don't matter. However, this means that two algorithms can

have the same big-O time complexity, even though one is always faster than the other. For example,

suppose algorithm 1 requires N2 time, and algorithm 2 requires 10 * N

2 + N time. For both algorithms, the

time is O(N2), but algorithm 1 will always be faster than algorithm 2. In this case, the constants and low-

order terms do matter in terms of which algorithm is actually faster.

However, it is important to note that constants do not matter in terms of the question of how an algorithm

"scales" (i.e., how does the algorithm's time change when the problem size doubles). Although an algorithm

that requires N2 time will always be faster than an algorithm that requires 10*N

2 time, for both algorithms,

if the problem size doubles, the actual time will quadruple.

When two algorithms have different big-O time complexity, the constants and low-order terms only matter

when the problem size is small. For example, even if there are large constants involved, a linear-time

algorithm will always eventually be faster than a quadratic-time algorithm. This is illustrated in the

following table, which shows the value of 100*N (a time that is linear in N) and the value of N2/100 (a time

that is quadratic in N) for some values of N. For values of N less than 104, the quadratic time is smaller than

the linear time. However, for all values of N greater than 104, the linear time is smaller.

N 100*N N2/100

102 104 102

103 105 104

104 106 106

105 107 108

106 108 1010

107 109 1012