intro to algorithms 2

8/14/2019 Intro to algorithms 2

1/23

11-711 Algorithms for NLP

Introduction to Analysis of Algorithms

Reading:Cormen, Leiserson, and Rivest,

Introduction to Algorithms

Chapters 1, 2, 3.1., 3.2.


2/23

Analysis of Algorithms

Analyzing an algorithm: predicting the computational resources that

the algorithm requires

Most often interested in runtime of the algorithm Also memory, communication bandwidth, etc.

Computational model assumed: RAM

Instructions are executed sequentially

Each instruction takes constant time (different constants)

The behavior of an algorithm may be different for different inputs.

We want to summarize the behavior in general, easy-to-understandformulas, such as worst-case, average-case, etc.

1 11-711 Algorithms for NLP


3/23


Generally, we are interested in computation time as a function of

input size.

Different measures of input size depending on the problem

Examples:

Problem Input Size

Sorting List of numbers to sort # of numbers to sort

Multiplication Numbers to multiply # of bits

Graph Set of nodes and edges # of nodes + # of edges

Running time of an algorithm: number of primitive steps

executed

Think of each step as a constant number of instructions, each of

which takes constant time.

Example: Insertion-Sort (CLR, p. 8)



4/23

Insertion Sort (A)

cost times

1. for j

2 to length[A]

n

2. do key

A[j]

n-1

3.

4. i

j - 1

n-1

5. while i 0 and A[i] key

6. do A[i+1]

A[i]

!

#

7. i

i-1

$

!

#

8. A[i+1] key % n-1



5/23

Insertion-Sort

&

'

: # of times the while loop in line 5 is executed for the value(

)

0

1

2

3

5

6

1

7

5

8

0

1

9

@

2

7

5

A

0

1

9

@

2

7

5

B

C

D

E

8

G

D

7

5

H

C

D

E

8

0

G

D

9

@

2

7

5

I

C

D

E

8

0

G

D

9

@

2

7

5

P

0

1

9

@

2

Q In the best case, when the array is already sorted, the running time is

a linear function ofR

.

)

0

1

2

3

S 1

7

T

Q In the worst case, when the array is opposite of sorted order, the

running time is a quadratic function ofR

.

)

0

1

2

3

S 1

8

7

T 1

7

5



6/23


U We are mostly interested in worst-case performance because

1. It is an upper bound on the running time for any input.

2. For some algorithms it occurs fairly often.

3. The average case is often roughly as bad as the worst case.

U We are interested in the rate of growth of the runtime as the input size

increases. Thus, we only consider the leading term.

Example: Insertion-Sort has worst caseV

W

X

Y

a

U We will normally look for algorithms with the lowest order of growth

U But not always:

b

for instance, a less efficient algorithm may be preferable if its an

anytime algorithm.



7/23

Analyzing Average Case of Expected Runtime

c Non-experimental method:

1. Assume all inputs of lengthd

are equally likely

2. Analyze runtime for each particular input and calculate theaverage over all the inputs

c Experimental method:

1. Select a sample of test cases (large enough!)

2. Analyze runtime for each test case and calculate the average and

standard deviation over all test cases



8/23

Recursive Algorithms (Divide and Conquer)

e General structure of recursive algorithms:

1. Divide the problem into smaller subproblems.

2. Solve (Conquer) each subproblem recursively.

3. Merge the solutions of the subproblems into a solution for the full

problem.

e

Analyzing a recursive algorithm usually involves solving arecurrence equation.

e Example: Merge-Sort (CLR, p. 13)



9/23

Recursive Algorithms

f

g

h

i

p

q time to divide a problem of sizei

f

r

h

i

p

q time to combine solutions of sizei

f Assume we divide a problem intos

subproblems, each tu

the size of

the original problem.

f The resulting recurrence equation is:

v

h

i

p

q

w

h

x

p

i

sv

h

u

p

g

h

i

p

r

h

i

p



10/23

Merge

: Initialization

times

1.

j

k

l

m

n

o

1

2. k m

l 1

3. create arrays

jo

and

o

1

4. fork

to

j

j

5. do

k

n

o

m

j

6. forz

k

to

m

j

7. do

z

k

l

o

z

m

j

8.

j

o

k {

1

9.

o

k {

1

10.k

1

11.z

k

1



11/23

Merge|

}

~ ~ ~

(cont.): Actual Merge

times

12. for

to

n

13. do if n

14. then

15.

16. else

17.



12/23

Merge Sort (A,p,r)

times

1. ifp

r

2. then q

(p+r)/2

3. Merge-Sort(A,p,q)

4. Merge-Sort(A,q+1,r)

5. Merge(A,p,q,r)



13/23

A note on Implementation

Note that in the pseudocode of the Merge operation, the for-loop is

always executed

times, no matter how many items are in each of

the subarrays.

This can be avoided in practical implementation.

Take-home message: dont just blindly follow an algorithm.



14/23

Merge-Sort

-

According to the Merge-Sort algorithm, we divide the problem into 2subproblems of size

.

The resulting recurrence equation is:

By solving the recurrence equation we get

,

much better than

for Insertion-Sort.

There are several ways of solving this recurrence equation. The

simplest one uses a recurrence tree, where each node in the tree

indicates what the cost of solving it is. Then inspecting the tree cangive us the solution.



15/23

In this case, we get

, because we divide the problem in half in

each recursive step.



16/23

Growth of Functions

We need to define a precise notation and meaning for order of growth

of functions.

We usually dont look at exact running times because they are toocomplex to calculate and not interesting enough.

Instead, we look at inputs of large enough size to make only the order

of growth relevant - asymptotic behavior.

Generally, algorithms that behave better asymptotically will be the

best choice (except possibly for very small inputs).



17/23

-notation (Theta notation)

Asymptotically tight bound

Idea:

if both functions grow equally fast

Formally:

there exist positive constants

,

, and

such that

for

all

means a constantfunction or a constant



18/23

-notation (Big-O notation)

Asymptotic upper bound

Idea:

if

grows faster than

, possibly much

faster

Formally:


and

such that

for all

Note:

Big-O notation often allows us to describe runtime by just

inspecting the general structure of the algorithm.

Example: The double loop in Insertion-Sort leads us to conclude

that the runtime of the algorithm is

.

We use Big-O analysis for worst-case complexity since that

guarantees an upper bound on any case.



19/23

-notation (Big-Omega notation)

Asymptotic lower bound

Idea:

if

grows slower than

, possibly

much slower

Formally:


and

such that for all

Theorem:

For any two functions

and

,

iff

and



20/23

-notation (Little-o notation)

Upper bound that is not asymptotically tight

Idea:

if

grows much faster than

Formally:

for any positive constant

, there exists

a constant

such that

for all

We will normally use Big-O notation since we wont want to make

hard claims about tightness.



21/23

-notation (Little-omega notation)

Lower bound that is not asymptotically tight

Idea:

!

"

#

$

%

!

&

!

"

# #

if&

!

"

#

grows much slower than

!

"

#

Note:

!

"

#

'

%

!

&

!

"

# #

if and only if&

!

"

#

'

(

!

!

"

# #

Formally:

%

!

&

!

"

# #

$

)

!

"

#0

for any positive constant1

2

3

, there exists

a constant"

4

2

3

such that3

5

1

&

!

"

#

6

!

"

#

for all"

7

"

4

8



22/23

Comparison of Functions

9 The@

,A

,B ,C

, andD

relations are transitive.

Example:

E

F

G

H

I

A

F

P

F

G

H H

Q

P

F

G

H

I

A

F

S

F

G

H H

I

T

E

F

G

H

I

A

F

S

F

G

H H

9 The@

,A

, andB relations are reflexive.

Example:E

F

G

H

I

A

F

E

F

G

H H

9 Note the analogy to the comparison of two real numbers.e.g.

E

F

G

H

I

A

F

P

F

G

H H

U

W

X

Y

e.g.E

F

G

H

I

D

F

P

F

G

H H

U

W

a

Y



23/23

Comparison of Growth of Functions

b Transitivity: all

b Reflexivity:

c

d

e

f

g

h

d

c

d

e

f f

c

d

e

f

g

i

d

c

d

e

f f

c

d

e

f

g

p

d

c

d

e

f f

b Symmetry:

c

d

e

f

g

h

d

q

d

e

f f

iffq

d

e

f

g

h

d

c

d

e

f f

b Transpose symmetry:

c

d

e

f

g

i

d

q

d

e

f f

iffq

d

e

f

g

p

d

c

d

e

f f

c

d

e

f

g

r

d

q

d

e

f f

iffq

d

e

f

g

s

d

c

d

e

f f

b Lack of Trichotomy:

Somec

d

e

f

are neitheri

d

c

d

e

f f

norp

d

c

d

e

f f


intro to algorithms 2

Documents