computational complexity - lecture 01 - introduction - tom kelsey

Computational ComplexityLecture 01 - Introduction and Revision

Tom Kelsey & Susmit Sarkar

School of Computer ScienceUniversity of St Andrews

http://www.cs.st-andrews.ac.uk/~tom/[email protected]

Tom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 1

Administrivia

Lectures every Thursday, 2 - 4:30, Physics CNo "standard" tutorials - one revision tutorial at the endThree practicals – check exact times & dates on MMS

1 P1 is an essay - you tell me about fundamental concepts –Thu Week 3

2 P2 – You write code that converts one problem into another,then solves it – Fri Week 6

3 P3 – I give you algorithm code, you tell me everything thereis to know about its complexity – Thu Week 9

4 P2 & P3 may change! I need to consult with Susmit Sarkar

2 hour exam, answer any 3 from 4 questions


Books & Sources

Computational Complexity: A Modern Approach bySanjeev Arora and Boaz BarakIntroduction to Automata Theory, Languages, andComputation, 2nd Ed. by John E. Hopcroft, Rajeev Motwaniand Jeffrey D. UllmanIntroduction to Automata Theory, Languages, andComputation by John E. Hopcroft and Jeffrey D. UllmanMatrix Computations by Gene H. Golub and Charles F. vanVan LoanIntroduction to the Theory of Computation by MichaelSipserA Practical Introduction to Data Structures and AlgorithmAnalysis by Clifford A. ShafferAlgorithm Design by Jon Kleinberg and Eva Tardos


Notes

Notes

Notes

http://www.cs.st-andrews.ac.uk/~tom/

mailto:[email protected]

Books & Sources

This course is a mashup between:

My notes for Machines & Computation (based on MikeAtkinson’s notes based on Sipser)Steve’s notes for CS3109 Computability & ComplexitySteve’s notes for Graphs & AlgorithmsKevin Wayne’s notes for COS 423 Theory of Algorithms(Princeton)Extra examples/results from the books listed on theprevious slideInteresting stuff from my research


Why are we all here?

If you design, modify or deploy an algorithm, someone willask you about its complexity

If you clearly don’t understand the question, we’ve failed asa schoolIf you answer the question incorrectly, you look like a halfwitTo give correct answers, you need a good understanding oftheoretical limits and practical assessment

If someone designs, modifies or deploys an algorithm, youwill want to know about its complexity

If you can’t correctly formulate the question, we’ve failed asa schoolIf you can’t spot incorrect answers, you look like a halfwitIf you can prevent inefficiency – or drastically improveefficiency – you look like a geniusTo assess the correctness of answers, you need a goodunderstanding of theoretical limits and practical assessment


Structure of the course

Theoretical Computer ScienceAbstract models of computationTuring machinesComplexity classes, translations

Practical Computer ScienceFlops, master theorem, complexity classesWorst and average case analysesComplexity of important classes of algorithms


Notes

Notes

Notes

Computational complexity explained in one slide

Nearly all algorithms that allow inputs of any length willrun forever, returning no solutionMany of those that can be solved, can be solved in constanttime for inputs of a fixed sizeThe honest answer to most questions about complexityclasses is "we have no idea"Much of the work done in complexity is irrelevantThere is a huge mismatch between theory and practiceNevertheless, ignoring complexity is not a good idea


This lecture

Go over three topics you covered in 2nd year

1 Finite state machines2 Context free grammars (maybe lecture 2)3 Big O notation

I’ll be using slightly different notation & definitions – go overyour 2nd year notes to make sure you understand everything


Automata

For now, we study various abstract machines - automata -which focus on different aspects of computing powerFor every such abstract machine we define the set ofproblems it can solve - the language of the machineOr, given a set of problems, define the automata that solvesthis set

and no other

Computer ≡ AutomataSyntactically correct program ≡ solvable problemAn automata solves a problem in a precise number ofmachine translations

this number measures the time taken


Notes

Notes

Notes

Automata Theory

Definitions & properties of mathematical models ofcomputationThe finite automaton

Text processingCompilersHardware design

The context-free grammar or pushdown automatonProgramming languagesArtificial intelligence

The Turing machineAny algorithmic process


Computability Theory

We can’t solve all problems using computersWhich problems can we solve?Which problems can’t we solve?

why not?

What if we use a different computer?No solution known or no solution possible?Where are the boundaries between possible and impossible?


Complexity Theory

Why are some problems hard?Why are others easy?Can we classify families of problems?Some problems are easy most of the timeWhat if new methods were devised?Time vs Space complexityWhat if better computers were invented?


Notes

Notes

Notes

This Course is Important

Many people canDevelop computer graphicsAnalyse operating systemsUtilize component technologywhatever

A Computer Scientist understands the practical & possiblelimits of computation


Mathematics

Basic mathematical toolsChapter 0 of Sipser gives a short review of these topicsSetsSequencesFunctions & relationsStrings & languagesBoolean logic – see your 2nd year notes

Proofs


Sequences

A sequence is an ordered list of objects( 7, 21, 57 )( apples, oranges, pears )( 0, 1, 1, 2, 3, 5, 8, 13, . . . )

Duplicates are allowedOrder matters

( jump, count, pull ripcord, land )

A tuple is a finite sequenceQuite often, the next term in a sequence depends on itspredecessors


Notes

Notes

Notes

Cartesian products

Suppose A = {1, 2, 3} and B = {x, y}A× B is the set of ordered pairs from A and B:

{(1, x), (1, y), (2, x), . . . , (3, y)}

|A× B| = |A|.|B|

We write A×A as A2

A×A× · · · ×A︸︷︷︸k times

= Ak


Functions

N is the set of natural numbers,Z is the set of integers,Q is the set of rational numbers andR is the set of real numbers.

A function takes input from a domain D, which is often aCartesian product

Output is in its range, which is a setµ : N 7→N

µ(n) =

1 if n = 10 if a2 | n for some a > 1(−1)r if n has r distinct prime factors


Predicates

A predicate is function with domain D and range {true, false}

pass? : {0, 1, . . . , 20} × {0, 1, 2, . . . , 60} 7→ {true, false}

se : N×N 7→ {true, false}

se(73, 6) = false

se(888888, 222220) = true


Notes

Notes

Notes

Relations

A relation is predicate with domain Ak

Often k = 2 and we use infix notation:

a ∼R b means R(a, b) = true

Properties:

Reflexive: a ∼ a ∀a ∈ ASymmetric: a ∼ b iff b ∼ a ∀a, b ∈ AAntisymmetric: a ∼ b and b ∼ a⇒ a = bTransitive: a ∼ b and b ∼ c⇒ a ∼ c

ReST & RoTA


Graph of a relation


Strings & languages

An alphabet is a set of symbols

Σ = {0, 1}

Γ = {a, b, c, . . . , z}

A string over Σ is a tuple of symbols from Σ

w = 010101110|w| = 9w2 = 1

x = penguin|x| = 8w2 = e


Notes

Notes

Notes

Strings & languages

Empty string is denoted by ε

|ε| = 0

Reverse of a string: wR = wnwn−1 . . . w1

z is a substring of w if z appears consecutively within w

Concatenation:

xy = x1x2 . . . xny1y2 . . . ym

xk = xx · · · x︸︷︷︸k times


Strings & languages

Lexicographic ordering of strings is as you’d expect,except thatlength matters

{0, 1} → (ε, 0, 1, 00, 01, 10, 11, 000, 001, . . .)

A language is a set of stringsEnglishEspañolJavaC++{ε, 0, 1, 00, 01, 10, 11, 000, 001, . . .}


Induction

Suppose we have a series of dominoes, each 5cm apart.If we roll a marble at the first domino, will they all fall overin sequence?Two questionsDoes the marble knock over domino 1?

can we get to the first step?is the 5cm gap small enough?

can we get from any step to the next?


Notes

Notes

Notes

Structural Induction

Works whenever we build incrementally from a small baseNatural numbers

base is 0 or 1, increment by adding 1Sets

base is ∅, increment by inserting membersStrings

base is ε, increment by appending symbols

Lists, Graphs, etc.


Structural Induction

The power set P(X) is the set of all subsets of XIf |S| = n, show that |P(S)| = 2n

Base case|∅| = 0, ∅ has one subset, and 20 = 1

Induction stepsuppose that if |S| = n, then |P(S)| = 2n

insert a new element e into Sfor each p ∈ P(S) we can form p∪ {e}giving another 2n members of P(S∪ {e})2n + 2n = 2n+1 and the proof is complete


More on Power Sets

Let P be a power set.Show that ⊆ defines an order relation on PLet X, Y and Z be any members of P.

1 X ⊆ X, so ⊆ is reflexive.2 If X ⊆Y and Y⊆ X, then X = Y, so ⊆ is antisymmetric.3 If X ⊆ Y and Y ⊆ Z, then X ⊆ Z, so ⊆ is transitive


Notes

Notes

Notes

Mathematical method

1 Read the problem carefully2 Draw a picture3 Look for patterns and counterexamples4 Choose a proof method5 Check each possible case6 Write in formal notation7 Look for ways to apply the result


"Proof" by diagrams

Scientifically, diagrams (not geometric constructions) areonly for (counter-)examplesReal and complete proofs might take too much time for anexam or even a practicalFor examination purposes it is sometimes sufficient to showthat you have understood the problem, found the solution,etc., e.g. by:

drawing diagramsgiving a proof sketch (in English)

Get a feeling from the lectures what is requiredRead the exam questions carefully!


Computation

Old way of thinkingOnly tangentially related to

operating systemsnetworksgraphicscommunicationsmediaetc


Notes

Notes

Notes

Computation

Solve a useful problem, quicklydomains: especially military, financialapproximate solutions often acceptedis an exact solution possible, if so how quickly?

Architecture not important for the principlemechanical "calculators" like Charles Babbage enginehowever: lack of flexibility can restrict the computationalpowerwhich is what this part of the course is about

We are interested in algorithmic problems: applying afunction f to an input x, what is f (x)?


Computation

Canonical useful problem from early 20th centuryDreadnought class battleships developed, having samecalibre gunsHow do you ensure a salvo hits a target?

Analog gun laying computers were developedx1 = range, x2 = barrel temp., x3 = propellant temp., x4 =latitude, x5 = own course & speed, x6 = target course &speed, etc.

For input x, what is f (x)?f returns direction & elevation for gunsapproximate values, since slow firing is sub-optimalaccurate results, since missing the target is pointless


Jacquard Loom (1801)

programmable: used punch cards to control a sequence ofoperationsno calculations, but important step in the history ofcomputing hardware


Notes

Notes

Notes

Zuse Z3 (1941)

programmablemathematical computationsfully automaticconditional branchingcomputer


First 5 "Computers"


Motivating example 1

Suppose we’re writing C/C++ codeWe send our code to a C pre-processor prior to compilationOne of its tasks is to strip out comments

Lines starting with // in C++Text within /* */ in C

How can we model this?


Notes

Notes

Notes


ASCII symbols are read oneby oneInitial state unless and untila / is seenIf the next symbol is

/ go to the C++ state;wait for an EOL* go to the C state; waitfor a "*/" pairanything else; return tothe initial state



Note that:always in exactly one of a finite number of statesthe next state depends on the next symbolthere is a rule for every symbol at each statewe don’t have to visit every statewe could think of the initial state as an accepting state



We can formalise this as:Q ={none, slash, star, C, C++}Σ = { ASCII chars} ∪{ ASCII device chars}δ : Q× Σ 7→ Q is thetransition functionnone ∈ Q is the startstate{none} ⊆ Q is the set ofaccept states

These 5 define a finiteautomaton M

accepts correctlycommented C, C++rejects incorrectlycommented C, C++we call correctlycommented C, C++ thelanguage of M, and writeL(M)

All other finite automataare defined in the same way


Notes

Notes

Notes

Motivating example 2 – grep nano

We have an informal diagram, and we can posit a ΣFrom this we can write down the formal definitionThe start and accept states are usually easyThe transition function is given by a table

a row for each statea column for each symbol

We can convert to and from diagrams and 5-part definitionsTom Kelsey and Susmit Sarkar CS3052-CC 2013-31-01 40

DFAs

Informally:A deterministic finite automaton (DFA) over a finitealphabet Σ is a finite directed graph with the property thateach node emits one labeled edge for each distinct elementof Σ. There is a unique start node, and a non-empty set ofaccept nodes.


DFAs

Formally:A deterministic finite automaton (DFA), M, over a finitealphabet Σ is a 5-tuple

M = (Q, Σ, δ, q0, F)

If M is a DFA, and A = {all strings accepted by M}, thenL(M) = AM recognises AM accepts strings from A


Notes

Notes

Notes

DFAs

We now have a simple model of computation"compute" means "decide if this string is in a language"For any input string, a DFA will either accept or reject itAlphabets aren’t important

simple pre-process can check that an input string is OKwe can always convert to binary stringsin our examples, we could use the ASCII bitstringrepresentationswe’d need more states and arrows, but the transition tableonly has 2 columns

Everything is deterministic – no gaps in the transition table


The computation of a DFA

Let s1s2s3 . . . sn ∈ Σ∗ be an input stringΣ∗ = { all finite strings over Σ}each sj ∈ Σ∗

Consider the sequence of states (q0, q1, q2, . . . , qn)q0 is the start state and, for j = 1 . . . n :qj = δ(qj−1, sj) is the sequence of states the input causes themachine to go through


Formal Acceptance

Let M = (Q, Σ, δ, q0, F) be a DFA and w = s1s2s3 . . . sn ∈ Σ∗

be an input stringM accepts w if a sequence of states r0, r1, r2, . . . , rn exists in Qsuch that:

1 r0 = q02 ri+1 = δ(ri, xi+1) for i = 1 . . . , n3 rn is a member of F


Notes

Notes

Notes

Asymptotic Order of Growth

K. Wayne – Princeton








Notes

Notes

Notes




Summary

This course builds on some concepts considered in pre-honoursAsymptotic complexityFinite state machinesBoolean propositions, truth tables, tautology, validity, ....Regular expressions

The idea is to assess what is and isn’t possible. If it is possible,how long is it likely to take?


Notes

Notes

Notes

computational complexity - lecture 01 - introduction - tom kelsey

Documents