complexity analsis
TRANSCRIPT
-
7/23/2019 Complexity Analsis
1/26
1
Computational Complexity
Computer Science Theory 2
The Turing Machine
Motivating idea
Build a theoretical human computer
Likened to a human with a paper and pencil that
can solve problems in an algorithmic way
The theoretical machine provides a means to
determine:
If an algorithm or procedure exists for a given problem
What that algorithm or procedure looks like
How long would it take to run this algorithm or
procedure.
-
7/23/2019 Complexity Analsis
2/26
2
Computer Science Theory 3
Decision Problem
Remember: A decision problem is a problem that has a
yes/no answer
Example:
Is a given string x a palindrome (Is x pal?)
Is a given context free language empty?
Computer Science Theory 4
What makes a good algorithm?
Suppose we know that a decision problem is
decidable, how do we know that there is a
good algorithm that solves the problem?
Consider running time on a TM
Actually, consider the complexity of a TM that
can solve the problem.
-
7/23/2019 Complexity Analsis
3/26
3
Computer Science Theory 5
Complexity
Complexity refers to the rate at which thestorage or time grows as a function of the
input to the algorithm
T(n) = time complexity (amount of time an
algorithm will take based on input)
S(n) = space complexity (amount of space an
algorithm will take based on input)
For the rest of this lecture, well only
consider T(n) although the discussion canalso be applied to S(n)
Computer Science Theory 6
Units
What does T(n) return?
A basic operation can be defined as a move a
TM makes in accepting / rejecting a string
T(n) = number of TM moves (note that the
actual running time will depend upon the
specific machine characteristics, but for
algorithm comparison purposes, the actual
machine doesnt matter).
-
7/23/2019 Complexity Analsis
4/26
4
Computer Science Theory 7
Asymptotic Analysis
Used to determine the growth in T(n) Based on the idea that as the input to an
algorithm gets large
The complexity will become proportional to a
known function.
Notations:
O (Big-O) upper bounds on the complexity
(Big-Theta) tight bounds on the complexity
Computer Science Theory 8
Asymptotic Analysis
Big-O (order of)
T(n) = O(f(n)) if and only if there are constants
c0 and n0 such that
T(n) c0f(n) for all n n0
What this really means:
Eventually, when the size of the input gets large
enough, the upper bounds on the runtime will be
proportional to the function f(n).
-
7/23/2019 Complexity Analsis
5/26
5
Computer Science Theory 9
Asymptotic Analysis
(Big-Theta) tight bounds on thecomplexity
T(n) = (f(n)) means that T(n) and f(n) have
(approximately) the same growth rate.
Computer Science Theory 10
Algorithm Efficiencies
-
7/23/2019 Complexity Analsis
6/26
6
Computer Science Theory 11
Are you a good algorithm?
So what should be the cutoff between agood algorithm and a bad algorithm?
In the 1960s, Jack Edmonds proposed:
A good algorithm is one whose running time is a
polynomial function of the size of the input
Other algorithms are bad
This definition was adopted:
A problem is called tractable if there exists a
good (polynomial time) algorithm that solves it.
A problem is called intractable otherwise.
Computer Science Theory 12
Is this a valid cutoff?
-
7/23/2019 Complexity Analsis
7/26
7
Computer Science Theory 13
Running Time on TMs
We can use the number of steps taken by aTuring Machine to represent the running
time of a program.
It doesnt really matter what we measure, as
long as we stay consistent with the
algorithms we are comparing.
Computer Science Theory 14
Our Friend, pal
Lets assume that we have a TM that
recognizes palindromes using the following
algorithm:
The machine will go all of the way to the end of
the input stream and retrieve the last character
(replacing it with a blank). It will then move all
of the way back to the beginning of the input
and compare the retrieved character to the first
character (replacing the first character with ablank). It continues doing this as long as there
are characters and it doesnt fail.
-
7/23/2019 Complexity Analsis
8/26
8
Computer Science Theory 15
Our Friend, pal
Assuming the length of the string is n, forthe first comparison it will make 4n + 4
moves. The second comparison will be 4(n
1) + 4. The third 4(n 2) + 4, etc.
Through the magic of math, it turns out that
this will be proportional to 4n2.
Computer Science Theory 16
Our Friend, pal - again
Lets try pal by another algorithm:
Read and remember the first character, move all of
the way to the end and read the last character
compare it to the one you remembered.
For the first comparison, you will need 2n + 3
moves. The second will need 2(n 1) + 3
moves, and so on.
It turns out that this algorithm is proportional
to 2n2.
-
7/23/2019 Complexity Analsis
9/26
9
Computer Science Theory 17
pal - analysis
Which of these grows faster?
Answer: they both have the same gowth rate
Which of these algorithms is better?
Answer: the second one. While they both
have the same growth rate, the second one
will run twice as fast.
Computer Science Theory 18
The class P
The class P contains all decision problems
that are decidable by an algorithm that
runs in polynomial time.
Does this define a class of languages?
Yes
The set of all problems whose encodings of yes
instances (a language) is recognized by a TM M
M recognizes the above language in Polynomial Time.
-
7/23/2019 Complexity Analsis
10/26
10
Computer Science Theory 19
The class P
If we take the Church-Turing Thesis to betrue:
P is robust:
Meaning the problems contained in P remain in P
even if we change our basic model of computation.
A basic TM
A Sun, a Mac, even a PC running Windows XP!
Computer Science Theory 20
The class P
Are there actually problems in P?
Most everyday algorithms (e.g. sorting, searching)
are in P
Are there problems not in P?
- Yes, we know that Self-Accepting is not in P
since it isnt even solvable
Are there decidable problems not in P?Lets take a look
-
7/23/2019 Complexity Analsis
11/26
11
Computer Science Theory 21
Satisfiability (SAT)
INSTANCE A Logical expression containing
variables xi
logical connectors &, |, and !
In conjuction normal form (C1 & C2 & C3 & Cn)
PROBLEM
Is there an assignment of truth values to each of
the variables such that the expression will
evaluate to true?
Computer Science Theory 22
Satisfiability (SAT)
Example of an instance (x1 | x2 | x3 ) & (x4 | !x5) & !x6 & (x7 | x8)
One Solution:
x1 = true x5 = false
x2 = false x6 = false
x3 = false x7 = true
x4 = false x8 = false
-
7/23/2019 Complexity Analsis
12/26
12
Computer Science Theory 23
Satisfiability (SAT)
Nave algorithm to solve Systematically consider all combinations of
assignments of True and False values for all
variables and test the expression
In the worst case, for n variables
T(n) = O (2n) exponential time
Computer Science Theory 24
Satisfiability (SAT)
Can we do better?
No known polynomial algorithm
Either one doesnt exist
One exists and we havent found it yet.
You might get lucky and find a solution on
the first try. However, there is no algorithm to
follow that guarantees a solution in polynomial
time.
-
7/23/2019 Complexity Analsis
13/26
13
Computer Science Theory 25
Non-deterministic TM
Lets consider the NDTM: Same as the ordinary TM except:
The transition function will return a set of triplets
(q, x, D)
For each state / symbol combination, zero or more
transitions can be defined.
The machine can choose which transition to take.
Computer Science Theory 26
Non-deterministic TM
If we map all paths that can possibly be
taken:
Number of paths will be exponential
Example:
Suppose at each of n states, for each character, there
are two possible moves.
Number of paths 2n
-
7/23/2019 Complexity Analsis
14/26
14
Computer Science Theory 27
Satisfiability (SAT)
Running SAT on a non-deterministic TM. Step 1: Guess a set of boolean assignments to
each variable (this is the non-deterministic part
2n different choices)
Step 2: Evaluate the truth of the entire
expression using the guessed assignment (this
part is deterministic)
Can certainly do each step in polynomial time
Computer Science Theory 28
The class NP (Nondeterministic Polynomial)
The class NP contains all decision problems that
are decidable by a non-deterministic algorithm
that runs in polynomial time.
Represents algorithms that run in exponential time
Does this define a class of languages? -- Yes
The set of all problems whose encodings of yes instances (a
language) is recognized by a NDTM M
M recognizes the above language in Polynomial Time.
Are there problems in NP?
Well, for one SAT is
-
7/23/2019 Complexity Analsis
15/26
15
Computer Science Theory 29
The class NP
Clearly P is a subset of NP
P
NP
Is there something in
here?
Computer Science Theory 30
Reducing One Language to Another
Worked well for decidabilityLets use it
here.
Basic idea
Take an encoding of one problem
Convert it to another problem we know to be in
P or NP
Conversion must be done in polynomial time!
-
7/23/2019 Complexity Analsis
16/26
16
Computer Science Theory 31
Reducing One Language to Another
Formally (for complexity) Let L1 and L2be languages over 1 and 2
We say L1 is polynomial time reducible to L2(L1 p L2) if
There exists a Turing computable function
f: 1* 2
* such that
x L1 iff f(x) L2
f can be computed in polynomial time.
Computer Science Theory 32
Reducing One Language to Another
Informally (for complexity)
We can take any encoded instance of one
problem
Use a TM to compute a corresponding encoded
instance of another problem.
In polynomial time
If this other problem has a TM that recognizes the
set of yes encodings in polynomial time (P), we
can run that TM to solve the first problem.
If this other problem has a NDTM that recognizesthe set of yes encodings in polynomial time (NP),
we can run that TM to solve the first problem.
-
7/23/2019 Complexity Analsis
17/26
17
Computer Science Theory 33
The class NP-complete
The hardest of the problems in class NP arein the class of NP-complete problems:
A language L is NP-complete if
L NP
For every other language L1NP
L1 p L
All NP-complete problems are equally
difficult.
Computer Science Theory 34
The class NP-complete
Implications
Hardest problem
Its probably not worth looking for a solution
for an NP-complete problem
How do we show a problem to be NP-complete
Reduction
We need a NP-complete problem to kick things off.
-
7/23/2019 Complexity Analsis
18/26
18
Computer Science Theory 35
The class NP-complete
Guess what? SAT can be shown to be NP-complete
See Cooks Theorem
Computer Science Theory 36
NP-Complete and P
If a polynomial algorithm is found for any
NP-complete problem
A polynomial algorithm can be found for all
NP-complete problems (via polynomial
reduction).
In other words
For any L NP-Complete,
If L P then
P = NP
Finding such a language is still an open
problem.
-
7/23/2019 Complexity Analsis
19/26
19
Computer Science Theory 37
NP-Complete and P
The world of NP
P
NP
Is there something in
here?
After 40 years of researchnobody
knows
NP-complete
Computer Science Theory 38
Practical considerations
-
7/23/2019 Complexity Analsis
20/26
20
Computer Science Theory 39
Practical considerations
Computer Science Theory 40
Practical considerations
-
7/23/2019 Complexity Analsis
21/26
21
Computer Science Theory 41
Some NP-Complete Problems
Traveling Salesman INSTANCE
A finite set of cities C = { c1, c2, , cn }
A distance function d:
d( ci, cj ) = distance between ci and cj
Upper bounds B
PROBLEM
Is there a tour of all cities such that the distance
traveled will be less than B?
Computer Science Theory 42
Some NP-Complete Problems
Traveling Salesman
PROBLEM
Said another way, is there an ordering of the cities
such that
Bccdccdiinii
n
i
ii ++
=
),(),( 111
1
-
7/23/2019 Complexity Analsis
22/26
22
Computer Science Theory 43
Some NP-Complete Problems
NDFA inequivalence INSTANCE
Two nondeterministic finite automata A1 and A2having the same input alphabet.
PROBLEM
Do A1 and A2 recognize different languages?
NOTE
Can be done in polynomial time if A1 and A2 are
deterministic.
Computer Science Theory 44
Some NP-Complete Problems
Crossword Puzzle Construction
INSTANCE
A finite set W * of words
An n by n matrix A consisting of 1s and 0s
PROBLEM
Can an n by n crossword puzzle be built up including
all words from W and blank squares consisting of
the 0 cells in A?
-
7/23/2019 Complexity Analsis
23/26
23
Computer Science Theory 45
Some NP-Complete Problems
You want more?
Computers and Intractability by Michael R.
Garey and David S. Johnson.
Computer Science Theory 46
Dealing with NP-completeness
Some NP-complete problems have solutions
that are, on the average, polynomial.
Restrict the problem
Approximation
Heuristics
Buy a Supercomputer
-
7/23/2019 Complexity Analsis
24/26
24
Computer Science Theory 47
Summary
Complexity P
NP
NP-completeness
Computer Science Theory 48
So There You Have It!
I bet you thought I forgot!
What is a language?
What is a class of languages?
Thanks for playing.
-
7/23/2019 Complexity Analsis
25/26
25
Computer Science Theory 49
Where to next?
Language Design / Compiler Construction Formal Languages
Compiler Construction Lab
Complexity & Computation
Complexity and Computability
Computer Science Theory 50
The Theory Hall of Fame
Stephen Cole Kleene
Regular expressions & FAs
Noam Chomsky
The grammar guy
W. Ogden
Pumping Lemma for CFLs
Kurt Godel
Kicked off study of computability
Alan Turing
Turing Machines
-
7/23/2019 Complexity Analsis
26/26
Computer Science Theory 51
The Theory Hall of Fame
Alonzo Church Unlimited support for TMs
Emil Post
Post Correspondence Problem
H.G. Rice
Decision problems for TMs
Stephen Cook
NP-completeness Aho, Hopcraft, and Ullman
Bell Labs guys work on programming
languages
Computer Science Theory 52
The Theory Hall of Fame
YOU
If you find a polynomial solution to any
problem in NP-complete!