CS 345: Chapter 10Parallelism, Concurrency, and

Alternative Models

Or, Getting Lots of Stuff Done at Once

• Parallelism is the use of many processors to solve a problem.

• When we want to ask the question,

“Will parallelism result in some intractable problems becoming tractable?”

we need to distinguish between fixed parallelism and expanding parallelism.

• Fixed Parallelism: The number of processors remains fixed as the problem size grows.

• Expanding Parallelism: The number of processors available grows as the problem size grows.

• Since the number of processors available affects the running time of a parallel algorithm, how do we balance the two?

• Product Complexity: Time x Size.

• See Figure 10.2, page 262.

• Note that since any parallel algorithm can be serialized on a single processor, the best product complexity cannot be lower than the lower bound of the problem’s sequential complexity.

• Among the difficulties of parallel algorithms is “How should processors communicate?”

• There are two basic approaches.– Shared Memory– Fixed Networks

Shared Memory

• These types of machines are called Multi‑Processors.

• Is the shared memory only for reading values or can it be used to write values?

• If the shared memory is for writing, then there must be some method for resolving conflicts.

Fixed-Network Machines

• These types of computers are called Multi‑Computers.

• A processor is connected to a fixed number of neighbors, and each processing element has its own private memory.

• There are many types of network designs: mesh/torus, fat trees, hypercubes, and others based on graph theory.

• Some networks are designed for specific problems. Others are for general purpose, but are better at some algorithms than others.

• Some issues with large multi-computers– What communication model is used?

• Store and forward• Worm-hole routing• Virtual cut-through

– Deadlock and Livelock– Fault-tolerance.

Systolic Networks• One type of network for parallel

processing is a systolic network.

• See figure 10.5 on page 266.

• This is similar to an assembly line. Each processor performs a set task on some data and passes it on to its neighbor.

• In the example, there is an n x m matrix. A sequential algorithm takes O( n x m ), but a systolic network takes O( n + m ).

No. Since any parallel algorithm can be serialized, a parallel algorithm cannot solve an undecidable or non-computable problem.

Can a parallel algorithm solve an undecidable problem?

Can a parallel algorithm turn an intractable problem into a tractable

problem?

• Many problems in NP, including some NP-Complete problems, have been shown to have a polynomial time parallel solution.

• However– The number of processors required for the

solution is exponential.

– NP-complete problems are not known to be intractable, so this does not necessarily mean that parallelism can remove inherent intractability.

– It is not clear that a parallel algorithm that uses only a polynomial number of instructions but requires an exponential number of processors can actually run in polynomial time on a real computer.

• Thus, the question of whether a parallel algorithm can remove inherent intractability, is still an open question.

The Parallel Computation Thesis

• Part of this thesis is the claim that parallel time is the same as sequential memory.

If P can be solved sequentially using S space for inputs of length N, then it can be solved in parallel time that is no worse than polynomial time in S.

Alternately, if P can be solved in parallel time T with inputs of length N, it can be solved sequentially using memory bounded by a polynomial in T.

• So, any algorithm solvable by a sequential algorithm in polynomial space can be solved by a parallel algorithm in polynomial time.

• That is:

Sequential-PSpace = Parallel-PTime

• Thus the question of whether there are intractable problems that become tractable when using parallelism comes down to the question:

Does PSpace contain intractable problems?

• This, like the question, “Does P=NP?”, is still an open question.

The Class NC(Nick’s Class)

• In general, a polynomial time parallel algorithm cannot claim to be tractable since it may require an exponential number of processors.

• A problem is in NC if – It runs in polylogrithmic time.– It requires only a polynomial number of

processors.

• All problems in NC are also in P, but it is not known if the converse is true.

• Most researchers believe that the two classes are not the same.

• Thus, we have:

NC P NP PSpace

Conjecture 1

There are problems solvable in reasonable sequential space – reasonable parallel time that cannot be solved in reasonable sequential time, even using non-determinism.

Conjecture 2

There are problems solvable in reasonable sequential time only if non-determinism is used.

Conjecture 3

There are problems solvable in reasonable sequential space, but not in very fast parallel time using reasonable hardware size.