cs4402 – parallel computing lecture 1: classification of parallel computers classification of...
TRANSCRIPT
CS4402 – Parallel Computing
Lecture 1:Classification of Parallel Computers
Classification of Parallel Computation
Important Laws of Parallel Compuation
How I used to make breakfast……….
How to set family to work...
How finally got to the office in time….
What is Parallel Computing?
In the simplest sense, parallel computing is the simultaneous use of multiple computing resources to solve a problem.
Parallel computing is the solution for "Grand Challenge Problems“: weather and climate biological, human genome chemical and nuclear reactions
Parallel Computing is a necessity for some commercial applications: parallel databases, data mining computer-aided diagnosis in medicine
Ultimately, parallel computing is an attempt to minimize time.
Grand Challenges Problems
Reason 1: Speedup
Reason 2: Economy
Resources already available. Taking advantage of non-local resources Cost savings - using multiple "cheap" computing resources instead of
paying for time on a supercomputer.
A parallel system is cheaper than a better processor. Transmission speeds. Limits to miniaturization. Economic limitations.
Reason 3: Scalability
13
14
Types of || Computers
Parallel Computers
Hardware Software
Shared memory
Distributed memory
Hybrid memory
SIMD MIMD
17
The Banking Analogy
Tellers: Parallel Processors
Customers: tasks Transactions: operations Accounts: data
Vector/Array
Each teller/processor gets a very fine-grained task
Use pipeline parallelism
Good for handling batches when operations can be broken down into fine-grained stages
SIMD (Single-Instruction-Multiple-Data)
All processors do the same things or idle
Phase 1: data partitioning and distributed
Phase 2: data-parallel processing
Efficient for big, regular data-sets
Systolic Array
Combination of SIMD and Pipeline parallelism
2-d array of processors with memory at the boundary
Tighter coordination between processors
Achieve very high speeds by circulating data among processors before returning to memory
MIMD(Multi-Instruction-Multiple-Data)
Each processor (teller) operates independently
Need synchronization mechanism by message passing or mutual exclusion (locks)
Best suited for large-grained problems
Less than data-flow parallelism
28
Important Laws of || Computing.
29
30
31
32
33
34
35
36
Important Consequences
f=0 when no serial part S(n)=n perfect speedup.
f=1 when everything is serial S(n)=1 no parallel code.
fn
nnS
)1(1)(
37
Important Consequences
S(n) is increasing when n is increasing
S(n) is decreasing when f is increasing.
fn
nnS
)1(1)(
38
Important Consequences
no matter how many processors are being used the speedup cannot increase above
Examples: f = 5% S(n) < 20 f = 10% S(n) < 10 f = 20% S(n) < 5.
ffn
nnS
1
)1(1)(
39
40
41
42
Gustafson’s Law - More
43
Gustafson’s Speed-up
pnsT
TpnTs
TimeParallel
TimeSequentialnS
)(
snnsnsnS )1()1()( When s+p=1
Important Consequences:
1) S(n) is increasing when n is increasing
2) S(n) is decreasing when n is increasing
3) There is no upper bound for the speedup.
44
To read:
1. John L. Gustafson, Re-evaluating Amdahl's Law,
http://www.scl.ameslab.gov/Publications/Gus/AmdahlsLaw/Amdahls.html
2. Yuan Shi, Re-evaluating Amdahl's and Gustafson’s Laws,
http://www.cis.temple.edu/~shi/docs/amdahl/amdahl.html
3. Wilkinson’s book,
1. sections of the laws of parallel computing
2. sections about types of parallel machines and compuation