![Page 1: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/1.jpg)
Computer Architecture
Parallel ProcessingOla Flygt
Växjö Universityhttp://w3.msi.vxu.se/users/ofl/
[email protected]+46 470 70 86 49
![Page 2: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/2.jpg)
Outline
Basic conceptsTypes and levels of parallelismClassification of parallel
architectureBasic parallel techniquesRelationships between languages
and parallel architecture
CH03
![Page 3: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/3.jpg)
Basic concepts
The concept of programordered set of instructions
(programmer’s view)executable file (operating system’s
view)
![Page 4: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/4.jpg)
The concept of process
OS view, process relates to executionProcess creation
setting up the process descriptionallocating an address spaceloading the program into the allocated address
space, andpassing the process description to the scheduler
process statesready to runrunningwait
![Page 5: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/5.jpg)
Process spawning (independent processes)
A
CB
D E
![Page 6: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/6.jpg)
The concept of thread
smaller chunks of code (lightweight)threads are created within and
belong to processfor parallel thread processing,
scheduling is performed on a per-thread basis
finer-grain, less overhead on switching from thread to thread
![Page 7: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/7.jpg)
Single-thread process or multi-thread (dependent)Thread tree
Process
Threads
![Page 8: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/8.jpg)
Three basic methods for creating and terminating
threads1. unsynchronized creation and
unsynchronized terminationcalling library functions: CREATE_THREAD,
START_THREAD
2. unsynchronized creation and synchronized terminationFORK and JOIN
3. synchronized creation and synchronized terminationCOBEGIN and COEND
![Page 9: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/9.jpg)
Processes and threads in languages
Black box view: T: thread
T2 T0 T1 T1 T2 T0 . . . Tn
FORK
JOIN
FORK
JOIN
COBEGIN
COEND
. . .
![Page 10: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/10.jpg)
The concepts of concurrent execution (N-
client 1-server)
Non pre-emptive Pre-emptive
Time-shared Prioritized
PrioritySever SeverSever
Client ClientClient
![Page 11: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/11.jpg)
Parallel execution
N-client N-server modelSynchronous or Asynchronous
SeverClient
![Page 12: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/12.jpg)
Concurrent and Parallel Programming Languages
Classification of programming languages
![Page 13: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/13.jpg)
Types and levels of parallelism
Available and utilized parallelismavailable: in program or in the problem
solutionsutilized: during execution
Types of available parallelismfunctional
arises from the logic of a problem solution
dataarises from data structures
![Page 14: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/14.jpg)
Available levels Utilized levels
User (program) level User level
Procedure level Process level
Loop level Thread level
Instruction level Instruction level
2
1
1.Exploited by architectures2.Exploited by means of operating systems
Available and utilized levels of functional
parallelism
![Page 15: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/15.jpg)
Utilization of functional parallelism
Available parallelism can be utilized by architecture,
instruction-level parallel architectures compilers
parallel optimizing compileroperating system
multitasking
![Page 16: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/16.jpg)
Concurrent execution models
User level --- Multiprogramming, time sharing
Process level --- MultitaskingThread level --- Multi-threading
![Page 17: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/17.jpg)
Utilization of data parallelism
By using data-parallel architecture
Convert into functional parallelism (i.e. loop constructs)
![Page 18: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/18.jpg)
Classification of parallel architectures
Flynn’s classificationSISDSIMDMISD (Multiple Instruction Single
Date)MIMD
![Page 19: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/19.jpg)
Part III Part II Part IV
Data-parallel architectures Function-parallel architectures
Instruction-level
PAs
Thread-level
PAs
Process-levelPAs
ILPS MIMDs
Vectorarchitecture
Associative
architecturearchitectureand neural
SIMDs Systolic Pipelinedprocessors processorsVLIWs Superscalar Distributed
memory
(multi-computer)
Sharedmemory(multi-MIMD
DPs
Parallel architectures
processors)
![Page 20: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/20.jpg)
Basic parallel technique
Pipelining (time)a number of functional units are employed in
sequence to perform a single computationa number of steps for each computation
Replication (space)a number of functional units perform multiply computation simultaneously
more processorsmore memorymore I/O
more computers
![Page 21: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/21.jpg)
Relation between basic techniques and
architectures
![Page 22: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/22.jpg)
Relationships between languages and parallel
architecture SPMD (Single Procedure Multiple data)
Loop: split into N threads that works on different invocations of the same loop
threads can execute the same code at different speeds synchronize the parallel threads at the end of the loop
barrier synchronization use MIMD
Data-parallel languages DAP Fortran
C = A + B (A, B and C are arrays) use SIMD
Other types like Vector machines or Systolic arrays does not require any specific language support.
![Page 23: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/23.jpg)
Synchronization mechanismsTest_and_set
Semaphore
ConditionalCritical region
Monitor
Rendezvous
Send/receivemessage
Broadcast Shiftnet_receive
(processor form)
Remote procedure calls
Net_send
![Page 24: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/24.jpg)
Using Semaphores to handle mutual execution
P1
Criticalregion
V(S)
S Semaphore
Shareddata
structureV(S)
P(S)
P2
Criticalregion
P(S)
BusyWait
![Page 25: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/25.jpg)
Parallel distributed computing
Ada used rendezvous concepts which combines
feature of RPC and monitorsPVM (Parallel Virtual Machine)
to support workstation clustersMPI (Message-Passing Interface)
programming interface for parallel computers
![Page 26: Computer Architecture Parallel Processing Ola Flygt Växjö University Ola.Flygt@msi.vxu.se +46 470 70 86 49](https://reader036.vdocuments.net/reader036/viewer/2022062716/56649e045503460f94aefd85/html5/thumbnails/26.jpg)
Summary of forms of
parallelism