Computer ArchitectureParallel ProcessingOla FlygtVxj Universityhttp://w3.msi.vxu.se/users/ofl/Ola.Flygt@msi.vxu.se+46 470 70 86 49

OutlineBasic conceptsTypes and levels of parallelismClassification of parallel architectureBasic parallel techniquesRelationships between languages and parallel architectureCH03

Basic conceptsThe concept of programordered set of instructions (programmers view)executable file (operating systems view)

The concept of processOS view, process relates to executionProcess creationsetting up the process descriptionallocating an address spaceloading the program into the allocated address space, andpassing the process description to the schedulerprocess statesready to runrunningwait

Process spawning (independent processes)ACBDE

The concept of threadsmaller chunks of code (lightweight)threads are created within and belong to processfor parallel thread processing, scheduling is performed on a per-thread basisfiner-grain, less overhead on switching from thread to thread

Single-thread process or multi-thread (dependent)Thread treeProcessThreads

Three basic methods for creating and terminating threads1. unsynchronized creation and unsynchronized terminationcalling library functions: CREATE_THREAD, START_THREAD2. unsynchronized creation and synchronized terminationFORK and JOIN3. synchronized creation and synchronized terminationCOBEGIN and COEND

Processes and threads in languagesBlack box view: T: threadT2 T0 T1 T1 T2 T0 . . . TnFORKJOINFORKJOINCOBEGINCOEND. . .

The concepts of concurrent execution (N-client 1-server) Non pre-emptivePre-emptiveTime-sharedPrioritizedPrioritySeverSeverSeverClientClientClient

Parallel executionN-client N-server modelSynchronous or AsynchronousSeverClient

Concurrent and Parallel Programming LanguagesClassification of programming languages

Languages

1_client

1-server model

N_client

1-server mode

1_client

N-server model

N_client

N-server model

sequential

+

-

-

-

concurrent

-

+

-

-

Data-parallel

-

-

+

-

Parallel

-

+

-

+

Types and levels of parallelismAvailable and utilized parallelismavailable: in program or in the problem solutionsutilized: during executionTypes of available parallelismfunctionalarises from the logic of a problem solutiondataarises from data structures

Available levelsUtilized levelsUser (program) levelUser levelProcedure levelProcess levelLoop levelThread levelInstruction levelInstruction level211.Exploited by architectures2.Exploited by means of operating systemsAvailable and utilized levels of functional parallelism

Utilization of functional parallelismAvailable parallelism can be utilized by architecture,instruction-level parallel architectures compilersparallel optimizing compileroperating systemmultitasking

Concurrent execution modelsUser level --- Multiprogramming, time sharingProcess level --- MultitaskingThread level --- Multi-threading

Utilization of data parallelismBy using data-parallel architecture Convert into functional parallelism (i.e. loop constructs)

Classification of parallel architecturesFlynns classificationSISDSIMDMISD (Multiple Instruction Single Date)MIMD

Part IIIPart IIPart IVData-parallel architecturesFunction-parallel architecturesInstruction-levelPAsThread-levelPAsProcess-levelPAsILPSMIMDsVectorarchitectureAssociativearchitecturearchitectureand neuralSIMDsSystolicPipelinedprocessorsprocessorsVLIWsSuperscalarDistributedmemory(multi-computer)Sharedmemory(multi-MIMDDPsParallel architectures

processors)

Basic parallel techniquePipelining (time)a number of functional units are employed in sequence to perform a single computationa number of steps for each computationReplication (space)a number of functional units perform multiply computation simultaneouslymore processorsmore memorymore I/Omore computers

Relation between basic techniques and architectures

Relationships between languages and parallel architectureSPMD (Single Procedure Multiple data)Loop: split into N threads that works on different invocations of the same loopthreads can execute the same code at different speedssynchronize the parallel threads at the end of the loopbarrier synchronizationuse MIMD Data-parallel languagesDAP FortranC = A + B (A, B and C are arrays)use SIMDOther types like Vector machines or Systolic arrays does not require any specific language support.

Synchronization mechanismsTest_and_setSemaphoreConditionalCritical regionMonitorRendezvousSend/receivemessageBroadcastShiftnet_receive(processor form)Remote procedure callsNet_send

Using Semaphores to handle mutual execution P1CriticalregionV(S)SSemaphoreShareddatastructureV(S)P(S)P2CriticalregionP(S)Busy Wait

Parallel distributed computingAda used rendezvous concepts which combines feature of RPC and monitorsPVM (Parallel Virtual Machine)to support workstation clustersMPI (Message-Passing Interface)programming interface for parallel computers

Summary of forms of parallelism

*