Slide 1

Module-2 : Parallel Programming1Dr.A.SrinivasPES Institute of TechnologyBangalore, Indiaa.srinivas@pes.eduMajor Forms of Decomposition

2Concurrency (TBC)How different tasks can be concurrently carried out.3Concurrency: Very much in Real WorldSteps in cooking a pasta meal

2009 Mathew J. Sottile, Timothy G. Mattson, and Craig E Rasmussen4Flynns Taxanomy

5Single Instruction Single Data (SISD) : - Only one Data Stream is processed by the CPU during a given clock cycle.Multiple Instruction Single Data (MISD)- Processing of Single Data with Multiple InstructionsSingle Instruction Multiple Data (SIMD) :Multiple Data Elements are processed in a single clock cycle(Intel MMX, SSE, SSE2, SSE3Multiple Instruction Single Data (MIMD) : Can execute multiple instruction streams, while working on a separate and independent Data Stream

6Granularity of ConcurrencyIntra-instruction (Pipelining)Parallel instructions (SIMD, VLIW)Tight-coupled MPLoosely-coupled MP7

8MetricsSpeed-up:

Efficiency:

Utilization:

Redundancy: 9Threads10

11

12Flow of ThreadsJava ThreadsA thread is not an objectA thread is a flow of controlA thread is a series of executed statementsA thread is a nested sequence of method calls13Illustration of a ThreadThread : Basic Unit of CPU Utilization- thread contains a PC pointing to the current instructions. - thread contains :- CPU state information for the current thread, other resources such as stack etc.I14Relationships among Processors, Processes and Threads

15Mapping Models of Threads to Processes

16

17The Thread ObjectA thread is not an objectA Thread is an object

void start()Creates a new thread and makes it runnable

void run()The new thread begins its life inside this method18Thread Creation Diagram19Thread t = new BThread();

t.start();

doMoreStuff();Object ABThread() {}

void start() {// create thread}

void run() {doSomething();}Object BThread (extends Thread) Thread Creation Diagram20Thread t = new BThread();

t.start();

doMoreStuff();Object ABThread() {}

void start() {// create thread}

void run() {doSomething();}Object BThread (extends Thread) Thread Creation Diagram21Thread t = new BThread();

t.start();

doMoreStuff();Object ABThread() {}

void start() {// create thread}

void run() {doSomething();}Object BThread (extends Thread) Thread Creation Diagram22Thread t = new BThread();

t.start();

doMoreStuff();Object ABThread() {}

void start() {// create thread}

void run() {doSomething();}Object BThread (extends Thread) Thread Creation Diagram23Thread t = new BThread();

t.start();

doMoreStuff();Object ABThread() {}

void start() {// create thread}

void run() {doSomething();}Object BThread (extends Thread) Thread Creation Diagram24Thread t = new BThread();

t.start();

doMoreStuff();Object ABThread() {}

void start() {// create thread}

void run() {doSomething();}Object BThread (extends Thread) Thread Creation Diagram25Thread t = new BThread();

t.start();

doMoreStuff();Object ABThread() {}

void start() {// create thread}

void run() {doSomething();}Object BThread (extends Thread) Thread Creation Diagram26Thread t = new BThread();

t.start();

doMoreStuff();Object ABThread() {}

void start() {// create thread}

void run() {doSomething();}Object BThread (extends Thread) Thread Creation Diagram27Thread t = new BThread();

t.start();

doMoreStuff();Object ABThread() {}

void start() {// create thread}

void run() {doSomething();}Object BThread (extends Thread) Runnable InterfaceA helper to the thread objectThe Thread objects run() method calls the Runnable objects run() methodAllows threads to run inside any object, regardless of inheritance28Runnable ExampleTalker talker = new Talker();Thread t = new Thread(talker);t.Start();---class Talker implements Runnable { public void run() { while (true) { System.out.println(yakitty yak); } }}29Tread Lifecycle

30Thread Lifecycle31BornBlockedRunnableDeadstop()start()stop()Activeblock on I/OI/O availableJVMsleep(500)wake upsuspend()resume()waitnotifyBlocking ThreadsWhen reading from a stream, if input is not available, the thread will blockThread is suspended (blocked) until I/O is availableAllows other threads to automatically activateWhen I/O available, thread wakes back up againBecomes runnableNot to be confused with the Runnable interface32Thread SchedulingIn general, the runnable thread with the highest priority is active (running)Java is priority-preemptiveIf a high-priority thread wakes up, and a low-priority thread is runningThen the high-priority thread gets to run immediatelyAllows on-demand processingEfficient use of CPU33Thread StarvationIf a high priority thread never blocksThen all other threads will starveMust be clever about thread priority34Thread Priorities: General StrategiesThreads that have more to do should get lower priorityCounterintuitiveCut to head of line for short tasksGive your I/O-bound threads high priorityWake up, immediately process data, go back to waiting for I/O35Race ConditionsTwo threads are simultaneously modifying a single objectBoth threads race to store their valueIn the end, the last one there wins the race(Actually, both lose)36Race Condition Exampleclass Account { int balance; public void deposit(int val) { int newBal; newBal = balance + val; balance = newBal; }}37Synchronization

38Thread SynchronizationLanguage keyword: synchronizedTakes out a monitor lock on an objectExclusive lock for that threadIf lock is currently unavailable, thread will block39Thread SynchronizationProtects access to code, not to dataMake data members privateSynchronize accessor methodsPuts a force field around the locked object so no other threads can enterActually, it only blocks access to other synchronizing threads40Shared Data Synchronization

41

42Recursive Lock

43Thread DeadlockIf two threads are competing for more than one lockExample: bank transfer between two accountsThread A has a lock on account 1 and wants to lock account 2Thread B has a lock on account 2 and wants to lock account 144Avoiding DeadlockNo universal solutionOrdered lock acquisitionEncapsulation (forcing directionality)Spawn new threadsCheck and back offTimeoutMinimize or remove synchronization 45Wait and NotifyAllows two threads to cooperateBased on a single shared lock object46Wait and Notify: CodeConsumer:synchronized (lock) { while (!resourceAvailable()) { lock.wait(); } consumeResource();}47Wait and Notify: CodeProducer:produceResource();synchronized (lock) { lock.notifyAll();}48Wait/Notify Sequence49Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }Wait/Notify Sequence50Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }Wait/Notify Sequence51Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }Wait/Notify Sequence52Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }Wait/Notify Sequence53Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }Wait/Notify Sequence54Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }Wait/Notify Sequence55Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }Wait/Notify Sequence56Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }Wait/Notify Sequence57Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }Wait/Notify Sequence58Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }Wait/Notify Sequence59Lock ObjectConsumerThreadProducerThread1. synchronized(lock){2. lock.wait();3. produceResource()4. synchronized(lock) {5. lock.notify();6.}7. Reacquire lock8. Return from wait()9. consumeResource();10. }59Wait/Notify: Details60Often the lock object is the resource itselfSometimes the lock object is the producer thread itselfWait/Notify: Details61Must loop on wait(), in case another thread grabs the resource...After you are notifiedBefore you acquire the lock and return from wait()Use lock.notifyAll() if there may be more than one waiting threadWait/Notify Example: Blocking Queue62class BlockingQueue extends Queue { public synchronized Object remove() { while (isEmpty()) { wait(); // really this.wait() } return super.remove(); } public synchronized void add(Object o) { super.add(o); notifyAll(); // this.notifyAll() }}Computer System Architectures 4 machine organizationsSISD - Single Instruction, Single DataSIMD - Single Instruction, Multiple DataMISD - Multiple Instruction, Single DataMIMD - Multiple Instruction, Multiple Data

63Flynn Architectures (1)64Array ProcessorSerial ProcessorCUPUIPU1PUnDID1DnCUCU control unitPU processor unitI instruction streamD data streamSISDSIMDOne data processed by CPU in a Clock CycleMultiple Data Elements processed in one clock cycleFlynn Architectures (2)65CU1PU1I1D1CUnPUnInDnMultiprocessorandMulticomputerMIMDCU1PU1I1DCUnPUnInMISDNo real examples- possibly some pipeline architecturesProcessing of multiple data by a single instructionCan execute multiple instruction streams, while working on an independent datastreamW

ith SIMD & MIMD machines of today, Software Developers can exploit Data Level & Task Level Parallelism. 66

67