master course in parallel computing from pen-and …...master course in parallel computing from...
TRANSCRIPT
![Page 1: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/1.jpg)
Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster
Novica NosovićUniversity of Sarajevo
Faculty of Electrical EngineeringDpt. of Computing and Informatics
e-mail: [email protected]
Workshop “Cooperation at Academic Informatics Education across Balkan Countries and Beyond: The Impact of Informatics to Society"Jelsa (CRO), 2nd - 6th September 2019
![Page 2: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/2.jpg)
Abstract
Serious course in parallel computer systems can hardly fit in 3 lecture hours per week, 14 weeks semester budget. Covering typical problems (what?) suitable for parallelization, solved with adequate algorithms (how?), coded with contemporary programming languages (tools?) and executed on accessible parallel machines (architectures?), requires more lecture hours and lab resources. Leading technical universities around the world do this in increased number of teaching hours and with extended access to variety of parallel hardware platforms. They also avoid "the textbook"-based teaching by offering extended list of conference papers in various parallel computers and programming technologies. Our approach is to maximize learning outcomes using (very) limited resources.
![Page 3: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/3.jpg)
On Difficulty of Parallel Software
If (sequential programming = difficult)
parallel programming = difficult2
![Page 4: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/4.jpg)
Course Goals (working in groups)
● Write parallel program● Achieve considerable speedup using “lessons learned”● Use “low hanging fruit”● Try to:
– Be relevant, coherent and consistent– Find “piece of gold”– Write conference paper – Publish paper
![Page 5: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/5.jpg)
Abstraction Layers
● Problem● Algorithm● Tools● Architecture
![Page 6: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/6.jpg)
Classroom Demo on Parallelization
● New concepts introduced: communication, synchronization, scheduling, load balancing, Amdhals law, message passing, task/data decomposition, map-reduce, hard/weak scaling, arithmetic intensity, high watermarking, sublinear/linear/superlinear speedup, multi/hyper-threading, shared/distributed memory, co-processors/accelerators, clustering,
![Page 7: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/7.jpg)
The Class
![Page 8: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/8.jpg)
Problem(s) in Measuring Speedup
![Page 9: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/9.jpg)
Parallella – credit card sized “supercomputer”
16 Core Epiphany Coprocessor
![Page 10: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/10.jpg)
ControlFlow vs. DataFlow
PCI Express x16 Card2 x Xilinx Virtex-5 SX240T24GB DDR2 memory
![Page 11: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/11.jpg)
Conventional Heterogeneous Multicore System Architecture
Global Interconnection Network
Node 1Node 2 Node 3 Node N
…Scratchpad memory
Core Array
Accelerator
Memory Banks
Multicore Sockets
NIC
![Page 12: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/12.jpg)
20 Dell PowerEdge SC1435`s with IB Switch
![Page 13: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/13.jpg)
NUMA Configuration with Broadcom Chipset
![Page 14: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/14.jpg)
Load balancing is hard
![Page 15: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/15.jpg)
My Expectations
![Page 16: Master Course in Parallel Computing From Pen-and …...Master Course in Parallel Computing From Pen-and-Paper to Heterogeneous NUMA Cluster Novica Nosović University of Sarajevo Faculty](https://reader034.vdocuments.net/reader034/viewer/2022042210/5eaed7f14037310648663509/html5/thumbnails/16.jpg)
Results achieved
● Predictable results can not be considered “piece of gold”