computing for ska (c4ska) colloquium 2015 · mahmoud mahmoud phd student institute for radio...
TRANSCRIPT
![Page 1: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/1.jpg)
Mahmoud Mahmoud
PhD Student
Institute for Radio Astronomy & Space Research (IRASR)
AUT University.
Computing for SKA (C4SKA) Colloquium 2015
![Page 2: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/2.jpg)
Middleware
Is software that facilitates combination of autonomous operating environments into a unified operating environment.
Communication management: Hides network protocols.
Common interface for communication.
Data type marshaling.
Resource management: Resource monitoring.
Task scheduling.
Load balancing.
Distributed application development: Programming language.
Integrated development environment (IDE).
Debugging and profiling tools.
![Page 3: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/3.jpg)
Top-level View for
Middleware Placement
![Page 4: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/4.jpg)
Top-level View for
Middleware Placement
![Page 5: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/5.jpg)
Top-level View for
Middleware Placement
![Page 6: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/6.jpg)
Top-level View for
Middleware Placement
![Page 7: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/7.jpg)
Top-level View for
Middleware Placement
![Page 8: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/8.jpg)
Top-level View for
Middleware Placement
![Page 9: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/9.jpg)
Top-level View for
Middleware Placement
![Page 10: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/10.jpg)
Top-level View for
Middleware Placement
![Page 11: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/11.jpg)
Top-level View for
Middleware Placement
![Page 12: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/12.jpg)
Top-level View for
Middleware Placement
![Page 13: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/13.jpg)
Top-level View for
Middleware Placement
![Page 14: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/14.jpg)
Examples of some HPC Middleware
Usage in Radio Astronomy
DiFX – MPI based parallel
software correlator.
IBM InfoSphere Streams –
correlation, RFI mitigation, and
imaging.
ARTEMIS Pelican/Panda – pulsar
search pipelines.
![Page 15: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/15.jpg)
Basic Parallel Processing
Scheduling Theory
A parallel application is defined by a set
of tasks and communications.
Tasks may be partially ordered and
represented as by a task graph.
The task graph is referred to as a job.
![Page 16: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/16.jpg)
Importance of Scheduling –
an example
![Page 17: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/17.jpg)
Importance of Scheduling –
an example
![Page 18: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/18.jpg)
Importance of Scheduling –
an example
![Page 19: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/19.jpg)
Importance of Scheduling –
an example
Shorter
![Page 20: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/20.jpg)
Classical Scheduling Theory
Optimality Criteria
Schedule length
Maximum lateness
Mean tardiness
Number of late tasks
Weighted number of late tasks
Mean flow time
Mean weighted
![Page 21: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/21.jpg)
SKA “Big Data” Problem
Implications on Scheduling
Big data volumes impractical to store for
later processing.
To avoid mass storage data must remain
in motion.
Data in motion is data streaming and
implies pipeline (stream) based
processing architecture.
Power efficiency criteria
![Page 22: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/22.jpg)
Stream Processing Paradigm
![Page 23: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/23.jpg)
Stream Processing Paradigm
![Page 24: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/24.jpg)
Minimum Power Dissipation
Based Scheduling
Dynamic Programming
Minimal Power Dissipation Path
Make simplifications to reduce number
of combinations
Initial conditions
Best effort no gaurantee
Multiple job scheduling
Adapting to change
![Page 25: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/25.jpg)
Metrics
![Page 26: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/26.jpg)
Example
![Page 27: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/27.jpg)
Example
![Page 28: Computing for SKA (C4SKA) Colloquium 2015 · Mahmoud Mahmoud PhD Student Institute for Radio Astronomy & Space Research (IRASR) AUT University. Computing for SKA (C4SKA) Colloquium](https://reader033.vdocuments.net/reader033/viewer/2022042313/5edbfb59ad6a402d666672e4/html5/thumbnails/28.jpg)
References
Drozdowski, M. (2009). Scheduling for parallel processing. London: Springer.
Sinnen, O. (2007). Task scheduling for parallel systems (Vol. 60). John Wiley & Sons.
Dijkstra, E. W. (1959). A note on two problems in connexion with graphs. Numerische mathematik, 1(1), 269-271.
Andrade, H., Gedik, B., & Turaga, D. (2014). Fundamentals of Stream Processing: Application Design, Systems, and Analytics. Cambridge University Press.