SKA SDP :A snapshot of recent technical directions

and conclusions

Chris BroekemaASTRON

Netherlands Institute for Radio Astronomy

Highlights

A whirlwind overview of recent prototyping and design work done within the SDP consortium.

Note: these are invariably proofs-of-concept. While concepts are taking shape, unless otherwise stated, no down-selects have been made.

Algorithm Reference Library

https://github.com/SKA-ScienceDataProcessor/algorithm-reference-library

Illustrative flow chart

4

DRAFT

Describing the data relationships between components

• All major calibration and imaging algorithms expressed in Python unencumbered by practicalities of I/O, memory management, optimisation

• Establlishes reference e.g. for later implementation by non-experts

• Prototypes of functional components (referentially transparent)

• Synthesis components: Fourier transforms (predict, invert) using 2D, wprojection, faceting, wslicing, snapshots

• Model for LOW sky(S3) and LOW station beam (OSKAR)

• Testing is memory-limited: largest image made so far is 25K by 25K pixels

• Largely complete

• 97% coverage test suite

Algorithm Reference Library (ARL)A Python prototype of the algorithms

5

Algorithm Reference LibrarySynthesis framework allows experimenting with components

6

Dask graph for ICAL11 ingests. 5 major cycles.

7

Execution framework: DaLiuge

https://arxiv.org/abs/1702.07617

Astronomer

Component Parametersdefault parameter values of components

Pipeline Logic reduction components and sequence

Data Parallelisationhints about the potential of parallelism

Algorithms what’s the best algorithm to get the desired

answer

Parallel Executionwhat is executed where

Parallel Codingwriting parallel code

Code Optimisationoptimise parallel code

I/O Optimisationoptimise I/O on hardware

OS and hardware co-designoptimise hardware for code to be run

HPC Software Engineer

OS level S/W Engineer

Computer H/W Engineer

Logical Graph Template

Component Parametersdefault parameter values of components

Pipeline Logic reduction components and sequence

Data Parallelisationhints about the potential of parallelism

Algorithms best algorithm to get the desired answer

Parallel Executionwhat is executed where

Parallel Codingwriting parallel code

Code Optimisationoptimise parallel code

I/O Optimisationoptimise I/O on hardware

OS and hardware co-designoptimise hardware for code to be run

Physical Graph Template

Physical Graph

Logical Graph

Astronomer

HPC SoftwareEngineer

OS level S/W Engineer

Computer H/W Engineer

Operator

SDP Consortium meeting, SKA Engineering Meeting, Rotterdam, Friday 16 th June 2017

SDP Integration Prototype (SIP)

Ben Mort, Fred Dulwich (University of Oxford) Brian McIlwrath, Chris Pearson, David Terrett, Nijin Thykkathu (Rutherford Appleton Laboratory)

Arjen Tamerus, Vlad Stolyarov, Verity Allan, John Taylor (University of Cambridge)

SDP System Integration Prototype (SIP) - activities

14

Current activities

• TANGO interfaces to TM• CSP interfaces (SPEAD, Pulsar search)• Master Controller interface to tasks and

execution frameworks• Prototyping a number of test capabilities

(pipelines)• LMC services such as LTS using distributed

Redis• Deployment of SIP code onto AlaSKA-P3

Next Steps

• Prototyping of the SDP buffer• Integration of LMC services (eg. LTS, Sky

model, QA) with execution frameworks

Exploring high-risk areas

Example outcome: resilience in running LMC services. By moving towards a micro-services like architecture using container orchestration the risk of single key service failure for our control interfaces is largely mitigated

Implemented

Aims of SIP

End-to-end SDP system prototype:• Prototype of all major external and internal

interfaces• Verification and testing of SDP (software)

architecture• Focused testing and analysis of

technology choices• Link / overlap with execution framework

work and vertical prototyping• Link to hardware prototyping

(AlaSKA-P3)

TM

CSP

SDP

LMC services

Capabilities / Execution framework(s)

LFAA?

Master Controller: Platform as a Service

• Service that starts, monitors and stops tasks– Manages CPU resources– Restarts failed tasks– Redundancy / failover– Service discovery

• Abstract interface– Multiple implementations for different sorts

of task• First implementation based on Docker swarm

mode• Mesos + Marathon & Apache Spark have now

also been implemented

SaaSSaaS

PaaSPaaS

IaaS (OpenStack)IaaS (OpenStack)

Middleware

Containers, logging & monitoring, visualisation

Containerisationa modern software paradigm

* http://diginomica.com/2014/07/02/virtualization-dead-long-live-containerization/

Container: A choice of engines

● Docker

● Rkt

● LXC/LXD

● Others: Singularity, runC

● Docker currently dominates – greatest set of tools, largest

community and adoption

Container Orchestration: choices

● Container build & run and Orchestration

● Container orchestration

● Batch processing - Old School!

● Bespoke – DALiuGE, Dask, AirFlow

We must schedule the pipelines

SDP: Logging Architecture

SDP: Visualisation & Integration

Software-defined networks

http://dl.acm.org/citation.cfm?doid=3075564.3075594https://www.astron.nl/~broekemahttps://gitlab.com/broekema/SDP_Controller

SDN capable switch

Host A

Host X

Host Y

./generator

./analyzer

./analyzer

Network controller

OpenFlowryu

SDP_controller

The following OpenFlow features are used in our proof-of-concept application:

– Set Field action destination MAC address

– Set Field action destination IP address

– Group Type=ALL: apply action instructions

– Group Type=ALL: output to multiple ports

All of the investigated switches worked to some degree. Many of the features we rely on are optional and are not guaranteed to be implemented by the switch vendor.

– The concept is viable

– We rely on optional features that are not always implemented

– performance and quality of implemented features varies wildly

– A software-defined network adds robustness over a conventional network

– Adding the network to the processing platform is a potentially disruptive new paradigm that offers great additional features

Numerical precision