Credit: Alex Cherney/terrastro.com
Ma#hew Whi*ng ASKAP Science Pipelines
CSIRO ASTRONOMY & SPACE SCIENCE
Processing so8ware suite for ASKAP Tailored to meet the peculiar requirements of ASKAP: very high data rates and quasi-‐real-‐Cme processing Designed to run on high-‐performance compuCng systems ASKAP processing approach described in detail in ASKAP-‐SW-‐0020 Covers all stages of processing: • Ingest of data from correlator • CalibraCon & Imaging • Source extracCon & cataloguing • Archiving
New version of SW-‐0020 due out soon!
The ASKAPso? package
ASKAP Science Processing
ASKAP-SW-0020
Version: 2.0Date: 20/12/2011Project: ASKAP
Prepared by: Tim Cornwell, Ben Humphreys, Emil Lenc, Maxim Voronkov, MatthewWhiting
Reviewed by: Ilana Feain,Review reference : Redmine issue 3280Approved by: Ilana Feain Date: 20/12/2011
Keywords: computing, science, processing
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
Designated as the ASKAP Real-‐Cme Computer Compute power: • 472 Cray XC30 compute nodes • Each 2x10 core, with 64GB RAM • 200 Tflop/s peak performance
Interconnect: • Cray Aries (dragonfly topology)
Storage: • 1.4 PB Lustre file system • Peak I/O performance 30GB/s
GPU nodes: • 64 nodes with Kepler GPU • Allocated to MWA (not ASKAP)
ASKAP Central Processor @ Pawsey Centre “Galaxy”
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
Prototype pipelines have been developed for BETA processing
Focus on matching processing to BETA observaConal model, while handling key types of science processing
Split by beam, image with w-‐project, self-‐calibraCon, mosaic a8er imaging
Bandpass calibraCon per beam
These pipelines provide a way to validate the ASKAPso8 tasks, and to prototype the pipeline processing to be used for Early Science
Processing for BETA data
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
Summary of current ASKAPso? capabili*es Program Purpose
cimager Imager for conCnuum datasets
simager Imager for spectral-‐line datasets – massively-‐distributed computaCon
ccalibrator Gains and leakage calibraCon
cbpcalibrator Bandpass calibraCon
cflag Flagging
ccalapply Apply calibraCon soluCons to measurement sets
ccontsubtract ConCnuum subtracCon
linmos Linear mosaicking
cmodel Model image generator
mssplit Take a subset of a measurement set
selavy Source finding (conCnuum, spectral-‐line, RM synthesis soon)
askappipelines module Pipeline scripts to Ce the above together
casdaupload CASDA archiving of key data products
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
Example: Tucana wide field con*nuum
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
E
N
Credit: Wasim Raja
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
SUMSS (Molonglo telescope, 1998)
ASKAPso8 (Wasim Raja, BETA)
CASA (Josh Marvil, BETA)
36-‐beam ADE observa*on of Apus
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
Credit: Wasim Raja
Early Science pipelines
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
CASS Science Team
Science Team
AddiConal features in development include: • Improved imaging framework to beOer uClise compute resources • Doppler correcCon – single correcCon per beam • Direct FITS output of images, with parallel write • RM Synthesis for extracted spectra of Stokes-‐I components • InvesCgaCng support for UV gridding experiments (DINGO)
We welcome tesCng & input from the community
Our Community Busy Weeks have provided training in and pracCcal experience with ASKAPso8
More will be held over coming months
The road to Early Science
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
Early science observaCons present limited modes for processing Features of ASKAP-‐12 array: • Shorter baselines – 2.18 km maximum baseline (c.f. 6.44 km for ASKAP-‐36) • Smaller data sizes – approx. 1 TB/hr (c.f 8.5 TB/hr for ASKAP-‐36)
Moving to full ASKAP requires scaling-‐up capacity in the ingest and imaging pipelines, to account for: • Increased data rates à increased I/O bandwidth • Increased UV grid & image sizes à increased memory/CPU requirements • Allow real-‐Cme processing à increased efficiency
From Early Science to full-‐scale ASKAP
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
Calibration Pipeline Services
Small-N (e.g. Continuum) Imager Pipeline
Large-N (eg. Spectral Line) Imager Pipeline
Ingest Pipeline
UV Data
16200 Channels(18.5kHz)
UV Data
300 Channels(1MHz)
Imager
Imager
Source Finder/Identifier
Source Finder/Identifier
Source Catalog
Source Catalog
Calibrator
Transient Detector Pipeline
TransientImager
Images
Transient Finder/Identifier
Transient Detections
16200 Channels(18.5kHz)
Calibration Solution
~30 Channels(10MHz)
Calibration Data
Service
Sky Model Service
Light Curve Service
Image Cube
Images
Pipelines for Full ASKAP Opera*ons
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
Real-‐Cme services: • CalibraCon + RFI flagging to be applied at ingest • Sky model subtracted off visibiliCes & kept up-‐to-‐date
Full spaCal resoluCon imaging (10” PSF) over full field (>10k pix) • ASKAPso8 capable of high resoluCon but large memory + compute impact • ProhibiCve at full spectral resoluCon – limit this to 30” PSF • Aim to produce small regions around specified points at 10” PSF
Transient pipeline • Imaging at 5-‐10sec cadence • No deconvoluCon, snapshot imaging without reprojecCon • Light-‐curve service
Zoom-‐mode capabiliCes • Improvements to Doppler correcCon? Correct for differenCal Doppler effect
within a field Improvements to source-‐finding & quality analysis following Early Science
New features/improvements for full ASKAP
ASKAP Science Pipelines | MaOhew WhiCng | ASKAP2016, June 6-‐10 2016
Thank you CSIRO Astronomy & Space Science MaOhew WhiCng ASKAP Science OperaCons t +61 2 9372 4683 E [email protected] w www.atnf.csiro.au/projects/askap
CSIRO ASTRONOMY & SPACE SCIENCE
We acknowledge the Wajarri Yamatji people as the tradi6onal owners of the Observatory site.