!"#$%&'&()(*"+,-)./("+01,(2"3-)"4.*5-"!1,)-6-20"
Online archive
Staging archive
ICRAR, Perth
Staging & processing
Fornax, Perth
Long-term archive
PBStore, Perth
MIT, USA
Mirrored archive
AARNet
UI
Research
VO
Web
Scripting
1/10Gbps
10Gbps
1 Gbps
10Gbps
10Gbps
~ 400 MB/s
Catalog VO-Table
VO-Table
Data distribution, placement, scheduling
Online processing
MRO 320Gb/s
**Volume Title**ASP Conference Series, Vol. **Volume Number****Author**c�**Copyright Year** Astronomical Society of the Pacific
The MWA Archive - A Multi-tiered Dataflow and Storage System
Chen Wu1, Andreas Wicenec1, Dave Pallot2, and Alessio Checcucci1
1ICRAR/University of Western Australia2ICRAR/Curtin University
Abstract. The Murchison Widefield Array (MWA) is a next-generation radio tele-scope, generating visibility data products continuously at about 400 MB/s. The entireMWA archive consists of dataflow and storage sub-systems distributed across threetiers.
1. MWA Data Products
The Murchison Widefield Array (MWA) is optimized for the observation of extremelywide fields at frequencies between 80-300 MHz. The complete array will provideabout 2000 m2 of collecting area at 150 MHz with 30.72 MHz instantaneous band-width at 40 kHz spectral resolution and a field of view between 200 and 2500 squaredegrees with a time resolution of 0.5 seconds. The initial MWA commissioning willinstall 32 antennae referred to as “tiles”. Each tile is a 4 by 4 grid of dual-linear-polarization, wide-beamwidth antenna elements. Each data sample is a complex num-ber representing the amplitude and phase of a signal at a particular point in time withina specific frequency channel. For each channel at each time step, the correlator carriesout (N × (N + 1)/2) × (2 × 2) = 2N(N + 1) pair-wise cross/auto-correlation, where Ndenotes the number of tiles, and 2 × 2 indicates that polarised samples of orthogonaldirections are cross-correlated. For example, under the N = 32-tile configuration dur-ing commissioning, each correlator produces 2, 112 visibilities per frequency channelper time step. Since a “visibility” is a complex number (2 × 32 bits = 8 Bytes), theaggregated data rate amounts to 2N(N + 1)× 8× 1/T ×M per second, where T denotesthe time resolution (i.e. 0.5 seconds). M is the total number of frequency channels:30.72 MHz bandwidth / 40 kHz spectral resolution = 768. Hence correlators under thefull 128-tile configuration will produce visibility data at 387 MB/s. This requires 24on-site GPU-based software correlators, each processing 32 channels, and producingfrequency-split visibilities at a rate of about 16 MB/s.
During the 32-Tile phase of the MWA commissioning, two GPU machines withsoftware correlators have been deployed at MRO. For the purpose of commissioning,the spectral resolution is set to 10kHz with a 1-second time resolution. Therefore, eachone of the two correlators shares 30.72 MHz / 10kHz / 2 = 1,536 frequency channels,generating visibilities at the rate of 24.75 MB/s. Visibilities are currently stored, alongwith several observation-specific metdata, as FITS image or binary table extensions. Inaddition to visibilities, it is envisaged that images produced by the real-time imagingpipeline (Ord et al. (2010)) running on those GPU boxes will also be archived online.
1
NG/AMS Servers
NG/AMS Servers
Data Capture (Online Processing – Tier 0)
DataCaptureMgr
DataHandler DataHandler
In-memory Buffer Staging Area
NGAS Client
Local NGAS Servers
Data Producer
Remote NGAS Servers
Wide Area Network
Data Network
• Plugin-based • Multi-threaded • Throughput-oriented • Fault-tolerant • Admission control • File handling
Archive DB
A Single Process
M&C Interface
Metadata Network
M&C Subsystem
HDD Ramdisk SSD
Web / Query / VO Interface
-200
0
200
400
600
800
1000
1200
1400
16 24 32 40 48 56 64 72
Tota
l arc
hive
thro
ughp
ut -
MB/
s
Simulated data rate per client - MB/s
Archiving simulation throughput
A - 1Gbps bandwidth for commissioningB - 12 clients / 4 servers on 6 / 2 Fornax nodes
C - aggregated data producing rate for 12 clientsD - 12 clients / 4 servers on 6 iDataplex / 1 SupermicroE - 12 clients / 2 servers on 6 iDataplex / 1 Supermicro
F - 24 clients / 4 servers on 6 / 1 Fornax nodesG - 24 clients / 4 servers on 24 / 2 Fornax nodesH - 24 clients / 4 servers on 12 / 2 Fornax nodes
I - 24 clients / 4 servers on 28 Fornax nodesJ - aggregated data producing rate for 24 clients
!"#$%&
'"()&'%*+,-&
.+/(0&.1"1%&2,(3%&
4",0&2(56&2,(3%&
7"8%&2,(3%&
!"#$%&!
!"#"$%&&'(($)"*'+$! ,('+$%&&'(($-".'+/!
!"#"$0&1'234'+!
0#5+"6'$78'+"+&1*!
'&(&$)&!
*+,-./&!
01/&!
21"3!
'&%+.&/&!
!"
#$$#
!%"&
&'&()%)!)")*)&)))+),)')(+%+!+"+*
!"
#$$#
!%"&
&'&()%)!)")*)&)))+),)')(+%+!+"+*
!"
#$$#
!%"&
&'&()%)!)")*)&)))+),)')(+%+!+"+*
!"
#$$#
!%"&
&'&()%)!)")*)&)))+),)')(+%+!+"+*
!"
#$$#
!%"&
&'&()%)!)")*)&)))+),)')(+%+!+"+*
!"
#$$#
!%"&
&'&()%)!)")*)&)))+),)')(+%+!+"+*
!"
#$$#
!%"&
&'&()%)!)")*)&)))+),)')(+%+!+"+*
!"
#$$#
!%"&
&'&()%)!)")*)&)))+),)')(+%+!+"+*
*4$(35&! !"#"$-4"&'9'/#$:/68/'!
662177$4#!
!"
#$$#
!%"&
&'&()%)!)")*)&)))+),)')(+%+!+"+*
!"
#$$#
!%"&
&'&()%)!)")*)&)))+),)')(+%+!+"+*
;'('"+&1$<5&3(!
66'&89&)%!
Calibration
Imaging
Quality Control
Source Extraction
Calibration
Imaging
Quality Control
Source Extraction
Offline (re)processing on Fornax (Tier 1) - DIRP
! !
!"#$%&'()*+,-.(/#012,-0,3#-(45-#52-6
!""#!""$ !"%& !"%'
()*+,$ ()*+,#
!"%-!"%. !"%% !$""
7"82$(9":-+
;".<3,-(9":-+
=-5->"<.-$,(9":-+
!/01#
!/01$ !)11$ !)11#
73+,#-(9":-+
!)112 !)113
!)11& !)11' !)11. !)11-
(045!)6,47879, (+:9,19 /),+;)6
?@(9":-+
!)6,478<,
A-%:(9":-
B$C2$2D%$: EF(G,1-#$-,
+)$ +)#
;"<*(9":-+
EHFG,1-#$-,G&,-#$%>
!""#$%%&'()*+,-')."'(/.012/'(3%3415
7TB per node
Calibration
Imaging
Quality Control
Source Extraction
Data-Aware DIRP Job Scheduler
Long-Term Archive
Staging Archive
78("#$)9851-6":5*(;(%*"!)).0"<#:!="51"."6(>,'/(6().&-6").*5-",(%(19-?(@"/(6().&6/").A"*.,."9-6&6$-$1%0".,"B"CDE1F"78("(6&)("#:!".)985G("9-6151,1"-3"*.,.H-A".6*"1,-)./("1$I'101,(21"*51,)5I$,(*".9)-11",8)(("&()1F"""!,"75()"J@"<#4K"'"#$)9851-6"4.*5-"KI1()G.,-)0@":(1,()6"!$1,).%5.=@"1,.&-6"I(.2"*.,."51"?)-9(11(*"-6%56(@"?)-*$956/"LG515I5%5,0M"1,)(.21".,".6".//)(/.,(*").,("-3"NOJJ"#DE1F"78("G515I5%5,0"*.,."51"561,.6,%0"9.?,$)(*"56",8("-6%56(".)985G("3.95%5,0F""!,"75()"P"<Q(),8@"RJJ"S2"1-$,8"-3"#4K=@",8("1,./56/".)985G("56/(1,1"G515I5%5,0"1?%5,1"3)-2"75()"J"<,8)-$/8";I)('-?&9":!T",).613()=".6*"8.6*%(1",8(2"-G()",-"I-,8"%-6/',()2".)985G56/".6*"-U56("?)-9(1156/F"!"85().)9859.%"1,-)./("2.6./(2(6,"<V+#="101,(2"51"*(?%-0(*@"A8598"%(G()./(1"-$)")(1(.)98"56"*.,.".99(11"?.W()6".6.%0151",-"25/).,("*.,.".9)-11",8("1,-)./("85(.)980F"!"*.,.'56,(615G("CQX'9%$1,()"<Y-)6.>="51"$1(*"3-)"-U56("*.,."?)-9(1156/@"A8598".%1-"198(*$%(1"*.,."2-G(2(6,F""!,"75()"Z"<#[7@"X+!=@",8("25))-)(*".)985G("3.95%5,0"1$I19)5I(1",-"1?(95;9"*.,."?)-*$9,1"A5,8"75()"P@".6*"9-6&6$-$1%0"56/(1,1"$?*.,(*"*.,."1,(.21"-3")(%(G.6,"*.,.",0?(1"-6".")(/$%.)"I.151F"""
{chen.wu, andreas.wicenec}@uwa.edu.au
!"#"$%#&'(&)"#*)&+$