swift archive

SwiftHUG 14 Feb 2003

1

Swift archive

Lorella Angelini/HEASARC


2

Swift Mission Characteristics

• Multi-wavelength observatory– Burst Alert Telescope (BAT): 10-150 keV

—detect ~ 150 GRB per year—onboard computation of positions—arc-minute positional accuracy

– Dedicated telescopes for X-rays, UV, and optical afterglow follow up:

—0.3-10 keV X-ray Telescope (XRT)—170-650 nm UV/Optical Telescope

(UVOT)—0.3-2.5 arc-second locations—existing hardware from JET-X and XMM—determine redshifts from X-ray

absorption, lines, and Lyman-a cutoff

Rapid response satellite

20 - 70 sec to slew within FOV of BAT

autonomous operations factor 100 improved response

time continue monitoring of fading

afterglow rapid dissemination of the data

CollaborationUSA/Italy/UK

Launch 5 Dec 2003


3

Latest on the Mission

• Spacecraft delivered to GSFC (August 2002)• XRT & UVOT completed & calibrated (XRT: Sep 02; UVOT: Nov 02)

— Integrated with the Spacecraft • BAT completed, under calibration and final delivery in May 2003• Approved of a SWIFT GO program :

– Cycle 1 AO via the ROSS-2003 NRA (Sep 1 2003; due Dec 1 2003)– Open only to scientists at US institutions (similar programs may be

possible in UK & Italy):– Cycle 1 begin ~4.5 months after launch for about 12 months – Proposals are for the following areas :

—New GRB projects not duplicative of team key projects & not requiring GI-specific observatory pointings

—Correlative observations of GRB’s with other telescopes —Theoretical studies to advance mission science return on GRBs


4

Swift rapid data dissemination plan

• Data are downloaded via :

– TDRSS. Series of messages after a trigger is detected on board.

— broadcast via the GCN (minutes) —later archived as part of the final Swift archive (< a week)

– Malindi. All the telemetry. About 6 -7 download per day

• Two processing types of the Malindi data (same pipeline

software) – Quick look (output data is not part of the final archive):

— Data processed every Malindi dump, e.g. the same observation is re-processed when new data arrive.

— Data public in 3-6 hours after each dump (from FTP SDC). – Permanent archive:

— Final processing after an observation (~ 1 day of data ) — Archived after all telemetry for observation is available ( < a

week)


5

Overview Swift Data Archive

• Archive includes all data levels, calibration, software and documents

• Data are in FITS format OGIP standards – GIF, JPEG and HTML used for products preview

or to record processing

• No proprietary period for data– But check-out (~45 days) & verification mission

phase (3 months) where data are available only to the Swift team

• Archival data at HEASARC, ISAC (Italy) & LEDAS (UK)• Data transfers are via DTS protocol (adapted from

XMM)


6

SDCGSFC

HEASARCarchive

Swift Stagingarea

MOCPSU

QL

World

Italy archiveISAC

UK archiveLEDAS

Swift Data Flow


7

HEASARC responsibilities

• Populate and maintain the Swift science and calibration data archive

• Provide the following services – Public access to the Swift archive– Science data analysis software environment – Web server for the Swift Science Support Center – Database system for the Swift tables– Long-term archive

• Generate GRB summary products• These responsibilities are documented in the Swift-

HEASARC MOU (16 Oct 2001)– Technical agreement via ICD in place with the SDC, ISAC, BAT

Team (completed and Signed) & MSSL (draft)


8

Archival Data • Production data populate the archive from the start of the

mission start & delivered in less than a week :– TDRSS messages (SDC) - by trigger number – Level 1,2,3 production data (SDC) - by observation – Monitoring & Calibration data (SDC/SSC) - by type & CALDB

• GRB summary data products (HEASARC) associated with the GRB catalog - by GRB– Populate the archive on routine basis 5 months after launch

• Surveys : XRT (ISAC), BAT (BAT team), UVOT (MSSL) - by time – Populate the archive not earlier than 1 yr into the mission

with updates every 3 monthsData volume estimate is 1 Gygabyte/day

(does not include the surveys or GRB products)


9

Database tables

• Each data set has at least one associated table – Allow user to select and retrieve data – Record high level information related to the data set (position, time

and other parameters)

• Additional tables associated with the production data:– As-flown timeline (log times of specific events) – Instrument configuration tables (log modes used by each of the

instruments during an observation)

• All tables associated with the production data contain keys to relate:– table entries with data files (e.g. sequence number)– different tables (e.g. target ID)

• Tables delivered daily for the production data – For the surveys and GRB products tables update with data delivery


10

Recent Activities• Modify and/or develop, package & distribute automatic

protocols for the archive (http://legacy.gsfc.nasa.gov/dts/dts.html) :– To transfer/deliver data Data Transfer System (DTS vn. 6) – To ingest Swift databases (DBSI vn. 1)– To ingest data in the archive (Data Archive System, DAS vn.

1)– Test plans: for DTS & end2end archive procedure

• Tests of the protocols (May-Jun, Sep-Oct 02) SDC, ISAC, HEASARC

• Prototype new interface to access table information & data archive

• Web prototype to ingest data from the follow-up Team• Ingest XRT & UVOT calibration data in their flight format • Ingest the Swift software in the HEAsoft package


11

Automatic ingest protocols• HEASARC modified DTS (originally written & operational for

XMM) :– To cope with multiple sites data transfers– To facilitate transfers of large files – Add extra the security capabilities

• HEASARC developed DAS to move data & tables from the staging area in the final archive location – DAS consists of perl scripts initiated by DTS – Includes capability to move & replace data– Makes a copy of the data set on a back-up storage– Sends e-mail to operator to ingest databases

– Keeps log of the data in the archive • HEASARC developed scripts to ingest the tables (DBSI)

– In the EXOSAT DBMS (used by the other DC sites)– Specific ingest procedures for the Swift tables


12

DTS test results


13

Swift archive interface • The Swift archive will be access via :

– HEASARC Browse – Dedicated Swift archive page

• The Swift archive page is ‘just’ a front end page to Browse: – To post news on the Swift archive– To provide up-front common Swift parameters for searches

(besides position & time) – To link the other data archive centers related to Swift & the GCN

• Query outputs from Browse and the Swift archive page will be identical (e.g. no changes in the underline software)

• We are implementing Swift databases by relating in the DBMS the Swift tables to facilitate user query & the retrieve of information .


14

Follow-up Team Web site

• Follow-up team collect data in response to a GRB alert– But there is no organized way to keep a record of these

data• Record this information is important :

– To log which observatories (and/or energy range) have performed /will perform observations for a given GRB

– To help plan future Swift observations – To assemble the results (and/or data) together with the

Swift results in the catalog (and in the archive) • We develop a WEB prototype to demonstrate a mechanism to

submit information from the follow up team• The prototype was presented at the last Swift SWG

– Currently gathering inputs & suggestions before final implementation


15

General Features

• Input form based on the Remote Proposal System (e.g. RPS) :– Cover page with the team member information (automatically

filled). – Observation form to input results and data file

— clone or add or delete previous input results– Submit : verify the form and email back all input information

• Observation form allows to :– Write results in pre-allocated boxes (makes easier to parse

numbers)– Upload files (format, content, and naming convention TBD)– Provide a link (http &/or URL ) to a site or to the data– Write a note to be shown with the data

• The input information is ingested in a database system :– It is tagged with GRB, team member and observatory name

• Search capability via GRB name or time (uses Browse HEASARC)


16

Ingest : Calibration & Software

• Calibration data

– Delivery of the Panther XRT calibration data (25 Nov 2002)– First delivery of the UVOT calibration data (29 Nov 2002)These data are in telemetry format are used to test the

reformatting to FITS and science software.

• SSC delivered the Build 2 Swift software to the HEASARC :

— It has been currently integrated within HEAsoft — Test distribution tar file (Build3 Feb, Build 4 April) to

ensure » no-crosstalk between different software packages » Properly run on different platforms


17

This year’s milestones

• DTS and end2end tests with UKDC (March 2003)• Final hardware configuration for the archive (May

2003)– Populate with the calibration data out the

pipeline• Implementation of the Swift data archive access

(June 2003)• Script framework to derive the GRB high level

data products• Software distribution : First release (Sep 2003)

Launch 5 Dec 2003


18

BeppoSAX Archive at HEASARC • HEASARC hosts currently 90% of the BeppoSAX public data:

• Data are from the NFI mainly LECS & MECS, 90% of the PDS and 10% of HPGSPC

• Data are delivered to the HEASARC via CDs by the BeppoSAX-SDC now ASDC (ASI Science Data center)

• The ASDC is now ready to re-process the NFI with the latest software & calibration & it s currently processing the WFC data

• Future delivery to the HEASARC : • WFC data FOT (not FITS, Starting from March 03) • WFC processed data (FITS, Starting from May 03) • NFI re-process data (FITS all NFIs, Starting in April 03)

• Delivery via DTS (already tested)

swift archive

Documents