science archive for sky surveys data providers and the vo - nesc 2003 march wide field astronomy...

Science Archive for Sky Surveys

Data Providers and the VO - NeSC 2003 March

Wide Field Astronomy UnitInstitute for Astronomy

Data Sources

• UKIRT Wide-Field Camera (WFCAM) early 2004– 100 GB/night (four 2048 x 2048 infrared arrays)– ~24 TB/year pixel data + ~2 TB/year object data

from data-reduction pipeline operated by CASU

• VISTA Infrared camera from 2006-07

• 400 GB/night (sixteen 2048 x 2048 infrared arrays)

• ?? from a future optical camera ...

What is the Science Archive?

Not a simple repository of data, but a system that:

• enables access to large data volumes• provides reduced-size results sets• provides tools to enable science exploitation• can become part of a Virtual Observatory

Design approach

• Requirements & specification developed in

collaboration with stakeholders (UKIDSS &c)• Design governed by agreed requirements• Build on existing experience (e.g. SSS, 6dF GS)• Scalability ..........

Phased approach

• V1 Archive System for WFCAM first light Jan 2004• V2 one year after survey operations begin allowing

further studies of DBMS scalability

• V3 Grid-enabled, at some future point

Overall project plan

VISTA first light

ID Task Name Duration Start Finish

7 Agree CASU ICD 61 d 21/01/03 15/04/03

8 Archive CDR 0 d 16/04/03 16/04/03

9 Buy/install V1 Hardware 30 d 17/04/03 28/05/03

10 Hardware installed 0 d 30/05/03 30/05/03

11 Install SQLserv+SSA 21 d 02/06/03 30/06/03

12 SSA on-line 0 d 30/06/03 30/06/03

13 Develop V1.0 code 116 d 22/04/03 30/09/03

14 Aqu/install pixel hardw. 30 d 18/08/03 26/09/03

15 get CASU extract tools 83 d 05/06/03 29/09/03

16 V1.0 Integration/test 58 d 01/10/03 19/12/03

17 V1.0 Operable 0 d 19/12/03 19/12/03

18 Build/deliver WFCAM 370 d 01/07/02 29/11/03

19 WFCAM First light 0 d 05/01/04 05/01/04

20 Commission WFCAM 30 d 05/01/04 13/02/04

21 Ingest commis. data 30 d 05/01/04 13/02/04

22 UKIDSS Survey begin 0 d 13/02/04 13/02/04

23 Ingest Survey data 776 d 16/02/04 05/02/07

24 Develop V2.0 archive code 200 d 06/01/04 11/10/04

25 V2.0 H/w & DBMS study 278 d 10/03/03 31/03/04

26 V2.0 H/W & DBMS CDR 1 d 01/04/04 01/04/04

27 Acq/install V2 Hardware 131 d 02/04/04 01/10/04

28 get V2 tools from CASU 132 d 01/04/04 01/10/04

29 Final integration/test 60 d 12/10/04 03/01/05

30 V2.0 Operable 0 d 03/01/05 03/01/05

31 VISTA archive Phase B 450 d 12/01/05 03/10/06

32 V1.0 VISTA archive Op. 0 d 03/10/06 03/10/06

33 Ingest/curate VISTA data 63 d 04/10/06 31/12/06

Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 4 Qtr 1 Qtr 2 Qtr 3 Qtr 42003 2004 2005 2006

Version 1 Archive System

• Following SDSS DR-1 (not EDR) for catalogues:

“SkyServer” system employing MS SQL Server

running on W2K PC(s); browser-based GUIs with

clickable options and SQL editor

• Additional simple web-page interfaces for pixel data

• Facilities for joins with other catalogues e.g. 2MASS

SSS pixel data interface

• WWW interface to extract pixel or object data by (RA,Dec) selection

• Or, access via CDS Aladin tool

SSS pixel/obj data interface

After extracting data from WWW interface,

upload them straight into GAIA.

Form access to 6dFGSgenerates SQL for you

e.g. choose table from menu

and get the results

or write SQL directly

Migrate SSS to SQL Server

• DB Schemas for SSS developed

subset of SSS installed on a laptop• Put SSS catalogue data in SQL Server

to gain experience and test scalability in preparation for Version 1 of Archive

SSS SQLServer database

• ultimate SSS archive and precursor to WFCAM/VISTA

Version 2.0 and beyond

• Version 2.0 will, in addition to V1.0: - enhanced GUI functionality (e.g. plotting) - on-the-fly cats and pix over arbitrary areas - generate stacked frames with user-specified options - provide new source extraction options

• Version 3.0 will, in addition to V2.0: - return data in VOTable using VO transfer protocols - recast the online services as web services - develop (or install) advanced visualisation tools - develop (or install) advanced analysis tools - allow upload of user analysis codes - ……

Remote client GUI access

SDSS JavaQT

• Complex

queries using

SQL