the dark energy survey data management system
DESCRIPTION
The Dark Energy Survey Data Management System. Ignacio Sevilla Noarbe CIEMAT (Madrid) o n behalf of the DES Collaboration. From photons to catalogs. DES in a nutshell. Cosmological survey in visible/near IR light using 4 complementary techniques to characterize dark energy : - PowerPoint PPT PresentationTRANSCRIPT
From photons to catalogs
• Cosmological survey in visible/near IR light using 4 complementary techniques to characterize dark energy:
I. Cluster Counts
II. Weak Lensing
III. Large-scale Structure
IV. Supernovae• Two multiband (photometric) surveys:
5000 deg2 grizY to 24th mag AB griz
10-30 deg2 repeat (SNe)• Build new 3 deg2 FOV multi-CCD camera,
Data management system, improve Blanco facilities
Blanco 4-meter at CTIO
2/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
Credit:NOAO
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 3/29
SDSS-II DES
Area 10000 5000
Nb. of CCDs 22 62
Resolution 120 Mpix 570 Mpix
Raw data/night 200 GB/night 300 GB/night
Catalog size 18 TB 100 TB (est.)
Total data volume 60 TB 4 PB (est.)
Credit: Kotwani et al. (2010)
Transfer
Process
Archive
Distribute
4/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
Transfer
CTIO NCSA 300 GB/night in 18 h
Process
Archive
Distribute
5/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
Transfer
Process
Orchestration:
NCSA HPC nodes
Archive
Distribute
6/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
Transfer
Process
Archive
Results Archive nodesOracle DB
Distribute
7/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
Transfer
Process
Archive
Distribute
Through web portals
8/2911/08/2011 APS-DPF 2011 DESDM I.Sevilla
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 9/29
Exposure consists of 62+ CCD images – 570 Mpix - 3deg2
Survey is ~150,000 100-sec exposures over 525 nights
Auxiliary CCDs record images for autoguiding and calibration
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 10/29
X 300 + calib. = RAW DATA
we send this to NCSA
Correct for cross-talk among CCDs.
Correct for bias levels, non-uniformities, other optical and electrical effects.
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 11/29
DETRENDED DATA
Credit:NOAO
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 12/29
We need reference star catalogs
Full focal plane is fit to single solution
Correct optical distortion at focal plane
Credit: E.Bertin
We use Sextractor, SCAMP software by E.Bertin
REDUCED DATA11/08/2011 APS-DPF 2011 DESDM I.Sevilla 13/29
Use standard star fields at different angles in the sky (X) with known fluxes (m_std and color_std) and relate with instrumental flux (m_inst):
Make big least squares solution for a,b,k; apply results to science images.
€
minst −mstd = a+ bCCDncolorstd + kXinputoutput
At this point we have, for every night, approx. 300 exposures corrected by:
• Instrumental effects
• Absolute position
• Absolute photometry
This is the nightly processing.
We store these in the archive (+ auxiliary images, info).
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 14/29
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 15/29
COADDED DATA
Go deeper; calibrate better
BUT
Point spread function is inhomogeneous: each exposure has different quality
single exposure
single exposure
single exposure
single exposure
single exposure
single exposure
single exposure
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 16/29
COADDED DATA
Find PSF in each image
Homogeneize per coadd tile
PSF HOMOGENEIZED
0.77 ’’0.77 ’’
1.32 ’’1.32 ’’
0.94 ’’0.94 ’’
0.94 ’’0.94 ’’
It takes 30x more time for coaddition with respect to nightly:
• PSF has to be extracted• PSF has to be homogeneized• Actual addition of image and recomputation of errors
Additionally:
• Global photometric calibration among all images of the season
We store these in the archive (+ auxiliary images, info).
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 17/29
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 18/29
CATALOGS
E.Bertin
This step is performed with the SExtractor package
Position, shape and photometry is calculated.
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 19/29
Hundreds of millions of objects with hundreds of columns each:
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 20/29
Hundreds of millions of objects with hundreds of columns each:
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 21/29
Hundreds of millions of objects with hundreds of columns each:
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 22/29
Photometric redshift pipeline: take fluxes in five bands -> estimate redshift
Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear.
Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena.
mag_band_g = 20.7mag_band_r = 19.2mag_band_i = 18.5… + errors, other estimates
(this only one kind of photo-z! More estimations foreseen)
DESDM Photoz pipeline(neural network)
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 23/29
Photometric redshift pipeline: take fluxes in five bands -> estimate redshift
Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear.
Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena.
Eliminate instrumentalsignature
Obtain true shape (intrinsic galaxy shape+shear)
DESDM
WL pipeline
local PSF
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 24/29
Photometric redshift pipeline: take fluxes in five bands -> estimate redshift
Weak lensing pipeline: identify stars and construct PSF -> deconvolve from galaxies to obtain shear.
Difference imaging pipeline (SN): subtract images from different epochs to look for transient phenomena.
Credit: Pan-STARRS
11/08/2011 APS-DPF 2011 DESDM I.Sevilla
Produce cosmological simulations
Process through atmosphere, detectors, include ‘nasty’ stuff
25/29
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 26/29
Galaxies
Stars
11/08/2011 APS-DPF 2011 DESDM I.Sevilla 27/29
BCS Images of First SPT Clusters
Credit: Blanco Cosmology Survey
Real data sets from the Blanco Cosmology Survey, SPT, SCS (Mosaic2 camera).
Large scale management with SDSS data.
11/08/2011 APS-DPF 2011 DESDM I.Sevilla
The Dark Energy Survey (next talks!) will make use of a large, scalable data management system to process and archive raw images and science ready data products.
Acceptance of the system is underway (results end of year). Tests on real DES data expected for first months of 2012.
Raw and reduced images to be released yearly, catalogs at midpoint and end of survey.
Community pipeline getting ready for usage of the DES camera starting May 2012.
28/29
11/08/2011 APS-DPF 2011 DESDM I.Sevilla
11/08/2011 APS-DPF 2011 DESDM I.Sevilla
Development Funding ~$6MDevelopment Funding ~$6M $4 million from NSF$4 million from NSF $1.78 million (in kind) from NCSA/U Illinois, Fermilab, IAP and $1.78 million (in kind) from NCSA/U Illinois, Fermilab, IAP and
MunichMunich $300K from DES collaboration for Community Pipeline$300K from DES collaboration for Community Pipeline
(new simulations with more DES-like systematics coming up by Stanford team)
Carnero et al. 2010
Perform full-depth observations in 100 sq.deg.
Area is off main survey
Overlap existing datasets when possible
Run acceptance tests on data