zeljko ivezic, robert lupton & mario juric lsst jtm ... · baseline: we plan to “catch up”...
TRANSCRIPT
Data Products Definition Document (LSE-163)
Zeljko Ivezic, Robert Lupton & Mario JuricLSST JTM Meeting, Mar 6, 2017
http://ls.st/lse-163
LSST Products in LSST Science Requirements
Large Synoptic Survey Telescope
Data Products Definition Document
LSST Document LSE-163
Mario Juric⇤, R.H. Lupton, T. Axelrod, J.F. Bosch,G.P. Dubois-Felsmann, Z. Ivezic, A.C. Becker, J. Becla,
A.J. Connolly, M. Freemon, J. Kantor, K-T Lim, D. Shaw,M. Strauss, and J.A. Tyson
for the LSST Project
May 5, 2016
Abstract
This document describes the data products and processing servicesto be delivered by the Large Synoptic Survey Telescope (LSST).
The LSST will deliver three levels of data products and services.Level 1 (nightly) data products will include images, di↵erence im-ages, catalogs of sources and objects detected in di↵erence images,and catalogs of Solar System objects. Their primary purpose is toenable rapid follow-up of time-domain events. Level 2 (annual) dataproducts will include well calibrated single-epoch images, deep coadds,and catalogs of objects, sources, and forced sources, enabling staticsky and precision time-domain science. Level 3 (user-created) dataproduct services will enable science cases that greatly benefit fromco-location of user processing and/or data within the LSST ArchiveCenter. LSST will also devote 10% of observing time to programswith special cadence. Their data products will be created using thesame software and hardware as Levels 1 and 2. All data products willbe made available using user-friendly databases and web services.
⇤Please direct comments to <[email protected]>.
1
DPDD
CONTENTS 2
Contents
1 Preface 4
2 Introduction 52.1 The Large Synoptic Survey Telescope . . . . . . . . . . . . . . 52.2 General Image Processing Concepts for LSST . . . . . . . . . 62.3 Classes of LSST Data Products . . . . . . . . . . . . . . . . . 72.4 Conceptual Design of Science Pipelines . . . . . . . . . . . . . 9
3 General Considerations 163.1 Estimator and Naming Conventions . . . . . . . . . . . . . . . 163.2 Image Characterization Data . . . . . . . . . . . . . . . . . . . 173.3 Fluxes and Magnitudes . . . . . . . . . . . . . . . . . . . . . . 183.4 Uniqueness of IDs across database versions . . . . . . . . . . . 193.5 Repeatability of Queries . . . . . . . . . . . . . . . . . . . . . 19
4 Level 1 Data Products 204.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204.2 Level 1 Data Processing . . . . . . . . . . . . . . . . . . . . . 21
4.2.1 Di↵erence Image Analysis . . . . . . . . . . . . . . . . 214.2.2 Solar System Object Processing . . . . . . . . . . . . . 23
4.3 Level 1 Catalogs . . . . . . . . . . . . . . . . . . . . . . . . . 244.3.1 DIASource Table . . . . . . . . . . . . . . . . . . . . . 264.3.2 DIAObject Table . . . . . . . . . . . . . . . . . . . . . 324.3.3 SSObject Table . . . . . . . . . . . . . . . . . . . . . . 344.3.4 Precovery Measurements . . . . . . . . . . . . . . . . . 364.3.5 Reprocessing the Level 1 Data Set . . . . . . . . . . . . 36
4.4 Level 1 Image Products . . . . . . . . . . . . . . . . . . . . . . 384.4.1 Visit Images . . . . . . . . . . . . . . . . . . . . . . . . 384.4.2 Di↵erence Images . . . . . . . . . . . . . . . . . . . . . 384.4.3 Image Di↵erencing Templates . . . . . . . . . . . . . . 38
4.5 Alerts to DIASources . . . . . . . . . . . . . . . . . . . . . . . 394.5.1 Information Contained in Each Alert . . . . . . . . . . 394.5.2 Receiving and Filtering the Alerts . . . . . . . . . . . . 40
CONTENTS 3
5 Level 2 Data Products 425.1 Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425.2 Level 2 Data Processing . . . . . . . . . . . . . . . . . . . . . 43
5.2.1 Object Characterization Measures . . . . . . . . . . . . 455.2.2 Supporting Science Cases Requiring Full Posteriors . . 485.2.3 Source Characterization . . . . . . . . . . . . . . . . . 495.2.4 Forced Photometry . . . . . . . . . . . . . . . . . . . . 505.2.5 Crowded Field Photometry . . . . . . . . . . . . . . . 50
5.3 The Level 2 Catalogs . . . . . . . . . . . . . . . . . . . . . . . 505.3.1 The Object Table . . . . . . . . . . . . . . . . . . . . . 515.3.2 Source Table . . . . . . . . . . . . . . . . . . . . . . . 575.3.3 ForcedSource Table . . . . . . . . . . . . . . . . . . . 60
5.4 Level 2 Image Products . . . . . . . . . . . . . . . . . . . . . . 615.4.1 Visit Images . . . . . . . . . . . . . . . . . . . . . . . . 615.4.2 Calibration Data . . . . . . . . . . . . . . . . . . . . . 615.4.3 Coadded Images . . . . . . . . . . . . . . . . . . . . . 61
5.5 Data Release Availability and Retention Policies . . . . . . . . 63
6 Level 3 Data Products and Capabilities 656.1 Level 3 Data Products and Associated Storage Resources . . . 656.2 Level 3 Processing Resources . . . . . . . . . . . . . . . . . . . 666.3 Level 3 Programming Environment and Framework . . . . . . 676.4 Migration of Level 3 data products to Level 2 . . . . . . . . . 69
7 Data Products for Special Programs 70
The main classes of LSST data products:1) Images: single visit, coadded images, difference images2) Catalogs: Level 1: DIA Sources, DIA Objects, SS Objects, Alerts Level 2: Sources, Forced Sources, Objects3) Alerts
Motivation for these revisions:
- it’s been ~3 years since the last revisions of DPDD (LSE-163, what) and “Science Pipelines” Document (LDM-151, how)
- a lot of DM development happened over last 2 years: useful feedback and potential for design improvements
- last chance for major edits (~3 years to first light!)
The main changes in DPDD:
- a pedagogical introduction to overall pipeline design, with high-level block diagrams: provide a reader (both stakeholders and developers) with an understanding of the complete DM system to be constructed or operated
- changes in nightly Level 1 processing and a simplification of its interaction with DRP (data release processing): tentative
- changes in forced photometry: measured on both direct and difference images: tentative
- revisions of measured parameters (e.g. dipole fit for DIA Sources, adaptive moments, options for galaxy models)
- general cleanup
Single Visit Processing- “classical” astronomical image processing (e.g. similar to SDSS)- prototype implementation operational on HSC data - rudimentary QA analysis tools available - PSF pipeline is a non-trivial task- calibration products pipeline is implied above
Image Coaddition- algorithmic research not
completed - rudimentary
implementation available
Detect and deblend sources- association issues not fully
resolved - rudimentary
implementation available - “StackFit” measurements
Multifit
- modeling issues not fully resolved
- requires major resources - direct impact on science
Image Differencing- algorithmic research not
completed (DCR, seeing)- rudimentary
implementation available
Difference image analysis
- rudimentary implementation available
- real time (60 sec) processing, which must be exceedingly robust: scary!
Alert Generation- if DIA works fine, it
shouldn’t be too hard to construct Alert stream
- lots of issues with external brokers
MOPS- rudimentary
implementation available
- recent work in the context of NEOs showed that MOPS is in a much more sorry state than we thought
- in addition to components that we don’t have, we might need to rewrite those that we thought we did have
Baseline: we plan to “catch up” with the new data acquiring during the DR processing period, and then replace the “live” Level 1 db with the new DRP one. This switch would likely be disruptive for Level 1 science and downstream brokers.
Alternative proposal:- use a running 6-12 month wide window to compute summary
quantities in DIA Objects: a “living” Level 1 db (DIA Objects retain the memory of only that window, and not of all the old data)
- freeze the dataset at the beginning of DR processing- produce Level 1 db as part of DRP (using all data), but do not
replace the “living” Level 1 db
Level 1 DR Processing: - produce difference images for all visits and find all DIA Sources- associate DIA Sources to DIA Objects and SS Objects- go back (2nd pass) to single-visit images and difference images and perform all required processing for all DIA Sources from DIAObjects and SS Objects (image models for SNR>5 DIA Sources and Forced PSF Photometry for all DIA Sources)
