![Page 1: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/1.jpg)
WIYN ODI:Observing Process, Data Analysis and Archiving
Pierre MartinYale Survey Workshop, October 2009
![Page 2: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/2.jpg)
ODI: Scientific Challenges
ODI is designed to take advantage of the best seeing conditions at WIYN.
ODI is a general facility instrument; Scientific programs will go from one single exposure to large-scale surveys and time-domain programs.
Optimization of ODI scientific output requires high observing efficiency, sophisticated calibrations and long-term availability of the data products.
![Page 3: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/3.jpg)
ODI: Operational Challenges
ODI offers five different acquisition modes including the default mode (e.g. full OTA correction) which requires a large number (~200) of guide stars.
ODI individual raw images are large (~ 2 Gbytes/image) and the (calibrations + scientific) data volume will be considerable (up to ~ 4 Tbytes per night !).
Data reduction will be complex, CPU intensive, and likely to evolve with time.
![Page 4: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/4.jpg)
ODI: The Observing Process ?
WIYN is planning to eventually offer queue observing
The many advantages of queue observing :•Adapting to sky conditions •Benefits highly ranked science•Well-suited for very short and long-term programs•Facilitates time-constrained programs•Provides high observing efficiency•Need photometry? You got it!•Balancing time between partners•Optimizes dark/gray/bright telescope time•More complete and controlled calibrations•Metadata add value to data
(The Plan: Start with the Yale Survey!)
![Page 5: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/5.jpg)
QSO: Three Steps Process
Queue Preparation&
Observations
ProgramSubmission
Data Evaluation
- Phase I (CADC) + TAC- Phase II (PH2)- Program Reviews and Prep
- Planning and Scheduling- Night Queues (4-5) - Observing Process- Calibration Plan
- Data Quality Assessment- Validation Cascade
E.g. The CFHT Queued Service Observing (QSO)
![Page 6: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/6.jpg)
Phase 2 Tier(Web client ?)
WOQ
Db
WOQ Tools
Scheduler
Obs Tool
LogBook
Data!
WIYN ODI QUEUE: The Concept?
![Page 7: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/7.jpg)
![Page 8: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/8.jpg)
The Daily Jigsaw Puzzle: Queuing!
![Page 9: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/9.jpg)
The Daily Jigsaw Puzzle: Queuing!
![Page 10: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/10.jpg)
The Daily Jigsaw Puzzle: Queuing!
![Page 11: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/11.jpg)
The Daily Jigsaw Puzzle: Queuing!
![Page 12: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/12.jpg)
Some CFHT QSO results :
A Programs : Completion >> 90%B Programs : Completion > 75%
Validation rate: 92-95%
Agency time balance: (e.g. MegaCam 2006B semester)
Caveat:Queue WORKS but it is not cheap ...
![Page 13: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/13.jpg)
ODI: Data Analysis and Archiving ?
WIYN is actively moving forward in developing a scientific data reduction pipeline and a long-term archiving solution for ODI
Main Challenges:
• ODI data will be complex to analyze and calibrate• ODI data volume will be very large (~150 TB/yr ?)• ODI science programs will explore the time-domain
![Page 14: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/14.jpg)
What the P&A system should do ...
(Most of the Yale Survey reduction requirements are covered in the PASRD)
![Page 15: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/15.jpg)
A process in several steps : • Tier 0
Quick look analysis for observers and time-domain programs: basic analysis done on site on local machines. [Included in ODI Project]
• Tier 1End-of-run, removal of instrument signatures and more advanced spectrometric and photometric calibrations on individual images. Updated master calibration products.
• Tier 2Production of optimum science products: Image stacking, high-accuracy astrometric and photometric solutions, PSF re-sampling, cosmic ray removal, fringing correction, etc. Fine-tuning on image stacking and catalogs.
• Tier 3 Image manipulation (e.g. filtering, image arithmetic), display, photometry, etc.
![Page 16: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/16.jpg)
A possibility: The “WIYN/IU” Model:
Pan-STARRS IPP, adapted to ODI and Teragrid, Data Mining through Science Gateway
Gateway
![Page 17: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/17.jpg)
WIYN Actualities: Done: Pipeline and Archiving Science Requirements
Document (PASRD)
Now: Design work continues on the “WIYN/IU” model
Now: UW and Yale committed to their share for funding the pipeline; waiting on Indiana. NOAO not committed yet.
Early 2010: Design review (+ management plan)
Now: Seeking WIYN ODI Pipeline Scientist
Now: Discussions with NOAO/Alternate Model ?
![Page 18: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/18.jpg)
ConclusionODI is going to be a very sophisticated camera,
designed to provide large volume of complex data for diverse scientific endeavors
Optimizing ODI Scientific Output :
Priority 1: Scientific Pipeline & Archiving Priority 2: Queue/Service Observing Mode
![Page 19: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/19.jpg)
ConclusionODI is going to be a very sophisticated camera,
designed to provide large volume of complex data for diverse scientific endeavors
Optimizing ODI Scientific Output :
Priority 1: Scientific Pipeline & Archiving Priority 2: Queue/Service Observing Mode
WIYN is working on those !
![Page 20: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/20.jpg)
![Page 21: Observing Process, Data Analysis and Archiving](https://reader030.vdocuments.net/reader030/viewer/2022021106/620568c94bbc36457e12923b/html5/thumbnails/21.jpg)