using the nsa presentation to noao users committee october 5, 2005
TRANSCRIPT
Using the NSA
Presentation to NOAO Users Committee
October 5, 2005
NSA is part of an End-to-End Data Flow System
• NOAO Science Archive– Release 2 running (since Jan, 2003)
• Includes all compliant reduced data from NOAO surveys• OpenSky Node serves catalog data from NDWFS• Contents available through NVO services (DataScope)• Contains about 2 TB total
– Release 3 in development• Will support access to raw data from NOAO instruments• Both proprietary and non-proprietary access• Will include pipeline-reduced data as available• Infrastructure for eventual full-feature archive
• Data Reduction Pipeline Development– CCD Mosaic Pipeline nearing completion– NEWFIRM Pipeline to be ready for instrument commissioning– Both based on in-house pipeline management system and IRAF
modules
End-to-end System, Continued
• NOAO Virtual Observatory Portal (prototype)– Integrated user interface– Access to NSA and external data through VO protocols– Access to local and external tools and services– New visualization tools that allow much more sophisticated
exploration of data• Including time-domain
• Data Transport System– Captures data streams from all NOAO instruments– Safestore function replaces save-the-bits– Multiple copies at NOAO, NCSA– 3 TB cached in last 12 months
How it works• Although each element of program has value, the real breakthrough is the
combination of all into end-to-end data flow and management system– New method for observers to acquire their data
• Most data will arrive already reduced– Automatic processing of data from heavily used (“survey-friendly”) instruments– Access to archived pointed observations and many small surveys
• Develop concept of a “virtual survey”
– Unified interface/access to VO data and tools– Infrastructure to expand and develop new VO-enabled analysis tools
• Value Added– Multiplies value of all observations– Ultimate “broad impact” - science with minimal additional resources– Another way of providing access within system context– Appropriate role for national observatory: to guarantee long term access to and
curation of data for community
Sky
Time
Query
Access
Analyze
NSA+
Sky
Time
Query
Access
Analyze
NSA+
Sky
Time
Query
Access
Analyze
Sky
Time
Query
Access
Analyze
Sky
Time
Query
Access
Analyze
NSA+
SDSS+
Chandra+
XMM+
Sky
Time
Query
Access
Analyze
Sky
Time
Query
Access
Analyze
Group by Spatial Overlap
Sky
Time
Query
Access
Analyze
Group by Spatial Overlap
Sky
Time
Query
Access
Analyze
Group by Spatial Overlap
Sky
Time
Query
Access
Analyze
Sky
Time
Query
Access
Analyze
Create CatalogsRe-bin ImageViewCreate Cut-out
View catalogRe-bin ImageViewImcombineCreate Cut-out
Create CatalogR-bin ImageViewImcombineCreate Cut-out
Image Statistics
Image LocationsData Volumes
Image Locations
Milestones• Had hoped to demo end-to-end system at January 2006 AAS• Have had to scale back expectations due to staff losses
– Demo portal only at AAS– End-to-end system begins to run in April-June timeframe– Ramp up of operations next summer
• Operations includes quality control, user support, maintaining throughput, fixing problems
• Eventually catch up and reduce cached data
• Next cycle of development will include (e.g.)– Portal transition from scientific prototype to fully-engineered system– Develop integrated catalog-image archive tools– Extend portal-archive to include virtual workspace, grid-enabled,
user-configurable pipelines– Tools for community to prepare their datasets for inclusion
Other Issues• Reviewing our previous decision to make all
metadata available immediately– Probably will allow PI’s to request (with justification)
proprietary status for metadata• IRAF
– Remains important toolkit, but, in its present form, not the future of data reduction
– New context will be Web and VO– Portal will provide access to IRAF Web services– Continue to maintain, but minimize new development and change
user support model• Understand that many in community depend on IRAF• Will not undermine their ability to continue using it
