VSO Status Update

HDMC Meeting

9 June 2008

Vision

To allow solar physicists to identify and search for data even if they don't know it exists.

Make it easier for them to get to the data whether they knew where it was before or not.

Why VSO?

Why not CDAWeb?Will they take > 2TB/day of data?

Why not FTP and VSPO:Do you want to scan > 70k files per day to

determine which ones meet your needs?Every time you perform a search?

(these numbers are just for SDO)

How to accomplish:

Standardized API for querying & requesting data using a consistent model at each provider

System to allow UIs to query a single API and have the queries distributed & responses aggregated.

Current Status

Data Providers : 14

Observatories : 26

Instruments : 55

Data Sets : 180?... dating back to 1915

Current Status (SPASE terms)

Repositories : 19 Observatories : 26 Observatory Groups : 3 Instruments : 66+ Instrument Groups : ~10 Numerical Data : ~180 products Display Data : (not tracked directly) Granules : ? millions

Data Sets Planned:

In progress: Hinode/SOT Huairou NRL EIT & LASCO TIMED/SEE

Planned: Hinode/EIS SMART SDO... and catalogs

SPASE use:

um ... no?

Current Interfaces / APIs

VSO SOAP API (finding & ordering): Web-based interface

3 custom VSO-derived interfaces (NSO, SAO, SSC) IDL-based interface

Dominic Zarro's IDL commands in SSW Perl command line interface AstroGrid

Registry API Catalog 'Display' API for HelioViewer

Interfaces / APIs

In progress: IDL client for Registry API More complex Catalogs API(s)

Planned: Cart API

and UI in IDL Catalog APIs for HELM

access via IDL SPASE compliant registry API

or at least SPASE descriptions of holdings

Data Coverage:

Incomplete.Measurement method:

People are still asking us to add data sets.

Goals:

Prepare for the flood of SDO data (and other recent missions: STEREO,

Hinode )

Plans for the Future:

Catalogs

Summary Rows

Looking to the work of others

Empowering new searches!

We’re doing some work under the hood!

Moving beyond time

• An event or feature driven approach

• Science catalogs• LASCO CME, GOES X-ray Flare, NOAA Active Region Number, Coronal

Holes, Bright points etc.

• Science queriesOne of the drivers for this

• SDO• So much data need to move to feature / event driven approach,• But useful now …

Give me all of these!

Actually I really want all XRT jets observed on the limb with coincident SOTand EIS data.

Would have made things much easier if I could haveperformed this query!

Science Queries

How do we know what queries to support?• We get asked!

• “It would be nice if … / I’d really like to be able to …”

Coincident observations• Observations of same features in multiple instruments within

specified time window.

Cadence• Find an hours worth of observations where there are images every

minute. (Helioseismology)

DEM Datasets• Find XRT datasets suitable for DEM analysis. (Closely spaced

observations, number of different filters/wavelengths). EIS data would be a bonus!

Science Queries

Investigating EIT dimming regions as proxies for CMEs

• Combine GOES Flare List + LASCO CME list to generate list of on disk flares without associated CMEs. Compare list to catalog of EIT dimming regions. Do spatial comparison. Generate list of EIT dimming regions whose properties are observed to be associated with CME list.

• Currently waiting on EIT dimming region catalog! Can do the first part to at least generate list of EIT images that should be analyzed.

• Support driven by science questions!

Conclusions

• We are moving from just data discovery in a certain time range, to finding datasets of interest that fit a number of scientific criteria.

• Searches for features and events and in the process blurring the distinction between data and catalogs.

• SDO will require these new approaches. • But it’s going to return a LOT of data.

Plans : Summary Rows

VSO data providers can return a record summarizing multiple data granules Data Providers can define their own groupings, UIs can

decide to expand them. Summary products still work like regular products

Data providers serve a tarball or data cube in response to a data transfer request

UIs that don’t understand summary products can still retrieve the collection

Changing over to a new JavaScript toolkit to provide AJAX summary row expansion

Plans : Look to Others

Catalogs Bobby Candey's HELM project (VxO funded) Peter Fox's SESDI project (NASA funded) Related IVOA & astronomy standards

Presentation / UI Use existing JavaScript toolkits

ExtJS and GWT... still requires customization to serve our needs, but gives

head start

Plans : Look to Others (2)

Massive amounts of data:How libraries handle interlibrary loanHow libraries handle 'de-duplicating' of

records Functional Requirements of Bibliographic

Records… is going to require significant changes at the data

providers.

The Future of VSO

• We are moving from just data discovery in a certain time range, to finding datasets of interest that fit a number of scientific criteria.

• Searches for features and events and in the process blurring the distinction between data and catalogs.

• SDO will require these new approaches.

http://virtualsolar.org/