ICSTI Workshop, 2012/03/05, F. Genova

The astronomical Virtual Observatory

ICSTI Workshop, 2012/03/05, F. Genova

Sharing astronomical data : why

• Major scientific objectives– Long term observations of variable natural phenomena– A large number of objects, complex interactions, many scales

• Observations with different techniques, at different scales (ground- and space-based observatories, large surveys)Multi-wavelength observations make a significant and increasing

fraction of publications• Re-using data for scientific objectives different from the

original ones, i.e. optimize the science return of large ground- and space-based instruments and of large surveys

IUE (1978-1996): five times more publications from data retrieved in the archive than from the selected observing teams (Wamsteker, Griffin, 1995) – a major precursor

ICSTI Workshop, 2012/03/05, F. Genova

Astronomy data landscape

• On-line information and services: heterogeneous, distributedobservatory archives, value-added services and tools (CDS), electronic journals and the NASA ADS bibliographic database, modelling data, tools, etc

• Lots of work behind the scene: re-using data requires that–it is properly described–users trust its QUALITY

• On-line services are widely used by the astronomical community

ICSTI Workshop, 2012/03/05, F. Genova

Data policy• Based on data sharing• Observational data obtained through competitive process,

available to all in observatory archives (after a 1 year proprietary period)

• Academic journals– A small number of major journals often managed by Astronomical

Societies or international agreement (Astronomy & Astrophysics, 25 member countries from Europe and South America)

– Table of contents and abstracts freely available, as well is ‘long’ tables at CDS (« attached data »)

– Open access after 1-3 years, immediate open access to a significant part of A&A

– Wide use of arXiv e-print archive

ICSTI Workshop, 2012/03/05, F. Genova

The VO: We did not start from nothingA network of on-line services

• The community is used to define exchange standards in international partnership – More than 30 years ago: FITS (observational data)

– Since 1983: bibcode (1999A&A…447…89T)

• Even before the Web: remote query to services• Since 1993, implementation and networking of

on-line services covering the whole range of ‘data’ – in particular for bibliographic information

A unified access to tables published in journals (CDS & journals)

• Common description– Improved quality: Checks

complementary to the referee’s– Data discovery: VO-enabled content

description– Data retrieval

• Also more general ‘additional data ’

ICSTI Workshop, 2012/03/05, F. Genova

http://vizier.u-strasbg.fr/viz-bin/vizHelp?cats/cats.htx

ICSTI Workshop, 2012/03/05, F. Genova

The astronomical Virtual Observatory• Enable seamless access to the wealth of on-line astronomy

resourcesall the world’s astronomical data should feel like they sit on the astronomer’s desktop workspaceAn ambitious goal and no pre-existing organisational model to follow (~2000)

• Pragmatic approach with a few basic principles– A global VO– Always keep in mind science usage and implementation by data centres– Fulfil astronomy’s needs (disciplinary VRE) BUT when possible use

generic building blocks to allow wider interoperability

• Global interoperability requires international agreementThe definition of interoperability standards isoverseen by the International Virtual Observatory Alliance

ICSTI Workshop, 2012/03/05, F. Genova

The IVOA

• Precursor: a Work Package of the European OPTICON network led by CDS (international already)

• IVOA created in 2002• Alliance of VO projects• Procedure adapted from W3C

(WD, PR, REC)• Working Groups and interest

groups on the different aspects • Now in implementation phase but

maintenance and new topics require sustainability

• Any data centre can join by implementing the interoperability layer

http://www.ivoa.net/Documents/

VO architecture

ICSTI Workshop, 2012/03/05, F. Genova

http://www.ivoa.net/Documents/Notes/IVOAArchitecture/index.html

Interoperability: current status

Continuing to work on standards remains mandatory–Feedback from implementation and scientific usage–Evolution of astronomy – new facilities, new science–Evolution of technology

From C. Arviset

Passage to maintenancemode for many standards

VO-enabled access to VizieR (in blue)

VO evolution• The VO has never been solely a technology

development• Scientists and data providers have been participating

from the beginning in the VO development• Things had to be made in the proper order

– The basic building blocks (standards and tools) had to be – and have been – built, with in mind take-up by data centres and science users

– Now towards operational phase – The focus is moving towards more support to take-up

by scientists and data providers, plus outreach towards education

Science requirements

• Science requirements have been present from the beginning– Scientists in VO projects– Science Advisory Committees of individual projects– Science demos

• When the basic standards have been developed, IVOA set up a Committee, then a Standing Committee for Science Priorities to identify high priority science cases, then performs a gap analysis to identify the lacking standards

ICSTI Workshop, 2012/03/05, F. Genova

Impact on Data Providers:The Euro-VO census

• Census of European Data Centres (EuroVO-DCA, EuroVO-AIDA, 2009, 2010)

• Inclusive definition : Data Centres populate the VO with data and services, service to the community, added-value, sustainability, quality

• 69 ‘data centres’ answeredData archives, services, theory data and services

Data centres in Europe (and elsewhere!)• A huge diversity in aims

– large archives of ground- and space-based telescopes provided by national or international agencies

– large systematic surveys of the sky, results of large simulations

– generalist data bases and services– smaller contributions of scientific teams which

share their expertise• Huge diversity in size and organisations• An ecosystem of data and service providers

willing to share data and knowledge - a distributed, heterogeneous system with no a central point nor hierarchical organisation

Strands of work during operational phase• Support to take-up by data providers• Support to take-up by the scientific community• Continuous technical development

– Standards (update of existing standards and new standards because of feedback/evolutions) – VO teams + IVOA

– Tools• Outreach towards education and the general

public • Interdisciplinarity a must in the current

« political » context

Interdisciplinary aspects

• IVOA had in mind to use generic components when possible. e.g. for two critical components for « wide » interoperability– Registry of Resources: OAI-PMH, Dublin Core (with

extensions)– Vocabulary: RDF + SKOS (semantic web)

• Re-use/adaptation by other disciplines: pragmatic approach through dissemination of knowledge by knowledgeable staff in EU projects in « nearby » disciplines (HELIO et al., VAMDC)

ICSTI Workshop, 2012/03/05, F. Genova

The astronomy knowledge infrastructure• Science driven information network widely used by the

scientific community• A model based on open access to data and services

(pragmatic open access strategy)• A fully distributed model

– Agencies responsible for large infrastructures provide data archives

– Established data centres provide value-added services and tools– Now smaller, motivated actors are appearing – labs willing to

share their knowledge• Links, portals, access tools• Mid-term sustainability

– Support to archive/data centres– Support to national projects which work on interoperability and

tools– Support to the International (and European) layers

ICSTI Workshop, 2012/03/05, F. Genova

European strategic exercisefor astronomy:

Astronet Roadmap (2008)

The data/service infrastructure isan important part of the disciplinary infrastructure

Riding the Wave report of the EU High Level Expert Group

on Scientific Data

Data is an infrastructureCollaborative Data Infrastructure

European strategies

ICSTI Workshop, 2012/03/05, F. Genova

Conclusion

• Astronomy : a global, heterogeneous, distributed, open data infrastructure widely used by the scientific community

• Multipolar, with no central point• Open to large organisations as well as to labs willing to share

their knowledge

• The sharing of knowledge is a global challenge• No single model – the way disciplines organize themselves to

share their data is strongly dependent on their culture• Any Collaborative Data Infrastructure should be able to

accommodate disciplinary « pillars » when they exist