SeaDataNet - Pan-European infrastructure for ocean and marine data management
Present state, relations to EMODnet, and future developments
By
Dick M.A. Schaap – MARIS (SeaDataNet Technical Coordinator)
Copenhagen – Denmark, May 2011, EEA – WISE-MARINE Meeting
Supported by the European Commission
SeaDataNet has set up and is operating a pan-European infrastructure for managing marine and ocean data by connecting 40 National Oceanographic Data Centres (NODC’s), national oceanographic focal points, and ocean satellite data centres, in Europe
EU FP6 2006 - 2011
Interoperability
Interoperability is the key to distributed data management system success. This is achieved in SeaDataNet via:
Adopting the ISO 19115 metadata standard for all metadata directoriesProviding standard metadata entry tools: MIKADO, and online CMSProviding XML Validation Services to quality control the metadata maintenance Using common and controlled vocabularies, including international content governanceUsing harmonised Data Transport Formats for data sets delivery and standard tools for format conversion, such as NEMO, .. Using common quality control protocols and flag scale Adopting of OGC standards for mapping and viewing servicesUsing SOAP Web Services in the SeaDataNet architectureOrganising training and instruction workshops for transfer of expertise and hands-on training
Portal with harmonised services, data products and tools for users and data centres
http://www.seadatanet.org
Pan-European Directory Services
The SeaDataNet portal provides overviews of marine organisations in Europe and their engagement in research, scientific cruises, monitoring and data management for European waters and global oceans:
EDMO: European Directory of Marine Organisations (>2200 entries)
EDMED: European Directory of Marine Environmental Data sets (>3000 entries)
EDMERP: European Directory of Marine Environmental Research Projects (>2500 entries)
CSR: Cruise Summary Reports (>31500 entries)
EDIOS: European Directory of Ocean-observing Systems (>270 programmes for the UK alone and many underway for other European countries)
These Directories are maintained by NODCs for their country and published at pan-European level
Pan-European Directory Services
All these Directories have been harmonised and mutually tuned in format, syntax and semantics (common vocabularies), a common XML editor (MIKADO), online CMS and online user interfaces.
EDMERP
EDIOS
CSR
EDMO
EDMED
Relations between Directory Services
From the EDMO interface one can oversee and retrieve cross related entries in each of the Directories.
SeaDataNet from data discovery to unified access to analysis
CDI is a fine-grained index to individual data measurements. The CDI Service provides users a highly detailed insight and unified access to the large volumes of marine and oceanographic data sets An intelligent middle tier connection is configured between the SeaDataNet portal and the local data management systems at each of the data centres.
A shopping basket allows users to submit a shopping request for multiple data providers in one go and to follow its processing by each of the providers via an online transaction register.
Common Data Index (CDI) Data Discovery and Access Service
Common Data Index (CDI) Data Discovery and Access Service
Distributed Data Management
Data Policy and implementation
Metadatafree and open access, no registration requiredeach data centre is obliged to provide the meta-data in standardized format to populate the catalogue services
Data and products services the general case is free and open accesshowever users must register once in order to get a personal login - password
Web form to provide necessary information User agrees with “SeaDataNet User Licence”After processing, login/password sent by email (email check)
Licence is part of the SeaDataNet Data Policy, that is intended to be fully compatible with the Directive of the European Parliament and of the Council on public access to environmental information, the INSPIRE Directive, IOC, ICES, WMO, GCOS, GEOSS and CLIVAR data principles.
SeaDataNet Data Policy is an overarching policy, respecting any local policy
Each user gets a SeaDataNet Role; while each CDI record has a ‘Data access restriction’ label. The combination of role and access label determines ‘access OK’, ‘access to be discussed’ or ‘access denied’
SeaDataNet CDI – Data retrieval and downloading
Search
Results
Include inBasket
Shopping list
Submit + Authentication
RequestConfirmed
Data
Check StatusIn RSM
Ready at DC x
Download
SDN format
SeaDataNet CDI Quick Search interface
Drilling down and very intuitive for any user
SeaDataNet CDI Extended Search interface
Searching for nutrients
SeaDataNet CDI Extended Search interface
Details of a selected CDI record
SeaDataNet CDI Search interface
Searching for nutrients – results map zoom in and also with Blue Marble
SeaDataNet CDI Extended Search interface
Shop for nutrient data around Danmark
SeaDataNet CDI Search interface
Shopping list for nutrient data around Danmark
Submit
CDI extra functionalities
Numerous options for searching and drilling down of search results
CDI XML format can handle points, detailed tracks and polygons as GML
CDI Summary function for search results with statistics and drill-down function
CDI Mapping supports OGC WMS services with option to add external WMS layers and vice versa to serve CDI as WMS – WFS layer to other portals
A query and GIS layer configuration can be stored by users as a favourite and can be forwarded to other users as a URL
Extra attribute included for linking to (pre)viewing services
SeaDataNet CDI Search interface
Searching for nutrients – results Summary
SeaDataNet CDI Search interface
Searching for nutrients – results Summary – Data centres - clickable
SeaDataNet CDI Search interface
Add OGC WMS layers – EMODNet Hydrography
SeaDataNet CDI Search interface
OGC WMS layers added – EMODNet Hydrography
OGC WMS – WFS support
CDI entries from the CDI portal are available as a series of OGC WMS and WFS services, e.g. for inclusion in the EMODNet portals.
CDI - INSPIRE compliance
CS-W service on top of the CDI portal to deliver CDI XML entries in the INSPIRE compliant ISO 19139 format (at present aggregated by discipline and by data centre)
SeaDataNet Viewing services
Ocean Data View (ODV)General data analysis and visualization software>10,000 registered users
ODV software upgraded to ODV4 for:Extending ODV‘s graphical display capabilities and interactive controls for automatic and visual quality control and data quality flaggingSeamless connection to SeaDataNet output: SDN ODV4 data formats, quality flag scaleIntegration of DIVA gridding softwareImportant tool for the SeaDataNet regional Data Products
DIVA software (Data-Interpolating Variational Analysis) allows to spatially interpolate observations on a regular grid in an optimal way. The analysis is performed on a finite element grid allowing for a spatial variable resolution and a good representation of the coastline and isobaths.
SeaDataNet Viewing services – ODV + DIVA
Regional Products services
Climatologies and interpolated data products for the Arctic seas, Atlantic Ocean,Baltic sea, Mediteranean sea and the Black Sea regions. Statistical output in NetCDF files, that are visible via WMS through Ocean Browser
Cross sector involvement
SeaDataNet cooperates and provides services for many other European and international groups and projects, contributing and safeguarding good data management. Examples:
EuroGOOS, maintaining the European Directory of Ocean observing systems (EDIOS) and quality controlling and archiving long-term data series. See www.edios.org GMES Marine Core Services for streamlining the provision of long-term archives for optimising marine forecast services; there is an MoU formulating the cooperation between MyOcean and SeaDataNet POGO, collecting and providing information on ocean-going research vessels, and their operators, planned and completed cruises. See www.pogo-oceancruises.orgServices for the international oil & gas industry for metocean data (SIMORC). See www.simorc.org
SeaDataNet Expansion
SeaDataNet standards and tools are adopted by other EU-funded projects:
Up-Grade Black Sea SCENE project, (2009 – 2011) involving 6 NODCs and 35 other data holding institutes from the 6 Black Sea countries, to provide metadata and data access and to strengthen their national NODC networks. EU-funding 3.4 Million Euro. CASPINFO project, (2009 – 2011) involving 12 institutes and private industry from the Caspian Sea region, to provide metadata and data access and to build their capacity for data management and user services. EU-funding 0.8 Million Euro. Geo-Seas project, (2009-2012) involving 24 geological and geophysical data centres from 16 European countries (EuroGeoSurveys), to provide metadata and data access. EU-funding 4.9 Million Euro. EuroFleets project, (2009-2012) involving 24 research institutes and data centres from 17 European countries, to optimise the management of research vessels and to streamline the flow of data from the research cruises to the data centre infrastructure. EU-funding 7.2 Million Euro.
In each project several SeaDataNet partners are participating for transfer of knowledge & expertise, to secure interoperability and to achieve crossfertilisation leading to upgrading of the SeaDataNet standards and tools as well as enriching of the services.
SeaDataNet and EMODnet
EU has adopted a new Marine Strategy Framework Directive (MSFD) which includes a requirement for an overarching European Marine Observation and Data Network (EMODnet)
EMODnet can be considered as a network of existing and developing European observation systems, linked by a data management structure covering all European coastal waters, shelf seas and surrounding ocean basins’.
SeaDataNet and EMODnet
SeaDataNet and EMODnet
SeaDataNet has qualified itself as leading infrastructure for the EMODnet data management component and is undertaking several preparatory EMODnet projects:
Chemistry – focus on the groups of chemicals required for monitoring the MSFD for establishing Good Environmental Status – cooperation with EEA, ICES and Regional Conventions Hydrography – to produce and deliver a higher resolution digital bathymetry for European seasPhysics – together with MyOcean and EuroGOOS Regions to give wide access to operational metocean data, real-time and archives
The Geology lot is producing harmonised seabed maps (WMS). The SeaDataNet infrastructure is adopted via the Geo-Seas project to give overview and access to the underlying geological and geophysical data sets. SeaDataNet is data partner in the Biology lot and a closer technical relation is foreseen in the SeaDataNet II project.
EMODnet geographic areas
Hydrography North Sea Celtic Seas, Channel, Mediterranean sea,NW Atlantic
Geology North Sea Baltic and Celtic Seas
Biology North Sea Bay of Biscay and the Iberian Coast
Chemistry North Sea Black Sea + Med spots
Physics North Sea All other European seas
Habitats North Sea Baltic, Celtic Seas and the Western Mediterranean
SeaDataNet and EMODnet
The EMODnet pilots and associated research projects stimulate a wider uptake of SeaDataNet standards and contribute to enriching the standards and services
SeaDataNet is a project with a fixed partnership. The EMODnet pilots are data driven and bring on board many more data providers that adopt the standards to connect to the infrastructure and to populate the data resources.
To illustrate this: SeaDataNet comprises 40 Data Centres, but already 60 Data Centres are now connected and we are making good progress towards ca 90 connected Data Centres
EMODnet is management and policy driven which also provides an excellent opportunity for SeaDataNet to upgrade its status from project to sustained operational infrastructure.
CDI Service as common service
www.geo-seas.eu
www.emodnet-hydrography.eu
www.blackseascene.net
(www.emodnet-physics.eu)
SeaDataNet CDI Service
www.caspinfo.net
www.emodnet-chemistry.eu
CDI – Data Coverage
CDI coverage at end May 2011: > 1.000.000 CDI entries from 29 countries and >60 data centres for physics, chemistry, geology, geophysics, bathymetry and biology; connecting another 30 data centres and increase in CDIs is well underway
SeaDataNet II proposal – Recently accepted
There are many challenges and recent innovations which will be taken up in SeaDataNet II that will run from 1st October 2011 – 30th September 2015 with a funding budget of 6 Million Euro. A number of objectives:
Achieving full INSPIRE compliance and contributing to the INSPIRE process for developing implementing rules for oceanography
Achieving data access and data products services that meet requirements of end-users and intermediate user communities, such as GMES Marine Core Service, establishing SeaDataNet as the core data management component of the EMODNet infrastructure, and contributing on behalf of Europe to global portal initiatives, such as the IOC-IODE - Ocean Data Portal (ODP), and GEOSS.
Achieving an improved capability for handling also marine biological data and interoperability with the emerging biodiversity data infrastructure in Europe
SeaDataNet II project – selected technical aims
Establishing machine-to-machine interfacing next to present user portal for technically serving specific user communities, such as GMES Marine Core Service and WISE-Marine (EEA)
Developing online visualisation (previewing, viewing) services
Extension of the CDI metadata format with Observations & Measurements module and SensorML module, where applicable for specific types of data, e.g. for operational oceanography
Formulation of NetCDF core data model including SeaDataNet metadata header in cooperation with UniData
Tackling the duplicates problem
Conclusions
SeaDataNet is a strong partnership, well established in the Member States and in the ocean and marine community
SeaDataNet operates and further develops a data management infrastructure that is adopted as core data management component in the EMODnet pilots and also adopted by many research projects and monitoring initiatives
SeaDataNet has established and is further developing marine data management standards that are adopted on a wide scale
SeaDataNet has established and is further developing interoperability solutions, such as machine-to-machine interfaces, for serving specific user communities
Associated projects and EMODnet pilots stimulate expansion of the number of connected data centres and increase of data population, which is very relevant for MSFD and WISE-Marine.
EMODnet opportunities
Additional funding (22.5 MEuro) is being arranged for 2012 – 2014 for more data pilots and extending and operating existing pilots.
Moreover EMODnet has the promise of a substantial and structural funding from 2014 onwards for the data management component and the observation systems.
Already the preparatory actions are accelerating the convergence between different data sectors (geology, hydrography, chemistry, ….). There is an increasing willingness to harmonise, to become interoperable and to enable sharing of data in sectors and between sectors.
These opportunities and the SeaDataNet – EMODnet momentum should be used by DG Environment and EEA.
EuropeanMarine Observation Data
NETwork MSFD
Country data
MSFD brings EMODnet and WISE-Marine together
WISE-Marine
Research data
Policy relevant data
Reference layers (maps)
Marine Core Service data