emodnet chemistry 2 service contract mare/2012/10 s12.656742 technical developments of the emodnet...
TRANSCRIPT
EMODnet Chemistry 2
Service Contract MARE/2012/10 S12.656742
Technical developments of the EMODNet Chemistry portal
ByDick M.A. Schaap – Technical Coordinator
Split – Croatia, 17 – 18 June 2014, 1st Plenary Progress Meeting
Contract requirements for the portal
gather all measurements of a particular chemical species with their appropriate metadata within a given space and time windowvisualise the measurement density in a given time and space windowvisualise a time evolution of a selected group of measurementsshow concentration plots for a given time and space window and also along the coastcalculate spatially distributed data products specifically relevant for selected Marine Strategy Framework Directive Descriptorsinclude the physical conditions under which the measurements were made (from EMODnet physical parameters portal or the GMES marine core service)show inflows from rivers of nutrients. The user should be able to select a section of coast, a country or a region (NUTS3) and obtain time series of inflows of parameters expressed as mass or moles per unit time per river (or section of coast)
EMODNet Chemistry portal
The EMODNet Chemistry portal sits on top of the SeaDataNet infrastructure and makes use of its services that have been adapted and are further developed for specific EMODNet Chemistry needs. It provides various services and functionalities to users for browsing and viewing the chemistry data products and for identifying and requesting access to the gathered chemistry data sets for the European waters.The primary services are:
CDI Data Discovery and Access Service giving facilities for searching and retrieving chemistry data sets; OceanBrowser Viewing Service giving facilities for viewing, browsing and downloading Chemistry data products; Sextant Products metadata catalogue giving facilities for searching Chemistry data products and linking to the viewing service.
CDI service for discovery and unified access of data
Already 96 data centres connected and more underway
CDI Data Discovery and Access Service
Based up ISO19115 content standard and ISO 19139 XML Schema, fully INSPIRE compliantDedicated tools and facilities for generating CDI entries, format associated data sets and populating the CDI serviceShopping basket mechanism for discovery, access request, and downloading of data sets from distributed data centres Downloading in harmonised SeaDataNet formats: SDN ODV ASCII, and soonSDN NetCDF (CF) Adopted in many projects and ongoing improvements Operational governance scheme
Chemistry CDI Data Discovery and Access Service
Dedicated CDI service for EMODNet Chemistry scope Three search interfaces for human users:
Quick Search with dynamic drilling down of search results; Extended Search with more flexibility for combining search options, including free search;Variables Vs Marine Regions with an interactive Matrix of variables in specific marine regions.
Chemistry CDI Service upgrading
ISO 19115 content standard and XML Schema have been migrated to INSPIRE compliant ISO 19139
Extra fields included:EDMED referencesCSR referencesPublication referencesQuality Info referencesVocabularies NVS 2.0 upgradingSea regions (C19) references via geo-tagging
User interface also upgraded to support new elements and extra search options
Shopping basket extended from 500 to 10.000 CDI’s requests
Geo-tagging CDI’s with Sea regions (C19)
C19 – polygons for marking up all CDI entries with C19 terms using spatial queries + extra tags via C19 BT relations
Sea regions vocabulary (C19 as hierarchical service)
C19
Chemistry CDI Service upgrading – search interface
Extra P02
Extra Duration
Extra Sea Region
Extra Multiple
Chemistry CDI Service upgrading – details
EDMED references
Sea regionreferences
Chemistry CDI Service upgrading – details
CSR references
EDMOreferences
EDMERPreferences
Chemistry CDI Service upgrading – details
Publicationreferences
Quality inforeferences
Chemistry CDI Service – filter
EMODNet Chemistry 2 project has an extended scope of chemical substances:
Pesticidides and biocides Antifoulants PharmaceuticalsHeavy metalsHydrocarbonsRadionuclidesFertilisers Organic matter(e.g. from sewers or mariculture)ChlorophyllPartial pressures of dissolved gasesAcidity (from pH, pCO2, Total Inorganic Carbon, alkalinity)Others
Mapping between new EMODNet Chemistry 2 groups and P02 Vocabulary doneSets filter for virtual subset of SeaDataNet CDI to EMODNet Chemistry CDI collection
Chemistry CDI Service – present coverage (16 June 2014)
661095 CDI records and data sets
European waters (N80 W-30; N20 E45)587538 CDI records and data sets 62 Data Centres31 Countries248 Originators1868 – 2014 years82% unrestricted18% to be negotiated
Chemistry CDI Service – WMS / WFS services
The CDI locations and related metadata can be shared with other portals by means of OGC WMS (Web Mapping Services) and WFS (Web Feature Services)
http://geoservice.maris2.nl/wms/seadatanet/cdi_v2/emodnet/chemistryGetcapabilities
http://geoservice.maris2.nl/wms/seadatanet/cdi_v2/emodnet/chemistry?service=WMS&request=GetCapabilitiesExample WMS request:
http://geoservice.maris2.nl/wms/seadatanet/cdi_v2/emodnet/chemistry?styles=&format=image/png&transparent=true&request=getmap&version=1.1.1&srs=EPSG:4326&layers=lines&width=580&height=290&bbox=-180,-90,180,90Getcapabilites indicates what is available. Implementing WFS is depending on the client and needs programming. Example of WFS request:
http://geoservice.maris2.nl/wfs/seadatanet/cdi_v2/emodnet/chemistry?service=WFS&version=1.0.0&request=getfeature&outputformat=gml3&typename=points&maxfeatures=10&bbox=-0.2793103448275872,57.4448275862069,1.5206896551724127,59.244827586206895
Further developing controlled vocabularies for EMODNet Chemistry
As part of SeaDataNet controlled vocabularies (NVS 2.0) are maintained and served by NERC BODC as web services for marking up all metadata and data entries. At present more than 160 vocabulary lists are served with more than 150.000 concepts and with established and active gouvernance
The parameter usage vocabulary list P01 is used for data sets while the parameter discovery vocabulary P02 is used for the CDI metadata. P01 are narrower terms of P02. At present P01 already contains more than 30.000 concepts.
Therefore activities are undertaken in EMODNet Chemistry for:New entries because of the extended scope of substancesMaking mapping to P01 easier and more efficient by exposing the semantic model behind P01 and making it retrievable by componentsGrouping various P01 terms under an aggegrated term in a new vocabulary P35 that will facilitate data aggegration and product labelling
Further developing controlled vocabularies for EMODNet Chemistry
Example of P01 term: ‘Concentration of tributyltin cation {tributylstannyl TBT+ CAS 36643-28-4} per unit dry weight of biota {Mytilus galloprovincialis (ITIS: 79456: WoRMS 140481) [Subcomponent: flesh]}’
Semantic model:
Discovery and harvesting of data sets for regional product groups
EMODNet Chemistry 2 project is generating chemistry products per sea region (as defined by MSFD (draft))
This requires discovery and harvesting of data sets per MSFD region for specific chemical parameters
In EMODNet Chemistry 1 this was done on a manual basis using the CDI Discovery and Shopping mechanism
In EMODNet Chemistry 2 this was done initially in a semi-automatic way and progress is made towards an almost full automatic method:
using a Robot harvester via the shopping mechanism to discover and retrieve specific data sets from distributed data centres to compile and maintain specific aggregate data sets as internal central data buffers that can be transferred to regional groups for further processing and products generation.
Discovery and harvesting of data sets for regional product groups – 1st year
Filter set to discover nutrients data and to MSFD regions (approx.)
Boundaries of regions approximated by VLIZ and then schematised by MARISwith extra margins as GEO-filter
Discovery and harvesting of data sets for regional product groups – 1st year
Robot harvester has gathered circa 440.000 CDI records and data sets for nutrients in the given regions
These sets were transferred to the EMODNet regional groups per region
An online Buffer Content Management System (Buffer CMS) has been developed and tested for configuring specific data buffer profiles in agreement (SLA’s) with data providers AND for specific data user communities (such as EMODNet Chemistry regional product groups, MyOcean, SeaDataNet regional dataproduct groups, …)
Configuration settings concern discovery filter, buffer group, motivation, users (by means of SeaDataNet AAA services)
Robot harvester can be activated to perform retrieval for each buffer profile and also to maintain the central metadata and data buffers automatically for new entries and updates of existing entries
Progress of the robot harvesters is administered in the existing online Request Status Manager (RSM) system which is part of the CDI Shopping mechanism for tracking and tracing requests by users, data providers and overall
Discovery and harvesting of data sets for regional product groups – now almost automatic
Buffer Content Management System (Buffer CMS)
logon
Overview of buffersConfiguring profile of specific buffer
Buffer CMS + Central User Interfacing
CDI User Interfaces
CDI Robot harvester
Specific data buffers
CDI User Interface + Shopper:
Access regulated via AAA
Agreed Settings
Dynamic Maintenance
Robot CMS to configure Robot harvesting profiles
MARIS master
RSM systemextended with
administering robot transactions and via
central interfaces
API
Central User Interface with logon (AAA service) following authorisation in buffer CMS profiles
logon
Overview of authorized buffersCentral buffer UI incl direct shopping
The new Buffer CMS and Central buffer UI and API (under development for full machine-to-machine interaction) together with the new central shopping mechanism and upgraded RSM will greatly facilitate the maintenance of central buffers and regular delivery of data sets incl metadata to the Chemistry regional groups
The Central shopping mechanism works on the data Buffers and can deliver (in delayed mode) large data sets which are divided over downloadable zip files with maximum 10.000 data sets each; all shopping transactions are administered in new section of the RSM
REMARK: The central buffers are exclusive for specific applications and acces is secured via AAA service only for authorised users. These buffers do not replace the distributed CDI infrastructure and its shopping process for regular users.
Discovery and harvesting of data sets for regional product groups
Request Status Manager (RSM) service extended with administering of Central buffer interfaces
Logon as user/provider/master
New functions for central buffer shopping
However the central buffers will contain and deliver ‘raw’ data sets for specific parameters and as harvested from the distributed data centres => further action is needed for making the collection more homogeneous and validated => aggregated data sets
Aggregation and validation for generating homogeneous data collections can be done by using ODV software and specific expertise per region and chemical substance
Use will also be made of the new P35 Vocabulary for aggregating P01 terms. The P35 population is making progress:http://seadatanet.maris2.nl/v_bodc_vocab_v2/welcome.asp
The final buffers of aggregated and validated data sets will provide the input for data products and advanced visualisation services
Converting buffer data sets to validated aggregated data sets
EMODnet Chemistry extension
Specific data buffers
Oceanotron &OpenEarth services
ODV QC + aggregation
QCd data buffers
Regional experts
DIVA + OceanBrowser service
CDI RobotHarvester
Advanced services
The validated and aggregated buffers of data sets will be input for the products and advanced services:
Interpolated maps as produced with DIVA software (Ulg)
Time series graphics for selected stations via WPS (Deltares)
OceanBrowser viewer (Ulg) as common service for viewing the DIVA maps and giving access to the time series graphics
In a later stage also using Oceantron server (IFREMER) in between ODV binary files and OceanBrowser for additional visualisation services of all data sets
OceanBrowser Viewing Service (Ulg)
The viewer provides access to two kinds of products:DIVA interpolated maps. Output images available as horizontal sections and vertical sections. The latter can be selected by drawing an appropriate transect; (via OpenDAP, NetCDF, ncWMS); inclusion of predefined coastal sections Time series plots of selected stations via WMS of station locations and WPS (Deltares) for graphics on-the-flyIntegration with CDI service via WMS – WFSIntegration with SEXTANT Product Catalogue .
Sextant Products metadata catalogue (IFREMER)
Used to describe the Chemistry data products such as DIVA maps. This facilitates searching for specific data products and the exchange and use of the Chemistry data products in other services, such as the Chemistry OceanBrowser, and other portals with OGC WMS support. Metadata format: ISO19115 - ISO19139 with SDN Controlled Vocabs; CSW service based upon GeoNetWork
Planned developments and extensions
Single-Sign-On (SSO): only for requesting data using CDI service; uses CAS; work ongoing for supporting also OpenID and ShibbolethSearch:
the CDI search facility with be expanded with MSFD sea regions (once officially established) and possibly user drawn geographical polygon OceanBrowser menu will integrate the Sextant catalogue to facilitate searching of specific products for viewing
Visualisation:Further developing the time series visualisation services and their structured integration in system architecture Further analysis of use of Oceantron service on top of ODV binary files towards OceanBrowser for visualisation of data fields (first in frame of SeaDataNet 2)
Exploring cloud hosting and computing
Exploring options for hosting of central aggregrated buffer data sets and applications in a cloud hosting and computing environment
Prepared a background paper and Terms of Reference for cloud hosting of the validated buffer data sets and the advanced services for visualisation
Cloud hosting as neutral environment and highly performing
Dialogue with Cineca about options and possible way forward. Cineca is a non profit Consortium, made up of 69 Italian universities, and 3 Institutions, including OGS and CNR. It is the largest Italian computing centre.
Cineca is also partner in EUDAT, an FP7 project towards a pan-European Collaborative Data Infrastructure which will allow researchers to share data within and between communities and enable them to carry out their research effectively.
We will keep partners informed about progress.
www.emodnet-chemistry.eu