seadatanet pan-european infrastructure for ocean and marine data management

OBSERVATIONS

& PRÉVISIONS CÔTIÈRES

www.seadatanet.org

SeaDataNetPan-European Infrastructure

for Ocean and Marine Data Management Gilbert Maudire and the SeaDataNet Consortium

SeaDataNet mid-term review – Bologna – September 2008, 19

OBSERVATIONS


2www.seadatanet.org


Data is of vital importance for marine researchand support to public decision

Marine observation and data sets status

o observation conducted by more than 1100 active scientific laboratories in the countries of partners, using various sensors and platforms

o not always quality controlled using comparable procedures

o not always easily accessible

Necessity of large background work in order to : secure them in long term archivesdocument available observations and data sets (metadata) improve coherence of data sets (references, quality, redundancy) make data sets accessible easily (formats, on line distribution)

OBSERVATIONS


3www.seadatanet.org


What is SeaDataNet? EU Sixth Framework Programme

Integrated Research Infrastructure Initiative (I3)

1 April 2006 to 31 March 2011 (5 years)

Objectives

o To network existing oceanographic data centres already nationally funded

o To develop an efficient distributed pan-European marine data management Infrastructure (a “unique Virtual Data Centre”)

o To provide on-line access to distributed databases of standardised quality by using adapted communication & information technology

This project integrates and develops several EU past initiatives and international cooperation, in particular Medar/Medatlas, Edmed, Edios, Sea-Search etc.

OBSERVATIONS


4www.seadatanet.org


49 Partners National Oceanographic Data Centres

o divisions of major national marine research institutes in 35 countries

o experienced as leaders of data Management of many scientific projects and of concerted actions (e.g. MTPII-MATER, CANIGO, OMEX, EDIOS, MERSEA, MEDAR/MEDATLAS, SEA-SEARCH)

3 satellite data centres

3 scientific modelling centres

2 SMEs experts in software development

3 International organisations

with strong regional interest/cooperation

OBSERVATIONS


5www.seadatanet.org


Primary focus on water column parameterso Physical

o Chemical

o Bio-chemical (non species dependant)

Projects will be proposed for other domains by involved communities

in strong relationships with SeaDataNet

Several SeaDataNet partners are already involved in those domains

o Geophysical and geological data

o Biology (species dependant)

SeaDataNet managed data

OBSERVATIONS


6www.seadatanet.org


SeaDataNet services Online services : web portal and web services

o Discovery services • Metadata directories • Common vocabularies : parameters, units, …

o Delivery services • Security services : Authentication, Authorization & Administration (AAA)• Data download and access including reformatting in standardized formats

o Viewing services• Web map services, data visualization.

Offline services : « back office » serviceso Safeguarding within the national repositories (NODCs)

o Quality Control and desktop viewing services• ODV analysis software

o Product generation adapted to each regions

o Monitoring and statistics : usage & system performance

OBSERVATIONS


7www.seadatanet.org


Infrastructure versions & planning Version 0 – 2006-2007

Continuation and maintenance of existing Sea-Search system : the data access needs several different requests to each data centres and the data sets are delivered in different formats

o Operated at the start of the project

Version 1 – 2008-2010Setup of the integrated online data services to users :

networking of 10 “interoperable” data centres of the Technical Task Teamunique request to the interconnected data centresand the data sets are delivered with a unique format

o Presently under test and progressive integration of 10 data centres during 2008

Version 2 – 2010-2011Improved version of the system (ease of use, data visualisation, data access)and by step, all data centres connected for providing online trans-national access to the data, metadata and products

o Operational in 2010

OBSERVATIONS


8www.seadatanet.org


Discovery services

All catalogues (European metadata directories)• are compliant with ISO19115 standard family for spatial metadata• make use of common vocabularies and references• are accessible and linked through web services• ease data access using various selection criterions adapted to

different users (scientists, observatory and fleet managers, …)

OBSERVATIONS


9www.seadatanet.org


Common data index : gate to data access

online searchable directory

where?, when?, what?, …

tool for managing user’s“Shopping Basket”, that is :o To prepare a list of distributed datasets

matching user’s requesto To ask concerned data centres to serve the user,o To display the status of user’s requesto To route the requested data from the distributed

data centres to the user

OBSERVATIONS


10www.seadatanet.org


SeaDataNet data policy and data licenceinspired by INSPIRE Directive to facilitate the data flow takes into account the local rules and the SeaDataNet users needs

Metadatao free and open access, no registration required

o each data centre should provide the meta-data in standardized format to populate the catalogue services

Data and productso visualisation freely available

o data download, in general, free for academic purposes with mandatory registrationo a “SeaDataNet role” (partner, academic, commercial etc.) is attributed to individual user

using the Authentication, Authorization and Administration (AAA) Service• Each NODC attributes the roles to the users of its of country

• Out of the partnership, the roles are assigned by SeaDataNet user-desk

o a common data licence has been adopted and users must agree on when registering

o each data centre node delivers data according to the user’s role and its local regulationo each data centre should provide the data sets necessary to develop the common products

OBSERVATIONS




QC procedures and tools

SeaDataNet protocol :

o Coherent automatic and visual checks are performed on the data by each NODC

o A unique quality flag scale has been adopted The automatic checks and the visualization could be performed

using either by Ocean Data View (ODV) or existing partner software (if the same protocol and checks are applied)

Example:Using derivatives for spike & discontinuity detection in ODV

OBSERVATIONS




Data products SeaDataNet develops added-value products at regional

scales:o Mediterranean Seao Black Seao Baltic Seao Arctic Seaso Atlantic and global oceanThe preparation of these products allows the

checking of the compatibility of the QCs performed on the data in each NODC.

First products => gridded climatologies and trends

The comparability of the different regional products is assessed by using a new version of DIVA (Data Inverse Variational Analysis) developed (University of Liege)

OBSERVATIONS




Capacity Building is a main issue Training program main objectives

to ensure that management procedures are coherent within the consortium

to transfer expertise and to train IT experts of the SeaDataNet data centers in the installation and operation of the SeaDataNet technical components

OBSERVATIONS




www.seadatanet.org

Please contact : [email protected]

OBSERVATIONS


www.seadatanet.org

SeaDataNetProject management


OBSERVATIONS




Role of partners (1/4) Coordination

o Ifremer : project coordinator

o Maris : technical coordinator animator of the Technical Task Group with the help of :

• NERC/ BODC : standard development

• AWI : data quality control and desktop viewing services

o University of Liege : scientific coordinator in charge of the regional products development with the help of :

• JRC : Product harmonization

• INGV : coordinator of Mediterranean products

• METU : coordinator of Black Sea products

• NERI : coordinator of Baltic products

• IMR & VLIZ : coordinators of Artic & North Seas products

• Ifremer : coordinators of Atlantic & Global Ocean products

OBSERVATIONS




Role of partners (2/4) Coordination (cont.)

o HCMR : Coordinator of the networking activities i.e. metadata and data management, system monitoring

with the help for coordinating the distributed network of

• Maris : management of Common Data Index (CDI) and of Directories of Marine Projects (EDMERP) and Marine Organizations (EDMO)

• NERC/ BODC : management of Directories of Marine Environmental Datasets (EDMED) and of Ocean Observing System (EDIOS)

• BSH : management of Cruise summary reports

• Ifremer : Management of SeaDataNet users

OBSERVATIONS




Role of partners (3/4) Technical task team (11 partners : Western Europe + Russia)

o Technical specifications,

o Check of the defined standards

o Integration and system setup, test and validation of technical components

o Running of System V1

Capacity building and training

o UNESCO/ IOC Project office with the help of Technical Task Team

Information and communication

o ENEA

OBSERVATIONS




Role of partners (4/4)

All partners which are mostly NODC’s (except international organizations) have to

o generate metadata : organizations, projects, cruises, observing systems, data sets

o manage data from their country : quality, format, … according to SeaDataNet agreed standards

o provide access to their data repository, automatically using SeaDataNet system or manually as a first stage

o manage users of their respective countries by providing information and help

OBSERVATIONS




External relationships / Collaborations

Advisory board

o MEDS Canada (represents IODE/WMO JCOMM, GTSPP and Argo international programs)

o US – NODC (represents North America initiatives like DMAC)

o EuroGOOS (represents Operational Oceanography community)

Standard summit (last January on behalf of COI)

o North America, Europe, Japan

“Memorandum of understanding” with other consortiums

o Unidata, Mersea-MyOcean (GMES), EuroBis-Marbef (FP6), GENESI (FP7)

International conference on Marine Data and Information Systems (IMDIS)

o April 2008, more than 200 participants from about 50 countries.

OBSERVATIONS




Project management tools SeaDataNet web site www.seadatanet.org

o Distributed management, based on a Content Management system (EZPublish)

o General information : Description of Work, Meetings, Conferences, Education (several languages)

o Project services (discovery, data access, …)

o Finalised documents (applicable version of standards) and software tools

Mailing listso Based on Sympa software

o One list per group (technical, …) owned by the chair person

o + user desk (SeaDataNet users point of contact)

Document managemento Based on BSCW software (Basic Support for Cooperative Work - Germany)

o Cooperative work on documents (non finalised versions) and software

Software management (versionning tool)o Based on GForge

OBSERVATIONS




System monitoring focused on system “availability”

Availability of SeaDataNet user serviceso availability of web pages and web services

general information, metadata browsing (EDMED, CSR, CDI), data selection users’ interfaces, web services

o availability of web data access servicesdata centre download services

o Service “available” if monitoring system get an answer• Search of keywords within web pages,• Web services can be reached : WSDL returned

Monitoring o done by IFREMER (coordinator) and HCMR

in order to avoid biases due to network troubles

o Using NAGIOS Open Source software• Polling of services (web pages, web services)

OBSERVATIONS




Polling of system componentsType(*) Name Address (URL) Comment (**)

BODC

Web page (static)

BODC home page http://www.bodc.ac.uk/ Apache web server check (HEAD)

Web page (dynamic content)

http://www.bodc.ac.uk/cgi-bin/dbcheck Oracle database availability check (via Perl) (GET)


http://www.bodc.ac.uk/data/online_delivery/gebco/ Tomcat availability check (HEAD)

ENEA

http MOON-VOS http://moon.santateresa.enea.itHCMR

User Interface

Oceanographic database

http://hnodc-data.hcmr.gr

Ifremer

Web page + User interface

Nautilus http://www.ifremer.fr/nautilus

Web page +User interface

Coriolis http://www.coriolis.eu.org

OGS

Web page (static)

NODC portal http://nodc.ogs.trieste.it/

BSH

Maris

Web site (static pages)

Sea-Search portal http://www.sea-search.net Will be integrated into SeaDataNet portal soon/ count only on static pages


http://www.bodc.ac.uk/maps/

http://www.bsh.de/de/Meeresdaten/Beobachtungen/DOD-Datenzentrum/index.jsp

Map server availability check (HEAD)

Home page DOD Home page

OBSERVATIONS




Monitoring board Example of one-month report for Ifremer http access

Electrical failure during 6 hours

OBSERVATIONS


www.seadatanet.org

SeaDataNetAdministrative and financial aspects


OBSERVATIONS




Known issues IOC Project Office partner

o In charge of training courses and capacity buildingo Report at 0 up to now even if work done (3 training courses organised yet)

no agreement on US $ / € change rate between UNESCO and EC)

o IOC is an « Additional Cost partner »not well adapted to training course organization (additional personnel)

Transnational activities report

o Partners with very small web site and/or very little audience

o Audience measurment : KiloPageViews unit difficult to report accurately

o 80% of partners report less than expected for TA

OBSERVATIONS




Transnational Activities amendment

Change reporting unit from KiloPageViews to KiloHits

o To take into account all hits on online services :web pages (for any usages), web services, ftp servers, …

o But, cost / unit must decrease (Project Officer comment)

However, most audiences will remain far less than expected

o Cost are not linearly proportional to the audience« Entry costs » to set up a web site funds transfer to JRA (web site maintenance)

o Who? How much?

All must remain in the same budget envelope

OBSERVATIONS


www.seadatanet.org

SeaDataNetPlanning


OBSERVATIONS




Infrastructure versions & planning Version 0 – 2006-2007

Continuation and maintenance of existing Sea-Search system : the data access needs several different requests to each data centres and the data sets are delivered in different formats

o Operated at the start of the project

Version 1 – 2008-2010Setup of the integrated online data services to users :

networking of 10 “interoperable” data centres of the Technical Task Teamunique request to the interconnected data centresand the data sets are delivered with a unique format

o Presently under test and progressive integration of 10 data centres during 2008

Version 2 – 2010-2011Improved version of the system (ease of use, data visualisation, data access)and by step, all data centres connected for providing online trans-national access to the data, metadata and products

o Operational by 2010

OBSERVATIONS




Project planning and budget

High level of effort during the first three years(80% of the budget)

o Common standard adoption, set up and training,

o Development / Adaptation of common tools

o System specifications, development and setup (V1 and then V2)

o Development of first release of regional products and initial harmonization

Lower level of effort during the last two yearso Continuous data management

o Exploitation of the system, system maintenance and upgrade

o Upgrade of regional products

OBSERVATIONS




More information about budget

Funds breakdown / year / partner after second report

o Funds breakdown : https://www.ifremer.fr/bscw/bscw.cgi/0/110042

Transnational access audience

o Evolution for TA- Seadatanet.xls

seadatanet pan-european infrastructure for ocean and marine data management

Documents