ocean glider data management: argo concepts, groom, ogc sensor observation services justin buck...
TRANSCRIPT
![Page 1: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/1.jpg)
Ocean glider data management:Argo concepts, GROOM,
OGC sensor observation services
Justin Buck ([email protected])+ many collaborators
British Oceanographic Data Centre
6th EGO Meeting – 16th June 2014
![Page 2: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/2.jpg)
The numerous collaborators
Sylvie PouliquenThierry CarvalJean-Philippe Rannou
Mark HebdenLise QuesnelAdam Leadbetter
Task 3.2 - Data managementCost action ES0904Data STSM partners
Marine Autonomous Robotic Systems (MARS) facility
![Page 3: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/3.jpg)
Outline
Introduction
GROOM data flow
Common data tools
Sensor observation services
![Page 4: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/4.jpg)
Introduction
![Page 5: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/5.jpg)
GOOS status (JCOMMOPS)
![Page 6: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/6.jpg)
User expectations
• Data easily accessible from a unique point/portal
• Data coherent in terms of:– Data format– Data Quality– Processing chain (clearly documented)
• Additional requirements for Monitoring and forecasting users:– Data are available in near real time (within 24 hours)– Data are available in delayed mode after calibration and /or
validation (typically within 12 months)
![Page 7: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/7.jpg)
Stakeholder expectations
• Opportunity to use more observations than they could afford alone
• Operate jointly part of the network
• Benefit from the other partners' experience from design to implementation to data management and user uptake
A key phrase for funding bodies:
Acquire once, use multiple times
![Page 8: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/8.jpg)
However ...Broadly speaking, two distinct types of deployment:
1) ‘Process’ type studies – short duration, spatially restricted, typically associated with a cruise. Designed to answer a specific question and can have data restrictions.
2) ‘Sustained observation’ studies – longer duration, regional scale missions. Typically repeated sections. Of interest to the ocean modelling and forecasting community.
So there is potential for conflict between ‘project’ interests and ‘operational’ interests
![Page 9: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/9.jpg)
Gliders/GROOM/EGO
![Page 10: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/10.jpg)
GROOM Task 3.2Data system goals
![Page 11: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/11.jpg)
A common data exchange format
EGO glider data format established by the GROOM community (October 2012):
Climate and Forecast (CF) and SeaDataNet compliant NetCDF.
Interoperable with data standards being developed internationally (e.g. IMOS in Australia and IOOS in the U.S).
Standard quality control protocols for both near real-time and ‘delayed-mode’ glider datasets (utilising Argo).
Ensures that glider data, metadata and technical information are stored and distributed in a consistent manner.
![Page 12: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/12.jpg)
Common data tools
![Page 13: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/13.jpg)
BODCworkflow
Data Provider
Data Scientist
Unauthorised user
Authorised user
Source data
Source meta
Delayed mode meta/data
Database meta
Database data
File System meta/data
Register Arrival
Archive
Convert to standards [meta/data]
QC/Calibration
Merge as relevant
Authorise
Authenticate
Prepare request
Store as relevant
External Database
Discovery
Manual Processing
Automated (Provider) upload
Automated download/upload
Automated (BODC) download
Checkout
Error Handling & Reporting
![Page 14: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/14.jpg)
BODCworkflow& commontools
Data Provider
Data Scientist
Unauthorised user
Authorised user
Source data
Source meta
Delayed mode meta/data
Database meta
Database data
File System meta/data
Register Arrival
Archive
Convert to standards [meta/data]
QC/Calibration
Merge as relevant
Authorise
Authenticate
Prepare request
Store as relevant
External Database
Discovery
Manual Processing
Automated (Provider) upload
Automated download/upload
Automated (BODC) download
Checkout
Error Handling & Reporting
Datacollection and securearchive
Internalstorage
Datadelivery
Reformatting & processing(EGO code)
![Page 15: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/15.jpg)
STSM to develop common tools
• Brest, December 2012
• Goal to develop first version of common tools– Tomeu Garau, Daniele Cecchi (NURC)– Thierry Carval, Jean-Philippe Rannou (Ifremer)– Justin Buck, Mark Hebden, Lise Quesnel (BODC)
![Page 16: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/16.jpg)
Common tools requirements
• Modular [for easy modifications and maintenance]• Consistent input/output interfaces between modules• Simple!• Flexible enough to handle varying use cases• Robust and handles any error with known rollback
points• Reports every failure• Similar tasks such as reading from the database are
done using similar approaches
![Page 17: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/17.jpg)
Common tools workflowSource data Seaglider Slocum Other gliders
Processingmodules
Datadelivery
Collection of source data readers, multiple readers per platform type. Data from single transmission converted to a .mat
Conversion of .mat output to EGO format NetCDF for transmission
Merge single transmission files to produce EGO NetCDF containing trajectory for deployment
Time series quality control routines
Single transmission quality control routines
Corrections/calibration routines
Originator GDAC GTS
![Page 18: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/18.jpg)
JSON files drive a generic system
EGO NetCDF writer
Raw data files
Merged EGO NetCDF file
JSON filesdescribing EGO format
JSON filesdescribing variables totransfer
JSON files describing deployment, glider and sensor configurations
*The flexibility means system is usable in other projects e.g. SMRU animal tags
![Page 19: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/19.jpg)
JSON files
EGO NETCDF writer controlled by JSON files.
Example on right describes CTD sensor metadata.
![Page 20: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/20.jpg)
Real time quality control• Test are adaptations of established Argo QC• RTQC configured via JSON files• Presently implemented tests
– Valid range (e.g. TEMP, PRES, speed etc)– Regional range– Gradient– Spike– Stationary– Position on land– Density inversion
![Page 21: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/21.jpg)
Sharing of code
• Code is available on the Ifremer SVN repository with Mantis used for bug tracking
• Pooling of common code from multiple centres
• Code has a reciprocal public license
• A SeaDataNet login is required• Thierry Carval is administrator
and can grant access
![Page 22: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/22.jpg)
Sensor Observation Services
![Page 23: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/23.jpg)
BODCworkflow& commontools
Data Provider
Data Scientist
Unauthorised user
Authorised user
Source data
Source meta
Delayed mode meta/data
Database meta
Database data
File System meta/data
Register Arrival
Archive
Convert to standards [meta/data]
QC/Calibration
Merge as relevant
Authorise
Authenticate
Prepare request
Store as relevant
External Database
Discovery
Manual Processing
Automated (Provider) upload
Automated download/upload
Automated (BODC) download
Checkout
Error Handling & Reporting
Datacollection and securearchive
Internalstorage
Datadelivery
Reformatting & processing(EGO code)
![Page 24: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/24.jpg)
Sensor webs - connecting data
From http://www.sensorsmag.com/networking-communications/wireless-applications/the-big-picture-sensor-webs-disaster-response-demo-1338
![Page 25: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/25.jpg)
OGC, SWE, O&M?
OGC – Open Geospatial ConsortiumSWE – Sensor Web Enablement initiative
OGC Defined prototyped and tested sensor web components:• Sensor Model Language (SensorML)• Observations & Measurements (O&M)• Sensor Observation Service (SOS)
Sensor Web Enablement
Sensor Model Language
Observations & Measurements
...
![Page 26: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/26.jpg)
Why SWE?
• Standardized web services will exist for accessing sensor information and sensor observations
• Sensor systems will be capable of real-time mining of observations to find phenomena of immediate interest
• Sensors will be capable of issuing alerts based on observations, as well as be able to respond to alerts issued by other sensors
• O&M required for EC INSPIRE directive
![Page 27: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/27.jpg)
OGC Sensor Observation Service (SOS)The SOS standard is applicable to use cases in which
sensor data needs to be managed in an interoperable way.
• Defines a Web service interface which allows querying observations, sensor metadata, as well as representations of observed features.
• Defines means to register new sensors and to remove existing ones.
• Defines operations to insert new sensor observations.
![Page 28: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/28.jpg)
International Harmonisation
• Meeting with IOOS and IMOS during December 2013 to harmonise data formats
• Adjusted format includes metadata structure changes for OGC SOS compatibility
![Page 29: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/29.jpg)
ncSOS – NetCDF implementation
• SOS service run on a repository of NetCDF files via a THREDDS server– C.F1.6 and Attribute Convention for Dataset
Discover (ACDD) attributes required• IOOS adapting THREDDS for 3D trajectory data• BODC investigating the addition of access
control methods for restricted data
![Page 30: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/30.jpg)
Potential BODC implementationNRT Source data Gliders NRT Ship underway Others e.g. Animal tags,
Argo, sea level, etc ...
Processingmodules
Datadelivery
Accession of data
Single transmission quality control routines
Originator via SFA GDACNODB (when delayed mode ready as per IDP project)
Met Office for push to GTS
Storage in repository of files
THREDDS server with ncSOS
Methods to interrogate & package data
Application of calibrations
Conversion to CF NetCDF with ACDD compliant attributes
web
![Page 31: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/31.jpg)
Summary
• First concepts for glider data management based on OceanSites, Argo, etc formed the basis of GROOM activity– Basic common tools available for conversion of glider
data to an international exchange format• Todays technology permits more advanced data
delivery methods than a decade ago and a prototype SOS services being developed– Several projects contributing ODIP & SenseOCEAN
(active), AltantOS (proposed)
![Page 32: Ocean glider data management: Argo concepts, GROOM, OGC sensor observation services Justin Buck (juck@bodc.ac.uk)juck@bodc.ac.uk + many collaborators British](https://reader036.vdocuments.net/reader036/viewer/2022062421/56649dc95503460f94abfba0/html5/thumbnails/32.jpg)
Questions?