BADC Workshop 2:BADC Services to Data Suppliers

Royal Met. Soc. Conference – 14 September 2005

Ag Stephens et al.

2

Workshop Outline

• Purpose of Workshop– To briefly present BADC provision to Data Suppliers– To gain feedback from the user community

• Workshop plan– Presentation: BADC Services to Data Suppliers– Points for discussion…

• How can we serve Data Suppliers better?• Which services need improving, and how? • What new services are required?

– Evaluation form

3

• Introduction and scope

- The BADC and its data suppliers- The NERC Data Policy

• Support to data suppliers

- Data management planning- Archival, distribution & service infrastructure- Metadata- File names- Data format- Data submission- Campaign support- Getting help

• Discussion

Presentation Outline

4

The BADC

• The NERC-designated Data Centre for atmospheric science is the BADC (under NCAS)

• It currently holds over 30 TB of data including NWP forecasts, climate runs, instrumental and satellite products

• It serves 7,000 users in the UK and overseas

The BADC’s data suppliers

• NERC-funded researchers (e.g. through Directed Mode Programmes or using a NERC facility such as the FAAM or UFAM instruments) — also the BADC data users!

• Other research or data centres (e.g. Met Office, ECMWF, Eumetsat, ESA, NASA)

• International research programmes (e.g. NDSC, EC-funded such as NitroEurope)

http://badc.nerc.ac.uk/

The BADC and its data suppliers

5

The NERC Data Policy stipulates:

• The Data Centres’ duties

• NERC grant holders’ duties, e.g.

– Get acquainted with the NERC Data Policy Handbook (*)

(*) http://www.nerc.ac.uk/data/documents/datahandbook.pdf

– Offer the data generated by a NERC-funded project to the designated Data Centre

– Ensure appropriate data custody, validation, documentation, cataloguing and dissemination

– Maintain and promote data stewardship standards

– Set up data protocols (conditions of submission, access and use)

– Assist UK researchers in locating and accessing data, including fetching data from external sources

– Handle data-related queries

The NERC Data Policy

6

Data Management Planning

At the outset of a research programme/project/experiment:

Scoping study to determine:

- scientific goals - external data needs

- project duration - data sharing needs

- staff and collaborators - investigators’ wishes

- details on data to be produced & archived (nature, volume, flow,…)

Data management plan (DMP) proposal and adoption (for large programmes): common dispositions and technical measures to meet the programme needs — in accordance with policies possibly already in force (e.g. international data policy, Freedom of Information Act, etc.)

Data protocol (DMP executive summary)

- submission time-frame- conditions of access

- retention time-period - conditions of use and publication

7

Metadata

Metadata = data about the data

Metadata are essential to enable the:- user (a human or computer) to understand the data (physical nature,

units, error estimates, scientific context, algorithms, instrument or model specifications, publication references, etc. — N.B. error flags, error bars may be integrated into the data body)

- get connected information on research context (experiment, project, platform, contact, etc.)

- read the data (format and layout)- find out about the existence of data, where the data are held and how to

access them (discovery metadata). This information is required by data portals, browsers and search engines. Our NERC DataGrid project is developing metadata formats following international ISO standards to improve data discovery (see: http://ndg.nerc.ac.uk/????).

Help on metadata - http://badc.nerc.ac.uk/help/metadata/

8

The BADC archive is based on a browsable file system.

We encourage meaningful file names to allow:

- identification of the file content without reading file- automated ingestion into the archive- automated use by handling software

Help on file names - http://badc.nerc.ac.uk/help/file_naming.html

File names

9

Under development:

- NetCDF/NASA Ames file format converter

Data Formats

- have a history of successive improvements - allow and encourage inclusion of significant metadata (such as

NetCDF’s Climate and Forecast [CF] Metadata Convention)- enable an ease of data exchange with collaborators- enable readability by existing software

Online BADC tools:

- NASA Ames file format checker- NetCDF file CF compliance checker

Help on formats - http://badc.nerc.ac.uk/help/formats

The BADC encourages the use of NASA Ames (ASCII) and NetCDF (binary) data formats which:

10

1. Requirements:

• you must register as a BADC user

• you must have been granted access to the relevant dataset

2. Data files are uploaded to the BADC incoming directory via:

• the BADC web-based data file uploader

• ftp to ftp.badc.rl.ac.uk

Data Submission

3. Data files are ingested (moved to the archive):

• with some checking of metadata and data format

NOTE:

This process is data submission to the final archive.

Uploading to the online workspace is not submitting the data to the BADC.

11

Campaign Support

Campaign Support includes:• The provision of meteorological data and products

(such as synoptic charts, rain radar images) in near real time

- Sources: Met Office, ECMWF• The provision of near real time satellite data (MSG)• The provision of forecast trajectories calculated with

forecast winds, either through the use of the BADC or Reading trajectory service

• The provision of dedicated online workspaces or FTP space allowing fast exchange of preliminary data

Near real time Met Office Products for the

Convective Storms Initiation Project (CSIP)

CSIP Collaborative Workspace allows secure upload and sharing of preliminary data for collaborators

12

Getting help/info

1st step - http://badc.nerc.ac.uk/

2nd step - badc@rl.ac.uk

Points for discussion… • How can we serve Data Suppliers better?• Which services need improving, and how? • What new services are required?