Download - CF Metadata Conventions: Governance, Support, and Future Karl E. Taylor Program for Climate Model Diagnosis and Intercomparison Lawrence Livermore National

CF Metadata Conventions:

Governance, Support, and Future

Karl E. Taylor

Program for Climate Model Diagnosis and IntercomparisonLawrence Livermore National Laboratory

Presented to theGO-ESSP Community Workshop

Paris, France

11 June 2007

PCMDI BASC / CRC17 May 2007 K. E. Taylor

Intro to CF metadata conventions

• Climate and Weather Forecast Metadata that makes data files self-describing

• Examples of CF metadata

Coordinate information needed to locate data in space and time

Further grid information (e.g., grid cell bounds, area)

Standard names – in conjunction with other attributes helps users determine whether data from different sources are the same physical quantity and helps distinguish variables within archives

Some processing info. (e.g., zonal mean, climatological processing)


Brief history of CF

• Started with “COARDS” standard, and developed through efforts of a handful of volunteers

• 2003: CF 1.0 released

• 2005: CF white paper discussing future governance circulated

• 2006: White paper revised and presented to WCRP WGCM

• 2006: Current governance structure established

• 2003-present: increasing acceptance of CF and adoption by several community coordinated projects:

IPCC AR4 archive conforms

PMIP, HTAP, regoinal groups, seasonal groups, ENSEMBLES …


Governance structure

• Original CF “authors” turned over control to two working committees:

Conventions Committee

Standard Name Committee

• Committee work done via email and web discussion (“Trac”)

• Support provided by:

BADC: Alison Pamment (50%) – standard names

PCMDI: Kyle Halliday (20%) – web site support

• WCRP/WGCM has been approached to appoint a CF Governance Panel charged with responsibility for stewardship of CF.


What happened at last GO-ESSP meeting?

• Discussed CF governance & procedure for making changes

• Some pressing CF issues were identified and in some cases individuals volunteered to try to make progress

Augmented grid information, especially for less structured grids.

Relationship between CF and GIS grids & metadata

Standard name issues (including dialects, profiles, hierarchy structures)

Relationship between CF and netCDF4

CF handling of in situ observations

How to handle discovery information

Sample datasets and reference implementations


Agenda for today

• Governance issues – what needs fixing?

• Underlying principles – any adjustments needed?

• Technical issues – what’s been done? Can we make progress on any of the discussion topics?

• Supportive software – status reports

Any additions? Other concerns?

Conventions Committee:

Support and Governance

Agenda Item

GO-ESSP Community Workshop

Paris, France

11 June 2007


Current committee members

• Karl Taylor (chair)

• Kyle Halliday (secretary)

• Balaji

• John Caron

• Jonathan Gregory

• Tom Gross

• Steve Hankin

• Jamie Kettleborough

• Russ Rew

• Rich Signell


Terms of reference

• Further develop (revise) CF conventions

• Consider implementation of CF metadata conventions in other file formats (besides netCDF)

• Determine what is meant by CF conformance


Current procedure for modifying/extending CF

• Anyone can propose a change

• Discussion opens & a member of the committee volunteers to moderate

• A provisional resolution is reached by consensus and summarized by moderator

• Reference files are produced illustrating the new feature

• If not a trivial change, trial implementations are carried out.

• If problems emerge iterate

• When consensus reached, the proposed change is provisionally accepted. (advanced implementers may begin adoption)

• Collection of changes evaluated as a package leading to new CF versions released by consensus

• Test reference files become part of the “test data”


Problems encountered.

• Progress on some proposals, but few have been carried through to implementation.

• Contrast with standard names committee

Few actions have taken place

With a couple of exceptions, the volunteer moderation of discussion topics has floundered

• Procedure for introducing changes outlined in white paper:

Email

Web-based Trac system

• Jonathan’s thoughts

Issues Concerning

Underlying Principles of CF

Agenda Item


Paris, France

11 June 2007


Underlying principles

• Special needs of observations

• Should the scope of metadata accommodated by CF include:

Description of experiment Description of model Description of higher-order grid-characteristics (e.g., relationship between u & v grids,

which grid cells are connected to which)

• Competing needs of data producers & application developers

CF extensions can be implemented faster by producers who sometimes are impatient for changes to be formally adopted

• Should requirement be relaxed that all CF metadata reside in the same file as the data itself?

• Bundling of common definitions of a quantity into I.D.’s (defined by a combination of attributes including the standard name + others)

• Aggregation layers

• CF profiles

Miscellaneous Technical Issues

Agenda Item


Paris, France

11 June 2007


Miscellaneous technical issues

• Ensembles (including realization_weight and other issues): Alison

• Time axis “issues” for forecasts: Jonathan

• Definition of the “geoid”: Alison

• Use of “anomaly”: Alison or Jonathan

• Subgrid variation description issues (use of “where”): Jonathan

• Time and calendar issues for paleoclimate simulations: Jean-Yves? Balaji?

The Future of CMOR

Karl E. Taylor

Program for Climate Model Diagnosis and IntercomparisonLawrence Livermore National Laboratory

Presented to theGO-ESSP Community Workshop

Paris, France

11 June 2007


The Climate Model Output Rewriter (CMOR)

• Provides a uniform interface for contributing CF-compliant data (adhering to “good practices”) to the growing number of MIP’s (e.g., CMIP, AMIP, CFMIP, PMIP, APE, HTAP)

• For IPCC AR4, nearly all model output in the archive was written through CMOR.

• Assures CF-compliance and all MIP-established output requirements are met.

• Traps many metadata errors when files are produced.

See http://www-pcmdi.llnl.gov/software/about_software.php


Project-specific CMOR tables facilitate and ensure consistency of model output from contributing groups

• Proper specification of several coordinate attributes, including:

Correct standard name

“axis”, “positive”, and “formula_terms” attributes, as appropriate

• Proper specification of several variable attributes, including:

Correct standard name

Required dimensions

“cell_methods” attribute

• A capability to

Reorder axis order

Reverse axis direction (or translate longitude dimension)

Convert units (through udunits)


CMOR flagged common errors, including

• Pointing out when required metadata are omitted.

• Rejecting incorrect metadata (wrong units, inadmissible attribute values, etc.)

• Rejecting inconsistent coordinate dimensions passed by user to CMOR.

• Rejecting non-monotonic coordinate values or inconsistent boundary values, as passed by user.

• Rejecting values that are clearly unrealistic (likely indicating improper units conversion or incorrect sign).


What changes will be implemented in the next 6 months?

• Station data (needed, for example, by the HTAP project), including

Station names

Station locations

• Output from regional models

With logically rectangular grids

But with non-cartesian longitudes and latitudes

i.e., data(i,j), lon(i.j), and lat(i,j)


Longer term prospects for CMOR

• Difficult to see extending CMOR to write the grid-description files mandated by the GridSpec proposal.

• Probably possible, however, to write the actual data files through CMOR (and leave it to other software to produce the special GridSpec files).

Download - CF Metadata Conventions: Governance, Support, and Future Karl E. Taylor Program for Climate Model Diagnosis and Intercomparison Lawrence Livermore National

Top Related