Integrated Plug-and-Play Modeling:An Overview of CSDMS and the

Earth System Bridge Project

Scott D. Peckham

Senior Research Scientist at INSTAAR

Lead PI for Earth System Bridge

Former Chief Software Architect for CSDMS

University of Colorado, Boulder

May 21, 2015

Community Surface Dynamics Modeling System (CSDMS)

csdms.colorado.edu

http://csdms.colorado.edu

http://csdms.colorado.edu/wiki/Model_download_portal

CSDMS: 1296 Members, 5 Working Groups, 6 Focus Research Groups

Some Key Concepts and Terms

Model: State variables for some system of interest are discretized in space (on a computational grid) and new values are computed from previous values by marching forward in time according to a set of rules (e.g. laws of physics).

Model Component: A model that has been specially prepared for plug-and-play reusability. (i.e. a standard interface & compiled for language interoperability).

Interface: A standardized set of functions, which a caller uses to interact with a resource such as a model, database or service. Heterogeneous resources wrapped with a standard interface then operate in a “familiar” way.

Modeling Framework: A software container in which pre-compiled model components are instantiated, configured and dynamically linked to rapidly create a customized model. (With a “birds-eye view” of all components.)

Workflow: Software that allows a chain of steps to be saved, modified and re-executed. Each step uses an application to create an intermediate product that is passed along to the next step, often as a file.

Linking Component-based Models:How Can Two Models Differ?

• Programming language

(C, C++, Fortran, Java, Python, etc.)

Solution: Babel and Bocca (CCA toolchain)

• Computational grid

(triangles, rectangles, Voronoi, points, etc.)

Solution: ESMF regridder (parallel, spatial interpol.)

• Timestepping scheme

(fixed, adaptive, local)

Solution: Temporal interpolation tool

• Variable names

Need some means of “semantic mediation”

Solution: CSDMS Standard Names

• Variable units

Solution: UDUNITS (Unidata)

Language Interoperability with Babel

Language interoperability

is a powerful feature of the

CSDMS framework.

Components written in

different languages can be

rapidly linked in HPC

applications with hardly any

performance cost. This

allows us to “shop” for

open-source solutions (e.g.

libraries), gives us access

to both procedural and

object-oriented strategies

(legacy and modern code),

and allows us to add

graphics & GUIs at will.

CSDMS Regridding Tools

• ESMF Regrid (multi-proc., Fortran)• Custom-built regridders

The Basic Model Interface (BMI)

http://csdms.colorado.edu/wiki/BMI_Description

https://github.com/csdms/bmi/blob/master/bmi.sidl

Taming Heterogeneity with Interfaces

Before:Each resource is unique.

Own ways of doing things.

Respond differently.

Can become unstable.

Difficult to control.

After:Uniform outward appearance.

Respond to same commands.

Interchangeable units.

Have a chain of command.

Work as a team.

The Basic Model Interface (BMI)

BMI requires a developer to make some relatively simple, noninvasive,

and framework-independent changes to his/her model source code,

mostly by adding some new functions. Functions can be grouped into:

Model Control Functions (initialize, update, finalize)

Model Information Functions (e.g. time-stepping scheme)

Variable Information Functions (e.g. dimensions, data type, units)

Variable Getters and Setters

Grid Information Functions

These functions make the model self-describing (so the framework can

“see” and reconcile model differences) and fully controllable.

CSDMS can automatically wrap BMI-enabled models to provide them

with a CMI interface which allows them to be used in the CSDMS

framework and to gain other new capabilities, including NetCDF output.

Follows the CS “Hollywood Principle”: Don’t call us, we’ll call you.

How BMI and CMI Work

Model Coupling Metadata and the

CSDMS Standard Names

http://csdms.colorado.edu/wiki/CSDMS_Standard_Names

http://csdms.colorado.edu/wiki/CSN_Metadata_Names

Semantic Matching for Model Variables

Hydro Model A

Output variables:

• streamflow• rainrate

Hydro Model B

Input variables:

• discharge• precip_rate

CSDMS Standard Names

• channel_exit_water_x-section__volume_flow_rate

• atmosphere_water__rainfall_volume_flux

Goal: Remove ambiguity so that

the framework can automatically

match outputs to inputs.

The CSDMS Standard NamesData Models like RDF and EAV use triples like:

Subject + Predicate + Object, and

Entity/Object + Attribute + Value (object-oriented)

CSDMS Standard Names use a similar pattern for creating

unambiguous and easily understood standard variable names or

“preferred labels” according to a set of rules. These are then

used to retrieve values and metadata. The pattern is:

Object name + [Operation name] + Quantity name

Examples:

atmosphere_carbon-dioxide__partial_pressure

atmosphere_water__rainfall_volume_flux

earth_ellipsoid__equatorial_radius

soil__saturated_hydraulic_conductivity

We have also started building a set of standard Attribute and Process Names.

The CSDMS Standard Names

Main Page: csdms.colorado.edu/wiki/CSDMS_Standard_Names

Basic Rules: csdms.colorado.edu/wiki/CSN_Basic_Rules

Object Names: csdms.colorado.edu/wiki/CSN_Object_Templates

Operation Names: csdms.colorado.edu/wiki/CSN_Operation_Templates

Quantity Names: csdms.colorado.edu/wiki/CSN_Quantity_Templates

Process Names: csdms.colorado.edu/wiki/CSN_Process_Names

Assumption Names: csdms.colorado.edu/wiki/CSN_Assumption_Names

Metadata Names: csdms.colorado.edu/wiki/CSN_Metadata_Names

Model Metadata Files: csdms.colorado.edu/wiki/CSN_MMF_Example

The CSDMS Standard Names can be viewed as a lingua franca that provides a

bridge for mapping variable names between models. They play an important role

in the Basic Model Interface (BMI). Model developers are asked to provide a BMI

interface that includes a mapping of their model's internal variable names to

CSDMS Standard Names and a Model Coupling Metadata (MCM) file that

provides model assumptions and other information.

IMPORTANT: Model developers continue to use whatever variable names they

want to in their code, but then "map" each of their internal variable names to the

appropriate CSDMS standard name in their BMI implementation.

NSF EarthCube Building Block:Earth System Bridge Project

www.earthsystemcog.org/projects/earthsystembridge

Earth System Bridge, CoG page

http://www.earthsystemcog.org/projects/earthsystembridge

ES - Framework Description Language

Earth System Bridge – BMI Adapters

BMI

CSDMS

ESMF

Pyre

OMS

OpenMI

Adapters will allow a BMI-enabled model to be used

in any component-based modeling framework.

Standard Name Crosswalks, Etc.

Develop crosswalks between the CSDMS Standard Names

(2573 names) and the:

• CUAHSI VariableName CV (786 names, 50%)

• CF Convention Standard Names (2630 names, 60%)

tendency_of_mass_concentration_of_ammonia_in_air

atmosphere_air_ammonia__time_derivative_of_mass_concentration

Work with deep Earth process modeling community to

develop new CSDMS Standard Names (PSU meeting)

Work with hydrologic modeling community (CUAHSI) to

further develop CSDMS Standard Names (NFIE mtg)

Earth System Bridge Demonstration Projects

Demonstrate the new capabilities provided by Earth System

Bridge technology by using them to couple

operational/federal atmosphere-ocean models at the global-

to-regional scale to academic (or operational)

hydrologic/land/coastal models at the regional-to-local scale.

• ESMF: Couple the MPIPOM-TC ocean model to a local-

scale hydrologic or inundation model. (MPIPOM-TC =

Message Passing Interface Princeton Ocean Model for

Tropical Cyclones)

• CSDMS: Complete implementation and testing of a Basic

Model Interface (BMI) for WRF model (Weather Research

and Forecasting) and link it within the CSDMS framework

to the local-scale, spatial hydrologic TopoFlow model.

• Peckham, S.D., E.W.H. Hutton and B. Norris (2013) A component-based approach to integrated modeling in the geosciences: The Design of CSDMS, Computers & Geosciences, special issue: Modeling for Environmental Change, 53, 3-12 .

• Peckham, S.D. (2014) The CSDMS Standard Names: Cross-domain naming conventions for describing process models, data sets and their associated variables, Proceedings of the 7th Intl. Congress on Env. Modelling and Software, International Environmental Modelling and Software Society (iEMSs), San Diego, CA. (Eds. D.P. Ames, N.W.T. Quinn, A.E. Rizzoli).

• Peckham, S.D. (2014) EMELI 1.0: An experimental smart modeling framework for automatic coupling of self-describing models, Proceedings of HIC 2014, 11th International Conf. on Hydroinformatics, New York, NY.

For More Information

Reconciling Differences with Standards

If we reconcile differences between

the resources in a pairwise manner,

the amount of work, etc. grows fast:

Cost(N) = N (N-1) / 2 ~ N2.

vs.

Introduce a new, generic or

standard representation (the

“hub”), then map resources to

and from it. The amount of work,

maintenance, etc. drops to:

Cost(N) = N.

Model Coupling Metadata

Models are based on a number of real-world objects, e.g.

atmosphere, river channel. (Think object-oriented.)

There are “assumptions” that apply to the model, its

computational grid, its objects and the attributes/quantities

that describe those objects.

MCM uses the CSDMS Standard Assumption Names and

the CSDMS Standard Variable Names. The assumption

names are used to “decorate” or “tag” other model entitities

(above).

Building a Modeling Framework

CSDMS has integrated a variety of powerful, open-source tools to build its modeling framework, such as:

Babel – Language interoperability (C,C++,Java,Python,Fortran)Bocca – Component preparation and project managementCcaffeine – Low-level model coupling (parallel environ.)ESMF Regrid – Multi-processor spatial regriddingOpenMI Regrid – Single-processor spatial regriddingNetCDF – Scientific data format (self-describing, etc.)VisIt – Visualization of large data sets (multi-proc.)

We developed our own interface standards, BMI & CMI, and greatly extended the original Ccaffeine GUI to create our CSDMS Web Modeling Tool for interactive model coupling.

Some Key Concepts and Terms

Model: State variables for some system of interest are discretized in space (on a computational grid) and new values are computed from previous values by marching forward in time according to a set of rules (e.g. laws of physics).

Model Component: A model that has been specially prepared for plug-and-play reusability. (i.e. a standard interface & compiled for language interoperability).

Interface: A standardized set of functions, which a caller uses to interact with a resource such as a model, database or service. Heterogeneous resources wrapped with a standard interface then operate in a “familiar” way.

Modeling Framework: A software container in which pre-compiled model components are instantiated, configured and dynamically linked to rapidly create a customized model. (With a “birds-eye view” of all components.)

Workflow: Software that allows a chain of steps to be saved, modified and re-executed. Each step uses an application to create an intermediate product that is passed along to the next step, often as a file.

How Can a Model be Converted to aReusable, Plug-and-Play Component?

CSDMS has developed two model interfaces, one called Basic Model

Interface (BMI) and one called Component Model Interface (CMI) for

this purpose.

BMI requires a developer to make some relatively simple, noninvasive,

and framework-independent changes to his/her model source code,

mostly by adding some new functions. These provide a caller with 3 key

things: (1) fine-grained control (i.e. IUF), (2) variable getters and setters

and (3) model metadata. The model can still be used in “stand-alone

mode,” just as before.

CSDMS automatically wraps BMI-enabled models to provide them with

a CMI interface. CMI makes function calls to the (1) framework, (2)

service components (the “reconcilers”), (3) the BMI of the wrapped

model and (4) the CMI of other plug-and-play components. CMI gives a

model many new capabilities, including NetCDF output.

Metadata for Model:TopoFlow – Meteorology

Author Scott D. Peckham

Email Scott.Peckham@colorado.edu

Domains Hydrology, meteorology

Language Python

License MIT

Purpose TopoFlow is a spatially-distributed hydrologic model that consists of many stand-alone components. This component provides meteorology variables, such as precipitation rate, temperature, relative humidity, and shortwave and longwave radiation.

Goto Model Objects Page

MCM Tool

A Prototype or

mock-up of a smart

phone app based on

Model Coupling

Metadata (MCM) for

describing a model.

(under development)

Objects for Model:TopoFlow - Meteorology

atmosphere_aerosol_dust

atmosphere_air-column_water~vapor

atmosphere_bottom_air

atmosphere_bottom_air_flow

atmosphere_bottom_air_land_heat~net~latent

atmosphere_bottom_air_land_heat~net~sensible

atmosphere_bottom_air_water~vapor

atmosphere_water

earth

land_surface

model_grid

physics

snowpack

water

Quantities for Object:land_surface

Quantity SI Units Grid Type

albedo 1 G1 config, output

aspect_angle radians G1 config, output

emissivity 1 G2 output

geodetic_latitude degrees G1 config, output

longitude degrees G1 config, output

slope_angle radians G1 output

temperature deg_C G2 output

+ +

Q AAO

Assumptions for Quantity:aspect_angle

Sign_and_Angle_Conventions

counter-clockwise_from_east

+

Objects for Model:TopoFlow - Meteorology

atmosphere_aerosol_dust

atmosphere_air-column_water~vapor

atmosphere_bottom_air

atmosphere_bottom_air_flow

atmosphere_bottom_air_land_heat~net~latent

atmosphere_bottom_air_land_heat~net~sensib

atmosphere_bottom_air_water~vapor

atmosphere_water

earth

land_surface

model_grid

physics

snowpack

water

+ Assumptions for Model:TopoFlow - Meteorology

Equations

some_named_equation

+

AO AO

The BMI Breakthrough: For Modelers

• Requires minimal effort.

• Is noninvasive (adds no dependencies on CSDMS data.

structures or code. No interference with developer’s design.

• Allows model to still be used as before in a stand-alone mode.

• Requires no new code intended to accommodate the needs of

other models (unlike OpenMI 1.4).

• Is framework agnostic and requires no modeling-framework

specific knowledge (e.g. the CCA concept of ports);

developers do not “code to” the framework.

• Only requires developer to do things that would be

necessary for the model to be used in any modeling

framework.

• Provides added value, e.g. ability to couple to other models

and to write output variables to standardized NetCDF files.

BMI Breakthrough: For SE at CSDMS

• One “CMI wrapper” (code) can be used for any BMI-enabled

model, instead of unique wrapper code for each model.

• Make it possible to largely automate the conversion of

contributed process models to CSDMS components.

• Dramatically reduces code maintenance time.

• Addresses the “frozen code” problem. i.e. minimal

additional effort to bring a new version of the same model

(e.g. with enhancements or bug fixes) into the framework.

• Minimal impact on performance of the model.

• Makes it possible for the CSDMS framework to

automatically accommodate differences between models

by calling service components when needed.

• The SE no longer needs to learn all about each model.

Earth System – Framework

Description Language (ES - FDL)

https://earthsystemcog.org/projects/es-fdl/

Metadata for Model:TopoFlow – Meteorology

Author Scott D. Peckham

Email Scott.Peckham@colorado.edu

Domains Hydrology, meteorology

Language Python

License MIT

Purpose TopoFlow is a spatially-distributed hydrologic model that consists of many stand-alone components. This component provides meteorology variables, such as precipitation rate, temperature, relative humidity, and shortwave and longwave radiation.

Goto Model Objects Page

MCM Tool

A prototype or

mock-up of a smart

phone app based on

Model Coupling

Metadata (MCM) for

describing a model.

(under development)

Standard Assumption Names

Assumption Type: Example

Boundary conditions: no_slip_boundary_conditionConserved quantities: momentum_conservedCoordinate system: cartesian_coordinate_systemAngle conventions: clockwise_from_north_conventionDimensionality: 2_dimensionalEquations used: navier_stokes_equationClosures: eddy_viscosity_turbulence_closureFlow-type assumptions: laminar_flowFluid-type assumptions: herschel_bulkley_fluidGeometry assumptions: trapezoid_shapedNamed model assumptions: green_ampt_infiltration_modelThermodynamic processes: isenthalpic_processApproximations: boussinesq_approximationAveraging methods: reynolds_averagedNumerical methods used: arakawa_c_gridState of matter: liquid_phase