Earth System Bridge:Spanning Scientific Communities

with Interoperable Modeling Frameworks

Scott D. PeckhamSenior Research Scientist at INSTAAR

Lead PI for Earth System BridgeFormer Chief Software Architect for CSDMS

University of Colorado, BoulderJuly 15, 2015

Community Surface Dynamics Modeling System (CSDMS)

csdms.colorado.edu

http://csdms.colorado.edu

http://csdms.colorado.edu/wiki/Model_download_portal

CSDMS: 1323 Members, 5 Working Groups, 6 Focus Research Groups

Some Key Concepts and TermsModel: State variables for some system of interest are discretized in space (on a computational grid) and new values are computed from previous values by marching forward in time according to a set of rules (e.g. laws of physics).

Model Component: A model that has been specially prepared for plug-and-play reusability. (i.e. a standard interface & compiled for language interoperability).

Interface: A standardized set of functions, which a caller uses to interact with a resource such as a model, database or service. Heterogeneous resources wrapped with a standard interface then operate in a familiar way.

Modeling Framework: A software container in which pre-compiled model components are instantiated, configured and dynamically linked to rapidly create a customized model. (With a birds-eye view of all components.)

Workflow: Software that allows a chain of steps to be saved, modified and re-executed. Each step uses an application to create an intermediate product that is passed along to the next step, often as a file.

Linking Component-based Models:How Can Two Models Differ?

Programming language(C, C++, Fortran, Java, Python, etc.)Solution: Babel and Bocca (CCA toolchain)

Computational grid (triangles, rectangles, Voronoi, points, etc.)Solution: ESMF regridder (parallel, spatial interpol.)

Timestepping scheme(fixed, adaptive, local)Solution: Temporal interpolation tool

Variable namesNeed some means of semantic mediation

Solution: CSDMS Standard Names Variable units

Solution: UDUNITS (Unidata)

Language Interoperability with Babel

Language interoperabilityis a powerful feature of the CSDMS framework.Components written in different languages can be rapidly linked in HPCapplications with hardly any performance cost. This allows us to shop foropen-source solutions (e.g. libraries), gives us access to both procedural andobject-oriented strategies (legacy and modern code), and allows us to addgraphics & GUIs at will.

CSDMS Regridding Tools ESMF Regrid (multi-proc., Fortran) Custom-built regridders

The Basic Model Interface (BMI)

http://csdms.colorado.edu/wiki/BMI_Descriptionhttps://github.com/csdms/bmi/blob/master/bmi.sidl

Taming Heterogeneity with Interfaces

Before:Each resource is unique.Own ways of doing things.Respond differently.Can become unstable.Difficult to control.

After:Uniform outward appearance.Respond to same commands.Interchangeable units.Have a chain of command.Work as a team.

The Basic Model Interface (BMI)BMI requires a developer to make some relatively simple, noninvasive, and framework-independent changes to his/her model source code, mostly by adding some new functions. Functions can be grouped into:

Model Control Functions (initialize, update, finalize)Model Information Functions (e.g. time-stepping scheme)Variable Information Functions (e.g. dimensions, data type, units)Variable Getters and SettersGrid Information Functions

These functions make the model self-describing (so the framework can see and reconcile model differences) and fully controllable.

CSDMS can automatically wrap BMI-enabled models to provide them with a CMI interface which allows them to be used in the CSDMS framework and to gain other new capabilities, including NetCDF output.

Follows the CS Hollywood Principle: Dont call us, well call you.

How BMI and CMI Work

Model Coupling Metadata and theCSDMS Standard Names

http://csdms.colorado.edu/wiki/CSDMS_Standard_Nameshttp://csdms.colorado.edu/wiki/CSN_Metadata_Names

Semantic Matching for Model Variables

Hydro Model A

Output variables:

streamflow rainrate

Hydro Model B

Input variables:

discharge precip_rate

CSDMS Standard Names

channel_exit_water_x-section__volume_flow_rate

atmosphere_water__rainfall_volume_flux

Goal: Remove ambiguity so thatthe framework can automaticallymatch outputs to inputs.

The CSDMS Standard NamesData Models like RDF and EAV use triples like:

Subject + Predicate + Object, andEntity/Object + Attribute + Value (object-oriented)

CSDMS Standard Names use a similar pattern for creating unambiguous and easily understood standard variable names or preferred labels according to a set of rules. These are then used to retrieve values and metadata. The pattern is:

Object name + [Operation name] + Quantity name

Examples:atmosphere_carbon-dioxide__partial_pressureatmosphere_water__rainfall_volume_fluxearth_ellipsoid__equatorial_radiussoil__saturated_hydraulic_conductivity

We have also started building a set of standard Attribute and Process Names.

The CSDMS Standard Names

Main Page: csdms.colorado.edu/wiki/CSDMS_Standard_NamesBasic Rules: csdms.colorado.edu/wiki/CSN_Basic_RulesObject Names: csdms.colorado.edu/wiki/CSN_Object_TemplatesOperation Names: csdms.colorado.edu/wiki/CSN_Operation_TemplatesQuantity Names: csdms.colorado.edu/wiki/CSN_Quantity_TemplatesProcess Names: csdms.colorado.edu/wiki/CSN_Process_NamesAssumption Names: csdms.colorado.edu/wiki/CSN_Assumption_NamesMetadata Names: csdms.colorado.edu/wiki/CSN_Metadata_NamesModel Metadata Files: csdms.colorado.edu/wiki/CSN_MMF_Example

The CSDMS Standard Names can be viewed as a lingua franca that provides a bridge for mapping variable names between models. They play an important role in the Basic Model Interface (BMI). Model developers are asked to provide a BMI interface that includes a mapping of their model's internal variable names to CSDMS Standard Names and a Model Coupling Metadata (MCM) file that provides model assumptions and other information.

IMPORTANT: Model developers continue to use whatever variable names they want to in their code, but then "map" each of their internal variable names to the appropriate CSDMS standard name in their BMI implementation.

NSF EarthCube Building Block:Earth System Bridge Project

www.earthsystemcog.org/projects/earthsystembridge

Earth System Bridge, CoG pagehttp://www.earthsystemcog.org/projects/earthsystembridge

ES - Framework Description Language

Basic Model Interface for Models & Data

Heterogeneous Geoscience Models & Spatial, Time-varying Datasets

Model Model Model Model Model Dataset Dataset

Web Service

Standard Name Crosswalks, Etc.

Develop crosswalks between the CSDMS Standard Names (2573 names) and the:

CUAHSI VariableName CV (786 names, 50%) CF Convention Standard Names (2630 names, 60%)

tendency_of_mass_concentration_of_ammonia_in_airatmosphere_air_ammonia__time_derivative_of_mass_concentration

Work with deep Earth process modeling community to develop new CSDMS Standard Names (PSU meeting)

Work with hydrologic modeling community (CUAHSI) tofurther develop CSDMS Standard Names (NFIE mtg)

Earth System Bridge Demonstration Projects

Demonstrate the new capabilities provided by Earth System Bridge technology by using them to couple operational/federal atmosphere-ocean models at the global-to-regional scale to academic (or operational) hydrologic/land/coastal models at the regional-to-local scale.

ESMF: Couple the MPIPOM-TC ocean model to a local-scale hydrologic or inundation model. (MPIPOM-TC = Message Passing Interface Princeton Ocean Model for Tropical Cyclones)

CSDMS: Complete implementation and testing of a Basic Model Interface (BMI) for WRF model (Weather Research and Forecasting) and link it within the CSDMS framework to the local-scale, spatial hydrologic TopoFlow model.

Peckham, S.D., E.W.H. Hutton and B. Norris (2013) A component-based approach to integrated modeling in the geosciences: The Design of CSDMS, Computers & Geosciences, special issue: Modeling for Environmental Change, 53, 3-12 .

Peckham, S.D. (2014) The CSDMS Standard Names: Cross-domain naming conventions for describing process models, data sets and their associated variables, Proceedings of the 7th Intl. Congress on Env. Modelling and Software, International Environmental Modelling and Software Society (iEMSs), San Diego, CA. (Eds. D.P. Ames, N.W.T. Quinn, A.E. Rizzoli).

Peckham, S.D. (2014) EMELI 1.0: An experimental smart modeling framework for automatic coupling of self-describing models, Proceedings of HIC 2014, 11th International Conf. on Hydroinformatics, New York, NY.

For More Information

Composition: Loose vs. Tight Coupling

Loose Coupling(Multi-Machine, Web, File IO, Humans)

Tight Coupling(Single Machine, RAM)

Key Architecture Questions:What is being coupled? (e.g. servers, models) What is the latency?What is being passed between the coupled resources? (e.g. files, variables)How do we reconcile differences between the resources?

Useful Reference: Jeff Dean of Google,Software Engineering Advice from Building Large-Scale Distributed Systems

Earth System Bridge BMI Adapters

BMI

CSDMS

ESMF

Pyre

OMS

OpenMI

Adapters will allow a BMI-enabled model to be usedin any component-based modeling framework.

Earth System Bridge:Spanning Scientific Communitieswith Interoperable Modeling FrameworksCommunity Surface Dynamics Modeling System (CSDMS)csdms.colorado.eduSlide Number 3Slide Number 4Slide Number 5Some Key Concepts and TermsSlide Number 7Slide Number 8CSDMS Regridding ToolsSlide Number 10Taming Heterogeneity with InterfacesSlide Number 12Slide Number 13Slide Number 14Semantic Matching for Model VariablesThe CSDMS Standard NamesThe CSDMS Standard NamesNSF EarthCube Building Block:Earth System Bridge Projectwww.earthsystemcog.org/projects/earthsystembridgeSlide Number 19Slide Number 20ES - Framework Description LanguageSlide Number 22Slide Number 23Slide Number 24For More InformationSlide Number 26Slide Number 27Earth System Bridge BMI Adapters