Development of an Integrated Earth System Climate Modeling Environment

Purdue University/RCACNCAR/ESG

NOAA/CIRES

XSEDE Science Gateway 2

A Great Team!

Purdue/RCAC Carol X. Song, Lan Zhao, Chris Thompson, Robert Campbell, Jungha Woo, Han Zhang, Madhavan Lakshminarayanan

NCARDon Middleton, Nathan Wilhelmi, Eric Nienhouse, Michael Burek, Rick Brownrigg

NOAA/CIRESKathy Saint (SGI), Cecelia DeLuca, Sylvia Murphy

“Developing an integrated end-to-end TeraGrid climate modeling environment”, TeraGrid 2011 conference, Salt Lake City, UT, July 2011.

05/11/2012

XSEDE Science Gateway 3

Background• Motivation for a climate modeling environment

CCSM/CESM Resource Expertise A collaborative online environment Existing synergistic community efforts

Earth System Grid, Earth System Curator, ESMF, METAFOR Common Information Model and Purdue CCSM3 portal

• Project goals Provide an integrated environment for climate modeling,

data/metadata archival, visualization and analysis Support research and education, lower barriers to using

climate models05/11/2012

XSEDE Science Gateway 4

System Architecture

Create Case

Configure Case Submit Case

Authentication/ Authorization

Track Status

Post-processDebugging

Transfer Files

Publish Metadata

ESG Data Publisher

iRODSScratch Storage

Token Mgr

Data/Metadata

JobsOutput

CESM Web Services

User requests

Job Management

CESM portalESG gateway

Output

TG MyProxy

Account DB

Publish Data

XD community account/Individual account

SOAP and Restful XML/JSON

05/11/2012

CESM Portal - Model Workflow• Insulate users from

cluster Hide command line

details Integrate CESM 1.0 +

metadata capabilities + ESG publishing + NCL

Web services (SOAP, Restful XML/JSON)

• Manage user’s experience Maintain job status

and metrics End-to-end modeling

from simulation to data/metadata publishing

Bind independent processes into unified workflow

CESM Portal

JBOSSJob Scripts Publish Scripts

SOAP

Create case

Configure case

Compile case

Execute case

CESM-WS/ gateway

Create

Configure

Compile

Execute

Query Status

CESM Model

create_case

./configure

<case>.compile

<case>.submit

Data

Post-processing Post-

processing EnginesAMWG

NCL scripts

…

Configure post-process

Run post-process

ESG Publicationworkflow

5XSEDE Science Gateway05/11/2012

XSEDE Science Gateway 6

ESMF Attributes and Self-describing Models

• Metadata is increasingly important in Earth science modeling Understand multi-component models with many options and parameters Distinguish simulations from one another Archive simulations with full descriptions for preservation and defensibility Enable run reproducibility

• ESMF Attributes store, aggregate, and output metadata Structured as name-value pairs Arranged in packages by purpose and convention:

International Standards Organization (ISO), Climate and Forecast (CF), METAFOR CommonInformation Model (CIM), etc.

Output in XML and other formats• ESMF Attributes in CESM

CESM components (land, atm, ice, ocean, sea ice) and driver are being instrumented with Attributes

Information captured includes component level, field level, andsystem level metadata

05/11/2012

XSEDE Science Gateway 7

CESM Metadata in the ESG Gateway

05/11/2012

Screenshot of CESM metadataImplemented in the model

by ESMF and Curator.

Exported by CESM at run time.

Ingested and displayed by ESG.

Uploaded to Purdue’s atom feed.

XSEDE Science Gateway 8

Data Discovery, Access and Analysis

• ESG: Science Gateway for Earth System Data Access Data discovery and access across federation of data nodes. Identity services and project specific access control. Data publishing services. Versioning and provenance support. Bulk data download, wget scripts. Data use tracking and metrics. Model documentation metadata browsing. ATOM CIM feed and ESG ingest.

• NCL Backend to Product Services Integrated 64-bit NCL visualization engine. Big data capable analysis functions. ParNCL as a drop-in replacement, for parallel computation. Plug-in component to NOAA/PMEL LAS Product Server.

05/11/2012

XSEDE Science Gateway 9

Demo

05/11/2012

XSEDE Science Gateway 10

Demo

05/11/2012

XSEDE Science Gateway 11

Demo

05/11/2012

XSEDE Science Gateway 12

Next Steps• Transition into XD

Release CESM portal Work with faculty and instructors to use in teaching Need help in connecting with users

• Federated authentication/identity between CESM and ESG gateways

• Making a rich set of tools for analysis and visualization available to CESM gateway users

• Add other resources to backend for better user experience

05/11/2012