Data for Climate and Energy Studies

Steven WorleyComputational and Information Systems Laboratory

NCAR

NCAR-CSM Symposium on Climate and Energy 2

Topics

Scope of the NCAR Research Data Archive (RDA)

Discovery and Access Highlights

User ranked popular datasets

Examples

Near-term service improvements

7 May 2010

NCAR-CSM Symposium on Climate and Energy 3

Scope of the NCAR Research Data Archive (RDA)

Focus on atmospheric, oceanographic, and related geo-sciences observational data and derived analyses.

Some weather forecast data Do not specialize in climate prediction datasets

7 May 2010

Active stewardship program to maintain and grow the RDA for 40+ years.

Large variety, 600+ datasets, ~ 400 TB, 4M files

NCAR-CSM Symposium on Climate and Energy 4

Discovery and Access Highlights

7 May 2010

Primary design feature for web portal• Data Discovery – Find Data!

NCAR-CSM Symposium on Climate and Energy 5

Discovery and Access Highlights

7 May 2010

Multiple Methods - simple to interoperable

1. Find the files in our lists and download• Through your browser – limit 2GB• We create a ‘wget’ script for you – run in background on your

machine – no limit

2. You select temporal, spatial, parameter domains• We build a file list for you• Download options as in 1

3. Data is not online to the web – but, is on archive storage• We automatically stage data to online, then download

4. You select temporal, spatial, parameter domains - we build CURL commands - you get only the grids you select• About CURL

• Client URL Library functions• Readily available on Linux OS• We use HTPPS protocols – others are available• Applies well to WMO GRIB data format

• Users modify the CURL commands and script them to perform routine data extractions from RDA

NCAR-CSM Symposium on Climate and Energy 6

User ranked popular datasets

7 May 2010

Unique users FY09 datasets Titles2878 ds082.0, ds083.2, ds083.0 NCEP FNL Operational Model Global Tropospheric Analyses924 ds090.0 NCEP/NCAR Global Reanalysis Products510 ds758.0, ds759.3, ds759.2 NGDC Global 2' and 5' Elevations, USGS 30 ARC-second

477ds461.0, ds351.0ds337.0, ds464.0,ds353.4 NCEP ADP/PREPBUFR Global Surface and Upper Air Observations

358 ds608.0 NCEP North American Regional Reanalysis (NARR)264 ds609.2 GCIP NCEP ETA model output262 ds540.1, ds540.0 International Comprehensive Ocean-Atmosphere Data Set (ICOADS)190 ds744.4 QSCAT/NCEP Blended Ocean Winds 173 ds277.0 NCEP V2.0 OI Global SST, V3.0 Extended Reconstructed Analyses153 ds335.0, ds336.0 Unidata (IDD) Observations and Model Data106 ds091.0 NCEP/DOE Reanalysis II106 ds552.1, ds552.0, ds556.0 River Discharge Data91 ds277.3 Hadley Centre Global Sea Ice and Sea Surface Temperature (HadISST)89 ds824.1, ds330.3 Global Tropical Cyclone "Best Track" Position and Intensity Data, TIGGE Cyclone Tracks 72 ds570.0 World Monthly Surface Station Climatology69 ds314.0 Global Meteorological Forcing Dataset for Land Surface Modeling68 ds900.0 U.S. AFGWC Station (Surface and Upper Air) Library61 ds260.3 NOCS Surface Flux Dataset v2.058 ds285.3 Japanese Subsurface Temperature And Salinity Analyses V6.756 ds512.0 CPC Global Summary of Day/Month Observations56 ds625.0 Japanese 25-year Reanalysis Project55 ds578.1, ds485.0 China Monthly Station Precipitation and Temperature, Daily Precip. and Monthly Soil Temperature53 ds285.0 World Ocean Database and World Ocean Atlas47 ds770.0 GISS Soil and Surface Slope

45 ds215.0Global Monthly Surface Temperature Anomalies (1856-2005), Precipitation (1900-1998), and Sea Level Pressure (1873-2000) from the University of East Anglia Climatic Research Unit

42 ds277.7 NOAA OI 1/4 Degree Daily SST Analysis42 ds330.2 TIGGE Near Real-time40 ds472.0 TDL U.S. and Canada Surface Hourly Observations36 ds232.2 Scatterometer Climatology of Ocean Winds32 ds131.1, ds131.0 NOAA-CIRES Twentieth Century Global Reanalysis Version I and II30 ds260.2 CORE.2 Global Air-Sea Flux Dataset27 ds885.1 NCDC TD9640 U.S. Palmer Drought Indices25 ds627.0 ERA-Interim Project25 ds510.0 NCDC TD3200 U.S. Cooperative Summary of Day24 ds564.0 Global Historical Climatology Network (GHCN) Temperature, Precipitation, Pressure

5921 All Datasets All DSS datasets

Top 30 datasets/groups FY09

~ 6000 Unique Users Annually

NCAR-CSM Symposium on Climate and Energy 7

One example

Final Global Analysis from NOAA/NCEP 4x Daily Updated in the RDA 1x/day 1° horizontal resolution 26 vertical pressure levels, plus surface Series starts in 1999 Over 55 parameter fields

7 May 2010

NCAR-CSM Symposium on Climate and Energy 8

One example

7 May 2010

NCAR-CSM Symposium on Climate and Energy 9

Re-analyses

7 May 2010

Table 1: Global atmospheric and oceanographic re-analyses are one of many valuable data resources provided by external organizations that employ the expertise of RDA consultants and are the most recent major reanalyses available in the Research Data Archive. Most time periods are ongoing, that is, providers continue to produce the products gong forward in time. In general, all reanalyses also have lower temporal and horizontal resolutions than those shown above. Most reanalyses also have variables on vertical model coordinate levels, as well as large numbers of surface specific fields, and vertically integrate values.

http://www.earthobservations.org/documents/geonewsletter/art008001_trenberth_article.pdf

NCAR-CSM Symposium on Climate and Energy 10

Near-term service improvements

Current and soon-to-be workflow

7 May 2010

Complete User CommunityAdvantages:

Fast access to online data – limited part of RDA

Access to all RDA content metadata

Access to RDA data processing services

Complete User CommunityDisadvantages:

Slow access to MSS data – delayed mode

Have to create a separate RDA account and log in

Data processing requests take a long time to finish

Slow download speeds for some users

HPC User CommunityAdvantages:

Access to full RDA Fast computing No login required

HPC User CommunityDisadvantages:

No access to online data Use MSS as a file server No direct access to RDA

metadata No direct access to RDA

data processing services Require separate account

to access RDA web server

HPC User CommunityImprovements:

Fast access to full RDA Access to all RDA content

metadata Access to RDA data

processing services Single CISL account Single “first point of contact”

Complete User CommunityImprovements:

Fast access to full RDA Expanded data processing

services available Single CISL account - no

separate RDA account Faster download speeds –

grid-based tools, e.g. GRIDFTP

Single “first point of contact” for user support

Resolved all the disadvantagesNew Challenges: GPFS and HPSS don’t have generic

file use logging • Need for metrics & services

HPSS doesn’t have sophisticated file access control• Some RDA assets have limited

access policies Abandon a functional RDA

registration system – retool a 20K+ user DB

Of course, there will be more!

Big transition while maintaining RDA content building and services

NCAR-CSM Symposium on Climate and Energy 13

End

Scope of the NCAR Research Data Archive (RDA) Discovery and Access Highlights User ranked popular datasets Examples Near-term service improvements

http://dss.ucar.edu/

7 May 2010