HDF5 OPeNDAP ProjectUpdate and Demo

MuQun Yang and Hyo-Kyung Lee (The HDF Group)

James Gallagher (OPeNDAP, Inc.)

1HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

OPeNDAP Review

• A software framework that allows simple access to remote data

• Data Access Protocol• Client-server model• Widely used by Earth Science Community

2HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

OPeNDAP client library

???

DAP 2.0/4.0

dap server(e.g. Apache)

OPeNDAP HDF/NetCDF

server

Visualization clients

(e.g. GrADS)

HDF/NetCDFView HDF/NetCDF

OPeNDAP Concept

3HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Example Usage

OPeNDAP Client

OPeNDAPHDF4 Server

HDF4

Syntactic Structure of DataSemantic Meaning of Data

Actual Content of Data

OPeNDAP Visualization Client4

HDF and HDF-EOS Workshop XII, Aurora, CO10/17/2008

Why Important?

• HDF5: NASA EOS / NPOESS

• NASA’s MISSION: ACCESS!

• Our MISSION: Build a DAP-HDF5 Bridge

5HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

OPeNDAP HDF5 Server

1. Map HDF5 Datatypes into DAP Datatypes

2. Plus, tweaks for OPeNDAP Clients to access HDF-EOS5 Files

Goal: Access HDF5 data via OPeNDAP

6HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Mapping HDF5 Datatypes to DAP• Compound Datatype

• Support for Group

• Object/regional References

7HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Some OPeNDAP Clients

• Cannot handle long variable names.

• Need special attributes on dataset.

• Handle only DAP Grid datatype.

• Handle only well-formed attributes.

Not all OPeNDAP clients are created equal!

8HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Tweaks for OPeNDAP Clients Two configuration options

• --enable-short-name

• --enable-CF

9HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

HDF EOS

• Grid with No Dimension data Clients expect Grid with Dimension data

• Some Attributes stored in Extremely Long String(s) (e.g. StructMetada.0) Clients expect well-structured attributes

Not all HDF5 files are created for DAP!

10HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Tweaks for HDF-EOS Two more configuration options

• --enable-eos-grid

• --enable-eos-meta

11HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Day After Server Tweaks

• Finally, Happy Clients!

GrADSNCLFerretMATLAB

ncBrowse

IDVODC

IDL

12HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Some Things

• Hard: HDF5 Opaque, Bitmap, Enum, Variable Length types

• Illegal: Reserved characters in DAP are used in Dataset/Group name in HDF5

(that OPeNDAP HDF5 server can’t serve)

13HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Goals

•1st : Access HDF5 via OPeNDAPThe OPeNDAP HDF5 handler was released on March, 2008 at OPeNDAP website http://www.opendap.org/

• 2nd: Construct HDF5 data from OPeNDAP Bonus: Help clients to better access EOS

Swath data 14

HDF and HDF-EOS Workshop XII, Aurora, CO10/17/2008

HDF5-Friendly OPeNDAP client library

15HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Example: Group in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.

HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5

16HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Example: Reference in HDF5

• Object / Regional Reference

• Map to DAP URL

• Important for NPOESS

17HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

• Only ODC can display Swath properly.• One more reason: Help Clients to view Swath

HDF5-Friendly OPeNDAP Client Library

• Easy but Rigid OPeNDAP NC Client Library! easy: same as netCDF C interface rigid: follows netCDF model

• Our client library must be Easy but Flexible

No Latitude and Longitude

Courtesy of NASA

18HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

libnc-dap

???

liboc-dap

NC-Friendly OPeNDAP Client Library

libdap

dapserver

OPeNDAP NC Server

GrADS

NetCDFView NetCDFHDF5

Group/Ref/Swath

OPeNDAP HDF5 Server

View HDF5

GrADS

View G/R/Swath

HDF5-Friendly OPeNDAP Client Library

19HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Summary of Client Library Prototype

• Finished coding • Tested with GrADS visualization client• Documented • Working on a demo DAP to HDF5 tool via

the client library prototype• May test with NCL

10/17/2008HDF and HDF-EOS Workshop XII, Aurora,

CO 20

Example: dap2h5

• A test application for the client library

• It can construct HDF5 from DAP output

ORIGINAL

CONSTRUCTED

21HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Demo: MLS swath via GrADS

• GrADS is modified to use our library

• The client library provides grid mapping

from swath data

• GrADS displays swath through HDF-EOS5

specific client library API calls

22HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Caution

• Our Client Library is ONLY PROTOTYPE! It does NOT support all DAP data types

It does NOT support all AURA files It does NOT support all Visualization clients

23HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Summary

• HDF5 Access via OPeNDAP Is Easy Is used by GES DISC to serve Aura files

• HDF5-Friendly OPeNDAP Client Library Prototype Serves HDF5 better(EOS swath) Benefits visualization clients

24HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Future Work• HDF5 to DAP2 Mapping Document

• Release HDF5-friendly OPeNDAP Client Library Prototype

• URL: http://hdfgroup.org/projects/opendap/index.html

25HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Credits

Mike Folk (THG)Robert McGrath (NCSA)

Peter Leonard, Daniel Kahn, Marghi Hopkins (ADNET)Christopher Lynnes, James Johnson, Denis Nadeau, Fan Fang (NASA)

Jennifer Adams (GrADS)Dave Brown (UCAR)

26HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008

Acknowledgement

This work was supported basing upon the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX06AG75A. .  Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA. 

27HDF and HDF-EOS Workshop XII, Aurora,

CO10/17/2008