hdf opendap project update
Post on 30-Dec-2015
52 Views
Preview:
DESCRIPTION
TRANSCRIPT
HDF OPeNDAP Project Update
MuQun Yang and Hyo-Kyung Lee
The HDF Group
March 31, 2009
1Annual briefing to ESDIS04/19/23
Outline
• HDF5 – OPeNDAP work review• HDF4 – OPeNDAP handler progress report• Future work plan
04/19/23 Annual briefing to ESDIS 2
Users
OPeNDAP Clients
IDV
OPeNDAP Servers
HDF4 Files
Hyrax AIRS
DAP
libnc-dap
Visualization Tools
Access data via OPeNDAP
HDF HANDLER
04/19/23 3Annual briefing to ESDIS
HDF5 – OPeNDAP Work
4Annual briefing to ESDIS04/19/23
HDF5-OPeNDAP handler
• Update with new Hyrax• Help GES DISC serve TES and OMI data• Finish HDF5 – DAP2 mapping document and
submit it as a technical note to NASA SPG
04/19/23 Annual briefing to ESDIS 5
04/19/23 Annual briefing to ESDIS 6
HDF5-Friendly OPeNDAP Client Library
• Retrieve the key HDF5 information• Easy to access other HDF5 data (e.g. EOS5
swath)
Example: Groups in HDF5Traditional OPeNDAP client library: It’s an attribute that I don’t understand. I’ll ignore it.
HDF5-Friendly OPeNDAP client library: I was waiting for this key attribute to re-construct HDF5
7Annual briefing to ESDIS04/19/23
04/19/23 Annual briefing to ESDIS 8
MLS swath via GrADS
CO at these elevations is a measure of pollution
lower stratosphere
upper stratosphere
Before
After
CO level near Beijing, China before, during and after Olympics
HDF4 – OPeNDAP Work
9Annual briefing to ESDIS04/19/23
Current HDF4-OPeNDAP handler
• Contributed by JPL in 1998• Part of the OPeNDAP distribution• Visualization client tools cannot display most
HDF-EOS2 and HDF4 data
04/19/23 Annual briefing to ESDIS 10
• Enhance the HDF4-OPeNDAP handler
• More NASA HDF-EOS2 and HDF4 data can be accessed via visualization clients
11Annual briefing to ESDIS04/19/23
Our work
Challenges – more products
12Annual briefing to ESDIS04/19/23
Hundreds of HDF-EOS2 and HDF4 Data products
Challenges – Requirement from Clients
• CF conventions
- Clients expect a certain data structure
- Clients expect some key attributes
13Annual briefing to ESDIS04/19/23
• Various HDF-EOS2 Grid projections• Geographic• Sinusoidal• Polar stereographic• …
04/19/23 Annual briefing to ESDIS 14
Challenges – Grid
Data [4][12]Lon[12]Data [4][8]Lon[4][8]
Geographic
Sinusoidal
• HDF-EOS2 Swath- Dimension map
04/19/23 Annual briefing to ESDIS 15
Challenges - SwathData field has three columnsLongitude field has two columns Longitude should have three columns
Challenges – various products • “Special” HDF-EOS2 files
• Hybrid HDF-EOS2 files- Some HDF4 objects that cannot be read by EOS
library
• HDF4 data products
04/19/23 Annual briefing to ESDIS 16
Enhanced HDF4 Handler
04/19/23 Annual briefing to ESDIS 17
• Follow CF conventions Generate the correct DAP data type. Add shared geo-location information. Insert additional key attributes.
How: Parse Metadata or Use EOS2 library
Implementation choice
• Use Metadata parser or HDF-EOS2 library• Metadata parser
• Pro: simple
• Con: limited products
• HDF-EOS2 library• Pro: maximum products
• Con: install HDF-EOS2 library first
04/19/23 Annual briefing to ESDIS 18
Solution Comparison
19Annual briefing to ESDIS04/19/23
Quick Implementation
• Parse Metadata – hdf5_handler
• Use HDF-EOS2 library – h4toh5
20Annual briefing to ESDIS04/19/23
Current Status
• MODIS / AIRS Grid and Swath
• Other Projections (Polar, Sinusoidal)
• Testing
21Annual briefing to ESDIS04/19/23
HDF-EOS2 in CF via DAP
2204/19/23
AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdfhttp://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10MODIS GridMOD08_D3.A2008284.005.2008289105354.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html
AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/
AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdfMODIS Swath
MOD06_L2.A2001222.0025.004.2003127111841.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.htmlAnnual briefing to ESDIS
More challenges: Missing Attributes
• MODIS Grid – Rain fall
23Annual briefing to ESDIS04/19/23
No Unit!
• AURA OMI Grid - Ozone
More challenges Match dimension names
• Horizontal geolocation information:
- Non-standard names ( e.g. nlon, nlat)
How to match?• Vertical geolocation information:
- Users provide the geolocation name and vaule but no conventions are enforced.
How to know?
04/19/23 Annual briefing to ESDIS 24
More challenges – Missing Values• In a swath file, a data field has three dimensions. But no dimension value for the third dimension.
25Annual briefing to ESDIS04/19/23
MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf
A Solution to challenges
• Provide an option for the handler to generate fully CF-compliant output
• Create a separate XML input file for the handler and feed it into the handler
• Insert attributes, renaming rules, and missing values by putting them in the XML input
• This requires a knowledge about the characteristics of data in advance
26Annual briefing to ESDIS04/19/23
One Solution An input file from data provider
• Example: Attributes, Renaming, Values
27Annual briefing to ESDIS04/19/23
<xml><variable name=‘RrLandRain’> <attr name=‘unit’ type=‘String’> inches</attr></variable><rename from=‘pressStd’ to=‘level’></rename><insert name=‘NAltitudeDim’ size=10 type=‘Int8’>0,1,2,3,4,5,6,7,8,9</insert></xml>
Future Work (Short Term)
• Expand test-suite with data pool
• Release through OPeNDAP website
• Support for Swath via Parsing
• Support for Point data type via HDF-EOS2 library
28Annual briefing to ESDIS04/19/23
Future Work (Long Term)
• General HDF4 Files Support
• Using an input XML file to provide full
CF support
• HDF-EOS2 to DAP2 Mapping Document
29Annual briefing to ESDIS04/19/23
Thank You!
October 15-18, 2008 Workshop Name (edit footer to change) 30
Acknowledgements
This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A, NNX08A077A and NNX06AG75A.
Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.
October 15-18, 2008 31Workshop Name (edit footer to change)
Questions/comments?
October 15-18, 2008 Workshop Name (edit footer to change) 32
top related