HDF4 OPeNDAP ProjectProgress Report

MuQun Yang and Hyo-Kyung Lee

1HDF Developers' Meeting04/21/23

OPeNDAP Review

• A software framework that allows simple access to remote data• Data Access Protocol• Client-server model• Widely used by Earth Science Community

2HDF Developers' Meeting04/21/23

Users

OPeNDAP Clients

IDV

OPeNDAP Servers

HDF4 Files

Hyrax AIRS

DAP

libnc-dap

Visualization Tools

OPeNDAP Reivew

Client Library(libnc-dap)

DAP Protocol (via http)

Server(apache)

Handler(hdf4_handler)

Remote Data(HDF4)

View Data (User)

OPeNDAP Concept

4HDF Developers' Meeting04/21/23

Visualization Tools

(gradsdap)

Typical Remote Access

FTP/HTTP

Visualization tools

(e.g. grads)

Better Access for HDF-EOS2 via DAP

• Faster download through selection

• Quicker analysis through visualization

5HDF Developers' Meeting04/21/23

Goals

You bought a HDTV but...

• No HD signal

• Needs more Channels in HD

• Cable Service Provider Problem

6HDF Developers' Meeting04/21/23

HDTV Problems

You downloaded a viewer but...

• No CF convention

• Needs more HDF4 data in CF

• Data Service Provider Problem

7HDF Developers' Meeting04/21/23

HDF4 Problems

Challenges: HDF-EOS2

• CF convention (e.g. NetCDF model)

• Syntax

• Clients expect a certain data structure

• Semantics

• Clients expect some key attributes

8HDF Developers' Meeting04/21/23

HD Signal Challenge

Challenges: HDF-EOS2

• DVD/VHS/VCD/LD >>> Blue-Ray

• Can all titles be delivered in HD?

• HDF4/HDF-EOS2 >>> HDF5/HDF-EOS5

• Can all HDF-EOS2 be delivered in CF?

9HDF Developers' Meeting04/21/23

More Channels in HD Challenge

Solution: New HDF4 Handler

10HDF Developers' Meeting04/21/23

HD Signal Solution

• Syntax

Generate the correct DAP data type. Add shared geo-location information.

• Semantics Insert additional key attributes.

Solution: New HDF4 Handler

• Use Metadata parser

• Pro: simple

• Con: limited channels

• Use HDF-EOS2 library

• Pro: maximum channels

• Con: install HE2 library first

11HDF Developers' Meeting04/21/23

More Channels in HD Solution

Solution Comparison

12HDF Developers' Meeting04/21/23

Quick Implementation

• Parse Metadata – hdf5_handler

• Use HDF-EOS2 library – h4toh5

13HDF Developers' Meeting04/21/23

Current Status

• MODIS / AIRS Grid and Swath

• Other Projections (Polar, Sinusoidal)

• Testing

14HDF Developers' Meeting04/21/23

HDF-EOS2 in CF via DAP

• Finally, Happy Data Consumer!

15HDF Developers' Meeting04/21/23

AIRS Grid AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdfhttp://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10MODIS GridMOD08_D3.A2008284.005.2008289105354.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html

AIRS Swath ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/

AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08331121849.hdfMODIS Swath

MOD06_L2.A2001222.0025.004.2003127111841.hdfhttp://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html

Limitations – Missing Attributes

• MODIS Grid – Rain fall

16HDF Developers' Meeting04/21/23

No Unit!

• AURA OMI Grid - Ozone

Limitations – Missing Names

• How to Match Dimension Names?

17HDF Developers' Meeting04/21/23

HDF4 Handler

NetCDF4 Handler

pressureStd → Level

Limitations – Missing Values

• How to Supply Missing Dimension Values?

18HDF Developers' Meeting04/21/23

MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf HTTP/1.0 200 OKXDODS-Server: 3.7.9XOPeNDAP-Server: 3.7.9XDAP: 3.1Content-Type: text/plainContent-Description: dods_errorCache-Control: no-cache

Error { code = 1002; message = "An internal error was encountered:This HDF-EOS2 file has multiple Grids with different dimension sizes.Please report this to support@opendap.org";};

Solution – Customization

• Provide an option for handler customization to generate fully CF-compliant output

• Create a separate XML input file for handler and feed it into the handler.

• Insert attributes, renaming rules, and missing values by putting them in the XML input.

• This requires a knowledge about the characteristics of data in advance.

19HDF Developers' Meeting04/21/23

Solution – Customization

• Example: Attributes, Renaming, Values

20HDF Developers' Meeting04/21/23

<xml><variable name=‘RrLandRain’> <attr name=‘unit’ type=‘String’> inches</attr> <attr name=‘_FillValue’ type=‘Int16’>-1</attr></variable><rename from=‘GeoXTrack’ to=‘lon’ ></rename><rename from=‘pressStd’ to=‘lev’></rename><insert name=‘NAltitudeDim’ size=10 type=‘Int8’>0,1,2,3,4,5,6,7,8,9</insert></xml>

Summary

• HDF4 Access via OPeNDAP Is Easy, Efficient and Fast Is Cool like HDTV Adding CF is not straightforward

• Use HDF-EOS2 library Is Essential Maximize the data coverage

21HDF Developers' Meeting04/21/23

Future Work (Short Term)

• Expand test-suite with data pool

• Release through OPeNDAP website

• Support for Swath via Parsing

• Support for Point data type via HDF-EOS2 library

22HDF Developers' Meeting04/21/23

Future Work (Long Term)

• General HDF4 Files Support

• Customization for CF option

• HE2 to DAP Mapping Documentation

23HDF Developers' Meeting04/21/23