improving the usability of hdf-eos2 data

Improving the usability of HDF-EOS2 data

Kent Yang, Joe Lee, Choonghwan Lee

The HDF Group

March 31st, 2009

04/21/23 Annual briefing to ESDIS 1


EOSDIS Evolution 2015 Vision TenetsVision Tenet Vision 2015 Goals

Archive Management NASA will ensure safe stewardship of the data through its lifetime. The EOS archive holdings are regularly peer reviewed for scientific merit.

EOS Data Interoperability

Multiple data and metadata streams can be seamlessly combined. Research and value added communities use EOS data interoperably with other

relevant data and systems. Processing and data are mobile.

Future Data Access and Processing

Data access latency is no longer an impediment. Physical location of data storage is irrelevant. Finding data is based on common search engines. Services invoked by machine-machine interfaces. Custom processing provides only the data needed, the way needed. Open interfaces and best practice standard protocols universally employed.

Data Pedigree Mechanisms to collect and preserve the pedigree of derived data products are readily

available.

Cost Control Data systems evolve into components that allow a fine-grained control over cost

drivers.

User Community Support

Expert knowledge is readily accessible to enable researchers to understand and use the data.

Community feedback directly to those responsible for a given system element.

IT Currency Access to all EOS data through services at least as rich as any contemporary science

information system.

Feb 3, 2005

Purpose of the talk

• Identify problems to access HDF-EOS2 data• Share our plan to help improve the EOS2

usability

04/21/23 3Annual briefing to ESDIS

The success of EOS mission


MODIS HDF-EOS data to detect sand storm from Gobi chinaCourtesy NASA Earth Observatory

The success of EOS mission


MODIS HDF-EOS data to detect fires in AfricaCourtesy NASA

Identify problems : technical part


Beauty of HDF-EOS2

• Very close to physical model• Convenient to handle geo-location projections• Files are generally compact

No needs to store latitude and longitude for grid Reduce the size of latitude and longitude for

swath



A compact HDF-EOS2 Grid

• MODIS (MYD10A2) Grid file• Two 2400 X 2400 grid data fields with gzip

compression• File size: 75 KB• If lat/lon are kept in the file

• File size may become 44 MB,

500 times bigger than the original file size

October 15-18, 2008 8Workshop Name (edit footer to change)

A compact HDF-EOS2 swath

• Small file size – Swath

04/21/23 9

Annual briefing

to ESDIS

cross

alon

gcross

alon

g

coarse_cross

coar

se_a

long

with dimension maps

CloudCover(along, cross)Latitude(along, cross)Longitude(along, cross)

CloudCover(along, cross)Latitude(along, cross)Longitude(along, cross)

CloudCover(along, cross)Latitude(coarse_along, coarse_cross)Longitude(coarse_along, coarse_cross)

CloudCover(along, cross)Latitude(coarse_along, coarse_cross)Longitude(coarse_along, coarse_cross)

with dimension maps

w/o dimension maps

along 406 4060

cross 271 2708

Longitude 0.5 MB(406 X 271 X 4)

42 MB(4060 X 2708 X 4)

There are issues to access some HDF-EOS2 files

Aims to improve the usability



First of all, there should be more communications among providers, tool developers, users and data storage software developers.

Focus on HDF-EOS2 and HDF4 in the next few slides.

Schematic of Grid


YD

imXDim

StructMetadata.0 =“… XDim= 14 YDim = 8 UpperLeftPointMtrs =(-156.0,71.0)LowerRightMtrs =(180.0,-75.0)… Projection = HE5_GCTP_GEO …” HDF-EOS5 GRIDS:Float32 Ozone(YDim, XDim)


Issue 1: lack of strict checking in HDF-EOS2 library


Lack of strict checking in HDF-EOS2 library

• XDim and YDim exist and have values .e.g, OMI (OMUVBL3) Generally, XDim and YDim do not exist because

they are implied. XDim and YDim do not need to have values

because they can be calculated. Values are just redundant.

04/21/23 14

Annual briefing

to ESDIS

• Inconsistency in the number of elements HDF-EOS2 informs and HDF-EOS2 actually reads e.g., MODIS (MAC021S0, MAC02QS0,

MAC02QS1, MAC021S1) We believe that this data field had four elements

once and it was appended later.


04/21/23 15

Annual briefing

to ESDIS

StructMetadata … Band_1KM_RefSB[4] …

StructMetadata … Band_1KM_RefSB[4] …

Data FieldBand_1KM_RefSB

4 elements


4 elements


15 elements


15 elementsconsistent

INCONSISTENT!INCONSISTENT!


• Wrong grid parameters e.g., MISR (MI3DAER) Projection is defined by code, projection

parameters, etc. For Lambert Azimuth projection, the fifth

parameter specifies “longitude of center of projection”.

Its value is entered in DDDMMMSSS.SS format. -92970000.00


degrees -92degrees -92 seconds 0seconds 0minutes 970?minutes 970?

Issue 2: lack of standard way to handle some geolocation

fields


Schematic of Swath


cross

alon

g

StructMetadata.0 =“ … DimensionName=”along”

size= 11DimensionName=”cross” size= 3 …” Data fields: Float32 CloudCover(along,cross)Geolocation fields: Float32 lat(along,cross) Float32 lon(along,cross)

No standard way to handle geolocation fields

• No standard way to figure out the vertical coordinate in an HDF-EOS2 file A data field

• TAirStd[GeoTrack][GeoXTrack] StdPressureLev is a dimension Additional data field supplies values, probably

• pressStd How can a user know for sure that the field

pressStd provides the vertical coordinate information?


[StdPressureLev][StdPressureLev]



Issue 3: No enforcement for the use of standard names


Schematic of Grid


YD

imXDim



No enforcement for the use of standard names

• Use of non-standard dimensions in grid e.g., MOPITT (MOP03) To be interpreted properly, each data field must

make use of the two predefined dimensions: XDim and YDim

However, none of data fields in that file refers to XDim or YDim. Instead, they refer to nlon and nlat, which are not standard way.


No enforcement for the use of standard names

• Object names are not restrictive e.g., MODIS (MAC03S0) Name of a data field called

• Land/SeaMask A dimension called

• nscans*10 These names may not be safe in other APIs.

Especially, HDF5 does not allow slash(/).


Other bad practices

• Extreme long attributes• Chunk size much bigger than dimensional size

for fixed size SDS

• These are not enforced by HDF4 library


Other technical difficulties

• Hybrid HDF-EOS2 files Some data cannot be accessed by HDF-EOS2

library

• Need to help users access these files


Identify problems : service part


A message from DAAC User Survey


I am a student and I need to read MODIS HDF-EOS data. But it is hard for me to extract the data. I need a fortran program to extract the data. I wish DAAC can provide this.

DAAC tool sites

• Provide most information users need


What’s left?

• Some tools are out of date• On some sites, no C/fortran examples to access

the data• Information on the tool page out of date


What’s the problem?

• A small percentage of EOS2 users needs help. Access “difficult” files Compensate tools/examples for DAACs


The HDF Group’s plan to DAACs

• Compensate fortran/C example code for DAACs• C/fortran examples for hybrid files and “difficult”

files• Evaluate/update tools listed at hdfeos.org• Evaluate tools at DAACs• EOS-forum

Web forum? EOS2 user and developer mailing lists?


Thank You!


Acknowledgements

This work was supported by the Cooperative Agreement with the National Aeronautics and Space Administration (NASA) under NASA grant NNX06AC83A and NNX08A077A.

Any opinions, findings, conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of NASA.


Questions/comments?


improving the usability of hdf-eos2 data

Documents

relevant data

data pedigreemechanisms

modis hdfeos data

future data access

grid data fields

processing data access

usability of hdf

hdfeos2 datashare