esdis metadata archive
DESCRIPTION
NASA's Earth Observing System (EOS) archive includes data collected over many years by many satellite instruments. These data are stored in the HDF format that includes data and metadata. The content of the metadata was examined for compliance with a set of conventions developed by the NASA science community at the beginning of the EOS Project (the HDF-EOS conventions). The initial results show that ~50% of the data files and 76% of the datasets have metadata that allows them to be used easily in standard tools. This talk was presented at the ESIP (espied.org) meeting during January 2014.TRANSCRIPT
ESIP Winter 2014 1
The EOSDIS Metadata Archive
Ted HabermannDirector of Earth Science The HDF [email protected]
January 8-10, 2014
ESIP Winter 2014 2
The EOSDIS Metadata Archive
Ted HabermannDirector of Earth Science The HDF [email protected]
January 8-10, 2014
ESIP Winter 2014 3
Metadata Types and Sharing
January 8-10, 2014
Discovery
Use / Mashup
Understanding
Core
Structural
Archive
HDF-EOS
ACDD
NUG,CF
netCDF
Data Tools Need Use Metadata
January 8-10, 2014ESIP Winter 2014
78 Tools for EOSDIS Data
Data Tools Need Use Metadata
January 8-10, 2014ESIP Winter 2014
5
78 Tools for EOSDIS Data
Data Tools Need Use Metadata
January 8-10, 2014ESIP Winter 2014
6
Structural Metadata
78 Tools for EOSDIS Data
HDF-EOS Grid Metadata
Dimension
+ DimensionName: CharacterString+ Size : Integer
Dimension
Grid
+ GridName : CharacterString+ Xdim : Integer+ Ydim ; Integer+ UpperLeftPointMtrs : CharacterString+ LowerRightMtrs : CharacterString+ Projection : CharacterString+ ProjParams : Real+ GridOrigin : Code+ SphereCode : Integer+ PixelRegistration : Code+ ZoneCode : Code
GridStructure
+ group 0..*
+ Dimension 0..*
+ group 0..1
DataField
+ group 0..1
+ DataField 0..*
MergedFields
+ group 0..1
StructMetadata + group
DataField
+ DataFieldName: CharacterString+ DataType: CharacterString+ DimList: CharacterString+ MaxDimList : CharacterString+ CompressionType : Code+ DeflateLevel : Integer+ BlockSize : Integer+ CompressionParameters : Integer
HDF-EOS Grid Metadata
Dimension
+ DimensionName: CharacterString+ Size : Integer
Dimension
Grid
+ GridName : CharacterString+ Xdim : Integer+ Ydim ; Integer+ UpperLeftPointMtrs : CharacterString+ LowerRightMtrs : CharacterString+ Projection : CharacterString+ ProjParams : Real+ GridOrigin : Code+ SphereCode : Integer+ PixelRegistration : Code+ ZoneCode : Code
GridStructure
+ group 0..*
+ Dimension 0..*
+ group 0..1
DataField
+ group 0..1
+ DataField 0..*
MergedFields
+ group 0..1
StructMetadata + group
DataField
+ DataFieldName: CharacterString+ DataType: CharacterString+ DimList: CharacterString+ MaxDimList : CharacterString+ CompressionType : Code+ DeflateLevel : Integer+ BlockSize : Integer+ CompressionParameters : Integer
Grid
+ GridName : CharacterString+ XDim : Integer+ YDim : Integer+ UpperLeftPointMtrs : CharacterString+ LowerRightMtrs : CharacterString+ Projection : CharacterString+ ProjParams : CharacterString+ GridOrigin : Code+ SphereCode : Integer+ PixelRegistration : Code+ ZoneCode : Code
HDF-EOS Swath Metadata
Dimension
+ DimensionName: CharacterString+ Size : Integer
Dimension
StructMetadata.N + group
Swath
+ SwathName : CharacterString
SwathStructure
+ group 0..*
+ group 0..1
DimensionMap
+ GeoDimension: CharacterString+ DataDimension: CharacterString+ Offset: Integer+ Increment: Integer
DimensionMap
+ group 0..1
+ Dimension 0..* + DimensionMap 0..*
IndexDimensionMap
+ group 0..1
GeoField
+ GeoFieldName: CharacterString+ DataType: Code+ DimList: CharacterString+ MaxDimList : CharacterString+ CompressionType : Code+ DeflateLevel : Integer+ BlockSize : Integer+ CompressionParameters : Integer
GeoField
+ group 0..1
+ GeoField 0..*
DataField
+ DataFieldName: CharacterString+ DataType: CharacterString+ DimList: CharacterString+ MaxDimList : CharacterString+ CompressionType : Code+ DeflateLevel : Integer+ BlockSize : Integer+ CompressionParameters : Integer
DataField
+ group 0..1
+ DataField 0..*
MergedFields
+ group 0..1
+ IndexDimensionMap 0..*
IndexDimensionMap
+ GeoDimension: CharacterString+ DataDimension: CharacterString
The NASA Metadata Archive
HDF4Mapper
ODL2XML.xsl
HDF4MapWriter was used to extract metadata from 600+ HDF4 Products for this analysis.
ESIP Winter 2014 11
Results
January 8-10, 2014
50%GranulesHDF-EOS
Metadata Groups
50%Granules
76%ProductsHDF-EOS
ESIP Winter 2014 12
Conclusion
January 8-10, 2014
Metadata from the entire HDF4 archive could be read using a single tool (HDF4Mapper) because the entire archive is in a single format (HDF4).
Roughly half of the archive includes conventional use metadata (StructMetadata.N) that allows it to be plotted / mapped with conventional tools.
ESIP Winter 2014 13
Conclusion
January 8-10, 2014
Metadata from the entire HDF4 archive could be read using a single tool (HDF4Mapper) because the entire archive is in a single format (HDF4).
Roughly half of the archive includes conventional use metadata (StructMetadata.N) that allows it to be plotted / mapped with conventional tools.
ESIP Winter 2014 14
The Earth Science Platform
January 8-10, 2014
Services
For
mat
s
Conventions
Tools
The Earth Science Platform depends on consistent formats and conventions to support efficient tool/service developmentFormats are working well but need to be updatedConventions need work
ESIP Winter 2014 15
The Earth Science Platform
January 8-10, 2014
Services
For
mat
s
Conventions
Tools
The Earth Science Platform depends on consistent formats and conventions to support efficient tool/service developmentFormats are working well but need to be updatedConventions need work
ESIP Winter 2014 16
The Earth Science Platform
January 8-10, 2014
Services
For
mat
s
Conventions
Tools
The Earth Science Platform depends on consistent formats and conventions to support efficient tool/service developmentFormats are working well but need to be updatedConventions need work
18BESSIG
Acknowledgements
July 24, 2013
This work was partially supported by contract number NNG10HP02C from NASA.
Any opinions, findings, conclusions, or recommendations expressed in this material are those of the author and do not necessarily reflect the views of NASA or The HDF Group.