9/19/2000 McGrath 9/19/2000 1

University of Illinois at Urbana-Champaign

HDFHDF

Transition from HDF4 to Transition from HDF4 to HDF5: IssuesHDF5: Issues

Robert E. McGrathRobert E. McGrath

NCSANCSA

University of Illinois at Urbana-ChampaignUniversity of Illinois at Urbana-Champaign

9/19/2000 McGrath 9/19/2000 2

University of Illinois at Urbana-Champaign

HDFHDF

Overall GoalOverall Goal

A discussion of compatibility, conversion, A discussion of compatibility, conversion, and interoperability issues for HDF4 and and interoperability issues for HDF4 and

HDF5, and HDFEOS.HDF5, and HDFEOS.

For more details see the white paper at:For more details see the white paper at:

http://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdfhttp://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdf

9/19/2000 McGrath 9/19/2000 3

University of Illinois at Urbana-Champaign

HDFHDF

Conclusions (Preview)Conclusions (Preview)

Transition from HDF4 to HDF5 is not Transition from HDF4 to HDF5 is not necessarily transparent or effortless.necessarily transparent or effortless.

9/19/2000 McGrath 9/19/2000 4

University of Illinois at Urbana-Champaign

HDFHDF

Conclusions: HDFConclusions: HDF

• Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be written or rewritten.

• Many tools will support both HDF4 and HDF5.

• Default translation from HDF4 to HDF5 will be supported as a utility and library.

9/19/2000 McGrath 9/19/2000 5

University of Illinois at Urbana-Champaign

HDFHDF

Conclusions: HDF-EOSConclusions: HDF-EOS

• Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be written or rewritten

• Translation of ‘pure’ HDF-EOS will be supported

• Translation of ‘hybrid’ HDF-EOS files is very difficult.

• Not possible to support V.2 and V.3 in a single program

9/19/2000 McGrath 9/19/2000 6

University of Illinois at Urbana-Champaign

HDFHDF

What Does Compatibility Mean?What Does Compatibility Mean?

9/19/2000 McGrath 9/19/2000 7

University of Illinois at Urbana-Champaign

HDFHDF

What does “compatibility” mean?What does “compatibility” mean?

• Ability to “do the same things” with new software– I.e., “compatibility” is relative to some purpose– Many dimensions and shades of compatibility– No golden solution sufficient for every use

• Assume HDF4 users want to continue doing what they are already doing, but move to HDF5

9/19/2000 McGrath 9/19/2000 8

University of Illinois at Urbana-Champaign

HDFHDF

Some Types of CompatibilitySome Types of Compatibility

CompatibilityIssue

Description

Conceptual The overall conceptual approach:the formal and informal models.

Format Storage layout and other formatdetails.

Software Compatibility of the software Programming model

The assumptions and approach to programming.

Features The set of features supported. Interfaces The programming interfaces.

9/19/2000 McGrath 9/19/2000 9

University of Illinois at Urbana-Champaign

HDFHDF

Compatibility of HDF and HDF-EOSCompatibility of HDF and HDF-EOS

Compatibility Issue HDF4/HDF5 HDFEOS2/HDFEOS3Conceptual Yes, H5 is a

supersetYes

Format No NoSoftware Programming model No? Yes Features Some Yes, V3 is a superset Interfaces No No

9/19/2000 McGrath 9/19/2000 10

University of Illinois at Urbana-Champaign

HDFHDF

InteroperationInteroperation

• My definition of “interoperation”: one program or tool may use HDF4 or HDF5 or both

9/19/2000 McGrath 9/19/2000 11

University of Illinois at Urbana-Champaign

HDFHDF

NCSA HDF4 and HDF5NCSA HDF4 and HDF5

9/19/2000 McGrath 9/19/2000 12

University of Illinois at Urbana-Champaign

HDFHDF

Single Combined HDF Library

dataset1 dataset2

Stored asHDF4 Objects

image1image2

Stored asHDF5 Objects

HDF4 file HDF5 file

HDF 4 Library HDF 5 Library

Interoperability: Any program may use either HDF4 or 5 Interoperability: Any program may use either HDF4 or 5 or both:or both:

Not Supported

9/19/2000 McGrath 9/19/2000 13

University of Illinois at Urbana-Champaign

HDFHDF

Application Specific Library

dataset1 dataset2

Stored asHDF4 Objects

image1image2

Stored asHDF5 Objects

HDF4 file HDF5 file

HDF 4 Library HDF 5 Library

Interoperability: Specific Application or tool may use Interoperability: Specific Application or tool may use either HDF4 or 5 or both:either HDF4 or 5 or both:

Up to Application and Tool Developers

Application specific Library

9/19/2000 McGrath 9/19/2000 14

University of Illinois at Urbana-Champaign

HDFHDF

HDF-EOS Version 2 and Version 3HDF-EOS Version 2 and Version 3

9/19/2000 McGrath 9/19/2000 15

University of Illinois at Urbana-Champaign

HDFHDF

Grid1 Grid2

Stored asHDF4 Objects

Grid3 Grid4

Stored asHDF5 Objects

HDF4 file HDF5 file

HDF 4 Library HDF 5 Library

HDFEOS Library

Interoperability: Interoperability: Any program may using HDF-EOS Objects with either Any program may using HDF-EOS Objects with either

HDF4 or HDF5 or both:HDF4 or HDF5 or both:Not supported

9/19/2000 McGrath 9/19/2000 16

University of Illinois at Urbana-Champaign

HDFHDF

HDFEOS V2 Library HDFEOS V3 Library

Grid1 Grid2

Stored asHDF4 Objects

Grid3 Grid4

Stored asHDF5 Objects

HDF4 file HDF5 file

HDF 4 Library HDF 5 Library

Interoperability: One application may use HDF-EOS Interoperability: One application may use HDF-EOS Objects with either HDF4 or HDF5 or both:Objects with either HDF4 or HDF5 or both:

Not supported

9/19/2000 McGrath 9/19/2000 17

University of Illinois at Urbana-Champaign

HDFHDF

TranslationTranslation

• Convert HDF4 objects (file) to HDF5 objects (file)

9/19/2000 McGrath 9/19/2000 18

University of Illinois at Urbana-Champaign

HDFHDF

Translation of HDF4 to HDF5Translation of HDF4 to HDF5

Stored asHDF4 Objects

Stored as AnalogousHDF5 Objects

/

HDF4 Library HDF5 Library

HDF4 File HDF5 File

Supported: NCSA mapping and h4toh5 tool and library

9/19/2000 McGrath 9/19/2000 19

University of Illinois at Urbana-Champaign

HDFHDF

Grid1 Grid2

Translation of HDF-EOS V2 to V3Translation of HDF-EOS V2 to V3

HDFEOSObjects

Stored asHDF4 Objects

Grid1 Grid2

SAMEHDFEOSObjects

Stored asHDF5 Objects

HDF4 file HDF5 file

HDF4 Library

/

HDF5 Library

HDF EOS V. 2.x Library HDF EOS V. 3 Library

Supported

9/19/2000 McGrath 9/19/2000 20

University of Illinois at Urbana-Champaign

HDFHDF

““Hybrid” HDF-EOS filesHybrid” HDF-EOS files

• HDF-EOS files may contain native HDF objects (images, arrays, etc.) as well as HDF-EOS objects (Grid, etc.)

• Almost all HDF-EOS files are ‘hybrids’.

• Very difficult to automatically translate hybrid files.

9/19/2000 McGrath 9/19/2000 21

University of Illinois at Urbana-Champaign

HDFHDF

““Hybrid” HDF-EOS FileHybrid” HDF-EOS File

NCSA HDF

HDFEOS

Science Application Code

Other

Components ofHDFEOS Objectsstored in HDFObjects

Other HDFObjects StoredDirectly ThroughHDF

One HDF file, objectsmixed together.

Very difficult to tell which

objects are part of HDF-EOS

9/19/2000 McGrath 9/19/2000 22

University of Illinois at Urbana-Champaign

HDFHDF

ConclusionsConclusions

Transition from HDF4 to HDF5 is not Transition from HDF4 to HDF5 is not necessarily transparent or effortless.necessarily transparent or effortless.

9/19/2000 McGrath 9/19/2000 23

University of Illinois at Urbana-Champaign

HDFHDF

Conclusions: HDFConclusions: HDF

• Transition from HDF4 to HDF5 is not transparent: in most cases source code will have to be rewritten.

• Many tools will support both HDF4 and HDF5.

• Default translation from HDF4 to HDF5 will be supported as a utility and library.

9/19/2000 McGrath 9/19/2000 24

University of Illinois at Urbana-Champaign

HDFHDF

Conclusions: HDF-EOSConclusions: HDF-EOS

• Transition from HDFEOS2 to HDFEOS3 is not transparent: in most cases source code will have to be rewritten

• Translation of ‘pure’ HDF-EOS will be supported

• Translation of ‘hybrid’ HDF-EOS files is very difficult.

• Not possible to support V.2 and V.3 in a single program

9/19/2000 McGrath 9/19/2000 25

University of Illinois at Urbana-Champaign

HDFHDF

For More InformationFor More Information

• NCSA White paper: “Transition From HDF4 to HDF5”: http://hdf.ncsa.uiuc.edu/HDF5/doc/TechNotes/TransitionIssues.pdf

• “Guidelines for Mapping HDF4 to HDF5 Objects”, http://hdf.ncsa.uiuc.edu/HDF5/papers/H4-to-H5MappingGuidelines.pdf

• h4toh5 tool: Soon