images of hdf5
DESCRIPTION
Images of HDF5. Gerd Heber The HDF Group The 15 th HDF and HDF-EOS Workshop April 17-19, 2012. Outline. Five long stories distilled into shorts: A model of the information in an HDF5 file A new XML representation of HDF5 HDF5 as a Service - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/1.jpg)
www.hdfgroup.org
The HDF Group
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 1
Images of HDF5
Gerd Heber The HDF Group The 15th HDF and HDF-EOS Workshop April 17-19, 2012
![Page 2: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/2.jpg)
www.hdfgroup.org2
Outline
Five long stories distilled into shorts:
• A model of the information in an HDF5 file
• A new XML representation of HDF5
• HDF5 as a Service
• The HDF5 user experience I always wanted
• An odd couple – HDF5 and databases
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 3: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/3.jpg)
www.hdfgroup.org3
HDF5 INFORMATION SET
“Language shapes the way we think, and determines what we can think about.”
(Benjamin L. Whorf)
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 4: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/4.jpg)
www.hdfgroup.org4
HDF5 Information Set
• Is a model of the content of an HDF5 file
• Provides a consistent set of definitions
• Gives an undistorted view of HDF5*
• Puts the simplicity of HDF5 center stage
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
*Not tainted by the idiosyncrasies of a particular API
![Page 5: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/5.jpg)
www.hdfgroup.org5
HDF5 Information Set
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 6: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/6.jpg)
www.hdfgroup.org6
Sources of Complexity
1. Productivity• Finite number of parts and combining-rules
yields an infinite number of unique structures• HDF5 groups and datatypes
2. Reference (Cohesion)• The ability to refer from one part to another • HDF5 groups, links, and references
(By comparison, databases are only weakly productive and their referential capabilities are limited by Codd’s Information Principle.)Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 7: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/7.jpg)
www.hdfgroup.org7
HDF5 Micro-Web
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
7
lat | lon | temp----|-----|----- 12 | 23 | 3.1 15 | 24 | 4.2 17 | 21 | 3.6
Experiment Notes:Serial Number: 99378920Date: 3/13/09Configuration: Standard 3
/SimOutViz
Every HDF5 file has a root group
Parameters10;100;1000
Timestep36,000
September 28-30, 2010 HDF/HDF-EOS Workshop XIV
IMG1
IMG2IMG3
TBL1 TBL3
TBL1TBL2
Ext
![Page 8: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/8.jpg)
www.hdfgroup.org8
Hypermedia
Hypermedia – An application that uses associative relationships among information contained within multiple media datafor the purpose of facilitating access to and manipulation of the information encapsulated by the data.
[Lowe & Hall 99]
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 9: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/9.jpg)
www.hdfgroup.org
The HDF Group
Questions?
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 9
![Page 10: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/10.jpg)
www.hdfgroup.org10
REPRESENTING HDF5 IN XML
“We find that the same word – Fidelity – can be used both in connection with the excellence of sound reproduction and picture reproduction.”
(1931 Electronics Oct. 137/1)
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 11: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/11.jpg)
www.hdfgroup.org11
Use Cases
1. Viewing structure and contents of an HDF5 file in a web browser (XSLT in the browser)
2. XML as a catalog record3. XML as a light-weight intermediate form for applications4. Generation, validation, and reconstruction of HDF5 files5. XML as intermediate to other data languages or file
formats (e.g., ISO, netCDF)6. XML as machine-readable documentation7. Templates, skeleton files, etc.
(Source: The XML DTD for HDF5: Design Notes. 12 June 2000)
10+ years on – still a pretty complete list! Where are we?
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 12: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/12.jpg)
www.hdfgroup.org12
HDF5/XML Survey
• http://www.surveymonkey.com/s/RMSZSSX• 13 replies to date (still open)• Users are fluent in XML Schema, XPath, XSLT,
and XLink/XPointer• Descriptive data are more important than a
full-fledged data element representation• Hardly anybody uses the HDF Group’s XML
schema, most respondents created their own• Split on the fidelity of the representation
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 13: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/13.jpg)
www.hdfgroup.org13
Why another schema?
• Address shortcomings• Omissions• Eliminate redundancies• De-normalized group structure representation• Dataset and attribute value serialization• Simplify tools
• Reflect simplicity of the HDF5 data model• High-fidelity representation• Be neutral with respect to application domains• Future proofing
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 14: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/14.jpg)
www.hdfgroup.org14
High-Level Structure
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
<domain xmlns=“http://www.hdfgroup.org/HDF5/XML/schema/2011/11/11” xmlns:xlink=“http://www.w3.org/1999/xlink”>
<!-- “Pointer” to the HDF5 root group --> <root xlink:href=“903d1d75-e617-4767-a3bf-0cb3ee509027”/>
<linkbase> <!–- Representations of HDF5 groups --> </linkbase>
<database> <!–- Representations of HDF5 datasets --> </database>
<encodingbase> <!-- Collection of representations of HDF5 datatypes --> </encodingbase>
</domain>
![Page 15: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/15.jpg)
www.hdfgroup.org15
HDF5/XML Summary
• HDF5/XML is a high-fidelity rendering of user-level HDF5 items in XML
• Communities/domain experts should create XML representations that work for their users• HDF5/XML cannot fill that role
• One can use XSLT or XQuery to connect to the HDF5/XML tool chain (to be developed)
See me for a demo and additional information / questions /comments / suggestions / donations
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 16: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/16.jpg)
www.hdfgroup.org
The HDF Group
Questions?
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 16
![Page 17: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/17.jpg)
www.hdfgroup.org17
HDF5/REST*
But let your communication bee, GET, PUT: POST, DELETE: For whatsoeuer is more then these, commeth of euill.”
(Matthew 5:37, KJV 1611, Tyndale 1526)
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
*The support of Wenming Ye and Daniel Odievich (Microsoft) for this project is gratefully acknowledged.
![Page 18: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/18.jpg)
www.hdfgroup.org18
Outline
• REST• Resources• Representations• URIs
• Cloud / Windows Azure• Summary
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 19: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/19.jpg)
www.hdfgroup.org19
REST*
the Internet as it was originally conceivedis perfectly suited for transferring both hypermedia-based documents and data
[Scribner & Seely 2009]
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
*REpresentational State Transfer [Fielding 2000]
Why create complex data service architectures when
?
![Page 20: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/20.jpg)
www.hdfgroup.org20
Four Simple Principles
1. The server maintains resources that are separate from representations returned to clients
2. Clients manipulate resources via the representations issued to them
3. The messages that convey representations to the client are self-describing
4. Application state is transferred using hypermedia techniques
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
[Scribner & Seely 2009]
![Page 21: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/21.jpg)
www.hdfgroup.org21
HDF5/REST Resources
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 22: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/22.jpg)
www.hdfgroup.org22
HDF5/REST URIs
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 23: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/23.jpg)
www.hdfgroup.org23
HDF5/REST URIs – Examples
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 24: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/24.jpg)
www.hdfgroup.org24
Examples
Get (a representation of) the HDF5 root
Create a new HDF5 group (unlinked)
Link the newly created group as ‘New Group’
Delete an HDF5 attribute
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
GET /root
POST /groups # server replies with {groupID}
POST /groups/{groupID1}/participants/New%20Group{groupID} # content
DELETE /datasets/{datasetID}/attributes/{name}
![Page 25: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/25.jpg)
www.hdfgroup.org25
Representations
• Clients express preferences via Accept header
• Server may reply with
or
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
Accept: application/json;0.9, text/xml, application/xml;q=0.8, application/octet-stream;q=0.7, image/png, image/gif, image/jpeg;q=0.2, */*; q=0.1Accept-Encoding: gzip, deflate, compress;q=0.9
Content-Type: text/xmlContent-Length: 2890…
HTTP/1.1 406 Not Acceptable
![Page 26: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/26.jpg)
www.hdfgroup.org26
Windows Azure
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
Fabric
Storage
SDK VS
Watch Steve Marx’s
“What is Windows Azure?”
![Page 27: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/27.jpg)
www.hdfgroup.org27
Windows Azure Implementation
Why it’s easy…• HDF5/XML proxy• XSLT does most of the
heavy lifting• HDF5DotNet for data
access• Great development
and deployment tools• Easy scale-out
Challenges• Cloud BLOB/block
stores aren’t file systems
• Performance from• Caching• Latency hiding• Parallelism
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 28: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/28.jpg)
www.hdfgroup.org28
HDF5/REST Summary
• HDF5/REST is an “HTTP API” for HDF5• RISC rather than CISC• Build more complex services on top of
HDF5/REST (e.g., HDF5DNS, HDF5WHOIS)• HDF5 domains = “virtual HDF5 files”
See me for a demo and additional information / questions /comments / suggestions / donations
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 29: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/29.jpg)
www.hdfgroup.org
The HDF Group
Questions?
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 29
![Page 30: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/30.jpg)
www.hdfgroup.org30
AN HDF5 MODULE FOR
A Winning Team:HDF5 + The Best Shell on the Planet
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 31: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/31.jpg)
www.hdfgroup.org31
A Word from the Author
“In the end, there’s no hard-and-fast distinction between a shell language and a scripting language. Some of the features that make a good scripting language result in poor shell user experience.
Because PowerShell’s goal is to be both a good scripting language and a good interactive shell, balancing the tradeoffs between user experience and scripting authoring was one of the major design challenges.”
(Bruce Payette)Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
Conversely, some of the features that make for a good interactive shell experience can interfere with scripting.
![Page 32: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/32.jpg)
www.hdfgroup.org32
Provider Core
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 33: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/33.jpg)
www.hdfgroup.org33
Show Time
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 34: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/34.jpg)
www.hdfgroup.org34
Windows PowerShell Resources
• Bruce Payette, Windows PowerShell in Action, 2nd Edition, Manning 2011
• Scripting with Windows PowerShell
• Windows PowerShell: Learn It Before It’s an Emergency – Part 1-5
• Windows PowerShell Blog
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 35: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/35.jpg)
www.hdfgroup.org
The HDF Group
Questions?
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 35
![Page 36: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/36.jpg)
www.hdfgroup.org36
HDF5 AND DATABASES
“Complaint for true loue vnrequited.”(Sir Thomas Wyatt, 1542)
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 37: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/37.jpg)
www.hdfgroup.org37
Fatal Attraction
• The power and simplicity of the relational model
• SQL is a declarative language• Optimizable• Data independence
• Greater productivity, because it’s easier to express intent at a high-level
(Source: Don Chamberlin on SQL in “Masterminds of Programming”, O’Reilly 2009)
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 38: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/38.jpg)
www.hdfgroup.org38
First Symptoms (Mid-Late 90s)
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 39: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/39.jpg)
www.hdfgroup.org39
HDF and HDF-EOS Workshop 1
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
An HDF-EOS DataBlade using Informix’s Object- Relational
Database
Renu Chaudhry ECOlogic www.ecologic.net
September 8-10, 1897 GSFC, Maryland
![Page 40: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/40.jpg)
www.hdfgroup.org40
BCS Universal File Interface
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
Source: Barrodale Computing Services Ltd.http://www.barrodale.com/universal-file-interface-ufi
![Page 41: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/41.jpg)
www.hdfgroup.org41Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 42: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/42.jpg)
www.hdfgroup.org42
SciQL Highlights
• An extension of SQL:2003 (pronounced as ‘cycle’)
• Array as first class citizens of DBMS• Seamless integration of tables and arrays• Named dimensions with constraints• Flexible structure-based grouping
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 43: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/43.jpg)
www.hdfgroup.org43Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 44: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/44.jpg)
www.hdfgroup.org44
HDF5/DBMS Summary
Three significant developments:• Arrays can be first class citizens• Database file systems offer the potential to
store Level 0 data and analyze Level 1 and Level 3 data within the same DBMS
• All vendors (IBM, Microsoft, Oracle) have rolled out BigData connectors
Databases have morphed into data hubs.We are working hard to get HDF5 connected!
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV
![Page 45: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/45.jpg)
www.hdfgroup.org
The HDF Group
Thank You!
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 45
![Page 46: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/46.jpg)
www.hdfgroup.org
Acknowledgements
This work was supported by Subcontract number 114820 under Raytheon Contract number NNG10HP02C, funded by the National Aeronautics and Space Administration (NASA) and by cooperative agreement number NNX08AO77A from the NASA. Any opinions, findings, conclusions, orrecommendations expressed in this material are those of the authors and do not necessarily reflect the views of Raytheon or the National Aeronautics and Space Administration.
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 46
![Page 47: Images of HDF5](https://reader036.vdocuments.net/reader036/viewer/2022062323/568166e1550346895ddb15a4/html5/thumbnails/47.jpg)
www.hdfgroup.org
The HDF Group
Questions/comments?
Apr. 17-19, 2012 HDF/HDF-EOS Workshop XV 47