05 december, 2002hdf & hdf-eos workhop vi1 esdis status richard ullman esdis project...
Post on 11-Jan-2016
218 Views
Preview:
TRANSCRIPT
05 December, 2002 HDF & HDF-EOS Workhop VI 1
ESDIS Status
Richard Ullman
ESDIS Project
Richard.E.Ullman@nasa.gov
05 December, 2002 HDF & HDF-EOS Workhop VI 2
Outline
• ESDIS General Status
• HDF-EOS Plans
• Website http://hdfeos.gsfc.nasa.gov
05 December, 2002 HDF & HDF-EOS Workhop VI 3
ESDIS Science Data Services Today
• EOSDIS provides support for high data volumes from Aqua, Terra, and Landsat 7, and continues to support QuikSCAT, ACRIMSat, SAGE-III, JASON and pre-EOS-era data including TRMM, UARS, TOPEX/Poseidon, RADARSat, and others. – EOSDIS finalizing preparations to support ICESat and SORCE
– In all, EOSDIS is providing data processing, archival, and/or distribution for over 15 Earth science satellite missions.
• EOSDIS has set a new benchmark for data management. The total volume of the science data in our archives totals over 2 Petabytes. Since 1998, the science data volume managed by the EOSDIS has increased eight-fold, and continues to grow at a rate of over 2 Terabytes per day
• In FY02, EOSDIS provided more than 16 million data and information products to over 1.8 million individuals.
05 December, 2002 HDF & HDF-EOS Workhop VI 4
FY94/95 FY96 FY97 FY98 FY99 FY00 FY01 FY02
Science Operations Supported
Heritage Missions
T/P, UARS, TOMS, ERBE
Terra ACRIMSatTRMM
Landsat 7 QuikSCAT
Aqua GRACE
SAGE III JASON
SeaWiFS Radarsat
Products Distributed
15 Million
Archive Growth
Archive Volume has doubled each year for the past three years
2 Petabytes
Science Operations Timeline
05 December, 2002 HDF & HDF-EOS Workhop VI 5
EOSDIS DAAC Data delivery FY’02October 1, 2002 through September 30, 2002
• Over 1.8 million distinct users
• 16,013,980 products delivered
• Notes to charts on following slides:– Distinct users includes users accessing DAAC web pages,
including web-crawlers
– Distinct user type is based on email addresses of users or URLs
– Product delivered is defined as the smallest deliverable unit of data
– Product delivery breakdown is based on email addresses of users receiving ECS and Non-ECS data
– “FTP Delivery” are to URLs not mapped to specific domains
– “Foreign Other” includes foreign email addresses whose country is known but domain-type could not be determined
05 December, 2002 HDF & HDF-EOS Workhop VI 6
16,013,980 Data Products DeliveredOctober 1, 2001 - September 30, 2002
05 December, 2002 HDF & HDF-EOS Workhop VI 7
Over 1.8 Million Distinct UsersOctober 1, 2001 - September 30, 2002
05 December, 2002 HDF & HDF-EOS Workhop VI 8
ESDIS Status
• After years of development, ECS is operational and generally recognized as successful.
• Primary ECS development contract is essentially completed.
• ECS Maintenance and Development (EMD) will emphasize maintenance more than development.
• Already the majority of ESDIS budget has shifted from development to operations.
• Still two major areas of new capability (see posters at AGU for more information):– Data Pools
– ECHO
05 December, 2002 HDF & HDF-EOS Workhop VI 9
HDF-EOS Plans
• HDF-EOS 5 development is nearly complete.
• Continue to maintain, port to newer operating systems, bug-fix.
• Need the advise of this community workshop - we will discuss this afternoon.– What tools or capabilities are now needed?
• HDF-EOS 2 and HDF-EOS 5
– When is it the right time to press EOS science teams to migrate to HDF-EOS 5?
– What steps should NASA take to facilitate?
05 December, 2002 HDF & HDF-EOS Workhop VI 10
Data Pools Concept: The Right Data, the Right Way, Right Now!
Value-Added Providers
Data Providers
Data TailoringWorkflow
Management
End Users
User-definedViews, Presentations and Data Access Requests
Dynamic Web and FTP Data Views,User-specified Data Access
Other Data Pools
Other Data Pools
Data Producers
DataServices
DataItems
Data ServiceDevelopers
GeoPIPE
PIPE = Personalized Information Presentation Engine
05 December, 2002 HDF & HDF-EOS Workhop VI 11
General Capabilities– The Right Data: Data Location
• Groups, themes, bookmarks and views
• Navigation and machine-based location
• Science views (e.g., science metadata)
• Applications views (e.g., OGIS coverage server)
• Location aids (e.g., geopolitical overlays)
• External location-support services
– The Right Way: Data Tailoring• Data reduction, manipulation and reformatting services
• Virtual data products
• Workflow management and execution monitoring
• External tailoring services
– Right Now: Rapid Access• Low latency data transfers
• Secure remote file access
• On-the-fly data compression
• Automated request routing and load balancing
• Near real-time data
05 December, 2002 HDF & HDF-EOS Workhop VI 12
EOS ClearingHOuse (ECHO) http://eos.nasa.gov/echo
• ECHO is a metadata clearinghouse– A single Internet portal for Earth science metadata search
– Index of data provider inventory-level data holdings metadata.
• .ECHO is a data order broker– Forwards orders for data discovered to the data providers to fill.
– Data providers retain customer fulfillment service
• ECHO is a data service broker– Registered service are associated with registered datasets
– Four kinds of service association • Advertised, Context Passing, Brokered, Order Option
• ECHO is an open client API for custom user clients – The EOS Data Gateway (EDG) is ESDIS’ ECHO client
05 December, 2002 HDF & HDF-EOS Workhop VI 13
ECHO Data Providers
• Current ECHO Data Providers– EOSDIS Core System DAACs
(17% thus far)• EDC Land Processes DAAC
• Goddard DAAC
• NSIDC DAAC
– ORNL DAAC (100%)
• ECHO holds the metadata for over 3 million granules, and growing
ECHO Metadata Holdings
0
500,000
1,000,000
1,500,000
2,000,000
2,500,000
3,000,000
3,500,000
4,000,000
Jan-06 Feb-06
Mar-06 Apr-06 May-06
Jun-06 Jul-06 Aug-06
Sep-06
Date
ORNL_DAAC
NSIDC_ECS
LP_ECS
GSFC_ECS
05 December, 2002 HDF & HDF-EOS Workhop VI 14
HDF-EOS Tools and Information Web Sitehttp://hdfeos.gsfc.nasa.gov
Richard.E.Ullman@nasa.gov
05 December, 2002 HDF & HDF-EOS Workhop VI 15
http://hdfeos.gsfc.nasa.gov
05 December, 2002 HDF & HDF-EOS Workhop VI 16
Website status
• A resource for discovering about hdf-eos in particular.– This workshop series’ presentation archive
– Links to hdf-eos tools
• Site has been revamped according to comments received at the last workshop.– Tools download page now has opportunity for user feedback.
– Workshop presentations are keyword searchable
• New features planned– Better introductory material.
• Post and organize documentation of HDF-EOS.
• Better navigation to NCSA site for HDF
– Host hdf-eos “web forum” • Incorporate the eostools@eos.nasa.gov listserv
05 December, 2002 HDF & HDF-EOS Workhop VI 17
HDF-EOS Profile
Richard Ullman
ESDIS Project
Richard.E.Ullman@nasa.gov
05 December, 2002 HDF & HDF-EOS Workhop VI 18
History
• September 1993, HDF adopted as baseline standard for EOSDIS Core System standard data product generation, archival, ingest, and distribution capabilities
• Dec. 94 - ECS Engineering Support Directive to create HDF-EOS
• June 1996, HDF-EOS v1.0 library released
• Upgrades every 6 mo., – Current version 2.8 on HDF 4
• HDF5 support (called HDF-EOS 5) beginning November 2000– Current version 5.1.3 on HDF5
05 December, 2002 HDF & HDF-EOS Workhop VI 19
HDF-EOS Data Objects
• Point
• Swath
• Profile (Swath subtype)
• Grid
• Zonal (HDF-EOS 5 only)
05 December, 2002 HDF & HDF-EOS Workhop VI 20
HDF-EOS 5
• Based on HDF5, a complete rewrite of HDF4 with a different interface. – First released in 2000.
• Designed to ‘resemble’ HDF-EOS 2 to the maximum extent possible. – Support same data structures
– Added prefix ‘HE5_’ to HDF-EOS 2 functions.
– Doesn’t preclude HDF5 functionality.
– Data Type changes, e.g. INT64 -> H5T_NATIVE_LONG
05 December, 2002 HDF & HDF-EOS Workhop VI 21
HDF-EOS 5 Functionality
• Basic File I/O
• Fill Values
• Compression
• Chunking/Tiling
• Swath Interface
• Grid Interface
• Point Interface
• Profile Interface
• Global (File), Group & Local Attributes
• External Data Files
• Subsetting
• Unix/Linux Support
• Threadsafe Version
• FORTRAN, C, C++
• General Table Interface (proposed)
05 December, 2002 HDF & HDF-EOS Workhop VI 22
The new ADDITIONAL Group has global (file) attributesThe new functionality is added to the EH(utility) interface.
STRUC. METADATA
Global (file) Attributes
HDFEOS
SWATH GRID POINT
Root -- “/”
ADDITIONAL
HDFEOSINFORMATION
Top Level of HDF-EOS 5
05 December, 2002 HDF & HDF-EOS Workhop VI 23
SwathName
Data Fields Geolocation Fields
Attribute Dataset
DataField.1
DataField.n
Longitude Latitude
Group Attribute<DataFields>:<AttrName>
Global Attribute<SwathName>:<AttrName>
Local Attribute<FieldName>:<AttrName>
CoLatitudeTime
Shaded Objects are implemented in a fixed way. User doesn’t have
direct access via the interface
Each Data Field can have Attributes and/or Dimension Scales
Profile Fields
ProfileField.1
ProfileField.n
Group
Swath Structure
05 December, 2002 HDF & HDF-EOS Workhop VI 24
Latitude Longitude Tempera-ture oC
Dew PointoC
61.12 -149.48 15.00 5.00
45.31 -122.41 17.00 5.00
38.50 -77.00 24.00 7.00
38.39 -90.15 27.00 11.00
M M M M
HDF-EOS Point
• Intended use– Discrete points in time and/or
location.
– Table of data linked to table of geographic information.
– 8-level Hierarchical, each level may contain indices to the level below
05 December, 2002 HDF & HDF-EOS Workhop VI 25
Latitude Longitude Buoy_ID25.2645 091.2564 012622.3549 -93.4657 356423.2564 -89.2546 1256
Buoy_ID
Time WaveHeight(ft)
Temp(C)
0126 01:26 2.54 18.40126 05:56 3.58 18.23564 06:28 12.64 16.41256 08:12 7.58 17.11256 09:58 7.76 17.20126 09:59 4.23 20.13564 10:16 10.23 17.5
HDF-EOS Point
• Hierarchical links:– Every level in a Point data set
must be linked into the hierarchy.
– Before two levels can be linked, a link field must exist
05 December, 2002 HDF & HDF-EOS Workhop VI 26
HDF-EOS Swath
• Intended use– Across track scanning
instruments.
– Sounding instruments
– Level 1: Geolocated Sensor Units
– Level 2: Geophysical Parameters
InstrumentPath
Scan Lines
Along Track
InstrumentPath
Along Track
05 December, 2002 HDF & HDF-EOS Workhop VI 27
“Longitude”
“Latitude”DimensionName: GeotrackSize: 21
DimensionName: GeotrackSize: 21
DimensionName: ScanSize: 16
DimensionName: ScanSize: 16
DimensionName: TrackSize: 42
DimensionName: TrackSize: 42
Map1DataDimension: “Track”
Geodimension: “Geotrack”Offset: 1
Increment: 2
Map1DataDimension: “Track”
Geodimension: “Geotrack”Offset: 1
Increment: 2
“Brightness Temperature” “Time”
HDF-EOS Swath
Data Fields Geolocation Fields
05 December, 2002 HDF & HDF-EOS Workhop VI 28
HDF-EOS Grid
• Intended use– Variables mapped on uniform
space-time grid scales
– Level 3 - Gridded single measurement parameters
– Level 4 - Modeled or derived from multiple measurements
05 December, 2002 HDF & HDF-EOS Workhop VI 29
HDF-EOS Grid
XdimSize: 2000
YdimSize: 800
Projinfo
AltitudeSize 30
05 December, 2002 HDF & HDF-EOS Workhop VI 30
HDF-EOS Grid
• Projections Supported
• Geographic
• Transverse Mercator
• Universal Transverse Mercator
• Hotine Oblique Mercator
• Space Oblique Mercator
• Polar Stereographic
• Lambert Azimuthal Equal Area
• Lambert Conformal Conic
• Polyconic
• Interrupted Goode’s Homolosine
• Integerized Sinusoidal
• Compression Methods
• Run-Length Encoding
• Adaptive Huffman
• Gzip
05 December, 2002 HDF & HDF-EOS Workhop VI 31
Product Levels
• Level 0 - Reconstructed, unprocessed instrument/pay-load data at full resolution; any and all communications artifacts, e.g., synch. frames, communications headers, duplicate data removed.
• Level 1A- Reconstructed, unprocessed instrument data at full resolution, time-referenced, and annotated with ancillary information, including radiometric and geometric calibration coefficients and georeferencing parameters, e.g., platform ephemeris, computed and appended but not applied to the Level 0 data.
• Level 1B - Level 1A data that have been processed to sensor units (not all instruments will have a Level 1B equivalent).
• Level 2 - Derived geophysical variables at the same resolution and location as the Level 1 source data.
• Level 3 - Variables mapped on uniform space-time grid scales, usually with some completeness and consistency.
• Level 4 - Model output or results from analyses of lower level data, e.g., variables derived from multiple measurements.
05 December, 2002 HDF & HDF-EOS Workhop VI 32
Resources
– HDF-EOS on the web:• http://hdfeos.gsfc.nasa.gov/
• http://newsroom.gsfc.nasa.gov/sdptoolkit/toolkit.html
– HDF and HDF5 on the web:• http://hdf.ncsa.uiuc.edu/
– HDF-EOS and HDF via email:• eostools@eos.nasa.gov
• hdfhelp@ncsa.uiuc.edu
top related