data are from mars, tools are from venus
TRANSCRIPT
![Page 1: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/1.jpg)
JL2-SESIP-0717
Data Are from Mars,
Tools Are from Venus
H. Joe Lee ([email protected])
The HDF Group
This work was supported by NASA/GSFC under
Raytheon Co. contract number NNG15HZ39C
All images used in this presentation are from
autodraw.com for public use.
![Page 2: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/2.jpg)
JL2-SESIP-0717
2
No “Earth” in title?
“Men Are from Mars, Women are from Venus”
- John Gray
![Page 3: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/3.jpg)
JL2-SESIP-0717
3
Are data from Mars?
Why can’t I use Earth tools?Correct geo-referencing
![Page 4: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/4.jpg)
JL2-SESIP-0717
4
Are tools from Venus?
Why can’t I open Earth data?
![Page 5: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/5.jpg)
JL2-SESIP-0717
5
Data Producers from Mars
• I want my data slim and efficient.
• How can I save money in managing data?
Tool Developers from Venus• I make my tool work for popular data first.
• Can I make money by supporting your data?
![Page 6: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/6.jpg)
JL2-SESIP-0717
6
Result: Frustrated Users
![Page 7: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/7.jpg)
JL2-SESIP-0717
7
We (HDFEOS.org) can help.
![Page 8: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/8.jpg)
JL2-SESIP-0717
8
• Hierarchical Data Format (HDF)
• Network Common Data Format (netCDF)
• Geospatial Tagged Image File Format
(GeoTIFF)
• Keyhole Markup Language (KML) /
zipped KML (KMZ)
• Comma-separated values (CSV)
• etc.
We identify gaps in File Formats
![Page 9: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/9.jpg)
JL2-SESIP-0717
9
• hdf
• netcdf-C
• netcdf-Java
• HDF – Earth Observing System (hdf-eos)
• Climate and Forecast Metadata (CF) conventions
• Geospatial Data Abstraction Library (GDAL)
• etc.
We identify gaps in Libraries
![Page 10: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/10.jpg)
JL2-SESIP-0717
10
• Microsoft Excel
• Esri ArcGIS
• Google Earth
• MATLAB
• Python
• Interactive Data Language (IDL)
• Panoply
• Integrated Data Viewer (IDV)
• HDFView
• h5dump
• Etc.
We identify gaps in Tools
![Page 11: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/11.jpg)
JL2-SESIP-0717
11
• Open-source Project for a Network Data
Access Protocol (OPeNDAP)
• Web Map Service (WMS)
• Web Map Tile Service (WMTS)
• Web Coverage Service (WCS)
• etc.
We identify gaps in Services
![Page 12: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/12.jpg)
JL2-SESIP-0717
12
• File conversion
• Libraries and tools usage
• NASA HDF product specific examples
• Demo services (e.g., Hyrax*, THREDDS**)
AND we provide Solutions…
*Hyrax is the data server from OPeNDAP.
**Thematic Real-time Environmental Distributed Data Services
![Page 13: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/13.jpg)
JL2-SESIP-0717
13
• Make HDF5 data work with
– GDAL
– netCDF
– Hyrax/THREDDS
• Don’t forget a few key CF conventions.
• Follow DIWG* recommendations.
Suggestions for data producers
*Data Interoperability Working Group
![Page 14: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/14.jpg)
JL2-SESIP-0717
14
• Download and test NASA HDF products.
• Support them natively.
• Support augmentation.
– VRT* in GDAL
– NcML** in netCDF
• Support 3D visualization for data in the air.
Suggestions for tool developers
*Virtual Dataset in XML format
**netCDF Markup Language
![Page 15: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/15.jpg)
JL2-SESIP-0717
15
3D Visualization
![Page 16: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/16.jpg)
JL2-SESIP-0717
16
• Try OPeNDAP first. CSV may be enough.
• Try netCDF conversion / augmentation.
• Correct metadata with NcML / VRT.
• Try GEE* instead of GDAL.
• Use CMR** wisely.
Suggestions for end-users
*GDAL Enhancement for ESDIS project
** Core Metadata Repository
![Page 17: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/17.jpg)
JL2-SESIP-0717
17
• Hadoop / Spark (streaming) / Dask
• Parquet / Arrow
• Elastic Search / Kibana
How about Big (fast) data?
![Page 18: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/18.jpg)
JL2-SESIP-0717
18
Streaming Swath
Hadoop + Spark Streaming + Elasticsearch & Kibana
S3 + AWS Lambda + Elasticsearch & Kibana
![Page 19: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/19.jpg)
JL2-SESIP-0717
19
• scikit-learn / keras / h2o.ai
• Please contact us at
[email protected] if you’d like to see
examples on machine learning.
Future: (Deep) Machine Learning?
![Page 20: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/20.jpg)
JL2-SESIP-0717
20
Deep Learning: Amazon
Rekognition
![Page 21: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/21.jpg)
JL2-SESIP-0717
21
Alexa, what does MODIS see now?
Alexa: I can recognize “Alps.”
![Page 22: Data Are from Mars, Tools Are from Venus](https://reader034.vdocuments.net/reader034/viewer/2022052514/5a64d07d7f8b9abb6f8b4a0b/html5/thumbnails/22.jpg)
JL2-SESIP-0717
22
This work was supported by
NASA/GSFC under Raytheon Co.
contract number NNG15HZ39C