![Page 1: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/1.jpg)
BigDataCube:
Flexible, Scalable User Services
for Massive Spatio-Temporal EO DataSymposium „Neue Perspektiven der Erdbeobachtung“, Köln, 2019-nov-12
Dimitar Misev, Peter Baumann, Bang Pham Huu, Vlad Merticariu, Heike Hoenig,
Dimitris Bellos, Sven Jacobsen, Stefan Wiehle
Jacobs University, rasdaman GmbH, cloudeo AG, DLRTechnically co-sponsored by:
![Page 2: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/2.jpg)
![Page 3: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/3.jpg)
![Page 4: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/4.jpg)
![Page 5: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/5.jpg)
BigDataCube
Mission: flexible & scalable services for massive
spatio-temporal EOdatacubes
Goals:
– standards-based datacubes
– enable industry & research collaboration across boundaries
– serve data ready for use free resources for core business
Approach: public & commercial datacubes, federated
Benefit: enable novel 3rd party services fast, flexible, scalable
![Page 6: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/6.jpg)
BigDataCube: Partners
Jacobs University (project coordinator),
rasdaman GmbH
– datacube R&D, OGC + ISO + INSPIRE standardization
– Federation, security
cloudeo AG
– commercial geo-infrastructure
DLR Maritime Safety and Security Lab
– maritime wind & sea state products, S1 derived
![Page 7: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/7.jpg)
BigDataCube:
Public/Private Datacube Partnership
On-demand standards-based datacube analytics
– Public CODE-DE + DLR service + private cloudeo services
– S1 & S2 (~400 TB), DTM/DEM, wind speed & sea state
Jacobs University (lead), rasdaman GmbH, cloudeo AG, DLR
Goals: Advancing federation; security!
![Page 8: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/8.jpg)
rasdaman: Actionable Datacubes
= „raster data manager“: SQL + n-D datacubes– pioneered actionable datacubes; intl patents, publications
massively scalable datacube analytics engine– 2.5+ PB; 1000x parallelization; intercontinental federation
standards blueprint, reference implementation, awards– 28k+ downloads of open-source rasdaman
![Page 9: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/9.jpg)
rasdaman Full-Stack Architecture
external
archives
rasserver
databasefile system
rasdamangeo services
Web clients (m2m, browser)
distributed query processingNo single point of failure
alternative
storage
[SSTD 2013]
tile access
optional compression
![Page 10: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/10.jpg)
Dataset C
Parallel, Distributed Processing
1 query 1,000+ cloud nodes
Dataset D
Dataset B
Dataset A
max( (A.nir - A.red) / (A.nir + A.red) )
+ avg(B.green)
+ max( (C.red + C.green + C.blue) / 3 )
+ max( (D.nir + D.red) / 2 )
![Page 11: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/11.jpg)
Bring Your Favourite Client
Open standards users in comfort zone
– Map navigation: OpenLayers, Leaflet, ...
– Virtual globe: NASA WorldWind, Cesium, ...
– Web GIS: QGIS, ArcGIS, ...
– Analysis: GDAL, R, python, ...
![Page 12: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/12.jpg)
Result: Federations
CODE-DE: Sentinel-x
CreoDIAS
cloudeo AG: DEM
Alfred-Wegener-Institut: maritime products
Taiwan Datacube– Agro services on FORMOSAT, drones
Helmholtz-Zentrum Geesthacht (under work)
DWD: climate variables (planned)
European Earth Datacube Federation- single common geo information space
![Page 13: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/13.jpg)
Datacubes too large for yes/no decision– Ex: ECMWF climate timeseries
Role-based access control + triggers– Any user/role, any op, any shape down to single pixel
– Full admin control
Authentication & authorization– External identity provider (OGC Testbed-15) or locally
Quota: access, download, processing
Billing
Security
free priced
![Page 14: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/14.jpg)
Results: SAFE for Services & Analytics?
Server effort for analysis tasks?
Zip archive extra tool invocation for extracting image file
– subdirectories
JPEG (lossless) extra CPU cycles for pixel reconstruction
– Wavelets suboptimal for spatio-temporal subsetting
File granularity: 100x100km GB sizes
– Benchmarks [Furtado et al]: ~3 MB suitable
SAFE is archive format, not service format!
![Page 15: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/15.jpg)
Results: Standardization
OGC– Coverage Implementation Schema (CIS) 1.1
• Web Coverage Service (WCS) 2.1
– Web Coverage Processing Service (WCPS) 1.1
• Integrated data / metadata retrieval & processing
ISO TC211
– 19123-1 Abstract Coverage Model (under work)
– 19123-2 Coverage Implementation Schema
ISO 9075-15:2019 SQL/MDA (Multi-Dimensional Arrays)
INSPIRE: coverage compatibility with OGC/ISO
![Page 16: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/16.jpg)
rasdaman as SME Business Enabler
EOfarm/GR: Big Data Analytics for farmers
& water quality monitoring
SBI/DE: intra-field variation analysis service for farmers
CropMaps/DE startup:
SAR-based crop classification
and yield prediction
![Page 17: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/17.jpg)
Related Projects
H2020 LANDSUPPORT– integrated land resource management and agriculture & forestry practices
H2020 EOSC-hub– European Open Science Cloud
H2020 PARSEC– EO business accelerator
BMEL BigPicture– Rule-based classification of field health status
– in-field anomalies, frost damage, drought
BMBF DeepRain
![Page 18: BigDataCube: Flexible, Scalable User Services for Massive ......BigDataCube: Flexible, Scalable User Services for Massive Spatio-Temporal EO Data Symposium „Neue Perspektiven der](https://reader033.vdocuments.net/reader033/viewer/2022060523/6052aaf33b864e3c3541aa99/html5/thumbnails/18.jpg)
Summary
BigDataCube = public/private datacube federation
– Platform: rasdaman datacube reference implementation
„Making federations a commodity“
– Location transparency
– Security, quota, billing
– Performance, scalability
Datacube standardization progress
Contributing to European technology independence & visibility