data flow operations concepts – product lifecycle evolution · slide 8. eo product lifecycle –...
TRANSCRIPT
ESA UNCLASSIFIED - For Official Use
December 2018
Data Flow Operations Concepts –Product Lifecycle Evolution
Slide 2
EO Product Lifecycle – Systematic Production
Downlink AcquisitionOnboard Sensing Processing
Archiving
Dissemination
Publication DeletionReprocessing
Archiving
Dissemination
Publication
A Simplistic Data Provider View of
Systematic Production
Auxilliary Data Collection
Slide 3
Downlink AcquisitionOnboard Sensing Processing
Archiving
Dissemination
Publication DeletionReprocessing
Archiving
Dissemination
Publication
Processing
Archiving
Dissemination
Publication DeletionReprocessing
Part of a complex
value adding chain
EO Product Lifecycle – Systematic Production
Slide 4
EO Product Lifecycle - Processing
Downlink AcquisitionOnboard Sensing Processing
Archiving
Dissemination
Publication DeletionReprocessing
Product Consolidation Data Quality Correction
Improvements in Sensor
Understanding
Improvements in Scientific
Understanding
Cost / Quality / Schedule
Level 0 Level 1 Level 2
Auxilliary Data Collection
Result of many trade off analysis and experiences Cost / Quality / Schedule
Slide 5
EO Product Lifecycle – Current Production Baselines
Catalogue Completeness
Well indentifiedAlgorithm and
Processor Versions
Data Format andArchive Integrity
Quality Control and Disclaimers
Fully documented Input/Output
Relying on the correct and reliable configuration of complete processing chain
Slide 6
EO Product Lifecycle – Current Implementation of Catalogue Completeness - APIs and listsProducts() -> uuid, md5, id/filename … metadataDeletedProducts() -> uuid, md5, id/filename … deletionCauseCatalogueView: .csv lists
Slide 7
EO Product Lifecycle – Current Implementation -Ingestion vs. Synchronisation in the DHR Network
BE FE
DHR
HUBX
Product UUID, checksum aremaintainedvia synchronisation… Integrity and provenance generically assured
No assurance of provenance
S1A_GRD_XYZ
S1A_GRD_XYZ
Slide 8
EO Product Lifecycle – Current Implementation –Already Suffering
BE FE
DHR
LTA Retrieval
Already suffering- .zip processing of archive data differs,
MD5 no longer uniquePossibilities are “workarounds”- Alternative MD5’s for one UUID - UUID generation based on core
components of the productAs many drawbacks as benefits
S1A_GRD_XYZ
S1A_GRD_XYZ
Slide 9
EO Product Lifecycle – Evolution to On Demand Processing
Downlink AcquisitionOnboard Sensing Processing
Archiving
Dissemination
Publication DeletionReprocessing
Archiving
Dissemination
Publication
On Demand
On Demand Processing – freeing the obligation to manage the full long term archive of all L1/L2
Auxilliary Data Collection
Slide 10
To sustain the current demand the distribution function will continue to need to manage rolling archive cache of latest “fresh” data
Full catalogue of processed data will remain available
~No long term archiving of all product levels, ….~As upgrade of processors are introduced they will be immediately
available for ad-hoc historic data requests, complete baselines no longer assured
?Bug fixes “versions” of data products will invalidate definitive list?No “guarantee” the same production for subsequent processing using
“same” input and processors (incl. aux data)
EO Product Lifecycle – Evolution to On Demand Processing - Impacts
Slide 11
Quite a few Provenance Starting Points
e.g. Provenance Management for Modern Distributed Systems. In Proceedings of USENIX Theory and Practice of Provenance, London, England, July 9–13, 2018
Centralised vs. Decentralised Approaches
Slide 12
Initial RequirementsBaseline RequirementsProvide per product proof of provenance for Sentinel dataflows
a record of all processing steps (ACQ, PRO, DISSEM, LTA ..)
Provide a secure, definitive reference for valid Sentinel data transformations
Attributable to contracted/trusted partnersAllow trusted partners to invalidate individual productsAllow trusted partners to invalidate entire product baselinesNo “lock-in” for trusted partnersAllow to document reason of invalidityAllow users to “certify” the validity of individual productsAllow users to “certify” the validity of lists of productsAllow users to identify product replacementsFully support the Sentinel on-demand production scenariosDistributed – Deployable in multiple facilities and cloud
infrastructures
Slide 13
Initial Requirements
Potential RequirementsApplicable to mission planningAllow to identify production gapsAuditable timelineInput for end-to-end dashboardExtendable to per product proof of provenance for Collaborative and Downstream individual products
Service Requirements
High availability, Robust to maintenance, network shortfall
Scalable - Sentinel-Scale, 100’s*10^6 products, many transformations per product
Including all components/operations necessary for management of digital signature
Capable to support data migration from initial sources (.csv etc.)
Open Source, API libraries / scriptable interfaces for integration to processor workflows
Slide 14
Way Forward
Product Lifecycle Operations Concept TN - 01/2019For comment and review - 02/2019
Hot topic for future prototype and interoperability demonstration