guidelines for provenance esip 2015 summer meeting asilomar conference grounds monterey, ca tuesday,...

Guidelines for Provenance

ESIP 2015 Summer MeetingAsilomar Conference GroundsMonterey, CATuesday, July 14, 2015

Hook HuaNASA / JPL / Caltech

2ESIP 2015 Summer: Guidelines for Provenance

RELEVANT SCOPE OF PROVENANCE FOR EARTH SCIENCE

PROV-ES

2015-07-14


Organizations holding relevant content during project life cycle

• Data and derived products are archived and managed by DAACs• Preserving long-term usability, understandability and reproducibility requires

significant additional information• Such “Preservation Content” is typically held by several different organizations and

individuals during project lifecycle

2015-07-14

Instrument Teams / PI’s

Instrument Developer/

Manufacturer

Data gathering project (e.g., flight

project)

Interdisciplinary Data User / PIProduct Generation

Support Teams (SIPSs)

DAACs

Calibration Teams

Mission Operations Team

Validation Teams

USGS

NOAA

NASA Technical Report Server

(NTRS)

NASA Aeronautics and Space Database

(NA&SD)”

General users

International Partner Archives


Towards a Preservation Content Standard• U.S. Earth Science Information Partners (ESIP) Data Stewardship Committee

– January 2011 - Proposed development of “Provenance and Context Content Standard (PCCS)” enumerating all content items to capture completely the provenance and context

– June 2011 - PCCS matrix developed based on U.S. Global Change Research Program Workshop Report (1998), experience with NASA mission instrument teams and NOAA’s Satellite Products Documentation Guidelines

• NASA– November 2011 – Adopted PCCS matrix to develop NASA Earth Science Data

Preservation Content Specification– In place since November 2011; current version dated January 2013

• Requirement for new missions• Checklist for missions in operation • Has been used for data from several spaceborne instruments (e.g., HIRDLS on EOS Aura satellite,

GLAS on ICESat-1)

• ESA– May 2011 – Long Term Data Preservation Preserved Data Set Composition

• “Living document” - subject to review and change via configuration management process

• Relevant to multiple Data Stewardship Interest Area (DSIA) working groups in NASA’s Earth Science Data System Working Groups (ESDSWG)

2015-07-14


Preservation Content Specification• Preflight/Pre-Operations: Instrument/Sensor characteristics including pre-flight/pre-

operations performance measurements; calibration method; radiometric and spectral response; noise characteristics; detector offsets

• Science Data Products: Raw instrument data, Level 0 through Level 4 data products and associated metadata

• Science Data Product Documentation: Structure and format with definitions of all parameters and metadata fields; algorithm theoretical basis; processing history and product version history; quality assessment information

• Mission Data Calibration: Instrument/sensor calibration method (in operation) and data; calibration software used to generate lookup tables; instrument and platform events and maneuvers

• Science Data Product Software: Product generation software and software documentation• Science Data Product Algorithm Input: Any ancillary data or other data sets used in

generation or calibration of the data or derived product; ancillary data description and documentation

• Science Data Product Validation: Records, publications and data sets• Science Data Software Tools: product access (reader) tools.• Checklist: “metadata” about the above 8 categories showing how and where items in each

category are preserved

2015-07-14

7

Summary of PROV-ES• “If PCS is the what, then PROV-ES is the how”

• An interoperable provenance specification for use in Earth Science Data Systems to enable:

– Capturing the increasing amount of contextual processing information of Earth Science Data Records (ESDRs).

– Improving transparency, understanding, and trust of ESDRs .– Providing an interoperable representation of provenance for NASA EOS data products.– Reducing the barrier of entry to infusing provenance capture and consumption in Earth Science Data

Systems.

• Timeline– Started as an ESDSWG 2013 Working Group– Developed first OPM/PROV approach from ACCESS 2009 project– 2014 WG focused on taking 2013 results and developing proof of concept working implementations– 2015 WG focused on implementations by early adopters

• Infusion– Employed additional use cases from ESIP, NASA-funded projects, DAACs, and USGCRP/GCIS– Initial specification and implementations employed– Automatic PROV-ES generation infused into initial NASA data systems– Faceted search interface to explore PROV-ES records

2015-07-14 ESIP 2015 Summer: Guidelines for Provenance

8

2015 Working Group Activity• Recently started 2015 activities

– Additional DAAC use cases• E.g. Assess the provenance and approach needed for supporting legacy

datasets.– Looking into early adopter implementations– Developing a “provenance primer”

• Assess related work– ISO 19115-* Lineage– DataOne’s ProvONE– GCIS

• What to capture to represent science publications for trust, scientific verifiability, and reproducibility?

• What is the provenance of the science analysis, findings, publication, citations, etc?

• How best to implement and infuse provenance into science data systems?



Approach for PROV-ES• Maximize infusion potential• Leveraging standards (W3C PROV)• Maximize reuse of open source tools

and libraries supporting W3C PROV• Minimize customizations / extensions• Conform to set of conventions

– Analogous to CF

• Focuses on Science Data Systems• Broadening for end-to-end life cycle

– Follows PCS

2015-07-14

PROV-ES Implementation

• Interoperability with W3C PROV specification– Maximize reuse of available tools interoperable with

PROV• Examples include (but not limited to): ProvToolbox (Java), Prov

(Python), Prov-O-Viz (Javascript)

• Support additional attributes to W3C PROV to encode PROV-ES concepts

– PROV Earth Science mapping to existing W3C PROV terms

• Support domain-specific extended concepts• Provenance capture instrumented into

processing steps– Entities, activities, and agents are kept loosely-coupled to

enable more versatile provenance curation– E.g. instrumentation Python API for PROV-ES

• Ease infusion through (re)use of existing interoperable W3C PROV tools


11

PROV-ES Concepts and Properties


ES conceptBase

PROV concept Additional Attributes Notes

key name value type cardinality eos:collection prov:Entity

prov:type eos:collection literalprov:label xsd:string display name. typically same as short name.prov:location xsd:anyURI [0..1] A URL to the collection. e.g. landing page.eos:shortName xsd:string eos:longName xsd:string human-readable long name of datasetbibo:doi doi [0..1] DOI for the collectiongcis:sourceInstrument eos:instrument [0..*] A dataset was collected by using one or more instruments.eos:level xsd:string [0..1] http://science.nasa.gov/earth-science/earth-science-data/data-processing-levels-for-eosdis-data-products/eos:version xsd:string [0..1]

eos:granule prov:Entity prov:type eos:granule literalprov:label xsd:string human-readable nameprov:location xsd:anyURI [0..1] A URL to the granule.eos:partOfCollection eos:collection [0..1] a granule is part of a collectiongcis:sourceInstrument eos:instrument [0..*] A dataset was collected by using one or more instruments.eos:level xsd:string [0..1] http://science.nasa.gov/earth-science/earth-science-data/data-processing-levels-for-eosdis-data-products/eos:version xsd:string [0..1]

eos:product prov:Entity prov:type eos:product literalprov:label xsd:string human-readable nameprov:location xsd:anyURI [0..1] A URL to the granule.eos:partOfCollection eos:collection [0..1] a granule is part of a collectiongcis:sourceInstrument eos:instrument [0..*] A dataset was collected by using one or more instruments.eos:level xsd:string [0..1] http://science.nasa.gov/earth-science/earth-science-data/data-processing-levels-for-eosdis-data-products/eos:version xsd:string [0..1]

eos:file prov:Entity prov:type eos:file literalprov:label xsd:string human-readable nameprov:location xsd:anyURI [0..1] A URL to the file.

eos:platform prov:Entity prov:type eos:platform literalprov:label xsd:string human-readable namegcis:hasInstrument eos:instrument [0..*]

eos:instrument prov:Entity prov:type eos:instrument literalprov:label xsd:string human-readable namegcis:inPlatform eos:platformgcis:hasSensor eos:sensor [0..*]gcis:hasGoverningOrganization

prov:Organization [0..*]

eos:sensor prov:Entity prov:type eos:sensor literalprov:label xsd:string human-readable namegcis:inInstrument eos:instrument

eos:software prov:Entity prov:type eos:software literalprov:label xsd:string human-readable namegcis:implements eos:algorithm [0..*]eos:version xsd:string [0..1]

eos:algorithm prov:Entity prov:type eos:algorithm literalprov:label xsd:string human-readable namegcis:implementedIn eos:software [0..*]eos:describedBy bibo:Document [0..*]eos:version xsd:string [0..1]

eos:document prov:Entity prov:type bibo:Document literalprov:label xsd:string human-readable nameprov:location xsd:anyURI [0..1] A URL to the document.bibo:doi doi [0..1]eos:version xsd:string [0..1]

eos:processStep prov:Activity prov:type eos:processStep literalprov:label xsd:string human-readable nameeos:usesSoftware eos:software [0..*]eos:runTimeContext eos:runTimeContext [0..1]

eos:softwareAgent prov:SoftwareAgent prov:type prov:SoftwareAgent literalprov:label xsd:string human-readable nameid xsd:string [0..1]host xsd:string [0..1]

eos:runtimeContext prov:Entity prov:type eos:runtimeContext literalprov:label xsd:string human-readable nameeos:hasRuntimeParameter xsd:string [0..*]

eos:download prov:Activity prov:type eos:download literalsize xsd:long [0..*] bytes. transfer rate can be calculated using activity start and end time with size.

eos:publication prov:Entity A bounded physical representation of a body of information designed with the capacity (and usually intent) to communicate, and which has been made available to the public.

prov:type gcis:publication literalprov:location xsd:anyURI [0..1] A URL to the publication.bibo:doi doi [0..1] DOI of the publication.

http://www.w3.org/TR/2013/REC-prov-o-20130430/%23Entity

http://www.w3.org/TR/2013/REC-prov-dm-20130430/Overview.html%23term-attribute-label

http://www.w3.org/TR/2013/REC-prov-dm-20130430/Overview.html%23term-attribute-location

http://purl.org/ontology/bibo/doi

http://data.globalchange.gov/gcis.owl%23d4e872

http://science.nasa.gov/earth-science/earth-science-data/data-processing-levels-for-eosdis-data-products/























http://www.w3.org/TR/2013/REC-prov-o-20130430/%23Organization










http://bibliontology.com/bibo/bibo.php


http://bibliontology.com/bibo/bibo.php




http://www.w3.org/TR/2013/REC-prov-o-20130430/%23Activity


http://www.w3.org/TR/2013/REC-prov-o-20130430/%23SoftwareAgent




http://www.w3.org/TR/2013/REC-prov-o-20130430/%23Activity


http://data.globalchange.gov/gcis.owl%23Publication



12

GUIDELINES: PROVENANCE FOR EARTH SCIENCE


13

One Size Doesn’t Fit All• No single provenance specification to meet all

NASA EOSDIS needs– Mission-specific adaptions vary– E.g GCIS information model effort making great

progress on representing various reports and datasets

• Define a minimum basis set– Use case-driven

• Defined approach for easily adding additional attributes


14

Activities• Examples of activities

– Processing step• A running PGE with inputs and outputs data

– Analysis• A science user performance an analysis with data generating

plots for a publicaiton

• Capturing internal provenance (eager)– Details only known at time of activity within the

activity– Capture appropriate provenance at time of activity

• Capturing external provenance (lazy)– Extrinsic characteristics such as data prov:used and

data prov:wasGeneratedBy2015-07-14 ESIP 2015 Summer: Guidelines for Provenance

15

Artifacts• Data resources are prov:Entity

– Datasets– Collections– Granules– Software

• prov:Entity or prov:Plan– Algorithm

• Prov:Entity or prov:Agent– Sensor– Platform

• Agents have a role in generating new data• Plan are a set of steps intended to be carried our by agents2015-07-14 ESIP 2015 Summer: Guidelines for Provenance

16

Lexicon Mappings• Differences in nomenclature will exists across organizations (and within!)• Good approach by GCIS Lexicons


17

Authoritative Identifier • Many unique identifier approaches exist• Who can be the authoritative organization for curating

different resource types?• Disambiguation needs coordination across all resource types

– Identifier for agents• People• Organizations

– Identifier for entities• “Datasets” (DOIs)• “Dark data”• Software• Algorithms• Journal articles (DOIs)

– Identifiers for Activities• Processing Steps• Analysis


18

Level of Detail• What level of detail provenance to capture?

– Source provenance– Transformative provenance

• Should be use case-driven– “A machine generate a massive amount of provenance, but a scientist

can manually generate a small focused set that is highly useful.”

D1 D2

P1 wasGeneratedByused

wasDerivedFrom



W3C PROV Data Model Types and Relations

2015-07-14

Source: http://www.w3.org/TR/prov-dm/#prov-dm-types-and-relations

20

Linking Provenance Concepts• Support real-world scenarios of independent

provenance records– Status quo of limited use of coordinated end-to-

end identifiers and provenance bundles• Support loosely-coupled and linked data

provenance resources• Ingest distributed provenance traces• Dynamically assemble provenance bundles

– Generate a reachability graph from any given provenance resource

– Via identifiers, with fallback to search terms2015-07-14 ESIP 2015 Summer: Guidelines for Provenance

21

Linked Bundles• Represent provenance in linked bundles• Want each organization to generate their own

provenance bundles• Enable linked relationships across these

bundles


L0 L1P1wasGeneratedByused

L2P2wasGeneratedByused

Organization 1 Organization 2

Foster provenance representation in linked form for reuse

org1:bundle1 org2:bundle2

22

Interactive Visualization for Linked Relationships

NCA3 Mapping into PROV-ES from GCIS API


23

Archiving Provenance Traces• Generate provenance traces (e.g. into files)

– Enable archiving– Enable (re)ingestion into provenance systems

• Enables decoupled consumption of provenance– Visualization– Analytics– Reproducibility

• Store as provenance as collocated serializations

• Provenance in ISO 19115-* Lineage2015-07-14 ESIP 2015 Summer: Guidelines for Provenance

From Instrumentation to Discovery

Processing Step prov-es

provenance

Provenance Discovery

e.g. Faceted Search

Prov storage

e.g. Elastic Searchused

datasets

generated

datasets

24

• PROV-ES “log lines” generated in JSON• Distributed PROV-ES traces crawled and ingested into Elastic Search datastore• Support discovery of multi-dimensionality of provenance through Faceted Search

– Enables drilling down across end-to-end provenance concepts such as datasets, processing steps, algorithms, documentation, instruments, sensors, platforms, etc.

Provenance Ingest

json

https://prov-es.jpl.nasa.gov/beta


Mapping From Other Sources

25

• Support Faceted Search of NCA3 Report provenance by mapping from GCIS into PROV-ES

NCA3

MappingScript

Provenance Discovery

e.g. Faceted Search

Prov storage

e.g. Elastic Search

Provenance Ingest

json

https://prov-es.jpl.nasa.gov/beta


26

Provenance for Discovery• What should the

provenance “atomic unit” be for discovery purposes?– Search result set

• As provenance bundles• As provenance triple

• Preference to handle provenance discovery at unit of “triples” instead of entire bundles– Entities– Activities– Agents– Relationships


Example result set from PROV-ES Faceted Search


WHERE WE ARE

2015-07-14


Initial Prototype Infusions

2015-07-14

• AIST 2011 on geodetic InSAR processing– PROV-ES production in NASA AIST’s Advanced Rapid

Imaging & Analysis for Monitoring Hazards (ARIA-MH) Hybrid Cloud Science Data System (HySDS)

• MEaSUREs 2012 on A-Train atmospheric science data fusion– PROV-ES production in NASA MEaSUREs’s “A Multi-Sensor

Water Vapor, Temperature and Cloud Climate Data Record”

• Mapping NCA3 provenance from GCIS into PROV-ES• OCO-2 L2 Full Physics Processing

– PROV-ES traces from OCO-2 L2 production in AWS Cloud Computing


DAAC Early Adopters• ESDSWG Provenance Working Group activities

– Strong interest in implementing PROV-ES with real use cases

2015-07-14

Organization POC Use Case

ORNL Yaxing Wei

A science user wants to find a soil dataset from the ORNL DAAC. He does a search and finds a few. He wants to choose the best one (e.g. based on most recent soil measurements,

using a certail soil classification legend, and/or derived from a dataset he trusts).

LPDAAC Jason Werpy

NSIDC Donna Scott Legacy datasets

GHRC Helen Conover

ASF Kirk Hogenson

ALOS PALSAR SAR data granules, including RTC (terrain corrected) and (potentially) intererometric products. Users will

want to find data over areas of interest including interferometric pairs, and granules from other data sets in the

same area.

MODAPS/LAADS Curt Tilmes

SEDAC Bob Downs

AMSR-E SIPSNSIDC DAAC

Helen Conover (AMSR-E SIPS), Doug Fowler (NSIDC DAAC)

Standard product generation - Level 2B Rain product from Level 2A Brightness Temperatures. Output collection is

AMSR-E/Aqua L2B Global Swath Surface Precipitation GSFC Profiling Algorithm (10.5067/AMSR-E/AE_Rain.003). This data

is produced at the SIPS and archived & distributed from the DAAC


ESIP 2015 Summer: PROV-ES Hack-a-thon• PROV-ES Hack-a-thon

– http://commons.esipfed.org/node/8219– Wednesday, July 15, 2015 - 1:30pm to 3:00pm– Summer Meeting 2015– Session Type: Breakout– Room Location: Merrill Hall– Expertise Level: Beginner

• Hands-on PROV-ES examples• Deploy your own PROV-ES Faceted Search on a

virtual machine!2015-07-14

http://commons.esipfed.org/node/8219




PROV-ES Faceted Search• Deployable as a service in a virtual machine• Come to our PROV-ES Hack-a-thon session!

2015-07-14

Resources• PROV-ES Faceted Search and API

– https://prov-es.jpl.nasa.gov/beta/• ESDSWG PROV-ES Working Group

– Wiki• https://wiki.earthdata.nasa.gov/display/ESDSWG/Provenance+for+Earth+Science+(PROV-ES)+Working+

Group (login required)

– Mailing list• [email protected]• Subscribe at https://lists.nasa.gov/mailman/listinfo/esdswg-prov-es

– Telecon notes• https://docs.google.com/document/d/1mw0ROVcW4-7W3IfZk2pizM1k6sHM2hgmRTvChrYtpO8/edit?pli=1

– Use Case Mapping• https://docs.google.com/spreadsheet/ccc?key=0AlQ95ca89UmYdEJXX05NVDRhRGRWV09hcmNENnFROHc#

gid=0

– PROV-ES Mapping to W3C PROV• https://docs.google.com/spreadsheets/d/1gsyuKY-xD1yQaHD6QVB6vJ3YmDWL9WKPiSopW8BtuB8/edit#gi

d=0

• NASA Earth Science Data Preservation Content Specification (NASA ESD PCS)– http

://earthdata.nasa.gov/sites/default/files/field/document/423-SPEC-001_NASA%20ESD_Preservation_Spec_OriginalCh01_0.pdf

• Provenance and Context Content Standard– http://wiki.esipfed.org/index.php/Provenance_and_Context_Content_Standard


http://prov-es.jpl.nasa.gov/beta/

https://wiki.earthdata.nasa.gov/display/ESDSWG/Provenance+for+Earth+Science+(PROV-ES)+Working+Group



mailto:[email protected]

mailto:[email protected]

https://docs.google.com/document/d/1mw0ROVcW4-7W3IfZk2pizM1k6sHM2hgmRTvChrYtpO8/edit?pli=1

https://docs.google.com/document/d/1mw0ROVcW4-7W3IfZk2pizM1k6sHM2hgmRTvChrYtpO8/edit?pli=1

https://docs.google.com/spreadsheet/ccc?key=0AlQ95ca89UmYdEJXX05NVDRhRGRWV09hcmNENnFROHc%23gid=0



https://docs.google.com/spreadsheets/d/1gsyuKY-xD1yQaHD6QVB6vJ3YmDWL9WKPiSopW8BtuB8/edit%23gid=0



http://earthdata.nasa.gov/sites/default/files/field/document/423-SPEC-001_NASA%20ESD_Preservation_Spec_OriginalCh01_0.pdf




http://wiki.esipfed.org/index.php/Provenance_and_Context_Content_Standard

http://wiki.esipfed.org/index.php/Provenance_and_Context_Content_Standard

guidelines for provenance esip 2015 summer meeting asilomar conference grounds monterey, ca tuesday,...

Documents

distribution of data

data flow

multiple data stewardship

raw instrument data

eosdis data centers

long term data preservation

science operations

relevant content