semantic extensions for scientific workflows on the grid

Semantic Extensions for Scientific Workflows on the GridBertram Ludscher([email protected])Associate ProfessorDept. of Computer Science & Genome CenterUniversity of California, Davis

FellowSan Diego Supercomputer CenterUniversity of California, San DiegoUC DAVISDepartment ofComputer Science

ISGC2005, April 25-29, 2005SWDBAug 29, 2004http://seek.ecoinformatics.org

OverviewScience Environment for Ecological Knowledge (SEEK)

Scientific Workflows What are they?Why do we need them?

The Kepler Scientific Workflow System

Adding Semantics to Scientific Workflows

Science Environment for Ecological KnowledgeLarge collaborative NSF/ITR (2002-2007)

Bringing together ecologists, IT experts, CS researchers,

SEEK.ecoinformatics.org


SEEK: Multidisciplinary research to facilitate

Access to ecological, environmental, and biodiversity dataEnable data sharing & re-useEnhance data discovery at global scales

Scalable analysis and synthesis Taxonomic, Spatial, Temporal, Conceptual integration of data, addressing data heterogeneity issuesEnable communication and collaboration for analysisEnable re-use of analytical components


SEEK Main Components KeplerProblem-solving environment for scientific data analysis and visualization scientific workflows

EcoGrid* Distributed data network for environmental, ecological, and systematics dataMaking diverse environmental data systems interoperate

Semantic Mediation SystemSmart data discovery and integration

Knowledge Representation WGTaxon WGBEAM WGEducation, Outreach, Training*name-clash: cf. other Eco-Grid project!







Ecology Scientific Workflow: Invasive Species PredictionSource: NSF SEEK (Deana Pennington et. al, UNM)


Scientific WorkflowsModel the way scientists work with their data and toolsMentally coordinate export and import of data among software systemsScientific workflows emphasize data flow ( business workflows)Metadata (incl. provenance info, semantic types etc.) is crucial for automated data ingestion, data analysis, Goals: SWF automation, SWF & component reuse, SWF design & documentationmaking scientists data analysis and management easier!


Commercial & Open Source Scientific Workflow (Dataflow) SystemsKensington Discovery Edition from InforSenseTavernaTriana







Kepler Starting Point: UC Berkeleys Ptolemy II Large, polymorphic component (Actors) and Directors libraries (drag & drop)Directors define the component interaction & execution semantics


Kepler Scientific Workflows e.g. from Web Services1234 Minute-made (MM) WS-based application integrationSimilarly: MM workflow design & sharing w/o implemented components


Job Management (here: with NIMROD) Job management infrastructure in place Results database: under development Goal: 1000s of GAMESS jobs (quantum mechanics)


Some Kepler Actor Additions


Ecological Niche Model in Kepler(200 to 500 runs per speciesx2000 mammal speciesx3 minutes/run)

=833 to 2083 days


Grid-enabled KeplerUtilize distributed computing resourcesExecute single steps or sub-workflows on distributed machinesInitially, focus on trivially parallel workflowsSupport collaboration through the formation of ad-hoc grids

ImplementationsPeer to peer using JXTATraditional HPC-based batch job submission (e.g., NIMROD, Condor)KeplerGrid for NicheModeling



A GEON Data Analysis Workflow


Statistics Packages (here: R) in KeplerSource: Dan Higgins, Kepler/SEEK



KEPLER: An OPEN SOURCE, cross-project collaborationIlkay Altintas SDM, Resurgence, NLADR,Kim Baldridge Resurgence, NMI Chad Berkley SEEK Shawn Bowers SEEKTerence Critchlow SDM Tobin Fricke ROADNetJeffrey Grethe BIRNChristopher H. Brooks Ptolemy II Zhengang Cheng SDM Dan Higgins SEEKEfrat Jaeger GEON Matt Jones SEEK Werner Krebs, EOLEdward A. Lee Ptolemy II Kai Lin GEONBertram Ludaescher SDM, SEEK, GEON, BIRN, ROADNetMark Miller EOLSteve Mock NMISteve Neuendorffer Ptolemy II Jing Tao SEEK Mladen Vouk SDM Xiaowen Xin SDM Yang Zhao Ptolemy IIBing Zhu SEEK Ptolemy IIwww.kepler-project.orgYour Logos& NamesHERE!!!


GEON Dataset Generation & Registration (a co-development in KEPLER)Xiaowen (SDM)Edward et al.(Ptolemy)Yang (Ptolemy)Efrat(GEON)Ilkay(SDM)SQL database access (JDBC)Matt,Chad, Dan et al. (SEEK)% Makefile$> ant run


Kepler todaySupports scientific workflowsEcology, molecular bio, geology, Variety of analytical components (including spatial data transformations)Support for R scripts and Matlab scriptsReal-time data access via Antelope ORBEcoGrid access to heterogeneous dataEML Data supportExperimental data, survey data, spatial raster and vector data, etc.DarwinCore Data supportMuseum collectionsEcoGrid registry to discover data sourcesOntology-based browsing for analytical componentsExploit semantics to improve the user experienceDemonstration workflowsEcology: Ecological Niche Modeling, Biodiversity Analysis, Genomics: Promoter Identification WorkflowGeology: Geologic Map Integration, Rock-type distribution analysisOceanography: Real-time Revelle example of data access


Kepler soon (this year mostly )Usability engineeringFull evaluation and user-oriented customization of all UI componentsDistributed computing/grid computingLarge jobs, lots of machinesDetached executionSmart data and component discoverySupport annotating data sourcesComponent repository / downloadable componentsAutomated data and service integration and transformation using ontologiesComplete EcoGrid accessFull EML supportSupport for large data and 3rd-party transferMore data sources and types of data sources (e.g., JDBC, GEON data)Provenance and metadata propagation


Joint Ptolemy/Kepler Meeting (in eigener Sache ;-)







Kepler Actor-Library w/ Concept IndexHow do you find the right component (actor)? Ontology-based actor organization / browsing Simple text-based and concept-based searchingNext: ontology-based workflow designWorkflowComponents(MoML)Ontologies(OWL)Default + OtherSemanticAnnotationsurn idsinstanceexpressions


Ecological ontologiesWhat was measured (e.g., biomass)Type of measurement (e.g., Energy)Context of measurement (e.g., Psychotria limonensis)How it was measured (e.g., dry weight)

SEEK intends to enable community-created ecological ontologies using OWLRepresents a controlled vocabulary for ecological metadata


Ontology O (in Description Logic cf. OWL-DL)


SEEK KR (Knowledge Representation) Working GroupCurrent OntologiesEcological Concepts, Models, NetworksMeasurementsPropertiesStatistical AnalysesTime and SpaceTaxonomic IdentifiersUnitsSymbiosisRecent DevelopmentsBiodiversity (measured traits, computation of traits)Descriptive Terminology for Plant CommunitiesOntology documentationFuture GoalsFill-in existing concepts, evolve the ontology frameworkMore domains


Need for Semantic Annotations of data & actorsLabel data with semantic types (concept expressions from an ontology)Label inputs and outputs of analytical components with semantic types

Example: Data has COUNT and AREA; workflow wants DENSITY via ontology, system knows that data can still be used (because DENSITY := COUNT/AREA)Use reasoning engines to generate transformation stepsUse reasoning engine to discover relevant componentsDataOntologyWorkflow Components


A Scientists Semantic View of ActorsS1 (life stage property)S2 (mortality rate for period)P1P2P4P3P5PhaseObservedPeriodPhasesEggsInstar IInstar IIInstar IIIInstar IVAdults44,0003,5132,5291,9221,4611,300Nymphal{Instar I, Instar II, Instar III, Instar IV}Population samples for life stages of the common field grasshopper [Begon et al, 1996]Periods of development in terms of phaseslife stage periodsk-value for each period of observation[(nymphal, 0.44)]observationsSource: [Bowers-Ludaescher, DILS04]


Structural Type (XML DTD) AnnotationsS1 (life stage property)S2 (mortality rate for period)P1P2P4P3P5

44,000 0.95 Eggs

root cohortTable= (measurement)*elem measuremnt= (phase, obs)elem phase= xsd:stringelem obs= xsd:integer

Eggs 44,000

structType(P2)structType(P3)Source: [Bowers-Ludaescher, DILS04]


Semantic Type AnnotationsTake concepts and relationships from an ontology to semantically type the data-in/out ports

Application: e.g., design support: smart/semi-automatic wiring, generation of adaptor actors

Actor(normalize)pinpoutTakes Abundance Count Measurements for Life StagesReturns Mortality Rate DerivedMeasurements for Life StagesSource: [Bowers-Ludaescher, DILS04]




A KR+DI+Scientific Workflow ProblemServices can be semantically compatible, but structurally incompatibleSourceServiceTargetServicePsPtSemantic Type PsSemantic Type PtStructural Type PtStructural Type PsDesired ConnectionIncompatibleCompatible()()(Ps)Ontologies (OWL)Source: [Bowers-Ludaescher, DILS04]


The Ontology-Driven FrameworkSourceServiceTargetServicePsPtSemantic Type PsSemantic Type PtStructural Type PtStructural Type PsDesired ConnectionCompatible()RegistrationMapping (Output)RegistrationMapping (Input)CorrespondenceOntologies (OWL)


Correspondence Example/population/sample == semType(P2) /population/sample/meas/cnt == semType(P2).itemMeasured /population/sample/meas/cnt/text() == semType(P2).itemMeasured.hasCount /population/sample/meas/acc == semType(P2).hasProperty /population/sample/meas/acc/text() == semType(P2).hasProperty.hasValue /population/sample/lsp/text() == semType(P2).hasContext.appliesTo /cohortTable/measurement == semType(P3) /cohortTable/measurement/obs == semType(P3).itemMeasured /cohortTable/measurement/obs/text() == semType(P3).itemMeasured.hasCount /cohortTable/measurement/phase/text() == semType(P3).hasContext.appliesToSourceTarget populationsample *meascntxsd:doublexsd:stringlspxsd:integeracccohortTablemeasurement *obsxsd:integerphasexsd:stringWe want to exploit the semantic information to obtain structural correspondences


Correspondence Example/population/sample == semType(P2) /population/sample/meas/cnt == semType(P2).itemMeasured /population/sample/meas/cnt/text() == semType(P2).itemMeasured.hasCount /population/sample/meas/acc == semType(P2).hasProperty /population/sample/meas/acc/text() == semType(P2).hasProperty.hasValue /population/sample/lsp/text() == semType(P2).hasContext.appliesTo /cohortTable/measurement == semType(P3) /cohortTable/measurement/obs == semType(P3).itemMeasured /cohortTable/measurement/obs/text() == semType(P3).itemMeasured.hasCount /cohortTable/measurement/phase/text() == semType(P3).hasContext.appliesToSourceTarget populationsample *meascntxsd:doublexsd:stringlspxsd:integeracccohortTablemeasurement *obsxsd:integerphasexsd:string/population/sample == semType(P2)/cohortTable/measurement == semType(P3)These fragments correspond


Correspondence Example/population/sample == semType(P2) /population/sample/meas/cnt == semType(P2).itemMeasured /population/sample/meas/cnt/text() == semType(P2).itemMeasured.hasCount /population/sample/meas/acc == semType(P2).hasProperty /population/sample/meas/acc/text() == semType(P2).hasProperty.hasValue /population/sample/lsp/text() == semType(P2).hasContext.appliesTo /cohortTable/measurement == semType(P3) /cohortTable/measurement/obs == semType(P3).itemMeasured /cohortTable/measurement/obs/text() == semType(P3).itemMeasured.hasCount /cohortTable/measurement/phase/text() == semType(P3).hasContext.appliesToSourceTarget populationsample *meascntxsd:doublexsd:stringlspxsd:integeracccohortTablemeasurement *obsxsd:integerphasexsd:string/population/sample/meas/cnt == semType(P2).itemMeasured/cohortTable/measurement/obs == semType(P3).itemMeasuredThese fragments correspond


Correspondence Example/population/sample == semType(P2) /population/sample/meas/cnt == semType(P2).itemMeasured /population/sample/meas/cnt/text() == semType(P2).itemMeasured.hasCount /population/sample/meas/acc == semType(P2).hasProperty /population/sample/meas/acc/text() == semType(P2).hasProperty.hasValue /population/sample/lsp/text() == semType(P2).hasContext.appliesTo /cohortTable/measurement == semType(P3) /cohortTable/measurement/obs == semType(P3).itemMeasured /cohortTable/measurement/obs/text() == semType(P3).itemMeasured.hasCount /cohortTable/measurement/phase/text() == semType(P3).hasContext.appliesToSourceTarget populationsample *meascntxsd:doublexsd:stringlspxsd:integeracccohortTablemeasurement *obsxsd:integerphasexsd:string/population/sample/meas/cnt/text() == semType(P2).itemMeasured.hasCount/cohortTable/measurement/obs/text() == semType(P3).itemMeasured.hasCountThese fragments correspond


Correspondence Example/population/sample == semType(P2) /population/sample/meas/cnt == semType(P2).itemMeasured /population/sample/meas/cnt/text() == semType(P2).itemMeasured.hasCount /population/sample/meas/acc == semType(P2).hasProperty /population/sample/meas/acc/text() == semType(P2).hasProperty.hasValue /population/sample/lsp/text() == semType(P2).hasContext.appliesTo /cohortTable/measurement == semType(P3) /cohortTable/measurement/obs == semType(P3).itemMeasured /cohortTable/measurement/obs/text() == semType(P3).itemMeasured.hasCount /cohortTable/measurement/phase/text() == semType(P3).hasContext.appliesToSourceTarget populationsample *meascntxsd:doublexsd:stringlspxsd:integeracccohortTablemeasurement *obsxsd:integerphasexsd:string/population/sample/lsp/text() == semType(P2).hasContext.appliesTo/cohortTable/measurement/phase/text() == semType(P3).hasContext.appliesToThese fragments correspond


Ontology-Guided Data TransformationSourceServiceTargetServicePsPtSemantic Type PsSemantic Type PtStructural Type PtStructural Type PsDesired ConnectionCompatible()Structural/SemanticAssociationStructural/SemanticAssociationCorrespondenceGenerate(Ps)Ontologies (OWL)TransformationSource: [Bowers-Ludaescher, DILS04]


Linking Structural and Semantic Types : S OSchema elements/Structural type SOntology /Semantic type O


Propagating Semantic Annotations Given: structural schemas S (input) and S (output), and an ontology Oa semantic annotation : S Oa query annotation q: S SProblem: compute


ApplicationsWF design time:Actor Actor connectionsData binding time:Actor Data connections (data binding)WF runtime:semantic tagging of derived data products


Semantic PropagationInfer annotations for derived products:When a (partial) specification of an actor is given (e.g., as a query q), then exploit this to propagate semantic annotations from S to Tminimize costly semantic annotationcheck for consistencyqTruSannotatedannotatedTraditional LAV query answeringChase & Backchase, e.g., via MARS maps tosource annotationnew targetannotationnquery


Biodiversity Workflow w/ Query Annotations q


Annotation Constraint : S O = x (s(x) c(x)) y o(z) % z = x y

s links the variables x to schema elements of Sc is conjunction of comparisons over x and constantso populates the ontology structure O

X : biom[seas=S], S = w X : observation[temporalContext = S : WinterSeason]s(x)c(x)o(z)


Example (Biodiversity Workflow)


Scientific Workflow (SWF) Design MethodologySupport SWF design & reuse, via:Structural data types Semantic typesAssociations (=constraints) between them Type checking, inference, propagationSeparation of concerns:structure, semantics, WF orchestration, etc.


Data IntegrationKnowledgeRepresentationProcess Integration(Scientific Workflows)Src: ECS-289 Scientific Data Management WQ05Data FederationEcoGrid


Q & A


KEPLER: An Open CollaborationInitiated by members from DOE SDM/SPA and NSF SEEK; now several other projects (GEON, Ptolemy II, EOL, Resurgence/NMI, )Open Source (BSD-style license)Intensive Communications: Web-archived mailing listsIRC (!)Meetings, HackathonsCo-development: via shared CVS repositoryjoining as a new co-developer (currently):get a CVS account (read-only)local development + contribution via existing KEPLER memberbe voted in as a member/co-developer

Software & social engineeringHow to better accommodate new groups/communities?How to better accommodate different usage/contribution models (core dev special purpose extender user)?


GrOWLGraphical Ontology Editing and Browsing Krivov and Villa (UVM)


Data Procurement using SemanticsFind all datasets that contain abundance measurements of Manica bradleyi inter-ant parasites observed within California

SEEK AHM 2004

SWDBAug 29, 2004

November, 2004

Our Initial Data-Procurement Prototype

Support external services for incorporating information not easily or conveniently expressed using conceptual modeling languages (description logic)

SMS Dataset Discovery

Dataset Repository

Dataset Query Expression

Semantic Descriptions

External Services

Semantic Conversions

Ontology Repository

Relevant Datasets/Data

ConversionRules

SEEK AHM 2004

SWDBAug 29, 2004

November, 2004

Results

SEEK AHM 2004

SWDBAug 29, 2004

November, 2004

Ontologies

Expressed in OWL, shown here graphicallyThis is a simple OWL ontology (in terms of formulas)

prop

Observation

SpatialContext

GeoSpatialRegion

GeoCoordPoint

LatLonPoint

UTMPoint

context

location

xsd:float

xsd:float

latDeg

lonDeg

xsd:float

xsd:float

UTMx

UTMy

xsd:int

xsd:int

zone

region

xsd:string

value

ObservableItem

item

EcoProperty

property

TaxonID

EcoEntity

Abundance

SciName

xsd:string

xsd:string

genus

species

property

SciName

Parasite

Host

InquilinismHost

InquilinismParasite

parasiteOf

inquilinismOf

parasiteOf

inquilinismOf

isa

role

Host

class

SEEK AHM 2004

SWDBAug 29, 2004

November, 2004

Results


Related PublicationsScientific WorkflowsScientific Workflow Management and the Kepler System, B. Ludscher, I. Altintas, C. Berkley, D. Higgins, E. Jaeger-Frank, M. Jones, E. Lee, J. Tao, Y. Zhao, Concurrency and Computation: Practice & Experience, Special Issue on Scientific Workflows, to appear, 2005. A Framework for the Design and Reuse of Grid Workflows, Ilkay Altintas, Adam Birnbaum, Kim Baldridge, Wibke Sudholt, Mark Miller, Celine Amoreira, Yohann Potier, and Bertram Ludaescher, Intl. Workshop on Scientific Applications on Grid Computing (SAG'04), LNCS 3458, Springer, 2005 Kepler: An Extensible System for Design and Execution of Scientific Workflows, I. Altintas, C. Berkley, E. Jaeger, M. Jones, B. Ludscher, S. Mock, 16th International Conference on Scientific and Statistical Database Management (SSDBM'04), 21-23 June 2004, Santorini Island, Greece. Kepler: Towards a Grid-Enabled System for Scientific Workflows, Ilkay Altintas, Chad Berkley, Efrat Jaeger, Matthew Jones, Bertram Ludscher, Steve Mock, Workflow in Grid Systems (GGF10), Berlin, March 9th, 2004.An Ontology-Driven Framework for Data Transformation in Scientific Workflows, S. Bowers and B. Ludscher, Intl. Workshop on Data Integration in the Life Sciences (DILS'04), March 25-26, 2004 Leipzig, Germany, LNCS 2994. A Web Service Composition and Deployment Framework for Scientific Workflows, I. Altintas, E. Jaeger, K. Lin, B. Ludaescher, A. Memon, In the 2nd Intl. Conference on Web Services (ICWS), San Diego, California, July 2004.


Related PublicationsSemantic Data Registration and IntegrationOn Integrating Scientific Resources through Semantic Registration, S. Bowers, K. Lin, and B. Ludscher, 16th International Conference on Scientific and Statistical Database Management (SSDBM'04), 21-23 June 2004, Santorini Island, Greece. A System for Semantic Integration of Geologic Maps via Ontologies, K. Lin and B. Ludscher. In Semantic Web Technologies for Searching and Retrieving Scientific Data (SCISW), Sanibel Island, Florida, 2003. Towards a Generic Framework for Semantic Registration of Scientific Data, S. Bowers and B. Ludscher. In Semantic Web Technologies for Searching and Retrieving Scientific Data (SCISW), Sanibel Island, Florida, 2003. The Role of XML in Mediated Data Integration Systems with Examples from Geological (Map) Data Interoperability, B. Brodaric, B. Ludscher, and K. Lin. In Geological Society of America (GSA) Annual Meeting, volume 35(6), November 2003. Semantic Mediation Services in Geologic Data Integration: A Case Study from the GEON Grid, K. Lin, B. Ludscher, B. Brodaric, D. Seber, C. Baru, and K. A. Sinha. In Geological Society of America (GSA) Annual Meeting, volume 35(6), November 2003. Query Planning and RewritingProcessing First-Order Queries under Limited Access Patterns, Alan Nash and B. Ludscher, Proc. 23rd ACM Symposium on Principles of Database Systems (PODS'04) Paris, France, June 2004. Processing Unions of Conjunctive Queries with Negation under Limited Access Patterns, Alan Nash and B. Ludscher., 9th Intl. Conference on Extending Database Technology (EDBT'04) Heraklion, Crete, Greece, March 2004, LNCS 2992. Web Service Composition Through Declarative Queries: The Case of Conjunctive Queries with Union and Negation, B. Ludscher and Alan Nash. Research abstract (poster), 20th Intl. Conference on Data Engineering (ICDE'04) Boston, IEEE Computer Society, April 2004.

Im going to talk about some of the work weve been doing using ontologies to help make scientific workflows easier to design and usesemType(P2) subtype_of semtype(P3)

semantic extensions for scientific workflows on the grid

Documents

scientific data analysis

scientific workflowsisgc2005

orgecology scientific

distributed data network

biodiversity dataenable

automated data ingestion

conceptual integration

reuseenhance data discovery