SCAPE

Alexandra Nenadic, David Withers University of Manchester

Practical Tools for Digital Preservation: A Hack-a-thon York, 28th September 2011

Taverna and myExperiment Tools for creating and sharing workflows

Taverna

SCAPE What is a workflow?

• Connecting a set of tools/services to create automated and repeatable processing/analysis

SCAPE

Design and run workflows

Taverna

SCAPE Taverna Features - Overview

• Access to remote, distributed or local services and resources

• Enables service interoperability and integration

• Automates data flow between services

• Implicit iteration over data sets, list handling and control links to determine order of service invocation

• Extensible • Large external developer base

• Various third party plugins available

• Data and provenance collection

SCAPE Taverna Workbench

• Graphical desktop tool

• Drag-and-drop services into diagram

• Connect services, run, reconnect, rerun

• Integrates diverse set of tools

SCAPE Workflow Design

Available services

Tree view of the workflow

structure

Workflow diagram

SCAPE Taverna Workflows – Features in Detail

• A set of (local and remote) services to analyze or manage data

• Data-links connects services • i.e. output from service A is input to

service B and C • Describes the desired dataflow

instead of process coordination • Nested workflows are also services • Automatic iterations • Parallelization • Can customize list handling and

control links • Fault tolerance

• Retry (with delay and back off) • Failover (alternate services)

SCAPE Supported Services

• SOAP/WSDL Web services • REST Web services • SoapLab Web services • R statistical services • Inline Beanshell scripts • External tools and scripts (via ssh or localhost) • Spreadsheet import • XPath and text manipulation services • SADI semantic Web services • Nested workflows (workflow within workflow) • BioMoby • BioMart • … your tool (write your own Taverna plugin)

SCAPE Workflow Results

Input data and results per port

Progress report

Previous runs

SCAPE Workflow Provenance

• Information about a workflow run • What happened?

• And when?

• Lineage tracing • Which input produced which output

• Intermediate data • Inputs and outputs for each workflow step

• Useful for debugging

• Saved in standard format (such as OPM)

SCAPE Taverna is Domain-Independent

• Bioinformatics

• Biomedicine

• Chemistry

Pharmacogenomics Association study of Nevirapine-induced skin rash in Thai Population

HIV and TB research in South Africa Sleeping Sickness in African Cattle

Systems Biology for Crop research, BioDiversity

SCAPE Taverna is Domain-Independent

• Astronomy

• Data and text mining

• Digital content preservation (IMPACT)

• Social simulations

Astronomy & HelioPhysics

Observing Systems Simulation Experiments JPL, NASA

Library Document Preservation British Library

SCAPE

Share, discover and reuse workflows

SCAPE myExperiment

• http://www.myexperiment.org

• Social networking for people to share workflows and collaborate

• Makes it easy for people to contribute to a pool of workflows, build communities and form relationships

• Enables people to share, describe, reuse and repurpose workflows, reduce time-to-production, share expertise and avoid reinvention

SCAPE myExperiment

SCAPE Workflow Sharing, Ownership and Attribution

• myExperiment can provide a central location for workflows from one community/group

• myExperiment allows you to say

– Who can look at your workflow

– Who can download your workflow

– Who can modify your workflow

– Who can run your workflow

• Workflow ownership and attribution

• Users do not need to start from scratch – reuse or modify existing workflows

• Attribute/credit original author

SCAPE Use myExperiment from Taverna

SCAPE Training

• Tutorials and Training • 58+ tutorials to >900 people • >20 Universities, institutes

and networks • Major conferences • Summer schools

• Developer and User Days • Annotation Jamborees

• Undergraduate and Postgraduate Bioinformatics in > 30 Universities

SCAPE Taverna and SCAPE

• SCAPE preservation components/actions as services in Taverna workflows

• Use Taverna Workbench to create and test SCAPE preservation workflows on local data

• Then scale-up and run the workflows on a parallelized platform using Hadoop MapReduce

• Share Taverna SCAPE workflows on myExperiment