taverna workflows in the cloud
DESCRIPTION
Robert HainesTRANSCRIPT
Taverna* and workflows
*Other workflow systems are available
Taverna workflows • Sophisticated analysis
pipelines• A set of services to analyze or
manage data (local or remote)• Workflows run through the
workbench or via a server• Automation of data flow
through services• Control of service invocation• Iteration over data sets• Provenance collection• Extensible and open source
Taverna Workbench
• Desktop application• GUI• Plug-in Framework• Intermediate results
views• Search for Web
Services in catalogues• Search and publish to
myExperiment
Taverna Server family
• Taverna Server – Multiple clients, Multi-user– Local and large scale infrastructures– Site Replication
• Taverna Server Amazon Image– Local R server– Multiple instances in Amazon Cloud and as required,
for multiple users/uses and different security scenarios• Taverna Virtual Machine• Taverna Command Line• Bundled Servers, Services and Tools
Users are not the same….any one individual can be all of these
• Pro Makers: Technical Experts– Rich power tools– Control, flexibility, expressivity
• In the Field Users– Re-modellers
• Simplified though limited tools• Revise variants, tweaking• Inspection and guidance
– Vanilla Users: Pre-cooked workflows• Point and click / form fill / ambient configuration• Web based / Bespoke / Embedded launch
Workbench
Lite
Taverna Tool Spectrum
Technical ComputationalScientist
DomainScientist
Workbench WorkbenchComponents
LiteDomain-SpecificWebsite / Tool
Workflow Visibility
Concept KnowledgeTaverna Domain
High Low
Player Command Line
The Taverna Suite of ToolsClient User Interfaces
User InterfacesWorkflow Repository
Service Catalogue
Third Party Tools
Web Portals
Activity and Service Plug-in Manager
WorkflowProvenance
Workflow Server
Secure Service AccessCredential Manager
Workflow Engine
Virtual Machine
Prog & APIs
Command Line
Taverna Lite
Player
Taverna Workbench
Freely availableopen source
Current Version 2.4
80,000+ downloads across versions
Part of the myGrid Toolkit
Windows/Mac OS X/Linux/unix
Katherine Wolstencroft, Robert Haines, Donal Fellows, Alan Williams, David Withers, Stuart Owen, Stian Soiland-Reyes, Ian Dunlop, Aleksandra Nenadic, Paul Fisher, Jiten Bhagat, Khalid Belhajjame, Finn Bacall, Alex Hardisty, Abraham Nieva de la Hidalga, Maria P. Balcazar Vargas, Shoaib Sufi, and Carole Goble: “The Taverna workflow suite: designing and executing workflows of Web Services on the desktop, Web or in the Cloud”, Nucleic Acids Res., May 2013. doi:10.1093/nar/gkt328
Taverna – www.taverna.org.uk
Workflows in the Cloud
Biodiversity Virtual e-Laboratory
Biodiversity Virtual e-Laboratory
• BioVeL is an international network of experts– Connects two scientific communities: IT and biodiversity
• “Pals” system
– Roughly a three-way split between:• Biodiversity scientists, Biodiversity Informaticians, Computer
Scientists
– Shares expertise in workflow studies among BioVeL’s users and friends
– Fosters an international community of researchers and partners on biodiversity issues
Biodiversity Virtual e-Laboratory
• BioVeL users want to be able to:– Import data from own research and/or from existing
libraries– Use workflows to process vast amounts of data.– Build their own workflows– Access a library of workflows and re-use existing
workflows– Cut down research time and overhead expenses– Contribute to other such initiatives, such as LifeWatch
and GEO BON
Species occurrence Environmental layers
Salinity
Temp bottom
Ice conc
Primary production
Ecological niche modeling of an invasive species
Model projection
Model test
Create model
Select parameter values for the chosen algorithm
Select algorithm
Test the performance of the parameter in the modelTest performance of the
distribution prediction on the model
Assemble the model on CRIA server
Project Model with prediction layers
High quality occurrence data set
Select layers with environmental factors that are likely to influence
the distribution of the speciesCh
angi
ng a
lgor
ithm
, par
amet
er
valu
es, a
nd s
et o
f lay
ers
Select prediction layers (e.g. 2050)
Project Model with original layers
Statistical analysis of the raster data
Semi-automatized ecological niche modeling workflow
• Scientist’s PowerPoint workflow– Used everyday
• Came toManchester
• Two days with a Taverna developer– Not Scalable!
• First iteration of workflow produced
Ecological niche modeling workflow
Scary!
Ecological niche modeling workflow
Better?
Population Modelling
• Is the population growing or declining?• What effect has exploitation or other stimulus
had on the population?• Which stage should be the focus of
conservation?
Year 1• Stage• # flowers/fruits• Other variable
S
J
V
G
D
Year 2• Survival• Stage 2• # flowers/fruits 2• # of seedlings recruited • Other variables 2
SURVIVALGROWTH RATEFECUNDITYRECRUITMENT
Population Modelling Workflow
Simplifications for users
• Pre-cooked workflows– In myExperiment
• Run from the Web– Taverna Player
• Wire into familiar tools – Spreadsheets– Community portals,
e.g. ViBRANT Scratchpads
• Packaging – Taverna VM
Making it “too simple” for users!?
• Portal– Can handle many users– Makes it very easy to run workflows
• So we see lots of workflow runs!– Which is GREAT!
• Taverna has big requirements– BioVeL workflows are BIG– High CPU/Memory– Per running workflow
• Taverna becomes the bottleneck
Scale workflows: More Taverna!
• Scale and load-balance Taverna– Now we can run loads more
workflows
• Users are happy
• Service providers are NOT!– Using services – Good– Overloading services – Bad
** Please imagine loads of arrows here!
Scale workflows: More services?
• We need to replicate services– Bundle local to Taverna?
• But we don’t “own” all services– Too big/complex for us to
replicate? (Data)– Closed source?
• BioVeL has (some) funds to help service providers– Scale, redesign, re-engineer?
• Partnerships/MOUs
Data: Local services
• Data can be uploaded once• It is:
– Within your firewall/DMZ/VPC– Secure– Easy to access by services– In the right place at the right time
• Data can be read/written by services– Quickly– Without worrying about security– At no cost (£)
Data: Remote services• Data should be uploaded once• It is:
– Within your firewall/DMZ/VPC– Secure
• It is not:– Easy to access by services– In the right place at the right time
• To pass data between services it must be moved– Need secure third-party access– Bandwidth costs in to and out of the
Cloud– Need “pass by reference”
Workflows in the Cloud
Cloud Analytics for Life Sciences
SNP annotation
Annotation task• Location, Gene, Transcript• Present in public databases, dbSNP, etc• Frequency in e.g. 1000 genome data• Conservation data (cross species)
Infrastructure Requirements
• Execute analysis workflows• Accessible to clinicians and genetic testers• Cope with expanding demands on compute• Provide a secure environment• Collect provenance
Architecture overview
Webinterface
Inputs
Results
Storage (S3)
Ensembl (mySQL)
Cache(S3)
Taverna Server
Taverna Server
Taverna Server
Workflow engine
orchestratore-Hive
Other?
Taverna
Com
mon
API
Application specific tools and Web Services
Application specific tools and Web Services
Application specific tools and Web Services
WS WS ToolToolWS
Secure area(OpenAM)
All user interaction via web interface
User data stored in the Cloud
Data for all tools and Web Services stored in the Cloud
Unified access to different workflow engines with our common REST API
Tools and Web Services for each workflow are installed together for easy replication
Orchestrating workflows in the CloudInput
Workflow
Datastore
Find virtual machine for
this workflow
Is one running?
Start one
Is there space on
it?
Wait until ready
Run workflow
Yes
No
Yes
No
Delete run
Is this instance empty?
Done
Terminate it
Yes
No
Status updates
The user’s view
• Curated set of workflows– Designed, built and tested by domain experts– Quality assurance tested (if appropriate)
• Workflows are presented as applications– The workflows themselves are hidden– Configured and run via a web interface
• All user data stored securely in the Cloud– User separation
• Workflows as a Service
Web interface: Getting started
Web interface: Creating a Run
Web interface: Checking run progress
Conclusions
The user’s view
• “Science”, “Tools”, “Applications”, “Data”– Not workflows– Not infrastructure
• But they ALL have workflows– On paper– In PowerPoint– In scripts– Run “by hand”– Too personal/specific – cannot share them– “Works on my machine”
Workflow as a Service
• The workflow IS the service– Users do not see the Workflows– Run restricted sets of Taverna workflows in the cloud
• Connects to other cloud based resources – storage, tools, etc.• Scale everything behind the scenes
– Users can tweak parameters, but not design their own– Web portal access for scientists– Data passed by reference instead of by file– Pay as you go – cheap at the point of use
Supporting end-users
• Make it easy– Automate workflows they are already using– Don’t get in the way of the science– Hide the infrastructure where possible
• But it is really hard– So much has to be co-ordinated– Scale everything– Stay secure
Acknowledgements/Partners• University of Manchester• Cardiff University• European Commission 7th
Framework Programme– 283359 - BioVeL
• Eagle Genomics• Technology Strategy Board
– 100932 - Cloud Analytics for Life Sciences
• National Health Service• Amazon Web Services
Thanks
• myGrid Team– Carole Goble (PI)– Shoaib Sufi– Alan Williams– Katy Wolstencroft
• CA4LS– Abel Ureta-Vidal (PI)– Mike Cornell– Madhu Donapudi– Helen Hulme– Nick James
• BioVeL– Alex Hardisty (PI)– Renato De Giovanni– Jonathan Giddy– Norman Morrison– Abraham Nieva de la Hidalga– Matthias Obst– Maria Paula Balcazar Vargas– Elisabeth Paymal– Hannu Saarenmaa
…and many, many more…