An Introduction to Designing, Executing and Sharing Workflows

with Taverna

Katy Wolstencroft

myGrid

University of Manchester

IMPACT/Taverna Hackathon 2011

Exercise 1: Exploring the Workbench

Taverna can be downloaded from http://www.taverna.org.uk/

Go to the page and find the latest (2.3) Download the correct version for your operating system Follow the instructions in the Taverna installer

The following page shows a screenshot of Taverna and the different panels that make up the workbench

Taverna Workbench

Workflow DiagramServices Panel

Workflow Explorer

1. Workflow Diagram

The visual representation of workflow Shows inputs/outputs, services and control flows Allows editing of the workflow by dragging and dropping

and connecting services together Enables saving of workflow diagrams for publishing and

sharing

1. Workflow Explorer

The Workflow Explorer shows the detailed view of your workflow. It shows default values and descriptions for service inputs and outputs and it shows where remote services are located. It also shows configuration details, such as iteration and looping (we will come back to these things later).

Workflow validation details can also be found here. Before a workflow is run, Taverna checks to see if it is connected correctly and if its services are available.

1. Available Services Panel

Lists services available by default in Taverna Local java services WSDL Web Service – secure and public RESTful Services R Processor services (for statistical analyses) Beanshell scripts Xpath scripts

Allows the user to add new services or workflows from the web or from file systems – there are loads more available!

In the Services panel, type ‘image’ into the search box. Select ‘Get Image from URL’ This is a local service, but web services work the same way Many historical documents are stored as images on the

web. This is a simple, but useful service to help gather data

Exercise 2: Building a Simple Workflow

Drag this service across to the workflow diagram panel

Exercise 2: Building a Simple Workflow

In a blank space in the workflow diagram, right-click and select “Add Workflow Input Port”

Type a name (e.g. URL) for this input in the pop-up window and click “ok”

Do the same to create a new workflow output. Call this output “image”

Exercise 2: Building a Simple Workflow

You now have 3 boxes in the diagram and we need to connect them up into a workflow

First, we need to find out how many inputs and outputs the ‘get image from URL’ service has

At the top of the workflow diagram, select the ‘show ports’ icon

Show Ports

Exercise 2: Building a Simple Workflow

Click on the workflow input box and drag the linking arrow across to the URL input of the ‘get_image_from_URL’ service.

Link the image output of ‘get_image_from_url’ to the workflow output port

Exercise 2: Building a Simple Workflow

You have now built your first workflow! It should look something like this.

In many cases, you have to supply input data for EVERY service input port. In this case, however, the ‘base’ input is optional, so we will leave it.

Save the workflow by going to file -> save workflow

Exercise 2: Building a Simple Workflow

Run the workflow by selecting “file -> run workflow”, or by clicking on the play button at the top of the workbench

Exercise 2: Building a Simple Workflow

An input window will appear. As you can see, we have not yet added a description of the workflow or of the input

Click on ‘New Value’ in the input window and add the urlhttp://www.archives.gov/exhibits/featured_documents/magna_carta/images/magna_carta.jpg where it says “some input data goes here”

Exercise 2: Building a Simple Workflow

Click “run workflow” In the bottom left of the results window, click on the

results. You will now see an image from the specified web page

Workflow results can be saved here if required by clicking on ‘save all values’

2: Adding a Workflow Description

Right-click on a blank part of the workflow diagram and select “show details”

In the workflow explorer panel, the details page will open up. Add some details about the workflow (e.g. who is the author, what does the workflow do).

You can also add examples and descriptions for the workflow inputs by selecting them in the explorer panel and selecting “details”

Adding this metadata makes the workflow much more reusable

Save the workflow by going to “File -> save workflow”

New services can be gathered from anywhere on the web We will find a new service and add it to the workbench IMPACT and SACPE have a whole suite of services. We

will add one (you will be using it later on today) Go to https://fue.onb.ac.at/synapse. Here you will find a list

of IMPACT services Click on IMPACTTesseractV3Proxy and copy the link you

are directed to. This is the WSDL address and is what Taverna needs to

run the service

Exercise 3: Adding New Services

3. Adding New Services

Go to the services panel in Taverna and click “import new services”. For each type of service, you are given the option to add a new service

Select ‘WSDL service…’ A window will pop-up asking for a web address

3. Adding New Services

Enter the service address you just copied

Scroll down the Services list, you will see your new service there

Exercise 4: Sharing and Reusing Workflows

Go to http://www.myexperiment.org myExperiment is a social networking site for sharing

workflows and workflow expertise and experiences Browse around the site and see what it contains Find everything that has been tagged with ‘text mining’,

for example Look at the text mining workflows. You will see some

that are specific to biology, some that are generally applicable, and some that are specific to other scientific disciplines

4. Sharing and Reusing workflows

IMPACT have many workflows on myExperiment, but they are not public. You must join an IMPACT group before you can see them and use them.

Create yourself an account and join the group called ‘IMPACT-myGrid-Hackathon’ (NOTE: you need to join this group to access content for future exercises)

Explore the shared items in this group. These are examples of the types of tasks IMPACT workflows can perform

5. Using Workflows from myExperiment

You can download and run the workflows from the myExperiment website, or you can use myExperiment directly from Taverna

To use workflows from the website, you can either download them, or copy the workflow file location into the ‘open workflows from the web’ option in Taverna’s file menu.

5. Using Workflows from myExperiment

Go back to Taverna and click on the myExperiment icon at the top of the workbench

Go to ‘my stuff’ and log in (using the same credentials as the web page)

Find the IMPACT-myGrid-Hackathon group by using the ‘search’ option.

Look at the shared items and find the workflow called ‘Text to List’

Click on ‘open’ and this workflow will be automatically imported into your Taverna design window

5. Validate your Workflow

Taverna checks to see that everything is connected properly and that all the required services are available

Go to the workflow explorer and click on ‘validation report’

See if Taverna has found any problems with the workflow. Errors will be displayed in red, warnings in yellow. Workflows with warnings often still run.

If there are problems, follow the instructions to resolve them by clicking on the ‘Solution’ tab

If not, run the workflow

5. Using Workflows from myExperiment

Use the default input suggested to run the workflow. The workflow will collect and list some example data stored at the given URL

It returns a list of image files We can now combine this workflow with the one we

made earlier to return the actual images.

In Taverna, you can add workflows as if they were any other kind of service – these are called ‘Nested Workflows’

6. Reusing and connecting Workflows

From the current workflow design window, go to

‘Insert -> Nested workflow Import the workflow you

made earlier, by selecting ‘import from file’

You can see a small version of the workflows, so you can check you are importing the correct workflow

6. Reusing and connecting Workflows

We now need to connect the two workflows together Connect the Text2List service to the input of the nested

workflow by dragging an arrow across. Make a new workflow output port (by right-clicking and

adding workflow output port) Connect the output of the nested workflow to the new

workflow output port

6. Reusing and connecting Workflows

Your new workflow should look something like this

Save and run the workflow This time, as it runs, you will

see Taverna automatically iterates over the list of data produced by Text2List

NOTE: some of the iterations will fail. See if you can tell which

Look at one of the resulting images

7. Looking at Intermediate Results

You can track intermediate workflow values through the results view. This is very useful for working out where unexpected results came from.

On the diagram, click the Text2List service and look at its inputs and outputs in the results.

You can save the workflow in myExperiment if you wish, but make sure you give credit to the nested workflow author and make sure you ONLY share it with the IMPACT-myGrid-Hackathon group

Controlling data flow in WorkflowsAdvanced Exercises

As you have already seen, Taverna can automatically iterate over sets of data.

When 2 sets of iterated data are combined, however, Taverna needs extra information about how they should be combined. You can have:

A cross product – combining every item from list 1 with every item from list 2 - all against all

A dot product – only combining item 1 from list 1 with item 1 from list 2, and so on – line against line

8. Iteration

Find and load the workflow ‘Demonstration of configurable iteration’ from myExperiment

Read the workflow metadata to find out what the workflow does (by looking at the ‘Details’)

Select the ‘ColourAnimals’ service and select the ‘Details’ in the workflow explorer and ‘configure list handling’

Click on ‘dot product’ in the pop-up window. This allows you to switch to cross product

8. Iteration

Run the workflow twice – once with ‘dot product’ and once with ‘cross product’.

Save the first results so you can compare them – what is the difference? What does it mean to specify dot or cross product?

8. Iteration

9. Retries: Making your Workflow Robust

Web services can sometimes fail due to network connectivity

If you are iterating over lots of data items, you can guard against these temporary interruptions by adding retries to your workflow

Upload the ‘Retry-Example’ workflow from the IMPACT-myGrid-Hackathon group. This workflow is designed to fail sometimes.

Run the workflow as it is and count the number of failed iterations

9. Retries: Making your Workflow Robust

Now, select the ‘sometimes_fails’ service and select the ‘details’ tab in the workflow explorer panel

Click on ‘advanced’ and ‘configure’ for retries In the pop-up box, change it so that it retries each

service iteration 2 times Run the workflow again – how many failures do you get

this time? Change the workflow to retry 5 times – does it work

every time now?

10. Looping

From myExperiment, download and open the workflow “dummy_example_of_looping”

This workflow is asynchronous. This means that when you submit data (by running the workflows), it will return a jobID and place your job in a queue. This is very useful if your job will take a long time!

The ‘CheckStatus’ service will query your job ID to find out if it is complete

10. Looping

The default behaviour in a workflow is to call each service only once for each item of data – so what if your job has not finished when ‘Status’ workflow asks?

Run the workflow Almost every time, the workflow will ‘fail’ (in this case,

that means it will return 0) because the results have not been returned before the workflow reaches the ‘getResults’ service

10. Looping

This is where looping is useful. Taverna can keep running the ‘status’ service until it reports that the job is done.

Select the ‘CheckStatus’ service and click on the ‘details’ tab in the workflow explorer

Select ‘advanced’ and click on ‘add looping’ Use the drop-down boxes in the looping window to set

‘state’ ‘is_not_equal_to’ RUNNING

10. Looping

Save the workflow and run it again This time, the workflow will run until the ‘CheckStatus’

service reports that it is either COMPLETE, or it has an ERROR.

You will see results for ‘GetResults’, but you will still get an error for ‘GetResults2’. This is because there is one more configuration to change – we also need ‘Control Links’

A control link specifies that there is a dependency of one service on another even though there is no data flowing between them.

A control link is a line with a white circle at the end that connects two services (see the link between ‘CheckStatus’ and ‘getResults’

11. Control Links

11. Control Links

We will add control link to getResults2 Right-click on getResult2 and select ‘Run after’ from the

drop down menu. Set it to ‘Run after’ -> ‘CheckStatus’ Save and run the workflow Now you will see both results returned