1

P-GRADE Portal and GEMLCA P-GRADE Portal and GEMLCA Legacy Code ArchitectureLegacy Code Architecture

Peter KacsukPeter KacsukMTA SZTAKIMTA SZTAKI

www.lpds.sztaki.hu

2

OutlineOutline

• Motivation of our work• P-GRADE (Parallel Grid Run-time and Application

Development Environment) Portal in a nutshell• Workflow application support• Parameter sweep application support• GEMLCA• P-GRADE Portal and Grid interoperability• Conclusions

3

EE--sscientistscientists’ concerns’ concerns

The P-GRADE Grid Portal gives you all the answers!

• How to concentrate on my own research if the technology I would like to use is in continuous change?

• Which is the most suitable grid for me?• How can I learn and understand the

usage of that technology?• How can I develop applications?• How can I execute applications? • How to tackle performance issues?• How to use several Grids at the same

time?• How to migrate my application from

one grid to another?• How can I collaborate with fellow

researchers?

4

Easy-to-use, middleware-neutral Easy-to-use, middleware-neutral workflow-oriented Grid portalworkflow-oriented Grid portal

• Graphical tools for workflow-oriented application development, execution and monitoring

• Sequential & parallel components can be integrated into large Grid applications

• Direct or broker-based resource allocation• The P-GRADE Portal hides differences of Grids

• Globus 2 and 4 – LCG2 – gLite Grid interoperability• Transparent switching between Grid technologies

• Your code does not have to contain grid specific calls

5

Proxy management

P-GRADE portal in a P-GRADE portal in a nutshellnutshell

Grid and Grid resource management

Graphical editor for workflow creation

Job mapping to Grid and Grid resources

Workflow management and

execution visualization

6

What is a P-GRADE Portal workflow?What is a P-GRADE Portal workflow?

• a directed acyclic graph where– Nodes represent jobs (batch

programs to be executed on a computing element)

– Ports represent input/output files the jobs expect/produce

– Arcs represent file transfer operations and job dependencies

• semantics of the workflow:– A job can be executed if all

of its input files are available

7

The typical user scenarioThe typical user scenarioPart 1 - development phasePart 1 - development phase

Certificate servers

Portalserver

Gridservices

START EDITOR

OPEN & EDIT or DEVELOP WORKFLOW

SAVE WORKFLOW

8

Certificate servers

Portalserver

Gridservices

TRANSFER FILES, SUBMIT JOBS

DOWNLOAD (SMALL)

RESULTS

DOWNLOAD (SMALL)

RESULTS

The typical user scenarioThe typical user scenarioPart 2 - execution phasePart 2 - execution phase

VISUALIZE JOBS and

WORKFLOW PROGRESS

MONITOR JOBS

DOWNLOAD PROXY CERTIFICATES

9

DemoDemo

• Workflow editing

• Workflow submission

• Workflow execution monitoring

• Examples:– VO site test– Equation Solver solving the equation

A*x = B

10

User’s feedback User’s feedback

• Users want parameter study (PS) support at workflow level

• It means:– If the user has an existing workflow in a repository, he

would like to run • the same workflow (without any change)

• with many different parameters

• Goal of the work:– Enable users to easily turn their workflow application

into a parameter sweep application

11

Parameter sweep (PS) workflow execution Parameter sweep (PS) workflow execution based on the black box conceptbased on the black box concept

PS port: 4 instances of the

input file

PS port: 3 instances of the input file

1 PS workflow execution

=

4 x 3 normal workflow execution

This provides the 3rd level of parallelism resulting a very large demand for Grid resources

12

Introducing three levels of parallelism

Each job can be a parallel program

– Parallel execution inside a workflow node

– Parallel execution among workflow nodes

Multiple jobs run parallel

– Parameter study execution of the workflow

Multiple instances of the same workflow with different data files

13

PS_1_2

PS_1_1PS_2_2

PS_2_1

PS_1_1 PS_2_1

PS_1_2 PS_2_1

PS_1_1 PS_2_2

PS_1_2 PS_2_2

WF

One or more Parameter Sets are defined. Each has different parameters. (Sets are represented by remote file directories, parameters by files)

PS DATA FLOW

Content of Parameter Sets are combined (Descartes product)

The workflow is executed as many times as many elements are in the Descartes product of input setsResult_1_1

Result_2_1

Result_1_2

Result_2_2

Store the results of the calculations(Results are represented by remote files of one or more common directories)

14

Data Model of advanced PSData Model of advanced PS

Gen

A_GenInput

Coll

Output

Coll

Output

Coll

Output

The Autogenerator is a special job:•May have no input•Runs on the Portal

15

Implementation of the example Auto

Generator

16

Parameter study workflowParameter study workflow

GEN

SEQ

COLL

SEQSEQSEQ

Parameter sweep grid

jobs

Grid job generates

input parameter

space

Collector grid job evaluates the results of the

simulationP-GRADE Portal

17

Summary of PS execution

First phase: executing

ones all the Generators

Last phase: executing

ones all the

Collectors

Second phase:

executing all generated

eWorkflows in parallel

18

Workflow Management Window showing Generator Workflow Management Window showing Generator and Collectorand Collector

Collector phase is

inactive until each

eWorkflow has terminated

19

PS-portal DemoPS-portal Demo

• PS-Workflow editing

• PS-Workflow submission

• PS-Workflow execution monitoring

• Example:– Equation Solver

20

GEMLCA/P-GRADE Portal GEMLCA/P-GRADE Portal MotivationsMotivations

• Need for using legacy codes in the Grid– Code from the past, maintained because it works– Often supports business critical functions– Complex dependencies, licence and/or hardware restrictions

cannot be (modified) gridified

• GEMLCA – Grid Execution Management for Legacy Code Architecture– To deploy legacy code applications as Grid services without

reengineering the original code and with only minimal user effort

• GEMLCA/P-GRADE Portal integration– To create Grid workflows where components can also be legacy code

applications– To make these functions available from a Grid Portal

21

Publish legacy applicationsPublish legacy applications as application services as application services

Grid(s)/VO(s)

Legacy applications

Legacy applications

Grid 1

Grid 2

. . .. . .

GEMLCAP-GRADE

Portal ServerDesktop 1

Web browser

LCLC

LCLC

““GEMLCA GEMLCA Administration Administration Tool” portletTool” portlet

22

Integrate legacy application Integrate legacy application services into your workflowservices into your workflow

Grid(s)/VO(s)

Legacy applications

Legacy applications

Grid 1

Grid 2

. . .. . .

GEMLCAP-GRADE

Portal Server

Desktop 2

Web browser ResultResult

InputInput

LC 1LC 1

LC 2LC 2

23

Combine legacy codeswith new codesinside the same workflow!

Code Code invocationinvocation

Code Code invocationinvocation

Job Job submissionsubmission

Job Job submissionsubmission

Combining legacy and Combining legacy and non-legacy (job) componentsnon-legacy (job) components

Publish successfully executed new code as legacy code

24

Sharing jobs with other Sharing jobs with other GEMLCA portal usersGEMLCA portal users

New button for sucessfully

executed jobs: create a legacy

code service from this jobs

25

GEMLCAP-GRADE

Portalserver

Sharing jobs with other Sharing jobs with other GEMLCA portal usersGEMLCA portal users

Job Job Repository Repository

Put job into Portallevel repository

26

Grid interoperability 1Grid interoperability 1

• P-GRADE portal supports simultaneous job submission in several Grids: – GT2 (OSG, TeraGrid and UK NGS)– LCG-2, gLite – EGEE, SEEGRID– NorduGrid

• P-GRADE Portal enables:– Connecting production Grids and make

them interoperable at workflow level

27

User

P-GRADE Portal

WPI Portal Server

Connecting various production GridsConnecting various production Grids

gLitebroker

Certificate server

Proxy

Workflow

ANL

Fermi Lab

OSG (GT2)

SEEGRID (gLite)

Job

Job

Job

Job

Budapest

Istambul

Athens

Supports both direct and brokered job submission

28

Grid interoperability 2Grid interoperability 2

• P-GRADE Portal with GEMLCA supports legacy code service invocation in 2nd and 3rd generation Grids:– GT2, GT4 – (OSG, TeraGrid and UK NGS)– LCG-2, gLite – (EGEE, SEEGRID)

• Combination of jobs and legacy code services enables GEMLCA/P-GRADE Portal to support:– Extending 2nd generation Grids with legacy code

services– Extending 2nd generation Grids with GT4 sites– Connecting 2nd generation and 3rd generation Grids

29

Connecting 2Connecting 2ndnd generation and generation and 33rdrd generation production Grids generation production Grids

Job Repository Job Repository (GEMLCA)(GEMLCA)

Fermi LabUser

ANL

GEMLCAP-GRADE

Portal

GIN Portal Server

Executable

OSG (GT2)

WestFocus GT4 Grid

UoW

Brunel

Poznan

Budapest

EGEE LCG/gLite

Executable

Executable

ServiceInvocation

Executable

30

GIN VO Portal

P-GRADE

GEMLCA

Portal

GEMLCA GEMLCA RepositoryRepository

31

RReferenceseferences

• P-GRADE Portal service is available for– SEE-GRID infrastructure– Central European VO of EGEE– GILDA: Training VO of EGEE– Many national Grids (UK National Grid Service,

HunGrid, etc.)– US Open Science Grid, TeraGrid– Economy-Grid, Swiss BioGrid, Bio and Biomed

EGEE VOs, BioInfoGrid, BalticGrid – GIN VO (see the presentation at the EGEE,

TeraGrid and Hungarian Grid booths)

32

P-GRADE portal installationsP-GRADE portal installations

33

How to learn the P-GRADE portal? How to learn the P-GRADE portal?

– Take a look at www.lpds.sztaki.hu/pgportal(manuals, slide shows, installation procedure, etc.)

– Visit or request a training event! (event list also on homepage)• Lectures, demos, hands-on tutorials, application development support

– Get an account for one of its production installations:• OSG portal - WPI• SEEGRID portal – SZTAKI• GILDA portal – SZTAKI• NGS portal – University of Westminster

– If you are the administrator of a Grid/VO then contact SZTAKI to get your own P-GRADE Portal!

– If you know the administrator of a P-GRADE Portal you can ask him/her to give access to your Grid through his/her portal installation! (Multi-Grid portal)

34

Thank you!Thank you!

www.lpds.sztaki.hu/pgportalpgportal@lpds.sztaki.hu

Learn once, use everywhere - portalDevelop once, execute anywhere - application