1 [email protected] advanced features of the p-grade portal peter kacsuk, gergely sipos peter...

1

www.lpds.sztaki.hu/[email protected]

Advanced features of the P-GRADE Advanced features of the P-GRADE portalportal

Peter Kacsuk, Gergely SiposPeter Kacsuk, Gergely Sipos MTA SZTAKI

Tamás KissUniversity of Westminster

2

ContentsContents

• Solving Grid interoperability

• Supporting collaborative Grid application development

• Classification of Grid portals

3

Grid Grid interoperabilitinteroperabilityy

4

Grid interoperability 1Grid interoperability 1

• P-GRADE portal supports job submission in 2nd generation Grids: – LCG – EGEE– GT2 – UK NGS and US OSG– NorduGrid

• P-GRADE Portal enables:– Connecting 2nd generation production Grids

5

User

P-GRADE Portal

v2.3

SZTAKI Portal Server

Connecting 2Connecting 2ndnd generation production Grids generation production Grids

LCG-2broker

Certificate server

Proxy

Workflow

Manchester

Leeds

UK NGS GT2

EGEE-VOCE LCG-2

Job

Job

Job

Job

Budapest

Poznan

Brno

Supports both direct and brokered job submission

6


• P-GRADE Portal with GEMLCA supports legacy code service invocation in 2nd and 3rd generation Grids:– LCG, gLite – EGEE– GT2, GT4 – UK NGS

• Combination of jobs and legacy code services enables P-GRADE Portal to support:– Extending 2nd generation Grids with legacy code

services

7

Extending 2Extending 2ndnd generation Grids with generation Grids with legacy code serviceslegacy code services

Job Repository Job Repository (GEMLCA)(GEMLCA)

ManchesterUser

Leeds

GEMLCAP-GRADE

Portal

UoW Portal Server

Executable

NGS GT2

Poznan

Budapest

EGEE/VOCEExecutable

Executable

8


• Combination of jobs and legacy code services enables P-GRADE Portal to support:– Transition from 2nd generation Grids to 3rd

generation Grids:• Extending 2nd generation production Grids with 3rd

generation volunteer Grid sites

• Connecting 2nd generation production Grids with 3rd generation volunteer Grids

– Connecting 2nd generation and 3rd generation production Grids

9

GEMLCA GEMLCA RepositoryRepository

Manchester

Westminster

User

Oxford

P-GRADE NGS

GEMLCA

Portal

UoW Portal Server

UoW site – GT4 with GEMLCA Service

Service Invocation

Executable

Executable

Executable

Extending 2Extending 2ndnd generation Grids with generation Grids with 33rdrd generation volunteer Grid sites generation volunteer Grid sites

UK NGS GT2

10

Extending the NGS with legacy code services Extending the NGS with legacy code services and GT4 Resourcesand GT4 Resources

GT2 Job submission to Rutherford

GT4 Service Invocation at UoW

GEMLCA Legacy code submitted

to Oxford

GT2 Job submission to Manchester

11


Manchester

User

Leeds

P-GRADE NGS

GEMLCA

Portal

UoW Portal ServerNGS GT2 Grid

WestFocus GT4 Grid

UoW

Brunel

ServiceInvocation

Extending 2Extending 2ndnd generation production Grids generation production Grids with 3with 3rdrd generation volunteer Grids generation volunteer Grids

Executable

Executable

12

Connecting 2Connecting 2ndnd generation and generation and 33rdrd generation production Grids generation production Grids

Job Repository Job Repository (GEMLCA)(GEMLCA)

ManchesterUser

Leeds

GEMLCAP-GRADE

Portal

UoW Portal Server

Executable

NGS GT2

WestFocus GT4 Grid

UoW

Brunel

Poznan

Budapest

EGEE LCG/gLite

Executable

Executable

ServiceInvocation

Executable

13

Plan: Connecting major Grids by GEMLCA and the P-GRADE Portal

P-GRADE

GEMLCA

Portal


In test phase Available

14

Regional coverage of Regional coverage of using P-GRADE Portalusing P-GRADE Portal

+ GILDAFor education & training

15

Conclusion:Conclusion:Easy-to-use, technology-neutral Easy-to-use, technology-neutral

Grid portal for e-ScientistsGrid portal for e-Scientists

• The P-GRADE Portal hides differences of Grids• Globus 2 and 4 – LCG2 – gLite Grid interoperability• Transparent switching between Grid technologies

• Graphical tools for application development, execution and monitoring

• Sequential & parallel components can be integrated into large Grid applications

• Direct or broker-based resource allocation• Your code does not have to contain grid specific calls

16

Collaborative work Collaborative work supportsupport

17

Grid usageGrid usage

Grid

Grid middleware and resources

Portal

Browser Special client

Grid portal

Executeapplication

Get results

Defineapplication

Gridapp.

18

The problemThe problem

• Grids have been designed for highly complex applications (The overhead is too big to execute small programs…)

• How can a single person define highly complex applications?

19

The solution today…The solution today…

Grid

Portal


Executeapplication

Get results

Defineapplication

Gridapp.

E-mail, phone, skype, MSN, …

20

The solution today…The solution today…

Grid

Portal


Executeapplication

Get results

Defineapplication

Gridapp.

E-mail, phone, skype, MSN, …

How can I integrate the How can I integrate the knowledge of many people knowledge of many people into a single and consistent into a single and consistent entity in the shortest time?entity in the shortest time?

21

Similar problems Similar problems have been already solved…have been already solved…

• Concurrent editing of Web pages: WIKI, MediaWiki

• Concurrent editing of documents: Online collaboration tool in MS Word

• Concurrent editing of tables: Online collaboration tool in MS Excel

• …

22

The solution: The solution: collaborative Grid portalscollaborative Grid portals

• Grid portals that support – concurrent engineering of applications

– concurrent management of application execution

Grid

Portal

Gridapplication

23

The CollaborativeThe Collaborative P-GRADE Portal P-GRADE Portal

Grid

P-GRADE Portal

Browser Special client Browser Special client

submitresults results

edit edit

24

The CollaborativeThe Collaborative P-GRADE Portal P-GRADE Portal

Grid

P-GRADE Portal

Browser Special client Browser Special client

submitresults results

edit edit

No No time-consuming time-consuming communicationcommunication

…………only efficient, only efficient,

workflow-workflow-oriented oriented

collaboration!collaboration!

25

Collaborative Grid Collaborative Grid workflowsworkflows

Collaborative workflow


• Integrate knowledge – development phase• Generate and distribute results – execution phase

26

An e-Science exampleAn e-Science example



Topology of the graph

Input protein molecules

Experiment results

Protein folding

algorithm

27

A business exampleA business example



Graph topology

Input for the market

prediction application

Results of the

prediction workflow

Market prediction

model

28

Collaborative workflow Collaborative workflow developmentdevelopment

– The portal has to guarantee that the result is a consistent workflow

– Protect against data loss locking mechanisms– As real-time as possible! to increase parallelism

during development

29

Workflow development Workflow development modelmodel

• Locking workflow components (jobs) exclusively• First come first served• Only the owner can edit the component

30


• Locking workflow components (jobs) exclusively• First come first served

31


• Locking workflow components (jobs) exclusively• First come first served

32

ScreenshotScreenshot

Locked Locked for mefor me

Locked for Locked for somebody somebody

elseelse

UnlockedUnlocked

33

Portal server

Workflow editor #2

Requesting a lockRequesting a lock

Workflow editor #1

Lockrequest

Lockaccepted

34

Portal server

Synchronizing the Synchronizing the different viewsdifferent views

Workflow editor #1

Push updatedlocal view

Workflow editor #2

Pull global view

35

Result of the collaborative Result of the collaborative developmentdevelopment

Job

Job Job

JobJob

data data

datadata data

Job

data

Grid resource

Grid resourceGrid resource

Grid resource

Grid resource

Grid resource

36

Executing collaborative Executing collaborative workflowsworkflows

1. Concurrent control requestsIt can happen in single-user case too (multiple browsers are opened)

2. Different users’ proxies to be used to execute different workflow nodes

Grid 1Grid 1

Grid 2Grid 2

Grid 3Grid 3

P

1

P

3

P

2

37

ConclusionsConclusions

• Extend the single-user version of the P-GRADE Portal:– Concurrent engineering of workflow-based

grid applications– Application execution by multiple users in a

concurrent fashion– Prototype version already available

38

Classification of Classification of Grid portalsGrid portals

39

Two orthogonal aspects of Grid Two orthogonal aspects of Grid portals usage portals usage

• Number of accessible Grids– Single isolated Grid (SI)– Multiple isolated Grids (MI)– Multiple collaborative Grids (MC)

• Collaborative vice not collaborative usage– Multiple isolated users (MI)– Multiple collaborative users (MC)

40

Portal classificationPortal classification

Multiple Isolated users

(MIxx)

Multiple Collaborative users

(MCxx)

Single Isolated Grid(xxSI)

MISI portals (current portals)

MCSI portals

Multiple Isolated Grids(xxMI) MIMI portals MCMI portals

Multiple Collaborative Grids (xxMC) MIMC portals MCMC portals

41

LCG-2

P-GRADE-Portal

LondonRome

Athens

MISI Portal: P-GRADE portal 1.0MISI Portal: P-GRADE portal 1.0

42



(MIxx)


(MCxx)



MCSI portals



Multi-Grid portals

43

GridLab

SEE-Grid

P-GRADE-Portal

MIMI Portal: P-GRADE portal (2.1)MIMI Portal: P-GRADE portal (2.1)

London Rome

Athens

44



(MIxx)


(MCxx)



MCSI portals



45

GridLab

SEE-Grid

P-GRADE-Portal

London Rome

Athens

MIMC Portal: P-GRADE portal (2.1)MIMC Portal: P-GRADE portal (2.1)

46



(MIxx)


(MCxx)



MCSI portals



47

SEE-Grid

P-GRADE-Portal

MCMC Portal: P-GRADE portal (3.0)MCMC Portal: P-GRADE portal (3.0)

GridLab

LondonRome

Athens

48

MI users WF GUI MC users WF GUI

SI GridWF

manager

MI Grids WF

manager

MC Grids WF

manager

Summary of classificationSummary of classification

.

.

.User

Grid

.

.

.

User

User

.

.. Grid

Grid

Grid

.

..

.

..

.

.

.User

.

.

.

User

User

.

..Grid

Grid

.

..

.

..

.

.

.

Grid

Grid

.

..

.

..

User

.

.

.

User

User

.

..Grid

Grid

.

..

.

..

P-GRADE 2.3P-GRADE 3.0+

49

ConclusionsConclusions

• xxMC portals: execute a workflow simultaneously in many Grids

• P-GRADE portals 2.1 - 2.3 already work as xxMC portal providing service for several production Grids:

– HunGrid– SEE-Grid– VOCE– UK NGS

www.lpds.sztaki.hu/pgportal

• MCxx portal: Enables to collaborate through workflow applications!

• Collaborative P-GRADE Portal: Prototype was demonstrated at SC05 exhibition

50

Thank you for the attention! Thank you for the attention! Questions…Questions…

kacsukkacsuk@@sztaki.husztaki.hu www.lpds.sztaki.huwww.lpds.sztaki.hu

1 [email protected] advanced features of the p-grade portal peter kacsuk, gergely sipos peter...

Documents