www.eu-eela.eu

JRA1 – Application and Infrastructure Grid Services

Francisco Brasileiro

Universidade Federal de Campina Grande – UFCG (Brazil)

Diego Scardaci

Istituto Nazionale di Fisica Nucleare – INFN (Italy)

EELA-2 1st Conference: All Members Meeting

Bogotá, 24.02.2009

www.eu-eela.eu

• Agenda–Role of the JRA1–Services developed in the context of

the activity–How to get help

2Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

• JRA1 Goals–Help in fostering the sustainability of the

e-InfrastructureMaking the e-Infrastructure more interesting

and wide spread by increasing its reach and its usability

Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 3

www.eu-eela.eu

• Increase the reach of the e-infrastructure by:– Allowing the scavenging of idle resources

Create the necessary mechanisms to allow resource centres that run the OurGrid middleware to co-exist with resource centres running gLite within the EELA platform

Provide some level of interoperation between these different kinds of resource centres and their associated applications

– Allowing the execution of the grid middleware on top of platforms other than Scientific Linux Port the gLite middleware to the Windows platform Leveraging on the multi-platform

characteristics of OurGrid

Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 4

www.eu-eela.eu

• Increase the usability of the e-infrastructure by:– Developing new application-oriented grid

services

– Leveraging the grid services provided by the OurGrid middleware to execute bag-of-tasks jobs

– Facilitating the management of resource centres

Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 5

www.eu-eela.eu

• Services developed– Infrastructure-oriented

The main development here allows the co-existence of service grids (gLite-based) with opportunistic grids (OurGrid-based)

More details will be presented tomorrow

– Application-oriented

Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009 6

www.eu-eela.eu

Grid2Win Porting of gLite middleware to Windows

Platform

Dario Russo, Fabio Scibilia, Elisa IngràINFN –Catania

7Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

• The Grid2Win project aims to create a gLite User Interface (UI) and a gLite Computing Element (CE + WNs) running on Microsoft Windows.

Gatekeeper + LRMS

User Interface

Linux WNs

WMProxy

User Interface

WindowsXP Wns

new users come in to the grid !

windows applications can run on the grid !

Grid2Win

• Cygwin emulates a POSIX environment on Windows;

• CE based on PBS and Microsoft Windows Server 2008

8Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

Grid2Win – Main Features

We have ported:

• Grid Security Infrastructure: Credentials management with VOMS extensions commands (voms-proxy-*)

• Workload Management: Job submission and output retrieving commands (glite-wms-job-*)

• File Transfer: Secure transfer with credential management GSIFTP (globus-url-copy) RFIO commands (rf*)

• Data Management: File Catalogue Browsing commands (lfc-*) and data movement & replication (lcg-*)

• Information System: Command lcg-infosites and existing LDAP Explorer integrated in our GUI.

9Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

Links

• Grid2Win on EELA-2 forge:– https://forge.eu-eela.eu/projects/grid2win/

• Grid2Win on EELA-2 wiki:– https://grid.ct.infn.it/twiki/bin/view/EELA2/Grid2Win

10Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

T-GSAF

Transactional Grid Storage Access Framework

Salvatore ScifoConsorzio Cometa – Catania

Salvo ParisiIR&T Engineering s.r.l. - Catania

11Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

GSAF as Design Help• We have common features, we have common problems we need a

Design Pattern

GRID FARM(Redundancy, High Availability, Data Backup&Recovery, High Storage Capability, Net Access Security)

GRID Metadata Service GRID Data Service

Grid Storage Access Framework(Data access, API Fragmentation, Vertical architecture, Knowledgment gap)

Healt Science Application Earth Science Application Cultural Heritage Application

• Built on top of the Grid Metadata Service and Grid Data Service– collects and implements functionalities shared among applications

according to “write once use anywhere” principle

– reduces the knowledge gap hiding the complexity and the fragmentation of the several gLite Data

management APIs exposing a unified interface more near to the developer mind (design patterns)

rather than the Grid stuff details (API syntaxes)

– acts as a black box providing classes and related methods for applications located above interfaces to extend the implemented capabilities

12Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

GSAF - Data Grid Application solution

• GSAF is an Object Oriented Framework– Java 100% - Design Pattern based– Clear design : object modeling / functional modeling– Solves the fragmentation of DMS APIs– Solves the natural (due to OGSA) inconsistence of services– Provides ACID Transactional Model (coming soon)

• Road Map– RFIO java native implementation– SRM encapsulation– Grelc integration– Secure Storage integration

GSAF

LFC API

CNS API RFIO API

Globus Security API

GridFTP API SRM API

Security

POSIX

File Management

File Catalogue

DMS API

AMGA API

Road Map• New– LFC API– CNS API

• AMGA is supported

• only one super interface for Data Grid Application

• modular and extensible

13Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

1° Use Case: DMS Web Interface

14Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

Transactional GSAF

• Transaction Manager– Provides developers with a Transaction Pattern

autocommit(), execute(), commit(), rollback() performs several data manipulation in Atomic Mode

– Hold Runtime Exceptions to limit Data Inconsistence– System Crash Recovery to avoid Data Loss

• Challenge– SOA of Grid middleware makes transactions impossible at

server side– GSAF works at client side

Not full access to resources and services (locks) Not cache mechanism to save temporary data (memory) Not backup system to store previous data status (restore) Not full access control to distributed resources (centralization) …..

15Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

State of the Art

• GSAF – Framework released– Adopters

ADAT Project (Archdiocese of Catania): Cultural Heritage Digital Archive (antique manuscripts)

BM Portal project (Bio-Lab, DIST University of Genoa ) :Service platform to deploy and delivery bio medical and bio informatic applications

Aiuri project (COPPE/UFRJ - BRAZIL):aims to implement a Grid Oriented platform to support data and text mining applications uses GS.

• T-GSAF– Analysis and Design (Completed!)– Development, testing, production (Almost DONE!)

16Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

Links

• GSAF on EELA-2 forge:– https://forge.eu-eela.eu/projects/gsaf/

• GSAF on EELA-2 wiki:– https://grid.ct.infn.it/twiki/bin/view/EELA2/GSAF

17Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

A Secure Storage Service for the gLite Middleware

Diego Scardaci – INFN Catania

Giordano Scuderi – UNICO S.R.L.

18Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

Insider Abuse: the problem

• A grid user could store sensitive data in a Storage Elements managed by external organizations.

• Storage Elements Administrators could access data (but the data are sensitive!). For this reason data MUST be stored in an encrypted format.

• Data Encryption/Decryption MUST be performed inside user secure environment (for example inside the user’s organization).

19Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

A Secure Storage service for the gLite Middleware

• Provides gLite users with suitable and simple tools to store confidential data in storage elements in a transparent and secure way.

The service is composed by the following components:• Command Line Applications: commands integrated in

the gLite User Interface to encrypt/upload and decrypt/ download files.

• Application Program Interface: allows the developer to write programs able to manage confidential data .

• Keystore: a new grid element used to store and retrieve the users’ keys.

20Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

Links

• Secure Storage on EELA-2 forge:– https://forge.eu-eela.eu/projects/securestorage/

• Secure Storage on EELA-2 wiki:– https://grid.ct.infn.it/twiki/bin/view/EELA2/SecureStorage

21Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

WatchDog Monitoring and controlling job execution

on the WorkerNode

Riccardo Bruno

INFN – Catania

22Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

Why do we need a WatchDog ?

• Why– Especially long term jobs require to be monitored and controlled during their

execution.• How

– Perform job control and monitoring using grid services in the less invasive way.• Observations

– Almost all jobs submitted on the grid are piloted by shell scripts Shell scripting allow to get precious info in case of faults Shell scripting can pilot more complex batch execution

– Both AMGA and SE+LFC can be used as the simplest IS on the grid. lfc-* and lcg-* tools already available for file creation and retrieve The latency of CLI tools for the storage is very low compared to long term jobs

• Requirements:– Monitor job execution watching snapshot of files produced by the job execution

File snapshot will be reported on LFC+SE or AMGA servers– It would be useful to configure the monitoring tool accordingly to the user needs

Few shell environment variables can be used to configure the watchdog tool– Control the job execution accessing directly on the WN

It is possible to send commands on the WN23Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

How does the WatchDog work ?

• The Watchdog is a shell script to be included in the JDL main script.

– Some watchdog features: It starts in background before to run the

long term job The watchdog runs as long as the main job The main script can control, stop and wait

until the watchdog has finished Easily and highly configurable and

customizable The watchdog does not compromise the

CPU power of the WN Amga LFC+SE

Commandsor Scripts

JDL

WN

File snapshots

WN

WN

UI

UI UI

24Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

Links

• Watchdog on EELA-2 forge:– https://forge.eu-eela.eu/projects/watchdog/

• WatchDog on EELA-2 wiki:– https://grid.ct.infn.it/twiki/bin/view/EELA2/WatchDog

25Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

Other JRA1 Services

• GENIUS:- https://glite-tutor.ct.infn.it

GFAL Java API:- https://grid.ct.infn.it/twiki/bin/view/EELA2/GFALJAVAAPI- https://grid.ct.infn.it/twiki/GFAL/

gLibrary/DRI:- https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-

UF3.pdf- https://glibrary.ct.infn.it/glibrary/downloads/DRI/gLibraryDRI-

CBMS08.pdfStorage Accounting (SAGE):

- https://forge.eu-eela.eu/projects/sage/

All information available on the JRA1 section of the EELA-2 wiki:

http://wiki.eu-eela.eu → JRA1

26Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009

www.eu-eela.eu

• JRA1 Resources– JRA1 Section in EELA-2 web site

Contains a summary of the activity goals and points to the Wiki site

https://www.eu-eela.eu/

– JRA1 Wiki Site Contains descriptions and links for each JRA1 services https://grid.ct.infn.it/twiki/bin/view/EELA2/JRA1Services

– EELA-Forge Site All released software https://forge.eu-eela.eu/

– Contact us at: Francisco Brasileiro (fubica@dsc.ufcg.edu.br) Diego Scardaci (diego.scardaci@ct.infn.it)

27Bogotá, EELA-2 TB+EAC Meeting, 24.02.2009