CERN-INTAS project No. 03-52-4297

Subtask 6.2: OGSA/Globus evaluation for data intensive

applications by A. Demichev (SINP-MSU)

Objectives & executed work

The subtask objectives : An evaluation of new OGSA/WS(RF)-based Grid technology considered as

a candidate for further stages of LCG Tested/evaluated

Toolkits & MW components potentially useful for gLite OMII Globus Toolkit 4 (GT4)

gLite components The subtask participants:

M. Lamanna (CERN IT) V. Pose (JINR) A. Demichev (SINP-MSU) L. Shamardin (SINP-MSU)

Objectives & executed work (2)

In collaboration with ARDA team (CERN) KIAM RAS (Moscow) JINR (Dubna) JRA1 team (CERN) OMII & GT4 teams

Russian participants visited CERN for works related to the subtask (~ 6 PM)

OMII testing/evaluation (while GT4 was being prepared for release...)

Testing/evaluation of OMII (www.omii.ac.ru) MW was interesting for JRA1 from the point of view of potential use of some its components for EGEE MW

The goal of the evaluation was to realize: how well the components of the OMII MW meets

the needs of a large grid environment with ten thousands of users, tens of big virtual organizations, hundreds of resource centres, ten thousands of jobs, Peta bytes of data etc.

OMII testing/evaluation (2)

Description of the functionality provided by the MW Installation and configuration of the MW on different platforms Testing of performance, concurrency, reliability, usability and

management of the services Job Services Data Services Security

Adding an application Adding a new service

Performance and concurrency of dummy services Interoperability with gLite Workload Management System

(WMS) Quality of OMII Support and Documentation

OMII: general features

OMII has several interesting features Web services architecture Account management Resource consumption accounting Conversation mechanism and authorization

module PBAC Process Based Access Control Easy and compact installation (but restricted to

the certain OS) The provided documentation clearly covers the main

topics, but at some points (e.g. budget management, service creation) it was not clear and comprehensive enough

OMII: Some results on performance, concurrency,

reliability and usability Job Service

remained stable with up to 20 concurrent clients a maximal job submission rate of about 6 jobs/min for a 2.4GHz P4 CPU server

node stability: in case of sequential submission of multiple jobs by a user an error

occurred near the 180-th job submission Data Service

reliability: all 1000 upload + download cycles of 10MB big files were successful concurrency: all upload + download cycles of 1..100KB big files were successful

with up to 5 concurrent clients Dummy Services

response time for a non-PBAC dummy service - about 0.1sec with PBAC about 0.7sec

Concurrency - 100 parallel clients executed successfully Adding simple applications is easy A simple new service has been easily deployed

OMII: Evaluation results

OMII is oriented towards web services rather than Grid architecture

There are no Workload Management System & Resource Broker, Information System, Logging and Bookkeeping, Replica Catalogs and Replica Management System

OMII does not support GSI and proxy credentials Virtual Organizations management

OMII: Evaluation results (2)

No interoperability with Globus Toolkit and the gLite WMS

Management and administrative techniques are intended to servicing individual servers and small projects but not large VOs or extended resource infrastructure

More details on OMII evaluation: http://lit.jinr.ru/Reports/annual-report05/omni-55.pdf http://theory.sinp.msu.ru/~demichev/omii/OMII_evaluation-

EGEE3.ppt

OMII's plans...

OMII Technical Roadmap (mid. 2005):“We will collaborate with EGEE (in particular CERN) to ensure that uses of the EGEE software (in particular gLite) will be able to have solutions that interoperate with OMII hosted components.”

Globus Toolkit 4 (GT4) Testing

GT4.0.0 was installed and properly configured on a

number of platforms (in total on 7 machines): Fedora Core 3 – using the Fedora Core 3 Binary Installer

Redhat 9 – using the RedHat 9 Binary Installer + Patches from

http://www.globus.org/toolkit/downloads/4.0.0/

SLC3 (Scientific Linux Cern Release 3.0.4) - using the

Source Instsaller

SL4 (Scientific Linux Release 4.0) - using the RPMs file

from ftp://ftp.linux-ink.ru/pub/SL/40/projects/GRID+HEP

GT4 studying, testing and evaluation

WSRF implementations comparison: GT4-Java (Java WS Core of GT4);

GT4-C (C WS Core of GT4); pyGridWare (Python WS Core); the Perl-based WSRF::Lite; Functional comparison:

Significant commonalities are seen with regard to

dispatching and SOAP processing techniques, though in

each implementation they are different

GT4 studying, testing and evaluation (2)

All the GT4 WSRF implementations Security: support three security protocols:

TLS/SSL transport-level security protocol; Secure Conversation SecureMessage.

WSRF::Lite supports only transport-level security. Persist WS-Resources in memory by default and come

with modules that allow resources to be saved to disk,

providing the ability to survive server failure at the cost of

some performance. Lifetime Management includes resource creation and

resource destruction.

GT4 studying, testing and evaluation (3)

WS-Notification. GT4-Java and pyGridWare do not implement WS-Brokered

Notification and have some other limitations. GT4-C does not implement producer-side notification. WSRF::Lite does not support any Notification specifications.

Authorization. GT4-Java and pyGridWare allows the service developer to provide

custom authorization. GT4-C implements three built-in mechanisms. WSRF::Lite client passes security information to the WS-Resource

through environment variables, leaving the WS-Resource

implementation to implement its own authorization.

GT4 studying, testing and evaluation (4)

Interoperability. Being consistent with the WS-Interoperability, the five

implementations achieve a base level of interoperability with

regard to XML, HTTP, SOAP, and WSDL. But the implementations, nevertheless, are not fully

interoperable. Namespace incompatibilities are the key

concern, as well as several technical details (HTTP headers,

application-specific portions of messages) which lies outside

the WSRF/WSN specifications). By now: OASIS has accepted the WSRF 1.2 (WS-Resource, WS-

ResourceProperties, WS-ResourceLifetime, WS-ServiceGroup, WS-

BaseFaults) as an OASIS standard.

GT4 Testing

Estimation of the overhead expenses of GT4 Java container overhead expenses of GT4 Java container (i.e. first part which is

common for invocation of all services) and client application. Estimation of average times for common WSRF operations

average times for common WSRF operations (in GT4 Java container) on server side and client side and investigation the influence of the number of created resources on the request time.

Estimation of GT4 Java container scalability and robustness GT4 Java container scalability = maximum possible number of

parallel invocations before server failure). GridFTP and RFT testing Further details and numerical results of the testing are available at

http://theory.sinp.msu.ru/~demichev/GT4evaluation.doc

gLite – GT4 WSS compatipility

Studying of the gLite – GT4 WorkSpace Service (WSS) compatibility was carried out by special request from JRA1.

WSS should allow to enforce VO politics in a way which does not contradict to local sites politics.

The goal was to verify compatibility of WSS with gLite and evaluate the service

Succeeded in combining gLite+WSS At the time of the study (Fall of 2005) WSS

proved to be too immature and unstable to be considered as real candidate for gLite component.

Comparative studying: GT4 - gLite

In each main area gLite has services, which are absent in

GT4, but are substantial: Security. Virtual organization support (VOMS gLite); Execution Management - Workload Management System; Data Management - support of storage resource internal

organization: SE, SRM, file catalogues R-GMA (gLite) vs. MDS4: essentially different

approaches: MDS4 - hierarchical R-GMA - relational

Glite Testing

The subtask team participated in testing gLite components: Fireman (file + replica) and Metadata catalogs

http://lcg.web.cern.ch/lcg/PEB/arda/public_docs/CHEP04/CaseStudies/Fireman1.doc

IO-server http://lcg.web.cern.ch/lcg/PEB/arda/public_docs/CHEP04/CaseStudies/gLiteIO

%20-%20testing-v2.pdf

Workload Management System

http://lcg.web.cern.ch/lcg/activities/arda/public_docs/2005/Q1/ARDAatTaiwan-apr05.ppt

AMGA Metadata Service (V.Pose's report at GRID'2006 on 28.06)

The work was carried out in ARDA framework (partially supported by the

INTAS-CERN project) and in close collaboration with JRA1 Team