grid interoperation on data movement between naregi and egee glite
DESCRIPTION
Grid Interoperation on Data Movement between NAREGI and EGEE gLite. Hideo MATSUDA 1,2 , Yoshiyuki KIDO 3,2 , Kentaro WAKATSUKI 4 1 NAREGI, 2 Osaka University, 3 Mitsui Knowledge Industry Co.,Ltd. 4 Hitachi Software Engineering Co.,Ltd. GIN (Grid Interoperation Now). - PowerPoint PPT PresentationTRANSCRIPT
EGEE-II INFSO-RI-031688
Enabling Grids for E-sciencE
www.eu-egee.org
EGEE and gLite are registered trademarks
GIN Grid Interoperation on Data Movement between NAREGI and EGEE gLite
Hideo MATSUDA1,2, Yoshiyuki KIDO3,2,
Kentaro WAKATSUKI 4
1 NAREGI, 2 Osaka University, 3 Mitsui Knowledge Industry Co.,Ltd.4 Hitachi Software Engineering Co.,Ltd.
To change: View -> Header and Footer 2
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
GIN (Grid Interoperation Now)
• An activity of OGF for interoperation among production grids
• Major grid projects are participating– EGEE, NAREGI, UK National Grid Service, NorduGrid, OSG,
PRAGMA, TeraGrid, ...
• Trying to identify islands of interoperation between production grids and grow those islands
• Areas– GIN-auth: Authorization and Identity Management– GIN-data: Data Management and Movement– GIN-jobs: Job Description and Submission– GIN-info: Information Services and Schema– GIN-ops: Operations Experience of Pilot Test Applications
To change: View -> Header and Footer 3
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
NAREGI GIN Activities
• Developing an interoperation island with EGEE– GIN-jobs, GIN-auth, GIN-info, and GIN-data
To change: View -> Header and Footer 4
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
Architecture
SC06 DemoNAREGI EGEE: using NAREGI WorkflowEGEE NAREGI: using gLite WMS commands
EGEE user NAREGI user
gLite-WMS gLite-BDII NAREGI-ISGIN-BDII
lcgCElcgCE
PreWS-GRAM
gLite-UI NAREGIPortal
Computing ResourceComputing Resource
NAREGIGridVM
WSGRAM
gliteCEgliteCE
NAREGI-GAHP
NAREGI Client LibNAREGI-SSNAREGI-SS
NAREGI-SC
Interop-SC
GIN-jobs: NAREGI-EGEE Architecture & SC06 Demo
To change: View -> Header and Footer 5
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
GIN-auth: Authentication
• IGTF is framework of International Grid Trust Federation.• IGTF consists of APGridPMA, EUGridPMA and TAGPMA.• NAREGI CA joined the APGrid PMA.• NAREGI CA has been approved as a production-level CA by
APGridPMA.
EUGridPMA TAGPMA
APGridPMA
NAREGI PMA
IGTF(International Grid Trust Federation)• GSI compliant with
x.509 proxy certificates for authentication.
• It has become available to use grid computing easily on the worldwide Internet by IGTF.
To change: View -> Header and Footer 6
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
All of grid information can be retrieved by each of grid in its fashion WRTresource description schema, data format, query language, client API, …
Each information service in grid acts as an information provider for the other and translator embedded in the provider performs conversion between different schemas.
Generic Information Provider
GIN-BDII
EGEE OSG NDGF NAREGI TeraGrid Pragma
Cell Domain connecting with BDII
LRPS
OSProcessor
Storage
CIM Providers withGlue=>NRG translator
● ●
JobQueue
Service
OGSA-DAI
Aggregator
RDB
CIM v2.12/w ext.
LDIF
xmlCIM
ARC
-BDII
Glue v1.2
NAREGI
TeraGrid/
MDS4
Glue v1.1
ARC
LDIF providers with X Glue translators :
“Site on a map”
GIN-info: Architecture
To change: View -> Header and Footer 7
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
GIN-data
• Data Management and MovementData Management and Movement
• Agreements:
– Grid FTP is the lowest common denominator for file transfer
– SRM and SRB islands for data management are being established
To change: View -> Header and Footer 8
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
NAREGI Software and Data Grid
RISMJob
LocalScheduler
GridVM
LocalScheduler
IMPIServer
GridVM
FMOJob
LocalScheduler
GridVM
SuperScheduler
WFT
Inputfiles
Work-flow
DataGrid
CA
Site ρ Site α Site μ
3: NegotiationAgreement
6: MPI job starts
9: Accounting
2: Monitoring
4: Reservation
5: IMPI starts
c: Editb2: Data import
2: Resource discovery InformationService
Co-Allocation
GridMPI
RISMSMP machine
64 CPUs
FMOPC cluster128 CPUs
GVS
Output files
Input files
IMPI
8: Visualization1: Submission
Application requirement definition
a: Sign-on
Portal
7: MPI init.
GridMPIDataGrid
Gfarm File System
Network monitor
To change: View -> Header and Footer 9
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
Gfarm File System (1)
• Developed by AIST, Japan.• Commodity-based distributed file system that federates local
disks of compute nodes• It can be shared among all cluster nodes and clients
– Just mount it as if it were high-performance NFS• It provides scalable I/O performance wrt the number of parallel
processes and users• It supports fault tolerance and avoids access concentration by
automatic replica selection
CPU CPU CPU CPU
LAN
. . .
Gfarm file system
To change: View -> Header and Footer 10
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
Gfarm File System (2)
• Files can be shared among all nodes and clients• Physically, it may be replicated and stored on any file
system node• Applications can access it regardless of its location• File system nodes can be distributed
GridFTP, samba, NFS server
Compute & fs node
Compute & fs node
Compute & fs node
Compute & fs node
Compute & fs node
Compute & fs node
GridFTP, samba, NFS server
Gfarm metadata server
Compute & fs node
Compute & fs node
Compute & fs node
Compute & fs node
Compute & fs node
ClientPC
NotePC
/gfarm
metadata
Gfarmfile system
…
File A
File A
File B
File C
File A
File B
File C
File C
File B
EUJapan
To change: View -> Header and Footer 11
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
GIN-data: Architecture
GridFTPServer
EGEE
gLite ClientgLite Client
SRM Client
NAREGI
NAREGI ClientNAREGI Client
SRMClient
GfarmAPI
NAREGIMetadata Server
LFC(Metadata Server)
GfarmServer
DPM(SRM Server)
StorageStorage
• NAREGI and EGEE gLite clients can access to both data resources (e.g., bi-directional file copy) using SRM interface.
• GridFTP is used as its underlying file transfer protocol.• File catalog (metadata) exchange is planned.
To change: View -> Header and Footer 12
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
GIN-data: File Transfer with GridFTP-DSI
• It is not easy to bridge between different file access protocols SRM (gLite) and Gfarm.
• As the 1st step, bi-directional file transfer between gLite and Gfarm using GridFTP.
• Problem: GridFTP authentication (delegation) cannot be directly passed to Gfarm file server.
• GridFTP-DSI (Data Storage Interface) has been used for integrating Gfarm API into GridFTP.
Gfarmmetadatasever
Gfarm file system
GridFTP client
Gfarm file server
GridFTP server
DSI for Gfarm
Gfarm API
Gfarm client lib
Proxy cert
export
Proxycert
GridFTP server for Gfarm access
To change: View -> Header and Footer 13
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
NAREGI(pbg1052.naregi.org)GridFTP server for Gfarm access
EGEE(lxdpm01.cern.ch)
srmcp gsiftp://pbg1052 srm://lxdpm01
GIN-data: SC06 Demo
• SRM copy (srmcp) command was ported in NAREGI.• Bi-directional file transfer can be performed using GridFTP with DSI for Gfarm.
SRM client
SRM (DPM)Server
Gfarm Server
srmcp srm://lxdpm01 gsiftp://pbg1052 Gfarm DSI
To change: View -> Header and Footer 14
Enabling Grids for E-sciencE
EGEE-II INFSO-RI-031688
GIN
Summary
• NAREGI developed EGEE-NAREGI island as an activity of GIN– Bilateral information exchange– Bilateral job submission– Bilateral file exchange– Interoperable security properties
• Next steps– Improve interoperation interfaces and functions
WS-GRAM, BES, JSDL, …– Grow the island with other EGEE partners– KEK will use NAREGI-EGEE interoperation environment for their
high energy physics calculations