grid’05 egee summer school budapest - 1

58
GRID’05 EGEE Summer School Budapest - 1 http://egee.hu/grid05/index.ph p?m=3

Upload: willis-thompson

Post on 20-Jan-2018

222 views

Category:

Documents


0 download

DESCRIPTION

GRID’05 EGEE Summer School Budapest - 3 Overview Introduction on Data Management (DM)  General Concepts  Some details on transport protocols  Data management operations  Files & replicas: Name Convention File catalogs  Cataloging requirements and catalogs in egee/LCG  RLS file catalog  LCG file catalog DM tools: overview Data Management CLI  lcg_utils Data Management API  lcg_utils  GFAL Advanced concepts  Advanced utilities: CLI&APIs  OutputData JDL attribute Conclusions

TRANSCRIPT

GRID’05 EGEE Summer School Budapest - 1

http://egee.hu/grid05/index.php?m=3

EGEE is a project funded by the European Union under contract IST-2003-508833

Grid Data Management

Gabor Hermann on the base of lecture of

Simone Campana LCG Experiment Integration and Support

CERN IT

www.eu-egee.org

GRID’05 EGEE Summer School Budapest - 3

Overview• Introduction on Data Management (DM)

General Concepts Some details on transport protocols Data management operations Files & replicas: Name Convention

• File catalogs Cataloging requirements and catalogs in egee/LCG RLS file catalog LCG file catalog

• DM tools: overview• Data Management CLI

lcg_utils• Data Management API

lcg_utils GFAL

• Advanced concepts Advanced utilities: CLI&APIs OutputData JDL attribute

• Conclusions

GRID’05 EGEE Summer School Budapest - 4

Overview• Introduction on Data Management (DM)

General Concepts Some details on transport protocols Data management operations Files & replicas: Name Convention

• File catalogs Cataloging requirements and catalogs in egee/LCG RLS file catalog LCG file catalog

• DM tools: overview• Data Management CLI

lcg_utils• Data Management API

lcg_utils GFAL

• Advanced concepts Advanced utilities: CLI&APIs OutputData JDL attribute

• Conclusions

GRID’05 EGEE Summer School Budapest - 5

Data Management: general concepts

• What does “Data Management” mean? Users and applications produce and require data Data may be stored in Grid files Granularity is at the “file” level (no data “structures”) Users and applications need to handle files on the Grid

• Files are stored in appropriate permanent resources called “Storage Elements” (SE) Present almost at every site together with computing

resources We will treat a storage element as a “black box”

where we can store data• Appropriate data management utilities/services hide

internal structure of SE• Appropriate data management utilities/services hide

details on transfer protocols

GRID’05 EGEE Summer School Budapest - 6

Data Management: general concepts

• A Grid file is READ-ONLY (at least in egee/LCG) It can not be modified It can be deleted (so it can be replaced) Files are heterogeneous (ascii, binary …)

• High level Data Management tools (lcg_utils, see later) hide transport layer details (protocols …) Storage location

• To use lower level tools (edg-gridftp, see later ) you need some knowledge of the transport layer some knowledge of Storage Element implementation

GRID’05 EGEE Summer School Budapest - 7

Some details on protocols

• Data channel protocol: mostly gridFTP (gsiftpgsiftp) secure and efficient data movement extends the standard FTP protocol Public-key-based Grid Security Infrastructure (GSI) support Third-party control of data transfer Parallel data transfer

• Other protocols are available, especially for File I/O rfio protocolrfio protocol:

• for CASTOR SE (and classic SE)• Not yet GSI enabled

gsidcap protocolgsidcap protocol:• for secure access to dCache SE

file protocolfile protocol: • for local file access

• Other Control Channel Protocols (SRM, discussed in SE lecture … )

GRID’05 EGEE Summer School Budapest - 8

Data Management operations

UI

SECECE

SECECE

Several Grid ComponentsSeveral Grid Components

Upload a file to the gridUpload a file to the grid

• U ser need to store data in SE (from a U I)

• Application need to store data in SE (from a CE)

• U ser need to store the application (to be retrieved and run on a CE) For small files the

InputSandbox can be used (see WMS lecture)

GRID’05 EGEE Summer School Budapest - 9

Download files from the gridDownload files from the grid

• User need to retrieve (onto the UI) data stored into SE For small files produced in WN

the OutputSandbox can be used (see WMS lecture)

• Applications need to copy data locally (into the CE) and use them

• The application itself must be downloaded onto the CE and run

Data Management operations

UI

SECECE

SECECE

Several Grid ComponentsSeveral Grid Components

GRID’05 EGEE Summer School Budapest - 10

Replicate a file across Replicate a file across different SEsdifferent SEs

• Load share balacing of computing resources Often a job needs to run at a

site where a copy of input data is present

See InputData JDL attribute in WMS lecture

• Performance improvement in data access Several applications might

need to access the same file concurrently

• Important for redundancy of key files (backup)

Data Management operations

UI

SECECE

SECECE

Several Grid ComponentsSeveral Grid Components

GRID’05 EGEE Summer School Budapest - 11

One of the base idea of LCG:

Let us bring the little programs close to the big files

Asymmetry in JDL:

• In given situation it is the task of the user to copy the GRID files mentioned in Input Data to the CE

•The JDL supports the creating of GRID files from local files via Output Data

GRID’05 EGEE Summer School Budapest - 12

Data management operations

• Data Management means movement and replication of files across/on grid elements

• Grid DM tools/applications/services can be used for all kind of files

HOWEVERHOWEVER

• Data Management focuses on “large” files large means greater than ~20MB Tipically on the order of few hundreds MB

• Tools/applications/services are optimized to deal with large files• In many cases, small files can be efficiently treated using

different procedures Examples:

• User can ship data to be used by the application on the WN (and possibly the application itself) using the InputSandbox (see WMS lecture)

• User can retrieve (on the UI) data generated by a job (on the WN) using the OutputSandbox (see WMS lecture)

GRID’05 EGEE Summer School Budapest - 13

Files & replicas: Name Conventions

• Logical File Name (LFN) An alias created by a user to refer to some item of data, e.g.

“lfn:cms/20030203/run2/track1”

• Globally Unique Identifier (GUID) A non-human-readable unique identifier for an item of data, e.g.

“guid:f81d4fae-7dec-11d0-a765-00a0c91e6bf6”

• Site URL (SURL) (or Physical File Name (PFN) or Site FN) The location of an actual piece of data on a storage system, e.g.

“srm://pcrd24.cern.ch/flatfiles/cms/output10_1” (SRM) “sfn://lxshare0209.cern.ch/data/alice/ntuples.dat” (Classic SE)

• Transport URL (TURL) Temporary locator of a replica + access protocol: understood by a SE, e.g.

“rfio://lxshare0209.cern.ch//data/alice/ntuples.dat”

Logical File Name 1

Logical File Name n

GUID

Physical File SURL n

Physical File SURL 1.. ..

TURL 1

TURL n

..

GRID’05 EGEE Summer School Budapest - 14

Overview• Introduction on Data Management (DM)

General Concepts Some details on transport protocols Data management operations Files & replicas: Name Convention

• File catalogs Cataloging requirements and catalogs in egee/LCG RLS file catalog LCG file catalog

• DM tools: overview• Data Management CLI

lcg_utils• Data Management API

lcg_utils GFAL

• Advanced concepts Advanced utilities: CLI&APIs OutputData JDL attribute

• Conclusions

GRID’05 EGEE Summer School Budapest - 15

File Catalogs

At this point you should ask:

1) How do I keep track of all my files on the Grid?2) Even if I remember all the lfns of my files, what about

someone else files?3) Anyway, how does the Grid keep track of associations

lfn/GUID/surl?

Well… we need a FILE CATALOGUEWell… we need a FILE CATALOGUE

GRID’05 EGEE Summer School Budapest - 16

Cataloging Requirements

• Need to keep track of the location of copies (replicas) of Grid files • Replicas might be described by attributes

Support for METADATA Could be “system” metadata or “user” metadata

• Potentially, milions of files need to be registered and located Requirement for performance

• Distributed architecture might be desirable scalability prevent single-point of failure Site managers need to change autonomously file locations

GRID’05 EGEE Summer School Budapest - 17

File Catalogs in egee/LCG

• Access to the file catalog The DM tools and APIs and the WMS interact with the catalog

• Hide catalogue implementation details Lower level tools allow direct catalogue access

• EDG’s Replica Location Service (RLS) Catalogs in use in LCG-2 Replica Metadata Catalog (RMC) + Local Replica Catalog (LRC) Some performance problems detected during LCG Data Challenges

• New LCG File Catalog (LCF) Already being certified; deployment in January 2005 Coexistence with RLS and migration tools provided Better performance and scalability Provides new features: security, hierarchical namespace, transactions...

GRID’05 EGEE Summer School Budapest - 18

Overview of File catalogues

GRID’05 EGEE Summer School Budapest - 19

• RMC: Stores LFN-GUID mappings Accessible by edg-rmc CLI + API

• LRC: Stores GUID-SURL mappings Accessible by edg-lrc CLI + API

File Catalogs: The RLS

Logical File Name 1

Logical File Name 2

Logical File Name n

GUIDPhysical File SURL n

Physical File SURL 1

RMCLRC

DMLRC

RMC

GRID’05 EGEE Summer School Budapest - 20

File Catalogs: The LFC

• One single catalog• LFN acts as main key in the database. It has:

Symbolic links to it (additional LFNs) Unique Identifier (GUID) System metadata Information on replicas One field of user metadata

GUID

Xxxxxx-xxxx-xxx-xxx-

System Metadata

“size” => 10234“cksum_type” => “MD5”“cksum” => “yy-yy-yy”

Symlink

/grid/dteam/mydir/mylink

Replica

srm://host.example.com/foo/barhost.example.com

Replica

srm://host.example.com/foo/barhost.example.com

Replica

srm://host.example.com/foo/barhost.example.com

Replica

srm://host.example.com/foo/barhost.example.com

Symlink

/grid/dteam/mydir/mylink

Symlink

/grid/dteam/mydir/mylink

LFN

/grid/dteam/dir1/dir2/file1.root

User Metadata

User Defined Metadata

GRID’05 EGEE Summer School Budapest - 21

File Catalogs: The LFC (II)

• Fixes performance and scalability problems seen in EDG Catalogs Cursors for large queries Timeouts and retries from the client

• Provides more features than the EDG Catalogs User exposed transaction API (+ auto rollback on failure of mutating method call) Hierarchical namespace and namespace operations (for LFNs) /grid/<VO>/….. Integrated GSI Authentication + Authorization Access Control Lists (Unix Permissions and POSIX ACLs) Checksums

• Interaction with other components Supports Oracle and MySQL database backends Integration with GFAL and lcg_util APIs complete New specific API provided

• New features will be added (requests welcome!) ROOT Integration in progress POOL Integration will be provided soon VOMS will be integrated

GRID’05 EGEE Summer School Budapest - 22

LFC commands

lfc-chmod Change access mode of the LFC file/directory

lfc-chown Change owner and group of the LFC file-directory

lfc-delcomment Delete the comment associated with the file/directory

lfc-getacl Get file/directory access control lists

lfc-ln Make a symbolic link to a file/directory

lfc-ls List file/directory entries in a directory

lfc-mkdir Create a directory

lfc-rename Rename a file/directory

lfc-rm Remove a file/directory

lfc-setacl Set file/directory access control lists

lfc-setcomment Add/replace a comment

Summary of the LFC Catalog commands

GRID’05 EGEE Summer School Budapest - 23

LFC C API

lfc_deleteclass

lfc_delreplica

lfc_endtrans

lfc_enterclass

lfc_errmsg

lfc_getacl

lfc_getcomment

lfc_getcwd

lfc_getpath

lfc_lchown

lfc_listclass

lfc_listlinks

lfc_listreplica

lfc_lstat

lfc_mkdir

lfc_modifyclass

lfc_opendir

lfc_queryclass

lfc_readdir

lfc_readlink

lfc_rename

lfc_rewind

lfc_rmdir

lfc_selectsrvr

lfc_setacl

lfc_setatime

lfc_setcomment

lfc_seterrbuf

lfc_setfsize

lfc_starttrans

lfc_stat

lfc_symlink

lfc_umask

lfc_undelete

lfc_unlink

lfc_utime

send2lfc

lfc_access

lfc_aborttrans

lfc_addreplica

lfc_apiinit

lfc_chclass

lfc_chdir

lfc_chmod

lfc_chown

lfc_closedir

lfc_creat

lfc_delcomment

lfc_delete

Low level methods (many POSIX-like):

GRID’05 EGEE Summer School Budapest - 24

• Important environment variables:

• export LCG_GFAL_INFOSYS=grid152.kfki.hu:2170 Must be set for each catalogue type

• export LCG_CATALOG_TYPE=lfc

Must be set only for LFC • export LFC_HOST=grid155.kfki.hu

Must be set only for LFC

GRID’05 EGEE Summer School Budapest - 25

Overview• Introduction on Data Management (DM)

General Concepts Some details on transport protocols Data management operations Files & replicas: Name Convention

• File catalogs Cataloging requirements and catalogs in egee/LCG RLS file catalog LCG file catalog

• DM tools: overview• Data Management CLI

lcg_utils• Data Management API

lcg_utils GFAL

• Advanced concepts Advanced utilities: CLI&APIs OutputData JDL attribute

• Conclusions

GRID’05 EGEE Summer School Budapest - 26

DM CLIs & APIs overview

User Tools

Cataloging Storage Data transfer

Data Management (Replication, Indexing, Querying)

EDG LFC SRM Classic SE

GridFTP bbFTP

File I/O

RFIO DCAP

lcg_utils: CLI + C API edg-rm: CLI + API

GFAL C API GFAL C API GFAL C API (GFAL C API)

edg-rmc edg-lrcCLI + API

rfio API

dcap API

edg- gridtp Globus API

bbFTP API

CLI+API

SRM API

GRID’05 EGEE Summer School Budapest - 27

SRM Storage Management

GRID’05 EGEE Summer School Budapest - 28

Data management tools

• Replica manager: lcg-* commands + lcg_* API Provide (all) the functionality needed by the egee/LCG user Combine file transfer and cataloging as an atomic transaction Insure consistent operations on catalogues and storage systems Offers high level layer over technology specific implementations Based on the Grid File Access Library (GFAL) API

• Discussed in SE section

• edg-gridftp tools: CLI Complete the lcg_utils with GridFTP operations Lower level layer w.r.t. Replica Manager

• Only for gridFTP protocol Functionality available in GFAL May be implemented as lcg-* commands

GRID’05 EGEE Summer School Budapest - 29

DM CLIs & APIs: Old EDG tools

• Old versions of EDG CLIs and APIs still available

• File & replica management edg-rm

• Implemented (mostly) in java

• Catalog interaction (only for EDG catalogs) edg-lrc edg-rmc

• Java and C++ APIs

• Use discouraged Worse performance (slower) New features added only to lcg_utils Less general than GFAL and lcg_utils

GRID’05 EGEE Summer School Budapest - 30

Overview• Introduction on Data Management (DM)

General Concepts Some details on transport protocols Data management operations Files & replicas: Name Convention

• File catalogs Cataloging requirements and catalogs in egee/LCG RLS file catalog LCG file catalog

• DM tools: overview• Data Management CLI

lcg_utils• Data Management API

lcg_utils GFAL

• Advanced concepts Advanced utilities: CLI&APIs OutputData JDL attribute

• Conclusions

GRID’05 EGEE Summer School Budapest - 32

lcg-utils commands

Replica Management lcg-cp Copies a grid file to a local destination

lcg-cr Copies a file to a SE and registers the file in the catalog

lcg-del Delete one file

lcg-rep Replication between SEs and registration of the replica

lcg-gt Gets the TURL for a given SURL and transfer protocol

lcg-sd Sets file status to “Done” for a given SURL in a SRM request

File Catalog Interactionlcg-aa Add an alias in LFC for a given GUID

lcg-ra Remove an alias in LFC for a given GUID

lcg-rf Registers in LFC a file placed in a SE

lcg-uf Unregisters in LFC a file placed in a SE

lcg-la Lists the alias for a given SURL, GUID or LFN

lcg-lg Get the GUID for a given LFN or SURL

lcg-lr Lists the replicas for a given GUID, SURL or LFN

GRID’05 EGEE Summer School Budapest - 33

Gathering informations: lcg-infosites

[scampana@grid019:~]$ lcg-infosites --vo gilda se

*************************************************************These are the related data for gilda: (in terms of SE)*************************************************************

Avail Space(Kb) Used Space(Kb) SEs----------------------------------------------------------1570665704 576686868 grid3.na.astro.it225661244 1906716 grid009.ct.infn.it523094840 457000 grid003.cecalc.ula.ve1570665704 576686868 testbed005.cnaf.infn.it15853516 1879992 gilda-se01.pd.infn.it

GRID’05 EGEE Summer School Budapest - 34

lcg_utils CLI : usage example

We have a local file in our UI in Catania

[scampana@grid019:~]$ lcg-cr --vo gilda -l lfn:simone-important \ -d grid3.na.astro.it file://`pwd`/important-file.txt

guid:08d02e56-bdf6-4833-a4da-e0247c188242

Upload the file in Naples (Italy)

[scampana@grid019:~]$ ls -l important-file.txt-rw-r--r-- 1 scampana users 19 Oct 31 17:09 important-file.txt

[scampana@grid019:~]$ lcg-lr --vo gilda lfn:simone-important

sfn://grid3.na.astro.it/flatfiles/SE00/gilda/generated/2004-10-31/ \file4c7c2ad6-4d93-4cd2-be24-bf4239f58208

The file is effectively there … …. Let’ s replicate it to Merida now …

[scampana@grid019:~]$ lcg-rep --vo gilda \-d grid003.cecalc.ula.ve lfn:simone-important

[scampana@grid019:~]$ lcg-lr --vo gilda lfn:simone-important

sfn://grid003.cecalc.ula.ve/flatfiles/SE00/gilda/generated/2004-10-31/ \file39568d15-e873-4f17-9371-b8862ae77c36sfn://grid3.na.astro.it/flatfiles/SE00/gilda/generated/2004-10-31/ \file4c7c2ad6-4d93-4cd2-be24-bf4239f58208

[scampana@grid019:~]$ lcg-del --vo gilda -a lfn:simone-important

[scampana@grid019:~]$ lcg-lr --vo gilda lfn:simone-importantlcg_lr: No such file or directory

Delete all the replicas in the storage elements.

IMPORTANTIMPORTANT

The lcg_utils (both CLI and API described later) need to access the Information System (BDII).

The name of the BDII host used by lcg_utils is specified in the environment variable LCG_GFAL_INFOSYS

REMEMBER THAT, ESPECIALLY WHEN PERFORMING REMEMBER THAT, ESPECIALLY WHEN PERFORMING DATA MANAGEMENT OPERATIONS FROM THE WNDATA MANAGEMENT OPERATIONS FROM THE WN

GRID’05 EGEE Summer School Budapest - 35

Overview• Introduction on Data Management (DM)

General Concepts Some details on transport protocols Data management operations Files & replicas: Name Convention

• File catalogs Cataloging requirements and catalogs in egee/LCG RLS file catalog LCG file catalog

• DM tools: overview• Data Management CLI

lcg_utils• Data Management API

lcg_utils GFAL

• Advanced concepts Advanced utilities: CLI&APIs OutputData JDL attribute

• Conclusions

GRID’05 EGEE Summer School Budapest - 36

lcg_utils API

• lcg_utils API: High-level data management C API Same functionality as lcg_util command line tools

• Single shared library liblcg_util.so

• Single header file lcg_util.h

(+ linking against libglobus_gass_copy_gcc32.so)

GRID’05 EGEE Summer School Budapest - 37

lcg_utils: Replica management

int lcg_cp (char *src_file, char *dest_file, char *vo, int nbstreams, char * conf_file, int insecure, int insecure);

int lcg_cr (char *src_file, char *dest_file, char *guid, char *lfn, char *vo, char *relative_path, int nbstreams, char *conf_file, int insecure, int verbose, char *actual_guid);

int lcg_del (char *file, int aflag, char *se, char *vo, char *conf_file, int insecure, int verbose);

int lcg_rep (char *src_file, char *dest_file, char *vo, char *relative_path, int nbstreams, char *conf_file, int insecure, int verbose);

int lcg_sd (char *surl, int regid, int fileid, char *token, int oflag);

GRID’05 EGEE Summer School Budapest - 38

lcg_utils: Catalog interaction

int lcg_aa (char *lfn, char *guid, char *vo, char *insecure, int verbose);

int lcg_gt (char *surl, char *protocol, char **turl, int *regid, int *fileid, char **token);

int lcg_la (char *file, char *vo, char *conf_file, int insecure, char ***lfns);

int lcg_lg (char *lfn_or_surl, char *vo, char *conf_file, int insecure, char *guid);

int lcg_lr (char *file, char *vo, char *conf_file, int insecure, char ***pfns);

int lcg_ra (char *lfn, char *guid, char *vo, char *conf_file, int insecure);

int lcg_rf (char *surl, char *guid, char *lfn, char *vo, char *conf_file, int insecure, int verbose, char *actual_guid);

int lcg_uf (char *surl, char *guid, char *vo, char *conf_file, int insecure);

GRID’05 EGEE Summer School Budapest - 39

Available APIs

#include <iostream> #include <stdlib.h> #include <string.h> #include <string> #include <stdio.h> #include <errno.h>

// lcg_util is a C library. Since we write C++ code here, we need to // use extern C // extern "C" { #include <lcg_util.h> } using namespace std; /***************************************************************************************/ /* The folling example code shows you how you can use the lcg_util API for */ /* replica management. We expect that you modify parts of this code in */ /* to make it work in your environment. This is particularly indicated */ /* by ACTION, i.e. your action is required. */ /**************************************************************************************/int main () { cout << "Data Management API Example " << endl; char *vo = "cms"; // ACTION: fill in your correct VO here: gilda ! cout << "---------------------------------------------------" << endl;

C APIs

GRID’05 EGEE Summer School Budapest - 40

Available APIs

// Copy a local file to the Storage Element and register it in RLS // char *localFile = "file:/tmp/test-file"; // ACTION: create a testfile char *destSE = "lxb0707.cern.ch"; // ACTION: fill in a specific SE char

*actualGuid = (char*) malloc(50); int verbose = 2; // we use verbosity level 2 int nbstreams = 8; // we use 8 parallel streams to transfer a file

lcg_cr(localFile, destSE, NULL, NULL, vo, NULL, nbstreams, NULL, 0, verbose, actualGuid);

if (errno) { perror("Error in copyAndRegister:"); return -1; } else { cout << "We registered the file with GUID: " << actualGuid << endl; } cout << "---------------------------------------------------" << endl;

Copy and Register

GRID’05 EGEE Summer School Budapest - 41

Available APIs

// Call the listReplicas (lcg_lr) method and print the returned URLs // // The actualGuid does not contain the prefix "guid:". We add it here and // then use the new guid as a parameter to list replicas // std::string guid = "guid:"; guid.insert(5,actualGuid); char ***pfns = (char***) malloc(200);

lcg_lr((char*) guid.c_str(), vo, NULL, 0, pfns);

if(errno) { perror("Error in listReplicas:"); free(pfns); return -1; } else { cout << "PFN = " << **pfns << endl; }

free(pfns);

cout << "---------------------------------------------------" << endl;

List Replicas

GRID’05 EGEE Summer School Budapest - 42

Available APIs

// Delete the replica again // int rc = lcg_del((char*) guid.c_str(), 1, destSE, vo, NULL, 0, verbose);

if(rc != 0) { perror("Error in delete:"); return -1; } else { cout << "Delete OK" << endl; }

return 0;

}

Delete Replica

GRID’05 EGEE Summer School Budapest - 43

Available APIs

CC = g++ GLOBUS_FLAVOR = gcc32

all: data-management

data-management: data-management.o $(CC) -o data-management \

-L${GLOBUS_LOCATION}/lib -lglobus_gass_copy_${GLOBUS_FLAVOR} \ -L${LCG_LOCATION}/lib -llcg_util -lgfal \ data-management.o

data-management.o: data-management.cpp $(CC) -I ${LCG_LOCATION}/include -c data-management.cpp

clean: rm -rf data-management data-management.o

Makefile used

GRID’05 EGEE Summer School Budapest - 44

Overview• Introduction on Data Management (DM)

General Concepts Some details on transport protocols Data management operations Files & replicas: Name Convention

• File catalogs Cataloging requirements and catalogs in egee/LCG RLS file catalog LCG file catalog

• DM tools: overview• Data Management CLI

lcg_utils• Data Management API

lcg_utils GFAL

• Advanced concepts Advanced utilities: CLI&APIs OutputData JDL attribute

• Conclusions

GRID’05 EGEE Summer School Budapest - 45

Grid File Access Library

• GFAL is a library to provide access to Grid files File I/O, Catalog Interaction, Storage Interaction

• Abstraction from specific implementations

• Transparent interaction with the information service, the file catalogs…

• Single shared library in threaded and unthreaded versions libgfal.so, libgfal_pthr.so

• Single header file gfal_api.h

Cataloging Storage Data transfer

Data Management (Replication, Indexing, Querying)

EDG LFC SRM Classic SE

gridftp RDT

File I/O

rfio dcap

GRID’05 EGEE Summer School Budapest - 46

GFAL: Catalog API

int create_alias (const char *guid, const char *lfn, long long size)

int guid_exists (const char *guid)

char *guidforpfn (const char *surl)

char *guidfromlfn (const char *lfn)

char **lfnsforguid (const char *guid)

int register_alias (const char *guid, const char *lfn)

int register_pfn (const char *guid, const char *surl)

int setfilesize (const char *surl, long long size)

char *surlfromguid (const char *guid)

char **surlsfromguid (const char *guid)

int unregister_alias (const char *guid, const char *lfn)

int unregister_pfn (const char *guid, const char *surl)

GRID’05 EGEE Summer School Budapest - 47

GFAL: Storage API

int deletesurl (const char *surl)

int getfilemd (const char *surl, struct stat64 *statbuf)

int set_xfer_done (const char *surl, int reqid, int fileid, char *token, int oflag)

int set_xfer_running (const char *surl, int reqid, int fileid, char *token)

char *turlfromsurl (const char *surl, char **protocols, int oflag, int *reqid, int *fileid, char **token)

int srm_get (int nbfiles, char **surls, int nbprotocols, char **protocols, int *reqid, char **token, struct srm_filestatus **filestatuses)

int srm_getstatus (int nbfiles, char **surls, int reqid, char *token, struct srm_filestatus **filestatuses)

GRID’05 EGEE Summer School Budapest - 48

GFAL: File I/O API (I)int gfal_access (const char *path, int amode);

int gfal_chmod (const char *path, mode_t mode);

int gfal_close (int fd);

int gfal_creat (const char *filename, mode_t mode);

off_t gfal_lseek (int fd, off_t offset, int whence);

int gfal_open (const char * filename, int flags, mode_t mode);

ssize_t gfal_read (int fd, void *buf, size_t size);

int gfal_rename (const char *old_name, const char *new_name);

ssize_t gfal_setfilchg (int, const void *, size_t);

int gfal_stat (const char *filename, struct stat *statbuf);

int gfal_unlink (const char *filename);

ssize_t gfal_write (int fd, const void *buf, size_t size);

GRID’05 EGEE Summer School Budapest - 49

GFAL protocol of File Open

GRID’05 EGEE Summer School Budapest - 50

GFAL: File I/O API (II)

int gfal_closedir (DIR *dirp);

int gfal_mkdir (const char *dirname, mode_t mode);

DIR *gfal_opendir (const char *dirname);

struct dirent *gfal_readdir (DIR *dirp);

int gfal_rmdir (const char *dirname);

GRID’05 EGEE Summer School Budapest - 51

Overview• Introduction on Data Management (DM)

General Concepts Some details on transport protocols Data management operations Files & replicas: Name Convention

• File catalogs Cataloging requirements and catalogs in egee/LCG RLS file catalog LCG file catalog

• DM tools: overview• Data Management CLI

lcg_utils• Data Management API

lcg_utils GFAL

• Advanced concepts Advanced utilities: CLI&APIs OutputData JDL attribute

• Conclusions

GRID’05 EGEE Summer School Budapest - 52

Advanced utilities: edg-gridftp

edg-gridftp-exists TURL Checks if file/dir exists on a SE

edg-gridftp-ls TURL Lists a directory on a SE

globus-url-copy srcTURL dstTURL Copies files between SEs

edg-gridftp-mkdir TURL Creates a directory on a SE

edg-gridftp-rename srcTURL dstTURL Renames a file on a SE

edg-gridftp-rm TURL Removes a file from a SE

edg-gridftp-rmdir TURL Removes a directory on a SE

Used for low level management of file/directories in SEsUsed for low level management of file/directories in SEs

GRID’05 EGEE Summer School Budapest - 53

edg-gridftp example

CreateCreate and and deletedelete a directory in a GILDA Storage Element a directory in a GILDA Storage Element

GRID’05 EGEE Summer School Budapest - 54

Other Advanced CLI&API

• globus-url-copy srcTURL destTURL low level file transfer

• Interaction with RLS components edg-lrc command (actions on LRC) edg-rmc command (actions on RMC) C++ and Java API for all catalog operations

• http://edg-wp2.web.cern.ch/edg-wp2/replication/docu/r2.1/edg-lrc-devguide.pdf• http://edg-wp2.web.cern.ch/edg-wp2/replication/docu/r2.1/edg-rmc-devguide.pdf

• Using low level CLI and API is STRONGLY discouragedUsing low level CLI and API is STRONGLY discouraged Risk: loose consistency between SEs and catalogues REMEMBERREMEMBER: a file is in Grid if it is BOTH:BOTH:

• stored in a Storage Element• registered in the file catalog

GRID’05 EGEE Summer School Budapest - 56

Overview• Introduction on Data Management (DM)

General Concepts Some details on transport protocols Data management operations Files & replicas: Name Convention

• File catalogs Cataloging requirements and catalogs in egee/LCG RLS file catalog LCG file catalog

• DM tools: overview• Data Management CLI

lcg_utils• Data Management API

lcg_utils GFAL

• Advanced concepts Advanced utilities: CLI&APIs OutputData JDL attribute

• Conclusions

GRID’05 EGEE Summer School Budapest - 57

Summary

• We provided a description to the egee/LCG Data Management Middleware Components and Tools

• We described how to use the available CLIs

• Use-case scenarios of Data Movement on Grid

• We presented the available APIs

• An example usage of lcg_util library is shown

GRID’05 EGEE Summer School Budapest - 58

Bibliography

• General egee/LCG information EGEE Homepage

http://public.eu-egee.org/ EGEE’s NA3: User Training and Induction

http://www.egee.nesc.ac.uk/ LCG Homepage

http://lcg.web.cern.ch/LCG/ LCG-2 User Guide

https://edms.cern.ch/file/454439//LCG-2-UserGuide.html GILDA

http://gilda.ct.infn.it/ GENIUS (GILDA web portal)

http://grid-tutor.ct.infn.it/

GRID’05 EGEE Summer School Budapest - 59

Bibliography

• Information on Data Management middleware LCG-2 User Guide (chapters 3rd and 6th)

https://edms.cern.ch/file/454439//LCG-2-UserGuide.html Evolution of LCG-2 Data Management. J-P Baud, James Casey.

http://indico.cern.ch/contributionDisplay.py?contribId=278&sessionId=7&confId=0

Globus 2.4http://www.globus.org/gt2.4/

GridFTPhttp://www.globus.org/datagrid/gridftp.html

GFALhttp://grid-deployment.web.cern.ch/grid-deployment/gis/GFAL/

GFALindex.html

GRID’05 EGEE Summer School Budapest - 60

Bibliography

• Information on egee/LCG tools and APIs Manpages (in UI)

• lcg_utils: lcg-* (commands), lcg_* (C functions)

Header files (in $LCG_LOCATION/include)• lcg_util.h

CVS developement (sources for commands)http://isscvs.cern.ch:8180/cgi-bin/cvsweb.cgi/?hidenonreadable=1&f=u&

logsort=date&sortby=file&hideattic=1&cvsroot=lcgware&path=

• Information on other tools and APIs EDG CLIs and APIs

http://edg-wp2.web.cern.ch/edg-wp2/replication/documentation.html Globus

http://www-unix.globus.org/api/c/ , ...globus_ftp_client/html , ...globus_ftp_control/html