A. Sim, CRD, L B N L 1OSG Applications Workshop 6/1/2005

OSG SRM/DRM Readiness and PlanOSG SRM/DRM Readiness and Plan

Alex Sim / Jorge Rodriguez Alex Sim / Jorge Rodriguez

Scientific Data Management GroupScientific Data Management Group

Computational Research DivisionComputational Research Division

Lawrence Berkeley National LaboratoryLawrence Berkeley National Laboratory

andand

High Energy Physics GroupHigh Energy Physics Group

Dept. of PhysicsDept. of Physics

University of FloridaUniversity of Florida

A. Sim, CRD, L B N L 2OSG Applications Workshop 6/1/2005

Service DescriptionService Description

• Brief overview of the DRM ServiceBrief overview of the DRM Service• core service

• Provides core functionality • Movement of files into and out of DRM cache

• Web Service components by Java wrapper • Based on SRM v.1.1 specification• interoperable with other SRMs

• DRM dependencies DRM dependencies • Functions on top of ordinary UNIX file systems: local, NFS…• Binaries available for RH7.x, RH8 and RHEL3

• Statically linked and self contained• Requires Java 1.4.x+ for web services components

• Self contained except Java VM• Data movement services such as Gridftp

A. Sim, CRD, L B N L 3OSG Applications Workshop 6/1/2005

InstallationInstallation

• Installation:Installation:• DRM is included in VDT 1.3.x

• DRM version 1.2.7 in the current VDT• Startup script, SXXsrm to start/stop/check processes

• Can be deployed with everything else from VDT…• Can also be deployed as a stand-alone

• VDT will handle dependencies

A. Sim, CRD, L B N L 4OSG Applications Workshop 6/1/2005

ConfigurationConfiguration

• Configuration: Configuration: • Recommend to install and run under a non-privileged user

• e.g. srm• Service provider recommends this for security reasons

• Should use a separate (SRM) service certificate• Default configuration is provided

• Java based configuration generator is provided for customization• Manually modified configuration parameters will also work

A. Sim, CRD, L B N L 5OSG Applications Workshop 6/1/2005

Installation TestingInstallation Testing

• Testing:Testing:• srm-tester is provided

• Testing DRM installation and its proper running• Testing compatibility of any SRMs based on SRM spec v.1.1

• This tool can be used for SRM/dCache or any other SRM installations

• Testing interoperability of two SRMs based on SRM spec v.1.1• E.g. It checks the interoperability of SRM/DRM and SRM/dCache.

A. Sim, CRD, L B N L 6OSG Applications Workshop 6/1/2005

Client tools for SRM/DRMClient tools for SRM/DRM

• Java based srmcp from FNAL based on SRM v.1.1 can be used Java based srmcp from FNAL based on SRM v.1.1 can be used • For general SRM v.1.1 based file movement requests: get, put, copy

• DRM specific client commands can also be usedDRM specific client commands can also be used• Works only for DRMs• drm-copy, drm-get, drm-put, drm-status,

drm-ping, drm-ls, drm-release, drm-remove

• Java based srm-copy can also be usedJava based srm-copy can also be used• Included in coming DRM v.1.2.8• For general SRM v.1.1 and SRM v.2.1.1 access

• Works for any versions of SRMs : based on v.1.1 or v.2.1.1

• File movement functionalities • get, put, copy ( for servers based on v.1.1 and v.2.1.1)

• directory management capabilities (for SRM servers based on v.2.1.1)• ls, mkdir

• explicit space reservations (for SRM servers based on v.2.1.1)• reserveSpace

A. Sim, CRD, L B N L 7OSG Applications Workshop 6/1/2005

Disk

Cache

HRM

Disk

Cache

HRM

Files request(thousands of files)

Disk

Cache

Disk

Cache

Disk

Cache

DRM

Disk

Cache

Disk

CacheMSS

DRM

Disk

Cache

Disk

Cache

HRM(performs writes)

HRM(performs writes)

HRM(performs writes)

GridFTPHTTP(s)FTPBBFTPservers

Client

DiskCache

SRMdCache

Disk

Cache

Disk

Cache

Disk

Cache

Disk

Cache

Disk

Cache

Disk

CacheDiskCacheC

OR

BA

Nam

e services

Web services gatewaySRMCP

DRM-COPYDRM-GETDRM-PUTDRM-LSDRM-REMOVEDRM-RELEASEDRM-PINGDRM-STATUS

OtherSRMs

Disk

Cache

Disk

Cache

Disk

Cache

Disk

CacheStorage

Overview of Overview of the service architecturethe service architecture

A. Sim, CRD, L B N L 8OSG Applications Workshop 6/1/2005

Sample Application Level Test Sample Application Level Test Scenario (1)Scenario (1)

• Simplified application test :Simplified application test :1. stage files into DRM cache from remote

SRM/DRM, SRM/dCache or any file repositories (with supported transport protocols – gsiftp, ftp, http, https or bbftp)

2. Execute jobs that access “staged in” files in DRM cache via POSIX calls

A. Sim, CRD, L B N L 9OSG Applications Workshop 6/1/2005

Scenario (1)Scenario (1)

DRM

Disk

Cache

DISK CACHE

Disk

Cache

HRMDisk

Cache

DRMDisk

Cache

Disk

Cache

SRM/dCache

Disk

Cache

Disk

Cache

Disk

Cache

Disk

Cache

Disk

Cache

OtherSRMs Disk

Cache

Disk

Cache

Disk

Cache

Disk

CacheStorage

HRM(performs writes)

HRM(performs writes)

HRM(performs writes)

GridFTPHTTP(s)FTPBBFTPservices

DiskCache

Client1. File Request

3. Job execution with file accesswhen ready

2. Bring files in

A. Sim, CRD, L B N L 10OSG Applications Workshop 6/1/2005

Sample Application Level Test Sample Application Level Test Scenario (2)Scenario (2)

• Simplified application test :Simplified application test :1. stage files into DRM cache from remote

SRM/DRM, SRM/dCache or any file repositories (with supported transport protocols – gsiftp, ftp, http, https or bbftp)

2. Execute jobs that access “staged in” files in DRM cache via POSIX calls

3. Job creates an output file on local storage ($WNTMP or $TMP)

4. Job moves the output file from local storage into the local DRM cache

5. Job moves the output file from DRM cache to remote SRM managed or remote archival location

A. Sim, CRD, L B N L 11OSG Applications Workshop 6/1/2005

Scenario (2)Scenario (2)

DRM

Disk

Cache

DISK CACHE

Disk

Cache

HRMDisk

Cache

DRMDisk

Cache

Disk

Cache

SRM/dCache

Disk

Cache

Disk

Cache

Disk

Cache

Disk

Cache

Disk

Cache

OtherSRMs Disk

Cache

Disk

Cache

Disk

Cache

Disk

CacheStorage

HRM(performs writes)

HRM(performs writes)

HRM(performs writes)

GridFTPHTTP(s)FTPBBFTPservices

DiskCache

Client

1. File Request

3. Job executionwith file accesswhen ready

4. Space Request

5. Puts the outputfile into the desginatedDRM space when ready

2. Bring files in

6. Send files out

A. Sim, CRD, L B N L 12OSG Applications Workshop 6/1/2005

Other Application Level TestsOther Application Level Tests

• Scientific Applications (willing or able to Scientific Applications (willing or able to participate)participate)

• Provide document recommendations for use of DRM services in job work flow

• Make DRM enabled sites available for application tests

• Schedule with application administrators • Monitor activity by site and app admins.

A. Sim, CRD, L B N L 13OSG Applications Workshop 6/1/2005

Service MonitoringService Monitoring

• Monitor performance of storage activityMonitor performance of storage activity• DRM has information of current activities in its

cache as well as logs of all past activities• Will need input from Service Provider as to what to

monitor, what to measure, and what to report

A. Sim, CRD, L B N L 14OSG Applications Workshop 6/1/2005

Documents and SupportDocuments and Support

• Document linksDocument links• http://sdm.lbl.gov/srm-dist

• Support of the serviceSupport of the service• We (LBNL SDM group) will support all SRM/DRM

related inquiries• All support requests to srm@lbl.gov

A. Sim, CRD, L B N L 15OSG Applications Workshop 6/1/2005

PlansPlans

• Where we are todayWhere we are today• DRM v.1.2.7 in current VDT• Documents for DRM v.1.2.7 on the web• DRM v.1.2.8 to release in a week or two

• Next steps Next steps • Documents to include use cases• Having more participations from OSG sites for

installation and testing• Testing other use cases if provided• Enhancements if any• Bug fixing if found