INFN - FerraraINFN - Ferrara

BaBar MeetingBaBar Meeting

SPGrid: status in FerraraSPGrid: status in Ferrara

Enrica Antonioli - Paolo VeronesiFerrara, 12/02/2003

Ferrara - 12/02/03 2

Topics Topics

The DataGrid project

Ferrara Farm Configuration

First SP submissions through the Grid

Work in Progress

Future Plans

Conclusions

Ferrara - 12/02/03 3

INFN-GRID

2001- 2003

To manage and use computing resources distributed on Garr-b sites

Deployment of Testbed sites, in order to validate EDG software release and to adapt them to High Energy Physics requests

Special project of INFN

European DataGrid and INFN-GRIDEuropean DataGrid and INFN-GRID

2001 - 2003

Funded by European Union

Computing Grids permit:

• High Throughput Computing

• Analysis of large dimension data

• Sharing resources and data

Applications involved:• Biomedical Sciences

• Earth Observation

• High Energy Physics

EDG

CT

BO

PD

TO

CA

ROMA

MI

To USA

To Russia/Japan

Current prototipe of INFN DataGrid testbed connected to EDG testbed – US and ASIA

Cern

FEFE

R.A.L

Manchester

Ferrara - 12/02/03 4

EDG Architecture and ServicesEDG Architecture and Services

OS & Net services

APPLICATION Layer

ALICE ATLAS CMS LHCb BaBarBaBar

High level GRID middleware

Basics ServicesGLOBUS

toolkit

DataGRIDArchitecture

Ferrara - 12/02/03 5

CECE//WNWN

SESE

The DataGrid Testbed consists of different types of machines (Grid Elements).

In Ferrara the farm is composed by one Computing Element (CE), three Worker Nodes (WN), one User Interface (UI) and one Storage Element (SE).

All these machines are managed by a LCFGng (Local ConFiGuration system new generation) server and they are automatically configured.

UIUILCFGngLCFGngServerServer

Grid Elements in FerraraGrid Elements in Ferrara

Ferrara - 12/02/03 6

UI (User Interface): component for accessing to the

workload management system.

Users can submit a job and retrieve the output, they

sholud have an account and a personal certificate

installed in their home directory.

UIUICertificateCertificate AuthoritiesAuthorities

To access the GRID you have to request a certificate to a certification authority. INFN-GRID users can obtain a certificate from the INFN CA (http://security.fi.infn.it/).

User InterfaceUser Interface

To use the BaBar Grid, you must register that certificate with the BaBar Virtual Organisation (BaBar VO).

http://www.slac.stanford.edu/BFROOT/www/Computing/Offline/BaBarGrid/registration.html

Ferrara - 12/02/03 7

Job SubmissionJob Submission

StorageElement

UIJDL

Logging &Book-keeping(LB)

ResourceBroker (RB)

Job SubmissionService (JSS)

ComputingComputingElementElement

Information Service (IS)

ReplicaCatalogue

Job SubmitEvent

Input Sandbox

Job Status

submitted

waiting

ready

scheduled

Input Sandboxrunning

Job Status

done

outputready

Job StatusJob Status Output Sandbox

Output Sandbox

cleared

Ferrara - 12/02/03 8

SPGrid Farm in FerraraSPGrid Farm in Ferrara

LCFGngServer

Management

Lock serverLock server

Data serverData server

SCSISCSI

250 GB250 GB

IR 0A DSESE

CE-WNCE-WN

UIUI

Ferrara - EDG 1.4.3

RBRB

CNAF - BolognaCERN

Ferrara - 12/02/03 9

ConfigurationConfiguration INFN Grid Testbed Status: EDG 1.4.3 (RedHat 6.2).

A BaBar software special release (12.3.2y) has been built and installed to:

Write Kanga files Run Moose on RH 6.2

A special tag of ProdTools has been installed to perform tests.

A pool of BaBar accounts (babar000, babar001,…) has been created in the EDG farm of Ferrara.

Each member of BaBar VO is able to submit jobs to the farm of Ferrara through the RB located at CNAF (grid009g.cnaf.infn.it).

Ferrara - 12/02/03 10

Current StatusCurrent Status

Created a JDL file to run Moose on Grid resources.

Created scripts containing EDG commands to submit jobs, to check their status and retrieve output files.

An user can submit a range of runs.

For each run a job is created and submitted to the Resource Broker, then it is sent to the Ferrara CE (grid0.fe.infn.it).

The output file is then transferred to the closest SE (grid2.fe.infn.it).

Ferrara - 12/02/03 11

Moose.jdlMoose.jdl

grid1> more Moose.jdl Executable ="Moose.csh";InputSandbox ={"Moose.csh",".cshrc","config.csh"};StdOutput ="Moose.txt";StdError ="Moose.log";OutputSandbox ={"Moose.txt","Moose.log"};

General environment configurations

Config file for BaBar.Similar to SP standard

scripts

Similar to SP standard scripts (Job.Xsh)

[…]tar -czvf run${RUNNUM}.tar.gz *.rootglobus-url-copy -vb file://`pwd`/run${RUNNUM}.tar.gz \gsiftp://grid2.fe.infn.it/flatfiles/SE00/paolo/run${RUNNUM}.tar.gz

Globus command:To copy output files from WN to SE

Ferrara - 12/02/03 12

grid1> more launch #!/bin/tcsh -v

@ num_f = $1@ fin = $2

while ( $num_f <= $fin )

####build the run directories[…]####build a config.csh with the appropriate environment variablesecho "#\!/bin/tcsh -v" > config.csh[…]#### now run the jobdg-job-submit -o run$num_f.jobid -r \grid0.fe.infn.it:2119/jobmanager-pbs-long Moose.jdl

cd ..@ num_f++end

The launch scriptThe launch script

A config file is created for each

run

EDG job submission command

Range of runs to submit

For each run a job is created runtime

Ferrara - 12/02/03 13

Job SubmissionJob Submission

grid1> ls1962016 1962017 Moose.csh Moose.jdl config.csh launch monitor retrieve

grid1> ./launch 1962016 1962017[…]dg-job-submit -o run$num_f.jobid -r grid0.fe.infn.it:2119/jobmanager-pbs-long Moose.jdl

Connecting to host grid009g.cnaf.infn.it, port 7771Logging to host grid009g.cnaf.infn.it, port 15830

================== dg-job-submit Success ================ The job has been successfully submitted to the Resource Broker. Use dg-job-status command to check job current status. Your job identifier (dg_jobId) is: https://grid009g.cnaf.infn.it:7846/193.206.188.102/104224188091275?grid009g.cnaf.infn.it:7771 The dg_jobId has been saved in the following file: /home/enrica/stress/1962016/run1962016.jobid[…]

grid1> ls 1962016/Moose.csh Moose.jdl config.csh run1962016.jobid

grid1> ls 1962017/Moose.csh Moose.jdl config.csh run1962017.jobid

Range of runs to submit

Job ID

CNAF RB

Ferrara - 12/02/03 14

grid1> ./monitor 1962016 1962017Run 1962016 is Status = Scheduled Status Reason = initial Run 1962017 is Status = Scheduled Status Reason = initial

grid1> ./monitor 1962016 1962017Run 1962016 is Status = Running StatusRun 1962017 is Status = Running Status

The monitor scriptThe monitor script

grid1> ./monitor 1962016 1962017Run 1962016 is Status = Ready Status Reason = job acceptedRun 1962017 is Status = Ready Status Reason = job accepted

grid1> more monitor#!/bin/tcsh

@ num_f = $1@ fin = $2

while ( $num_f <= $fin )echo Run $num_f is `dg-job-status -i \ $num_f/run$num_f.jobid | grep Status`

@ num_f++end

EDG command

grid1> ./monitor 1962016 1962017Run 1962016 is Status = OutputReady Status Reason = terminated Run 1962017 is Status = OutputReady Status Reason = terminated

Ferrara - 12/02/03 15

The retrieve scriptThe retrieve scriptgrid1> more retrieve #!/bin/tcsh -v@ num_f = $1@ fin = $2while ( $num_f <= $fin )cd $num_f#### get logfilesdg-job-get-output -i run$num_f.jobid --dir $PWD#### get rootfilesglobus-url-copy \ gsiftp://grid2.fe.infn.it/flatfiles/SE00/paolo/run$num_f.tar.gz \ file://`pwd`/run$num_f.tar.gz

tar -xzvf run$num_f.tar.gzrm -f run$num_f.tar.gz#### delete rootfiles form SEglobus-job-run grid2.fe.infn.it /bin/rm \/flatfiles/SE00/paolo/run$num_f.tar.gzcd ..@ num_f++end

EDG command

Globus command:Direct copy of file from SE to UI

Globus command:delete file fromSE

Ferrara - 12/02/03 16

Retrieving OutputRetrieving Output

grid1> ls1962016 1962017 Moose.csh Moose.jdl config.csh launch monitor retrieve

grid1> ls 1962016/150546318633191 Moose.jdl rootdef-tru.root Moose.csh config.csh run1962016.jobidrootdef-tag.root rootdef-aod.root

grid1> ls 1962017/150551318931039 Moose.jdlMoose.csh config.csh run1962017.jobidrootdef-tru.root rootdef-tag.root rootdef-aod.root

grid1> ls 1962016/150546318633191/Moose.log Moose.txt

grid1> ls 1962017/150551318931039/Moose.log Moose.txt

Ferrara - 12/02/03 17

Future PlansFuture Plans

Data serverData server

Lock serverLock server

Objectivity DBObjectivity DBRBRB(UK)(UK)

Ferrara

MOOSEMOOSERPM

RPM

3) Install Objy DB on the SE

2) MOOSE in RPM format

1) Use of IC RB and others

Integration of MooseIntegration of Moose Application withApplication with

EDG software releasesEDG software releases

SPGrid Farm

Management

LCFGngServer

SESE

CE-WNCE-WN

UIUI

Ferrara - 12/02/03 18

DocumentationDocumentation

The DataGrid Project: http://eu-datagrid.web.cern.ch/eu-datagrid/default.htm

EDG tutorials Archive Web Site: http://hep-proj-grid-tutorials.web.cern.ch/hep-proj-grid-tutorials/loginex.html

INFN-Grid Testbed: http://server11.infn.it/testbed-grid/

BaBar-Grid: http://www.slac.stanford.edu/BFROOT/www/Computing/Offline/BaBarGrid/

Status of the Farm in Ferrara: http://print.fe.infn.it/status/