lcg-2 manual installation g - uomlcg-2 manual installation guide document identiﬁer:...

LHC COMPUTING GRID

LCG-2 MANUAL INSTALLATION GUIDE

Document identifier: CERN-LCG-GDEIS-434070

EDMS id: 434070

Version: v1.8

Date: June 3, 2004

Section: LCG Experiment Integration and Support

Document status: DRAFT

Author(s): Flavia Donno, Simone Campana, PatriciaMendez Lorenzo, Roberto Santinelli, AndreaSciaba’

File: LCG2Install

Abstract: This document describes the manual steps necessary to set-up a testbed using the LCG-2distribution of the middleware.

CONTENTS

1. INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

1.1. OBJECTIVES OF THIS DOCUMENT. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.2. REFERENCE DOCUMENTS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31.3. TERMINOLOGY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.4. DEFINITIONS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41.5. ASSUMPTIONS. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

2. RPMS DOWNLOAD . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

3. SOFTWARE INSTALLATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

3.1. GENERAL INFORMATION FOR SECURITY/GRID-MAPFILE. . . . . . . . . . . . . . 6

4. GENERAL REQUESTS FOR THE LCG-2 TESTBED INSTALLATION . . . 8

5. USER INTERFACE (UI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

6. COMPUTING ELEMENT (CE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

7. WORKER NODE (WN) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

8. STORAGE ELEMENT (SE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

9. RESOURCE BROKER (RB) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

10. TOP GRID INFORMATION INDEX SERVER (GIIS OR MDS) . . . . . . . . . . . 53

11. BERKLEY DB INFORMATION INDEX (DBII) . . . . . . . . . . . . . . . . . . . . . . . . . . . 55

12. RLS INSTALLATION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57

13. ACKNOLEDGEMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

CERN-LCG-GDEIS-434070 LCG-2 Manual Installation Guide Page2

1. INTRODUCTION

This document describes how to install and configure a LCG-2 testbed consisting on the following ele-ments:

• A User Interface (UI)

• A Resourse Broker (RB)

• A Computing Element (CE)

• A Worker Node (WN)

• A Storage Element (SE)

• A Info System (GIIS and BDII)This testbed does not included Proxy Server and Replica Location Server (RLS) systems and thereforetheir installations have not been included in this guide. All RPMs needed for the installation can be foundon the web.All instructions have been tested on RedHat7.3 CERN certified distribution. For a general description ofthe components listed above and their role, please refer to [R6].

1.1. OBJECTIVES OF THIS DOCUMENT

The goal of this document is to give instructions to site administrators to setup manually an LCG-2 site.A site can be a core site, providing main grid services, or just a support site providing computing andstorage resources. A core site is one that runs any of the following services: RB, TOP GIIS, BDII, RLS.A support site instead runs services such as CE, WN and SE. Therefore, in order to setup your site,please read the General Notes and then follow the sections of your interest with regards to the servicesyou intend to setup.

1.2. REFERENCE DOCUMENTS

[R1 ] LCG-1 Tutor Manual Installation Guide,http://grid-deployment.web.cern.ch/grid-deployment/cgi-bin/index.cgi?var=eis/tutorial

[R2 ] UI LCG-1 Manual Software Installation and Configuration,http://lcgapp.cern.ch/cgi-bin/viewcvs/viewcvs.cgi/lcg1/manualInstConf/UI instConf.txt?cvsroot=lcgdeploy

[R3 ] WN LCG-1 Manual Software Installation and Configuration,http://lcgapp.cern.ch/cgi-bin/viewcvs/viewcvs.cgi/lcg1/manualInstConf/WN instConf.txt?cvsroot=lcgdeploy

[R4 ] DBII Configuration files,http://lcgapp.cern.ch/cgi-bin/viewcvs/viewcvs.cgi/lcg1/BDII/?cvsroot=lcgdeploy

[R5 ] LCFGng-lite documentation,http://www.cern.ch/grid-deployment/documentation/LCFGng-lite

[R6 ] LCG-1 User Guide, (LCG-2 User Guide in preparation)http://grid-deployment.web.cern.ch/grid-deployment/cgi-bin/index.cgi?var=eis/docs


1.3. TERMINOLOGY

BDII BerkleyDB Information IndexCE Computing ElementGIIS Grid Information Index ServerGRIS Grid Resource Information ServerLRC Local Replica CatalogMDS Monitoring and Discovery ServiceRB Resource BrokerRLS Replica Location ServerRMC Replica Metadata CatalogSE Storage ElementUI User InterfaceVO Virtual OrganizationWN Worker Node

1.4. DEFINITIONS

Core Services They are BDII, TOP GIIS, RLS, RBCore Site A site which runs core servicesSite GIIS A GIIS that publishes the status of resources available at a siteSupport Services They are CE, WN, SE, UISupport Site A site which runs only support servicesTOP GIIS A GIIS that publishes the status of all resources part of the dissemination testbed

1.5. ASSUMPTIONS

• On UI, WN and CE install the Development Tools of RedHad 7.3 before proceesing further.

• For the rest of machines install RedHad 7.3 CERN Workstation, choose a medium level of securityallowing SSH, turn off SUE and be sure of the proper installation of HEPIX environment. Indeed,you will have to install by hand the subdirectory into /usr/local/lib/hepix/shells together with itscontent, copy it from lxplus machines at CERN.


2. RPMS DOWNLOAD

From the LCG-deployment home page it is possible to find all the RPMs needed to install all the machinetypes included in the following testbed:

http://grid-deployment.web.cern.ch/grid-deployment/cgi-bin/index.cgi?var=homepage

From the“Download” link on the left, use the downloadlcg20040204 1500-edgt20031027 1450 ver-sion of LCG-2. There you have all the machine types needed in this installation. Click on UI-rpm andyou will have all the RPMs needed to install a UI. It may be better if you used the general command:

wget -r -nd -l 1 http://grid-deployment.web.cern.ch/grid-deployment/download/UI-rpm-.html

To download all the RPMs at the same time.


3. SOFTWARE INSTALLATION

As recommendation we suggest you to follow these steps to install your RPMs in an arranged way:

• FOR ALL MACHINES INSTALLATION YOU MUST BE LOGGED IN AS ROOT

• Create a temporary repository into the machine:

mkdir /opt/RPMScd /opt/RPMS

• Download the RPMs needed for the installation inside this repository (use the wget command andthe syntax that you will find on the above web page)

• Create a sub-directory for packages To Be Installed (TBI), for packages Not To Be Install (NTBI)and for packages Already Installed into the machine (AI).

mkdir TBImkdir NTBImkdir AI

• Move from the repository to the NTBI directory all the packages you do not need to install (LCFG-related ones)

mv *lcfg-* NTBI

• Move the rest of .rpm files to the TBI directory:

mv *.rpm TBI

• Go into TBI and install all the RPMs as follows:

cd TBIrpm -ivh *.rpm

• In the case you need to resolve conflicts coming from the actuall installation on the machine, takenote of the RPMs producing the conflicts (normally it means that they are already installed on themachine), and move them to the AI subdirectory. Then repeat the RPM installation.

3.1. GENERAL INFORMATION FOR SECURITY/GRID-MAPFILE

On each machine (except UI and WN) copy the fileshostcert.pem andhostkey.pem into the directory:

/etc/grid-security/


and execute:

chmod 400 /etc/grid-security/hostkey.pemchmod 444 /etc/grid-security/hostcert.pem


4. GENERAL REQUESTS FOR THE LCG-2 TESTBED INSTALLATION

Once you have installed the RPMs needed for each machine, some general steps must be executed on allmachines before proceeding further.

• On all the machines, the file/etc/ld.so.conf should contain the following lines:

/opt/gcc-3.2.2/lib/opt/globus/lib/opt/edg/lib/opt/lcg/lib/usr/local/lib(edit the file and just add them by hand)

- Now run the command

/sbin/ldconfig -v

• A general requirement for the LCG-2 nodes is that they should be synchronized. This requirementmay be fulfilled in several ways. If your nodes run under afs, most likely they will be alreadysynchronized. Otherwise, you can use the NTP protocol with a time server.In order to do that you must,

– Install NTP:The list of the rpms for NTP for our basic OS configuration, with the corresponding links fordownload is:

ntp-4.1.1-1: http://grid-deployment.web.cern.ch/grid-deployment/download/RpmDir/release/ntp-4.1.1-1.i386.rpm

libcap-devel-1.10-8:http://grid-deployment.web.cern.ch/grid-deployment/download/RpmDir/release/libcap-devel-1.10-8.i386.rpm

libcap-1.10-8: http://grid-deployment.web.cern.ch/grid-deployment/download/RpmDir/release/libcap-1.10-8.i386.rpm

– Configure the file/etc/ntp.conf by adding the lines dealing with your time server con-figuration such as, for instance:

restrict mask 255.255.255.255 nomodify notrapnoquery server

Additional time servers can be added for better performance esults. For each server, thehostname and IP address are required. Then, for each time-server you are using, add a coupleof lines similar to the ones shown above into the/etc/ntp.conf file.

– Edit the file/etc/ntp/step-tickers adding a list of your time server(s) hostname(s),as in the following example:


ip-time-1.cern.ch137.138.17.69 )

– Activate the ntpd server with the following commands:

service ntpd stopntpdate service ntpd onchkconfig ntpd start

• On all the machines install the files/etc/sysconfig/globus and/etc/sysconfig/edgas specified further per machine type.


5. USER INTERFACE (UI)

1. Create the file/etc/globus.conf as follows:

[common]GLOBUS LOCATION=/opt/globusglobus flavor name=gcc32dbg[mds][gridftp][gatekeeper]default jobmanager=forkjob manager path=$GLOBUS LOCATION/libexecjobmanagers="fork "[gatekeeper/fork]type=forkjob manager=globus-job-manager

2. Create the directory/tmp/jobOutput and make it world-writeable:

mkdir /tmp/jobOutputchmod 777 /tmp/jobOutput

3. Create the file/etc/sysconfig/edg as follows:

# Root directory for EDG software# usual value: /opt/edgEDG LOCATION=/opt/edg# Directory for machine-specific files.# usual value :$EDG LOCATION/varEDG LOCATION VAR=/opt/edg/var# World writable directory for temporary files.# usual value: /tmpEDG TMP=/tmp

4. Create the file/etc/sysconfig/globus as follows:

GLOBUS LOCATION=/opt/globusGLOBUS CONFIG=/etc/globus.conf

5. Launch Globus Initialization Script:

/opt/globus/sbin/globus-initialization.sh

This Globus script is meant to be launched on the whole series of LCG-2 nodes, as a generalconfiguration step. It may therefore perform some configuration actions which are not specificallyrelated to a UI. Because of this it can give warning as well as some error messages.


6. Create the following two directories:

/opt/edg/var/etc/opt/edg/var/etc/profile.d

7. Configure the Replica Manager.

• Edit the file:/opt/edg/etc/edg-replica-manager/edg-replica-manager.conf.valuesas follows, filling with the right values for LOCALDOMAIN, DEFAULT.SE (the SE at yoursite), DEFAULT.CE (the CE at your site), INFOSERVICE (it should be equal to MDS),MDS.HOST (to point to the tutor TOP GIIS). The general syntax for this file is:

-------------------------@LOCALDOMAIN@||the local [email protected]@||the host of the close [email protected]@||the host of the close CE@INFOSERVICE@||The info provider to use. It can beStub, MDS or RGMA@STUBFILE@|/opt/edg/etc/edg-replica-manager/info-service-stub.properties|The properties file for the static file ’info service’. To be specifiedonly if INFOSERVER is equal to stub. @MDS.HOST@||Thehost of the [email protected]@|2170|The port of the [email protected]@|true|Fail if no ROS is available. Must be set to false.-------------------------

WARNING: This file is created by the edg-replica-manager-config-1.6.2-1 rpm and thuswhen created manually should be given a different name so that possible upgrades of therpm do not overwrite the already existing file.An example of this file configured in this testbed:

@EDG.LOCATION@|/opt/edg|location of edg the directory@LOCALDOMAIN@|cern.ch|the local [email protected]@|tbed0101.cern.ch|the host of the close [email protected]@|pceis01.cern.ch|the host of the close CE@INFOSERVICE@|MDS|The info provider to use. It can be Stub, MDS [email protected]@|LrcOnly|The mode the RLS should be run in. LrcOnly or WithRli@STUBFILE@||The properties file for the static file ’info service’@MDS.HOST@|tbed0102.cern.ch|The host of the [email protected]@|2170|The port of the [email protected]@|false|Fail if no ROS is [email protected]@| gcc3 2 2|The gcc suffix as used on the build box (emptyfor 2.95, gcc3 2 2 for 3.2.)

• Run the following command:


/opt/edg/sbin/edg-replica-manager-configure/opt/edg/etc/edg-replica-manager/edg-replica-manager.conf.values

The successful running of the script can be verified looking at the content of the file:

/opt/edg/var/etc/edg-replica-manager-configure/edg-replica-manager.conf

Which now should result to be configured accordingly.

8. Configure the Workload Management system (WMS) as follows:

• Create a configuration directory for each VO you want to support at your site. Possible VOsare alice, atlas, cms, lhcb, dteam:

mkdir /opt/edg/etc/alicemkdir /opt/edg/etc/atlasmkdir /opt/edg/etc/cmsmkdir /opt/edg/etc/lhcbmkdir /opt/edg/etc/dteam

• For each VO, the file/opt/edg/etc/ /edg wl ui.conf needs to be editedand configured as follows:

--------------------------------------------[VirtualOrganisation = "";NSAddresses = ":7772";LBAddresses = ":9000";## HLR location is optional. Uncomment and fill correctly for## enabling accounting# HLRLocation = "fake HLR Location"## MyProxyServer is optional. Uncomment and fill correctly for## enabling proxy renewal. This field should be set equal to## MYPROXY SERVER environment variable## MyProxyServer = ""]--------------------------------------------

An example of this file in our own configuration:

[VirtualOrganisation="alice";NSAddresses="pceis02.cern.ch:7772";LBAddresses="pceis02.cern.ch:9000";MyProxyServer="adc0032.cern.ch";]

• Create the files/opt/edg/etc/edg wl ui cmd var.conf as follows:

WARNING: in the directory/opt/edg/etc/ there is a template fileedg wl ui cmd var.conf.templategenerated during the rpm installation. Please IGNORE it and edit the file as shown below:


--------------------------------------------[rank = - other.GlueCEStateEstimatedResponseTime;requirements = other.GlueCEStateStatus == "Production";RetryCount = 3;ErrorStorage = "/tmp";OutputStorage = "/tmp/jobOutput";ListenerPort = 44000;ListenerStorage = "/tmp";LoggingTimeout = 30;LoggingSyncTimeout = 30;LoggingDestination = ":9002";# Default NS logger level is set to 0 (null)# max value is 6 (very ugly)NSLoggerLevel = 0;DefaultLogInfoLevel = 0;DefaultStatusLevel = 0;DefaultVo = "";]--------------------------------------------

Fill the required placeholders inserting the RB and Default VO. Also the job output and errordefault directories can be configured by this file. An example of a realedg wl ui cmd var.conffile is reported below:

[rank = - other.GlueCEStateEstimatedResponseTime;requirements = other.GlueCEStateStatus == "Production";RetryCount = 3;ErrorStorage = "/tmp";OutputStorage = "/tmp/jobOutput";ListenerPort = 44000;ListenerStorage = "/tmp";LoggingTimeout = 30;LoggingSyncTimeout = 30;LoggingDestination = "pceis02.cern.ch:9002";NSLoggerLevel = 0;DefaultLogInfoLevel = 0;DefaultStatusLevel = 0;DefaultVo = "dteam";]

9. Copy the following files from the directory/opt/edg/etc/profile.d to the directory/opt/edg/var/etc/profile.d:

edg-wl-ui-env.csh


edg-wl-ui-env.shedg-wl-ui-gui-env.cshedg-wl-ui-gui-env.sh

10. Cron Table Configuration

• Add the services to the crontab

crontab -e

This will allow the editing of the crontable with the default editor. The following lines haveto be added:

PATH=/sbin:/bin:/usr/sbin:/usr/bin56 3,9,15,21 * * * /opt/edg/etc/cron/edg-fetch-crl-cron >>/var/log/edg-fetch-crl-cron.log 2>&1

If needed, an automatic rotation of the log file can be obtained editing the file/opt/edg/etc/cron/edg-fetch.An example of configuration of the/etc/logrotate.d/edg-fetch for a monthly log rota-tion follows:

-----------------------------------/var/log/edg-fetch-crl-cron.log{compressmonthlyrotate 12missingokifemptycreate}-----------------------------------


6. COMPUTING ELEMENT (CE)

A Computing Element is the Grid Gateway machine which acts as a front-end between the Grid and yourlocal farm of WNs. If you decide to run only one CE then the machine runs a local GRIS and the siteGIIS. You do not need to run a site GIIS if you want to register with some other site GIIS. We stronglyrecommend that you run a site GIIS. In what follows we give instructions on how to configure one CEthat runs a GRIS and a site GIIS. For special configuration, please contact the LCG Experiment andSupport Team, sending e-mail [email protected].

1. On the CE machine, copy host certificate and key to:/etc/grid-security/hostcert.pem/etc/grid-security/hostkey.pem

2. Create different UNIX groups corresponding to the supported VOs (e.g. “cms”, “atlas”, “alice”,“lhcb”, “dteam”) will run on your site. Then, for each group create the special accounts VO-name “sgm” (namely,alicesgm), which are the “Experiment Software Manager” accounts. For thecreation of the grid-mapfile, we need to describe how to configureedg-mkgridmap.


# Root directory for EDG software# usual value: /opt/edgEDG LOCATION=/opt/edg# Directory for machine-specific files.# usual value :$EDG LOCATION/varEDG LOCATION VAR=/opt/edg/var# World writable directory for temporary files.# usual value: /tmpEDG TMP=/tmp# Host certificateX509 USER CERT=/etc/grid-security/hostcert.pem# Host keyX509 USER KEY=/etc/grid-security/hostkey.pem# grid mapfileGRIDMAP=/etc/grid-security/grid-mapfileGRIDMAPDIR=/etc/grid-security/gridmapdir/


GLOBUS LOCATION=/opt/globusGLOBUS CONFIG=/etc/globus.confGLOBUS TCP PORT RANGE=’’20000 25000’’


5. In the following we only support the configuration where home directories are shared between CEand WN. No poolaccount are still supported. For more informations about poolaccount please referto [R1]. On the CE machine, you should use NFS to export the home directories and mount themon the WNs. Same user and group ids should be used between CE and WNs. In what follows, weassume that user’s home directories can be found under the directory/home. To export the/homedirectory to the WNs you need to add the following line to the file/etc/exportfs as follows (inthe example the WNs are indicated as pceis*.cern.ch):

/etc/exportfs:--------------/home pceis*.cern.ch(rw,no root squash)

Then execute the commands:

/sbin/chkconfig nfs on (to allow nfs to be activated at boot time)/etc/rc.d/init.d.nfs start (to start the nfs service)/sbin/chkconfig nfslock on/etc/rc.d/init.d/nfslock startexportfs -rexportfs -a

6. You can configure the ipchain service to allow for GRID connections editing the file/etc/sysconfig/ipchainsas in the following example:# Firewall configuration written by lokkit# Manual customization of this file is not recommended.# Note: ifup-post will punch the current nameservers through the# firewall; such entries will *not* be listed here.:input ACCEPT:forward ACCEPT:output ACCEPT-A input -s 0/0 -d 0/0 21 -p tcp -y -j ACCEPT-A input -s 0/0 -d 0/0 22 -p tcp -y -j ACCEPT-A input -s 0/0 -d 0/0 -i lo -j ACCEPT-A input -s pceis01.cern.ch -d 0/0 -p tcp -y -j ACCEPT-A input -s pceis01.cern.ch -d 0/0 -p udp -j ACCEPT-A input -s tbed0101.cern.ch -d 0/0 -p tcp -y -j ACCEPT-A input -s tbed0101.cern.ch -d 0/0 -p udp -j ACCEPT-A input -s 137.138.0.0/16 -d 0/0 -p tcp -y -j ACCEPT-A input -s 137.138.0.0/16 -d 0/0 -p udp -j ACCEPT-A input -p tcp -s 137.138.0.0/16 -d 0/0 sunrpc -y -j ACCEPT-A input -p udp -s 137.138.0.0/16 -d 0/0 sunrpc -j ACCEPT-A input -p tcp -s 137.138.0.0/16 -d 0/0 nfs -y -j ACCEPT-A input -p udp -s 137.138.0.0/16 -d 0/0 nfs -j ACCEPT


-A input -p tcp -s 0/0 -d 0/0 0:1023 -y -j REJECT-A input -p tcp -s 0/0 -d 0/0 2049 -y -j REJECT-A input -p udp -s 0/0 -d 0/0 0:1023 -j REJECT-A input -p udp -s 0/0 -d 0/0 2049 -j REJECT-A input -p tcp -s 0/0 -d 0/0 6000:6009 -y -j REJECT-A input -p tcp -s 0/0 -d 0/0 7100 -y -j REJECT

After your changes are made, you just have to reactivate the service:/etc/init.d/ipchains restartand then re-start all the portmap daemon:/etc/init.d/portmap restart

7. Check the permissions on the directory/etc/grid-security/gridmapdir and if not yet granted,allows the owner and the group owner to write-read and execute and other to execute only.

8. If you are planning to use more than one WNs, you need to authorize the users to access directlythe WNs via the ssh key. To do so, create the scriptauth.sh as in the following example:#!/bin/shmkdir /home/$1/.ssh/usr/bin/ssh-keygen -t rsa -q -N "" -C $1 -f /home/$1/.ssh/id rsa/usr/bin/ssh-keygen -t rsa1 -q -N "" -C $1 -f /home/$1/.ssh/identitycat /home/$1/.ssh/identity.pub > /home/$1/.ssh/authorized keyscat /home/$1/.ssh/id rsa.pub >> /home/$1/.ssh/authorized keys

and, on the CE as root, for each user, run the command

./auth.shuser name

9. Now you need to configure PBS. Here are the instructions:

9.a) In case the PBS daemons are running, stop them:

/ets/rc.d/init.d/pbs server stop

/ets/rc.d/init.d/pbs sched stop

/ets/rc.d/init.d/pbs mom stop

9.b) Create the file :

/var/spool/pbs/server namewhich should contain one line with the full name of the local host


9.c) Add batch nodes (WNs) to the file/var/spool/pbs/server priv/nodes with a syntax likethe following:pceis01.cern.ch np=1 eis.Here with “eis” we mean the name of the local farm while with “np” the maximum numberof job you can submit on such node. (=the number of CPU available on the machine)

9.d) Add the following line to/etc/services:pbs 15001/tcppbs sched 15004/tcppbs mom 15002/tcppbs remom 15003/tcppbs remom 15003/udp

9.e) Be sure that the pbsserver has been stopped (/sbin/service pbs server stop and/sbin/servicepbs sched stop) then create the new PBS server configuration:

/usr/sbin/pbs server -t create

9.f) Configure the pbsserver using a pbsserver.conf file.The file to be create is/var/spool/pbs/pbs server.conf and it should be as the in thefollowing example:# Create queues and set their attributes.### Create and define queue medium#create queue mediumset queue medium queue type = Executionset queue medium resources max.cput = 00:45:00set queue medium resources max.walltime = 03:00:00set queue medium enabled = Trueset queue medium started = True## Create and define queue short#create queue shortset queue short queue type = Executionset queue short resources max.cput = 00:05:00set queue short resources max.walltime = 00:20:00set queue short enabled = Trueset queue short started = True## Create and define queue infinite#create queue infinite


set queue infinite queue type = Executionset queue infinite resources max.cput = 48:00:00set queue infinite resources max.walltime = 192:00:00set queue infinite enabled = Trueset queue infinite started = True## Create and define queue long#create queue longset queue long queue type = Executionset queue long resources max.cput = 03:00:00set queue long resources max.walltime = 12:00:00set queue long enabled = Trueset queue long started = True## Set server attributes.#set server scheduling = Trueset server acl host enable = Falseset server managers = root@set server operators = root@set server default queue = mediumset server log events = 511set server mail from = admset server query other jobs = Trueset server scheduler iteration = 600set server default node = set server node pack = FalseConfigure the pbsserver as follows:/usr/bin.qmgr < /var/spool/pbs/pbs server.confwhere is the full hostname of your CE machine.

9.g) Start the PBS server daemons:/sbin/service pbs server start/sbin/service pbs sched start

9.h) Create the file/opt/edg/etc/edg-pbs-knowhosts.conf following the example:NODES = pceis01.cern.chPBSBIN = /usr/binKEYTYPES = rsa1,rsa,dsaKNOWNHOSTS = /etc/ssh/ssh known hostsThen run the command:/opt/edg/sbin/edg-pbs-knownhostRemember to change the edg-pbs-knownhost.conf file every time you add a new WN. Thecommand has to run every 6 hours via a cron job as root. The following line should be added


to the root cron file using the command crontab -e as root:PATH=/sbin:/bin:/usr/sbin:/usr/bin24 1,7,13,19 * * * /bin/sleep ‘expr $RANDOM % 1800‘ &&/opt/edg/etc/cron/edg-fetch-crl-cron24 2,8,14,20 * * * /sbin/service edg-wl-locallogger proxy03 1,7,13,19 * * * /opt/edg/sbin/edg-pbs-knownhosts

9.i) Add the file/var/spool/pbs/mom priv/config as is in the example:$clienthost localhost$clienthost $restricted $logevent 255$ideal load 1.6$max load 2.1$usecp *.cern.ch:/home /home

9.l) Add the file/opt/edg/libexec/ceinfo-wrapper.sh as shown in the following lines:#!/bin/sh/opt/edg/libexec/edg-ce-all-lrms pbs-cluster pceis01.cern.ch-queue medium short long infinite-globus-config-file /opt/globus/etc/globus-gatekeeper.conf-static /opt/edg/var/etc/ce-static.ldif-auth-users-from-grid-mapfile /dev/null-globus-gatekeeperport 2119-globus-gatekeeperhost -globus-jobmanager pbs (this value should be also lcgpbs. Once set a valueyou have to adopte always this one in both ce-static.ldif and infoprovider.conffile. ‘‘lcgpbs’’ will identify all the sites without a shared filesystem)-grisport 2135-remotefiles /opt/edg/var/info/edg-scl-desc.txt-ttl 120-cesebind ’.’

9.m) Restart the PBS service as follows:/ets/rc.d/init.d/pbs server restart

/ets/rc.d/init.d/pbs sched restart

/ets/rc.d/init.d/pbs mom restart

Hints: execute the previous commands twice!!!


10. Configure the GRIS as follows:

10.a) Create the special account edginfobelonging to the groupedginfo on the CE.

10.b) Create the file/etc/globus.conf as shown in the following lines and change all the fieldincluded in:[common]GLOBUS LOCATION=/opt/globusglobus flavor name=gcc32dbgx509 user cert=/etc/grid-security/hostcert.pemx509 user key=/etc/grid-security/hostkey.pemgridmap=/etc/grid-security/grid-mapfilegridmapdir=/etc/grid-security/gridmapdir/

[mds]globus flavor name=gcc32dbgpthruser=edginfo[mds/gris/provider/edg][mds/gris/registration/site]regname=reghn=[mds/giis/site]name=allowreg=":2135"allowreg=":2135"[mds/giis/site/registration/topeis]name=regname=reghn=[gridftp]log=/var/log/globus-gridftp.log[gatekeeper]default jobmanager=forkjob manager path=$GLOBUS LOCATION/libexecglobus gatekeeper=/opt/edg/sbin/edg-gatekeeperextra options= \"-lcas db file lcas.db -lcas etc dir /opt/edg/etc/lcas/ -lcasmod dir/opt/edg/lib/lcas/ -lcmaps db file lcmaps.db -lcmaps etc dir /opt/edg/etc/lcmaps-lcmapsmod dir /opt/edg/lib/lcmaps\"logfile=/var/log/globus-gatekeeper.logjobmanagers="fork "[gatekeeper/fork]type=forkjob manager=globus-job-manager[gatekeeper/pbs]type=


10.c) Creates the files :-/opt/edg/var/etc/ce-static.ldif as shown (please, use only VOs supported at yoursite):dn: GlueCEUniqueID=:2119/jobmanager-pbs-shortGlueCEAccessControlBaseRule: VO:aliceGlueCEAccessControlBaseRule: VO:atlasGlueCEAccessControlBaseRule: VO:cmsGlueCEAccessControlBaseRule: VO:lhcbGlueCEAccessControlBaseRule: VO:dteam

dn: GlueCEUniqueID=:2119/jobmanager-pbs-mediumGlueCEAccessControlBaseRule: VO:aliceGlueCEAccessControlBaseRule: VO:atlasGlueCEAccessControlBaseRule: VO:cmsGlueCEAccessControlBaseRule: VO:lhcbGlueCEAccessControlBaseRule: VO:dteam

dn: GlueCEUniqueID=:2119/jobmanager-pbs-longGlueCEAccessControlBaseRule: VO:aliceGlueCEAccessControlBaseRule: VO:atlasGlueCEAccessControlBaseRule: VO:cmsGlueCEAccessControlBaseRule: VO:lhcbGlueCEAccessControlBaseRule: VO:dteam

dn: GlueCEUniqueID=:2119/jobmanager-pbs-infiniteGlueCEAccessControlBaseRule: VO:aliceGlueCEAccessControlBaseRule: VO:atlasGlueCEAccessControlBaseRule: VO:cmsGlueCEAccessControlBaseRule: VO:lhcbGlueCEAccessControlBaseRule: VO:dteam

dn: GlueSubClusterUniqueID=, GlueClusterUniqueID=GlueHostApplicationSoftwareRunTimeEnvironment: LCG-TESTGlueHostNetworkAdapterOutboundIP: TRUEGlueHostNetworkAdapterInboundIP: FALSEGlueHostArchitectureSMPSize: 2GlueHostOperatingSystemName: RedhatGlueHostOperatingSystemRelease: 7.3GlueHostOperatingSystemVersion: 3GlueHostBenchmarkSF00: 400GlueHostBenchmarkSI00: 380GlueHostMainMemoryRAMSize: 512


GlueHostMainMemoryVirtualSize: 1024GlueHostProcessorClockSpeed: 1000GlueHostProcessorModel: PIIIGlueHostProcessorVendor: intel

-/opt/edg/var/info/info-provider.conf as shown:EDG LOCATION=/opt/edgGRID INFO USER=REP MET PRESENT=noREP LOC PRESENT=noREP OPT PRESENT=noSITE INFO=yesCE PRESENT=yes

-/opt/edg/var/info/siteinfo.ldif as shown:siteName: EIS (just a label...)sysAdminContact: "[email protected], [email protected],[email protected], [email protected]"userSupportContact: "[email protected], [email protected],[email protected], [email protected]"siteSecurityContact: "[email protected], [email protected],[email protected], [email protected]"dataGridVersion: t20031027 1450installationDate: 20031107153000Z

where you have to replace and other parameters.

10.d) Your final step on the GRIS configuration is the execution of the commands:export GLOBUS LOCATION=/opt/globus/opt/globus/sbin/globus-initialization.sh/opt/edg/sbin/edg-info-provider-setup/sbin/chkconfig globus-mds on/etc/rc.d/init.d/globus-mds start

10.e) Make sure the file/etc/host is properly configured. Otherwise the Information Providermay not work. The hostname and its IP must be included into this file.

11. Now we need to start the Globus Gatekeeper and the Globus GridFtp Server. To do so execute thefollowing commands:/sbin/chkconfig globus-gatekeeper on/sbin/chkconfig globus-gridftp on/etc/rc.d/init.d/globus-gatekeeper start/etc/rc.d/init.d/globus-gridftp start


12. Test the GRIS and the GIIS as follows:ldapsearch -h -p 2135 -b ‘‘mds-vo-name=local,o=grid’’ -xldapsearch -h -p 2135 -b ‘‘mds-vo-name=,o=grid’’-xwhere you must replace the and the (the value youinsert in the /etc/globus.conf file). If the output of these commands shows some entries like thecharacteristics of the PBS queues that you setup, then it is very possible that the GRIS and theGIIS just configured on the machine works.

13. Increase some system parameters to improve CE scalability and add these commands torc.localto let them survive reboots. The following line will do the trick:echo 120000 > /proc/sys/fs/file-maxcp -f /etc/rc.d/rc.local /etc/rc.d/rc.local.origcat >> /etc/rc.d/rc.local

@CONF.GCC@| gcc3 2 2|The gcc suffix as used on the build box (empty for 2.95,gcc3 2 2 for 3.2.)

• Configure the poolaccount. To do so you have to edit the file/opt/edg/etc/lcmaps/lcmaps.dbwith the following syntax:path = /opt/edg/lib/lcmaps/modules# module definitionslocalaccount = "lcmaps localaccount.mod -gridmapfile /etc/grid-security/grid-mapfile"poolaccount = "lcmaps poolaccount.mod -override inconsistency -gridmapfile/etc/grid-security/grid-mapfile -gridmapdir /etc/grid-security/gridmapdir/"posixenf = "lcmaps posix enf.mod -maxuid 1 -maxpgid 1 -maxsgid 32 "

# policiesstandard:localaccount -> posixenf | poolaccountpoolaccount -> posixenf

• Configure LCAS. To do so you need to create/edit the files

-/opt/edg/etc/lcas/lcas.db in this way:# LCAS database/plugin list## Format of each line:# pluginname="", pluginargs=""##pluginname="lcas userallow.mod",pluginargs="allowed users.db"pluginname="lcas userban.mod",pluginargs="ban users.db"pluginname="lcas timeslots.mod",pluginargs="timeslots.db"pluginname="lcas plugin example.mod",pluginargs="Some bogus arguments"

-/opt/edg/etc/lcas/timeslots.db as the following:## This file contains the time slots for which the fabric# is available for Grid jobs# Format:# minute1-minute2 hour1-hour2 mday1-mday2 month1-month2 year1-year2 wday1-wday2# max range: [0-59] [0-23] [1-31] [1-12] [1970-...] [0-6]## wday:# 0-6 = Sunday-Saturday# 5-3 = Friday-Wednesday# # ’*’ means the maximum range


# - means from to maximum value## The wall clock time should match at least one time slot for authorization# The wall clock time matches if:# (hour1:minute1)

/etc/obj/nfsmount stop/etc/obj/nfsmount start

on all the machines


7. WORKER NODE (WN)

In order to configure a WN, you need to have already a CE configured at your site.

1. Create on the WN the same accounts as on the CE, making sure you use the same UID/GIDmapping. (Use the-u option on the adduser and groupadd commands specifying the id. In orderto produce a complete list of LCG users and groups you can query password and group files on theCE.You do not need to have a grid map file on the WN any way.

2. Mounting of SE. Job running on the WN must have access to files on a SE. This can be achievedin different ways, using several protocols such as GridFtp. If “file” access is specified, the WN canmount a SE partition, as shown below.

2.a) Edit the file/etc/sysconfig/ipchains (create it if it does not exist) as in the example:# Firewall configuration written by lokkit# Manual customization of this file is not recommended.# Note: ifup-post will punch the current nameservers through the# firewall; such entries will *not* be listed here.:input ACCEPT:forward ACCEPT:output ACCEPT-A input -s 0/0 -d 0/0 21 -p tcp -y -j ACCEPT-A input -s 0/0 -d 0/0 22 -p tcp -y -j ACCEPT-A input -s 0/0 -d 0/0 -i lo -j ACCEPT-A input -s .cern.ch -d 0/0 -p tcp -y -j ACCEPT-A input -s .cern.ch -d 0/0 -p udp -j ACCEPT-A input -s .cern.ch -d 0/0 -p tcp -y -j ACCEPT-A input -s .cern.ch -d 0/0 -p udp -j ACCEPT-A input -s 137.138.0.0/16 -d 0/0 -p tcp -y -j ACCEPT-A input -s 137.138.0.0/16 -d 0/0 -p udp -j ACCEPT-A input -p tcp -s 137.138.0.0/16 -d 0/0 sunrpc -y -j ACCEPT-A input -p udp -s 137.138.0.0/16 -d 0/0 sunrpc -j ACCEPT-A input -p tcp -s 137.138.0.0/16 -d 0/0 nfs -y -j ACCEPT-A input -p udp -s 137.138.0.0/16 -d 0/0 nfs -j ACCEPT-A input -p tcp -s 0/0 -d 0/0 0:1023 -y -j REJECT-A input -p tcp -s 0/0 -d 0/0 2049 -y -j REJECT-A input -p udp -s 0/0 -d 0/0 0:1023 -j REJECT-A input -p udp -s 0/0 -d 0/0 2049 -j REJECT-A input -p tcp -s 0/0 -d 0/0 6000:6009 -y -j REJECT-A input -p tcp -s 0/0 -d 0/0 7100 -y -j REJECT

where and are respectively the Storage Elemment (e.g.tbed0101) and the Com-puting Element (e.g. pceis01) machines.


2.b) Restart the ipchains daemon with the command:/etc/rc.d/init.d/ipchains restart

2.c) Create the new mounting point :mkdir -p flatfiles/SE00

2.d) Edit the file/etc/fstab accordingly. In the following example the fstab file will mount alsothe directories exported by the CE (see CE instruction):append line

:/home /home nfs rw 0 0.cern.ch:/flatfiles/SE00 /flatfiles/SE00 nfs rw 0 0

2.e) Remount all partitions:mount -achkconfig netfs on/etc/rc.d/init.d/netfs start

2.f) Copy from the directory/opt/edg/etc/profile.d to /opt/edg/var/etc/profile.d thefollowing files:edg-wl-ui-env.cshedg-wl-ui-env.shedg-wl-ui-gui-env.cshedg-wl-ui-gui-env.sh

3. Configure Replica Manager. For this you can follow the instruction specified in the CE section ofthis manual (see step 14 on the CE instructions).

4. Add the WN to the CE PBS configuration file (point 7.c of CE installation)

5. Create the file :

/var/spool/pbs/server namewhich should contain one line with the full name of the CE host.

6. Create the file/opt/edg/etc/edg-pbs-knownhosts.conf following the example below wherepceis01 is the CE.

NODES = pceis01.cern.ch

PBSBIN = /usr/bin

KEYTYPES = rsa1,rsa,dsa


KNOWNHOSTS = /etc/ssh/ssh known hostsThen run the command:/opt/edg/sbin/edg-pbs-knownhost Remember to change theedg-pbs-knownhost.conf fileevery time you add a new WN. The command has to run every 6 hours via a cron job as root. Thefollowing line should be added to the root cron file using the commandcrontab -e.(Hints: crontab -e will use the default editor , usual “vi”. If you want edit your crontab with a differ-ent editor you should set the $VISUAL env variable with your preferred editor like/usr/bin/emacs).Add the following lines:PATH=/sbin:/bin:/usr/sbin:/usr/bin03 1,7,13,19 * * * /opt/edg/sbin/edg-pbs-knownhosts56 3,9,15,21 * * * /opt/edg/etc/cron/edg-fetch-crl-cron >>/var/log/edg-fetch-crl-cron.log 2>&1If needed, an automatic rotation of the log file can be obtained editing the file :/etc/logrotate.d/edg-fetch.

7. Add the following line to/etc/services:

pbs mom 15002/tcppbs remom 15003/tcppbs remom 15003/udp

8. As root, create the file/var/spool/pbs/mom priv/config and edit it as shown below:$clienthost localhost$clienthost .cern.ch$restricted .cern.ch$logevent 255$ideal load 1.6$max load 2.1$usecp *.cern.ch:/home /home

9. The pbsmom daemon (local batch system) needs to be configured and started:

chkconfig pbs mom on/etc/init.d/pbs mom start


8. STORAGE ELEMENT (SE)

The Storage Element serves storage space to the Grid. At each site you can have more than one SE. TheSE GRIS registers with the site GIIS which normally runs on the CE.

1. Create the accounts as specified in the Step 2 of the CE instructions.

2. Create the special account edginfobelonging to the groupedginfo.

3. On the SE machine, you need to have an area which is served via NFS to the WNs. In what fol-lows, we assume that this area is/flatfiles/SE00. If you have a second SE, since the name ofthis area must be unique per SE, you can name the second SE area/flatfiles/SE1. Under thedirectory/flatfiles/SE00 create the subdirectories corresponding to the groups created at theStep 1 with privileges set as shown:

[root@tbed0101 SE00]# ls -ltotal 20drwxrwxr-x 2 root alice 4096 Dec 12 12:03 alicedrwxrwxr-x 2 root atlas 4096 Dec 12 12:03 atlasdrwxrwxr-x 2 root cms 4096 Dec 15 10:12 cmsdrwxrwxr-x 2 root dteam 4096 Dec 12 12:03 dteamdrwxrwxr-x 2 root lhcb 4096 Dec 12 12:03 lhcb


# Root directory for EDG software# usual value: /opt/edgEDG LOCATION=/opt/edg# Directory for machine-specific files.# usual value :$EDG LOCATION/varEDG LOCATION VAR=/opt/edg/var# World writable directory for temporary files.# usual value: /tmpEDG TMP=/tmp# Host certificateX509 USER CERT=/etc/grid-security/hostcert.pem# Host keyX509 USER KEY=/etc/grid-security/hostkey.pem# grid mapfileGRIDMAP=/etc/grid-security/grid-mapfileGRIDMAPDIR=/etc/grid-security/gridmapdir/



GLOBUS LOCATION=/opt/globusGLOBUS CONFIG=/etc/globus.confGLOBUS TCP PORT RANGE="20000 25000"

6. Add groups “edginfo” and “edguser”, and users “edginfo” and “edguser” in their correspondinggroups and create a user and a group for every VO with the corresponding name of this VO.

7. Under the directory/flatfiles/SE00, for each VO create:

mkdir /flatfiles/SE00/chown -R /flatfiles/SE00/chmod g+rwx /flatfiles/SE00/

8. Configue GRIS.

8.a) Create the special user/group “edguser”

8.b) Create the file/etc/globus.conf as reported below:

[common]GLOBUS LOCATION=/opt/globusglobus flavor name=gcc32dbgx509 user cert=/etc/grid-security/hostcert.pemx509 user key=/etc/grid-security/hostkey.pemgridmap=/etc/grid-security/grid-mapfilegridmapdir=/etc/grid-security/gridmapdir/globus tcp port range="20000 25000"

[mds]globus flavor name=gcc32dbgpthruser=edginfo

[mds/gris/provider/edg]

[mds/gris/registration/site]regname=reghn=

[gridftp]log=/var/log/globus-gridftp.log

[gatekeeper]default jobmanager=forkjob manager path=$GLOBUS LOCATION/libexec


jobmanagers = ‘‘fork’’

[gatekeeper/fork]type=forkjob manager=globus-job-manager

8.c) Create the file/opt/lcg/var/lcg-info-generic.conf as follows:(Please note: it is very important to avoid blank space at the end of each line)ldif file=/opt/lcg/var/lcg-info-static.ldifgeneric script=/opt/lcg/libexec/lcg-info-genericwrapper script=/opt/lcg/libexec/lcg-info-wrapperdynamic script=/opt/lcg/libexec/lcg-info-dynamic-classic

:/flatfiles/SE00/GlueSEUniqueID=GlueSEName=:diskGlueSEPort=2811GlueInformationServiceURL=ldap://:2135/Mds-Vo-name=local,o=gridGlueSLUniqueID=GlueSLName=GlueSLArchitectureType=diskGlueSEAccessProtocolType=gsiftpGlueSARoot=: : ...GlueSAPolicyMaxFileSize=100000GlueSAPolicyMinFileSize=0GlueSAPolicyMaxData=0GlueSAPolicyMaxNumFiles=0GlueSAPolicyMaxPinDuration=0GlueSAStateAvailableSpace=0GlueSAStateUsedSpace=0GlueSAPolicyFileLifeTime=GlueServiceURI=GlueServiceAccessPointURL=GlueServiceType=GlueServicePrimaryOwnerName=GlueServicePrimaryOwnerContact=GlueServiceHostingOrganization=GlueServiceMajorVersion=GlueServiceMinorVersion=GlueServicePatchVersion=GlueServiceAccessControlRule=GlueServiceInformationServiceURL=GlueServiceStatus=


GlueSEAccessProtocolPort=2811rfio GlueSAAccessControlBaseRule=: GlueSAAccessControlBaseRule=: GlueSAPolicyFileLifeTime=permanent

If more than one VO is supported by the SE, the last part must be repeated for each VO. Besure that all fields have been properly updated.

8.d) Execute the command:/opt/lcg/sbin/lcg-info-generic-config /opt/lcg/var/lcg-info-generic.conf/opt/lcg/etc/GlueSE.templateNote1: execute again this command everytime the file:/opt/lcg/var/lcg-info-generic.conf is modified.Note2: be sure the file just created in/opt/lcg/libexec namedlcg-info-wrapper isexecutable.

8.e) Create the directories:/opt/edg/var/info and/opt/edg/etc/se.

8.f) Create the files:/opt/edg/var/info/info-provider.conf/opt/edg/etc/se/se-static-info.ini/opt/edg/etc/se/se-paths.confas shown in what is following and where you have to replace and is just a string of your choice.

/opt/edg/var/info/info-provider.confEDG LOCATION=/opt/edgGRID INFO USER=REP MET PRESENT=noREP LOC PRESENT=noREP OPT PRESENT=noSE PRESENT=yesSE MOUNT=/flatfiles/SE00

/opt/edg/etc/se/se-static-info.ini# This section contains general information. [information]SEName = :diskSLName = SLArchType = diskSEPort = 8080hostname = SEdirectory = /flatfiles/SE00entryttl = 3600GlueSchemaVersionMajor = 1GlueSchemaVersionMinor = 1


vos = alice atlas cms lhcb dteamprotocols = gridftp rfio file# gridftp protocol parameters.

[protocol/gridftp]port=2811

# rfio protocol parameters.

[protocol/rfio]port=5001

# file protocol parameters.

[protocol/file]

# Values for VOs.

[vo/alice]SARoot = alice:/aliceSAPolicyFileLifeTime = permanentSAPolicyMinFileSize = 1SAPolicyMaxFileSize = 100000SAPolicyMinFileSize = 1SAPolicyMaxData = 10000000SAPolicyMaxNumFiles = 100000SAPolicyMaxPinDuration = 100000

[vo/atlas]SARoot = atlas:/atlasSAPolicyFileLifeTime = permanentSAPolicyMinFileSize = 1SAPolicyMaxFileSize = 100000SAPolicyMinFileSize = 1SAPolicyMaxData = 10000000SAPolicyMaxNumFiles = 100000SAPolicyMaxPinDuration = 100000

[vo/cms]SARoot = cms:/cmsSAPolicyFileLifeTime = permanentSAPolicyMinFileSize = 1SAPolicyMaxFileSize = 100000SAPolicyMinFileSize = 1


SAPolicyMaxData = 10000000SAPolicyMaxNumFiles = 100000SAPolicyMaxPinDuration = 100000

[vo/lhcb]SARoot = lhcb:/lhcbSAPolicyFileLifeTime = permanentSAPolicyMinFileSize = 1SAPolicyMaxFileSize = 100000SAPolicyMinFileSize = 1SAPolicyMaxData = 10000000SAPolicyMaxNumFiles = 100000SAPolicyMaxPinDuration = 100000

[vo/dteam]SARoot = dteam:/dteamSAPolicyFileLifeTime = permanentSAPolicyMinFileSize = 1SAPolicyMaxFileSize = 100000SAPolicyMinFileSize = 1SAPolicyMaxData = 10000000SAPolicyMaxNumFiles = 100000SAPolicyMaxPinDuration = 100000

/opt/edg/etc/se/sepaths.conf## File automatically maintained by EDG-LCFGng# (edg-lcfg-se)# Changes will be discarded.# Steve Traylen # Version 1.0.32 : 16/10/03 10:34##

#location of the SE’s runtime configuration

EDG-SE-SERVER-PATH-RUN-TIME-CONF=/opt/edg/var/se/etc

# Maximum number of days data files may remain visible to outside world# and users.

EDG-SE-SERVER-TIME-OUT-CACHE-MAX-DAYS=7


# EDG-SE-SERVER-CASTOR-RFIO-PORT=5001#client CASTOR rfio port number. It is 5001 for Castor at CERNEDG-SE-SERVER-CASTOR-RFIO-PORT=5001#sets rfiodpresent key to either value yes or no. Dependent on#rfiod running on serverEDG-SE-SERVER-RFIO-DAEMON-PRESENT=yes

#EDG-SE-SERVER-WEBSERVICE-SEND-HEADER# variable used in rmanman-webservice.conf for passing info between webservice# and rmanman

EDG-SE-SERVER-WEBSERVICE-SEND-HEADER=no

# Username of user who runs the se.EDG-SE-SERVER-AS-GROUP=se

#mass storage hsm device,# can be: disk# rfio# adsEDG-SE-SERVER-MSS-TYPE=disk

# Groupname of user runs the se.EDG-SE-SERVER-AS-USER=se

# Following MSS-PREFIX is the path prefix for the SE mass storage# type ie /opt/edg/var/se/mss is for disk SEEDG-SE-SERVER-MSS-PREFIX=/bigdisk

# This is the location of the GACL librariesEDG-SE-SERVER-PATH-ACL=/opt/edg/var/se/gacl

# Location of the data files visible to outside world and users.EDG-SE-SERVER-PATH-DATA=/flatfile/SE00

# Location of grid map file; used by SE software to# validate usersEDG-SE-SERVER-PATH-GRIDMAP-NAME=/etc/grid-security/grid-mapfile

# Location of gridmapdir directory; used by SE software to# validate usersEDG-SE-SERVER-PATH-GRIDMAPDIR-NAME=/etc/grid-security/gridmapdir/


# Location of NFS root path Not currently usedEDG-SE-SERVER-PATH-NFS-ROOT=/flatfile/SE00

# Location of the SE file metadata catalogue ie f.filename. These# files are owned by the SE userEDG-SE-SERVER-PATH-VFS-ROOT=/opt/edg/var/se/vfs

# Location of userprofiles generated from gridmap file.EDG-SE-SERVER-PATH-PROFILES=/opt/edg/var/se/users

# Location of the tmp rmanman pipe for each transaction/request# in progress.EDG-SE-SERVER-PATH-RMANMAN-FIFO=/tmp

# Location of the tmp rmanman log file of request in progress.EDG-SE-SERVER-PATH-RMANMAN-LOG=/tmp/rman.log

# Location of the symbolic link path to the metadata catalogue# owned by the SEEDG-SE-SERVER-PATH-SYMLINKS=/opt/edg/var/se/links

# Location of the log files SE’s audit trail.EDG-SE-SERVER-PATH-TRANSACTION-ARCHIVE=/opt/edg/var/se/logs

# Location of the log directories SE’s audit trail again.EDG-SE-SERVER-PATH-TRANSACTION-DIR=/opt/edg/var/se/transaction

# Location of the XML templates used within the SE.EDG-SE-SERVER-PATH-TRANSACTION-XML=/opt/edg/etc/se/transaction-xml

# Location of the XSL templates used within the SE.EDG-SE-SERVER-PATH-TRANSACTION-XSL=/opt/edg/etc/se/transaction-xsl

# Location of the Apache logsEDG-SE-SERVER-PATH-GSI-APACHE-VAR=/opt/edg/var/se/httpd

# Location of SE XmlDatabaseEDG-SE-SERVER-PATH-XML-DATABASE=/opt/edg/var/se/XmlDatabase

• Make sure the file/etc/host is properly configured. Otherwise the Information Providermay not work. The hostname and its IP must be included into this file.


9. Execute the following commands:

export GLOBUS LOCATION=/opt/globus/opt/globus/sbin/globus-initialization.sh/opt/edg/sbin/edg-info-provider-setup/sbin/chkconfig globus-mds on/etc/rc.d/init.d/globus-mds start

10. If you have an RLS at your site, you can use the GRIS on the SE to publish the RLS information.To do so execute the following steps:

10.a) Check that/opt/edg/libexec/edg-info-service and/opt/edg/etc/info/service.ldif.inare installed

10.b) mkdir /opt/edg/etc/info/info-provider-sevicecd /opt/edg/etc/info/info-provider-seviceln -s ../service.ldif.in SR.ldif.template

10.c) Edit the file/opt/edg/var/info/info-provider.conf and make sure the following linesappear:

REP MET PRESENT=yesREP LOC PRESENT=yes

10.d) Run the script:/opt/edg/sbin/edg-info-provider-setup

10.e) Edit the file just created :/opt/edg/var/info/edg-globus.ldif and change the line:args: /opt/edg/etc/info/glue-infoprovider-service-lrc.configwith the lineargs: /opt/edg/etc/info/glue-infoprovider-service-lrc-.configfor both the lrc and rmc. Make sure also the endpoints forlrc andrmc are correct. Theyshould be:dn: GlueServiceURI=http://:7777//edg-local-replica-catalog/services/edg-local-replica-catalog, Mds-Vo-Name=local, o=Gridanddn: GlueServiceURI=http://:7777//edg-replica-metadata-catalog/services/edg-replica-metadata-catalog, Mds-Vo-Name=local, o=GridAdd GlueServiceURI block for each VO you want to support.

10.f) Under/opt/edg/etc/info create the endpoint configuration files:glue-infoprovider-service-rmc-.configglue-infoprovider-service-lrc-.configfollowing the example below:


#file:glue-infoprovider-service-rmc-.config# Only ONE EMPTY LINE BETWEEN SECTIONS# section names,only two letters#

[SR]@URI: http://:7777//edg-replica-metadata-catalog/services/edg-replica-metadata-catalog@AccessPointURL: http://:7777//edg-replica-metadata-catalog/services/edg-replica-metadata-catalog@Type: edg-replica-metadata-catalog@PrimaryOwnerName: LCG@PrimaryOwnerContact: mailto:[email protected]@HostingOrganization: CERN@MajorVersion: 1@MinorVersion: 0@PatchVersion: 1@AccessControlRule: @InformationServiceURL: MDS2GRIS:ldap://:2170/mds-vo-name=local,o=grid@Status: running[SR end]

# End configuration fileand#file:glue-infoprovider-service-lrc-.config# Only ONE EMPTY LINE BETWEEN SECTIONS# section names,only two letters#

[SR]@URI: http://:7777//edg-local-replica-catalog/services/edg-local-replica-catalog@AccessPointURL: http://:7777//edg-local-replica-catalog/services/edg-local-replica-catalog@Type: edg-local-replica-catalog@PrimaryOwnerName: LCG@PrimaryOwnerContact: mailto:[email protected]@HostingOrganization: CERN@MajorVersion: 1@MinorVersion: 0@PatchVersion: 1@AccessControlRule: @InformationServiceURL: MDS2GRIS:ldap://:2170/mds-vo-name=local,o=grid@Status: running


[SR end]

# End configuration file

10.g) Execute the commands :export GLOBUS LOCATION=/opt/globus/opt/globus/sbin/globus-initialization.sh

10.h) Stop and start globus-mds:/sbin/chkconfig globus-mds on/etc/rc.d/init.d/globus-mds stop/etc/rc.d/init.d/globus-mds start

11. Test the GRIS as follows:ldapsearch -h -p 2135 -b ‘‘mds-vo-name=local,o=grid’’ -xif the output contains the status of the SE, then the service is working

12. Start the GridFtp server with:/sbin/chkconfig globus-gridftp on/etc/rc.d/init.d/globus-gridftp start

13. Enter the following lines to the crontab with crontab -e command:PATH=/sbin:/bin:/usr/sbin:/usr/bin45 6,8,10,12,14,16,18,20 * * * /opt/edg/sbin/edg-mkgridmap -output/etc/grid-security/grid-mapfile > /dev/null 2> /dev/null25 1,7,13,19 * * * /usr/bin/perl /opt/edg/sbin/edg se mkprofile.pl -g/etc/grid-security/grid-mapfile25 1,7,13,19 * * * /opt/edg/etc/cron/edg-fetch-crl-cron > /dev/null 2> /dev/null

14. Reboot the machine


9. RESOURCE BROKER (RB)

In this section we describe how to install a RB machine. If you do not want to install an RB at your site,please contact the EIS team to know which RB you can use.

1. Create the local user and groupedguserand make run the bash as default shell.

2. Create the local users and groupsalice, atlas, cms, lhcb, dteam, one for each VO and make themrun bash as default shell.

3. Create the following directories and modify ownerships and permissions:

chmod a+x /opt/edg/etc/profile.d/edg-wl-config.shmkdir -p /tmp/SandboxDirchown -R edguser:edguser /tmp/SandboxDirmkdir -p /var/edgwl/jobcontrol/condmkdir -p /var/edgwl/jobcontrol/logmkdir -p /var/edgwl/jobcontrol/submitmkdir -p /var/edgwl/loggingmkdir -p /var/edgwl/logmonitor/CondorG.logmkdir -p /var/edgwl/jobcontrol/internalmkdir -p /var/edgwl/logmonitor/CondorG.log/recyclemkdir -p /var/edgwl/networkserver/logmkdir -p /var/edgwl/SandboxDirchmod g+w /var/edgwl/SandboxDirmkdir -p /var/edgwl/workload manager/logchown -R edguser:edguser /var/edgwlmkdir -p /opt/globus/var/condor/logmkdir -p /opt/globus/var/condor/spoolmkdir -p /opt/globus/var/condor/log/GridLogsmkdir -p /opt/condor/setup/globus/condorchmod ug+rwx /opt/globus/var/condor/log/GridLogschown -R edguser:edguser /opt/globus/varchown -R edguser:edguser /opt/condor/var/logchown -R edguser:edguser /opt/condor/var/condormkdir -p /opt/condor/var/condor/log/GridLogschown -R edguser:edguser /opt/condor/var/condor/log/GridLogs


# Root directory for EDG software. (mandatory)# Usual value: /opt/edgEDG LOCATION=/opt/edg# Directory for machine-specific files.


# Usual value: $EDG LOCATION/varEDG LOCATION VAR=/opt/edg/var# World writable directory for temporary files. (mandatory)# Usual value: /tmpEDG TMP=/tmp# The directory containing trusted certificates and CRLs (CERTDIR).# Usual value: /etc/grid-security/certificates# Host certificate (X509 USER CERT) for services which don’t have their own.# Usual value: /etc/grid-security/hostcert.pemX509 USER CERT=/etc/grid-security/hostcert.pem# Host key (X509 USER KEY) for services which don’t have their own.# Usual value: /etc/grid-security/hostkey.pemX509 USER KEY=/etc/grid-security/hostkey.pem# Location of the grid mapfile (GRIDMAP).# Usual value: /etc/grid-security/grid-mapfileGRIDMAP=/etc/grid-security/grid-mapfile# Location of the grid map directory for pooled accounts (GRIDMAPDIR).# Usual value: /etc/grid-security/gridmapdirGRIDMAPDIR=/etc/grid-security/gridmapdir/


GLOBUS LOCATION=/opt/globusGLOBUS CONFIG=/etc/globus.confGLOBUS TCP PORT RANGE="20000 25000"

6. Create the file/opt/edg/etc/edg wl.conf as follows:

[Common = [DGUser = "$EDG WL USER";HostProxyFile = "/var/edgwl/networkserver/ns.proxy";];JobController = [CondorSubmit = "$CONDORG INSTALL PATH/bin/condor submit";CondorRemove = "$CONDORG INSTALL PATH/bin/condor rm";CondorQuery = "$CONDORG INSTALL PATH/bin/condor q";CondorSubmitDag = "$CONDORG INSTALL PATH/bin/condor submit dag";CondorRelease = "$CONDORG INSTALL PATH/bin/condor release";SubmitFileDir = "$EDG WL TMP/jobcontrol/submit";OutputFileDir = "$EDG WL TMP/jobcontrol/cond";Input = "$EDG WL TMP/jobcontrol/queue.fl";LogFile = "$EDG WL TMP/jobcontrol/log/events.log";


LogLevel = 5;ContainerRefreshThreshold = 1000;];LogMonitor = [JobsPerCondorLog = 1000;LogFile = "$EDG WL TMP/logmonitor/log/events.log";LogLevel = 5;MainLoopDuration = 10;CondorLogDir = "$EDG WL TMP/logmonitor/CondorG.log";CondorLogRecycleDir = "$EDG WL TMP/logmonitor/CondorG.log/recycle";MonitorInternalDir = "$EDG WL TMP/logmonitor/internal";IdRepositoryName = "irepository.dat";AbortedJobsTimeout = 600;];NetworkServer = [II Port = 2170;II Timeout = 30;II DN = "mds-vo-name=local,o=grid";II Contact = "tbed0102.cern.ch";Gris Port = 2135;Gris Timeout = 20;Gris DN = "mds-vo-name=local,o=grid";LogFile = "$EDG WL TMP/networkserver/log/events.log";LogLevel = 5;ListeningPort = 7772;MasterThreads = 8;DispatcherThreads = 10;SandboxStagingPath = "$EDG WL TMP/SandboxDir";EnableQuotaManagement = false;MaxInputSandboxSize = 10000000;EnableDynamicQuotaAdjustment = false;QuotaAdjustmentAmount = 10000;QuotaInsensibleDiskPortion = 2.0;];WorkloadManager = [PipeDepth = 1;NumberOfWorkerThreads = 1;DispatcherType = "filelist";Input = "$EDG WL TMP/workload manager/input.fl";MaxRetryCount = 10;LogFile = "$EDG WL TMP/workload manager/log/events.log";LogLevel = 5;];]


7. Configure the mysql database for lbserver:

/sbin/chkconfig mysql on/etc/rc.d/init.d/mysql startmysqladmin password datagridmysqladmin -p create lbserver20mysql -p lbserver20 < /opt/edg/etc/server.sqlmysql -p lbserver20Enter password: *****mysql> grant all on lbserver20.* to lbserver@localhost / gmysql> / qj


[common]GLOBUS LOCATION=/opt/globusglobus flavor name=gcc32dbgx509 user cert=/etc/grid-security/hostcert.pemx509 user key=/etc/grid-security/hostkey.pemgridmap=/etc/grid-security/grid-mapfilegridmapdir=/etc/grid-security/gridmapdir/[mds][gridftp]log=/var/log/globus-gridftp.log[gatekeeper]default jobmanager=forkjob manager path=$GLOBUS LOCATION/libexecjobmanagers="fork "[gatekeeper/fork]type=forkjob manager=globus-job-manager

9. Execute:

export GLOBUS LOCATION=/opt/globus/opt/globus/sbin/globus-initialization.sh

10. Open the existing file:/opt/edg/etc/edg-mkgridmap.conf and modify it as follows:

# EDG Virtual Organisations# eg ’group ldap://grid-vo.cnaf.infn.it/ou=testbed1,o=infn,c=it .infngrid’# Map VO members alicesgmgroup ldap://grid-vo.nikhef.nl/ou=lcgadmin,o=alice,dc=eu-datagrid,dc=orgalicesgm


# Map VO members alicegroup ldap://grid-vo.nikhef.nl/ou=lcg1,o=alice,dc=eu-datagrid,dc=org alice# Map VO members atlassgmgroup ldap://grid-vo.nikhef.nl/ou=lcgadmin,o=atlas,dc=eu-datagrid,dc=orgatlassgm# Map VO members atlasgroup ldap://grid-vo.nikhef.nl/ou=lcg1,o=atlas,dc=eu-datagrid,dc=org atlas# Map VO members cmssgmgroup ldap://grid-vo.nikhef.nl/ou=lcgadmin,o=cms,dc=eu-datagrid,dc=orgcmssgm# Map VO members cmsgroup ldap://grid-vo.nikhef.nl/ou=lcg1,o=cms,dc=eu-datagrid,dc=org cms# Map VO members lhcbsgmgroup ldap://grid-vo.nikhef.nl/ou=lcgadmin,o=lhcb,dc=eu-datagrid,dc=orglhcbsgm# Map VO members lhcbgroup ldap://grid-vo.nikhef.nl/ou=lcg1,o=lhcb,dc=eu-datagrid,dc=org lhcb# Map VO members dteamsgmgroup ldap://lcg-vo.cern.ch/ou=lcgadmin,o=dteam,dc=lcg,dc=org dteamsgm# Map VO members dteamgroup ldap://lcg-vo.cern.ch/ou=lcg1,o=dteam,dc=lcg,dc=org dteam# List of auth URIs# eg ’auth ldap://marianne.in2p3.fr/ou=People,o=testbed,dc=eu-datagrid,dc=org’# If these are defined then users must be authorised in one of the following# auth servers.# A list of athorised users.auth ldap://lcg-registrar.cern.ch/ou=users,o=registrar,dc=lcg,dc=org# DEFAULT LCLUSER: default lcluser lcluser# default lcuser .# ALLOW and DENY: deny|allow pattern to match# allow INFN# Local grid-mapfile to import and overide all the above information.# eg, gmf local /opt/edg/etc/grid-mapfile-localgmf local /opt/edg/etc/grid-mapfile-local

11. Create the file/opt/edg/etc/grid-mapfile-local which in principle will be empty:

# A lists of user mappings that will be imported into the # grid-mapfile.

12. Join the following command to the crontab:

45 6,8,10,12,14,16,18,20 * * * /opt/edg/sbin/edg-mkgridmap --output/etc/grid-security/grid-mapfile 2>&1

13. Create the file/etc/myproxy-server.config as follows:


accepted credentials "/C=TW/*"accepted credentials "/C=CH/O=CERN/OU=GRID/*"accepted credentials "/C=CZ/O=CESNET/*"accepted credentials "/O=CESNET/*"accepted credentials "/C=FR/O=CNRS/CN=CNRS-Projets"accepted credentials "/C=FR/O=CNRS/CN=CNRS"accepted credentials "/*"accepted credentials "/C=FR/O=CNRS/CN=Datagrid-fr"accepted credentials "/C=FR/O=CNRS/CN=CNRS-Projets"accepted credentials "/C=CY/O=CyGrid/*"accepted credentials "/DC=org/DC=DOEGrids/*"accepted credentials "/DC=org/DC=doegrids/*"accepted credentials "/O=DOEGrids.org/*"accepted credentials "/O=doegrids.org/*"accepted credentials "/O=doesciencegrid.org/*"accepted credentials "/DC=org/DC=doesciencegrid/*"accepted credentials "/O=doesciencegrid.org/*"accepted credentials "/DC=org/DC=doesciencegrid/*"accepted credentials "/O=DOE Science Grid/OU=Certificate Authorities/CN=CertificateManager"accepted credentials "/DC=net/DC=es/OU=Certificate Authorities/OU=DOEScience Grid/CN=pki1"accepted credentials "/DC=org/DC=DOEGrids/OU=Certificate Authorities/*"accepted credentials "/DC=net/DC=ES/*"accepted credentials "/C=DE/O=FZK-Grid/*"accepted credentials "/O=GermanGrid/OU=*"accepted credentials "/C=DE/O=FZK-Grid/*"accepted credentials "/C=DE/O=GermanGrid/*"accepted credentials "/O=GermanGrid/OU=*"accepted credentials "/C=IE/O=Grid-Ireland/*"accepted credentials "/C=CA/O=Grid/*"accepted credentials "/O=Grid/O=UKHEP/*"accepted credentials "/C=GR/O=HellasGrid/*"accepted credentials "/C=it/O=INFN/*"accepted credentials "/C=IT/O=INFN/*"accepted credentials "/C=PT/O=LIP/*"accepted credentials "/C=NL/O=NIKHEF/CN=NIKHEF medium-security certificationauth"accepted credentials "/O=dutchgrid/O=users/*"accepted credentials "/O=dutchgrid/O=hosts/*"accepted credentials "/O=Grid/O=NorduGrid/*"accepted credentials "/C=PL/O=GRID/*"accepted credentials "/C=RU/O=DataGrid/*"accepted credentials "/C=SK/O=SlovakGrid/*"


accepted credentials "/C=ES/O=DATAGRID-ES/*"

Execute now:

/sbin/chkconfig myproxy on/etc/rc.d/init.d/myproxy start

14. Execute the commands:

/sbin/chkconfig edg-wl-ftpd on/etc/rc.d/init.d/edg-wl-ftpd start

15. Execute:

/sbin/chkconfig edg-wl-lbserver on/etc/rc.d/init.d/edg-wl-lbserver start

16. Execute:

mkdir -p /etc/grid-security/gridmapdirchown -R root:edguser /etc/grid-security/gridmapdir

17. Modify the following two scripts:

/etc/init.d/edg-wl-ns/etc/init.d/edg-wl-wm

to include the line:

export EDG LOCATION=$EDG WL LOCATION

just after the line:

EDG WL LOCATION=${EDG WL LOCATION:-/opt/edg}

and restart the services. Otherwise the matchmaking with input files will not work.

18. Execute:

/sbin/chkconfig edg-wl-ns on/etc/rc.d/init.d/edg-wl-ns start/sbin/chkconfig edg-wl-wm on/etc/rc.d/init.d/edg-wl-wm start

19. Create the file/opt/condor/etc/condor.conf as follows:


## PART 1#CONDOR HOST = $(FULL HOSTNAME)RELEASE DIR = /opt/condorLOCAL DIR = /opt/condor/var/condorCONDOR ADMIN = S̈[email protected], [email protected],[email protected], [email protected]¨MAIL = /bin/mailUID DOMAIN = $(FULL HOSTNAME)FILESYSTEM DOMAIN = $(FULL HOSTNAME)## PART 2#DAEMON LIST = MASTER, SCHEDDFLOCK NEGOTIATOR HOSTS = $(FLOCK TO)FLOCK COLLECTOR HOSTS = $(FLOCK TO)HOSTALLOW ADMINISTRATOR = $(CONDOR HOST)HOSTALLOW OWNER = $(FULL HOSTNAME), $(HOSTALLOW ADMINISTRATOR)HOSTALLOW READ = *HOSTALLOW WRITE = $(FULL HOSTNAME), $(GLIDEIN SITES)HOSTALLOW NEGOTIATOR = $(NEGOTIATOR HOST)HOSTALLOW NEGOTIATOR SCHEDD = $(NEGOTIATOR HOST), $(FLOCK NEGOTIATOR HOSTS)HOSTALLOW WRITE COLLECTOR = $(HOSTALLOW WRITE), $(FLOCK FROM)HOSTALLOW WRITE STARTD = $(HOSTALLOW WRITE), $(FLOCK FROM)HOSTALLOW READ COLLECTOR = $(HOSTALLOW READ), $(FLOCK FROM)HOSTALLOW READ STARTD = $(HOSTALLOW READ), $(FLOCK FROM)LOCK = $(LOG)MAX SCHEDD LOG = 64000000SCHEDD DEBUG = D COMMANDMAX GRIDMANAGER LOG = 64000000GRIDMANAGER DEBUG = D COMMANDMAX COLLECTOR LOG = 64000000COLLECTOR DEBUG = D COMMANDMAX NEGOTIATOR LOG = 64000000NEGOTIATOR DEBUG = D MATCHMAX NEGOTIATOR MATCH LOG = 64000000MAX SHADOW LOG = 64000000## PART 3#MINUTE = 60HOUR = (60 * $(MINUTE))StateTimer = (CurrentTime - EnteredCurrentState)


ActivityTimer = (CurrentTime - EnteredCurrentActivity)ActivationTimer = (CurrentTime - JobStart)LastCkpt = (CurrentTime - LastPeriodicCheckpoint)STANDARD = 1PVM = 4VANILLA = 5IsPVM = (JobUniverse == $(PVM))IsVANILLA = (JobUniverse == $(VANILLA))IsSTANDARD = (JobUniverse == $(STANDARD))NonCondorLoadAvg = (LoadAvg - CondorLoadAvg)BackgroundLoad = 0.3HighLoad = 0.5StartIdleTime = 15 * $(MINUTE)ContinueIdleTime = 5 * $(MINUTE)MaxSuspendTime = 10 * $(MINUTE)MaxVacateTime = 10 * $(MINUTE)KeyboardBusy = (KeyboardIdle < $(MINUTE))ConsoleBusy = (ConsoleIdle < $(MINUTE))CPU Idle = ($(NonCondorLoadAvg) = $(HighLoad))BigJob = (ImageSize >= (50 * 1024))MediumJob = (ImageSize >= (15 * 1024) && ImageSize < (50 * 1024))SmallJob = (ImageSize < (15 * 1024))JustCPU = ($(CPU Busy) && ($(KeyboardBusy) == False))MachineBusy = ($(CPU Busy) || $(KeyboardBusy))## PART 4#DISABLE AUTH NEGOTIATION = trueLOG = $(LOCAL DIR)/logSPOOL = $(LOCAL DIR)/spoolEXECUTE = $(LOCAL DIR)/executeBIN = $(RELEASE DIR)/binLIB = $(RELEASE DIR)/libSBIN = $(RELEASE DIR)/sbinHISTORY = $(SPOOL)/historyMASTER LOG = $(LOG)/MasterLogSCHEDD LOG = $(LOG)/SchedLogGRIDMANAGER LOG = $(LOG)/GridLogs/GridmanagerLog.$(USERNAME)SHADOW LOG = $(LOG)/ShadowLogCOLLECTOR LOG = $(LOG)/CollectorLogNEGOTIATOR LOG = $(LOG)/NegotiatorLogNEGOTIATOR MATCH LOG = $(LOG)/MatchLogSHADOW LOCK = $(LOCK)/ShadowLock


RESERVED DISK = 5MASTER = $(SBIN)/condor masterSCHEDD = $(SBIN)/condor scheddNEGOTIATOR = $(SBIN)/condor negotiatorCOLLECTOR = $(SBIN)/condor collectorMASTER ADDRESS FILE = $(LOG)/.master addressPREEN = $(SBIN)/condor preenPREEN ARGS = -m -rSHADOW = $(SBIN)/condor shadowSHADOW PVM = $(SBIN)/condor shadow.pvmGRIDMANAGER = $(SBIN)/condor gridmanagerGAHP = $(SBIN)/gahp serverSCHEDD ADDRESS FILE = $(LOG)/.schedd addressSHADOW SIZE ESTIMATE = 1800SHADOW RENICE INCREMENT = 10QUEUE SUPER USERS = root, condorPVMD = $(SBIN)/condor pvmdPVMGS = $(SBIN)/condor pvmgsDEFAULT UNIVERSE = globusCRED MIN TIME LEFT = 120VALID SPOOL FILES = job queue.log, job queue.log.tmp, history, Accountant.log,Accountantnew.logINVALID LOG FILES = coreGLIDEIN SERVER NAME = gridftp.cs.wisc.eduGLIDEIN SERVER DIR = /p/condor/public/binaries/glideinAUTHENTICATION METHODS = CLAIMTOBEENABLE GRID MONITOR = TRUEGRID MONITOR = $(SBIN)/grid monitor.shGRIDMANAGER MINIMUM PROXY TIME = 600GRIDMANAGER MAX SUBMITTED JOBS PER RESOURCE = 32000GRIDMANAGER MAX PENDING SUBMITS PER RESOURCE = 20

20. Execute:

/sbin/chkconfig edg-wl-jc on/etc/rc.d/init.d/edg-wl-jc start/sbin/chkconfig edg-wl-lm on/etc/rc.d/init.d/edg-wl-lm start/sbin/chkconfig edg-wl-locallogger on/etc/rc.d/init.d/edg-wl-locallogger start /sbin/chkconfig edg-wl-proxyrenewalon/etc/rc.d/init.d/edg-wl-proxyrenewal start

21.


22. Configure theedg-mkgridmap:

• Add the line:45 6,8,10,12,14,16,18,20 * * * /opt/edg/sbin/edg-mkgridmap --output/etc/grid-security/grid-mapfile 2>& 1

• Edit the file/opt/edg/etc/edg-mkgridmap.conf and move each VO member to its corre-sponding account without the dot “.” , for example:# Map VO members atlas groupldap://grid-vo.nikhef.nl/ou=lcg1,o=atlas,dc=eu-datagrid,dc=org atlas

• This file above contains a call to the file/opt/edg/etc/grid-mapfile-local. Into grid-mapfile-local you can join users which are locally map into your site (although is not mapinto the general gridmap server) as follows:# A lists of user mappings that will be imported into the grid-mapfile."/C=CH/O=CERN/OU=GRID/CN=Heinz Stockinger 8894" cms

23. Put the following lines into the cron tab:

PATH=/sbin:/bin:/usr/sbin:/usr/bin23 1,7,13,19 * * * /opt/edg/etc/cron/edg-fetch-crl-cron23 2,8,14,20 * * * /sbin/service edg-wl-locallogger proxy23 2,8,14,20 * * * /sbin/service edg-wl-lbserver proxy23 2,8,14,20 * * * /sbin/service edg-wl-proxyrenewal proxy23 2,8,14,20 * * * /sbin/service edg-wl-ns proxy23 */1 * * 1-6 /opt/edg/libexec/edg-wl-purgestorage.sh hourly23 */4 * * 0 /opt/edg/libexec/edg-wl-purgestorage.sh weekly

24. Configure the EDG Replica Manager as follows:

@EDG.LOCATION@|/opt/edg|location of edg the directory@LOCALDOMAIN@|cern.ch|the local [email protected]@|tbed0101.cern.ch|the host of the close [email protected]@|pceis01.cern.ch|the host of the close CE@INFOSERVICE@|MDS|The info provider to use. It can be Stub, MDS or [email protected]@|LrcOnly|The mode the RLS should be run in. LrcOnly or WithRli@STUBFILE@||The properties file for the static file ’info service’@MDS.HOST@|tbed0102.cern.ch|The host of the MDS info [email protected]@|2170|The port of the MDS info [email protected]|false|Fail if no ROS is [email protected]@| gcc3 2 2|The gcc suffix as used on the build box (empty for2.95, gcc3 2 2 for 3.2.)

And execute:

/opt/edg/sbin/edg-replica-manager/configure/opt/edg/etc/edg-replica-manager/edg-replica-manager.conf.values


10. TOP GRID INFORMATION INDEX SERVER (GIIS OR MDS)

In this section we describe the steps necessary to install a TOP GIIS. For further details about the archi-tecture of the Globus MDS, GRIS, TOP GIIS and BDII, please consult the LCG-1 User Guide [R6].

1. Create the user/groupedginfo.


[common]GLOBUS LOCATION=/opt/globusglobus flavor name=gcc32dbgx509 user cert=/etc/grid-security/hostcert.pemx509 user key=/etc/grid-security/hostkey.pemgridmap=/etc/grid-security/grid-mapfilegridmapdir=/etc/grid-security/gridmapdir/[mds]globus flavor name=gcc32dbgpthruser=edginfo[mds/gris]suffix="mds-vo-name=dummy,o=grid"[mds/gris/provider/alledg][mds/giis/topmds]name=topeisallowreg="tbed0102.cern.ch:2135"allowreg="pceis01.cern.ch:2135"[mds/giis/local]name=localallowreg="tbed0102.cern.ch:2135"[mds/giis/topmds/registration/local]name=topeisregname=localreghn=tbed0102.cern.chregperiod=600ttl=1200[gridftp][gatekeeper]default jobmanager=forkjob manager path=$GLOBUS LOCATION/libexecjobmanagers="fork "[gatekeeper/fork]type=forkjob manager=globus-job-manager

3. Execute the following commands:


export GLOBUS LOCATION=/opt/globus/opt/globus/sbin/globus-initialization.sh/sbin/chkconfig globus-mds on/etc/rc.d/init.d/globus-mds start


11. BERKLEY DB INFORMATION INDEX (DBII)

• Create the following directories and modify ownerships and permissions:

mkdir -p /opt/lcg/var/bdiimkdir -p /opr/lcg/var/bdii/LDAPchown -R edginfo:edginfo LDAP

• In the directory

/opt/lcg/var/bdii

create and edit accordingly the filelcg-bdii.conf observing that the node name of the bdii andthe url of thelcg2-bdii-update.conf should be entered. The template of the file is the follow-ing:

BDII HOST=BDII PORT=2170BDII USER=edginfoBDII BIND=mds-vo-name=local,o=gridBDII TTL=300BDII TIMEOUT=30BDII AUTO UPDATE=yesBDII UPDATE=/opt/lcg/libexec/lcg-bdii-updateBDII UPDATE CONFIG=/opt/lcg/libexec/lcg-bdii-update-configBDII UPDATE CONFIG FILE=/opt/lcg/var/bdii/lcg-bdii-update.confBDII BOOTSTRAP URL=BDII CLEAN=yes

SLAPD=/opt/openldap/libexec/slapdSLAPADD=/opt/openldap/sbin/slapaddLDAP CONF=/opt/lcg/etc/lcg-bdii-slapd.confLDAP DB=/opt/lcg/var/bdii/LDAPPID FILE=/opt/lcg/var/bdii/slapd.pid

Here you have an example of this file:

BDII HOST=tbed0101.cern.chBDII PORT=2170BDII USER=edginfoBDII BIND=mds-vo-name=local,o=gridBDII TTL=300


BDII TIMEOUT=30BDII AUTO UPDATE=yesBDII UPDATE=/opt/lcg/libexec/lcg-bdii-updateBDII UPDATE CONFIG=/opt/lcg/libexec/lcg-bdii-update-configBDII UPDATE CONFIG FILE=/opt/lcg/var/bdii/lcg-bdii-update.confBDII BOOTSTRAP URL=http://grid-deployment.web.cern.ch/grid-deployment/eis/testbed/eis-bdii-update.confBDII CLEAN=yes

SLAPD=/opt/openldap/libexec/slapdSLAPADD=/opt/openldap/sbin/slapaddLDAP CONF=/opt/lcg/etc/lcg-bdii-slapd.confLDAP DB=/opt/lcg/var/bdii/LDAPPID FILE=/opt/lcg/var/bdii/slapd.pid

• Start up the servers:

3.1) lcg-bdii-refresh/opt/lcg/libexec/lcg-bdii-refresh /opt/lcg/var/bdii/lcg-bdii.conf

The script generates the file

lcg-bdii-update.conf in /opt/lcg/var/bdii

and will update it regularly once the cronjob in the crontab is specified. The file contains allthe relevant information about the LDAP urls.

3.2) lcg-bdii system script

/etc/init.d/lcg-bdii start

• Add the following lines to the crontab:

PATH=/sbin:/bin:/usr/sbin:/usr/bin/2 * * * * /opt/lcg/libexec/lcg-bdii-refresh /opt/lcg/var/bdii/lcg-bdii.conf> /dev/null


12. RLS INSTALLATION

The Testbed used to develop the present guide (EIS-testbed) has not included a RLS node. Its corre-sponding installation is therefore not included here. However you can find details of its installation intothe WP2 documentation web page.

http://edg-wp2.web.cern.ch/edg-wp2/replication/documentation.html


13. ACKNOLEDGEMENTS

We would like to thank the persons of the IT/GD division for their great support and suggestions towrite this guide. In particular to Heinz Stockinger as author of the LCG-1 Manual Installation Guide forTutorials and for all his constributions to this current manual and to Alessandro Usai and Antonio Reticofor giving us their guides on the UI and WN configuration. Finally to the users and experiments for theirconstributions.


INTRODUCTIONOBJECTIVES OF THIS DOCUMENTREFERENCE DOCUMENTSTERMINOLOGYDEFINITIONSASSUMPTIONS

RPMS DOWNLOADSOFTWARE INSTALLATIONGENERAL INFORMATION FOR SECURITY/GRID-MAPFILE

GENERAL REQUESTS FOR THE LCG-2 TESTBED INSTALLATIONUSER INTERFACE (UI)COMPUTING ELEMENT (CE)WORKER NODE (WN)STORAGE ELEMENT (SE)RESOURCE BROKER (RB)TOP GRID INFORMATION INDEX SERVER (GIIS OR MDS)BERKLEY DB INFORMATION INDEX (DBII)RLS INSTALLATIONAcknoledgements

lcg-2 manual installation g - uomlcg-2 manual installation guide document identiﬁer:...

Documents