A module to customize CREAM A module to customize CREAM jobs according to site policiesjobs according to site policies
Tsukuba, KEK, 21 December 2010
Sylvain Reynaud
JWGEN :JWGEN :
2
The problemThe problem
Adapt the job…– to user needs
• set user job environment– middleware version, operating
system, environment variable
• set LRMS queue• set hardware resources
– CPU, memory, disk
• set logical resources– hpss, xrootd, dCache, oracle
• set permissions (sw manager)
– to site policy• accept/deny job submission• enforcement of security policy• options and information for :
– quota, priorities, accounting
According to the information associated with this job
– Computing Element (CE)
– CE Queue
– User's virtual organization
– User's role
– JDL requirements (if specified)• CPU• memory• disk
3
A solution : JWGENA solution : JWGEN
JWGEN : a module to adapt grid job to…– user needs
– site policy
…according to information associated with this job– CE, queue, VO, requirements…
CREAM-CECREAM-CE
BLAHBLAH GIPGIP APELAPEL
jobcontrol
jobmonitor
info.provider
accoun-ting
BQSBQS
JWGEN
4
A solution : JWGENA solution : JWGEN
CREAM-CECREAM-CE CREAM-CECREAM-CE
Pado
vaPa
dova
CN
AF
CN
AF
BLAHBLAH GIPGIP APELAPEL BLAHBLAH GIPGIP APELAPEL
jobcontrol
jobmonitor
info.provider
accoun-ting
jobcontrol
jobmonitor
info.provider
accoun-ting
BQSBQS Grid EngineGrid Engine
JWGEN JWGEN
5
A solution : JWGENA solution : JWGEN
CREAM-CECREAM-CE
Pado
vaPa
dova
CN
AF
CN
AF
BLAHBLAH GIPGIP APELAPEL
jobcontrol
jobmonitor
info.provider
accoun-ting
Grid EngineGrid Engine
JWGEN
CREAM-CECREAM-CE
BLAHBLAH GIPGIP APELAPEL
jobcontrol
jobmonitor
info.provider
accoun-ting
BQSBQS
JWGEN
JDL
JSDL
options+
wrapper
7
OutputOutput : SGE options + job wrapper : SGE options + job wrapper
$ ./libexec/jwgen-core/generate_wrapper.sh /tmp/753627923.jsdl
#!/bin/bash
#$ -q admin
#$ -v SITE_NAME=Test
#$ -A "GRID=EGI SITE=Test TIER=tier3 VO=dteam ROLEVOMS=&2Fdteam&2FRole=NULL&2FCapability=NULL DN=&2FO=GRID-FR&2FC=FR&2FO=CNRS&2FOU=CC-LYON&2FCN=Sylvain&20Reynaud"
#$ -l os=sl5
#$ -N ccsyre_crm01_753627923
#$ -l s_cpu=00:01:40
#$ -l s_vmem=20M
#$ -l s_fsize=5G
#$ -l dcache
[ -r /afs/in2p3.fr/grid/profiles/glite/3.1.25-0/SL4_64/WN32/lcg-prologue.sh ] && . /afs/in2p3.fr/grid/profiles/glite/3.1.25-0/…
… etc …
8
Configuration fileConfiguration file
CREAM-CECREAM-CE
Pado
vaPa
dova
CN
AF
CN
AF
BLAHBLAH GIPGIP APELAPEL
jobcontrol
jobmonitor
info.provider
accoun-ting
Grid EngineGrid Engine
JWGEN
configfile
CREAM-CECREAM-CE
BLAHBLAH GIPGIP APELAPEL
jobcontrol
jobmonitor
info.provider
accoun-ting
BQSBQS
JWGEN
9
Configuration fileConfiguration file
Need to factorize information– thanks to CE profiles
configfile
10
Configuration fileConfiguration file
Need to factorize information– thanks to CE profiles
CC-IN2P3 siteCC-IN2P3 site
CC-IN2P3 Tier 1CC-IN2P3 Tier 1
CC-IN2P3 Tier 2CC-IN2P3 Tier 2 TestTest
EGIEGI LCGLCGprofile A profile B
LCG-CE 04LCG-CE 04CREAM
02
CREAM 02LCG-CE 08LCG-CE 08
CREAM VM-07
CREAM VM-07CREAM 05
CREAM 05
LCG-CE 11
LCG-CE 11
ROC-FRROC-FR
LCG-CE 02LCG-CE 02
LCG-CE 07LCG-CE 07
LCG-CE 01LCG-CE 01
LCG-CE 03LCG-CE 03
LCG-CE 05LCG-CE 05
LCG-CE 06LCG-CE 06
CREAM 03
CREAM 03
11
Configuration fileConfiguration file
Need to factorize information– thanks to CE profiles– thanks to operations on sets
Sets are used for– queues, groups, users, logical resources
Sets can be– created : ignore ancestor sets– subtracted : exclude some items– refined : without adding new items– extended : with adding new items
12
Query configuration fileQuery configuration file
$ ./sbin/jwgen-ce-query.sh -h
./sbin/jwgen-ce-query.sh: option requires an argument -- h
Usage: ./sbin/jwgen-ce-query.sh [-h <hostname>] [-q <queue>] [-g <group>] [-u <user>] [-x]
Where:
-h : hostname of the CE (default value: 'ccsyre.in2p3.fr')
-q : CE queue (default value: '_' = any value)
-g : unix group (default value: '_' = any value)
-u : unix user account (default value: '_default' = non-special users)
-x : print output as XML
13
Query configuration fileQuery configuration file
$ ./sbin/jwgen-ce-query.sh -h cclcgceli05.in2p3.fr -u atlas099
CEInstance 'cclcgceli05.in2p3.fr'
CEQueue 'long'
Group 'atlas'
User 'atlas099'
LogicalResourceSet = {dcache,xrootd}
OperatingSystem = sl5
ProfilesDirectory=/afs/in2p3.fr/grid/profiles/glite/3.2.2-0/SL5_64/WN64
Project = P_atlas_T2ana
ResourceCPUMax = max
ResourceFileSizeMax = max
ResourceMemoryMax = max
ResourceQueue = long
SubmissionPolicy = ACCEPT
14
Core package independent ofCore package independent of
grid middleware– based on JSDL– can be used with CREAM job control plug-in (BLAH) or Globus job
manager for example
LRMS– developed, tested and used with BQS– will now be used with Grid Engine
site policy– configuration keywords and templates are provided separately
15
ConclusionConclusion
JWGEN has proven to be useful with BQS, now with GE It can be reused with other grid middlewares, LRMS and
site policies
But…– configuration is of course more complex that the BLAH flat file– customization is not so easy to do (even if customizable files are
well identified)
We plan to propose JWGEN as an optional module for grid sites that need this kind of advanced features