[email protected] lcg accounting john gordon grid deployment board 13 th january 2004
TRANSCRIPT
Overview
Aim is to gather accounting information at the job level from each site and store in a central database so that arbitrary queries can be done against it.
Four parts Schema Gathering Sending Querying
Schema
Defined our own
Then discovered GGF Usage Schema http://www.psc.edu/~lfm/Grid/UR-WG/
Accounting Schema
LocalGroup - the group that the job ran as on the system LocalJobID - the ID of the job on the system on which it ran ** LCGJobID - the job ID assigned by the RB (null for jobs submitted locally or via globus) ** ExecutingSite - the site at which the job ran ** ExecutingCE - the queue from which the job ran SubmittingRB - the RB used to submit the job LocalUserID - the ID of the user on the system on which the job ran ** LCGUserDN - the DN of the user within LCG ** LCGUserVO - the virtual organisation of this user for this job ** LocalStartTime - the local wallclock date/time at which the job commenced execution LocalStopTime - the local wallclock date/time at which the job ceased execution UTCStartTime - the wallclock date/time at which the job commenced execution in UTC (GMT) ** UTCStopTime - the wallclock date/time at which the job ceased execution in UTC ** ElapsedTime - StopTime-StartTime (calculated seconds) BaseCpuTime - the actual CPU time recorded by the system on which the job ran (seconds) BaseCpuPower - the power of the system on which the job ran in SpecInts (needs more precise definition) SpecIntSecs - the computing power consumed by the job in SpecInt-Seconds (calculated from previous two
fields) ** DiskSpace - the maximum aggregated amount of local disk space used by the job (Mb) DiskIO - the aggregated volume of disk data read/written by the job (Mb)
Gathering
PBS doesn’t produce job accounts, only logs events Four solutions
RAL processes event logs and populkates db NIKHEF does similar pbsacct in SourceForge Job info available in pbs prolog. Do it direct.
Still considering Assume unix group = vo name Several solutions to mapping unix username to DN
Not yet chosen Initial version probably VO only
LSF will be easier as it has job accounts Site ultimately responsible.
They can filter out certain info (eg non-LHC VOs) Or remap groups to Vos Or replace the whole thing by calls to their own internal
database.
Sending
Send schema info over grid to central database Considered
Spitfire Develop our own web service R-GMA
• R-GMA Archiver is a web-service which takes a row of information and inserts it in a persistent database
Chose R-GMA Stream producer and general archiver Retain records in local Mon to provide resilience against network
failure. Sites could also run archivers to keep a copy of their data local.
Querying
Simple SQL query to start Canned queries with plotting General queries later
Progress
Evaluated different components separately Now integrating
Our Schema PBS prolog Calling R-GMA stream producer With general archive consuming
Prototype by 23rd.