k.harrison cern, 6th march 2003 ganga: gaudi/athena and grid alliance - aims and design - progress...
TRANSCRIPT
K.HarrisonCERN, 6th March 2003
GANGA: GAUDI/ATHENA AND GRID ALLIANCE
- Aims and design - Progress with low-level software- Progress with Graphical User Interface- Conclusions and next steps
6th March 2003 2
Aims
- The Indian goddess Ganga descended to Earth to flow as a river (English: Ganges) that carried lost souls to salvation- Ganga software is being developed jointly by ATLAS and LHCb to provide an interface for running Gaudi/Athena applications on the Grid Deal with all phases of a job life cycle: configuration, submission monitoring, error recovery, output collection, bookkeeping Carry jobs to the Grid underworld, and hopefully bring them back- Idea is that Ganga will have functionality analogous to a mail system, with jobs having a role similar to mails Make configuring a Gaudi/Athena job and running it on the Grid as easy as sending a mail
6th March 2003 3
Ganga: schematic representation
Gaudi/Athenaapplication
GangaG
UI/C
LI
JobOptionsAlgorithms
Collective&
ResourceGrid
Services
HistogramsMonitoringResults
6th March 2003 4
Design considerations
- Ganga should not reproduce what already exists, but should make use of, and complement, work from other projects, including AtCom, AthASK, DIAL and Grappa in ATLAS Should also follow, and contribute to, developments in Physicist Interface (PI) project of LCG - The design should be modular, and the different modules should be accessed via a thin interface layer implemented using a scripting language, with Python the current choice- Ganga should provide a set of tools that can be accessed from the command line (may be used in scripts), together with a local GUI and/or a web-based GUI that simplifies the use of these tools- Ganga should allow access to local resources as well as to the Grid
6th March 2003 5
Tentative Ganga architecture
Server
EDG UI
PYTHON SW BUS
XML RPC server
XML RPC module
GANGA Core Module
OS Module
Gaudi/Athena
GaudiPython PythonROOT
PYTHON SW BUS
GU
I
DB
Remote user
(client)
LAN/WAN
GRID
LRMS
Local Job DB
ProductionDB
BookkeepingDB
JobConfiguration
DB
6th March 2003 6
Low-level software
- In Ganga, components used to define a job are mapped to Python classes- The low-level software needs to provide representations of the most basic components, and must allow flexibility in manipulating them- Users will have access to low-level data and methods at the command line, but in most cases shouldn’t need them Common tasks will be simplified through higher-level functionality, available both at the command line and through a GUI- Low-level software presented here works, but is preliminary Details of component representations are still being discussed by development team
6th March 2003 7
Job components- In the preliminary implementation, the component breakdown is as follows: Job: a WorkFlow and a set of methods for submitting a script to a particular batch system For now, consider LSF Grid will be modelled as just another batch system WorkFlow: a series of WorkSteps, defining the sum of the actions to be performed when the job is run WorkStep: a set of sub-components providing all information necessary to run one instance of an executable WorkStep sub-components: Command, InputFile, OutputFile, CMTPackage (produces a library), CMTApplication (has an executable associated with it) Other sub-components to follow
6th March 2003 8
Defining and running jobs
- For now, consider case where experiment’s standard software is pre-installed at the site where the job runs (as for EDG), and user software has been configured (i.e. cmt requirements and job options are defined) Ganga should help with configuration in the future, but user will always have to do the work of developing his/her own analysis code- User software does not have to be pre-installed on remote site- Command sequences may be entered at Ganga prompt, or may be used in a Python script that imports the Ganga classes: from GangaKernel.GangaImport import
6th March 2003 9
Atlas example (Atlfast)
atlasSetup = GangaCommand(“source /afs/cern.ch/user/h/harrison/public/atlasSetup.sh”)atlfast = GangaCMTApplication(“TestRelease”, “TestRelease-00-00-15”,”athena.exe”, “run/AtlasfastOptions.txt”)atlfastOutput = GangaOutputFile(“atlfast.ntup”)workStep1 = GangaWorkStep([atlasSetup,atlfast,atlfastOutput])workFlow = GangaWorkFlow([workStep1])lsfJob = GangaLSFJob(“atlfastTest”,workFlow)lsfJob.build()lsfJob.run() At this level, there is less built-in intelligence than in ASK, but a lot of flexibility
6th March 2003 10
LHCb example (DaVinci)lhcbSetup = GangaCommand(“source /afs/cern.ch/lhcb/scripts/lhcbenv.sh”)daVinciSetup = GangaCommand(“source /afs/cern.ch/lhcb/scripts/ProjectEnv.sh DaVinci v7r2”)daVinci = GangaCMTApplication(“Phys/DaVinci”, “v7r2”,”DaVinci.exe”, “options/DaVinci.opts”)daVinciOutput = GangaOutputFile(“dvhistos.hbook”)workStep1 = GangaWorkStep([lhcbSetup,daVinciSetup,daVinci, daVinciOutput])workFlow = GangaWorkFlow([workStep1])lsfJob = GangaLSFJob(“daVinciTest”,workFlow)lsfJob.build()lsfJob.run() Command sequences for Atlas and LHCb are very similar
6th March 2003 11
Graphical User Interface
- GUI development has been undertaken by A.Soroko- Implementation is based on wxPython extension module- Current version is mainly illustrative Right buttons are there; full functionality will come later
6th March 2003 12
GUI: basic design
Embedded Python
interpreter
6th March 2003 13
GUI: job creation
Create a new job
6th March 2003 14
GUI: specifying WorkFlow
Give name and version of
the application
6th March 2003 15
GUI: defining job options
Job (application)
options downloaded
from DB
6th March 2003 16
GUI: defining batch system
Define batch
system
6th March 2003 17
GUI: defining resource requirements
Define necessary resources
6th March 2003 18
GUI: job submission
Submit the job (after
configuration)
6th March 2003 19
GUI: notification of job submission
Job has been submitted
successfully
6th March 2003 20
Conclusions and next steps
- Low-level tools that allow (pre-configured) jobs to be handled in a similar manner for Atlas and LHCb have been developed- Implementation of GUI is at an advanced stage- Jobs have been submitted successfully through Ganga to Cern LSF Ready to start experimenting with job submission to Grid- C.L.Tan is working to understand how AtCom and Ganga can best complement one another- C.E.Tull has started investigating possibilities for exploiting Grid Monitoring Architecture in Ganga- Ganga abstract has been accepted for presentation at CHEP03