ganga 4 basics - tutorial jakub t. moscicki arda/lhcb ganga tutorial, september 2006

Ganga 4 Basics - Tutorial Jakub T. MoscickiARDA/LHCb

Ganga Tutorial, September 2006

Ganga Tutorial, April 2005 2

Contents

• What is Ganga 7 things to know about Ganga from K.Harrison

• Setup and running for the first time

• Quick introduction to Python

• Simple Operations in Interactive Text Shell

• Copying jobs and submitting to other backends

• Scripting interface

• Shipping files

• Splitting


What is Ganga?

AtlasPROD

DIAL

DIRAC

LCG2

gLite

localhost

LSF

submit, kill

get outputupdate status

store & retrieve job definition

prepare, configure

Ganga4

JobJobJobJob

scripts

Gaudi

Athena

AtlasPROD

DIAL

DIRAC

LCG2

gLite

localhost

LSF


• Ganga Web Site: http://cern.ch/ganga

• Ganga Tutorial• Bug reports

• Ganga is based on Python 2.2 Python Website:

• http://www.python.org Excellent Python Tutorial:

• http://docs.python.org/tut/tut.html

• Other resources: IPython page:

• http://ipython.scipy.org/

Documentation


Setting up Ganga

• ATLAS tutorial instructions!

• • Note: we are using a beta version – new release by the end of

this month % ganga*** Welcome to Ganga ***Version: Ganga-4-2-0-beta7Documentation and support: http://cern.ch/gangaType help() or help('index') for online help.

• You will be asked few questions. Answer “yes” by pressing ENTER.• GRID certificate: type your password or press Control-D several

times.

• To quit press Control-D at the interactive prompt

In[1]: ^DDo you really want to exit ([y]/n)?%


Ganga Text Prompt

• Ganga is based on python 2.2 and has an enhanced python prompt (IPython): python programming/scripting

• var = 5• print var*10

easy access to the shell commands• !less ~/.gangarc # personal config file• !pwd

history <arrow up> TAB completion

• it works on keywords, variables, objects, try:– va<TAB>

many more features


• x = 2print x*3 # you may also skip print

if x==2: print "yes, x==2" # NOTE INITIAL SPACES!

alist = [1,2,3]for y in alist: print y

print len(alist) # built-in function lenhelp(len)

Python: Statements, Variables


• print range(11)help(range)

def square(v): return v*vprint square(x)

alist = [1,2,3]for i in range(len(alist)): alist[i] = square(alist[i])blist = [square(x) for x in alist]

Python: Functions


• import sysprint sys.argv

import osprint os.environ['HOME']from os import environfrom os import *import math, cmathprint math.cos(math.pi)v =(1+1j)print v*vmath.log(-1) # exception

Python: Modules


• # standard library & user classesimport randomr = random.Random()r.choice([1,2,3,4])r.randint(30,40)#built-in classes: for example stringss = "hello at the tutorial" # double quotess.split()s.count('a')s.upper()dir(s)if 'tut' in s: print 'tut is in ',s print '-'*20 # single quotes

Python: Classes and Objects


• # file objectsf = file('/etc/password')lines = f.readlines()print len(lines)# writing filesf2 = file('testfile','w')print >> f2, “this is a test”,20*'-'f2.close()!cat testfile

• NOTE: Ganga defines a special class File. • Do not confuse File and file!

Python: files


• Job Objects and Job Registry

[In:] jobs[In:] Job()[In:] jobs[In:] j = jobs[1][In:] print j.application[In:] print j.backend

# job registry: a list of all your jobs[In:] jobs[In:] jobs[1]

Ganga Scripting


Jobs

Job ( status = 'new' , inputdir = '/afs/cern.ch/user/k/kuba/gangadir/workspace/Local/2/input/' , name = '' , outputdir = '/afs/cern.ch/user/k/kuba/gangadir/workspace/Local/2/output/' , outputsandbox = [] , id = 2 , application = Executable ( exe = '/bin/hostname' , env = {} , args = [] ) , inputdata = None , inputsandbox = [ ] , backend = Local ( exitcode = None , id = None ) )


• [In:] j = Job()[In:] j.application.exe = '/bin/hostname'[In:] j.name = "MyTest"[In:] print j[In:] j.submit()

• # wait until job is completed and look# at the output directory[In:] print j.status[In:] print j.outputdir[In:] !ls $j.outputdir[In:] !cat $j.outputdir/stdout# the ! and $ syntax is IPython – you # cannot do it in normal python# it is useful for interactive work# instead of mouse copy-paste

Basic Job Submission


• # once a job is submitted you cannot modify# it. if you want to submit a new job you# should create a new job object[In:] j2 = j.copy()[In:] j2.backend = LSF()[In:] j2.submit()

•# if you have GRID certificate you can try[In:] j3 = j.copy()[In:] j3.backend = LCG()[In:] j3.submit()# print jobs to see all your jobs

More submission


• # see predefined plugins• # (only some combinations supported)• list_plugins()• help('index')• # submission in a loopfor i in range(5): j = Job() j.application.exe="/bin/echo" j.application.args=["hello ",str(i)]) print j.id, j.inputdir print j.submit()

More ideas


• # For frequently used jobs – use templates!t = JobTemplate(j)print templatesj = Job(t) # Create a new job from t

• # Other useful methodsj.kill()j.remove() # Removes the output as well!for j in jobs: j.kill()jobs.clean() # Faster! Deletes ALL jobs.

More Ideas (2)


• export(j,'myjob')• # press ^D to quit ganga• cat myjob• ganga submit myjob• ganga query

• *restart ganga*• load('myjob')

Scripting possibilities


• j = Job()• j.application.exe=File('spy')• j.outputsandbox = ['*.txt']• j.submit()

Do not confuse File with file!

Shipping files


• Prime number crunching: cryptography (also Grid certificates) you break the key by factoring large integers into prime numbers

• Application setup: we have 15 tables with 1M primes each

• the files are downloaded from a web server we have a “prime_factor.py” script which scans the table and returns the factors it finds:

• [(p1,n1),(p2,n2),...,(pk,nk)]• so that product of pow(pi,ni) = the number

Splitting Demo


• Job 1: find all primes factors of 4643871513989286 among the first

million primesj = Job()j.application.exe=File('prime_factor.py')j.application.args =['4643871513989286','http://cern.ch/diane/download/primes/primes1.zip']j.outputsandbox=['*.dat']j.submit()

cat $j.outputdir/*.dat

execfile('primes.py') # load helper functions for primes

check_prime_job(j)

Splitting Demo


• Job 2: find more factors using 5 tablesj = j.copy()split_prime_job(j,5)

• Exercise: find all primes of: 118020903911855744138963610

• Notes: you cannot not crack real codes this way ;-) instead of the Web server one could/should use a Grid SE

Splitting Demo


• Try: ganga -h to see options

• Have a look also in the config file: ~/.gangarc

Hints and Pitfalls


• Questions?

• Practical exercises

Questions?

ganga 4 basics - tutorial jakub t. moscicki arda/lhcb ganga tutorial, september 2006

Documents