gangatasks automating of job submission wit athena/athenamc in ganga hepcg workshop, siegen,...

GangaTasksGangaTasksAutomating of job Automating of job submission wit submission wit Athena/AthenaMC in GangaAthena/AthenaMC in Ganga

hepcg workshop, siegen, 13.12.2007hepcg workshop, siegen, 13.12.2007

johannes ebke, johannes elmsheuser, johannes ebke, johannes elmsheuser, tariq tariq mahmoudmahmoud

tariq mahmoud hepcg workshop, siegen, 13122007

automating of job submission in Athena/AthenaMC

outline

I. introduction, motivation

II. concept of tasksIII. some illustration IV. conclusion


I. motivation, mc

simul:5

recon:10

1

1 3 5 97

2 4 6 108

1

evgen:50

7 9


3 5

3

6

3 5

63

3

6

5

6

3

3

3

need automate job

(re)submission


I. motivation, mc


1 3 5 972 4 6 108

1

1 9

7

7

7 1 3 5 972 4 6 108

1

1 7 9

36

3 5

633

5

56

566

6

1 3 5 972 4 6 108

1

1 7

36

3 5

63

1 3 5 972 4 6 108

1

1 7 9

3 5

3 5

53

1 3 5 972 4 6 108

1

1 7 9

36

3 5

63756

1 3 5 972 4 6 108

1

9

7

7

3

5

55

1 3 5 972 4 6 108

1

1 7 9

36

3 5

63


I. motivation, mc

1 3 5 972 4 6 108

1

1 7 9

36

3 5

63756

1 3 5 972 4 6 108

1

1 9

7

7

3

5

55

1 3 5 972 4 6 108

1

1 7 9

36

3 5

63

1 3 5 972 4 6 108

1

1 9

7

7

3

5

5

6 7

4

4

4

4need automate job

(re)submission

need job chaining

(interconnection)



I. motivation, analysis

complete dataset


A,B,C: f1,f2,f3, …,fn

A,B,C f1,f2,f3A,B,C f4,f5,f6

A,B,C f7,f8,f9A,B,C f7,f8,f9

A,B,C f4,f5,f6

A,B,C f7,f8,f9A,B,C fn-2,fn-1,fn

incomplete datasetA: f1,f2,f3B: f1,f5,f6,f7,f8,f9C: f1,f2,f7,f8,f9,f10

A,C f1,f2B f5,f6

A f3

B,C f7,f8,f9C f10

automating of job submission in Athena/AthenaMC II. concept of tasks

taskfew parameters: task parameters: float, name, number of events etc.

process parameters: datasets, process name, geometry etc.

input/output specifications Athena version

AbstractJobgenerated automatically: name, task, status, necessary, prerequisites, run limit, number of attempts and attempt jobs, …

AbstractJobAbstractJob

jobjob

job Ganga jobs


automating of job submission in Athena/AthenaMC II. concept of tasks: GangaTasks

MCTask

MCJobMCJobMCJob

AnaTask

AnaJob

Task

AnaJobAnaJob

repository/tools.data

TaskList(GangaObject)

GPI.tasks


automating of job submission in Athena/AthenaMC III. illustration, resubmission


automate job (re)submission

need job chaining

(interconnection)


III. illustration: chaining task stepsmc.total_events = 200;

mc.events_per_job={"evgen": 50, "simul": 5, "recon":10}

1 3 5 7 9

1 3 5 7 9

1 3 5 7 9

1 3 5 7 9

1 3 5 972 4 6 108

1 3 5 972 4 6 108

1 3 5 972 4 6 108

1 3 5 972 4 6 108

2 3 41 evgen:i-0

simul:i-j

recon:i-k


mc.total_events = 200 mc.events_per_job= {"evgen": 50, "simul": 5, "recon":10} mc.float=10

AbstractJob.prerequisites()

tariq mahmoud atlas sw week, cern, 24102007

automating of job submission in Athena/AthenaMC III. illustration: interconnections,

prerequisites

automate job (re)submission

job chaining (interconnection)

the user sets few parameters … and gets

RESULTS


IV. conclusion

GangaTask package: automates job management (small scale production system)works with Ganga Athena and Ganga AthenaMC

(available from Ganga443 onwards) works at cernwiki with tutorial: https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTasksapplicable to any job submission-software on the grid

To do: 1. implementing site-test-code 2. making use of GangaRobot3. act on user suggestionstariq mahmoud hepcg workshop, siegen, 13122007

Configure once – run anywhere

https://twiki.cern.ch/twiki/bin/view/Atlas/GangaTasks

gangatasks automating of job submission wit athena/athenamc in ganga hepcg workshop, siegen,...

Documents