atlas grid

ATLAS Grid

Kaushik DeKaushik DeUniversity of Texas at ArlingtonUniversity of Texas at Arlington

HiPCAT THEGrid Workshop, UTAHiPCAT THEGrid Workshop, UTA

July 8, 2004July 8, 2004

July 8, 2004July 8, 2004Kaushik De, HipCAT THEGrid WorkshopKaushik De, HipCAT THEGrid Workshop 2

Goals and Status

Grids - need no introduction, after previous talksGrids - need no introduction, after previous talks

ATLAS - experiment at the LHC, being installed, over 2000 ATLAS - experiment at the LHC, being installed, over 2000 physicists, 1-2 petabytes of data per year, starting 2007-08physicists, 1-2 petabytes of data per year, starting 2007-08

ATLAS needs the GridATLAS needs the Grid Deployment and testing well under way Series of data challenges (using simulated data) to build

computing/grid capability slowly up to full capacity before 2007 UTA has played a leading role in ATLAS grids since 2002

Focus of this talk - how to get involved in grid projects and Focus of this talk - how to get involved in grid projects and make contributions as physicists and computer scientistsmake contributions as physicists and computer scientists

Need huge amount of application specific software to be Need huge amount of application specific software to be developed, tested, optimized, deployed...developed, tested, optimized, deployed...


ATLAS Data Challenges

Original Goals (Nov 15, 2001)Original Goals (Nov 15, 2001) Test computing model, its software, its data model, and to ensure

the correctness of the technical choices to be made Data Challenges should be executed at the prototype Tier centres Data challenges will be used as input for a Computing Technical

Design Report due by the end of 2003 (?) and for preparing a MoU

Current StatusCurrent Status Goals are evolving as we gain experience Computing TDR ~end of 2004 DC’s are ~yearly sequence of increasing scale & complexity DC0 and DC1 (completed) DC2 (2004), DC3, and DC4 planned Grid deployment and testing is major part of DC’s


ATLAS DC1: July 2002-April 2003

Goals : Produce the data needed for the HLT TDR Get as many ATLAS institutes involved as possible

Worldwide collaborative activityParticipation : 56 Institutes

Australia Australia

AustriaAustria

CanadaCanada

CERN CERN

ChinaChina

Czech RepublicCzech Republic

Denmark Denmark **

France France

Germany Germany

GreeceGreece

IsraelIsrael

ItalyItaly

JapanJapan

Norway Norway **

PolandPoland

RussiaRussia

SpainSpain

Sweden Sweden **

TaiwanTaiwan

UKUK

USA USA ** * using Grid


(6300)(6300)(84)(84)2.5x102.5x1066ReconstructionReconstruction

+ Lvl1/2+ Lvl1/2

14141650165022224x104x1066Lumi02 Pile-upLumi02 Pile-up

22960096001251253x103x1077SimulationSimulation

Single part.Single part.

6060

2121

2323

TBTB

Volume of Volume of datadata

51000 (+6300)51000 (+6300)

37503750

60006000

3000030000

CPU-daysCPU-days

(400 SI2k)(400 SI2k)

kSI2k.monthskSI2k.months

4x104x1066

2.8x102.8x1066

101077

No. of No. of eventsevents

CPU TimeCPU TimeProcessProcess

690 (+84)690 (+84)TotalTotal

5050ReconstructionReconstruction

7878Lumi10 Pile-upLumi10 Pile-up

415415SimulationSimulation

Physics evt.Physics evt.

DC1 Statistics (G. Poulard, July 2003)


U.S. DC1 Data Production

Year long process, Summer 2002-2003Year long process, Summer 2002-2003

Played 2nd largest role in ATLAS DC1Played 2nd largest role in ATLAS DC1

Exercised both farm and grid based productionExercised both farm and grid based production

10 U.S. sites participating10 U.S. sites participating Tier 1: BNL, Tier 2 prototypes: BU, IU/UC, Grid Testbed sites: ANL,

LBNL, UM, OU, SMU, UTA (UNM & UTPA will join for DC2)

Generated Generated ~2 million~2 million fully simulated, piled-up and fully simulated, piled-up and reconstructed eventsreconstructed events

U.S. was largest grid-based DC1 data producer in ATLASU.S. was largest grid-based DC1 data producer in ATLAS

Data used for HLT TDR, Athens physics workshop, Data used for HLT TDR, Athens physics workshop, reconstruction software tests...reconstruction software tests...


U.S. ATLAS Grid Testbed

BNL - U.S. Tier 1, 2000 nodes, 5% for BNL - U.S. Tier 1, 2000 nodes, 5% for ATLAS, 10 TB, HPSS through MagdaATLAS, 10 TB, HPSS through Magda

LBNL - pdsf cluster, 400 nodes, 5% for LBNL - pdsf cluster, 400 nodes, 5% for ATLAS (more if idle ~10-15% used), 1TBATLAS (more if idle ~10-15% used), 1TB

Boston U. - prototype Tier 2, 64 nodesBoston U. - prototype Tier 2, 64 nodes

Indiana U. - prototype Tier 2, 64 nodesIndiana U. - prototype Tier 2, 64 nodes

UT Arlington - 200 cpu’s, 70 TBUT Arlington - 200 cpu’s, 70 TB

Oklahoma U. - OSCER facilityOklahoma U. - OSCER facility

U. Michigan - test nodesU. Michigan - test nodes

ANL - test nodes, JAZZ clusterANL - test nodes, JAZZ cluster

SMU - 6 production nodesSMU - 6 production nodes

UNM - Los Lobos clusterUNM - Los Lobos cluster

U. Chicago - test nodesU. Chicago - test nodes


U.S. Production Summary

Number of Number of CPU hours CPU hours CPU hours

Files in Magda Events Simulation Pile-up Reconstruction

25 Gev di-jets 41k 1M ~60k 56k 60k+

50 Gev di-jets 10k 250k ~20k 22k 20k+

Single particles 24k 200k 17k 6k

Higgs sample 11k 50k 8k 2k

SUSY sample 7k 50k 13k 2k

minbias sample 7k ? ?

* Total ~30 CPU YEARS delivered to DC1 from U.S.* Total produced file size ~20TB on HPSS tape system, ~10TB on disk.* Black - majority grid produced, Blue - majority farm produced

Exercised both farm and grid based productionExercised both farm and grid based production Valuable large scale grid based production experienceValuable large scale grid based production experience


DC1 Production Systems

Local batch systems - bulk of productionLocal batch systems - bulk of production

GRAT - grid scripts, generated ~50k files produced in U.S.GRAT - grid scripts, generated ~50k files produced in U.S.

NorduGrid - grid system, ~10k files in Nordic countriesNorduGrid - grid system, ~10k files in Nordic countries

AtCom - GUI, ~10k files at CERN (mostly batch)AtCom - GUI, ~10k files at CERN (mostly batch)

GCE - Chimera based, ~1k files producedGCE - Chimera based, ~1k files produced

GRAPPA - interactive GUI for individual userGRAPPA - interactive GUI for individual user

EDG/LCG - test files onlyEDG/LCG - test files only

+ systems I forgot…+ systems I forgot…

Great diversity - but need convergence soon!Great diversity - but need convergence soon!


GRAT Software

GRid Applications ToolkitGRid Applications Toolkit developed by KD, Horst Severini, Mark Sosebee, and students

Based on Globus, Magda & MySQLBased on Globus, Magda & MySQL

Shell & Python scripts, modular designShell & Python scripts, modular design

Rapid development platformRapid development platform Quickly develop packages as needed by DC

Physics simulation (GEANT/ATLSIM) Pileup production & data management Reconstruction

Test grid middleware, test grid performanceTest grid middleware, test grid performance

Modules can be easily enhanced or replaced, e.g. EDG Modules can be easily enhanced or replaced, e.g. EDG resource broker, Chimera, replica catalogue… (in progress)resource broker, Chimera, replica catalogue… (in progress)


DC1 Production Experience

Grid paradigm works, using GlobusGrid paradigm works, using Globus Opportunistic use of existing resources, run anywhere, from

anywhere, by anyone...

Successfully exercised grid middleware with increasingly Successfully exercised grid middleware with increasingly complex taskscomplex tasks Simulation: create physics data from pre-defined parameters and

input files, CPU intensive Pile-up: mix ~2500 min-bias data files into physics simulation files,

data intensive Reconstruction: data intensive, multiple passes Data tracking: multiple steps, one -> many -> many more mappings


DC2: goals

The goal includes:The goal includes: Full use of Geant4; POOL; LCG applications Pile-up and digitization in Athena Deployment of the complete Event Data Model and the Detector Description Simulation of full ATLAS and 2004 combined Testbeam Test the calibration and alignment procedures Use widely the GRID middleware and tools Large scale physics analysis Computing model studies (document end 2004) Run as much as possible of the production on Grids Demonstrate use of multiple grids

(slides are from Gilbert Poulard’s talk couple of weeks ago in ATLAS)


Task Flow for DC2 data

HitsMCTruth

Digits(RDO)

MCTruth

BytestreamRaw

Digits

ESD

ESD

Geant4

Reconstruction

Reconstruction

Pile-up

BytestreamRaw

Digits

BytestreamRaw

Digits

HitsMCTruth

Digits(RDO)

MCTruth

Physicsevents

EventsHepMC

EventsHepMC

HitsMCTruth

Digits(RDO)

MCTruthGeant4

Geant4

Digitization

Digits(RDO)

MCTruth

BytestreamRaw

Digits

BytestreamRaw

Digits

BytestreamRaw

DigitsEventsHepMC

HitsMCTruth

Geant4Pile-up

Digitization

Mixing

Mixing Reconstruction ESD

Pyt

hia

Event generation

Detector Simulation

Digitization (Pile-up)

ReconstructionEventMixingByte stream

EventsHepMC

Min. biasEvents

Piled-upevents Mixed events

Mixed eventsWith

Pile-up

~5 TB 20 TB 30 TB20 TB 5 TB

TBVolume of datafor 107 events

Persistency:Athena-POOL


DC2 operation

Consider DC2 as a three-part operation:Consider DC2 as a three-part operation: part I: production of simulated data (June-July 2004)

needs Geant4, digitization and pile-up in Athena, POOL persistency “minimal” reconstruction just to validate simulation suite will run “preferably” on “Grid”

part II: test of Tier-0 operation (August 2004) needs full reconstruction software following RTF report design, definition of AODs and

TAGs (calibration/alignment and) reconstruction will run on Tier-0 prototype as if data were

coming from the online system (at 10% of the rate) Input is ByteStream (Raw Data) output (ESD+AOD) will be distributed to Tier-1s in real time for analysis

part III: test of distributed analysis on the Grid (Sept.-Oct. 2004) access to event and non-event data from anywhere in the world both in organized and

chaotic ways

in parallel: run distributed reconstruction on simulated data (from RODs)


DC2 resources (based on release 8.0.5)

~30~3077353560060011101077Pile-up (Pile-up (**))

Digitization RDODigitization RDO

~65~65

55

10 10

~50~50

00

1616

1616

TBTB

OffOff

sitesite

PhasePhase

IIII(mid-(mid-

AugustAugust))

Phase Phase II

(June(June

-July)-July)

005560060022101077ReconstructionReconstruction

Tier-1Tier-1

55556006000.50.5101077ReconstructionReconstruction

Tier-0Tier-0

~45~45110110101077TotalTotal

~40~40~100~1002800280011101077Total Phase ITotal Phase I

11

11

11

monthsmonths

Time Time durationduration

2525

2020

2020

TBTB

Volume Volume of dataof data

2525

44

44

TBTB

AtAt

CERNCERN

kSI2kkSI2k

101077

101077

101077

No. of No. of eventsevents

CPU CPU powerpower

ProcessProcess

(small)(small)Event mixing & Event mixing & Byte-streamByte-stream

~100~100RDO (Hits)RDO (Hits)

20002000**SimulationSimulation


New Production System for DC2

GoalsGoals Automated data production

system for all ATLAS facilities Common database for all

production - Oracle currently Common supervisor run by all

facilities/managers - Windmill Common data management

system - Don Quichote Executors developed by

middleware experts (Capone, LCG, NorduGrid, batch systems, CanadaGrid...)

Final verification of data done by supervisor


ATLAS Production system

LCG NG Grid3 LSF

LCGexe

LCGexe

NGexe

G3exe

LSFexe

super super super super super

prodDBdms

RLS RLS RLS

jabber jabber soap soap jabber

Don Quijote

Windmill

Lexor

AMI

CaponeDulcinea


ATLAS Production System

ComponentsComponents Supervisor: Windmill (Grid3) Executors:

Capone (Grid3) Dulcinea (NG) Lexor (LCG) “Legacy systems” (FZK; Lyon)

Data Management System (DMS): DonQuijote (CERN) Bookkeeping: AMI (LPSC)


Supervisor -Executors

Windmill

numJobsWantedexecuteJobsgetExecutorDatagetStatusfixJobkillJob

Jabber communicationpathway executors

Don Quijote(file catalog)

Prod DB(jobs database) execution

sites(grid)

1. lexor2. dulcinea3. capone4. legacy

supervisors

execution sites(grid)


Windmill - Supervisor

Supervisor development/U.S. DC production teamSupervisor development/U.S. DC production team UTA: Kaushik De, Mark Sosebee, Nurcan Ozturk + students BNL: Wensheng Deng, Rich Baker OU: Horst Severini ANL: Ed May

Windmill web pageWindmill web page http://www-hep.uta.edu/windmill

Windmill statusWindmill status DC2 started June 24th, 2004 Windmill is being used by all three grids in ATLAS thousands of jobs worldwide every day exciting and busy times - bringing interoperability to grids


Windmill Messaging

supervisoragent

executoragent

XML switch(JabberServer)

XMPP(XML)

XMPP(XML)

Webserver

SOAP

All messaging is XML based All messaging is XML based

Agents communicate using Jabber (open chat) protocolAgents communicate using Jabber (open chat) protocol

Agents have same command line interface - GUI in futureAgents have same command line interface - GUI in future

Agents & web server can run at same or different locationsAgents & web server can run at same or different locations

Executor accesses grid directly and/or thru web servicesExecutor accesses grid directly and/or thru web services


Intelligent Agents

Supervisor/executor are intelligent communication agentsSupervisor/executor are intelligent communication agents uses Jabber open source instant messaging framework Jabber server routes XMPP messages - acts as XML data switch reliable p2p asynchronous message delivery through firewalls built in support for dynamic ‘directory’, ‘discovery’, ‘presence’ extensible - we can add monitoring, debugging agents easily provides ‘chat’ capability for free - collaboration among operators Jabber grid proxy under development (LBNL - Agarwal)

JabberServer

Jabber Clients

Jabber Clients

XMPP


Capone Executor

Various executors are being developedVarious executors are being developed Capone - U.S. VDT executor by U. of Chicago and Argonne Lexor - LCG executor mostly by Italian groups NorduGrid, batch (Munich), Canadian, Australian(?)

Capone is based on GCE (Grid Computing Environment)Capone is based on GCE (Grid Computing Environment) (VDT Client/Server, Chimera, Pegasus, Condor, Globus)

Status:Status: Python module Process “thread” for each job Archive of managed jobs Job management Grid monitoring

Aware of key parameters (e.g. available CPUs, jobs running)


Capone Architecture

Message interfaceMessage interface Web Service Jabber

Translation levelTranslation level Windmill

CPE (Capone Process Engine)CPE (Capone Process Engine)

Processes Processes Grid Stub DonQuixote

from Marco Mambellifrom Marco Mambelli

Message protocols

Translation

Web Service

CPE

Jabber

Windmill ADA

Stu

b

Grid

Don

Qu

ixote


Don Quijote

Data Management for the ATLAS Automatic Production SystemData Management for the ATLAS Automatic Production System

Allow transparent registration and movement of replicas between all Allow transparent registration and movement of replicas between all Grid “flavors” used by ATLAS (across Grids)Grid “flavors” used by ATLAS (across Grids) Grid3; Nordugrid; LCG

(support for legacy systems might be introduced soon)

Avoid creating yet another catalogAvoid creating yet another catalog which Grid middleware wouldn't recognize (e.g Resource Brokers) use existing “native” catalogs and data management tools provide a unified interface

Accessible as a serviceAccessible as a service lightweight clients


Monitoring & Accounting

At a (very) early stage in DC2At a (very) early stage in DC2 Needs more discussion within ATLAS

Metrics being defined Development of a coherent approach

Current efforts: Job monitoring “around” the production database

Publish on the web, in real time, relevant data concerning the running of DC-2 and event production

SQL queries are submitted to the Prod DB hosted at CERN Result is HTML formatted and web published A first basic tool is already available as a prototype

On LCG: effort to verify the status of the GridØ two main tasks: site monitoring and job monitoringØ based on GridICE, a tool deeply integrated with the current

production Grid middleware On Grid3: MonaLisa On NG: NG monitoring


DC2 MonitoringDC2 Status

http://www.http://www.nordugridnordugrid.org/applications/.org/applications/prodsysprodsys/index.html/index.html


DC2 production (Phase 1)

Will be done as much as possible on GridWill be done as much as possible on Grid We are ready to use the 3 grid flavours

LCG-2, Grid3+ and NorduGrid All 3 look “stable” (adiabatic evolution)

Newcomers: Interface LCG to Grid Canada

UVic, NRC and Alberta accept LCG jobs via TRIUMF interface CE

Keep the possibility to use “legacy” batch systems

Data will be stored on Tier1sData will be stored on Tier1s

Will use several “catalogs”; DQ will take care of themWill use several “catalogs”; DQ will take care of them

Current planCurrent plan ~20% on Grid3 ~20% on NorduGrid ~60% on LCG To be adjusted based on experience


Current Grid3 Status (3/1/04)

(http://www.ivdgl.org/grid2003)

• 28 sites, multi-VO• shared resources• ~2000 CPUs• dynamic – roll in/out


NorduGrid middleware is deployed in:NorduGrid middleware is deployed in: Sweden (15 sites) Denmark (10 sites) Norway (3 sites) Finland (3 sites) Slovakia (1 site) Estonia (1 site)

Sites to join before/during DC2 (preliminary): Sites to join before/during DC2 (preliminary): Norway (1-2 sites) Russia (1-2 sites) Estonia (1-2 sites) Sweden (1-2 sites) Finland (1 site) Germany (1 site)

Many of the resources will be available for Many of the resources will be available for ATLAS DC2 via the NorduGrid middlewareATLAS DC2 via the NorduGrid middleware

Nordic countries will coordinate their shares For others, ATLAS representatives will

negotiate the usage

NorduGrid Resources: details


ROMA1

• 22 Countries• 58 Sites (45 Europe, 2 US, 5 Canada, 5 Asia, 1 HP)

• Coming: New Zealand, China, other HP (Brazil, Singapore)

• 3800 cpu

Sites in LCG-2: 4 June 2004


After DC2: “continuous production”

We have requests forWe have requests for Single particles simulation (a lot)!

Already as part of DC2 production (calibrations) To be defined

The detector geometry (which layout?) The luminosity if pile-up is required Others? (eg. Cavern background)

Physics samples for the Physics workshop studies (June 2005) DC2 uses ATLAS “Final Layout” It is intended to move to “Initial Layout” Assuming that the geometry description is ready by beginning of August we can

foresee an intensive MC production starting ~mid-September Initial thoughts:

~ 50 Million Physics events; that means ~10 Million events per month from end-September to March 2005

Production could be done either by the production team or by the Physics groups

The production system should be able to support both The Distributed Analysis system will provide an interface for Job submission


Conclusion

Data Challenges are important for ATLAS software and Data Challenges are important for ATLAS software and computing infrastructure readinesscomputing infrastructure readiness

Grids will be the default testbed for DC2Grids will be the default testbed for DC2

U.S. playing a major role in DC2 planning & productionU.S. playing a major role in DC2 planning & production

12 U.S. sites ready to participate in DC212 U.S. sites ready to participate in DC2

Major U.S. role in production software developmentMajor U.S. role in production software development

New grid production system, led by UTA, being adopted New grid production system, led by UTA, being adopted worldwide for grid production in ATLASworldwide for grid production in ATLAS

Physics analysis will be emphasis of DC2 - new experiencePhysics analysis will be emphasis of DC2 - new experience

Stay tunedStay tuned

atlas grid

Documents

hipcat thegrid workshopu

hipcat thegrid workshopgoals

atlas grid testbedbnl

grid deployment

grid projects

data model

simulated data

petabytes of data