The ALICE Distributed The ALICE Distributed Computing Computing

Federico CarminatiFederico CarminatiALICE workshop, Sibiu, Romania, ALICE workshop, Sibiu, Romania,

20/08/200820/08/2008

Sibiu 20/08/2008Sibiu 20/08/2008 22

Perspectives…Perspectives…

This workshop: exactly 30 days before the This workshop: exactly 30 days before the official start of LHC (2.6Mio seconds)official start of LHC (2.6Mio seconds)

10 September 200810 September 2008

Is the Grid Is the Grid readyready to register, process and to register, process and provide the ALICE physicists with a provide the ALICE physicists with a platform to efficiently analyze the first dataplatform to efficiently analyze the first data

33

The ALICE Grid is bornThe ALICE Grid is born

Working prototype in 2002Working prototype in 2002The Vision From the Very BeginningThe Vision From the Very Beginning

Single interface to distributed computing for Single interface to distributed computing for all ALICE physicistsall ALICE physicists

File catalogue, job submission and control, File catalogue, job submission and control, application software management, end user application software management, end user analysisanalysis

And this is….And this is….

AliEn – Alice EnvironmentAliEn – Alice Environment Sibiu 20/08/2008Sibiu 20/08/2008

44

Toddler years 2003-2004Toddler years 2003-2004

First MC productions…First MC productions…Full vertical Grid – interfaced down to local Full vertical Grid – interfaced down to local

batch system level and any type of local batch system level and any type of local storage the site provides (capability storage the site provides (capability retained and refined up to today)retained and refined up to today)

Few hundred CPUs at 13 sitesFew hundred CPUs at 13 sitesVery ambitious goals – validation of the Very ambitious goals – validation of the

entire ALICE Computing model entire ALICE Computing model

Sibiu 20/08/2008Sibiu 20/08/2008

55

Toddler years (2)Toddler years (2)First report of yearly ALICE data First report of yearly ALICE data

challenge: Sep. 2004challenge: Sep. 2004

Sibiu 20/08/2008Sibiu 20/08/2008

CPU work: 285 MSI-2K hours (one 2.8 GHz PC working for 35 years)

CR4 22.03.2004CR4 22.03.2004 C.W.FabjanC.W.Fabjan 66

At that time ALICE was…At that time ALICE was…

Services

77

Terrible twos (and a bit beyond)…2005-2006Terrible twos (and a bit beyond)…2005-2006

Increased sophistication of the central AliEn Increased sophistication of the central AliEn servicesservices

Full integration with the LCG servicesFull integration with the LCG services AliEn is moving toward ‘Top level Grid management’AliEn is moving toward ‘Top level Grid management’

Beginning of the story with storage (still ongoing)Beginning of the story with storage (still ongoing) Increase the number of participating sites and Increase the number of participating sites and

CPUs usedCPUs used And more data challenges…this time with a solid And more data challenges…this time with a solid

backup of the Computing TDRbackup of the Computing TDRPublished in June 2005Published in June 2005

Sibiu 20/08/2008Sibiu 20/08/2008

88

HP ProLiant DL580 AliEn central services

HP ProLiant DL360 AliEn to CASTOR (MSS) interface

HP ProLiant DL380 gShell API service

Disk SE log files storage 1TB SATA Disk server

AliEn services in 2005 AliEn services in 2005

Sibiu 20/08/2008Sibiu 20/08/2008

99

PDC results@end of 2005PDC results@end of 2005 Data Challenge IV with LCG SC3: distributed MC production works Data Challenge IV with LCG SC3: distributed MC production works

routinely routinely Monitoring: 25 sites operational, up to 1800 concurrent jobs, : 25 sites operational, up to 1800 concurrent jobs, all using all using

LCG resources, LCG services and ALICE services, thanks to our LCG resources, LCG services and ALICE services, thanks to our experience with AliEnexperience with AliEn

Running jobs (8 November)

Farm Min Avg Max

Sum 1160 1651 1771

CCIN2P3 134 210 231

CERN-L 268 286 304

CNAF 255 362 394

FZK 0 531 600

Houston 0 3 14

Münster 2 58 81

Prague 43 61 71

Sejong 2 2 2

Torino 33 41 43

1010

First records – Dec. 2005First records – Dec. 2005

2450 jobs

Sibiu 20/08/2008Sibiu 20/08/2008

1111

At that time ALICE was…At that time ALICE was…

1212

Beginning of the endless DCBeginning of the endless DCApril 2006 – decision to run in a quasi-April 2006 – decision to run in a quasi-

permanent modepermanent mode Integration of new sitesIntegration of new sites

Workload and storage Workload and storage Improving the robustness of central AliEn Improving the robustness of central AliEn

servicesservicesOperational experienceOperational experienceTest of AliRoot, new MC productionsTest of AliRoot, new MC productions

Sibiu 20/08/2008Sibiu 20/08/2008

PRODUCTION!!Deploy in prodn

PPSCertification

Tuesday 28/2/06gLite 3.0β exitscertification andenters PPS.

February April May JuneMarch

Friday 28/4/06gLite 3.0 exits PPSand enters production.

Wednesday 15/3/06gLite 3.0β availableto users in the PPS.

Deployment ofgLite 3.0β in PPS

Thursday 1/6/06SC4 starts!!

PPS Schedule for gLite 3.0PPS Schedule for gLite 3.0

Patches for bugscontinually passedto PPS.

YOU ARE HERE

gLite enters the Grid world

1414

April 2006 - xrootd is adoptedApril 2006 - xrootd is adopted xrootd with MSS backend is operational at CERNxrootd with MSS backend is operational at CERN Similar setup is being brought online at LyonSimilar setup is being brought online at Lyon For other sites – experts are preparing the software and For other sites – experts are preparing the software and

instructions for site experts how to install xrootd on the instructions for site experts how to install xrootd on the local storage serverslocal storage servers

Discussion of interoperation of various storage solutions Discussion of interoperation of various storage solutions (CASTOR2, DPM, dCache and xrootd ongoing(CASTOR2, DPM, dCache and xrootd ongoing

SRM standard is still being discussedSRM standard is still being discussed ALICE will use xrootd based storage and is actively ALICE will use xrootd based storage and is actively

pursuing its inclusion in the standard LCG packagepursuing its inclusion in the standard LCG package

Sibiu 20/08/2008Sibiu 20/08/2008

StatStatus of PDC’06us of PDC’06 1515

History of PDC’06History of PDC’06

Continuous running since Aplil 2006Continuous running since Aplil 2006 Test jobs, allowing to debug all site services and test the Test jobs, allowing to debug all site services and test the

stabilitystability From July – production and reconstruction of p+p MC eventsFrom July – production and reconstruction of p+p MC events

1616

Resources statistics PDC’06Resources statistics PDC’06

Resources contribution (normalized Si2K units): 50% Resources contribution (normalized Si2K units): 50% from T1s, 50% from T2sfrom T1s, 50% from T2s

6 T1s, 30 T2s

Sibiu 20/08/2008Sibiu 20/08/2008

1717

At that time ALICE was…At that time ALICE was…

Sibiu 20/08/2008Sibiu 20/08/2008

1818

Kindergarten 2007-todayKindergarten 2007-today Toward fully redundant central servicesToward fully redundant central services Integration of gLite transfer and storage Integration of gLite transfer and storage

solutionssolutions Automatic site and central services management Automatic site and central services management

tools ‘auto-pilot mode’tools ‘auto-pilot mode’ Automatic production toolsAutomatic production tools Integration of more sites, deployment of storageIntegration of more sites, deployment of storage MC and RAW data productionMC and RAW data production

Sibiu 20/08/2008Sibiu 20/08/2008

1919

Number of jobs evolutionNumber of jobs evolution

Sibiu 20/08/2008Sibiu 20/08/2008

Max. attained 10325

2020

Sites contributionSites contribution

50% resources contribution from T2s!

Sibiu 20/08/2008Sibiu 20/08/2008

2121

The ALICE Grid in numbersThe ALICE Grid in numbers73 participating sites73 participating sites

1 T0 (CERN/Switzerland)1 T0 (CERN/Switzerland)6 T1s (France, Germany, Italy, The 6 T1s (France, Germany, Italy, The

Netherlands, Nordic DataGrid Facility, UK)Netherlands, Nordic DataGrid Facility, UK)66 T2s spread over 4 continents66 T2s spread over 4 continents

As of today the ALICE share is some As of today the ALICE share is some 7000 (out of ~30000 total Grid) CPUs and 7000 (out of ~30000 total Grid) CPUs and 1.5 PB of distributed storage1.5 PB of distributed storage

In ½ year ~15K CPUs, x2 storageIn ½ year ~15K CPUs, x2 storage

Sibiu 20/08/2008Sibiu 20/08/2008

2222

The ALICE Grid MapThe ALICE Grid MapEurope

Asia

North America

Africa

Here is the live picture Sibiu 20/08/2008Sibiu 20/08/2008

2323

Control of the ALICE Grid Control of the ALICE Grid Fully redundantFully redundant

DB (MySQL) DB (MySQL) master-slave master-slave structure and structure and backupbackup

All central All central services run services run multiple instancesmultiple instances

Build servers for Build servers for I686, x86_64, I686, x86_64,

ia63, MacOS 32- ia63, MacOS 32- and 64- bitand 64- bit

Sibiu 20/08/2008Sibiu 20/08/2008

2424

ALICE Grid task list today ALICE Grid task list today Registration of data at MSS T0 and on the GRIDRegistration of data at MSS T0 and on the GRID Replication T0->T1Replication T0->T1 Quasi-online reconstructionQuasi-online reconstruction

Pass 1 at T0Pass 1 at T0 Pass 2 at T1sPass 2 at T1s

MC production and user analysisMC production and user analysis

Sibiu 20/08/2008Sibiu 20/08/2008

2525

RAW data from cosmic tests RAW data from cosmic tests

Sibiu 20/08/2008Sibiu 20/08/2008

230K RAW data files

2626

Replication of RAW Replication of RAW 60MB/sec rate (p+p data taking scenario)60MB/sec rate (p+p data taking scenario)Using gLite transfer tools (FTS), operated Using gLite transfer tools (FTS), operated

through AliEn FTDthrough AliEn FTD

Sibiu 20/08/2008Sibiu 20/08/2008

2727

Data and MC production - emphasis Data and MC production - emphasis Fast MC production for first physics Fast MC production for first physics

with various LHC startup scenarioswith various LHC startup scenariosAnd analysis for first publicationAnd analysis for first publicationAlready 2 cycles made Already 2 cycles made

Fast analysis of detector calibration Fast analysis of detector calibration datadataEssentially immediately after data taking Essentially immediately after data taking

(same day/night)(same day/night)Crucial for feedback to detector expertsCrucial for feedback to detector experts

2828

Production of RAW Production of RAW Major activity in the past 6 months, very Major activity in the past 6 months, very

successful despite rapidly changing conditionssuccessful despite rapidly changing conditions Both in the code and detector operationBoth in the code and detector operation In total some 120 TB of RAW passed through the In total some 120 TB of RAW passed through the

reconstructionreconstruction

Sibiu 20/08/2008Sibiu 20/08/2008

2929

MC production since 2006 MC production since 2006 Some 330Mio events with various physics contentSome 330Mio events with various physics content

Sibiu 20/08/2008Sibiu 20/08/2008

3030

Storage evolution Storage evolution Subject of intense development and deploymentSubject of intense development and deployment

For ALICE – xrootd as single access protocolFor ALICE – xrootd as single access protocol

4 de-facto storage solutions4 de-facto storage solutions Pure MSS – CASTOR2Pure MSS – CASTOR2 Hybrid disk+MSS – dCacheHybrid disk+MSS – dCache Disk – DPM, xrootdDisk – DPM, xrootd Critical for analysis is the disk-based storageCritical for analysis is the disk-based storage

Good overall deployment progress and stability of all Good overall deployment progress and stability of all storage typesstorage types

Secure access through the ALICE security envelope – Secure access through the ALICE security envelope – every operation is authorized through unique set of every operation is authorized through unique set of

encrypted keysencrypted keys

Sibiu 20/08/2008Sibiu 20/08/2008

3131

Storage deployment Storage deployment 38 storage endpoints over 21 sites, ~1.5 PB

Sibiu 20/08/2008Sibiu 20/08/2008

3232

Current operation principleCurrent operation principle

Central AliEn services

Site VO-boxSite VO-box Site VO-box

Site VO-boxSite VO-box

WMS (gLite/ARC/OSG/Local)

SM (dCache/DPM/CASTOR/xrootd)

Monitoring, Package management

The VO-box system (very controversial in the beginning)• Has been extensively tested• Allows for site services scaling• Is a simple isolation layer for the VO in case of troubles

Sibiu 20/08/2008Sibiu 20/08/2008

3333

Operation – central/site supportOperation – central/site support Central services support (2 FTEs equivalent)

There are no experts which do exclusively support – there are 6 highly-qualified experts doing development/support

Site services support - handled by ‘regional experts’ (one per country) in collaboration with local cluster administrators Extremely important part of the system In normal operation ~0.2FTEs/regions

Regular weekly discussions and active all-activities mailing lists

3434

User analysis activitiesUser analysis activitiesGenerally successful User jobs priorities are well mastered in the AliEn system Simple priority scheduling seem to work well, will be

expanded soon to ‘pay for what you use’ principles Storage remains a weak point

Only the lack of it – available amount does not allow full (as per compiuting model) replication of data to be analysed

As of today, 180 registered, 110 active users on the Grid Not counting the MC/RAW production and CAF

3535

User analysis statistics, one yearUser analysis statistics, one year

~1Mio user jobs completed, 2.7K/day

Chaotic analysis only!

3636

Today, ALICE is ready…Today, ALICE is ready…

Sibiu 20/08/2008Sibiu 20/08/2008

3737

Getting ready for the first school dayGetting ready for the first school day In 6 short yearsIn 6 short years the ALICE Grid has been transformed from a the ALICE Grid has been transformed from a

‘proof-of-concept’ into a fully operational computational platform for ‘proof-of-concept’ into a fully operational computational platform for ALICE offline work ALICE offline work

The ALICE computing model has been validated thoroughly through The ALICE computing model has been validated thoroughly through a series of data challenges with increasing complexity and scopea series of data challenges with increasing complexity and scope

The ALICE Grid today uses CPU and storage resources at some 73 The ALICE Grid today uses CPU and storage resources at some 73 computing centres, fully integrated into the local fabric and servicescomputing centres, fully integrated into the local fabric and services

The MC and RAW data production is a routine exercise, moving The MC and RAW data production is a routine exercise, moving rapidly into an automated mode of operationrapidly into an automated mode of operation

User analysis is a routine exercise too…User analysis is a routine exercise too… Services support on all levels is well understoodServices support on all levels is well understood

The ALICE Grid is ready for its first day in The ALICE Grid is ready for its first day in school, coming, as usual, at the beginning of school, coming, as usual, at the beginning of SeptemberSeptember

Sibiu 20/08/2008Sibiu 20/08/2008