the worldwide lhc computing grid

The Worldwide LHC Computing Grid

Processing the Data from the World’s Largest Scientific

Machine---

Jamie Shiers, CERN, Geneva, Switzerland

The Worldwide LHC Computing Grid - [email protected] - CCP 2006 - Gyeongju, Republic of Korea

Abstract

• The world's largest scientific machine will enter production about one year from the time of this conference

• In order to exploit its scientific potential, computational resources way beyond those needed for previous accelerators are required

• Based on these requirements, a distributed solution based on Grid technologies has been developed

• This talk describes the overall requirements that come from the

Computing Models of the experiments, the state of deployment of the production services, on-going validation of these services as well as the offline infrastructure of the experiments and finally the remaining steps that need to be achieved in the remaining months before the deluge of data arrives.


Overview

• Brief Introduction to CERN & LHC

• Data Processing requirements

• The Worldwide LHC Computing Grid

• Status and Outlook

LHC Overview

The Large Hadron ColliderProton-proton collider using an existing tunnel 27km in circumference, ~100m underground

Lies beneath French / Swiss border near Geneva

September 2006 The Worldwide LHC Computing Grid - [email protected] - CCP 2006 - Gyeongju, Republic of Korea

CERN – European Organization for Nuclear Research


The LHC Machine

HEPiX Rome 05apr06

LCG

[email protected]


8

CMS

Data Rates:• 1PB/s from detector• 100MB/s – 1.5GB/s to ‘disk’• 5-10PB growth / year• ~3GB/s per PB of data

Data Processing:• 100,000 of today’s

fastest PCs

Level 1

Level 2

40 MHz

40 MHz (1000 TB/sec)

(1000 TB/sec)

Level 3

75 KHz 75 KHz (75 GB/sec)

(75 GB/sec)5 KHz5 KHz (5 GB/sec)

(5 GB/sec)100 Hz 100 Hz (100

(100 MB/sec)MB/sec)

Data Recording &

Data Recording &

Offline Analysis

Offline Analysis


9

simulation

reconstruction

analysis

interactivephysicsanalysis

batchphysicsanalysis

batchphysicsanalysis

detector

event summary data

rawdata

eventreprocessing

eventreprocessing

eventsimulation

eventsimulation

analysis objects(extracted by physics topic)

Data Handling and Computation for

Physics Analysisevent filter(selection &

reconstruction)

event filter(selection &

reconstruction)

processeddata

les.

rob

ert

son

@ce

rn.c

h

RAW

ESD

AOD


RAW

ESD

AOD

TAG

randomseq.

1PB/yr (1PB/s prior to reduction!)

100TB/yr

10TB/yr

1TB/yr

Data

Users

Tier0

Tier1

Physics @ LHC

Concluding talk, ≠ SummaryKraków, July 2006

John Ellis, TH Division, PH Department, CERN

• Principal Goals:• Explore a new energy/distance scale

• Look for ‘the’ Higgs boson

• Look for supersymmetry/extra dimensions, …

• Find something the theorists did not expect

All charged tracks with pt > 2 GeV

Reconstructed tracks with pt > 25 GeV

(+30 minimum bias events)

selectivity: 1 in 1013

- 1 person in a thousand world populations- A needle in 20 million haystacks

LHC: Higgs Decay into 4 muons


14The Worldwide LHC Computing Grid - [email protected] - CCP 2006 - Gyeongju, Republic of Korea

ATLAS Getting Ready for LHC

H ZZ 4

e,

Z

e,

e,

e, mZ

Hg

g

tZ(*)

“Gold-plated” channel for Higgs discovery at LHC

Simulation of a H ee event in ATLAS

Signal expected in ATLASafter ‘early' LHC operation

Physics example

(W)LCG Overview

The LHC Computing GridA Worldwide Grid build on existing Grid Infrastructures, including

OpenScience Grid (OSG), EGEE and NorduGrid

Grid Computing

Today there are many definitions of Grid computing:

The definitive definition of a Grid is provided by [1] Ian Foster in his article "What is the Grid? A Three Point Checklist" [2].

The three points of this checklist are:

1. Computing resources are not administered centrally;

2. Open standards are used;

3. Non trivial quality of service is achieved.

… Some sort of Distributed System at least…

that crosses Management / Enterprise domains

October 7, 200517

LCG Status report [email protected] LHCC Open Meeting; 28th June 2006

LCG depends on 2 major science grid infrastructures …

The LCG service runs & relies on the grid infrastructures provided by:

EGEE - Enabling Grids for E-SciencEOSG - US Open Science Grid


EGEE – Close-up

• Many EGEE regions are Grids in their own right

• In some cases these too are build out of smaller, regional Grids

• These typically have other, local users, in addition to those of the ‘higher-level’ Grid(s)

• Similarly, OSG also supports communities other than those of the LCG…


WLCG

• WLCG:– A federation of fractal Grids…– A (small) step towards “the” Grid

• (rather than “a” Grid)

EGEE OSGWLC

G

GridGrid

Grid Grid GridGrid


Why a Grid Solution?

• The LCG Technical Design Report lists:

1. Significant costs of [ providing ] maintaining and upgrading the necessary resources … more easily handled in a distributed environment, where individual institutes and … organisations can fund local resources … whilst contributing to the global goal

2. … no single points of failure. Multiple copies of the data, automatic reassigning of tasks to resources… facilitates access to data for all scientists independent of location. … round the clock monitoring and support.

October 7, 200521

LCG Status report [email protected] LHCC Open Meeting; 28th June 2006

WLCG Collaboration

The Collaboration ~130 computing centres 12 large centres

(Tier-0, Tier-1) 40-50 federations of smaller

“Tier-2” centres 29 countries

Memorandum of Understanding Agreed in October 2005, now being signed

Purpose Focuses on the needs of the 4 LHC experiments Commits resources

Each October for the coming year 5-year forward look

Agrees on standards and procedures

LCG Service ModelTier0 – the accelerator centre (CERN) Data acquisition & initial processing Long-term data curation Distribution of data Tier1s

Canada – Triumf (Vancouver)France – IN2P3 (Lyon)Germany – Forschungszentrum KarlsruheItaly – CNAF (Bologna)Netherlands – NIKHEF (Amsterdam)

Nordic countries – distributed Tier-1 Spain – PIC (Barcelona)Taiwan – Academia Sinica (Taipei)UK – CLRC (Didcot)US – FermiLab (Illinois) – Brookhaven (NY)

Tier1 – “online” to the data acquisition process high availability

Managed Mass Storage – grid-enabled data service

Data intensive analysis National, regional support Continual reprocessing activity

Tier2 – ~100 centres in ~40 countries Simulation End-user analysis – batch and interactive

Les Robertson

CERN18%

All Tier-1s39%

All Tier-2s43%

CERN12%

All Tier-1s55%

All Tier-2s33%

CERN34%

All Tier-1s66%

CPU Disk Tape

Summary of Computing Resource RequirementsAll experiments - 2008From LCG TDR - June 2005

CERN All Tier-1s All Tier-2s TotalCPU (MSPECint2000s) 25 56 61 142Disk (PetaBytes) 7 31 19 57Tape (PetaBytes) 18 35 53

Networking Requirements: GB/s out of CERN (1.6GB/s

nominal + factor 6 safety 100s of MB/s into Tier1s 10s of MB/s into / out of

Tier2s

Provisioned: (Backbone at CERN)

10Gbps link to each Tier1 site 1Gbps minimum to Tier2s

Summary of Tier0/1/2 Roles

Tier0: safe keeping of RAW data (first copy); first pass reconstruction, distribution of RAW data and reconstruction output to Tier1; reprocessing of data during LHC down-times;

Tier1: safe keeping of a proportional share of RAW and reconstructed data; large scale reprocessing and safe keeping of corresponding output; distribution of data products to Tier2s and safe keeping of a share of simulated data produced at these Tier2s;

Tier2: Handling analysis requirements and proportional share of simulated event production and reconstruction.

N.B. there are differences in roles by experimentEssential to test using complete production chain of each!

Dario Barberis: ATLAS SC4 Plans 26

WLCG SC4 Workshop - Mumbai, 12 February 2006

ATLAS Computing Model

Tier-0: Copy RAW data to Castor tape for archival

Copy RAW data to Tier-1s for storage and reprocessing

Run first-pass calibration/alignment (within 24 hrs)

Run first-pass reconstruction (within 48 hrs)

Distribute reconstruction output (ESDs, AODs & TAGS) to Tier-1s

Tier-1s: Store and take care of a fraction of RAW data

Run “slow” calibration/alignment procedures

Rerun reconstruction with better calib/align and/or algorithms

Distribute reconstruction output to Tier-2s

Keep current versions of ESDs and AODs on disk for analysis

Tier-2s: Run simulation

Keep current versions of AODs on disk for analysis



ATLAS Tier-0 Data Flow

EF

CPUfarm

T1T1T1sCastorbuffer

RAW

1.6 GB/file0.2 Hz17K f/day320 MB/s27 TB/day

ESD

0.5 GB/file0.2 Hz17K f/day100 MB/s8 TB/day

AOD

10 MB/file2 Hz170K f/day20 MB/s1.6 TB/day

AODm

500 MB/file0.04 Hz3.4K f/day20 MB/s1.6 TB/day

RAW

AOD

RAW

ESD (2x)

AODm (10x)

RAW

ESD

AODm

0.44 Hz37K f/day440 MB/s

1 Hz85K f/day720 MB/s

0.4 Hz190K f/day340 MB/s

2.24 Hz170K f/day (temp)20K f/day (perm)140 MB/s

Tape



ATLAS “average” Tier-1 Data Flow (2008)

Tier-0

CPUfarm

T1T1OtherTier-1s

diskbuffer

RAW

1.6 GB/file0.02 Hz1.7K f/day32 MB/s2.7 TB/day

ESD2


AOD2

10 MB/file0.2 Hz17K f/day2 MB/s0.16 TB/day

AODm2


RAW

ESD2

AODm2

0.044 Hz3.74K f/day44 MB/s3.66 TB/day

T1T1OtherTier-1s

T1T1Tier-2s

Tape

RAW


diskstorage

AODm2


ESD2


AOD2

10 MB/file0.2 Hz17K f/day2 MB/s0.16 TB/day

ESD2


AODm2


ESD2


AODm2


ESD1


AODm1


AODm1


AODm2


Plus simulation Plus simulation && analysis data analysis data

flowflow

Real data storage, reprocessing and

distribution

Tier1 Centre ALICE ATLAS CMS LHCb Target

IN2P3, Lyon 9% 13% 10% 27% 200

GridKA, Germany 20% 10% 8% 10% 200

CNAF, Italy 7% 7% 13% 11% 200

FNAL, USA - - 28% - 200

BNL, USA - 22% - - 200RAL, UK - 7% 3% 15% 150

NIKHEF, NL (3%) 13% - 23% 150

ASGC, Taipei - 8% 10% - 100

PIC, Spain - 4% (5) 6% (5) 6.5% 100

Nordic Data Grid Facility - 6% - - 50

TRIUMF, Canada - 4% - - 50

TOTAL 1.6GB/s

Nominal Tier0 – Tier1 Data Rates (pp)H

eat

Centre T0->T1 T1->T2 T2->T1 T1->T1

Predictable – Data Taking

Bursty – User Needs

Predictable – Simulation

Scheduled Reprocessing

IN2P3, Lyon 200 286.6 85.5

GridKA, Germany 200 353.0 84.1

CNAF, Italy 200 278.0 58.4

FNAL, USA 200 403.1 52.6

BNL, USA 200 64.5 24.8

RAL, UK 150 76.0 36.0

NIKHEF, NL 150 36.0 6.1

ASGC, Taipei 100 114.6 19.3

PIC, Spain 100 106.6 23.3

Nordic Data Grid Facility 50 - -

TRIUMF, Canada 50 - -

Global Inter-Site Rates


The Scoville Scale

• The Scoville scale is a measure of the hotness of a chilli pepper. These fruits of the Capsicum genus contain capsaicin, a chemical compound which stimulates thermoreceptor nerve endings in the tongue, and the number of Scoville heat units (SHU) indicates the amount of capsaicin present. Many hot sauces use their Scoville rating in advertising as a selling point.

• It is named after Wilbur Scoville, who developed the Scoville Organoleptic Test in 1912[1]. As originally devised, a solution of the pepper extract is diluted in sugar water until the 'heat' is no longer detectable to a panel of (usually five) tasters; the degree of dilution gives its measure on the Scoville scale. Thus a sweet pepper, containing no capsaicin at all, has a Scoville rating of zero, meaning no heat detectable even undiluted. Conversely, the hottest chiles, such as habaneros, have a rating of 300,000 or more, indicating that their extract has to be diluted 300,000-fold before the capsaicin present is undetectable. The greatest weakness of the Scoville Organoleptic Test is its imprecision, because it relies on human subjectivity.

http://en.wikipedia.org/wiki/Chilli_pepper

http://en.wikipedia.org/wiki/Chilli_pepper

http://en.wikipedia.org/wiki/Capsicum

http://en.wikipedia.org/wiki/Capsaicin

http://en.wikipedia.org/wiki/Chemical_compound

http://en.wikipedia.org/wiki/Thermoreceptor

http://en.wikipedia.org/wiki/Nerve_ending

http://en.wikipedia.org/wiki/Tongue

http://en.wikipedia.org/wiki/Hot_sauce

http://en.wikipedia.org/wiki/Wilbur_Scoville

http://en.wikipedia.org/wiki/Wilbur_Scoville


Scoville Scale – cont.

Scoville rating Type of pepper

No heat Bell Pepper

600 – 800 Green Tabasco Sauce

30,000 - 50,000 Cayenne Pepper

100,000 - 325,000 Scotch Bonnet

15,000,000 - 16,000,000

Pure capsaicin

LCG Status

The LHC Computing GridStatus of Deployment of Worldwide Production Grid Services


The LCG Service

• The LCG Service has been validated over the past 2 years via a series of dedicated “Service Challenges”, designed to test the readiness of the service infrastructure

• These are complementary to tests by the experiments of the offline Computing Models – the Service Challenges have progressively ramped up the level of service in preparation for ever more detailed tests by the experiments

The target: full production services by end September 2006!• Some additional functionality is still to be added, resource

levels will continue to ramp-up in 2007 and beyond• Resource requirements are strongly coupled to total volume

of data acquired to date


The Service Challenge Programme

• Significant focus on Data Management, including data export from Tier0-Tier1

• Services required by VO / site agreed in mid-2005 with small but continous evolution expected

Goal is delivery of stable production services• Status: after several iterations, requirements and

plans of experiments understood, required services by site established

• Still some operational and functional problems, being pursued on a regular basis

Service Maximum delay in responding to operational problems

Average availability[1] on an annual

basis

DOWN Degradation > 50%

Degradation > 20% BEAMON

BEAMOFF

Raw data recording

4 hours 6 hours 6 hours 99% n/a

Event reconstruction / data distribution (beam ON)


Networking service to Tier-1 Centres (beam ON)


All other Tier-0 services

12 hours 24 hours 48 hours 98% 98%

All other services[2] – prime service hours[3]

1 hour 1 hour 4 hours 98% 98%

All other services – outside prime service hours

12 hours 24 hours 48 hours 97% 97%

CERN (Tier0) MoU Commitments

R.Bailey, Chamonix XV, January 2006R.Bailey, Chamonix XV, January 2006 3737

Breakdown of a normal yearBreakdown of a normal year

7-8

~ 140-160 days for physics per yearNot forgetting ion and TOTEM operation

Leaves ~ 100-120 days for proton luminosity running? Efficiency for physics 50% ?

~ 50 days ~ 1200 h ~ 4 106 s of proton luminosity running / year

- From Chamonix XIV -S

ervi

ce u

pgra

de s

lots

?

July-August 2006 Disk-Tape Rates

Centre ATLAS(4/4)

ATLAS tape

CMS(1/4)

LHCb ALICE(HI)

CombinedTape Rates

Nominal ppAll to tape

ASGC 60.0 24 10 - - 35 100

CNAF 59.0 24 25 ~4 60 113 200

PIC 48.6 20 30 ~4 - 54 100

IN2P3 90.2 36 15 ~4 60 115 200

GridKA 74.6 30 15 ~4 60 109 200

RAL 59.0 24 10 ~4 30 68 150

BNL 196.8 80 - - - 80 200

TRIUMF 47.6 20 - - - 20 50

SARA 87.6 36 - ~4 30 70 150

NDGF 48.6 20 - - - 20 50

FNAL - 50 - - 50 200

Totals ~800 1600

Testing of experiment driven data export at 50% ofnominal rate > 1 year prior to first collisions

Easter w/eTarget 10 day period


Experiment Production

• Experiments currently testing full production chain

• Elements include:– Data export– Job submission– Full integration of

Tier0/Tier1/Tier2 sites


Plans Prior to First Collisions

• Between now and first collisions these activities will continue, progressively ramping up in scope and scale

• Still significant work to involve ~100 Tier2s in a distributed, reliable service

• Still much work to do to attain data rates for prolonged periods (weeks) including recovery from site failure – power, cooling, service issues


• First collisions LHC expected November 2007– These will be at ‘low’ energy – 450 GeV per beam– Main target will be understanding detectors, trigger and offline software– ‘Re-discover’ existing physics – excellent for calibration! Data rates will be full nominal values! (Machine efficiency?)

• First full energy run in 2008: 7 + 7 TeV– Physics discovery run!– Heavy Ions in 2009? Data export schedule?

• Typically takes ~years to fully understand detector and software chain– Much of the initial ‘analysis’ will be done starting from RAW/ESD datasets– Big impact on network load – larger datasets, transferred more frequently Potential mismatch with ‘steady-state’ planning? Much larger initial bandwidth requirement (but do you really believe it will

go down?)– Those sites that have it will be more ‘competitive’ (and vice-versa…)

• Rate calculations have overhead for recovering backlogs due to down-time– But not for recovery from human and / or software error! e.g. bug in alignment / calibration / selection / classification code -> junk data!

And Beyond…


Summary & Conclusions

• Deploying a Worldwide Production Grid is not without its challenges

• Much has been accomplished; much still outstanding• My two top issues?

– Collaboration & communication at such a scale requires significant and constant effort

• We are not yet at the level that this is just basic infrastructure

– “Design for failure” – i.e. assume that things don’t work, rather than hope that they always do!

• A lesson from our “founding fathers” – the creators of the Internet?

the worldwide lhc computing grid

Documents

lhc simulation

computing models

grid technologies

republic of koreacmsdata

republic of koreawlcg

higgs discovery

higgs bosonlook

scientific potential