karlsruhe, april 27th, 2009desy it1 computing services for large experiments in physics volker glzow...

Karlsruhe, April 27th, 2009 DESY IT 1

Computing Services for large Experiments in Physics

Volker Gülzow

April 27th, 2009

Karlsruhe, April 27th, 2009 Gülzow DESY IT 2

Outline

Computing for the LHC Experiments Open the Grid to other fields Change of landscape Conclusion


Computing for the physics

Simulation Experiments


Large Experiments do need Computing TDR‘s


The Eventflow

Rate

[Hz]

RAW

[MB]

ESDrDST

RECO[MB]

AOD

[kB]

MonteCarlo

[MB/evt]

MonteCarlo

% of real

ALICE HI 100 12.5 2.5 250 300 100ALICE pp 100 1 0.04 4 0.4 100ATLAS 200 1.6 0.5 100 2 20CMS 150 1.5 0.25 50 2 100LHCb 2000 0.025 0.025 0.5 20

107 seconds/year pp from 2009 on ~109 events/experiment106 seconds/year heavy ion

Karlsruhe, April 27th, 2009 DESY IT 6

GridKA

Each layer:Specialised for certain tasks

e.g. T2:

-Analysis

-User access

-AOD storage


GRID

MIDDLEWARE

Visualizing

Supercomputer, PC-Cluster

Data Storage, Sensors, Experiments

Internet, Networks

Desktop

Mobile Access

The Grid Dream


Grid Computing

Grid Computing is about virtualization of global resources.

• It is about transparent access to globally distributed resources such as data and compute cycles

• A Grid infrastructure consists of services to access resources and (of course) of the resources itself

Opposite to distributed computing, Grid resources are not centrally controlled

Hence it is mandatory to use standard, open, general-purpose protocols and interfaces

A Grid must deliver nontrivial qualities of services

• In general Grid infrastructures are generic; without any dependencies of the applications / experiments


Grid Types

• Data Grids:• Provisioning of transparent access to data which can be

physically distributed within Virtual Organizations (VO)• Computational Grids:

• allow for large-scale compute resource sharing within Virtual Organizations (VO)

• Information Grids:• Provisioning of information and data exchange, using well

defined standards and web services• Collaborative Grids:

• allow networking people from various locations

There is not „THE Grid“


Grid Infrastructure and T2

Site3Site2

outputUI

LFC

WN WN WNWN

SEGR

ISCE GR

IS

BDII

JDL RBWMS

Batch

LFC

VOMSRBCESE

/etc/grid-security/grid-mapfile

DESY

ssh

$HOME/.globus/

certs

GIIS

HMS

world

R-GMA

VO exclusive


The Storage Element basic characteristics

LAN data access method/protocol for POSIX style file access

Has a information provider system, which can be queried externally

Has a wide area data access capability – “FTP”and “Web”-style

Has a Storage Management Interface (SRM), for accessibility and communication with other SE and/or data management systems


OSMHPSS

SRM SRM

Virtual Storage LayerSRM

SRM ClientTransfer Protocol Negotiation

Data Transfer

Storage Element

EnstoreTsmCastor

SRM

Jasmine

SRMSRM

Desy GridKarlsruhe

Fermi San Diego CERN Jefferson

dCache.ORG

gsiFTP(ssl) HTTP

Grid Storage Fabric Abstraction

dCap

GRID Middleware


dCache - a few ‘what it is’ lines

distributed cache system to optimize data access on tertiary storage

distributed Peta Byte Disk Store with single rooted file-system namespace, providing posix like and wide area access protocols

Grid Storage Element coming with standard data access protocols, Information Provider Protocols and Storage Resource Manager

Karlsruhe, April 27th, 2009 Gülzow

dCac

he.O

RGdC

ache

.ORG

dCache topology

dCache is a collaboration between DESY, FERMILab and the Nordic Data Grid Facility (NDGF) (Leader: Patrick Fuhrmann, DESY).

Beside providing core function of dCache, DESY is managing the project infrastructure

Code repository, Web, Wiki, code download pages Ticket system and e-mail support lists Regression test suite Customer relations, workshops, tutorials

dCache is distributed by gLite and US Open Science Grid Virtual Data Tool (VDT)

dCache is supported by d-grid, the HGF “Physics at the Terascale” and the US Open Science Grid (OSG).

dCache will hold the largest share of LHC data outside CERN within the next years.

dCache is in production at 7 of 11 LHC Tier 1 centres and more than 40 Tier 2's.

dCache.ORG

gLite


Issues being a Grid Site

• The local installation is operated in a global environment• There is always day light somewhere on the globe• Core Grid services are used everywhere (VOMS, LFC)

• One common infrastructure for multiple VOs of multiple disciplines• Different groups want different things• Computing models differ fundamentally• Use and user patterns differ• Software requirements differ

• User support is a big issue• Not scalable• Underestimated• Has a huge social factor


Reliability of the DESY Tier 2

march-09 Dec-08 jan-09 feb-09


Pushing the Grid: The D-Grid Initiative

National Grid Initiative BMBF-funded with > 100 Mio € Up to now 3 Calls 2 strategic lines: Communities & Integration

(DGI) All major Grid Labs are in DGI Many many communities Work in progress: sustainable infrastructure


Pushing Grid Technology

Generic vs Application driven strategy: Generic: Find/develope generic tools,

blueprints and generic strategies for many communities

Application driven: help communities from the neighbourhood to get started with the grid (-> my preferred approach)

Why? „You need to speak the language of the partner and have to have a feeling for the problem“


Research with Photons

Light Source DORIS @ DESY


Get the Grid to the „Photons“

Differences to HEP Short experiments Many experiments Data Volumes/Year: FDE ~ 3 PB, MID ~ 1.8 PB (1

MPIX CCD) Data stored in small files Data compression: open Many beamlines, different experiments Lifetime of data unclear Opportunistic users The „mental“ problem


Methods are different


Grids for „Photons“

Certificates for user access Computing for analysis under debate (~1000

„MPI-“cores / beamline) Storage: Access via Grid Tools Short term analysis Long term archiving

-> Concept: To start joint door opener projects between research groups and IT, it needs missionery effort


Everything on the grid?

Grid and the Tier model well suited for• Global & coordinated tasks

Analysis• Local & uncoordinated, chaotic data access

Strong need for additional resources for interactive use but with Grid-technology

Access of interactive resources to the SE


The landscape has changed

Virtualization Clouds Energy Networks and international collaborations


What is virtualization?

technique for hiding physical characteristics of computing resources from the way in which other systems, applications, or end users interact with those resources. • making a single physical resource appear to function as multiple logical

resources– server, operating system, application, storage device

• making multiple physical resources appear as a single logical resource.– storage devices or servers

In detail different technologies: bare metal full virtualization, paravirtualization etc.

It’s a concept to make use out of multicore systems Vendor provide HW-support


Can we profit from virtualization?

Yes, definitely, we can

Pro: We will have multiple core systems, which we can efficiently

use We can run legacy applications We can share resources between large experiments Gives portability of the codeCon: Security issues, what about OS patches? Lasy-factor „we don‘t want to port our application“ Support and maintainance costs


Simple Idea:

Setup huge facility and profit from:• Economies of scale (hardware, management,

operation)• Efficient resource scheduling for high utilisation• Tax-efficient locations• Cheap energy, cheap labor• Use virtualization techniques

Clouds


Google had 36 data centers in 2008**

WLCG had 134

BUT

One Google Data Center is estimated to cost ~$600M

An order of magnitude more than the new centre being planned at CERN

Google’s data center at the Dalles on the Columbia river

**source: royal.pingdom.com

Different Approaches


Microsoft’s data center in Quincy, WA • 44K m2 - 10 rooms each with up to 30K servers and storage for 6 trillion

photos Yahoo, Amazon, IBM -- also building giant data centers These major companies are expecting to build new markets for utility

computing (clouds), software as a service (Google Apps), as well as absorbing the expansion of traditional computing services...

Clouds


Clouds vs Grids

Clouds aim at efficient sharing of the hardware• low-level execution environment, Isolation between users• Operated as a homogeneous, single-management domain• Straight-forward I/O and storage• Expose only a high-level view of the environment -

scheduling, data placement, performance issues are hidden from the application and the user

Grids aim at collaboration• Add your resources to the community, but retain management

control• Expose topology – location of storage, availability of

resources• Choice of tools to hide the complexity from the user,


Clouds vs. Grids

Both need complex middleware to function• Grids had a problem in trying to provide a universal high-

functionality environment (OS, data management, ....), with intersecting collaborations and a naturally competitive environment

• Clouds have an advantage in offering a simpler base environment, leaving much of the functionality to the application - where universal solutions are not necessary – and what they do have to provide can be decided within a single management hierarchy

• Both have problems with licenceing As the names suggest –

the grids are transparent and the clouds are opaque


“Clouds” are just starting, will they survive these day’s? What to do, if not? Are there standards? Arn‘t we running a cloud ourselves already? We have to look on costs and potentially have to be ready to

integrate clouds May be use clouds as „overload resource“ And also learning from the innovative technology that is appearing

stimulated by the cloud market• to manage, operate and supply these giant facilities with

data• to package applications to use (heterogeneous) clouds

Clouds for scientific environment


Energy Energy cost of a computing infrastructure are not negletable Look for way‘s to save energy What helps? New processor technology (eg Nehalem) New peripherals concepts Efficient cooling concepts Efficient service provisioning concepts (eg virtualization) Power management and monitoring …


DOE Energy Datacenter Efficiency Initiative

www.energystar.gov


Networks & Mobility

We do have excellent connectivity in parts of the world But we have „digital divide“ We have large user communities accessing our resources at

any time Travelling scientists: We are close to having good bandwidth data connections

almost everywhere we go And we already have a powerful high capacity computer in the backpack This is where end-user analysis is going to be done The physicist’s notebook must be integrated with the experiment

environment, the physics data, and the grid resources Without burdening the notebook or its user The grid environment is too complex to be extended to the notebook


Role of a Scientific Computer Centre

Provide excellent operational service Have flexible and open concepts Know How source Own but adjusted scientific agenda Active partner in scientific collaborations Driving force for IT-innovation Solution (and SW) development Member in international projects


Conclusion

Computing is a major component in large physics experiments

Computing TDR‘s are definitely needed very early Grid‘s are a possible good solution Clouds will not replace the Grid but we can learn

and merge Virtualization is a evident concept In general there are excellent opportunities for vital

scientific Computer Centres


Backup slides


Risks

Requirements of the experiments are not fully settled

Member of a multisite chain, you depend on others There is often money around for invest, not for

personell Operation of huge installations is often

underestimated Financing ist not always long term secured Huge software effort is needed for following the

technology


Summary Grids are all about sharing.

• groups distributed around the world can pool their computing resources• large centres and small centres can all contribute• users everywhere can get equal access to data and

Grids are also flexible • place the computing facilities in the most effective and efficient places• exploiting funding wherever it is provided

HEP and others have shown that• grids can support computational and storage resources on a massive scale• that can be operated around the clock• running hundreds of thousands of jobs every day

The grid model has stimulated high energy physics to organise its computing• in a widely distributed way• building a collaboration involving directly a large fraction of the LHC members and their

institutes

This will be the workhorse for production data handling for many yearsand as such must be maintained and developed through the first waves of data taking


But – the landscape has changed dramatically over the past decade

The Web, the Internet, powerful PCs, broadband to the home, …• have stimulated the development of new applications that generate a

massive demand for computing remote from the user• …. that is being met by giant, efficient facilities deployed around the world• .... and creates a market for new technologies capable of operating on a

scale equivalent to that of HEP Whether or not commercial clouds become cost-effective for HEP data handling

is only a financial and funding-agency issueBUT Exploiting the associated technologies is an obligation

Could there be a revolution here for physics analysis?


The MoU for WLCG

See: http://lcg.web.cern.ch/LCG/mou.htm Stating rules, Tier 1&2 centres&responsibilities&

pledges, 11 Tier 1‘s, ~ 120 Tier 2‘s

http://lcg.web.cern.ch/LCG/mou.htm


The German Science Net XWIN


Different tasks: Different requirements

MC Production• Event Generation: no I; small O; little CPU• Detector Simulation: small I; large O & CPU

Event Reconstruction/Reprocessing• Reprocessing: full I; full O; large CPU• Selections: large I; large O; large CPU

Analysis• Usually: large I; small O; little CPU• Performed by many users, many times!• LHC StartUp phase: Short turn-around

Coordinated &global tasks

Uncoordinated, chaotic & local tasks


SE + surroundings

File

Transfer

ServiceFTS DM App

SRM SRM

SE SEFTS Channel

FTP connections

CE

LANAccess

Information System


The Grid Installation @ DESY, a complex system DESY’s obligation to WLCG (Tier 2 for Atlas, CMS, LHCb) The NAF for the Helmholtz Alliance (grid, interactive, batch),

seamless, site independent Support of the HERA analysis Hosting & support of various Virtual Organisations like ILC,

Icecube, ILDG etc. (total 15 VO’s) with a complete Grid infrastructure (example: running 7 Workload Mgmt Sys)

DESY’s obligation to D-Grid DESY’s obligation to EGEE Federation with RWTH Aachen (CMS) Federation with Uni Göttingen (Atlas) Support and cooperation with the Experiments


Storage & Computing Resources

• Zeuthen: – ~500 Cores -> 700 kSI2k for Tier2 & NAF Grid & ILDG &

Icecube– 460 TB disk for dCache, Lustre

• Hamburg:– ~2000 Cores -> 3000+ kSI2k for Tier 2 & NAF Grid & Hera &

ILC & others– 900 TB disk for dCache, Lustre – 450 TB for NAF dCache

• Total: ~ 3700+ kSI2k for the Grid, • NAF: interactive/batch (Zeuthen&Hamburg)

– 868 Cores ->1200 kSI2k for NAF interactive/batch


links

http://lcg.web.cern.ch/LCG/ http://naf.desy.de/ http://www.sun.com/software/products/lustre/ http://gridmap.cern.ch

http://lcg.web.cern.ch/LCG/

http://lcg.web.cern.ch/LCG/


EGEE …

Objectives:

The EGEE project brings together experts from more than 50

countries with the common aim of building on recent advances in

Grid technology and developing a service Grid infrastructure which is

available to scientists 24 hours-a-day.

The project provides researchers in academia and business with

access to a production level Grid infrastructure, independent of their

geographic location. The EGEE project also focuses on attracting a

wide range of new users to the Grid.

Because of its needs and its tradition and because of its *simple* use

cases, HEP has become the pilot application for the Grid (in EGEE).


NAF: Schematic basic layout

Interactive

Proof?

Batch-Type(SGE)

gsissh

grid-submit

qsub

??

AFS

Parallel Cluster FS

(Lustre)

SRM (dCache)DedicatedSpace in DESY T2 space

Desy Tier 2 (Batch)grid-submit

AFS/Kerberos

?? SRM ??

Grid-ftp /

srmcp

scp

NAFInterconnect


sketch of dCache components

tertiarystorage

Pool Nodes

namespaceservice

NFS v2/3

dCache Core

FTP

local access:- dCap- xrootd- http

SRMworld

put/get

local client

loca

l clie

nt


Virtualization on the Worker Nodes

Surprising idea: Virtualization costs performance, but many benefits:• More OS types and flavors can be supported, also old OS on new

hardware possible• Each jobs runs in his own OS instance, does not affect other jobs:

security through encapsulation• Separation of local and grid environment/users• Desktop harvesting?• Each job might get a clean system at start: No trojans• Buy a general purpose cluster, and use it for many different

purposes• Job migration and checkpointing: Interesting for MPI and very long

jobs• Distributed administration: Local admin installs VMM, generic

Virtual Machine provided by user or third party


Energy

www.koomey.com

karlsruhe, april 27th, 2009desy it1 computing services for large experiments in physics volker glzow...

Documents