cern intro 2010-10-27-snw
Post on 05-Dec-2014
503 Views
Preview:
DESCRIPTION
TRANSCRIPT
Grid Computingat the Large Hadron Collider:
Massive Computing at the Limit of Scale, Space, Power
and Budget
Dr Helge MeinhardCERN, IT Department
SNW Frankfurt, 27 October 2010
CERN (1)
§ Conseil européenpour la recherchenucléaire – aka European Laboratory for Particle Physics§ Facilities for
fundamental research§ Between Geneva and
the Jura mountains, straddling the Swiss-French border§ Founded in 1954
CERN (2)
§ 20 member states§ ~3300 staff members,
fellows, students, apprentices§ 10’000 users registered
(~7’000 on site)§ from more than 550
institutes in more than 80 countries
§ 1026 MCHF (~790 MEUR) annual budget§ http://cern.ch/
Physics at the LHC (1)
Matter particles: fundamental building blocks
Force particles:bind matter particles
Physics at the LHC (2)
§ Four known forces: strong force,weak force, electromagnetism, gravitation§ Standard model
unifies three of them§ Verified to
0.1 percent level§ Too many free
parameters§ E.g. particle masses
§ Higgs particle§ Higgs condensate fills
vacuum§ Acts like ‘molasse’,
slows other particles down, gives them mass
Physics at the LHC (3)
§ Open questions in particle physics:§ Why are the parameters of the size as we observe
them?§ What gives the particles their masses?§ How can gravity be integrated into a unified theory?§ Why is there only matter and no anti-matter in the
universe?§ Are there more space-time dimensions than the 4 we
know of?§ What is dark energy and dark matter which makes
up 98% of the universe?§ Finding the Higgs and possible new physics with
LHC will give the answers!
The Large Hadron Collider (1)
§ Accelerator for protons against protons – 14 TeVcollision energy§ By far the world’s
most powerful accelerator
§ Tunnel of 27 km circumference, 4 m diameter, 50…150 m below ground§ Detectors at four
collision points
The Large Hadron Collider (2)
§ Approved 1994, first circulating beams on10 September 2008§ Protons are bent by
superconducting magnets (8 Tesla, operating at 2K = –271°C) all around the tunnel§ Each beam: 3000
bunches of 100 billion protons each§ Up to 40 million bunch
collisions per second at the centre of each of the four detectors
LHC Status and Future Plans
Date Event10-Sep-2008 First beam in LHC19-Sep-2008 Leak when magnets ramped to full field for 7 TeV/beam20-Nov-2009 First circulating beams since Sep-200830-Nov-2009 World record: 2 * 1.18 TeV, collisions soon after19-Mar-2010 Another world record: 2 * 3.5 TeV30-Mar-2010 First collisions at 2 * 3.5 TeV, special day for the press26-Jul-2010 Experiments present first results at ICHEP conference14-Oct-2010 Target luminosity for 2010 reached (10^32)Until end 2011 Run at 2 * 3.5 TeV to collect 1 fb-1
2012 Shutdown to prepare machine for 2 * 7 TeV2013 - …(?) Run at 2 * 7 TeV
LHC Detectors (1)ATLAS
CMS
LHCb
LHC Detectors (2)
3’000 physicists (including 1’000 students) from 173 institutes of 37 countries
LHC Data(1)
The accelerator generates 40 million bunch collisions (“events”) every second at the centre of each of the four experiments’ detectors§ Per bunch collision, typically
~20 proton-proton interactions
§ Particles from previous bunch collision only 7.5 cm away from detector centre
LHC Data (2)
Reduced by online computers that filter out a few hundred “good” events per second …
… which are recorded on disk and magnetic tape at 100…1’000 Megabytes/sec
15 Petabytes per year for four experiments
15’000 Terabytes = 3 million DVDs
1 event = few Megabytes
LHC Data (3)
CERN18%
All Tier-1s39%
All Tier-2s43%
CERN12%
All Tier-1s55%
All Tier-2s33%
CERN34%
All Tier-1s66%
CPU Disk Tape
30’000 CPU servers, 110’000 disks:
Too much for CERN!
Summary of Computing Resource RequirementsAll experiments – 2008From LCG TDR – June 2005
TotalCPU (MSPECint2000s) 142Disk (Petabytes) 57Tape (Petabytes) 53
CERN All Tier-1s All Tier-2s25 56 617 31 19
18 35
Worldwide LHC Computing Grid (1)§ Tier 0: CERN§ Data acquisition and
initial processing§ Data distribution§ Long-term curation
§ Tier 1: 11 major centres§ Managed mass storage§ Data-heavy analysis§ Dedicated 10 Gbps lines
to CERN§ Tier 2: More than 200
centres in more than 30 countries§ Simulation§ End-user analysis
§ Tier 3: from physicists’ desktops to small workgroup cluster§ Not covered by MoU
Tier3physics
department
a
b
g
Desktop
Germany
USAUK
France
Italy
Taiwan
NordicCountries
Nether-lands
CERN Tier 0
Tier2
Lab a
Uni a
Lab c
Uni n
Lab m
Lab b
Uni bUni y
Uni x
grid for a physicsstudy group
SpainTier 1
grid for a regional group
Worldwide LHC Computing Grid (2)
§ Grid middleware for “seemless” integration of services§ Aim: Looks like single huge compute facility§ Projects: EDG/EGEE/EGI, OSG§ Big step from proof of concept to stable,
large-scale production§ Centres are autonomous, but lots of
commonalities§ Commodity hardware (e.g. x86 processors)§ Linux (RedHat Enterprise Linux variant)
CERNComputer CentreFunctions:§ WLCG: Tier 0,
some T1/T2§ Support for smaller
experiments at CERN§ Infrastructure for
the laboratory§ …
Requirements and Boundaries (1)
§ High Energy Physics applications require mostly integer processor performance§ Large amount of processing power and storage
needed for aggregate performance§ No need for parallelism / low-latency high-speed
interconnects§ Can use large numbers of components with
performance below optimum level (“coarse-grain parallelism”)
§ Infrastructure (building, electricity,cooling) is a concern§ Refurbished two machine rooms
(1500 + 1200 m2) for total air cooledpower consumption of 2.5 MWatts§ Will run out of power in about 2014…
Requirements and Boundaries (2)
§ Major boundary condition: cost§ Getting maximum
resources with fixed budget…
§ … then dealing with cuts to “fixed” budget
§ Only choice: commodity equipment as far as possible, minimisingTCO / performance§ This is not always the
solution with the cheapest investment cost!
Purchased in 2004, now retired
The Bulk Resources – Event Data
Tapes/ servers
Disk servers
CPU servers
Router
R
R
R
REthernetbackbone
(multiple 10GigE)
10GigE
Permanent storage on tape
Disk as temporary buffer
Data paths:tape « diskdisk « cpu
(simplified network topology)
CERN CC currently (September 2010)
§ 8’500 systems, 54’000 processing cores§ CPU servers, disk servers, infrastructure
servers§ 49’900 TB raw on 58’500 disk drives§ 25’000 TB used, 50’000 tape cartridges
total (70’000 slots), 160 tape drives§ Tenders in progress or planned
(estimates)§ 800 systems, 11’000 processing cores§ 16’000 TB raw on 8’500 disk drives
Disk Servers for Bulk Storage (1)§ Target: temporary event data storage§ More than 95% of disk storage capacity
§ Best TCO / performance: Integrated PC server§ One or two x86 processors, 8…16 GB, PCI RAID card(s)§ 16…24 hot-swappable 7’200 rpm SATA disks in server chassis§ Gigabit or 10Gig Ethernet§ Linux (of course)
§ Adjudication based on total usable capacity with constraints§ Power consumption taken into account§ Systems procured recently: depending on specs,
5…20 TB usable§ Looking at software RAID, external iSCSI disk enclosures
§ Home-made optimised protocol (rfcp) and HSM software (Castor)
Disk Servers for Bulk Storage (2)
Disk Servers for Bulk Storage (3)
Other Disk-based Storage
§ For dedicated applications (not physics bulk data):§ SAN/FC storage§ NAS storage§ iSCSI storage
§ Total represents well below 5% of disk capacity§ Consolidation project ongoing
Procurement Guidelines
§ Qualify companies to participate in calls for tender§ A-brands and their resellers§ Highly qualified assemblers/integrators
§ Specify performance rather than box counts§ Some constraints on choices for solution§ Leave detailed system design to bidder
§ Decide based on TCO§ Purchase price§ Box count, network connections§ Total power consumption
The Power Challenge – why bother?
§ Infrastructure limitations§ E.g. CERN: 2.5 MW for IT equipment§ Need to fit maximum capacity into given power envelope
§ Electricity costs money§ Costs likely to raise (steeply) over the next few years
§ IT responsible of significant fraction of world energy consumption§ Server farms in 2008: 1…2% of the world’s energy
consumption (annual growth rate: 16…23%)§ CERN’s data centre is 0.1 per mille of this…
§ Responsibility towards mankind demands using the energy as efficiently as possible
§ Saving a few percent of energy consumption makes a big difference
CERN’s Approach
§ Don’t look in detail at PSU, fans, CPUs, chipset, RAM, disk drives, VRMs, RAID controllers, …§ Rather: Measure apparent (VA) power consumption
in primary AC circuit§ CPU servers: 80% full load, 20% idle§ Storage and infrastructure servers: 50% full load, 50% idle
§ Add element reflecting power consumption to purchase price§ Adjudicate on the sum of purchase price and power
adjudication element
Power Efficiency: Lessons Learned
§ CPU servers: power efficiency increased by factor 12 in a little over four years§ Need to benchmark concrete servers§ Generic statements on platform are void
§ Fostering energy-efficient solutions makes a difference§ Power supplies feeding more than one
system usually more power-efficient§ Redundant power supplies are inefficient
Future (1)
§ Is IT growth sustainable?§ Demands continue to rise exponentially§ Even if Moore’s law continues to apply, data
centres will need to grow in number and size§ IT already consuming 2% of world’s energy –
where do we go?§ How to handle growing demands within a
given data centre?§ Demands evolve very rapidly, technologies less
so, infrastructure even at a slower pace – how to best match these three?
Future (2)
§ IT: Ecosystem of§ Hardware§ OS software and tools§ Applications
§ Evolving at different paces: hardware fastest, applications slowest§ How to make sure at any given time that
they match reasonably well?
Future (3)
§ Example: single-core to multi-core to many-core§ Most HEP applications currently single-
threaded§ Consider server with two quad-core CPUs as
eight independent execution units§ Model does not scale much further
§ Need to adapt applications to many-core machines§ Large, long effort
Summary§ The Large Hadron Collider (LHC) and its experiments is a
very data (and compute) intensive project§ LHC has triggered or pushed new technologies§ E.g. Grid middleware, WANs
§ High-end or bleeding edge technology not necessary everywhere§ That’s why we can benefit from the cost advantages of
commodity hardware§ Scaling computing to the requirements of LHC is hard
work§ IT power consumption/efficiency is a primordial concern§ We are steadily taking collision data at 2 * 3.5 TeV, and
have the capacity in place for dealing with this§ We are on track for further ramp-ups of the computing
capacity for future requirements
Summary of Computing Resource RequirementsAll experiments - 2008From LCG TDR - June 2005
CERN All Tier-1s All Tier-2s TotalCPU (MSPECint2000s) 25 56 61 142Disk (PetaBytes) 7 31 19 57Tape (PetaBytes) 18 35 53
Thank you
BACKUP SLIDES
CPU Servers (1)
§ Simple, stripped down, “HPC like” boxes§ No fast low-latency interconnects
§ EM64T or AMD64 processors (usually 2),2 or 3 GB/core, 1 disk/processor§ Open to multiple systems per enclosure§ Adjudication based on total performance
(SPECcpu2006 – all_cpp subset)§ Power consumption taken into account§ Linux (of course)
CPU Servers (2)
Tape Infrastructure (1)
§ 15 Petabytes per year§ … and in 10 or 15 years’ time physicists will
want to go back to 2010 data!
§ Requirements for permanent storage:§ Large capacity§ Sufficient bandwidth§ Proven for long-term data curation§ Cost-effective
§ Solution: High-end tape infrastructure
Tape Infrastructure (2)
Mass Storage System (1)
§ Interoperation challenge locally at CERN§ 100+ tape drives§ 1’000+ RAID volumes on disk servers§ 10’000+ processing slots on worker nodes
§ HSM required§ Commercial options carefully considered
and rejected: OSM, HPSS§ CERN development: CASTOR (CERN
Advanced Storage Manager)http://cern.ch/castor
Mass Storage System (2)
§ Key CASTOR features§ Database-centric layered architecture§ Stateless agents; can restart easily on error§ No direct connection from users to critical services
§ Scheduled access to I/O§ No overloading of disk servers§ Per-server limit set according to type of transfer§ servers can support many random access style
accesses, but only a few sustained data transfers§ I/O requests can be scheduled according to priority§ Fair shares access to I/O just as for CPU§ Prioritise requests from privileged users
§ Performance and stability proven at the level required for Tier 0 operation
Box Management (1)
§ Many thousand boxes§ Hardware management (install, repair, move,
retire)§ Software installation§ Configuration§ Monitoring and exception handling§ State management
§ 2001…2002: Review of available packages§ Commercial: Full Linux support rare, insufficient
reduction on staff to justify licence fees§ Open Source: Lack of features considered
essential, didn’t scale to required level
Box Management (2)
§ ELFms (http://cern.ch/ELFms)§ CERN development in collaboration with
many HEP sites and in the context of the European DataGrid (EDG) project
§ Components:§ Quattor: installation and configuration§ Lemon: monitoring and corrective actions§ Leaf: workflow and state management
Box Management (3): ELFms Overview
NodeConfigurationManagement
NodeManagement
Leaf
LemonPerformance& ExceptionMonitoring
LogisticalManagement
Box Management (4): Quattor
Node Configuration Manager NCM
CompA CompB CompC
ServiceAServiceBServiceC
RPMs / PKGs
SW Package ManagerSPMA
Managed Nodes
SW server(s)
HTTP
SWRepository RPMs
Install server
HTTP / PXE System
installer
Install Manager
base OS
XML configuration profiles
Configuration server
HTTP
CDB
SQL backend
SQL
CLIGUI
scriptsXML backend
SOAP
Used by 18 organisationsbesides CERN; including two distributed implementations
with 5 and 18 sites.
Box Management (5): Lemon
CorrelationEngines
User Workstations
Web browser
Lemon CLI
User
MonitoringRepository
TCP/UDP
SOAP
SOAP
Repositorybackend
SQL
Nodes
Monitoring Agent
Sensor SensorSensor
RRDTool / PHP
apache
HTTP
Box Management (6): Lemon
§ Apart from node parameters, non-node parameters are monitored as well§ Power, temperatures, …§ Higher-level views of Castor, batch queues
on worker nodes etc.
§ Complemented by user view of service availability: Service Level Status
Box Management (7): Leaf
§ HMS (Hardware management system)§ Track systems through lifecycle§ Automatic ticket creation§ GUI to physically find systems by host name
§ SMS (State management system)§ Automatic handling and tracking of high-level
configuration steps§ E.g. reconfigure, drain and reboot all cluster
nodes for a new kernel version
Box Management (8): Status
§ Many thousands of boxes managed successfully by ELFms, both at CERN and elsewhere, despite decreasing staff levels§ No indication of problems scaling up further§ Changes being applied wherever necessary§ E.g. support for virtual machines
§ Large-scale farm operation remains a challenge§ Purchasing, hardware failures, …
top related