the cms online cluster: setup, operation and maintenance of an evolving cluster

The CMS Online Cluster: Setup, Operation and Maintenance

of an Evolving Cluster"J.A. Coarasa "

CERN, Geneva, Switzerland"

for the CMS TriDAS group.""

ISGC 2012, 26 February - 2 March 2012, Academia Sinica, Taipei, Taiwan

ISGC 2012, 26/2-2/3 2012, Academia Sinica, Taipei, Taiwan The CMS Online Cluster

Outline"•  Introduction"

•  On/Off-line Computing Model"•  The Compact Muon Solenoid (CMS) Data Acquisition

system (DAQ)"

•  The CMS Online Cluster"–  IT Infrastructure"

•  Computing"•  Networking"•  Services"

– Operation"

J.A. Coarasa 2


Introduction"– The CMS Data Acquisition system (DAQ)"

J.A. Coarasa 3


Rate on tape 100 Hz!

On/Off-line computing"

~3µs"

~ s "

Level-1. Massive parallel processing !Particle identification (High pT, e, µ, jets, missing ET)"Hardwired custom systems (ASIC, FPGA). Synchronous clock driven!!

High Level Triggers !Physics process identification !Clusters of PCs. Asynchronous event driven!

!

First Level 100 kHz!

Readout: Data to Surface and Event Builder!2 Tb/s optical data links and 2 Tb/s switch networks!!

Distributed computing GRID (Tiers 0-4)!Analysis, production and archive!

!

Collisions rate ~1 GHz!

High Level Triggers!


The CMS Data Acquisition system (DAQ). The challenge"

Large Data Volumes (~100 Gbytes/s data flow, 20TB/day)!–  After Level 1 Trigger ~100 Gbytes/s (rate ~O(100) kHz) reach the

event building (2 stages, ~1600 computers).!–  HLT filter cluster select 1 out 1000. Max. rate to tape: ~O(100) Hz "

⇒  The storage manager (stores and forwards) can sustain a 2GB/s traffic.!

⇒  Up to ~300 Mbytes/s sustained forwarded to the CERN T0. (>20TB/day).!

Detector Front-end

Computing Services

Readout Systems

Builder and Filter Systems

Event Manager Builder Networks

Level 1 Trigger

Run Control

40 MHz

100 kHz

100 Hz

J.A. Coarasa 5


The CMS Online Cluster"–  IT Infrastructure"

J.A. Coarasa 6


IT Infrastructure. The purpose and qualities"

"The IT infrastructure (computing, networking and services) of the CMS Online Cluster takes care of the CMS data acquisition and experiment control. "

•  Autonomous (i.e. independent from all other networks) and provides uninterrupted operation 24/7 on two far apart (~200 m) physical locations, with two control rooms;"

•  Redundant services design!–  Losing 1 min of data is wasting accelerator time (worth ~O(1000)CHF/min)."–  Our system serves online data taking! If not taken, data is gone!"

•  Scalable services design to accommodate expansions;"•  Fast configuration turnaround copes with the evolving nature

of DAQ applications and large scale of cluster;"•  Serving the needs of a community of more than 900 Users."

J.A. Coarasa 7


More than 2700 computers mostly under Scientific Linux CERN 5:"–  640 (2-core) as a 1st stage building, equipped with 2 Myrinet and 3

independent 1 Gbit Ethernet lines for data networking. (1280 cores);"–  1008 (720 (8-core) + 288 (12-core allowing HT)) as high level trigger

computers with 2 Gbit Ethernet lines for data networking. (9216 cores);"–  16 (2-core) with access to 300 TBytes of FC storage, 4 Gbit Ethernet

lines for data networking and 2 additional ones for networking to Tier 0;"

Computing. Variety of roles"

J.A. Coarasa 8

Hig

h ba

ndw

idth

netw

orki

ng


More than 2700 computers mostly under Scientific Linux CERN 5:"–  640 (2-core) as a 1st stage building, equipped with 2 Myrinet and 3

independent 1 Gbit Ethernet lines for data networking. (1280 cores);"–  1008 (720 (8-core) + 288 (12-core allowing HT)) as high level trigger

computers with 2 Gbit Ethernet lines for data networking. (9216 cores);"–  16 (2-core) with access to 300 TBytes of FC storage, 4 Gbit Ethernet

lines for data networking and 2 additional ones for networking to Tier 0;"–  More than 400 used by the subdetectors;"–  90 running Windows for Detector Control Systems;"–  12 computers as an ORACLE RAC;"–  12 computers as CMS control computers;"–  50 computers as desktop computers in the control rooms;"–  200 computers for commissioning, integration and testing;"–  15 computers as infrastructure and access servers;"–  250 active spare computers;"

!⇒ Many different Roles!

Computing. Variety of roles"

J.A. Coarasa 9

Hig

h ba

ndw

idth

netw

orki

ng


Networking.Overall Picture"

CMS Networks:"–  Private Networks:"

•  Service Network ""(~3000 1 Gbit ports);"

•  Data Network ""(~4000 1Gbit ports)"

–  Source routing on computers"–  VLANs on switches "

•  Central Data Recording ""(CDR). Network to Tier 0."

•  Private networks for Oracle RAC"

•  Private networks for subdetectors"

–  Public CERN Network"

"J.A. Coarasa 10

CMS Networks

CM

S Si

tes

Computer gateways

Readout, HLT Control…

Firewall

Internet

Service Network"

Data Networks

CDR Network

CERN Network

Storage Manager


The Network Attached Storage"

•  Our important data is hosted in a Clustered Network Attached Storage:"–  User home directories;"–  CMS data: calibration, data quality

monitoring…;"–  Admin data."

•  2 NetApp filer heads in failover configuration (in two racks);"

•  With 6 storage drawers (2 of them in mirror) with internal Dual Parity RAID;"

•  Snapshot feature active (saves us from going to Backup)."

•  Deduplication active (saves ~25-55%)"•  Tested throughput > 380 MBytes/s."

J.A. Coarasa 11

4Gbit 2x10Gbit redundant

routers

shelves

heads

NAS


IT Structural Services. Redundancy and Load balancing. The Concept"

•  Pattern followed:"–  1 master + N slave/replicas ""(now N=3 for most of the services) ""hosted in different racks;""⇒Easy scalability.""⇒Needs replication for all services."

–  Services working under DNS alias ""where possible.""⇒Allows to move the service.""⇒No service outage."

–  Load balancing of primary server ""for client:"

•  DNS Round Robin;"•  explicit client configuration ""segregating in groups of computers."

J.A. Coarasa 12

1 master

N slave/replicas

Explicit client configuration segregating in groups of computers


IT Structural Services. Replication and Load balancing. Basic Services. "

•  Critical data local to the servers (hosted in RAID 1 protected servers);"

•  Manual load balancing through “in groups” segregation."

J.A. Coarasa 13

Service! replication through! Load Balancing through!DNS! named" Round Robin"

DHCP! in-house scripts" No. First who answers"Kerberos! in-house scripts" explicit segregation"

LDAP! slurpd" explicit segregation"NTP! -" explicit segregation"

syslog! -" No. Single server, used for stress test purposes"

Nagios monitoring! -" explicit segregation"


CMS Management and Configuration Infrastructure: The tools"

•  IPMI (Intelligent Platform Management Interface) is used to manage the computers remotely:"–  reboot, console access,…;"

•  PXE and anaconda kickstart through http are used as bootstrap installation method;"

•  Quattor (QUattor is an Administration ToolkiT for Optimizing Resources) is used as the configuration management system;""⇒All Linux computers configured through it or rpms distributed with it (even the Quattor servers themselves): BIOS, all Networking parameters…"

J.A. Coarasa 14


CMS Management and Configuration Infrastructure: Quattor implementation"

•  Based on Quattor 1.3 (cdb 2.0.4, swrep 2.1.38, PANC 6.0.8)"

•  Manages ~2500 installed computers in ~90 types!⇒One fully reinstalled computer in 9-30 min"⇒Big (~Gbyte) change cluster-wise in less than 25 min;""Small change only few minutes."

•  Uses in-house:!–  restricted format in templates: “hierarchical”+other conventions;"–  areas to define subdetector software and versioning in them;"⇒Allowed in-house easy developments:"•  Template summarizer/“inventory maker”"•  Dropbox for rpms"•  Template updater"

""

J.A. Coarasa 15


Monitoring infrastructure"•  We monitor 2500 hosts, 25000 services"

–  The usual things (ping, ssh, disk usage…)"–  In house:"

•  Electronics modules working?"•  Myrinet links working?"•  Quattor spma finished properly?"•  …"

•  Based on Nagios. 1+3 servers with manually split conf."•  Performance:"

–  Every 1 minute latency for the ping test;"–  Every more than 5 minutes for the rest of the services."⇒Needed improving. About to move to production the new version

of monitoring."

J.A. Coarasa 16


New Monitoring infrastructure"•  Addresses issues:"

–  Easy configuration"–  Scalability"

•  Based on icinga (1 server + N workers)"–  mod_gearman: allows to split functionality (Master/Worker)."–  check_multi: groups checks, can cascade."–  PNP4nagios: provides performance data."–  rrdcached: caches files and improves performance."

•  We monitor more services grouped in less checks and get historical performance data"

•  Performance:"–  Every 1 minute latency including performance data."

•  Adding system to notify on a fine grain self-enroll basis"

J.A. Coarasa 17


Databases"•  Save all bookkeeping data:"

–  configuration of the detectors;"–  conditions for the experimental data

taken…"•  Oracle 11.2.0.3 (just migrated, still on compatibility

mode to 10.2.0.5) "–  hosting two databases @ CMS

totalling ~30 Tbytes (more, including replicated/synced ones @ CERN);"

–  hardware:"•  6 blades + standby nodes;"•  netApp filer as a backend for storage;"•  Completely redundant at hardware level."

J.A. Coarasa 18


Hardware Tracking system"Our Hardware Tracking system (HTS) allows us to know everything about the systems (soft/ hardware)"It is based on:"•  OCS (www.ocsinventory-ng.org)"

–  Inventory system"–  Counts everything related to a machine"

•  GLPI (www.glpi-project.org)"–  Tracking system"–  Works in relation with OCS"–  Set of tools to maintain machines history"

J.A. Coarasa 19


The CMS Online Cluster"– Operation""

J.A. Coarasa 20


Constraints and implications"

•  Man power (~6 FTE):""⇒Needs effort on implementing scripts and automation:"

•  Computers will monitor the temperature and shutdown when too high;"•  Scripts to Check and fix serviceʼs problems."⇒Allows faster recovery after unexpected power cut"

"⇒Act only as a second level support in 24/7 operation.""⇒Try to interact with users through a ticketing system (savannah)"

•  Cluster 5 years old and still growing, adding resources as needed"–  Some hardware out of warranty"–  Some hardware with extended warranty"

•  Computers connected to the electronics need swift replacement (1h) if broken. ""⇒Need for standby spares in location.""⇒Fast turnaround in reinstallation or reconfiguration. "

•  Other computers, dealt with fault tolerant software and reconfiguration. Enough to change it the next day."""J.A. Coarasa 21


Operation Emphasis on continue running"

•  Learning from previous failures"•  Preventing service failures before they happen"

–  Redundancy where possible"•  Added full redundancy to the new database"•  Added redundancy to the control system (DCS)"•  Virtualization explored to add redundancy to other services"

–  Preventive campaigns carried out"•  Battery replacement on old computers"•  aging SAS controller (still under warranty) replacement (THS priceless)"•  Clean up/reinstallation of the whole cluster beginning of 2011"•  Clean up/selective beginning of 2012"

•  Identify the failures before they affect the running:"–  Carried out tests on the redundancy of the services"–  Improved the monitoring system"–  Scripts reporting problems before they affect (SAS controller)"

•  Testing before deployment"–  2 Testing and integration areas and 2 more being built"

"J.A. Coarasa 22


Operation Large Scale Effect on Failures"

Failure Happens! For the last year of operation (2011):"•  Unscheduled power cuts: ~1 every 2 months"

–  By far the worst to recover from. Affects also the detectors electronics."

•  Unscheduled cooling cuts: ~1 every few months"•  Network switch failures: ~ 1 every 3 months"

–  Database Failure"–  Failure of redundancy of kerberos and ldap (in the way of

being solved)"–  Affected the storage manager (still running under reduced

bandwidth"•  Computer failures: ~3 every week"

–  SAS aging problem responsible for most of them till exchange"

"J.A. Coarasa 23


Summary"•  A scalable cluster, 2700 computers

growing to 3000 in the next months"•  autonomous,"•  with service redundancy, "•  capable of more than 100GB/s data

throughputs, "•  has been working successfully since

2007.!

J.A. Coarasa 24


""""""""

Thank you. Questions?"J.A. Coarasa 25


Backup slides"

J.A. Coarasa 26


CMS Networks Configuration"•  Service Network (.cms)"

–  Service IPs "•  DHCP (fixed for central servers)"

–  IPMI IPs"•  Fixed at boot time from the DNS name"(in house development that fixes IPMI and BIOS

parameters)"

•  Data and Central Data Recording Networks"–  IPs and routing rules"

•  Fixed at boot time from DNS (A and TXT records)"(in house development: auto_ifconfig)"

J.A. Coarasa 27

x

CMS networks"

J.A. Coarasa 28

x 80 x 8

Readout Unit

Myrinet Switches

Frontend Detector Readout

x 8

Force 10

2x1G

bit

3x1Gbit

x (90+36) x 8

Builder Filter Unit

x 16

Storage Manager

4x1Gbit 4x3Gbit

HP switch

2x10Gbit

Tier 0 Force 10

redundant HP switch


Installation/Configuration Services. Replication and Load balancing"

•  Data being read from the NAS once and kept in memory cache from computers.""⇒The NAS is not loaded even at peak installation time when the installation servers’ network is saturated."

J.A. Coarasa 29

Service! replication through! Load Balancing through!PXE boot/TFTP !

No replication. NFS mount from the NAS."

Through DHCP"Kickstart Installation

server!DNS Round Robin"

yum repository Installation server!

DNS Round Robin"

Quattor repository Installation server!

DNS Round Robin"

Quattor cdb configuration server!

No. "


CMS IT Security"•  We are behind the CERN firewall, and services are not

reachable from the internet, unless explicitly allowed."•  The CMS Networks are private networks: not even

reachable from the CERN network."•  There is a Reverse Proxy, that requires authentication,

to access some internal information (selected http traffic)."

•  All users have to log in into gateway computers submitted to tighter security (ssh traffic to some computers). "

•  Traffic from internals computers to the outside world goes through proxying servers."

J.A. Coarasa 30


The Compact Muon Solenoid experiment (CMS). Design Parameters."

J.A. Coarasa 31

Detector Channels Ev. Data Pixel 60000000 50 (kB) Tracker 10000000 650 Preshower 145000 50 ECAL 85000 100 HCAL 14000 50 Muon DT 200000 10 Muon RPC 200000 5 Muon CSC 400000 90 ⇒  >50 000 000 Channels,

~630 data origins ⇒  Average of 1 Mbyte/event

(bigger for HI ~20 Mbytes/event)

Detectors


pp collisions in CMS"

FM – TIPP2011 32


Pb-Pb collisions in CMS"

FM – TIPP2011 33


Networking. Service Network Redundancy"

•  Consists of networking in two physical locations:"–  Each switch in each rack is

connected to 2 routers;"–  The 2 routers are connected in

a redundant failover configuration with 10 Gbit lines to the routers in the other location."

J.A. Coarasa 34

2x10Gbit redundant

1Gbit 1Gbit

1Gbit 1Gbit

SCX5

USC

NAS

the cms online cluster: setup, operation and maintenance of an evolving cluster

Technology

online data

data networking

cms control computers

gbytess data

academia sinica

evolving cluster

gbit ports data network

overall picture cms