DESY November 2, 1998DESY November 2, 1998
CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PC Farms at CERN
Frédéric HemmerCERN-IT/PDP
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 22CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Disclaimer
! This will cover farms which imply an involvement of CERN’s computer center.
! There are other farms in strict online environments or “private” farms in building.
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 33CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Overview
! Off line farms• Linux farms• NT farms• Issues
! PC Technology & Performance! Online Farms & quasi online farms! Cost of ownership! Conclusions
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 44CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Linux Farms - Nomad
! Proof of concept in Summer 97! Straight NQS port! SHIFT SW client port! CERNLIB port! NOMAD observed a quasi linearity with
clock frequency compared to Alpha’s !!!• I.e. Alpha@266 MHz = PII@266 MHz
! Now 17 PC’s dual, 3 types of MB
0
200
400
600
800
1000
1200
1400
1600
1800
2000
Cern Units
3Q97 4Q98 1Q98 2Q98 3Q98
NOMAD Installed Capacity
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 55CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Linux Farms - NA49
! NA49 already deployed privately a PC farm in their premises
! Request a new farm to be deployed in order to benefit from the computer center infrastructure (people and equipment …) in 1 H98
! Trivial deployment, running with NQS! Most PC’s are branded PC’s (HP)! Now completely off RISC for CPU! 18 DUALS @ 300->400 MHz
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 66CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
NA49 Analysis - data accessUnixUnixServerServerUnixUnixServerServerUnixUnixServerServer
CORECORETapeTape
ServersServers
HiPPIHiPPI
HPHPK260K260HPHPK260K260HPHPK260K260HPHPK260K260HPHPK260K260
FDDIFDDI
600 GB600 GB1 Run1 Run
PCPCPCPCPCPCPCPCPCPCPCPC
100BT
100BT
SGISGIChallengeChallenge
From experimentFrom experiment1010--12 TB / month12 TB / month
1 month/year1 month/yearManual FeedManual Feed
100 GB CartridgesSONY DMSSONY DMS 100 GB Cartridges
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 77CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Linux Farms (NA48)
! NA48 was using the QSW CS/2 (128 proc.)! CS/2 overload -> investigate PC’s in late
97! Installation of 12 Dual machines in 1Q98
and more ...
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 88CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Linux Issues
! EEPRO 100 B MP crashes! AFS support (MP)! NFS support (MP)! Commercial software! Manufacturer support for Linux! Very few Linux experts
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 99CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
NT offline Farms
! PCSF• Simulation facility but …
! COMPASS• Evaluating & benchmarking technology
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 1010CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PCSF - Overview
! Configuration ! Applications! Data access! Specific work & solutions! Key issues! Conclusions
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 1111CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PCSF - Goals
! Make PC+NT a standard option for Physics Data Processing, starting with simulation
! Establish a minimum management model for NT farm management
! Address scalability issues ! Gain Windows NT experience
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 1212CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PCSF Milestones
! Joined RD47 in Autumn 96! Price inquiry issued in 12/96! Hardware delivered 4/97! Ready to use 6/97! RD47 report 10/97! Expansion 5/98
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 1313CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PCSF Configuration (1)! Server running NT 4.0 Server SP3
• 1 dual capable Ppro @ 200 MHz, 96 MB, with 9 GB data disk (with mirroring). LSF central queues.
! Server running NT Terminal Server Beta 2• 1 dual Ppro @ 200 MHz, 128 MB, with 4 GB data disk.
Runs IIS 3.0 and is accessible from outside CERN. It also host the asp’s for Web access
! Servers running NT 4.0 Workstation SP3• 9 dual Ppro’s @ 200 MHz, 64 MB, 2*4GB • 25 dual PII’s @ 300 MHz, 128 MB, 2*4GB
All equipped with boot proms
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 1414CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PCSF Configuration (2)
! Machines interconnected with 4 3com 3000 100BaseT switch
! Display/Keyboard/Mouse connected to a Raritan multiplexor
! PC Duo for remote admin access" There were problems with other products! All running LSF 3.0." LSF 3.2 does not work, support weak! Completely integrated with NICE
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 1515CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Applications on PCSF
! ATLAS Dice simulation! NA45 1996 reconstruction! CMS reconstruction with Objectivity being
tested! LHCB simulation code ready! ATLAS reconstruction being ported! ATLAS/Marseille event filter prototype
scalability tests
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 1616CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Data access
NT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PC
Network
Unix RFIOUnix RFIOServerServerUnix RFIOUnix RFIOServerServerUnix RFIOUnix RFIOServerServerUnix RFIOUnix RFIOServerServer
Unix TapeUnix TapeServerServer
stagexxxstagexxx commandscommands
RFIORFIO
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 1717CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
ATLAS Level 3 DAQ
Processor FarmProcessor Farm
Event BuilderEvent Builder
# # ## # # # # ## # # # # ## # #
SFISFISFISFISFISFI
Readout BuffersReadout Buffers
1 GB/s1 GB/s
Storage (100 MB/s)Storage (100 MB/s)
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 1818CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
ATLAS Event Filter
! Testbed for evaluating algorithms & sizing! Architecture & simulation studies! Monitoring, system management,
feedback, etc…! Interface prototypes (SFI, SFO)! Timescale : prototype -1 (I.e. end 98)! Status : sizing of an initial farm
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 1919CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PCSF Usage
0
1000
2000
3000
4000
5000
6000
7000
8000
43 45 47 49 51 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41
Week #
NC
U h
ours
Idle
Used
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 2020CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 2121CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Specific work so far
! Installation (Remote Boot, Winstall, NICE replica’s, Install Server)
! User codes, CERNLIB, SHIFT! Job Starter! PC MGR! WNTS! Web Interface
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 2222CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Installation! Disk cloning + change SID" Fastest method, but not very automated! Remote boot
• Remote boot install procedures with virtual disk• Use unattended setup, installs Winstall and other
things• Third party packages installed through Winstall
" boot prom support on some hardware
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 2323CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Porting
! Usually porting code from Unix to NT is easy (NA45 code ported in 1 week)
! Usually porting production environment from Unix to NT is difficult (shell scripts)
! Porting build environment is difficult, better to use native tools (Dev Studio)" Mixing Unix and NT build environment,
revision control, etc.
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 2424CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Jobstarter
! Initially inherited from Unix LSF CERN JobStarter
! Rewritten in C++, using PcMgrSvc for drive mapping
! Check execution preconditions! Clean up normal and abnormal job end! Kill popup dialog windows" Excel & Winzip in batch
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 2525CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PcMgrSvc/Ctl
! Checks• Status of monitored processes/services• Amount of scratch space• Drive mapping(s)
! Map/Unmap drives! Sync. with time servers! Generate alarms on request! Gets all parameters from registry
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 2626CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Web Interface
! As a solution to• Remote access from outside CERN• Access from non NT hosts
! Implemented as ASP’s with VB! Requires IIS on the server
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 2727CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Web Interface - authentication
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 2828CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Web Interface - Overview
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 2929CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Web Interface - bjobs
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 3030CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Web interface - bjobs result
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 3131CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Windows NT Terminal Server
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 3232CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Next Steps
! Finish and understand remote boot issues! Complete remote boot - remote install! AFS Integration! Build up resilience! Investigate how to use the new WfM, DMI,
PXE, ACPI, etc. initiatives! Investigate whether WSH is an alternative! Investigate NT’s I/O capabilities
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 3333CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Key Issues
! AFS access! LSF support! Boot proms, equipment interoperability! CODE reintegration (Physics & CERNLIB)! Think Windows! Scalability & Management (home grown
solution vs. commercial apps.)! Remote & external access
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 3434CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PC with NT
! PC+NT has proven to work in batch environment, and is now an option for Physics Data Processing
! Farm management is less of a concern after have built a few tools (alternatives would be to use SMS or TNG), but some work is still needed
! Scalability has started to be addressed, but the relatively small number of nodes does not help here
! Considerable NT experience has been gained
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 3535CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Issues so far
! Linux• EEPRO 100 B MP support• Commercial software• Manufacturer support• Very few local Linux experts
! NT• AFS access• LSF support
• Think Windows• Remote and external access
! PC• Interoperability (cards/MB combination• Remote Boot support
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 3636CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PC Technology evolution in 97
! Pentium Pro " Pentium II• 50 % raw performance increase• but 50 % cache performance reduction
! SEC " new motherboards! 440 FX " 440 LX (SDRAM, AGP)! Recent MB’s " embedded SCSI, E’net,
VGA! 100 Mbit E’net switches standard, 1000
Mbit arriving
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 3737CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PC Technology evolution in 98! Pentium II @300 MHz " Pentium Xeon @
450 MHz• MP support• 50 % cache performance increase
! Slot 2 " new motherboards! 440 LX " 440 BX, 440 NX (100 MHz, EDO)! Recent MB’s " No more available through
Intel, TYAN! 1000 Mbit/s E’net switches standard, >>
1000 Mbit/s arriving
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 3838CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Racking evolution19981997
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 3939CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
At the back ...
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 4040CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Console multiplexors
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 4141CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Fast Ethernet switches (Sep. 98)
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 4242CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Fast Ethernet Switches (Oct. 98)
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 4343CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
At the back of Fast Ethernet Switches (Oct. 98)
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 4444CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Gigabit Ethernet Switches
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 4545CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Network performance: Results! PC’s interconnected through 100 BaseT
3Com 3000 switch! Repeated with other H/W! Half duplex behavior! Block size does not matter! Linux uses less CPU than NT
" Good unidirectional performance" Disappointing CPU consumption on NT" Disappointing bi-directional performance
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 4646CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PC to PC Network performance
Linux Windows NT MaxMB/s CPU % MB/s CPU % MB/s
1->1 9->11 17 9->11 40 12.51<-> 1 4.6 10 5.3 44 12.5
7.5 21 5.2 44 12.512.1 10.5 25
1->3 11.7 55 12.53.9 20 4.23.9 20 4.23.9 20 4.2
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 4747CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Network performance: issues
! Unexplained 0.5 MB/s observed with some eepro100 versions on PCRD hardware, but OK on PCSF
! Recent DEC E'net boards with chipset > 21140 give poor performance on Linux
! Surprising results PC/Alpha
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 4848CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PC/Alpha Network performance
Linux Alpha DUX MaxMB/s MB/s MB/s
1->1 11.1 11.1 12.51<-> 1 6.7 11.1 12.5
11.1 6.7 12.517.8 17.8 25
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 4949CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PC High Performance Networking
Gigabit Ethernet (10/98)! PII, 400 MHz, 440 BX,
100 MHz SDRAM, PCI 32/33, Tigon I
! 1500 bytes/packet: 28 MB/s, 40% CPU
! 9000 bytes/packet, 90 MB/s, 90% CPU
HiPPI (5/98)! PII, 300 MHz, 440LX,
SDRAM, Roadrunner to SGI O2000, 4 CPU, IRIX 6.4
! Transmit: 50 MB/s! Receive: 50 MB/s (53
MB/s with SMP)
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 5050CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Disk performance! PC’s connected to SEAGATE ST19171W
using two Adaptec 2940 UW! NT needs a lot of tuning (default behavior
is to swap data out!)! Block size, BIOS settings, EDO/FPM does
not matter" Poor performance
" Windows NT even worse" Memory bandwidth is suspected
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 5151CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Disk performance
# Streams Linux Windows/NT MaxMB/s CPU % MB/s CPU % MB/s
1 10.5 33 8.5 35 112 21 63 9.2 35 703 21 100 13.5 60 70
• Striping has no effect
••1 stream 2 stripes : 21 MB/s (22 max)1 stream 2 stripes : 21 MB/s (22 max)
••1 stream 3 stripes : 21 MB/s (33 max)1 stream 3 stripes : 21 MB/s (33 max)
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 5252CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Disk performance: issues
! Memory bandwidth suspected! Need to test with LX/SDRAM, BX
SDRAM@100 Mhz! RISC PCI does not support variety of
boards! Combined disk/network performance even
worse : 5-6 MB/s on Linux
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 5353CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Memory bandwidth (lmbench)
PCRD IBM DEC Prioris DEC PWS
Read MB/s 160 160 216 190Write MB/s 55 55 69 190
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 5454CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Memory bandwidth (lmbench)
0
50
100
150
200
250
300
350
MB/s
Taho
e2
DK44
0LX
Thun
der2
Tige
r2
GA6
86DL
X
GA6
86(C
PU1)
GA6
86(C
PU2)
DEC
PWS4
33
SUN
Ultra
5
Thun
der1
00
N440
BX
Kaya
k XA
's
Com
paq
Prol
iant
160
0
Equipment
Mem readMem write
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 5555CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Technology issues
! Technology evolves too fast (processors, chipsets, memory, motherboards, networking,...)• Changing environment/interoperability issues• Hard to maintain (obsolescence)• New NIC’s, drivers• Measurements valid only a few months" Difficult to establish stable environments
! Wide variety of solutions" Some combinations work, other not
! Local suppliers cannot help to solve problems
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 5656CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PC Performance summary! CPU performance fine! Network performance
• Some configurations do not work• Some configurations can saturate Fast Ethernet• Recent tests show excellent performance
! Memory performance• Now better than low-end RISC
! Disk Performance disappointing! Linux better than NT
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 5757CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Online and quasi online farms
! NA48 Data Recording! NA45 Data Recording in Objectivity
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 5858CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
NA48 Central Data Recording
Cisco 5505Cisco 5505
3Com
39003C
om 3900
FDDIFDDI
Fast EthernetFast Ethernet
Fast EthernetFast Ethernet
XLNT GbitXLNT Gbit
FDDIFDDI
HiPPIHiPPI
GigaRouterGigaRouter
3Com 93003Com 9300Gigabit EthernetGigabit Ethernet
HiPPIHiPPI
CS/2CS/22.5 TB Disk space2.5 TB Disk space
SUN E450SUN E450500 GB Disk space500 GB Disk space
Event BuilderEvent BuilderOnline PC FarmOnline PC Farm
Sub detectorSub detectorVME cratesVME crates
7 KM7 KM
OfflineOfflinePC FarmPC Farm
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 5959CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
NA 48 Data Recording in 98! May " September 1998! Raw Data on Tape
• 68 TB (1450 tapes, mainly 50 GB tapes)• 12.5 TB Selected Reconstructed Data• Total with 97 data : 96 TB
! Average Data Rate : 18 MB/s (peaks @ 23 MB/s)! CDR system can do 40-50 MB/s; limitation is CPU
Time available! Data recorded as files (4 million)
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 6060CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
NA48 On Line Farm
! 11 Subdetector PC’s (dual PII-266, 128 MB)! 8 Event Building PC’s (dual PII-266, 128 MB, 18 GB SCSI)! 4 CDR routing PC’s (dual PII-266, 64 MB, FDDI)! All running Linux! Software event building in the interburst gap! Optional Software Filter (tags data)! Send data to computer center (local disk buffers : 144 GB , 2
hours)! On CS/2 : L3 Filtering and tape writing
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 6161CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
NA48 Plans for 1999
Fast EthernetFast Ethernet
HiPPIHiPPI
4 * SUN E4504 * SUN E4504.5 TB Disk space4.5 TB Disk space
EventEventBuilderBuilder
Sub detectorSub detectorVME cratesVME crates
7 KM7 KM
3Com
39003C
om 3900
HiPPIHiPPI3Com 93003Com 9300
Gigabit EthernetGigabit Ethernet
Fast EthernetFast Ethernet
Cisco 5505Cisco 5505
Gigabit EthernetGigabit Ethernet
On/OfflineOn/OfflinePC FarmPC Farm
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 6262CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
NA45 Data Recording
Fast EthernetFast Ethernet
HiPPIHiPPI
2 * SUN E4502 * SUN E450500 GB Disk space500 GB Disk space
Event BuilderEvent BuilderOn Line PC FarmOn Line PC Farm
Sub detector VME cratesSub detector VME crates
7 KM7 KM
3Com
39003C
om 3900
HiPPIHiPPI
Gigabit EthernetGigabit Ethernet
Fast EthernetFast Ethernet
SCISCI
3Com 39003Com 3900
3Com 93003Com 9300
NA48NA48
PCSFPCSF
Gigabit EthernetGigabit Ethernet
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 6363CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
NA45 Raw Data recording in Objectivity
! October 98 ; November 98! Estimated bandwidth : 15 MB/s! Processes translate Raw Data format to Objectivity! Database files (1.5 GB) are closed, then written on tape! Steering done using a set of perl scripts on the disk
servers! On line filtering/reconstruction/calibration possible! Farm is running Windows NT! Reconstruction can use PCSF
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 6464CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Current & Future Data rates at CERN
Year Experiments BandwidthMB/s
Raw DataTB/year
ProcessingSPECInt95
1990-2000
LEP 0.5 1 100
1997-2000
SPS 15-20 30-70 500
2000-2008
SPS 35 300 2000
2004- LHC 100-1000 3000 50000
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 6565CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Summary
! On line PC farms are being used to record data at sensible rates (Linux)
! Off line PC farms are being used for reconstruction/filtering/analysis (Linux/NT)
! Still a lot to do on scalable farm management, global steering, CDR monitoring, etc..
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 6666CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
PC Total Cost of Ownership
42%
51%
4%1%2%
HWMUXRackNetworkSysadm
• Software not included
• Install labor not included
• Assumes 3 years lifetime
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 6767CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
DEC 8400 (12-Way) Cost of Ownership
• Software & SW maintenance not included
• Assumes 5 years lifetime
72.8%
0.1%13.8%0.9%12.4%
HWMUXHW MaintNetworkSysadm
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 6868CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
General Conclusions (1)
! PC’s are now used for online, quasi online and offline environments
! The “offline” is now part of the online! The I/O is still done using RISC/Unix but
recent MP Xeon may change this …
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 6969CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
General Conclusions (2)
! PC technology is moving very fast• Good for performance• Not so for stability, interoperability• Not so for understanding issues
! The general management of large farms is not solved but …• Number of initiatives/standards/tools may
help us here : WfM, DMI, PXE, ACPI, SMS, TNG, etc.
DESY November 2, 1998DESY November 2, 1998Frédéric Hemmer CERNFrédéric Hemmer CERN--IT/PDPIT/PDP 7070CER
N
CER
N --
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
Euro
pean
Lab
orat
ory
for P
artic
le P
hysi
cs
General Conclusions (3)
! Linux vs. NT … the battle is over• Choose the one suitable to your application• NT can be used• Linux is usable (and offers more performance).
! PC real costs are usually not well understood