cern - european laboratory for particle physics desy november 2, 1998 pc farms at cern frédéric...

DESY November 2, 1998

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PC Farms at CERN

Frédéric HemmerCERN-IT/PDP

DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 2C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

Disclaimer

This will cover farms which imply an involvement of CERN’s computer center.

There are other farms in strict online environments or “private” farms in building.


ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

Overview

Off line farms• Linux farms• NT farms• Issues

PC Technology & Performance Online Farms & quasi online farms Cost of ownership Conclusions


ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

Linux Farms - Nomad

Proof of concept in Summer 97 Straight NQS port SHIFT SW client port CERNLIB port NOMAD observed a quasi linearity with

clock frequency compared to Alpha’s !!!• I.e. Alpha@266 MHz = PII@266 MHz

Now 17 PC’s dual, 3 types of MB

0

200

400

600

800

1000

1200

1400

1600

1800

2000

Cern Units

3Q97 4Q98 1Q98 2Q98 3Q98

NOMAD Installed Capacity


ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

Linux Farms - NA49

NA49 already deployed privately a PC farm in their premises

Request a new farm to be deployed in order to benefit from the computer center infrastructure (people and equipment …) in 1 H98

Trivial deployment, running with NQS Most PC’s are branded PC’s (HP) Now completely off RISC for CPU 18 DUALS @ 300->400 MHz


ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

NA49 Analysis - data access

SONY DMSSONY DMS

UnixUnixServerServerUnixUnixServerServerUnixUnixServerServer

CORECORETapeTape

ServersServers

HiPPIHiPPI

From experimentFrom experiment10-12 TB / month10-12 TB / month

1 month/year1 month/yearManual FeedManual Feed

100 GB Cartridges100 GB Cartridges

HPHPK260K260

HPHPK260K260

HPHPK260K260

HPHPK260K260

HPHPK260K260

FDDIFDDI

600 GB600 GB1 Run1 Run

PCPCPCPCPCPCPCPCPCPCPCPC

100BT

100BT

SGISGIChallengeChallenge


ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

Linux Farms (NA48)

NA48 was using the QSW CS/2 (128 proc.)

CS/2 overload -> investigate PC’s in late 97

Installation of 12 Dual machines in 1Q98 and more ...


ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

Linux Issues

EEPRO 100 B MP crashes AFS support (MP) NFS support (MP) Commercial software Manufacturer support for Linux Very few Linux experts


ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

NT offline Farms

PCSF• Simulation facility but …

COMPASS• Evaluating & benchmarking technology

DESY November 2, 1998Frédéric Hemmer CERN-IT/PDP 1

0

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PCSF - Overview

Configuration Applications Data access Specific work & solutions Key issues Conclusions


1

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PCSF - Goals

Make PC+NT a standard option for Physics Data Processing, starting with simulation

Establish a minimum management model for NT farm management

Address scalability issues Gain Windows NT experience


2

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PCSF Milestones

Joined RD47 in Autumn 96 Price inquiry issued in 12/96 Hardware delivered 4/97 Ready to use 6/97 RD47 report 10/97 Expansion 5/98


3

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PCSF Configuration (1) Server running NT 4.0 Server SP3

• 1 dual capable Ppro @ 200 MHz, 96 MB, with 9 GB data disk (with mirroring). LSF central queues.

Server running NT Terminal Server Beta 2• 1 dual Ppro @ 200 MHz, 128 MB, with 4 GB data

disk. Runs IIS 3.0 and is accessible from outside CERN. It also host the asp’s for Web access

Servers running NT 4.0 Workstation SP3• 9 dual Ppro’s @ 200 MHz, 64 MB, 2*4GB • 25 dual PII’s @ 300 MHz, 128 MB, 2*4GB

All equipped with boot proms


4

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PCSF Configuration (2)

Machines interconnected with 4 3com 3000 100BaseT switch

Display/Keyboard/Mouse connected to a Raritan multiplexor

PC Duo for remote admin access There were problems with other products All running LSF 3.0. LSF 3.2 does not work, support weak Completely integrated with NICE


5

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Applications on PCSF

ATLAS Dice simulation NA45 1996 reconstruction CMS reconstruction with Objectivity being

tested LHCB simulation code ready ATLAS reconstruction being ported ATLAS/Marseille event filter prototype

scalability tests


6

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Data access

NT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PCNT PC

Network

Network

Unix RFIOUnix RFIOServerServer




Unix TapeUnix TapeServerServer

stagexxx commandsstagexxx commands

RFIORFIO


7

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

ATLAS Level 3 DAQ

Processor FarmProcessor Farm

Event BuilderEvent Builder

SFISFISFISFISFISFI

Storage (100 MB/s)Storage (100 MB/s)

Readout BuffersReadout Buffers

1 GB/s1 GB/s


8

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

ATLAS Event Filter

Testbed for evaluating algorithms & sizing

Architecture & simulation studies Monitoring, system management,

feedback, etc… Interface prototypes (SFI, SFO) Timescale : prototype -1 (I.e. end 98) Status : sizing of an initial farm


9

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PCSF Usage

0

1000

2000

3000

4000

5000

6000

7000

8000

43 45 47 49 51 1 3 5 7 9 11 13 15 17 19 21 23 25 27 29 31 33 35 37 39 41

Week #

NC

U h

ou

rs

Idle

Used


0

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s


1

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Specific work so far

Installation (Remote Boot, Winstall, NICE replica’s, Install Server)

User codes, CERNLIB, SHIFT Job Starter PC MGR WNTS Web Interface


2

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Installation Disk cloning + change SID Fastest method, but not very

automated Remote boot

• Remote boot install procedures with virtual disk

• Use unattended setup, installs Winstall and other things

• Third party packages installed through Winstall

boot prom support on some hardware


3

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Porting

Usually porting code from Unix to NT is easy (NA45 code ported in 1 week)

Usually porting production environment from Unix to NT is difficult (shell scripts)

Porting build environment is difficult, better to use native tools (Dev Studio)

Mixing Unix and NT build environment, revision control, etc.


4

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Jobstarter

Initially inherited from Unix LSF CERN JobStarter

Rewritten in C++, using PcMgrSvc for drive mapping

Check execution preconditions Clean up normal and abnormal job end Kill popup dialog windows Excel & Winzip in batch


5

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PcMgrSvc/Ctl

Checks• Status of monitored processes/services• Amount of scratch space• Drive mapping(s)

Map/Unmap drives Sync. with time servers Generate alarms on request Gets all parameters from registry


6

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Web Interface

As a solution to• Remote access from outside CERN• Access from non NT hosts

Implemented as ASP’s with VB Requires IIS on the server


7

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s Web Interface -

authentication


8

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Web Interface - Overview


9

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Web Interface - bjobs


0

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Web interface - bjobs result


1

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Windows NT Terminal Server


2

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Next Steps

Finish and understand remote boot issues Complete remote boot - remote install AFS Integration Build up resilience Investigate how to use the new WfM, DMI,

PXE, ACPI, etc. initiatives Investigate whether WSH is an alternative Investigate NT’s I/O capabilities


3

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Key Issues

AFS access LSF support Boot proms, equipment interoperability CODE reintegration (Physics & CERNLIB) Think Windows Scalability & Management (home grown

solution vs. commercial apps.) Remote & external access


4

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PC with NT

PC+NT has proven to work in batch environment, and is now an option for Physics Data Processing

Farm management is less of a concern after have built a few tools (alternatives would be to use SMS or TNG), but some work is still needed

Scalability has started to be addressed, but the relatively small number of nodes does not help here

Considerable NT experience has been gained


5

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Issues so far

Linux• EEPRO 100 B MP support• Commercial software• Manufacturer support• Very few local Linux experts

NT• AFS access• LSF support

• Think Windows• Remote and external access

PC• Interoperability (cards/MB combination• Remote Boot support


6

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PC Technology evolution in 97 Pentium Pro Pentium II

• 50 % raw performance increase• but 50 % cache performance reduction

SEC new motherboards 440 FX 440 LX (SDRAM, AGP) Recent MB’s embedded SCSI, E’net, VGA 100 Mbit E’net switches standard, 1000

Mbit arriving


7

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s PC Technology evolution in

98 Pentium II @300 MHz Pentium Xeon @ 450 MHz

• MP support• 50 % cache performance increase

Slot 2 new motherboards 440 LX 440 BX, 440 NX (100 MHz, EDO) Recent MB’s No more available through Intel,

TYAN 1000 Mbit/s E’net switches standard, >> 1000

Mbit/s arriving


8

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Racking evolution

1997 1998


2

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Fast Ethernet Switches (Oct. 98)


3

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

At the back of Fast Ethernet Switches (Oct. 98)


4

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Gigabit Ethernet Switches


5

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Network performance: Results PC’s interconnected through 100 BaseT 3Com

3000 switch Repeated with other H/W Half duplex behavior Block size does not matter Linux uses less CPU than NT

Good unidirectional performance Disappointing CPU consumption on NT

Disappointing bi-directional performance


6

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PC to PC Network performance

Linux Windows NT Max

MB/s CPU % MB/s CPU % MB/s

1->1 9->11 17 9->11 40 12.5

1<-> 1 4.6 10 5.3 44 12.57.5 21 5.2 44 12.5

12.1 10.5 251->3 11.7 55 12.5

3.9 20 4.23.9 20 4.23.9 20 4.2


7

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Network performance: issues

Unexplained 0.5 MB/s observed with some eepro100 versions on PCRD hardware, but OK on PCSF

Recent DEC E'net boards with chipset > 21140 give poor performance on Linux

Surprising results PC/Alpha


8

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PC/Alpha Network performance

Linux Alpha DUX Max

MB/s MB/s MB/s

1->1 11.1 11.1 12.5

1<-> 1 6.7 11.1 12.511.1 6.7 12.517.8 17.8 25


9

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PC High Performance Networking

HiPPI (5/98) PII, 300 MHz,

440LX, SDRAM, Roadrunner to SGI O2000, 4 CPU, IRIX 6.4

Transmit: 50 MB/s Receive: 50 MB/s

(53 MB/s with SMP)

Gigabit Ethernet (10/98)

PII, 400 MHz, 440 BX, 100 MHz SDRAM, PCI 32/33, Tigon I

1500 bytes/packet: 28 MB/s, 40% CPU

9000 bytes/packet, 90 MB/s, 90% CPU


0

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Disk performance PC’s connected to SEAGATE ST19171W using

two Adaptec 2940 UW NT needs a lot of tuning (default behavior is to

swap data out!) Block size, BIOS settings, EDO/FPM does not

matter Poor performance

Windows NT even worse Memory bandwidth is suspected


1

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Disk performance

# Streams Linux Windows/NT MaxMB/s CPU % MB/s CPU % MB/s

1 10.5 33 8.5 35 112 21 63 9.2 35 703 21 100 13.5 60 70

• Striping has no effect

•1 stream 2 stripes : 21 MB/s (22 max)

•1 stream 3 stripes : 21 MB/s (33 max)


2

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Disk performance: issues

Memory bandwidth suspected Need to test with LX/SDRAM, BX

SDRAM@100 Mhz RISC PCI does not support variety of

boards Combined disk/network performance

even worse : 5-6 MB/s on Linux


3

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Memory bandwidth (lmbench)

PCRD IBM DEC Prioris DEC PWS

Read MB/s 160 160 216 190Write MB/s 55 55 69 190


4

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Memory bandwidth (lmbench)

0

50

100

150

200

250

300

350

MB/s

Ta

ho

e2

DK

440

LX

Th

un

de

r2

Tig

er2

GA

686

DL

X

GA

686

(CP

U1

)

GA

686

(CP

U2

)

DE

C P

WS

43

3

SU

N U

ltra

5

Th

un

de

r10

0

N4

40

BX

Ka

ya

k X

A's

Co

mp

aq

Pro

lia

nt

16

00

Equipment

Mem read

Mem write


5

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Technology issues

Technology evolves too fast (processors, chipsets, memory, motherboards, networking,...)• Changing environment/interoperability issues• Hard to maintain (obsolescence)• New NIC’s, drivers• Measurements valid only a few months Difficult to establish stable environments

Wide variety of solutions Some combinations work, other not

Local suppliers cannot help to solve problems


6

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PC Performance summary CPU performance fine Network performance

• Some configurations do not work• Some configurations can saturate Fast Ethernet• Recent tests show excellent performance

Memory performance• Now better than low-end RISC

Disk Performance disappointing Linux better than NT


7

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Online and quasi online farms

NA48 Data Recording NA45 Data Recording in Objectivity


8

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

NA48 Central Data Recording

Cisco 5505Cisco 5505

3Co

m 3900

3Co

m 3900

FDDIFDDI

Fast EthernetFast Ethernet


XLNT GbitXLNT Gbit

FDDIFDDI

HiPPIHiPPI

GigaRouterGigaRouter

3Com 93003Com 9300Gigabit EthernetGigabit Ethernet

HiPPIHiPPI

CS/2CS/22.5 TB Disk space2.5 TB Disk space

SUN E450SUN E450500 GB Disk space500 GB Disk space

Event BuilderEvent BuilderOnline PC FarmOnline PC Farm

Sub detectorSub detectorVME cratesVME crates

7 KM7 KM

OfflineOfflinePC FarmPC Farm


9

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

NA 48 Data Recording in 98 May September 1998 Raw Data on Tape

• 68 TB (1450 tapes, mainly 50 GB tapes)• 12.5 TB Selected Reconstructed Data• Total with 97 data : 96 TB

Average Data Rate : 18 MB/s (peaks @ 23 MB/s)

CDR system can do 40-50 MB/s; limitation is CPU Time available

Data recorded as files (4 million)


0

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

NA48 On Line Farm

11 Subdetector PC’s (dual PII-266, 128 MB) 8 Event Building PC’s (dual PII-266, 128 MB, 18 GB

SCSI) 4 CDR routing PC’s (dual PII-266, 64 MB, FDDI) All running Linux Software event building in the interburst gap Optional Software Filter (tags data) Send data to computer center (local disk buffers :

144 GB , 2 hours) On CS/2 : L3 Filtering and tape writing


1

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

NA48 Plans for 1999


Gigabit EthernetGigabit Ethernet

HiPPIHiPPI

4 * SUN E4504 * SUN E4504.5 TB Disk space4.5 TB Disk space

EventEventBuilderBuilder

Sub detectorSub detectorVME cratesVME crates

7 KM7 KM

3Co

m 3900

3Co

m 3900

HiPPIHiPPI3Com 93003Com 9300



Cisco 5505Cisco 5505

On/OfflineOn/OfflinePC FarmPC Farm


2

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

NA45 Data Recording



HiPPIHiPPI

2 * SUN E4502 * SUN E450500 GB Disk space500 GB Disk space

Event BuilderEvent BuilderOn Line PC FarmOn Line PC Farm

Sub detector VME cratesSub detector VME crates

7 KM7 KM

3Co

m 3900

3Co

m 3900

HiPPIHiPPI



SCISCI

3Com 39003Com 3900

3Com 93003Com 9300

NA48NA48

PCSFPCSF


3

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

NA45 Raw Data recording in Objectivity October 98 ; November 98 Estimated bandwidth : 15 MB/s Processes translate Raw Data format to Objectivity Database files (1.5 GB) are closed, then written on

tape Steering done using a set of perl scripts on the disk

servers On line filtering/reconstruction/calibration possible Farm is running Windows NT Reconstruction can use PCSF


4

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s Current & Future Data rates at

CERN

Year Experiments BandwidthMB/s

Raw DataTB/year

ProcessingSPECInt95

1990-2000

LEP 0.5 1 100

1997-2000

SPS 15-20 30-70 500

2000-2008

SPS 35 300 2000

2004- LHC 100-1000 3000 50000


5

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

Summary

On line PC farms are being used to record data at sensible rates (Linux)

Off line PC farms are being used for reconstruction/filtering/analysis (Linux/NT)

Still a lot to do on scalable farm management, global steering, CDR monitoring, etc..


6

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

PC Total Cost of Ownership

42%

51%

4%1%2%

HW

MUX

Rack

Network

Sysadm

• Software not included

• Install labor not included

• Assumes 3 years lifetime


7

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

DEC 8400 (12-Way) Cost of Ownership

• Software & SW maintenance not included

• Assumes 5 years lifetime

72.8%

0.1%13.8%0.9%

12.4%

HW

MUX

HW Maint

Network

Sysadm


8

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s

General Conclusions (1)

PC’s are now used for online, quasi online and offline environments

The “offline” is now part of the online The I/O is still done using RISC/Unix

but recent MP Xeon may change this …


9

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s


PC technology is moving very fast• Good for performance• Not so for stability, interoperability• Not so for understanding issues

The general management of large farms is not solved but …• Number of initiatives/standards/tools may

help us here : WfM, DMI, PXE, ACPI, SMS, TNG, etc.


0

CE

RN

- E

uro

pea

n L

abo

rato

ry f

or

Par

ticl

e P

hys

ics

C

ER

N -

Eu

rop

ean

Lab

ora

tory

fo

r P

arti

cle

Ph

ysic

s


Linux vs. NT … the battle is over• Choose the one suitable to your

application• NT can be used• Linux is usable (and offers more

performance). PC real costs are usually not well

understood

cern - european laboratory for particle physics desy november 2, 1998 pc farms at cern frédéric...

Documents