daq for test&commissioning at point 2

DAQ for Test&Commissioningat Point 2

Klaus Schossmaier

CERN PH-AID

ALICE TPC Collaboration Meeting

Heidelberg, Germany

12-13 February 2004

TPC Collaboration meeting, 12-13 Feb 2004

2DAQ Test&Commissioning at Point 2

Planning Parameters

System: DAQ system for Test&Commissioning

(= LHCC milestone)

Size: 12 DDLs for TPC8 DDLs for others

Place:LHC Point 2

Date: Q1 2005

People: ALICE DAQ team



DAQ Installation at Point 2

SXL HallSX Hall

UX25 (Experimental Area)

PX24/CR1 (DAQ)

PX24/CR2 (HLT)

SR Hall

Access Shaft

DDLLAN

(ALICE sub-detector assembly)(Networking)

PX24/CR3 (DCS)PX24/CR4 (Misc.)

LAN

ACR WR1 WR2



Installation at PX24/CR1

10 LDCs (Local Data Concentrator):

rackmount PCsU1, U2, U4 heightequipped with 20 D-RORC cards DDL readout, sub-event building

HLT interface

4 GDCs (Gobal Data Collector):

rackmount PCsU1 height equipped with FC cards full-event building

1 DSS (DAQ Server Services):

rackmount PCU4 heighthot-swap SCSI disks RunControl, Logger

On-line Monitoring

1 KVM switch

1 FC switch

1 TDS (Transient Data Storage):

rackmount disk arrayU2 heightIDE of FC disks event buffering

GE switches

PDS (Permanent Data Storage)

CERN computer center

20 DDLs

DDL patch

panels

HLT (HighLevel Trigger)

PX24/CR2



Installation Issues

Hardware/Mechanical issues: Equipment: rackmount PCs, switches, disk array, … Rack assembly, DDL installation, cabling, … Infrastructure: power supply, networking, cooling, …

Software/Testing issues: Operating System: Linux + customizations DAQ Software: DATE + configuration Standalone tests Integration tests

Reference System in our AID lab: small, complete DAQ system with 6 DDLs to spot problems in advance to tackle problems during operation



Reference Setup – Front

KVM switch:

Raritan Paragon UMT2161

4x LDCs:

- 1U, 2U, 4U height- dual Xeon- 6x D-RORC cards DDL

2x GDCs:

- 1U height- dual Xeon- Qlogic QLA2310F cards

2x disk arrays:

- Infortrend IFT-6330

- DotHill SANnet II- Fibre Channel

DSS:

- 4U height- quad Xeon- 3x 36GB SCSI disks

L3 rack

2x GB Ethernet switch:

3COM SuperStack 3

Fibre Channel switch:

Broacade SilkWorm 3800

DDLs:

- to the DDL lab



Reference Setup - Rear

Power distributor

Cat5 cables:

- Ethernet- KVM- RJ45 connectors

Fiber Channel cable:

- 2 Gbit/s multimode - LC-LC connectors

L3 rack

Mounting rails (!)



DDL & D-RORC

DDL link card

Uniform half-CMC card for both SIU (= Source Interface Unit) and DIU (= Destination Interface Unit)

2.125 Gbit/s components (Agilent) for 200 MB/s bandwidth

bridge up to 200 meters

DDL patch panel with D-RORC

Two D-RORC types: 1 DDL link card plugged in, 2 integrated DDL channels

PCI 64 bit, 66 MHz

ALTERA FPGA, IP core from PLDA

200 MB/s throughput with DATE



Rackmount PCs

Evaluation/Benchmarking item LDC GDC DSS

Rackmount height 1U, 2U, 4U 1U 4U

CPU chipsetmotherboard

dual Xeon 2.4 GHzE7501

Supermicro X5DPE-G2

dual Xeon 2.66 GHzE7501

Supermicro X5DPI-G2

quad Xeon 1.5 GHzServerWorks GC-HE

Intel SRSH4

Main memory 2 GBDDR 266 MHz

2 GBDDR 266 MHz

4 GBDDR 266 MHz

Ethernet dual Gigabit Ethernet on-board

dual Gigabit Ethernet on-board

Gigabit and Fast Ethernet on-board

Disk 40 GB IDE 40 GB IDE hot-swap 36 GB SCSI disks

PCI slots up to 6 PCI-X for D-RORC cards

1 PCI-Xfor Fibre Channel card

6 PCI-X2 PCI 5V

Linux installation Red Hat 7.3 Red Hat 7.3 Red Hat 7.3 (rather old)



Disk Array Systems

front view

(IFT-6330)

rear view

(IFT-6330)

Infortrend IFT-6330-12F2D:

12x 300 GB IDE-based disks

dual 2 Gbit/s Fibre Channel ports

Embedded manager via front panel, RS232C, or IP

2x logical volumes: each one has1.1 TB, RAID5, with 5 disks

DotHill SANnet II 200:

12x 36.7 GB FC-based disks

dual 2 Gbit/s Fibre Channel ports

RAID configurations via RS232C



DAQ Software: DATE

DATE (= Data Acquisition and Test Environment)is the software framework of the ALICE DAQ.

Runs on Linux platforms (e.g. Red Hat 7.3) Released (latest is version 4.7) and documented DATE evolves with requirements and technology

Features/Packages: Dataflow: DDL readout, event building System configuration and Run control Monitoring: on-line performance, on/off-line data quality Utilities: message logger, control/configure front-end electronics, …



Readout with DATE

TPC RCU 3 with SIU

D-RORC with DIU

DDL

FEC

LDC

Step 0: Hardware is in place

Step 1: Run Linux with the physmem and the rorc driver

Step 2: Install DATE with a general configuration for the LDC

Step 3: Set up configuration file to enter parameters like D-RORC serial number, channel number, page size, checks, etc.

Step 4: Start/Stop the acquisition

Step 5: Analyse the raw data files, e.g. /tmp/run123.000, /tmp/run123.001, …



DATE Run Control



FEE Configuration/Control

TPC RCU 3 with SIU

D-RORC with DIU

DDL

FEC

LDC

Step 0: Hardware is in place

Step 1: Run Linux with the the rorc driver on the LDC

Step 2: Write a script (ASCII file) for the FeC2 interpreter

Step 3: Run this script, e.g. type FeC2 123 example.fec2

# send command, 19 bit information

write_command 0x10f

# read status, 19 bit address, 19 bit reply

read_and_print 0x100 “Register XYZ: 0x%x”

# download data block, 19 bit address, file

write_block 0x600 pedestal.hex “%x”

# verify data block, 19 bit address, file

read_and_check_block 0x600 pedestal.hex “%x”

stop_if_failed -1

S/N of D-RORC



On-line Monitoring

MOOD Framework: DATE + ROOT environment Interfaces to detector code

MOOD Applications: Visualization, e.g. TPC sector

test data (Hall 167) Data integrity Detector performance

Execution: DSS machine



DAQ Performance

ALICE Computing Data Challenge:

1750 MB/s event building without recording, 5 days

280 MB/s event building and recording to CASTOR, 7 days

D-RORC readout of one DDL channel with DATE:

206 MB/s readout only

118 MB/s readout with event building



DAQ Installations

DAQ System Location DDL LDC GDC Networking Storage

Development / Evaluation AID lab 2 4 4 Gb Ethernet disk

DDL Development DDL lab 2 1 0 Gb Ethernet disk

Detector Groups (e.g. SDD, HMPID, TOF, Muon, TRD)

Institutes 1 1 0 Fast/Gb Ethernet Disk

TPC Sector Test § PS T10 2 1 0 Fast Ethernet disk or CASTOR

ALICE Data Challenges CERN IT 0 80 80 Gb Ethernet TDS + CASTOR

Reference Setup AID lab 6 4 2 Gb Ethernet TDS

Test&Commissioning Point 2 20 10 4 Gb Ethernet TDS + CASTOR

Final System Point 2 395 185 50 Gb Ethernet TDS + CASTOR

§ Given that the RCU 3 is fully functional in time, see Joachim’s presentation.



Summary + Future Work

Objective: DAQ system for Test&Commissioning at Point 2 Surface tests of ALICE sub-detectors (e.g. TPC sector) Operational in Q1 2005

Installation Layout: 20 DDL links between the PX24/CR1 counting room and the

SXL mounting hall L3 racks: DDL patch panels, 15 rackmount PCs, 1 disk array,

switches for Gigabit Ethernet, Fiber Channel and KVM DAQ Reference System in the AID lab

Future work concerning TPC and DAQ: Bergen: Development of RCU 3 firmware for the DDL interface Luciano’s lab: Integration test of RCU 3 with the DDL TPC sector test at T10§: DAQ chain with RCU 3 + DDL + DATE

daq for test&commissioning at point 2

Documents