Simulations and Prototyping of the LHCb

L1 and HLT triggers

Tara Shears

For A. Barczyk, J.P. Dufey, B. Jost, T. Kechadi, R.McNulty, N. Neufeld, T. Shears

On behalf of the LHCb Collaboration

Overview

• Introduction to LHCb trigger and data transfer requirements

• Network simulation studies• Evaluating performance in software• Evaluating performance in hardware• Conclusions

LHCb Trigger System

Must reduce 10MHz visible interaction rate 200 Hz output rate

3 level trigger using hardware + software:L01 MHz (~4 s)L1 40 kHz (~50 ms)HLT 200 Hz

Average event size - L1 / HLT

5 kB / 40 kB

Events per MEP - L1 / HLT

25 / 10

Total bandwidth of readout network

7.1 GB/s

Data sources – L1 / HLT

126 / 323

Data destinations (SFC)

94

At L1, 125kB must be received each 25 s by one of the destinations

Data FlowLevel 0 trigger

1MHz

L1 / HLT Farm

MultiplexingLayer

FE FE FE FE FE FE FE FE FE FE FE FE

Switch Switch

Level-1Traffic

HLTTraffic

126Links

44 kHz5.5 GB/s

323Links4 kHz

1.6 GB/s29 Switches

32 Links

94 SFCs

Front-end Electronics

Gb EthernetLevel-1 Traffic

Mixed TrafficHLT Traffic

94 Links7.1 GB/s

TRM

Sorter

TFCSystem

L1-Decision

StorageSystem

Readout Network

Switch Switch Switch

SFC

Switch

CPUCPU

CPU

SFC

Switch

CPUCPUCPU

SFC

Switch

CPUCPU

CPU

SFC

Switch

CPUCPU

CPU

SFC

Switch

CPUCPU

CPUCPUFarm

62 Switches

64 Links88 kHz

~1800 CPUs

Big disk

Methods of evaluating trigger networks

Software simulation:– Parametrise dataflow, switch response

predicted system behaviour – Two approaches; Custom (models specific

network topology), Ptolemy

Hardware simulation:– Evaluate network / switching response in situ– Can examine microscopic or macroscopic

properties of system

Software Simulation Studies

Objectives:

Simulate performance of switch + cpu network arrangement

Predict data loss, latency, network robustness

Provide facility for testing alternative network arrangements

Approach:

Parametrise data packing / transfer with specified network to study response in custom simulation

Develop alternative, flexible simulation to allow other networks to be studied (Ptolemy)

Simulation; Custom code

Existing custom code:Implemented in C LHCb TDR network

configuration modelled and response studied (LHCb_2003_079)

Network simulated by parametrising data packing / queuing

sources destinationsmultiplexors

Crossbar switches

Custom simulation results

L1 latency for events in subfarm:

• TDR network configuration modelled

• 25 events in a MEP

• Simulated processing times cut off at 50 ms

If processing time cut at 30 ms < 0.5% of events lost.

multiplexors

Simulation; Ptolemy

Ptolemy:Freeware Java based

packageUses ‘actors’, self

contained units that execute simple tasks when called upon

Program then depends upon the way in which these actors are linked together in the graphical editor

Crossbar switches

Ptolemy simulation results

Investigate buffer occupancies:– Change inter-trigger

delays – Change data routing

source – destination– Essentially L1 data flow.

• Results of custom code/Ptolemy identical so far Facility available to test

any network configuration

Custom

Ptolemy

(Example buffer occupancy)

Hardware simulation studies

Switch fabric and logic – hard to simulate

Another approach is to characterise performance in hardware- Can evaluate each network component

separately- Or evaluate performance of given network

configuration (cpu + ethernet link + switch)- Results:

- parameters for input to software simulation- study errors / data loss for LHCb traffic patterns- performance characterisation in own right

Testing switch behaviour

Data source: – PCI card with NP4GS3

network processor, 3 GbE ports per card.

– Programmable data patterns– Up to 24 sources available

Synchronisation:– frame rate: up to ~75kHz– Synchronisation: O(100ns)

Software control:– Use Python interface– RH9 + 2.6.0 kernel

Connect ≥24 port switch to data sources to test switch characteristics

DST

SRC SRC

SW

125MB/s

125MB/s125MB/s

+ …. ( 24)

Results

Check buffer occupancies, switch latencies, packet loss and error rates.

Eg. Buffer occupancy

4MB (as expected) for this switch

Eg. Switch latency

linear, switching time negligible wrt.L1 latency for this switch

Studies ongoing

Large-scale network behaviour

Hardware:

100 3GHz cpu (source or dest.)

500 Mb/s cpu ethernet links

168 port switch (1 Gb/s links)

Software:

Synchronise data sends / receives with MPI

RH9 2.4.24 kernel

Tests:

Measure data transfer rate

Investigate different traffic patterns

500 Mb/s

Results

~LHCb L1 rate

(1 kB sent per source takes 1.25 ms for 50 destinations to receive)

Median receive time for 50 destinations [s]

Source data size [B]

50 sources send synchronously to each destination in turn– Find median time for data to

be received at all 50 destinations

– Each experiment repeated 1000 times

– Different data sizes testedResults:

– Measurements limited by system bandwidth

– Data transfer rates ~2.8 GB/s achieved at LHCb L1 timings (1KB/25s per source)

• Studies ongoing

Conclusions

• Software simulation:– Simulations of LHCb trigger network developed to

evaluate response / performance– LHCb TDR network parametrised and studied in custom

simulation– Alternative Ptolemy-based simulation developed

• Hardware simulation:– Allow simulation of network response where details of

fabric unknown– Testbed developed to analyse switch characteristics – Large scale testbed devised to study general network

response

Backup

Example: Building a switch

• Implementation– Routing– Output queuing– Timing– Buffer occupancy– Scaleable

Banyan Network Topology

Multiplexors Crossbars DestinationsSources