atlas trigger & data acquisition system: concept & architecture

Kostas KORDAS

INFN – Frascati

XI Bruno Touschek spring school, Frascati,19 May 2006

Higgs → 2e+2O(1/hr)

Higgs → 2e+2O(1/hr)

~25 min bias events ( >2k particles )

every 25 ns

~25 min bias events ( >2k particles )

every 25 ns

ATLAS

Trigger & Data Acquisition

system:

concept & architecture

ATLAS

Trigger & Data Acquisition

system:

concept & architecture

XI Bruno Touschek school, Frascati, 19 May '06

ATLAS TDAQ concept & architecture - Kostas KORDAS 2

LHCTeVatron

Process (pb) N/s N/year

Total collected before start of LHC

W l 3104 30 108 104 LEP / 107 FNAL

Z ee 1.5103 1.5 107 107 LEP

t t 830 1 107 104 Tevatron

b b 5108 106 1013 109 Belle/BaBar ?

Low lumi = 10 fb-1/y

ATLAS Trigger & DAQ: the need (1)ATLAS Trigger & DAQ: the need (1)

Total cross section is at ~100 mb, While the very interesting physics is at

~1 nb to ~1 pb,i.e., a ratio of 1:108 to 1:1011



ATLAS Trigger & DAQ: the need (2)ATLAS Trigger & DAQ: the need (2)

40 MHz

~ 200 Hz ~ 300 MB/s

Full info / event: ~ 1.6 MB/25ns ~60k TB/s

p p

Need high luminosity to get to observe the

very interesting events

Need on-line selection to write to disk

mostly interesting events



Dataflow

EBHigh LevelTrigger

L2

ROS

Level 1Det.

R/O

Trigger DAQ

2.5 s

~10 ms

Calo MuTrCh Other detectors

Read-Out Systems

L2P L2N

RoI

RoI data (~2%)

RoI requests

L2 accept (~3.5 kHz)

SFO

L1 accept (100 kHz)

40 MHz

EFEFP

~ sec

EF accept (~0.2 kHz)

ROD ROD ROD

ROB ROB ROB

SFI

EBN

Event Builder

EFN

DFM

L2SVROIB

Event Filter

Level 2

ATLAS Trigger & DAQ: architectureATLAS Trigger & DAQ: architecture

40 MHz

~ 200 Hz ~ 300 MB/s

100 kHz

~ 3.5 kHz

Full info / event: ~ 1.6 MB/25ns

~3+6 GB/s

160 GB/s



From the detector into the Level-1 TriggerFrom the detector into the Level-1 Trigger

• Interactions every 25 ns:

…in 25 ns particles travel 7.5 m• Cable length ~100 meters:

…in 25 ns signals travel 5 m

Total Level-1 latency = 2.5 sec(TOF + cables + processing + distribution)

For 2.5 sec, all signals must be stored in electronic pipelines

Weight: 7000 t

44 m

22m

Level 1

Trigger DAQ

2.5 s

Calo MuTrChOther detectors

FE Pipelines

40 MHz

40 MHz



Upon LVL1 accept: buffer data & get RoIsUpon LVL1 accept: buffer data & get RoIs

ROS

Level 1Det.

R/O

Trigger DAQ

2.5 s


Read-Out Systems

RoI

L1 accept (100 kHz)

40 MHz

40 MHz

160 GB/sROD ROD ROD

ROB ROB ROBROIB

Read-Out Drivers

Region of Interest Builder Read-Out Buffers

Read-Out Links (S-LINK)

100 kHz



LVL1 finds Regions of Interest for next levelsLVL1 finds Regions of Interest for next levels

4 RoI addresses

In this example:4 Regions of Interest:

2 muons,2 electrons



Upon LVL1 accept: buffer data & get RoIsUpon LVL1 accept: buffer data & get RoIs

ROS

Level 1Det.

R/O

Trigger DAQ

2.5 s


Read-Out Systems

RoI

L1 accept (100 kHz)

40 MHz

40 MHz

160 GB/sROD ROD ROD

ROB ROB ROBROIB

Read-Out Drivers

Region of Interest Builder Read-Out Buffers

Read-Out Links (S-LINK)

100 kHz

On average, LVL1 finds~2 Regions of Interest (in ) per event

Data in RoIs is a few % of the Level-1 throughput



LVL2: work with “interesting” ROSs/ROBsLVL2: work with “interesting” ROSs/ROBs

For each detector there is a simple correspondence Region Of Interest ROB(s)

LVL2 Proccessing Units:

for each RoI, the list of ROBs with the corresponding data from each detector is quickly identified

L2

ROS

Level 1Det.

R/O

Trigger DAQ

2.5 s

~10 ms


Read-Out SystemsL2P L2N

RoI

RoI data (~2%)

RoI requests

L1 accept (100 kHz)

40 MHz

40 MHz

100 kHz 160 GB/s

~3 GB/s

ROD ROD ROD

ROB ROB ROBL2SVROIBLevel 2

LVL2 Supervisor

LVL2 Network

LVL2 Processing Units

Read-Out Buffers

RoI-based Level-2 trigger:A much

smaller ReadOut network

… at the cost of a higher control traffic



Trigger DAQCalo

MuTrCh

EB

L2

ROS

Level 1Det.

R/O

2.5 s

~10 ms

Other detectors

Read-Out Systems

L2P L2N

RoI

RoI data (~2%)

RoI requests


L1 accept (100 kHz)

40 MHz

40 MHz

100 kHz

~3.5 kHz

160 GB/s

~3+6 GB/s

ROD ROD ROD

ROB ROB ROB

SFI

EBN

Event Builder

DFM

L2SVROIBLevel 2

Sub-Farm Input

Dataflow ManagerEvent Building Network

After LVL2: Build full eventsAfter LVL2: Build full events



LVL3: Event Filter deals with Full Event infoLVL3: Event Filter deals with Full Event infoTrigger DAQ

EB

L2

ROS

Level 1Det.

R/O

2.5 s

~10 ms



RoI

RoI data (~2%)

RoI requests


L1 accept (100 kHz)

40 MHz

40 MHz

100 kHz

~3.5 kHz

160 GB/s

~3+6 GB/s

EFEFP

~ sec

ROD ROD ROD

ROB ROB ROB

SFI

EBN Event Builder

EFN

DFM

L2SVROIB

Event Filter

Level 2

Farm ofEvent Filter Processors

Event Filter Network

Full Event Sub-Farm Input

~ 200 Hz



EB

L2

ROS

Level 1Det.

R/O

2.5 s

~10 ms



RoI

RoI data (~2%)

RoI requests


L1 accept (100 kHz)

40 MHz

40 MHz

100 kHz

~3.5 kHz

160 GB/s

~3+6 GB/s

EFEFP

~ sec

ROD ROD ROD

ROB ROB ROB

SFI

EBN

Event Builder

EFN

DFM

L2SVROIB

Event Filter

Level 2

Event Filter Processors

Event Filter Network

SFOEF accept (~0.2 kHz)

~ 200 Hz ~ 300 MB/s

Sub-Farm Output

From Event Filter to Local (TDAQ) storageFrom Event Filter to Local (TDAQ) storage



Dataflow

EBHigh LevelTrigger

L2

ROS

Level 1Det.

R/O

Trigger DAQ

2.5 s

~10 ms


Read-Out Systems

L2P L2N

RoI

RoI data (~2%)

RoI requests


SFO

L1 accept (100 kHz)

40 MHz

40 MHz

100 kHz

~3.5 kHz

~ 200 Hz

160 GB/s

~ 300 MB/s

~3+6 GB/s

EFEFP

~ sec


ROD ROD ROD

ROB ROB ROB

SFI

EBN

Event Builder

EFN

DFM

L2SVROIB

Event Filter

Level 2

TDAQ, High Level Trigger & DataFlowTDAQ, High Level Trigger & DataFlow



Dataflow

EBHigh LevelTrigger

L2

ROS

Level 1Det.

R/O

Trigger DAQ

2.5 s

~10 ms


Read-Out Systems

L2P L2N

RoI

RoI data (~2%)

RoI requests


SFO

L1 accept (100 kHz)

40 MHz

40 MHz

100 kHz

~3.5 kHz

~ 200 Hz

160 GB/s

~ 300 MB/s

~3+6 GB/s

EFEFP

~ sec


ROD ROD ROD

ROB ROB ROB

SFI

EBN

Event Builder

EFN

DFM

L2SVROIB

Event Filter

Level 2


High Level Trigger (HLT)

• Algorithms developed offline (with HLT in mind)

• HLT Infrastructure (TDAQ job):– “steer” the order of

algorithm execution– Alternate steps of “feature

extraction” & “hypothesis testing”)

fast rejection (min. CPU)

– Reconstruction in Regions of Interest min. processing time &

network resources



Dataflow

EBHigh LevelTrigger

L2

ROS

Level 1Det.

R/O

Trigger DAQ

2.5 s

~10 ms


Read-Out Systems

L2P L2N

RoI

RoI data (~2%)

RoI requests


SFO

L1 accept (100 kHz)

40 MHz

40 MHz

100 kHz

~3.5 kHz

~ 200 Hz

160 GB/s

~ 300 MB/s

~3+6 GB/s

EFEFP

~ sec


ROD ROD ROD

ROB ROB ROB

SFI

EBN

Event Builder

EFN

DFM

L2SVROIB

Event Filter

Level 2


DataFlow

• Buffer & serve data to HLT

• Act according to HLT result, but otherwise HLT is a “black box” which gives answers

• Software framework based on C++ code and the STL



Dataflow

EBHigh LevelTrigger

L2

ROS

Level 1Det.

R/O

Trigger DAQ

2.5 s

~10 ms


L2P L2N

RoI

RoI data (~2%)

RoI requests


SFO

L1 accept (100 kHz)

40 MHz

EFEFP

~ sec


ROD ROD ROD

ROB ROB ROB

SFI

EBN

EFN

DFM

L2SVROIB500nodes

100nodes

150nodes

1600nodes

Infrastructure Control Communication Databases

High Level Trigger & DataFlow: PCs (Linux)High Level Trigger & DataFlow: PCs (Linux)



TDAQ at the ATLAS siteTDAQ at the ATLAS site

SDX1

USA15

UX15

ATLASdetector

Read-Out

Drivers(RODs) First-

leveltrigger

Read-OutSubsystems

(ROSs)

UX15

USA15

Dedicated links

Timing Trigger Control (TTC)

1600Read-OutLinks

Gig

abit

Eth

erne

t

RoIBuilder

Reg

ions

Of I

nter

est

VME~150PCs

Data of events acceptedby first-level trigger

Eve

nt d

ata

requ

ests

Del

ete

com

man

ds

Req

uest

ed e

vent

dat

a

Event data pushed @ ≤ 100 kHz, 1600 fragments of ~ 1 kByte each

LVL2Super-visor

SDX1CERN computer centre

DataFlowManager

EventFilter(EF)

pROS

~ 500 ~1600

stores LVL2output

dual-CPU nodes

~100 ~30

Network switches

Event data pulled:partial events @ ≤ 100 kHz, full events @ ~ 3 kHz

Event rate ~ 200 HzData

storage

LocalStorage

SubFarmOutputs

(SFOs)

LVL2 farm

Network switches

EventBuilder

SubFarmInputs

(SFIs)

Second-leveltrigger

“pre-series” system: ~10% of final TDAQ

in place

http://atlaseye.web.cern.ch/atlaseye/Pictures/tec_coord_web/jura/webjura.jpg



Example of worries in such a system: CPU powerExample of worries in such a system: CPU power

• At Technical Design Report we assumed:

– 100 kHz LVL1 accept rate– 500 dual-CPU PCs for LVL2– 8 GHz per CPU at LVL2

• So:– each L2PU does 100Hz– 10ms average latency per event in each L2PU

• 8 GHz per CPU will not come– But, dual-core dual-CPU PCs show scaling!

Scaling of a dual-core dual-CPU Processor

0

100

200

300

400

500

1 2 3 4 5 6

Number of Processes

Ach

ieved

LV

L2 R

ate

(H

z)

Series1

Preloaded ROS w/ muon events, run muFast @ LVL2

Test with AMD dual-core, dual CPU @ 1.8 GHz, 4 GB total

We should reach necessary performance per PC at cost of higher memory needs & latency (shared memory model would be better here)

18



Last Sept: cosmics in the Tile hadronic calorimeter, brought via the pre-series (monitoring algorithms)

Cosmics in ATLAS in the pitCosmics in ATLAS in the pit

This July: Cosmic run with

LAr EM + Tile Had Cal (+Muon detectors?)



• ATLAS TDAQ:– 3-level trigger hierarchy– Use Regions of Interest from previous level: small data movement– Feature extraction + hypothesis testing: fast rejection min. CPU

power

SummarySummary

• We are in the installation phase of system• Cosmic run with Central Calorimeters (+muon system?) this summer

TDAQ will be ready in time

for LHC data taking

• Triggering at Hadron Colliders:– Need high luminosity to get rare events

– Can not write all data to disk• No sense otherwise: offline, we’ll be wasting our time looking for a needle in

the hay!



Thank you



ROS units contain 12 R/O Buffers150 units needed for ATLAS (~1600 ROBs)

A ROS unit is implemented with a 3.4 GHz PC housing 4 custom PCI-x cards (ROBIN)

ReadOut Systems: 150 PCs w/ special cardsReadOut Systems: 150 PCs w/ special cards12 ROS in place, more arriving

Performance of final ROS (PC+ROBIN)

is above requirements

Note: we have also ability to access individual ROBs if

wanted/needed50

60

70

80

90

100

110

120

130

140

0 2 4 6 8 10

Level 2 Trigger acceptance (%)

Le

ve

l 1

Tri

gg

er

rate

(kH

z)

“Hottest” ROS from paper model

2. Measurements on real ROS H/W

Low Lumi. operating region

High Lumi.

operating

region

LV

L1

acce

pt

rate

(kH

Z)

LVL2 accept rate (% of input)

Not all ROSs are equal in rate of data requestsRODROS re-mapping can reduce requirements on busiest (hottest) ROS



So, we need:• 5600 MB/s into EB system / (70MB/s in each EB node)

need ~80 SFIs for full ATLAS • When SFI serves EF, throughput decreases by ~20%

actually need 80/0.80 = 100 SFIs

Event Building needsEvent Building needsThroughput requirements:

• 100 KHz LVL1 accept rate• 3.5% LVL2 accept rate 3.5 KHz EB• 1.6 MB event size 3.5 x 1.6 = 5600 MB/s total input

Network limited (fast CPUs):• Event building using 60-70% of Gbit network ~70 MB/s into each Event Building node (SFI)

6 prototypes in place, evaluation of PCs now, expect big Event Building needs from day 1: > 50 PCs till end of year



Data File

LVL2 Ltcy

ProcessTime

RoI CollTime

RoI Coll Size

# Req /Evt

(ms) (ms) (ms) (bytes)

3.4 2.8 0.6 287 1.3

di-jet 3.6 3.3 0.3 2785 1.2

e 17.2 15.5 1.7 15820 7.4

Tests of LVL2 algorithms & RoI collection Tests of LVL2 algorithms & RoI collection

2) Processing takes ~all latency:

small RoI data collection time

Note: Neither Trigger menu, nor data files representative mix of ATLAS (this is the aim for a late 2006 milestone)

3) Small RoI data request

per event

Electron sample

is pre-selected

1) Majority of events rejected

fast

Di-jet, & e simulated events preloaded on ROSs; RoI info on L2SV

L2SV

L2PU

pROSEmulated

ROS

8

1

1

pROS1

DFM

1

Plus:• 1 Online Server• 1 MySQL data base server



• L2SV gets RoI info from RoIB• Assigns a L2PU to work on event• Load-balances its’ L2PU sub-farm

• Can scheme cope with LVL1 rate?• Test with preloaded RoI info into RoIB, which triggers TDAQ chain, emulating LVL1

• LVL2 system is able to sustain the LVL1 input rate:– 1 L2SV system for LVL1 rate ~ 35 kHz– 2 L2SV system for LVL1 rate ~ 70 kHz (50%-50% sharing)

Scalability of LVL2 system Scalability of LVL2 systemRoIB -> 1 or 2 L2SVs.

Each L2SV->1-8L2PUs

34

35

36

1 3 5 7

# L2PUs

LV

L1

ra

te (

KH

z)

1L2SV-1

2L2SV-1

2L2SV-2

Rate per L2SV stable within 1.5%

ATLAS will have a handful of L2SVs can easily manage 100 kHz LVL1 rate



• Previous Event Filter I/O protocol limited rate for small event sizes (e.g., partially built)

changed in current TDAQ software release

EF performance scales farm sizeEF performance scales farm sizeDummy algorithm: always accept,

but with fixed delay

Event size1 MB

Initially CPU limited, but eventually bandwidth limited

• Test e/ & selection algorithms– HLT algorithms seeded by L2Result– pre-loaded (e & ) simulated events

on 1 SFI Emulator serving EF farm– Results here are for muons:

Running muon algorithms: scaling with EF farm size (still CPU limited with 9 nodes)



ATLAS Trigger & DAQ: philosophy ATLAS Trigger & DAQ: philosophy

40 MHz

~100 kHz

2.5 s

~3 kHz

~10 ms

~ 1 s

~200 Hz

Muon

LVL1

Calo Inner

PipelineMemories

Read-OutDrivers

RatesLatency

RoI

LVL2

Event builder cluster

Local Storage: ~ 300 MB/s

Read-Out Subsystems

hosting Read-Out

Buffers

Event Filter farm

EF

ROBROBROBROBROBROBROBROBROBROBROBROB

RODRODRODRODRODROD

RODRODROD

ROBROBROBROBROBROBROBROBROBROBROBROB

Hardware based (FPGA, ASIC)Hardware based (FPGA, ASIC)

Calo/Muon (coarse granularity)Calo/Muon (coarse granularity)

Software (specialised algs)Software (specialised algs)

Uses LVL1 Uses LVL1 Regions of InterestRegions of Interest

AllAll sub-dets, sub-dets, fullfull granularity granularity

Emphasis on early rejectionEmphasis on early rejection

Offline algorithmsOffline algorithms

Seeded by Seeded by LVL2 resultLVL2 result

Work with Work with full eventfull event

Full calibration/alignment infoFull calibration/alignment info

Hig

h L

evel Tri

gg

er

Hig

h L

evel Tri

gg

er



L2PU

pROS

(EB node)

RoIB

L2SV

ROS

LVL1 Trigger

EFD/PT (EF node)

4: RoI Request

5: RoI Data

6: L2Result

7: Ack

16: Full Event

12: Data Request

15: Clear event

13: ROS Data

2: L1Result3: L1Result

8: L2Decision

9: L2DescisionGroup

10: Ack

1: L1 Trigger streams

15: Clear event

13: ROS Data

DFM

11: AssignmentSFI

12: Data Request

14: End Of Event

SFO

17: Full Event

L2PU

pROS

(EB node)

RoIB

L2SV

ROS

LVL1 Trigger

EFD/PT (EF node)

4: RoI Request

5: RoI Data

6: L2Result

7: Ack

16: Full Event

12: Data Request

15: Clear event

13: ROS Data

2: L1Result3: L1Result

8: L2Decision

9: L2DescisionGroup

10: Ack

1: L1 Trigger streams

15: Clear event

13: ROS Data

DFM

11: AssignmentSFI

12: Data Request

14: End Of Event

SFO

17: Full Event

Data Flow and Message PassingData Flow and Message Passing



A Data Collection application example: the Event Builder

A Data Collection application example: the Event Builder

Event Assembly Activity

Input Activity

Request Activity Event Handler ActivityAssignment

*EventFragments *Event

Event Sampler Activity

ROS &

pROSEvent Fragments Data Requests

Data Flow Manager

Assignment

Event

Event

Trigger(EventFilter)

EventMonitoring

SFI:Event

Builder

atlas trigger & data acquisition system: concept & architecture

Documents

final tdaq

mtotal level

mbsfull info event

event filter deals

buffer data

khz lvl1

interesting physics

ns signals travel