1

John Baines

Commissioning of the ATLAS High Level Trigger

2

Overview of Talk• ATLAS

• LHC Parameters

• The ATLAS Trigger

• UK & RAL involvement

• Commissioning– System Tests

– Single Beam

– Cosmics

• Successes & Lessons Learned

• Commissioning in 2009

• Summary

Material taken from conference talks by: S. Farrington, C. Padilla, R.Hauser, F. Winklmeier, W. Wiedenmann, R. Goncalo, A. Ventura

3

The ATLAS Detector

4

LHC Parameters

-

Example signal & background rates:• 100 GeV Higgs: ~0.1 Hz• SUSY <1 Hz• W ~500 kHz• Z ~80 kHzBackground• Inelastic: ~1GHz•Jets >1kHz

• Bunch crossing interval : 25ns (40MHz)• No. overlapping events : 23

=> event rate ~ 1GHz• Average no. particles : 1400About 108 channels to read out.

➔ Event size: 1.5 Mbyte ➔ Larger during special runs: > 15 Mbyte ➔ Tier0/Reconstruction/Grid/Storage:

output limit about 200 Hz/300 MByte/s

Parameters at full luminosity (L=1034cm-2s-1)

5

Trigger Architecture

-

Level 1

Level 2

Event Filter

75kHz

2 - 3kHz

200Hz

p p

40MHz Calorimeter or Muon (or TRTfastOR)Hardware: FPGA, ASICIdentify Regions of Interest for HLT

Software Trigger, commodity PCsSeeded by L1 ROI Full detector granularity.Requests data in RoI from Read Out Buffers

Software Trigger, commodity PCsSeeded by L1 & L2Has access to entire event.

2.5s

40ms2GHz CPU

4s2GHz CPU

6

ATLAS Trigger & DataFlow

~40ms

~4s

7

ATLAS UK HLT

RAL:Fred WickensMonika WielersDmitry EmeliyanovJulie KirkBill ScottJohn BainesStudent:Rudi Apolle

ManchesterOxfordRoyal HollowayRALUCL

Trigger Selection SoftwareInner Detector TriggerElectron/photon TriggerB-Physics TriggerTrigger Release CoordinationTrigger ValidationTrigger Hardware & Farms

8

Level-13 sub-systems:

• L1- Calorimeters

• L1- Muons

• Central Trigger Processor (CTP)

Signature Identification

• e/, /h, jets, μ

• Multiplicities per pT threshold

• Isolation criteria

• Missing ET, total ET, jet ET

CTP

• Receive and synchronize trigger information

• Generate Level-1 trigger decision (L1A)

• Deliver L1A to other subdetectors

• Sends the Regions of Interest to the Level 2 trigger

9

The HLT FarmUltimately: 2300 processors (L2+EF) Now: ~1600 processors

10

Multi-core processors

Resource requirements are multiplied with number of process instances•Memory ~ 1–1.5 GByte/Application•file descriptors•network sockets,•number of controlled applications

•~ 7k presently•~ 20k final systemTrigger

11

HLT Framework• Level-2

– HLT selection software runs in the Level-2 Processing Unit (L2PU).

– Selection algorithms run in a worker thread.

• Event Filter(3 kHz→200 Hz)

– Independent Processing Tasks (PT) run selection software on Event Filter (EF) farm nodes

• HLT Event Selection Software is based on the ATLAS Athena offline Framework

• HLT framework interfaces the HLT event selection algorithms to online• Driven by run control and data flow software • Event loop managed by data flow software• Allows HLT algorithms to run unchanged in the trigger and offline environment

12

HLT Selection Software

LVL2: Reduce rate from up to 75 kHz to 2-3kHz in av. 40msCustom algorithms with some offline components

EF: Reduce rate from 2-3 kHz to 200-300Hz in av. 4s.Offline algorithms run from HLT-specific wrappers

HLT:• Processing in Region of Interest

Only process ~few % of event At LVL2, request data over network for few % of event

• Early rejection – stepwise processing to minimize execution time for rejected events

13

match?

RoI-based, stepwise processing : e/ example EMROI

L2 calorim.

L2 tracking

cluster?

E.F.calorim.

track?

E.F.tracking

track?

e/ OK?

Level 2 seeded by Level 1Fast reconstruction algorithms Reconstruction within RoI

Level1 Region of Interest is found and position in EM calorimeter is passed to Level 2

Ev.Filter seeded by Level 2Offline reconstruction algorithms Refined alignment and calibration

Event rejection possible at each step

Electromagneticclusters

e/ reconst.

14

Trigger Menus• Trigger Menu defines chains of processing steps starting from LVL1 RoI

• Menu specified in terms of signatures e.g. mu6, e10, 2j40_xe30 etc.

• Chains can be prescaled at Level-1 or the HLT

• Signatures assigned to inclusive data-streams:

– egamma, jetTauEtmiss, muons, minbias, Lar and express

Example of electron signatures

15

B-physics Triggers

16

Trigger Rates & Streams

17

Commissioning• System tests with simulated & previously recorded cosmic data

– Download data to Read Out Buffers

Can test with collision events

Exercise system at max. LVL1 rate

• Cosmic tests:– Individual detectors (“slice weeks”)

– Combined runs

=> Expose algorithms to real detector noise, data errors etc.

• Beam:– Single beam

– Collisions

18

System Tests with simulated data

19

Single Beam

-

Single beam configuration – injection energy protons circulating in LHCOn collision with a collimator, a spray of particles entered the detector

OnlineOffline

10:19 10/9/2008

20

Level-1 Commissioning in Single Beam-

Each trigger component needs to be synchronised with the beam pick up

-10 10 -8 8Bunch crossing Bunch crossing

21

Commissioning with Cosmics

22

Cosmic Event

23

Differences in Cosmic v. Beam running• No beam clock

• Muon trigger chambers provide timing• Phase issues in read-out of TRT (straw detector) & Muon Drift Chambers

• No beam/no IP• Tracks distributed over d0, z0• L2 dedicated algorithms for fast muon reconstruction (in MDTs) and fast tracking algorithms in inner detector optimized for trajectories pointing towards the beam line

• Muons in HLT• The r-z view could not be fully reconstructed at L2 because algorithms are designed for pointing tracks and data access request is in trigger towers pointing to the IP• Possible to relax pointing requirements to study rejection/efficiency• Timing issues cause percent-level loss

• Tracking • Level-2 algorithms optimized for tracks from Interaction Point

24

Calorimeter in e/ & Triggers

Study of performance of clustering algorithm in Tau trigger

25

e/

Example plot from e FEX algorithms comparing L2 and EF: Shower shape in 2nd EM sampling Rη=E(3×7)/E(7×7).

26

Muon Trigger

=0.007 =17mRad

27

Muons in the Tile Calorimeter

between tile cluster and ID track

28

Commissioning the InDet trigger

Want to commission the LVL2 collisions algorithms with cosmic.But speed-optimisation of Level-2 algos means they are inefficient for tracks more that a ~5 mm from the nominal beam position.Three strategies:1)Use only the small fraction of events that pass close to the I.P.2)Loosen cuts in Pat. Rec. (not possible for all Algs.)3)Shift points.

29

Commissioning Level-2 trackingAdd an initial step that applies a shifts to all the points, so the track seems to come from the Interaction Point

30

Level-2 ID Efficiency w.r.t. Tracks reconstructed offline

31

Cosmics for ID

alignment

HLT trigger used to select events passing through the ID, sent to the the IDCosmic stream & used for offline alignment

32

Commissioning with Cosmics

216 millions events453 TB data400k filesseveral streams

33

Data Streaming

34

Online Handling of Time-Out Events• Time-out Events go to the

DEBUG stream

• The events are re-processed and streamed as if they had been processed online. The only difference is the file name.

• Files registered to the corresponding offline DB and processed normally, producing ESD, AOD, etc. , but still be separated and with the “recovered” tag.

35

Successes & Lessons learntSome highlights:• Trigger ready for First Beam

• Single beam events triggered with LVL1 & HLT streaming based on Level-1• HLT run offline on the CERN Analysis Farm

• Trigger including HLT algorithms exercised in cosmic running• ~2 months running, 220 million events• incl. long runs of >2M events

• Successfully streamed events incl. IDCosmic stream used for alignment.• Exercised processing of events from the Debug stream• Exercised procedures for evaluating new menus & code fixes on CAF prior to online deployment• Successfully exercised release management in data-taking conditions

• deployed patch releases for P1 and HLT

36

Successes & Lessons learnedImprovements for 2009 Running:

• Ability to change LVL1 pre-scales during a run was invaluable

put in place infrastructure to enable HLT prescales to also be updated during run

• Change of magnetic field required a menu change:

=> Algorithms now able to configure magnetic field automatically based on magnet current

• Problems with calculating online Level-2 & EF trigger rates

– Old system too susceptible to problems collecting information from farm nodes.

– Improvements in rate calculation and collection of information from nodes

• Removal of detectors from readout caused errors in HLT => events in debug stream

– Allow algorithms to access Mask saying which detectors are in the run => modify error response

• Problems with noisy detectors

– Consolidate procedures for making noisy detector masks available online

– Improve monitoring, especially detector & trigger info. displayed side-by-side

37

Plans for 2009/10

Luminosity : ~2x1032

Integrated : ~200pb-1

38

Collisions

-Cosmics

Cosmics with combined L1 muon triggers

First beam menu: Cosmics + beam pickup trigger

Bunch groups commissioned (requires clock commissioning)

High Level Trigger performs streaming

HLT algorithms run offline

Add HLT one piece at a time in tagging modeSwitch on HLT rejection after algorithms validated online

Full 1031 Menu

39

Collisions

-Cosmics

Cosmics with combined L1 muon triggers

First beam menu: Cosmics + beam pickup trigger

Bunch groups commissioned (requires clock commissioning)

40

Collisions

-Cosmics

Cosmics with combined L1 muon triggers

First beam menu: Cosmics + beam pickup trigger

Bunch groups commissioned (requires clock commissioning)

High Level Trigger performs streaming

HLT algorithms run offline

Add HLT one piece at a time in tagging modeSwitch on HLT rejection after algorithms validated online

Full 1031 Menu

41

Conclusion

• The trigger was successfully commissioned in Single Beam and Cosmic running in Autumn 2008

• Data has been analysed to validate the trigger operation.

• Improvements have been made in the light of experience from these runs

Eagerly awaiting collisions!!

42

Backup Slide

43

44

45

46

High Level Trigger

-

47

Level 1 Cosmic Rates

-