calibration streams in the event filter. status report mainz, thursday 13 october 2005 sander klous...

Calibration streams in the Event Filter.

Status report

Mainz, Thursday 13 October 2005

Sander Klous – NIKHEFOn behalf of the EF calibration team:

Martine Bosman, Andrea Negri, Serge Sushkov and Sarah Wheeler.

13 October 2005 TDAQ workshop - Mainz 2

Physics streams40 MHz x 1.5 MByte

Level 1

Level 2

75 kHz x 1.5 MByte

Definition and scope.

Calibration issues in the EF.– At the moment only EF output.

• Identify calibration types.– Size, rate, contents.

• Requirements for the EF.– Data flow.– Processing issues.

• Implementation scenarios.• Design and modifications.

– Processing.– Memory management.– Networking/Timing.

• Plan of work.

Processing time:1 Sec/Evt

Number of nodes:1500

Output: 200 Hz320 MB/s

Calibrationstreams

Level 1

Level 2

Other people

Event Filter

?2 kHz x 1.5 MByte


Use cases(identification of calibration streams).

Based on the Hawkings/Gianotti document. – All listed calibration types are identified at HLT level (after PESA).

• Known calibration types in the HLT.– Various duplicates of physics streams: e.g. inclusive high pT electrons and

muons (tracking), Z to di-lepton (energy), minimum bias (background).• Total: 35 MB/s (10% of physics data).

– Liquid Argon Calorimeter.Pulse shape analysis, timing calibration and tuning of filter coefficients.

• High pT electron sample• Electro Magnetic data only. ROI only.• Raw data, 5 consecutive samples (i.e. special event type).

– Calorimeters and TRT.Hadronic response studies, comparison to test beam data.TRT: e/ separation.

• High pT isolated hadrons.• All subdetectors. ROI only.• RAW data.


Use cases – continued…(identification of calibration streams).

– MDT small chambers.Hourly realignment.

• Small muon sample.• MDT information only. Overlap regions only.• Reprocessing of raw data.

– Inner Detector subdetectors (Pixel, SCT and TRT).ROD monitoring (TRT only) and alignment.

• Generic high pT events.• All subdetectors. ROI only.• Post-processing of track fit information on HLT level.

• Other foreseen calibration types.– Liquid Argon Calorimeter might need Z ee calibration at HLT level.– High statistics (1 kHz) ROI muon sample, containing MDT, CSC and

RPC/TGC information.– Your favorite missing calibration stream…

Special events

EF Node n

EF Node n Transporttime: 19 ms

EF Node n

Data flow characterization.

SFO

320 MB/s

Physics

Full calibration events

SFO

32 MB/s

High pT

SFO

1.6 MB/s

Z di-lepton

SFO

1.6 MB/s

Min. bias

EF processing: 1 second/event

SFOSFO

5 kB/s

OverlapMu

2.5 MB/s

LAr

SFO

2 MB/s

IsoHad

SFO

4 MB/s

GenPT

Stripping/CollectingPartial calibration events

Transport times/rates:LArOverlapMuIsoHadGenPT

0.5 ms0.05 ms

5 ms0.5 ms

50 Hz5 Hz5 Hz

100 Hz

Output

1 MB/s

LVL2Cal

SubdetectorFragment 1

ROI info

Transport time:LVL2Cal 0.01 ms

Partial event

SFI?

Transport times: 19 msSubdetectorFragment 1

SubdetectorFragment N

Lvl 1/2 info

Lvl 1/2 info

SFI

AdditionalProcessing

1 kHz!

Sorting

EF processing:1 second/event


EF processing issues.• Definitions:– Sorting for calibration: CalID.– Stripping/Collecting for calibration: CalCollect.– Detector calibration: CalDetect or Calibration.

• Full event streams.– Very similar to physics streams.– Output to multiple SFOs/streams.– Sorting of events for calibration (CalID).

• Partial event streams.– Processing similar to physics stream.– Stripping and collecting (CalCollect).– Handling of different output event size.– Sometimes requires additional processing

(CalDetect/Calibration).

• Special event streams.– Processing times completely different from

physics stream.– Handling of different input and output event

sizes.

• Central issue: Robustness of the EFD

Main output stream

SFO

Diagnostic

SFO

Node n

EFD

PT#1

PTIO

SFI

Input

ExtPTs

OutputOutput

Trash

ExtPTsPT

cal

PTIO

Calibration Stream

Output

SFO

PT#2

PTIO

EventResult

SharedHeap


Main output stream

SFO

EF processing issues.

Node n

EFD

SFI

Input

ExtPTs

Output

Event

SharedHeap

Dataflow application

• Definitions:– Sorting for calibration: CalID.– Stripping/Collecting for calibration: CalCollect.– Detector calibration: CalDetect or Calibration.

• Full event streams.– Very similar to physics streams.– Output to multiple SFOs/streams.– Sorting of events for calibration (CalID).

• Partial event streams.– Processing similar to physics stream.– Stripping and collecting (CalCollect).– Handling of different output event size.– Sometimes requires additional processing

(CalDetect/Calibration).

• Special event streams.– Processing times completely different from

physics stream.– Handling of different input and output event

sizes.

• Central issue: Robustness of the EFD


Calibration stream scenarios (1).

Additional functionality:• CalID algorithm.

• Parallel output streams.

Node n

EFD

SFI

Input

ExtPTs

Trash

Main output stream

SFO

Output

EventResult

Physics only events Full calibration events. e.g. Z di-lepton

PT#1

PTIO

PESA

Main output stream

SFO

Node n

EFD

SFI

Input

ExtPTs

OutputTrash

EventResult

PT#1

PESA

PTIO

Calibration Stream

SFO

Output

CalID

Calibration Stream 1

Main output stream

SFO SFO

Node n

EFD

SFI

Input

ExtPTs

OutputOutput

EventResult

PT#1

PESACalID

PTIO

Calibration stream scenarios (2).

Additional functionality:• PT for calibration.

– Information handling– Stripping/collecting.

• Memory management.

Partial calibration events. e.g. GenPT Node n

EFD

SFI?

Input

ExtPTs

Event

Calibration Stream

EF output

Output

PT#1

CalID

PTIO

Calibration Stream 2

Output

SFO

ExtPTs

PT cal P

TIO StrippingCollecting

CalResult

Special streams. e.g. LVL2Cal

Calibration

• Networking/Timing issues.

Sorting


Design and modifications (1).• CalID algorithm.

– Lightweight algorithm.– Runs after PESA in physics PT – Stability issues.

Athena configuration: multiple top algorithms.

– Workload:Low – implementation only, thorough testing required.

– Impact:High – required for (almost) all calibration streams.

– Coordination:Sorting – New PT answers should be discussed.

• Parallel output streams.– Slight modification of existing algorithm.– Runs in EFD, probably required for PESA as well.– Workload:

Low – Modification of standard EFD task.

– Impact:High – Required for most (calibration) streams.


Design and modifications (2).• PT for calibration.

1. Create stripping/collection algorithm.• Requires new eformat (see next slide).• Requires modifications in output task.

2. Allow multiple PTs to run consecutively (works already).3. Transfer information between these PTs.

– Should be possible with new eformat for EFResult / CalResult.

4. It might be interesting to transfer “intermediate results”.– Would avoid to

• Run calibration algorithms in the same PT as PESA.• Reanalyze complete event in second PT.

– Since EF is a dataflow application, this should be accomplished by writing an extra “Intermediate Result” object in the Shared Heap.• This requires ByteStream conversion for complex classes.

– Workload:Medium – With exception of item 4 (no use cases yet).

– Impact:High – Maybe with exception of item 4.


Memory management and information handling.

New eformat.

Node n EFD

SFI

Input

ExtPTs

ExtPTs

SharedHeap

EventEFResult

PT#1

PESACalID

PTIO

CalResult

SFO

Output

PT cal

PTIOStripping

Collecting

1 0 1 0 0 1

- x - 1 - 0 x

1 1 1 0 0 1

• Integrate with:– Virtual event.

– SharedHeap.

– Event handling in the EF.

– Event modification in EF, i.e. stripping/collecting.

Event fragments

1 0 1 0 0 1 1

1 0 1 0 0 1 1

EFResult

- - - - - - - 1

Stripping

CalResult

Virtual event


Open issuesMany new developments on a very tight schedule.• Memory and performance.

– Management: move from open/close backpressure mechanism (barrier) to analog (Nano sleeps).

– Timing: revise SFI – EFD – SFO protocol.

• Coordination…– Investigate common/similar design issues with monitoring– 128 bit header word. First discussion yesterday.

• 32 bits to register appropriate output streams.• Investigate usage of this header. Streaming only, since EFResult fragment

contains much more detail and is “just around the corner”.

– PT answer to EFD. Composite structure.– EFD – SFO sorting, i.e. how is an output stream defined?

• Load balancing between calibration and physics.• Distribution of calibration constants to EF software.

– Communication between Athena algorithms and configuration and calibration databases.

– Could this be done via e.g. the information service?


Plan of work.• Short time scale.

– Run calibration algorithm in PT.– Implement parallel output streams.

• Medium time scale.– Implement CalID algorithm and sorting.– Eliminate dead time in input and output tasks.

• Medium – Long timescale.– Change memory management.– Implement new eformat.– Implement stripping/collecting.

• Long timescale.– Transfer of intermediate results between first and second PT


Conclusions.

• Our understanding of calibration streams in the Event Filter improved a lot.

• We think we have a realistic overview of the workload involved in modifications of the Event Filter.

• Implementation has started and this will lead to even better understanding of the topic (and of the work involved).

• There are still some (many) open issues. Coordination is important, especially because of the tight schedule.

• More information:https://uimon.cern.ch/twiki/bin/view/Atlas/EventFilterCalibration


Appendix


A distributed trigger for calibration?Appears to fit with solutions shown in LVL2Mu presentations.• Ultralight project (Manuela Cirilli).• LVL2Mu calibration stream (Speranza Falciano).• Etc. (Enrico Pasqualucci, Alessandro de Salvo).

Additional functionality:

• HLT output

Node n

EFD

SFI?

Input

ExtPTs

EventPT#1 Event

Distributor

PTIO

Output

Moore’s law for networkingGary Stix, Scientific American, January 2001

The Event Filter is CPU dominated. You would like it to be bandwidth dominated…

Calibration Stream

HLToutput


ByteStream conversion

• Write converters?– No, a lot of work.

– Robustness issues.

• Generic ByteStream conversion?– No support for complicated classes

(e.g. multiple inheritance, polymorphism).

• Something else?• Not on the priority list.• Under discussion…


Memory management and networking/timing

Node n EFD

SFI

Input

ExtPTs

ExtPTsPT

cal PTIO

SFO

SharedHeapEvent

EFResultEvent

EFResultEvent

EFResultCalResult

IntermediateResults

PT#1

PESACalID

PTIO

Virtual event

OutputStripping

Collecting

19 milliseconds transport time

19 milliseconds transport time

0.25 ms dead time

0.25 ms dead time

1+ second processing time

Barrier



Node n EFD

SFI

Input

ExtPTs

ExtPTsPT

cal PTIO

SFO

SharedHeapEvent

EFResultCalResult

EventEFResultCalResult


IntermediateResults

IntermediateResults

IntermediateResults

PT#1

PESACalID

PTIO

Virtual event

Output

Barrier

0.01 ms transport time


25 ms dead time

25 ms dead time

0 sec. processing time

• Eliminate dead time.– Redesign of SFI – EFD – SFO

communication protocol.– Coordination with networking

group.

• Barrier is insufficient.– Oscillations.– Other memory requests.

Stripping

Collecting



Node n EFD

SFI

Input

ExtPTs

ExtPTsPT

cal PTIO

SFO

SharedHeapEvent

EFResultCalResult



IntermediateResults

IntermediateResults

IntermediateResults

PT#1

PESACalID

PTIO

Virtual event

Output

Barrier



25 ms dead time

25 ms dead time


• Eliminate dead time.

• Barrier is insufficient.– Oscillations.

– Other memory requests.

Stripping

Collecting



Node n EFD

SFI

Input

ExtPTs

ExtPTsPT

cal PTIO

SFO

SharedHeapEvent

EFResultCalResult



IntermediateResults

IntermediateResults

IntermediateResults

PT#1

PESACalID

PTIO

Virtual event

Output

• Eliminate dead time.

• Barrier is insufficient.– Oscillations.

– Other memory requests.Nano sleeps



25 ms dead time

25 ms dead time

Nano sleeps

Stripping

Collecting


• Solution: Nano sleeps,

However…

– multiple control loops.

– Risk of oscillations.

– Additional complexity.

• Workload:

High – Especially testing.

• Impact:

???

calibration streams in the event filter. status report mainz, thursday 13 october 2005 sander klous...

Documents

calibration calid

calibration issues

detector calibration

timing calibration

ef calibration team

foreseen calibration

listed calibration types

z ee calibration