Towards a Homogeneous Software Environment for DAQ Applications

Luciano Orsini

Johannes Gutleber

CERN EP/CMD

2

Topics

• The CMS DAQ model– RU, BU, EVM, FU– Event building process

• Interoperability (DAQ Column, demonstrator)• Configuration Management• Use of the model in testbeams• Summary

3

Data Acquisition Baseline

Detector Frontend

Computing Services

ReadoutSystems

FilterSystems

Event Manager Builder Networks

Level 1Trigger

RunControl

4

DAQ Main Subsystems

FS

Interconnect

Interconnect

Interconnect

CSN

EVM

RCN

BCN BDN

Interconnect

FUFU

BUO BUS

BUMBUI

RUO RUS

RUMRUI

RU linkFMU

DDUDDUFED

RU

BU

LV1

5

Readout Unit

RU

FEDGTP

EVM BCN

RCN

DSNInterconnect

RUM

RUO RUS

RUI

RDL

BDN

RU Readout UnitRUI Readout Unit InputRUM Readout Unit MemoryRUO Readout Unit OutputRUS Readout Unit Supervisor

GTP Global Trigger ProcessorFED FrontEnd DriverRDL Readout Data LinkEVM Event ManagerRCN Readout Control NetworkBCN Builder Control NetworkBDN Builder Data NetworkDSN DAQ Service Network

6

Filter Column

DSN

FS

BDN

CSN

BCNEVM

BU

RURCN

BU Builder UnitFS Filter Subfarm

RU Readout UnitEVM Event ManagerRCN Readout Control NetworkBCN Builder Control NetworkBDN Builder Data NetworkDSN DAQ Service NetworkCSN Computing Service Network

7

Builder Unit

BU

FS

EVM BCN

RCN

DSN

BUM

BUO BUS

BUI

RU

BDN

InterconnectBU Builder UnitBUI Builder Unit InputBUM Builder Unit MemoryBUO Builder Unit OutputBUS Builder Unit SupervisorFS Filter Subfarm

EVM Event ManagerRCN Readout Control NetworkBCN Builder Control NetworkBDN Builder Data NetworkDSN DAQ Service NetworkRU Readout Unit

8

BU vs. FU

allocate

take (id, data)

FUBU

RU

EVM

9

BU vs. EVM

allocate

confirm (id)

FUBU

RU

EVM

10

EVM vs. RU

readout (id)

FUBU

RU

EVMtrigger

data

11

BU vs. RU

send (id)

FUBU

RU

EVM

cache (id, data)

12

BU to EVM

FUBU

RU

EVM

clear (id)

13

DAQ Interfaces

BU

allocate

RU

cache

send

allocate

take

readout

trigger

data

clear

confirm

EVM

FU

14

Test Beams

RUI RUI

RUO RUO

BUI BUI

BM

RMFLT

Generic reusable module

Test beam specific

Detector frontend

...

...

15

BM

RMFLT

Test Beam Scenarios

RUI

RUO

RUI

RUO

BU BU

BM

RMFLT

Detector frontend

...

...

RUI

RUO

BU

Detector frontend

...

...

CPU

FU

FU

16

Trigger and Readout

trigger()

busyLow()

readout(id)

readyToRead(id) readyToSend(id)

FLT RM

interrupt

BM

data

RUM

RUI

17

Interoperability

RUOBM

BUIInterfaces

18

Interoperability

Detector Frontend

Computing Services

ReadoutSystems

FilterSystems

Builder (Control & Data) Networks

Level 1Trigger

RunControl

ReadoutManager

FilterManager

RCS

BU

19

Interoperability

• Step 1 – Identify Interface requirements between components

(RU vs. BU, EVM vs. BU, etc.) = IRS

• Step2 – Definition of Interface as messages and their format

+ configuration = IDD

• Step3 – Implementation of API supporting interfaces = XDAQ

20

Interoperability

• Support integration of independently developed DAQ sub-systems in step 3

• Interoperability through – established application message formats

• (using I2O private message frames)

– transport (available communication media)– configuration (physical/logical addressing scheme)

21

Configuration Management

• Version Control through CVS– Controlled software repository – Organised multi-platform releases

• Release notes document (SRN) with every software baselines to track status and changes– Build and installation instructions– Description and capabilities– Changes from previous releases– Reference to Documentation

22

Summary

• Support development of DAQ applications according to a common model

• Implement compatible interfaces between subsystems (interoperability)

• Assemble event builders from basic software components

• Common approach to DAQ in testbeams and CMS DAQ – smooth transition from testbeam to CMS DAQ – reuse software written for offline and testbeams

DAQ Column Software Internals

24

Topics

• Design Issues• DAQ Column set-up and configuration• Software internals• Status and plans

25

Design Issues

• Reflect I2O core environment – Event driven model, peer operations, homogeneous

message passing model– Software download, predefined commands– Fill the blanks approach

• Every subsystem maps to a self contained software module– Encapsulation of a subsystem behaviour– Clean and narrow interface from/to other subsystems

• Location transparency of components– Automatic mapping of physical to logical addressing

26

DAQ Column Configuration

DAQ Executives tasks

RUIO ( iop480)

PPC(mvme2304)

PersonalComputer (pcPentium)

27

Idle

Waiting for I2O configuration messages

Zzz..zzz…zzz..

28

Received Configuration

• Where (e.g. IOP 34)• Who (e.g RUO 3)• How (e.g TCP, DLPI)• What

(e.g. BM, BUI 1, etc.)

Detector Frontend

Computing Services

ReadoutSystems

FilterSystems

Event Manager Builder Networks

Level 1Trigger

RunControl

29

Downloading

• What (e.g.libRU.so, libEVM.so)

BM

RMFLT RUI

RUO

30

RUOBUI

Operational

Send (eventId)

BM

31

Local Communication Efficiency

Through executive transparent local and remote use

higher overhead

no infinite recursion

Direct invocation fast

only local

limits the reusability in a distributed environment

danger of infinite recursion

RM

RUI

32

Message Delivery

I2O h

eade

rda

ta

Buffer pool

RUO

Peer T

rans

ports

TCP

Myri

DLPI

FIFO

Peer Transport Agent

33

Func() { f… do…}

Message Access

I2O h

eade

rda

ta

Buffer pool

RUO

Peer T

rans

ports

TCP

Myri

DLPI

FIFO

34

BU

FU

XDAQ Deployment

FU BU RUO BM

RUO BM

35

Buffer Policies

• Buffer pools of several sizes– 1 KB, 4KB, 8KB … 256 KB

• Long messages are built from scatter-gather lists– SGL header contains a list of

• memory locations and • data sizes

– Quality of Service parameters in receiving peer-transport allow

• Building of SGL in one contiguous memory block (must be provided, e.g. buffer pool 1MB or dynamically allocated)

• Building of a new SGL consisting of several blocks

36

Transport Configurations

TCP

Myri

DLPI

FIFOTCP

Myri

DLPI

FIFO

Thread per Peer Transport- higher OS service overhead+ no CPU monopolisation+ allows integration with other software

Polling Peer Transport Agent+ low OS service overhead- executive uses CPU continuously- no mixing of polling/blocking PTs

37

I2O Overview

• An architecture to develop hardware and OS independent device drivers– Uniform message passing protocol– Hardware access wrapper API– Event driven execution model– Predefined parameters and protocol for configuration– Dynamic configuration– Space for private extensions– Split of processing responsibilities to offload host CPU

• Provides interfaces (.h files and annotation)– e.g. message headers, scatter-gather lists

38

I2O Issues

• Use of documented messages– for configuration, control and data

• Provide means for configuration• I2O model maps well to distributed environments• Allows hardware evolution without the need to

change the application components• Use parts of the I2O architecture for modelling

distributed DAQ systems

XDAQ toolkit uses a subset of I2O features

39

I2O Private Message Format

MessageSize MessageFlags VersionOffset

TargetAddressInitiatorAddressFunction = 0FFh

InitiatorContext

TransactionContext

XFunctionCodeOrganizationID = 100h

PrivatePayload = Function Parameters

PrivatePayload

3 2 1 031 24 23 16 15 8 7 0

40

I2O and the XDAQ Toolkit

• Every DAQ subsystems maps into a set of I2O private device classes– (RU = RUI + RUO)

• Each class can be instantiated several times– (RU#1, RU#2, BU#1)

• Each class exposes a private and an I2O interface that can be accessed by other class instances– (private, configuration and control messages)

• Instances exchange messages• Toolkit takes care of send and delivery

41

The XDAQ Toolkit• Is a framework with C++ API

– Messages from network are received and delivered to registered applications (callback mechanism)

• Application class method is associated with a specific message (binding at compile time)

• Classes dynamically register with the framework when they are downloaded

• User provides implementations for the predefined interfaces (abstract functions)

• Access to remote classes through asynchronous method invocation – through various configurations (TCP, PCI, Myrinet, ...)

42

I2O Based Configuration

• ExecSysTabSet– Tell the executive about the

physical addresses of other machines

• ExecDeviceAssign– Tell the executive about remote software modules with which it

can communicate

• ExecSwDownload– Tell the executive which software module it hosts(and initialise)

• ExecSysEnable– Tell the executive to start the local software modules (if required)

• ExecIOPReset– Tell the executive to put the system into its initial state

43

I2O Based Configuration

• UtilParamSet– Set parameters for a DAQ software element– Set executive parameters

• UtilParamGet– Get parameters from a DAQ software element– Retrieve executive settings

44

External Interface Issues

• I2O provides messages for– Application control– Application configuration– Executive configuration

• Systems that generate I2O configuration and control messages can communicate with XDAQ

• Interface design documents exist– I2O specification: http://www.i2osig.com

– EVB data format: http://cmsdoc.cern.ch/cms/TRIDAS/html/wTDR/EB_pres_layer.pdfhttp://cmsdoc.cern.ch/cms/TRIDAS/html/wTDR/IRS.pdf

45

Status and Plans

• Interoperability test with DAQ demonstrator and benchmarks (ongoing)

• Availability– Solaris, VxWorks, Linux – LynxOS by Frederic Drouhin

• XDAQ in small DAQ setup provides well founded input to the DAQ TDR– True user and system requirements and design options

• XDAQ data acquisition toolkit– as vehicle for smooth transition from testbeam to CMS DAQ – tools for stand-alone data acquisition test software