stefan koestnertutorial – user guideseptember 2007 tell1 - control guidelines for users &...

Stefan Koestner Tutorial – User Guide September 2007

TELL1 - CONTROL Guidelines for Users & Developers

Stefan Koestner Tutorial – User Guide September 2007

Contents:

• framework components (fwCcpc, fwHw, fwTell1)• configuration of Tell1 boards• finite state machines• design issues - special requirements• user specific - what you still have to do

Stefan Koestner Tutorial – User Guide September 2007

fwCcpc – in a nutshell:

• interface between PVSS and CCPC using DIM and a generic server

• basic test panels & PVSS library

doxygen documentation and tutorial under:http://lhcb-online.web.cern.ch/lhcb-online/ecs/PVSS_CCPC/default.html

Stefan Koestner Tutorial – User Guide September 2007

fwCcpc – in a nutshell:

• communication via Ccpc DP• each Tell1 has its own DP and each bus has its own DP element

• Note: use fwHw for accessing registers and not this DP! (model your hardware)

Stefan Koestner Tutorial – User Guide September 2007

fwHw – in a nutshell:

• a tool (panels & library) for modeling hardware as PVSS DPs and creating recipes. generic: SPECS & CCPC

Stefan Koestner Tutorial – User Guide September 2007

fwHw – in a nutshell:

• each register has its own DP and DIM srvcs/cmnds associated

• subscription: - register settings transferred to ccpc and stored there in list. - create DIM server cmnds/srvcs.

(establishes communication between server & PVSS)

doxygen documentation and tutorial under:http://lhcb-online.web.cern.ch/lhcb-online/ecs/FwHw/default.html

Stefan Koestner Tutorial – User Guide September 2007

Recipes – in a nutshell:• a recipe is a collection of registers, whose

values can be stored in the configuration DB - from where they can be applied to hardware at any time.

• Panels and framework functions to create and operate on recipes are provided.

documentation under:http://lhcb-online.web.cern.ch/lhcb-online/ecs/FwHw/default.html

Stefan Koestner Tutorial – User Guide September 2007

fwTell1 – in a nutshell:

• datapoint representations of Tell1 types (including detector specific parts)

• standard recipes of all recipe types (including detector specific part) To be imported from hardware: myTell1_’type’

• panels for device units (stopped counting ) – adopting automatically to Tell1 type

• control unit with user interface• finite state machines

Stefan Koestner Tutorial – User Guide September 2007

fwTell1 – in a nutshell:

• use fwHw tool to instantiate your Tell1 boards (instantiate datapoints)

• use DEN (device editor/navigator) to associate device unit with datapoint

• create your FSM tree

screenshot of Device Editor/Navigator

Stefan Koestner Tutorial – User Guide September 2007

fwTell1:Tell1 control unit:

pof – file:stores the firmware of FPGAs

cfg – file:server can access EPFL C-codeto configure the boards

each board has its own ccpc

Stefan Koestner Tutorial – User Guide September 2007

fwTell1:Tell1 control unit:

Commonly modified configurations can be sent to enabled boards

Send down commandsto enabled boards.(configure uses recipes e.g. “PHYSICS\Configure”

Stefan Koestner Tutorial – User Guide September 2007

fwTell1:Tell1 device unit:

A tab register is providedthat allows you to access the common part of eachTell1 board.

A tab for specific part is foreseen.

The panel configures automatically according to board type.

Some checks are performed.

Watch the info buttons andtooltips for documentation.

And many registers to be monitored … (a few hundred)

Stefan Koestner Tutorial – User Guide September 2007

fwTell1:Tell1 device unit:

in fact a lot of panels …We provide tool-tips and info buttons for help!And perhaps a dedicated tutorial on device unit panels.

Stefan Koestner Tutorial – User Guide September 2007

fwTell1:Tell1 device unit:

This is whereyou can put yourown panels!

Stefan Koestner Tutorial – User Guide September 2007

fwTell1:Tell1 device unit: We are not providing

the monitoring of each sub-detector specificparameters. (but a few…)

Stefan Koestner Tutorial – User Guide September 2007

fwTell1:Tell1 device unit:

We allow for spying on thedata of the MEP buffer.

The structure of the MEPbuffer differs – we do not provide parsersfor each sub-detector.

Stefan Koestner Tutorial – User Guide September 2007

fwTell1:Tell1 device unit:

Some examples for parsingthe MEP buffer…

Stefan Koestner Tutorial – User Guide September 2007

fwTell1:Tell1 device unit:

We divide between 3 sub-recipes.Each sub-recipe has its ownpanel.

You can export/import recipesor simply read them directly fromhardware.

Stefan Koestner Tutorial – User Guide September 2007

fwTell1:

Define your specific recipe type.Create the appropriate panels.

specific recipes:

Stefan Koestner Tutorial – User Guide September 2007

fwTell1 – conclusion:• for commissioning you can just use the common

part• BUT: start developing the specific part NOW!

• NOTE: the specific part is related to your firmware. Get in touch with EPFL Lausanne before you start! You might profit from already existing things.

doxygen documentation and tutorial under:http://lhcb-online.web.cern.ch/lhcb-online/ecs/PVSS_TELL1/default.html

Stefan Koestner Tutorial – User Guide September 2007

Ctrl managers – in a nutshell:

• fwCcpcScript.ctl: connects to running servers, automatic register subscription (crash/restart)

• fwTell1FSM.ctl: needed to stop monitoring of registers when switching panels in DUs

• fwTell1DisableArchives: disables the archive bit of registers gain speed for configuration

• fwFSMServer

Stefan Koestner Tutorial – User Guide September 2007

Configuration of many boards:

• trials and errors• benchmarking results

Stefan Koestner Tutorial – User Guide September 2007

Configuration of many boards:A first impression of scaling:

a snapshot from the talk of June 2007:

50 boards ~ 6 minutes

Stefan Koestner Tutorial – User Guide September 2007

first steps …

black: old - no changes applied

yellow/yellow: disabling archiving (but 35% CPU load from panels) lpink: disabling archiving & 1 ctrl manager per DU (still 35% CPU load from panels) green: disabling archiving (no CPU load from panels) red: disabling archiving & 1 ctrl manager per DU (no CPU load from panels) blue: reference when applying just specific part of recipes (disabled & 1 ctrl/DU)

50 boards ~ 3.5 minutes

• disabling archiving significant• additional ctrl managers in FSM significant under high load• panels consuming much CPU?

Stefan Koestner Tutorial – User Guide September 2007

same as before – two (lower) lines added …

black: checks on serverside - comparison of applied values with hardware values read back

cyan: common part of recipes (16 registers of 32 bit)

50 boards ~ 2.5 minutes

•changes on server significant but still too small

complete recipe of type ST: • 236 registers• ~80 kB

grouping and packing together memory blocks would help, but not so easy to implement in FPGAs

Stefan Koestner Tutorial – User Guide September 2007

so where does the time go …

• most of the time consumed by DIM, Event and Data manager• in current design each register is sending a DIM command

Stefan Koestner Tutorial – User Guide September 2007

parallelizing helps …

… but not yet enough!

Stefan Koestner Tutorial – User Guide September 2007

new approach – without changing the hardware representation …

getApplyRecipes() for fast Tell1 configuration:

• retrieve recipes from configuration DB• pack them together in one buffer• send them as one buffer/DIM command • decode it on server

Stefan Koestner Tutorial – User Guide September 2007

\0 &… …

C& C &…

\0

The buffer - array of unsigned char:

# registers register-names size of registers

data checks buffer corruption

…

command character

Stefan Koestner Tutorial – User Guide September 2007

\0 & …& C &

on the server side:

parsing through buffer

• get number of registers• get strings, compare them with list• fill array of pointers to list of registers (all hardware info stored) note: subscription is necessary!• check for corruption• fill array of register sizes• check for corruption• execute commands with appropriate data• final check

Stefan Koestner Tutorial – User Guide September 2007

the results for getApplyRecipes():red: one DIM command per register (old design) – but improved framework

blue: sending one single buffer containing all registers (new design)

grey: applying .cfg file

1.5 sec/board 1 sec/board no slope

50 boards < 1 minute

most time gets lost when retrieving the recipes from DB storing the whole buffer?I am optimistic that we could come closer to the .cfg file!

Stefan Koestner Tutorial – User Guide September 2007

how to create recipes without typing several thousand parameters:

• ‘get image from current board settings’ reads registers from hardware and saves them as a recipe(you can upload the .cfg file before)

• ‘import recipe from other hardware’ takes the recipe previously defined for another hardware.(you can use ‘myTell1_type’ as default recipe)

Stefan Koestner Tutorial – User Guide September 2007

Conlcusion:• creating recipes is not a big deal (tools are available)• downloading recipes to hardware is already fast & sufficient• we still support the interface to the .cfg files (original C-code)

Note: some parameters are providedby Gaudi jobs. interface under development

Stefan Koestner Tutorial – User Guide September 2007

Finite State Machines:

towards a fully automatized expert system …

Stefan Koestner Tutorial – User Guide September 2007

Tell1 boards belong to the DAQ domain.Predefined transitions and states:

FSM – in a nutshell:

Recipes are applied under ‘Configure’ –transition from NOT_READY to READY.

Checks are done to verify the statusof the board.If they fail status moves to ERROR.

If connection to ccpc is lost, it moves to UNKNOWN.

Stefan Koestner Tutorial – User Guide September 2007

FSM – in a nutshell:

• functions to be called by FSM are provided in fwTell1.ctl

• if needed you can provide a specific function to be called by the FSM (e.g. for calibration) fwTell1_Muon_Cmnd(string domain, string device, string command, string parameter)

Please read Integration Guidelines:https://edms.cern.ch/file/732486/3/LHCb_ECS_Guidelines.pdfand FSM Guidelines:https://edms.cern.ch/file/655828/3/LHCb_ECS_FSM_Guidelines.pdf

Stefan Koestner Tutorial – User Guide September 2007

FSM – from a higher point of view:

OTADCS

OTAHV

OTADAI

OTADAQ

INF DCS HV DAI DAQ L0 TFC HLT LHC

OTALV

OTATEMP

OTAFEE

OTATELL1

OTACRATES

OTACOOLING

OTA OTC

OT

Templates available:

Please use them as they are

Contact Clara for integration issues!

Storage

ECS

Stefan Koestner Tutorial – User Guide September 2007

FSM – Run Control:• Run Control is like a

Matrix, can exclude/include:– by SubDetector part– Or by Domain

• “Standalone” run, need to allocate:

• TFC• HLT• Storage• Monitoring

• Allocate uses the Partition Settings

• TFC, HLT, Storage, etc. will be ready soon.

Stefan Koestner Tutorial – User Guide September 2007

FSM – Run Sequences:

1 - Configure/RUN_TYPE=“CALIB_TIMMING”1’ - Configure/RUN_TYPE=“CALIB_TIMMING”/N_STEPS=102 - READY3 – Start4 – RUNNING5 - Start/N_TRIGGERS=100006 - RUNNING7 - TFC Paused (after N_TRIGGERS)8 - Step/RUN_TYPE =“CALIB_TIMMING”/STEP_NR=19 - RUNNING10 - Continue/N_TRIGGERS=10000

OTA OTA_DAQ TFC1

1’

234

5

67

89

10

N S

tep

s

OTADAQ

OTATFC

OTAFEE

OTATELL1

OTA

OTADev1OTADev1OTADev1OTADevN

OTADev1OTADev1OTADev1OTATell1N

Stefan Koestner Tutorial – User Guide September 2007

FSM – Run Sequences:• The Activity can be

saved, contains:– Nr Triggers

– Nr Steps

– Nr Sub-Farms, etc.

• It corresponds to the RUN_TYPE that gets propagated down and used to form a recipe name by all devices.

Stefan Koestner Tutorial – User Guide September 2007

FSM – Run Sequences:• Configure Command

– As usual propagates down so that each device can apply a recipe (or more) based on:

– RUN_TYPE (and N_STEPS)

• Step Command– Again propagates down, should be accepted in

state RUNNING (and return RUNNING) • Each device should change only the necessary

values based on:

– RUN_TYPE and STEP_NR

Stefan Koestner Tutorial – User Guide September 2007

FSM – Conventions:

• Recipe Names:– Guideline: <Run Type>/<FSM Command>, ex.:

» PHYSICS/Configure, PHYSICS/Start» CALIBRATION/Configure, etc.

• Necessary because a Configurator Object will preload recipes from DB to cache

Stefan Koestner Tutorial – User Guide September 2007

• started with a generic/altruistic server

• more and more intelligence moved to server side

few comments on design:

Stefan Koestner Tutorial – User Guide September 2007

clear separation between hardware and control issues – a ‘smarter’ server:

• server keeps track of all the information about the register (e.g. type, address,…) and stores it in a list• server treats each register in the appropriate manner(e.g. Fifo vs. standard blockwrite or different I2C derivates)• Once the register is defined the control system (PVSS) knows nothing about the hidden details and just writes to or reads from a register less and more readable code & better performance (interpreted language)

server-side PVSS-side

Intelligence related to hardware Intelligence related to ECS

Stefan Koestner Tutorial – User Guide September 2007

how to keep the design open:

• a ‘commonOperation’ part of the Ccpc DP allows to send a string to the server. Anything can be done.Identifier and parameters are separated by comas.

• a ‘user specific’ register type which offers many parameters to be filled in inside the fwHw tool. Any type of register can be modeled.

• the new recipe block executed via ‘configurationOperation’ on the Ccpc DP is in fact a set of instructions, which in addition can be stored in the Configuration DB.

\0 & …& C &

commands

Stefan Koestner Tutorial – User Guide September 2007

What is left for the sub-detectors?in fact not much …

• Specific part has to be inserted into datapoint type• The panels related to the specific part: - MEP buffer spy - Monitoring of specific registers/values - Recipe related things• specific functions to be called by FSM (e.g. for calibration) preferably packed as a library (obey to naming conventions)• Scripts inside device units might have to be extended

… but still a bit!