clara gaspar, march 2006 lhcb’s experiment control system step by step

Clara Gaspar, March 2006

LHCb’s Experiment Control System

Step by Step

22Clara Gaspar, March 2006

Overview

❚ LHCb’s Experiment Control System❙ What do we (JCOP/LHCb) provide

❙ What sub-detectors/sub-systems need to implement

❚ PVSS & Framework reminder

❚ Interfacing Electronics Boards❙ SPECS & CC-PC Tools

❙ The Configuration DB

❚ Hierarchical Control❙ The FSM Toolkit

❚ Note: This tutorial is meant as an overview❙ The PVSS & Framework and the FSM courses are still

required in order to use the tools!


ECS Scope

Detector Channels

Front End Electronics

Readout Network

High Level Trigger

Storage

L0

Experi

men

t C

on

trol S

yst

em

DAQ

DCS Devices (HV, LV, GAS, Temperatures, etc.)

External Systems (LHC, Technical Services, Safety, etc)

TFC


ECS Generic Architecture

...

To Devices (HW or SW)

Com

mands

Sta

tus

&

Ala

rms

ECS

DCS DAQ

DetDcs1

DetDcsN

SubSys1

SubSys2

Dev1

Dev2

Dev3

DetDaq1

SubSysN

DevN

LHCT.S.

...GAS

DSS

Ab

stra

ct levels


What do we provide?

❚JCOP + LHCb Online provide:❙Not complete applications, but:❙A Framework, i.e. a set of tools to help sub-

systems create their control systems:❘Complete, configurable components (ex. CAEN HV)❘Tools for defining User Components:

〡Electronics boards (SPECS/ CC-PC)〡Specific equipment/software tasks (DIM protocol)

❘Other Tools, for example:〡FSM for Building Hierarchies〡Configuration DB 〡Archiving, Alarm handling, etc.


We also provide:❚ Integration of Infrastructure Services:

❘Power Distribution and Rack/Crate Control❘Cooling and Ventilation Control❘Magnet Control (Monitoring)❘Gas Control❘Detector Safety System

❚And interface to:❘LHC machine❘Access Control System❘CERN Safety System

❚Sub-detectors can use these components:❙For defining logic rules (using their states)❙For high-level operation (when applicable)

❘Switch ON, Switch Off, Set parameters


And also Database Tools

PVSSPVSS

PVSSPVSS

. To Offline

...

...PVSS

PVSSPVSS

PVSS

Cond..DB

Conf.DB

To Offline

...

...

PVSSArch.

❚ Interfaces to the three Logical Databases in the Online System

Exp

eri

men

tal E

qu

ipm

en

t


Online Database Contents

❙Configuration DB contains:❘All data needed to configure the HW (or SW) for the

various running modes〡Ex.: HV V0 Settings, Pedestal settings, trigger settings, etc.

❙PVSS Archive contains:❘All monitoring data read from HW for monitoring and

debugging of the Online System〡Ex.: HV Vmon Readings, temperatures, pedestal readings, etc.

❙Conditions DB contains:❘A subset of the monitoring data read from HW if it is

needed for Event processing (prob. packaged differently)〡Ex.: HV Vmon Readings if changed by more than n Volts

❘Some configuration data once it has been used〡Ex.: Trigger settings used by a particular run


The Configuration DB❚The Configuration DB will contain:

❙All "static" information about the devices❘Connectivity, addresses, etc. (also inventory and history)➨ Developed within LHCb (supports queries)

❙All "dynamic" data needed by the devices (for different running modes and different versions):

❘Settings (voltages, alarm limits, etc.), Calibration constants, Pedestals, FPGA code (probably a pointer to it), etc.

➨The settings for a particular running mode are called a “Recipe” (partial recipes available)

➨The JCOP FW component implements a cache:〡Can be used without Oracle for tests〡Can pre-load several recipes before “Start of Run”


What needs to be done:

❚Start bottom up❙Integrate each device into PVSS❙Define configuration recipes

❘for the various running modes

❙Build a hierarchy for each sub-system❘According to the guidelines

❙Integrate the devices in the hierarchy


Device Integration

❚Device Types❙HV & LV channels

❘CAEN, ISEG, WIENNER -> JCOP Framework

❙Analog inputs❘ELMB -> JCOP Framework

❙Electronics boards❘SPECS & CC-PC -> Tools to describe boards❘TELL1 -> FW component (for common part)

❙Other Components❘HW or SW -> FwDIM component

➨ Needs: PVSS, Framework, DIM,…


PVSS


PVSS Distribution


Datapoint Concept

❚DP type -> DP

Configs


Graphical Objects

❚Reference Panels❙Can be “inherited” dynamically❙“$parameters” get replaced by instance

value


Building User Interfaces

❚Static Part -> Drag & Drop❚Dynamic part -> Control Scripts ("C" like)

❙A few usefull calls for accessing DPs:❘dpGet (string dpName, <data_type> value)❘dpSet (string dpName, <data_type> value)❘dpConnect (string callback, string dpName)

❙A few usefull calls for accessing Widgets:❘getValue (string widgetName, string

widgetProperty, <widget dependent data>)❘setValue (string widgetName, string

widgetProperty, <widget dependent data>)


PVSS Features

❚Open Architecture❙We can write our own managers➨It can be interfaced to anything (FSM, DIM)

❚Highly Distributed❙130 Systems (PCs) tested➨No major problem found

❚Standard Interface❙All data of all sub-systems defined as

DataPoints!


Demo-1

❚Start PVSS console❚Create a project

(add installation tool)

❚PVSS basic functionality❙PVSS Managers❙Parameterization Module

❘Datapoint structures

❙Graphic editor


Demo-2

❚Install Framework❘fwCore❘fwAnalogDigital❘fwCaen❘fwConfigurationDB❘fwDIM❘fwSpecs❘fwHw

❚CAEN component:❙Create Crates/Boards/Channels

❘“Crate0” will be used by FSM later

❙Show Operation panels


DIM Distributed Information Management System

❙Publish/Subscribe mechanism❘Servers publish Services.❘Clients subscribe to Services:

〡On change or at regular intervals

❘Clients can send commands to Servers

❙Services❘A set of data

〡any type or size〡Identified by a name

❙A Name Server❘Keeps a list of available Services


DIM Some Characteristics

❙Transparency❘DIM clients do not know where their interlocutors are.

❘DIM components can move from one machine to another, all connections are transparently re-established.

❙Available on mixed environments:❘UNIX (HP-UX, Sun-OS, Sun-Solaris, IBM-AIX, DEC-OSF, Linux),

Windows, VMS, Real-time OSs (OS9, LynxOS, VxWorks)

〡API available in “C”, C++ and Java

❙Easy to Use❘One “call” and a process can become a server or a client.

❘Monitoring and Visualization Tools Available.

❘Documentation and examples at: http://www.cern.ch/dim


PVSS<->DIM❚FwDIM component:

❙Server is a DIM Server❙Client is a PVSS Manager (PVSS00dim)❙Correspondence: PVSS DPs <-> DIM Services

❘Can be setup graphically via fwDIM panel❘Or via a script library

❙When setup❘When Server updates Service data goes into DP❘Writing to DP will send a DIM Command

❙Documentation at: ❘http://www.cern.ch/lhcb-online/ecs/fw/FwDim.html


Non-standard components

❚Integrating user components:❙Create a DIM server (C or C++)

❘Publishes device status & data❘Receives Commands

❙ Create a PVSS Datapoint❘That matches the structure of DIM services

❙Connect the DP to the DIM services❘Using the FwDIM tools

❙Make a PVSS panel to control the device❚ Used for: farm monitoring, trigger algorithms, etc.


Demo-3

❚FwDIM❙Configure DIM_DNS_NODE❙Start a DIM server (ex.: pvss_dim_server)❙Start DIM visualization tool

❘DIMTree on Windows❘DID on Linux

❙Start fwDIM.pnl❘Connect services to DPs❘Visualize from PVSS


Electronics Interface

❚CC-PC & SPECS tools:❙Low-level Software

❘A “C” library for accessing board components〡Via I2C, JTAG or parallel bus (and FPGA programming)

❘A Generic DIM server for PVSS Communication

❙PVSS Tools (FW components: fwCcpc/fwSpecs)❘A library (PVSS scripting) for accessing board

components on any board with a CC-PC/Specs (equivalent to the low-level library)

❘A graphical user interface providing the functionality available in the library


Electronics Integration❚Electronics Boards:

❙Can use the CCPC/SPECS FW Tools for tests, but accessing the “chips” is not enough

❙Boards have to be modeled in PVSS according to guidelines (ex. registers have to correspond to datapoints) in order to:❘Provide access to the Conf. DB

〡Select a device/group of devices and say:Save as “Physics” recipe.

❘Be able to archive the data❘Be able to send the data to the Cond. DB❘Integrate into the FSM, Generate alarms, etc.


Electronics Integration

❚We provide a tool for modeling boards and their components (FWcomponent: FwHw)

❘Declaring boards (access via SPECS or CC-PC)Containing:

〡Groups of Chips (recursive) Containing:

❘Chips (TTCrx, Beetle, etc.)Containing:❘Registers

(access via I2C/JTAG/Parallel Bus)❙Contacts:

❘Ricardo Fernandes: SPECS❘Stefan Koestner: CC-PC


Electronics boards

❚Demo Setup

SPECSMaster SPECS

Mezzanine

I2Cwidget

Croquette

ServerPC: pclbcecs03

SPECS

I2C

ClientPC: Portable

SpecsSrv

PVSS

PVSS00dim

Ethernet

DNS

SupportPC: pclhcb155

PVSS

PVSS00dim

Note: The DNS should run on a stable machine (same as PVSS), not on a portable…


Demo-4❚FwSpecs:

❙Server PC: pclbcecs03❘Configure DIM_DNS_NODE❘Start SpecsServer remotely

❙Client PC: portable❘Configure DIM_DNS_NODE❘Start SpecsClient “direct access” panel

〡Exercise I2C, JTAG, DCU❘Explain the “Monitoring” feature❘Show Advanced Panel (User Scripts)

❙Documentation at (not this version yet):❘ http://www.cern.ch/lhcb-online/ecs/PVSS_SPECS

❚ FwCcpc: very similar❙ Tools will be presented at Online meeting


Custom Electronics

❚Demo Example

SPECSMaster SPECS

Mezzanine

Velo Board

Server PC

SPECS

SpecsSrv

TTCrxBeetle1

Beetle2

I2C


Demo-5

❚FwHw❙Create HW types:

❘ TTCrx, Beetle and VeloBoard

❙Configure Default Settings❙Create veloBoards

❘“Operate” the board

❙Interface to Configuration Database (cache)❘Save recipes (“PHYSICS”, “TEST”, etc.)❘Download recipes


Electronics guidelines

❚FwHw: Some Guidelines❙ If a “chip” has many registers

❘If they can be written in one single operation〡Declare them as 1 register of size N〡This will optimize configuration time

❘Some (a few) can also be declared separately〡If they are often accessed individually

❚After using FwHw to define the boards:❙Design a user interface to operate each board type

❘The library fwSpecs or fwCcpc will give you access to the data to be visualized or sent to the board ex.: fwSpecs_read(“board1.ttcrx1.reg2”,…)


Control Hierarchy

...

ECS

DCS DAQ

DetDcs1

DetDcsN

SubSys1

SubSys2

Dev1

Dev2

Dev3

DetDaq1

SubSysN

DevN

LHCT.S.

...GAS

DSS

❚Building a Control Hierarchy❙And integrating Devices

❚Needs: FwFSM, LHCb guidelines


Control Units❚Each node is able to:

❙Summarize information (for the above levels)

❙“Expand” actions (to the lower levels)

❙Implement specific behaviour& Take local decisions❘Sequence & Automate operations❘Recover errors

❙Include/Exclude children (i.e. partitioning)❘Excluded nodes can run is stand-alone

❙User Interfacing❘Present information and receive commands

DCS

Temp

Tracker

Muon

HV

GAS

HV


Device Units

❚Device Units❙Provide the interface to real devices:

(Electronics Boards, HV channels, trigger algorithms, etc.)

❘Can be enabled/disabled❘In order to integrate a device within FSM

〡Deduce a STATE from device readings (in DPs)〡Implement COMMANDS as device settings

❘ Commands can apply the recipes previously defined

DevN


❚The FwFSM Component is based on:❙PVSS for:

❘Device Description (Run-time Database)

❘Device Access (OPC, Profibus, drivers)

❘Alarm Handling (Generation, Filtering, Masking, etc)

❘Archiving, Logging, Scripting, Trending

❘User Interface Builder

❘Alarm Display, Access Control, etc.

❙SMI++ providing:❘Abstract behavior modeling (Finite State Machines)

❘Automation & Error Recovery (Rule based system)

The Control FrameworkD

evic

e U

nit

s

Con

trol U

nit

s


SMI++

❚Method❙Classes and Objects

❘Allow the decomposition of a complex system into smaller manageable entities

❙Finite State Machines❘Allow the modeling of the behavior of each

entity and of the interaction between entities in terms of STATES and ACTIONS

❙Rule-based reasoning❘Allow Automation and Error Recovery


SMI++

❚Method (Cont.)

❙SMI++ Objects can be:❘Abstract (e.g. a Run or the DCS)

❘Concrete (e.g. a power supply or a temp. sensor)

❙Concrete objects are implemented externally either in "C", in C++, or in PVSS (ctrl scripts)

❙Logically related objects can be grouped inside "SMI domains" representing a given sub-system (Framework: Control Unit)


SMI++ Run-time Environment

ProxyProxyProxy

Hardware Devices

Obj

Obj

Obj

SMI Domain

ObjObjObj

Obj

Obj SMI Domain

❙Device Level: Proxies❘ drive the hardware:

〡deduceState〡handleCommands

❘ C, C++, PVSS ctrl scripts

❙Abstract Levels: Domains❘ Implement the logical model❘ Dedicated language - SML❘ A C++ engine: smiSM

❙User Interfaces❘ For User Interaction

❙All Tools available on: ❘ Windows, Unix (Linux) ❘ All communications are

transparent and dynamically (re)established


SMI++

❚SMI++ - The Language❙SML –State Management Language

❘Finite State Logic〡Objects are described as FSMs

their main attribute is a STATE

❘Parallelism〡Actions can be sent in parallel to several objects.

Tests on the state of objects can block if the objects are still “transiting”

❘Asynchronous Rules〡Actions can be triggered by logical conditions on the

state of other objects


SML – The languageclass: HighVoltage state: NOT_READY /initial_state action: GOTO_READY do SWITCH_ON all_in PSS if (all_in PSS in_state ON) then move_to READY endif move_to ERROR state: READY when ( any_in PSS in_state TRIP ) do RECOVER action: RECOVER do CLEAR all_in PSS do SWITCH_ON all_in PSS … action: GOTO_NOT_READY … state: ERROR … object: SubDetHV is_of_class HighVoltage

class: PowerSupply /associated state: UNKNOWN /dead_state state: OFF action : SWITCH_ON state: ON action : SWITCH_OFF state: TRIP action : CLEAR … object: PS1 is_of_class PowerSupply object: PS2 is_of_class PowerSupply object: PS3 is_of_class PowerSupply … objectset: PSS {PS1, PS2, PS3, …}

❚ Devices: ❚ Sub System:

❚ Objects can be dynamically included/excluded in a Set


SML example (automation)

object: RUN_CONTROL state: TEST_MODE when (LHC::STATE in_state PHYSICS) do PHYSICS action: PHYSICS do GOTO_READY all_in SubDetHVS if (all_in SubDetHVs in_state READY) do START_RUN DAQ … move_to PHYSICS_MODE state: PHYSICS_MODE …

object: LHC::STATE /associated state: UNKNOWN /dead_state state: PHYSICS state: SETUP state: OFF …

❚ External Device:

❚ Sub System:


PVSS/SMI++ Integration

❚ Graphical Configurationof SMI++ Using PVSS


Building Hierarchies

❚Hierarchy of CUs❙Distributed over

several machines❘"&" means reference

to a CU in another system

❙Editor Mode:❘Add / Remove /

Change Settings

❙Navigator Mode❘Start / Stop / View


Control Unit Run-Time

❚Dynamically generated operation panels(Uniform look and feel)

❚ Configurable User Panels


Features of PVSS/SMI++

❚Task Separation:❙SMI Proxies/PVSS Scripts execute only

basic actions – No intelligence❙SMI Objects implement the logic

behaviour❙Advantages:

❘Change the HW -> change only PVSS

❘Change logic behavioursequencing and dependency of actions, etc -> change only SMI rules


Features of PVSS/SMI++

❚Error Recovery Mechanism❙Bottom Up

❘SMI Objects react to changes of their children〡In an event-driven, asynchronous, fashion

❙Distributed❘Each Sub-System recovers its errors

〡Each team knows how to recover local errors

❙Hierarchical/Parallel recovery❙Can provide complete automation even

for very large systems


Demo-6

❚Show a simple Hierarchy❙Install fwLHCb_FsmDomains❙In installs standard LHCb FSM Domain

Types❙And it creates:

❙Show Include/Exclude and Enable/Disable❙Show “Temp” FSM and Alarm Handling

VELODCS

VELOMotors

VELOTemp


Sub-detector FSM Guidelines

❚Started defining naming conventions.❚Defined standard “domains” per sub-detector:

❙ DCS❘ DCS Infrastructure (Cooling, Gas, Temperatures, pressures, etc) that is

normally stable throughout a running period❙ HV

❘ High Voltages or in general components that depend on the status of the LHC machine (fill related)

❙ DAQ❘ All Electronics and components necessary to take data (run related)

❙ DAQI❘ Infrastructure necessary for the DAQ to work (computers, networks,

electrical power, etc.) in general also stable throughout a running period.

❚And standard states & transitions per domain.❚ Doc available in EDMS:

❘ https://edms.cern.ch/document/655828/1


FSM Guidelines

❚State Diagram for Trigger and DAQ Domains:

❙Possible intermediate “CONFIGURING” and “STARTING” states if operations slow…


MUONDCS

MUONHV

MUONDAQI

MUONDAQ

Hierarchy

VELODCS

Infrast. DCS HV DAQI DAQ L0 TFC HLT LHC

VELOHV

VELODAQI

VELODAQ

VELODCS_1

VELODCS_2

VELODAQ_1

VELODAQ_2

ECS

VELODev1VELO

Dev1VELODev1VELO

DevN

SubFarm1SubFarm

N


MUONDCS

MUONHV

MUONDAQI

MUONDAQ

Hierarchy & Conf. DB

VELODCS


VELOHV

VELODAQI

VELODAQ

VELODCS_1

VELODCS_2

VELODAQ_1

VELODAQ_2

ECS

VELODev1VELO

Dev1VELODev1VELO

DevN

Conf.DB

3

2

1

1 Configure/mode=“PHYSICS”(Get “PHYSICS” Settings)Apply Settings

2

3

1

1


Demo-7

❚Using type: DAQ_Domain❚Create:

❚Create type VeloBoard❚Integrate veloBoard1, veloBoard2,…

❙Apply recipes on “Configure” command❙Note: There will be a “configurator” object per CU

which gets recipes from DB to cache

VELODAQ

VELODev1VELO

Dev1VELOBoard1

VELOFEE

VELOTELL1


Demo-8

❚Using type: ECS_Domain❚Create:

VELODAQ

VELODev1VELO

Dev1VELOBoard1

VELOFEE

VELOTELL1

VELODCS

VELOMotors

VELOTemp

VELOHV

VELO


The End

❚Questions?


MUONDCS

MUONHV

MUONDAQI

MUONDAQ

Hierarchy & Partitioning

VELODCS


VELOHV

VELODAQI

VELODAQ

VELODCS_1

VELODCS_2

VELODAQ_1

VELODAQ_2

ECS

VELO

clara gaspar, march 2006 lhcb’s experiment control system step by step

Documents

clara gaspar

control systems

configuration data

step slide

tfc slide

database tools pvss

conditions db

monitoring data