The Framework for Control Systems: recent progress and

multi-disciplinary applications

Claudio Bisegni, Salvatore Caschera, Paolo Ciuffetti, Claudio Di Giulio, Giampiero Di Pirro , Alessandro D'Uffizi, Luca Gennaro Foggetta, Francesco Galletti, Riccardo Gargana, Eliana Gioscio, Andrea Michelotti, Alessandro Stecchi

INFN-LNF

Luciano Catani INFN-Roma Tor Vergata

what’s • R&D project aiming at the development of a framework for CS

& DAQ

• started in 2011 with the objective to develop solutions for future INFN needs in CS & DAQ

• introduce new technologies and native DAQ

• the development follows the concept of CS as ‘service’ (for data producers and data readers)

• -> data producers are !CHAOS clients, too!

2

PCaPAC 2012 (Kolkata)

PCaPAC 2014 (Karlsruhe)

• Concept of !CHAOS • proof of concept

• CDS, Cache, CU developed • first !CHAOS installation

XLDB 2012 (Stanford)

XLDB 2014 (Stanford)

• Concept of CDS: !CHAOS Data Service

• prototipe of CDS

SuperB Project (study for the accelerator CS)

multidisciplinary application (CPS)

collaboration with industry (IMA)PCaPAC 2016 (Campinas) • multi-applications project

milestones

3

UI web

UI LabVIEW

Storage

cacheCDSMDS

EU

CU virtual

machine

CU SBC

CU embedded

CU Bus + Ctrl

- cache: live data collection and distribution

- storage: continuous (history) data archiving (DAQ) and history data retrieving

- system configuration and info management

- messaging, alarms, timing

- javascript and Labview modules of User Interface Toolkit

- phy / virt machine: physical computer or virtual machine

- SBC single board computer, e.g. beagle bone)

- standard acquisition bus + its embedded controller, e.g. VME

- embedded: oscilloscope, Libera, ... -

- measurements, feedbacks

components & services

4

commands & data flowUI

CDS

MDS

CU

Meta data

cache

DatasetInputChangeandCommandSubmission

HardwareMetaData

History data

MetaData

HardwareStatusData

LiveData

HistoryData

5

HardwareMetaData

HardwareStatus

!CHAOS services: which sw technologies

MongoDB

CouchBaseC++

C++UI

CDS

MDS

CU

Meta data

cache

DatasetInputChangeandCommandSubmission

HardwareMetaData

History data

MetaData

HardwareStatusData

LiveData

HistoryData

HardwareMetaData

HardwareStatus

ZeroMQ

6

components of !CHAOS framework: libraries hierarchy

7

!CHAOS Common

MDSClient

CDSServiceCU API MDS

Service

CCS

UI API

RPC System

DirectIO System

Event Notification

Logging System

Network Broker

Message Channel

Algorithm &UtilityCode

REST

components of !CHAOS frameworkUI web

UI LabVIEW

CU virtual

machine

CU SBC

CU embedded

CU Bus + Ctrl

CCS

CU API

MDS

MDSCDS Storage

cacheCDS

EU

UI LabVIEW

8

!CHAOS Common

MDSClient

CDSServiceCU API MDS

Service

!CCS

UI API

RPC System

DirectIO System

Event Notification

Logging System

Network Broker

Message Channel

Algorithm &UtilityCode

REST

UI API UI APIREST

UI UI

Storage

cacheCDSMDS

EU

CU CU CU CU

HEP (BTF) HEP (Dafne) HEP (EUSPARC)

UI UI

Storage

cacheCDSMDS

EU

CU CU CU CU

UI UI

Storage

cacheCDSMDS

EU

CU CU CU CU

UI UI

Storage

cacheCDSMDS

EU

CU CU CU CU

Industry (IMA)

UI UI

Storage

cacheCDSMDS

EU

CU CU CU CU

Cyber Physical Systems

!CHAOS applications

9

UI web

UI LabVIEW

Storage

cacheCDSMDS

EU

CU virtual

machine

CU SBC

CU embedded

CU Bus + Ctrl

Experimental application of !CHAOS as control system of the DAFNE Beam-Test Facility (BTF), a part of the DAFNE accelerator complex at Laboratori Nazionali di Frascati.

!CHAOS has been successfully employed in three test-runs to drive and monitor the beam along the transfer line arriving at the BTF facility.

Power Supplies, beam detector DAQ on VME, beam scrapers

VME bus

HEP - BTF

10

!CHAOSCUonBeagleBone+

customCAPE

UI web

UI LabVIEW

Storage

cacheCDSMDS

EU

CU virtual

machine

CU SBC

CU embedded

CU Bus + Ctrl

Development of a CU running directly on a Libera BPM acquisition system of the DAFNE accelerator complex at Laboratori Nazionali di Frascati.

A full acquisition and display system has been developed to monitor the orbit of the beam in the DAFNE damping ring.

BPMs

HEP - DAFNE

11

Libera

UI web

UI LabVIEW

Storage

cacheCDSMDS

EU

CU phy / virt machine

CU SBC

CU embedded

CU Bus + Ctrl

Pilot project in collaboration with and co-funded by an industrial partner, world leader in the design and manufacturing of automatic machines for the processing and packaging of pharmaceuticals, cosmetics and food.

!CHAOS will be used for monitoring assembly lines and to prognosticate failures.A LabVIEW CU running on cRIO reads @16kHz some diagnostic (basically accelerometers) and sends the buffers @1Hz to the CDS

Diagnostics components

cRIO-9068

Industry -

12

UI web

UI LabVIEW

Storage

cacheCDSMDS

EU

CU virtual

machine

CU SBC

CU embedded

CU Bus + Ctrl

A two years pilot Project funded by Ministry for education, university and research

A cloud-aware installation of !CHAOS with CUs running on SBC for sensors monitoring. The CUs and UIs can be distributed over WAN.A configuration tool — developed on purpose — creates on OpenStack the virtual machines hosting the !CHAOS services and installs the backend and custom services. The number of VMs and services instances is configurable.

Sensors monitoring over WAN

Configuration tool

!CHAOS for Cyber Physical Systems

13

UI web

UI LabVIEW

Storage

cacheCDSMDS

EU

CU virtual

machine

CU SBC

CU embedded

CU Bus + Ctrl

The EUSPARC scientific program has three main objectives:• High gradient (laser-

plasma) acceleration for the next FEL and e+/e- colliders

• Advanced radiation sources for photons science

• Physics of high field interactions with matter

Technical Design Report of the new facility is in progress.!CHAOS is the candidates CS for the control of the whole facility, accelerator and experiments.

accelerator, experiments, beamlines

UI MathLab

HEP - EUSPARC

14

Benchmarking a distributed control system

• No standards benchmarks for CS found • results are deeply affected by network interconnection • results are affected by the use of virtual machines that impose HW

resource sharing • There are many “observables” (determinism, bandwidth, latencies,

robustness, reliability, high availability, resource occupation…) in many HW configurations

Control system community may consider working on CS benchmark standardisation

15

!CHAOS benchmark HW setup

• 5 physical Dell M610 (Intel X5550 2.67GHz x4 cores,48GB Ram),running 24x2 (2 test/production environment) virtual machines (Ubuntu 14.04 LTS x86_64).

• virtual machines host: • backend services

(Couchbase, MongoDB, Ceph filesystem)

• !CHAOS front end services (MDS,CDS)

• !CHAOS Clients (CU,UI) • 10 Gbit ethernet (intermediated

by supervisor)

16

ovc3n01 ovc3n02 ovc3n03 ovc3n04 ovc3n05

chaosdev-cb1 backendchaosdev-cb2 backendchaosdev-cds1 CDS1chaosdev-cds2 CDS2chaosdev-cu1 CU1chaosdev-cu2 CU2chaosdev-cu3 CU3

VMs 5 x Dell M610

!CHAOS benchmark: SW setup• Backend services:

• 2 instances of Couchbase (live data) • 2 instance of MongoDB (administration/logging/history).

• Front End services: • 2 CDS services that realise the abstraction of the backend services, • 1 instance of MDS service is

used for administration and orchestration (registration, start, stop) of chaos nodes.

• Test is performed onto the test environment and involves: • up to 3 data producers (CU) • up to 9 consumers (UI).

• Results are compared with performance of localhost which is not affected by the network but just by the its resources (CPU and RAM speed).

17

ovc3n01 ovc3n02 ovc3n03 ovc3n04 ovc3n05

chaosdev-cb1 backendchaosdev-cb2 backendchaosdev-cds1 CDS1chaosdev-cds2 CDS2chaosdev-cu1 CU1chaosdev-cu2 CU2chaosdev-cu3 CU3

VMs 5 x Dell M610

!CHAOS benchmark SW setup

• A CU — set up on purpose — implements a commands to generate traffic (calcBandwidth) at a given rate with a given packet size.

• The CU upon the calcBandwidth command starts pushing a buffer of packet size bytes at the requested rate into the !CHAOS infrastructure.

• The UI issues the calcBandwidth command varying the packet size and the rate. • The UI measures the time needed to the deliver and execute the command and get

the first result back in the live cache. It correspond to the overall round-trip time of the system.

• During the test the UIs and CUs are completely asynchronous, UIs poll the live cache at their maximum rate for new incoming packets from the CUs.

• Tests are executed in safe-mode that guarantees each packet is delivered appropriately to the back end services.

18

!CHAOS benchmark CU cycles (producer)

Cycle time in us varying packet size, strongly dependent by network constraints. Localhost ten times faster. Increasing the number of CDS enhance overall bandwidth.

19

from 1 to 2 CDS

!CHAOS benchmark UI cycles (consumer)

Cycle time in us varying packet size, strongly dependent by network constraints. Localhost ten times faster. Increasing the number of CDS enhance overall bandwidth.

20

from 1 to 2 CDS

!CHAOS benchmark Command Round Trip Time

Command RTT is the time that measure an UI that issues a command and see the result back. Since the UI has to read the payload packet this interval of time depends on the payload.

21

!CHAOS benchmark CPU%

The figures below show the time spent by a CU in user space (so it is the time spent hopefully doing useful work). Increasing the number of CDS we reduce bandwidth bottlenecks.

22

Conclusions• learning (and growing, and fixing) by doing

• not yet fully mature and ready for more complex and critical applications, a.

• new challenging projects (i.e. EUSPARC) are pushing for an early completion of the framework