A PROTOTYPE APPLICATION FOR EVALUATING EPICS AND CAN BUS   Abstract : This note presents the evaluation of EPICS (Experimental Physics and Industrial Control System) and CAN bus (Controller Area Network) made within a collaboration between ATLAS, CMS and ECP/CO. It was performed implementing a prototype for the control of fan trays of VME crates. The description of the system, a brief description of EPICS and CAN, and an appraisal of the tools used are presented.

TABLE OF CONTENTS

 1. INTRODUCTION. 

 2.EPICS.

 2.1 Hardware architecture  2.2 Software architecture  2.3 Application development

3. CAN BUS3.1 Structure 3.2 Characteristics 3.3 Application layer

4. EPICS-CAN PROJECT4.1 Setup of hardware 4.2 Functions implemented

4.2.1 Values and remote commands provided by the fan tray 4.2.2 Characteristics of the software implemented with EPICS

4.3 Development issues4.3.1 Database 4.3.2 OPI 4.3.3 Security 4.3.4 Sequencer 4.3.5 Recipes 4.3.6 Drivers

4.4 Coexistence with commercial applications in the same bus5. APPRAISAL OF THE TOOLS

5.1 Architecture and phylosophy 5.2 Installation 5.3 Compilation 5.4 Database development with CAPFAST 5.5 Channel Access 5.6 Security 5.7 OPI tools 5.8 Sequencer 

5.9 Drivers 5.10 Recipes

6.FUTURE WORK6.1 CAL compliant driver 6.2 Proportional Integral Derivative (PID) control 6.3 Channel Access Client 6.4 Record modification and development

7. CONCLUSION

8. GLOSSARY

9. REFERENCES

10.AUTHORS

  1. INTRODUCTION

A prototype was developed for the control of fan trays and power supplies of VME crates. It was used for evaluating the Experimental Physics and Industrial Control System (EPICS) [1]and the Controller Area Network field bus (CAN bus) [8]. The prototype was built by using the EPICS tool kit for the supervisor and front-end control system and using CAN for the I/O. The design assured scalability by allowing easily the addition of new crates with little development.

Initially those people working on the project had no experience with EPICS nor with CAN. Thus, the project has been very useful to become familiar with EPICS, to learn what can be done easily and what has to be improved, how difficult it is to learn and to use, as well as its various advantages and problems. Also, we learned about CAN and how to use it with EPICS. Another project using EPICS was initiated for doing the control of the gas mixture in the B1 CMS test beam (R. Barrillere, R. Stampfli and M. Zurek). It uses IP modules for front-end I/O.

After a brief introduction to EPICS and CAN, a description of the prototype is given. There follows a discussion and appraisal of the EPICS tools we used followed by possible continuations of the project. Finally we give our conclusions.

2. EPICS

EPICS has a distributed architecture for building scalable control systems and contains a collection of code and documentation comprising a software tool kit. It was created by the Accelerator Technology group at Los Alamos (LANL) and Advanced Photon Source (APS) at Argonne (ANL). It is distributed in source form, so that every site has control of every piece of

software locally. It is used currently by a large collaboration of laboratories and several commercial versions also exist.

Although one needs to be registered to download the software, it is a freeware product available to academic institutions and all the documentation and software is accessible via the Internet. As a consequence it is a very dynamic product, new versions of tools and manuals are available frequently. There is also a "mail exploder" <listserv@apl.anl.gov> that serves as a forum for the interchange of ideas, problems and support from experts in the field.

EPICS has been used for systems of up to 100 crates, 30 Workstations and well over 30000 channels. It is fully distributed which means that it requires no central device or software entity at any level. It has no single point of failure and it is scalable as new front-end systems can easily be added or removed. As long as the network is not saturated no single bottleneck is present. If a specific front-end machine becomes saturated, its functions can be spread over several front-end machines. The name-to-location resolution is done by using broadcasting or multicasting.

2.1 Hardware architecture

EPICS has a two level architecture with front-end machines and operator workstations communicating through a Local Area Network (LAN) via TCP and UDP protocols (Figure 1). In the EPICS terminology, the front-end machines are called Input Output Controllers (IOCs) and the workstations OPerator Interfaces (OPIs). OPI workstations can also act as gateways to other EPICS control systems or to external facilities, such as databases and so on.

 

Figure 1: EPICS hardware architecture.

2.2 Software architecture

The control process running in the IOC is called a "process database" and is made of instances of process elements called records. A record can be considered as something like an object with a set of properties (fields) and associated functions. It must not be regarded as a record in a traditional database. In some documents a record is also referred as a "process block". In the current implementation the minimum size of a record is 200 bytes and specialized records may be larger.

The software architecture is client-server based, providing read/write access directly between any two points (Figure 2). The communication is done via a protocol called Channel Access(CA) which has an Application Programming Interface (API) provided by a set of libraries. CA provides connections between clients and servers throughout the network via TCP/IP.  

Figure 2: Channel Access. Client-server architecture.

In every IOC there is a Channel Access server and a Channel Access client. Clients request values from the process database in the server. Channel Access allows existing components as well as user-written customized applications to perform channel connections, including reading, writing, and monitoring of every field in the database. Channel Access libraries are available under UNIX on 68xxx and SPARC processors, under VMS and under VxWorks on 68xxx based processors. They are written in C.

The process database works as a server, and some examples of its clients are the synoptic display manager, the alarm handler and the finite state machines. Channel Access makes transparent the way that the database is distributed over the IOCs. One can think of Channel Access as a logical bus between IOCs and OPIs.

Man Machine Interfaces (MMIs) run in the workstations. The synoptic screens are executed by display managers such as MEDM (Motif Editor and Display Manager) or DM (Display Manager). Alarms are managed by the ALarm Handler (ALH), and there are other OPI programs like the archiver, BURT (Back Up and Restore Tool) used to save/restore recipes, striptool (for trending), etc. The display manager provides a connection between the operator and the control system, and is able to monitor or modify fields in the database via Channel Access.

The connections are specified by channel names such as "record_name<.field_name>. The run-time display manager (DM) monitors channels and only updates the screen when there are changes. DM is able to do 2000 updates per second on a SPARC IPC.

The alarm handler warns the user when an alarm occurs and provides alarm viewing and logging. Alarms are detected in the process database and then reported to the alarm handler running in the workstation. The alarm handling is configured by an ASCII file into a hierarchical structure.

2.3 Application development

Application development is done on both OPI and IOC sides. IOC development concerns basically the process databases made of records, of which more than 40 types are available in the standard EPICS release 3.12. A database is a set of instances of records linked to each other to define process chains. The idea behind the chains is that a record in such a chain can either obtain information from other records or trigger some action by the following records. There are several tools available for creating databases one of which, CAPFAST, is the most frequently used.

CAPFAST was originally created for electronic circuit design and adapted to EPICS. The EPICS library contains all the standard records and these are represented as graphical symbols and connected via lines. The rest of the properties of each symbol (fields of the record) are edited in a properties window. The schematics created with this library are translated into the EPICS database format by a script.

There are other tools available to produce databases, such as the Database Configuration Tool (DCT) or the Graphical Database Configuration Tool (GDCT). The main differences are that CAPFAST provides hierarchical schematics and DCT and GDCT do not. On the other hand, DCT and GDCT are free but CAPFAST needs a license.

Application development normally means database configuration, but this is not always enough. Although there are drivers available for many types of hardware, in some cases it might be necessary to write a driver for a specific device. This is possible since the sources are provided and there are also templates available for building drivers. Drivers are programmed in C, and they can be downloaded into the IOC as object files for testing before including them in the general device driver support.

It is possible to write new record types if necessary and the source of the standard records is available to be used as a template. In this sense EPICS is fully extendible and there are several sites where they have developed their own record types according to their specific needs. It is not necessary to know any programming language for developing an application (configuration of database), but a good knowledge of C and VxWorks is necessary for writing drivers and record types.

EPICS provides a facility to implement Finite State Machines easily by writing code in State Notation Language (SNL). It has a syntax close to C, where all the actions are executed in the transitions. It is possible to define different state sets (machines) synchronised by event flags.

Variables are assigned to database records by specifying the name of the channel and this can be reassigned dynamically. SNL is converted to C code, from where it is compiled into an object file for being loaded into the IOC. There is also an FSM for the OPI, called Xseq, but it is not widely used. SNL has a syntax designed to implement the transitions easily. The object code is loaded at start up, and several instances of the same finite state machine can be called with different parameters. Since the FSMs are started after EPICS, they can be stopped and started at any time.

Besides the database configuration there is also work to be done on the OPI side. User interface displays are created by using graphical "object oriented" editors such as EDD and MEDM. These both use Xwindow but MEDM is based on Motif and EDD is not. The standard distribution contains a lot of different objects (monitors, valuators, etc) which can be configured in terms of static colour, dynamic colour, associated channels, size, etc. Quite complicated synoptic screens can be created by non-specialised people. Tools like the alarm handler and archiver are configured by ASCII files loaded either when they start or chosen by menus. They can be started from the general display (DM or MEDM).

3. CAN BUS

CAN bus is a serial data communication bus for real-time applications and was developed by the German company Robert Bosch in the mid 1980's for the automotive industry. It has the ability to connect together electronic control systems in vehicles with reduced cabling and high reliability. Thus, it is used as a field bus to get signals from sensors and send commands to actuators as well as to connect to Programmed Logic Controllers (PLCs). CAN is an international standard (ISO 11898) and various semiconductor manufacturers (Philips, Intel, Motorola, etc) and software companies offer CAN products. A wide range of complete systems, interface cards, I/O devices and development kits can be found on the market. As it is an international standard it is an open system and there are no license fees.

Several users groups have been formed, the most important of which is CAN in Automation (CiA), created in 1992. This group is mainly concerned with standards issues. CAN bus has been recommended as a CERN standard by the Working Group on Fieldbuses [9] together with PROFIBUS and WORLDFIP.

3.1 Structure

The original CAN protocol specification corresponds to the data link layer (layer 2) of the ISO/OSI reference model. The ISO 11898 standard includes the physical layer but the application layer definition is not included in the CAN specification. The system designer has to define it or use one of the layered standards (see 3.3).

Figure 3. OSI/ISO model and relation with CAN bus.

3.2 Characteristics

The network architecture has a flat bus topology (Figure 4). The basic unit for data transmission is a frame which contains 8 bytes of data, as well as the identifier, CRC, start field, etc.

Figure 4. CAN bus topology.

The capacity of the bus depends on its length. According to ISO 11898, the maximum capacity of the bus is 1 Mbit per second for a 40 m bus length, although higher rates can be reached (e.g. the Philips CAN controllers are specified for 1.6 Mbit/s over 10 metres).

The addressing mechanism is content-oriented and nodes do not have an address. Instead, each data frame has a unique identifier and frames are broadcast to all the nodes on the bus. Frames are then accepted by a node if the node is configured to deal with identifiers of that type.

Any real-time data communication requires rapid bus allocation as well as fast data transmission rates. In CAN bus, the access to the bus is arbitrated using CSMA/CA (Carrier Sense, Multiple Access with Collision Avoidance) and the priority of the messages is determined by the identifier. This arbitration mechanism is more efficient than the one used in Ethernet (CSMA/CD, CSMA with Collision Detection), where collisions are not avoided just detected, hence wasting part of the capacity of the network.

As a consequence of the addressing and bus allocation mechanisms, the bus is multimaster. There is no functionality concentrated on a single node, and the number of nodes is limited only by the physical drivers.

CAN is extremely reliable and has several mechanisms for error detection and confinement. For error detection five mechanisms are used: three at the message level (CRC, frame check,

acknowledgement check) and two at the bit level (bit monitoring and bit stuffing). The confinement is a method for discriminating between temporary errors and permanent failures and localising station failures.

Robustness is another CAN feature. It is capable of operation in extremely harsh environments. If the ISO physical layer is used, operation will continue even if either of the two wires in the bus is broken, shorted to power or to ground, or the two wires are shorted together.

The CAN protocol is always implemented on a chip (CAN controller). There are two kinds of controllers: basic CAN and full CAN. A basic CAN controller does not perform the acceptance filtering based on the identifiers, which has to be done by the CPU (usually a micro-controller) that it is connected to. On the other hand, a full CAN controller performs the filtering and passes to the CPU only the relevant messages. This scheme does not load the CPU, but has a drawback as only a limited number of identifiers can be accepted.

In the first version of the protocol 11 bits were used for the identifiers (Standard CAN). This allows 2032 possibilities as the last 4 bits cannot all be 0 simultaneously. A later standard,"Extended CAN", provides 29 bit identifiers. This appeared in CAN 2.0, where Standard CAN is specified in part A and Extended CAN in part B.

The simplicity of the protocol makes the cost of related products cheap. In particular, very low cost controllers (~ 10$) and interface devices exist. Devices can be custom built with the help of chip sets. Microcontrollers with embedded CAN controllers are also available. Its good properties have led to CAN being used in a wide range of applications other than the automotive industry, including medical systems, nautical instrumentation and so forth.

3.3 Application layers

CAN is a protocol that specifies only how to interchange data (level 2 in the OSI layered model). In a real application there are other things that have to be defined: the meaning of the data within the frames, the assignment of identifiers, how to deal with errors reported by the CAN layer, acknowledgement of messages, transmission of data longer than 8 bytes, etc. This additional functionality is in the CAN application layer that corresponds to layer 7 in the ISO/OSI reference model. CAN is very flexible because every system designer can have his own solution, but an effort has been made to standardise. This effort is not in vain because, when standards are produced, the benefits of open systems appear: equipment from different manufacturers can be mixed, tools for designing applications appear, scale economies, etc.

Four main protocol standards have been produced: Devicenet, Smart Distributed System (SDS), CAN Kingdom and CAN Application Layer (CAL). There are others but they are not so widely used. All are implemented in software and provide the facilities needed for the communication. Further standardisation has been carried out with device profiles, trying to standardise how to build the application and not only the communication issues. An example is CANopen, which is a new layer on top of CAL.

CAL, defined by the CiA group, is the most used, versatile and open high-level protocol. It is divided in four parts:

CMS (CAN Message Specification): provides object-oriented communication with a client-server model.

NMT (Network Management): provides functions like initialisation of the network or monitoring of correct functioning.

DBT (Identifier Distributor): permits dynamic distribution of identifiers used by CMS even while an application is running in the bus.

LMT (Layer Management): configures parameters of CAL and CAN.

4. EPICS-CAN PROJECT

The EPICS and CAN evaluation was done by developing a prototype for controlling the crate fan trays made by Wiener [10]. We defined some reusable software modules at both OPI and IOC levels in order to built an application that is easily extendible to many crates and to make some of the software reusable in future applications. With this prototype we learnt about EPICS, its features, its advantages and its shortcomings.

4.1 Setup of hardware

We used a Sun Workstation UltraSparc 1 (Solaris operating system) for development and running MMIs (OPI). We used a VME crate with a MVME167 CPU running VxWorks 5.2 and EPICS R3.12, one GreenSpring VIP616 IndustryPack carrier board and two CAN IP modules TIP810, and two VME crates with CAN Wiener fan trays as objects to be controlled (Figure 5). Both Wiener fan trays and TIP810 follow the basic CAN specification with 11 bit identifiers (standard CAN). TIP810 CAN IP modules are manufactured by TEWS DATENTECHNIC GmbH. The fan trays use a layer 7 protocol called lowCAL. The protocol is a subset of CAL that provides some of the CMS services.    

Figure 5: Hardware architecture of the prototype. 

A PC running a commercial CAN application (under Windows 3.1) was also connected to the bus. Thus, I/O communications were done by a CAN bus with four nodes (two fan trays, one PC, one IOC). The CERN network was used for the communication between OPIs and IOCs.

4.2 Functions implemented

4.2.1 Values and remote commands provided by the fan tray

The fan trays have a CAN node built in, through which it is possible to read values by request, such as currents, voltages, fan speeds, temperatures, status and configuration data. The fan tray sends also a frame status when an error occurs. It is possible to send commands to a specific fan tray or to every fan tray on the bus (general call). The commands available are: switch on/off, system-reset and change speed. The speed and switch on/off can be disabled.

4.2.2 Characteristics of the software implemented with EPICS.

Two operational modes, Manual and Automatic, were specified. The system provides a regulation of the temperature (control loop) when it works in automatic mode. It increases or decreases the fan speed according to the temperature inside the crate. When manual mode is selected the operator can change the fan speed by hand. Each mode has different alarm limits for the temperature or the fan speed. When the mode is changed, the recipe associated with the new mode is automatically loaded. Moreover, another mechanism is provided to assure that the configuration the system had before stopping is loaded automatically at startup.

Three user levels were defined: observer, operator and expert. The observer is able to see all the data (temperatures, voltages, currents, fan speeds and alarms) but has no permission to send any commands. The operator, who must give a password, has permissions to switch ON/OFF the

crate and to perform a SYS RESET. The expert, who must give a different password, is in charge of the operational maintenance of the crates. He is able to set up all limits and change mode of operation (automatic, manual).

The MMI is organised hierarchically. The top level contains an overview of all the crates from where the specific information can be shown in a bottom window with also the air temperature and the fan speed plot in a chart. The alarm handler may be called from all the screens and includes a tree structure with many crates, each one has several branches (voltages, currents, communication, etc.). The archiver tool is used for archiving the different temperatures and fan speeds read from the crates.

The Figure 6 shows the most complete synoptic display (expert user), which is composed of two windows. The one on the top is a menu from where the user chooses the fan tray he is interested in. This window includes two special buttons on the right side, one of them for exiting, and the other for sending commands to all the crates.  

Figure 6: MMI of the system. 

The bottom window displays the information related to the selected fan tray. The fan tray provides values of currents and voltages for the eight channels available. These values are presented in two rows on the top of the window.

The six individual fan speeds, the average speed and the speed requested (nominal) are presented on the left side of the display together with the commands for changing the speed. This speed can be changed with a scroll bar, three buttons (low, middle and high) or by typing the requested speed in revolutions per second.

In the middle of the panel the three temperatures and the operational status are shown. The commands for switching the crate on/off, performing a system reset or reading the configuration data (reinit) appear on the right side of the panel.

On the bottom the user has the button for calling the alarm handler, the status, the commands for changing the period of polling and the button for exiting the current fan tray.

4.3 Development issues

All the software was developed so that it can be reused in the same application when adding new elements to be controlled (fan trays), or also in future similar applications.

4.3.1 DATABASE

The EPICS database was created using CAPFAST and developed with a hierarchical approach. The database is configured by schematics made of symbols that represent records or another schematic, forming a tree structure without limitation on the number of levels. For our project we chose to develop a symbol that represented a fan tray. The fan tray symbols are used on the second level of the hierarchy (Figure 7). With these building blocks the whole system is built so it can be easily extended. It is just necessary to give the appropriate name and properties (like the CAN address, name etc.) of each symbol. The hierarchy is further used in the design of the fan tray symbol since there are tasks that appear more than once with different parameters.    

Figure 7: CAPFAST hierarchy.

Every fan tray has an assigned address. Moreover, there is a global identifier to address all of them with the same call. Normally it is the identifier 127, but it is also configurable. From the software point of view the general call is implemented just like another fan tray.

A fan tray symbol is subdivided in different modules: reading configuration data that are needed for the conversion of the currents and voltages, reading of values, sending commands and regulation. A module is a set of records connected together and triggered either periodically, by I/O interrupt or by operator command.

The module used to read values is processed periodically. The readout of the data (temperatures or fan speed) is done by sending a request CAN frame to the hardware, which answers with the value for the requested data. In the protocol used, a frame carries more than one parameter to use fully its data field (a sort of multiplexing). Each parameter is associated with an input record in the database. If each record were to send a request for information, then multiple identical frames would be received. To avoid this duplication, one particular record within a frame requests information via the bus and the other associated records just wait for its arrival. For instance, three temperatures are encapsulated in the same frame and in the readout schematic there is a record for each of them. One record is periodically scanned (master) and the other two are processed automatically when the frame arrives.

The command sending module is processed upon request of the user (switching ON or OFF a crate etc.). As a result, a frame with the command is sent.

The regulation module is composed of a subroutine record. The C function called from this record takes the value of the air temperature and generates a command to speed up or slow down the fan. The decision of what to do is made depending on some thresholds provided for the alarm limits of the records for the read of the temperatures.

A "fan tray symbol" which represents a fan tray has 100 records due to the choice of implementing the application layer of CAN within the database. This approach needs extra records that would not be necessary if the application layer was included in the EPICS device driver. We did not implement it in this way because the main goal of the project was to evaluate the whole EPICS toolkit, not to write specific drivers.

4.3.2 OPI

Man machine interfaces are created using EDD and executed using DM for synoptic design, ALH for alarm handling, the archiver for trending and archiving, etc. Displays are hierarchically organised. The top level contains a menu from which all crate synoptics can be selected. This main display includes buttons to login as operator or expert user. The default user level is observer. The displays were designed in a generic way, which means that they can be used for many fan trays depending on the parameters they are called with.

4.3.3 SECURITY

Security is implemented using Channel Access security procedures. Security implies OPI username (UNIX username) and hostname. It is configured in an ASCII file loaded at start-up. The file contains definitions of User Access Groups (UAGs), which are groups of UNIX OPI users; Host Access Groups (HAGs), groups of OPI or IOC hostnames; and Access Security Groups (ASGs). An ASG is composed of read and/or write rules. A rule allows UAGs and/or

HAGs to access the records in the database with the appropriate ASG field. The rules can be made dependent on values of database records. We defined three ASGs: expert, operator and observer. For the implementation, a workaround was used to make a login every time the user or level was changed. We created a UNIX account for every user with a call to DM included in the UNIX ".login" file. It is then easy to create different display interfaces for each user. This is completely transparent to the user, who just sees a prompt asking for operator or expert password, and then the appropriate screen, or the error message in case of typing a password incorrectly. With DM the feedback for a forbidden command is a white box masking the parameter. It is the same kind of feedback as for disconnected channels.

4.3.4 SEQUENCER

Two Finite State Machines (FSM) are implemented for each fan tray. One of them manages the operational modes and changes recipes and permission when the expert user changes between AUTOMATIC and MANUAL mode. The other triggers the module for reading configuration data when the system starts, when a fan tray is switched on or when the expert user calls the "re-init command". The two finite state machines are implemented in the same program (two state sets). The same code is reused for all fan trays and is started with different parameters.

4.3.5 RECIPES

EPICS core files were modified to be able to restore the Status recipe before the start-up of the database in order to start the application in the same state as it stopped the previous time. This is called "bumpless" restart in the EPICS community.

The system uses a set of C functions for saving/restoring data running in the IOC. There is another tool named BURT, but it loads the recipes after the startup of the database and these functions run in the OPI workstation. The prototype implements three recipes: Status, Automatic and Manual. The Status recipe contains the value of ON/OFF, AUTOMATIC/MANUAL and the nominal speed and is saved when a value changes. Both Automatic and Manual recipes are used to keep the values of alarm limits when the operational mode changes.

4.3.6 DRIVERS

To access the CAN bus from EPICS there are three drivers available which meet different needs. We used the driver developed by Andrew Johnson at the Royal Greenwich Observatory (Oxford). It provides the basic CAN functionality, send and receive message, with no application layer built in. The driver is suited for tip810 IP modules, and has a modular structure, so it is easy to develop a driver for another IP module by changing a small specific part within the driver. The driver from BESSY (Berlin) includes the application layer for CAN (lowCAL, which is a subset of the CMS part of CAL), but it is not suitable for our hardware. A third driver exists at DESY (Hamburg) that is specific for their system, hence not easily adaptable for other installations.

Some small modifications were needed to adapt the driver to the Wiener protocol and to simplify the configuration of the database with a hierarchical approach.

4.4 Coexistence with commercial applications on the same bus

In theory, there should not be any hardware incompatibility when connecting equipment from different manufacturers that conforms to the CAN standard. Problems can arise when different CAN application-level software is used or when there is more than one controller issuing commands.

We added a PC running the control software provided by Wiener to the CAN bus used by EPICS. Both control systems operated correctly without any incompatibilities. A pre-requisite for this was, of course, that the same high level protocol (e.g. same CAN identifiers) was used for both. In addition, the EPICS driver and application were able to accept unsolicited messages from the fan trays triggered by the program running on the PC.

5. APPRAISAL OF THE TOOLS

This section covers the subset of EPICS tools we used.

5.1 Architecture and philosophy.

The EPICS source files are organised in a hierarchy of directories that probably the developer has to be aware of. For applications using existing drivers and records, detailed knowledge of this is not needed, but the EPICS administrator has to know it very well.

Everything was easily accessible. For instance, since sources were available, it was possible to change library modules, procedures in driver support, devices and drivers as necessary. As EPICS is used at many sites there are a lot of templates for records, devices and drivers. Unfortunately, the directory architecture depends on the platform. It is not the same on HP as on Sun, for instance. The scripts are different and the process of compilation as well. As a consequence, maintenance is not easy.

Whilst some EPICS documentation is very good, in general the documentation is variable in both quality and level depending on its origin. In particular, whilst tools normally have reference documents they do not all have explanations of how they should be used. In addition, the documents are distributed over different Web sites.

EPICS is easily scalable and extendible by adding new drivers, records, OPIs, etc. Databases can be moved from one IOC to another one. As EPICS source files are available, there are people in various laboratories working with database interfaces as well as new products like JAVA and CORBA.

There are no tools to find the reason for a crash. A debug tool would be desirable.

Programming errors in drivers, records or finite state machines (such as VME bus error) crash the IOC and reboot it. This is a result of the lack of interprocess protection in VxWorks.

The name resolution mechanism uses broadcast. In principle this is positive, since one automatically finds where process variables reside. However, for large systems it is inconvenient because all IOCs receive all the broadcast messages, consuming CPU time and making the initialization process long. The network could also be a bottleneck. The ideal case would be to have the possibility to choose between a broadcast and a name server approach.

5.2 Installation

The installation is not straightforward. A lot of environment variables have to be set and some GNU packages have to be installed before compiling. There is no installation procedure that helps the user to make the system run (like Macintosh installation programs). As EPICS is open and flexible, the installations are not standard. Normally it is necessary to deal with Makefiles.

5.3 Compilation

Once the system is installed, compilation is very intuitive. An empty application skeleton is created and set up by script commands. The only thing the developer must take care of is to put everything in the correct place. SNL and subroutines in the ~/src directory, displays in ~/dl, CAPFAST files in ~/capfast.

If the application requires new drivers or records, they can be developed and debugged in a local directory without disturbing other users.

If there is any problem, it is necessary to deal with Makefiles, which are not very intuitive and have a lot of cross-references. When something is wrong in the compilation of the CAPFAST schematics it may be necessary to go through the produced ASCII database to find the error.

5.4 Database development with CAPFAST

Database development, for which we used CAPFAST, is the most important part of the control system. Databases are created hierarchically with schematics composed of graphical symbols that represent EPICS records. The hierarchy feature allows one to have user symbols that contain records and other user symbols and, therefore, to build schematics with a tree structure. Thus, several instances of the same entity may use the same symbol with different names and properties. Schematics are very intuitive since they represent the data flow of the values in a process. Unfortunately, there is no array of records.

There is not a possibility of choosing via menu between the different values of a field. Moreover, it does not provide any utility to check errors at edit time or at compilation time (check erroneous links, for instance). Modifying a database structure or adding a new one requires rebooting the IOC, which is inconvenient. There is no run-time debugging tool.

CAPFAST licenses are costly. It is not clear if this tool would scale for a large system. Once a part that has to be repeated many times is identified a script could replicate it. This script would not be very complicated as the database is an ASCII file.

A record is composed of common fields and specific fields for the purpose of the record. The common field has a size of 200 memory bytes in the IOC.

5.5 Channel Access

Channel Access can be considered as a software bus. It is a set of functions comprising a library for providing transparent communications between clients (e.g. OPI tools, sequencer) and servers (process database). It makes the IOC level completely transparent to the OPI level. Everything at OPI is specified by its channel name, without taking care about where the data are.

5.6 Security

The Channel Access Security facility, based on UNIX usernames and passwords, can be used for protection. There is no way to let IOCs know which OPIs (user id) are writing/reading channels. It is also impossible to know which users are logged in. There is no possibility of changing the user identity from the OPI with the standard EPICS tools. We had to do a workaround, a login command within a new terminal was executed to be able to change to a new user identity.

The feedback mechanism for a non-allowed operation is poor.

5.7 OPI tools

Many tools for creating graphical user interfaces are available. There are tools for displaying synoptic diagrams, for alarm handling and others for archiving, etc.

The first feature we missed was the integration of the different tools for application development and in the MMI run-time environment. The tools run in independent processes in the workstation, so the operators have a set of applications running concurrently (displays manager, alarm handler, archiver, etc.) in different windows. This can be confusing.

On the other hand, operator interfaces are created intuitively. Channels are connected assigning the channel name, without needing to know where the records are physically located (broadcast mechanism). It is possible to modify a display and the channels are reconnected without problems. It is even possible to create new displays at run time, keeping the old ones running.

The alarm handler is very poor. It misses fundamental things like customization of the alarm and command logging and global alarm masking (R3.12 release). The alarms are generated at IOC level, but they are handled at OPI level, so all users do not see the same information as the alarm handling is not system-wide. Modifying the configuration, for instance when adding a new fan tray, is not convenient. As the alarm handler knows nothing about users it doesn�t know who masked or acknowledged an alarm or whether they had sufficient privilege.

The alarm handler can trigger a command on alarm transition, like for example a Web browser to present a help window to the user.

5.8 Sequencer

SNL code is similar to C. The link to C code is very flexible, and it is possible to add C functions (even isolated C instructions) by typing escape sequences. It is easy to learn for C programmers, but it could be difficult for non-trained people. It is automatically compiled into object modules together with the rest of the application.

Finite state machines can be developed easily. Variables can be assigned to channels in the database dynamically and, what is more important, a FSM can be started/stopped at run time, since it is started after EPICS.

Theoretically, the same piece of code can be executed with different parameters, so we can define a FSM for a piece of equipment, and reuse the same program as many times as necessary. Unfortunately, it creates as many tasks as instances of the FSM, and at least in our system it crashed quite often.

There is no graphical design tool, such as for some commercial packages, where you draw a state diagram and it is automatically converted to executable code. There is no debugger.

The developer cannot specify pre- and post- transition actions, which can be needed for a large production system. Also, nested states are not included in its capabilities.

5.9 Drivers

Although there are a lot of EPICS drivers for many types of hardware, most of them available are targeted to the American market. Since hardware development is continuously growing it may be necessary to have new drivers for attaching new hardware.

The code of existing drivers is available so there are templates to help in writing new ones, which can be developed and tested separately, without including them in the EPICS base until they are completely operational. However, there are no tools for debugging EPICS drivers and one can only use "printf".

5.10 Recipes

The name of the recipe to be loaded at start-up cannot be chosen. A workaround is to rename any file that should be restored before starting the system.

The save/restore functions can be called whenever needed for creating or restoring recipes (e.g. in a subroutine record or SNL code). The fields to be restored or saved are specified in an ASCII file. It is possible to save them periodically, when a value changes within a given period of time, or manually.

Due to NFS access, reading or writing recipe files slows down the system.

6. FUTURE WORK

6.1 CAL compliant driver

To reduce the number of records in the database and put the application layer in the device support layer, we could adapt the driver developed at BESSY for the tip810 hardware. In this case the database would be smaller and independent of the type of field bus used.

6.2 Proportional Integral Derivative (PID) control

The regulation of the temperature by modifying the fan speed was carried out by a subroutine record. However, EPICS provides a PID record that can be used to perform the control.

6.3 Channel Access client

As already mentioned, there is an API which is available for developing Channel Access client applications. There are also some templates for helping beginners. It would be useful to have some experience in creating client applications, in order to evaluate how easy it would be to improve EPICS according to specific needs.

6.4 Record modification and development

Depending on the type of applications, it can be necessary to develop a new EPICS record to fit the requirements. For instance, if the prototype presented were going to be put in production, it would have many more fan trays (perhaps hundreds), and the current approach would not be valid. Possibilities would be to create a record for implementing a fan tray and adding the CAN application layer to the device driver.

7. CONCLUSION

In the following, we put the experience gained from this work in the context of the possible use of EPICS for controls in the LHC experiments.

The two-level architecture of EPICS fits very well to the distributed nature and the organisation of an LHC experiment. Functions (e.g. GUI, alarms) can be mapped onto workstations and IOCs can take care of sub-units of the detector in an autonomous way. They could be supervised either in stand-alone mode or integrated in the overall detector. The client-server model using the Channel Access mechanism is well suited to the application.

The availability of the source code gives complete flexibility; modules can be modified or new ones created using old ones as templates. The EPICS community is very dynamic and there is continuous development, improvements and ports to new hardware. Of course, local changes to the source code would imply a commitment to long-term maintenance unless they were to be accepted by the EPICS development team.

On the other hand, as pointed out in Chapter 5, many tools would need functionality to be added before using them in an experiment. The main points are the following: the tools are not integrated but run more or less independently from each other; connection of the EPICS configuration (records) and logging facilities to a powerful commercial data base system would be needed; a global alarm handler with additional features would be needed.

In general, EPICS needs a substantial level of training and expertise, both for the developer and the end user (operator). This is available in the accelerator community, but much less so for the experiments. Operators have very little training and are normally only knowledgeable about a limited fraction of the detector. Those people developing controls applications do this job only a small fraction of their time. If such people were to implement procedures in the IOCs, more security, diagnostic and debugging features would be essential. In addition, more checks at configuration and compile time would be needed.

In summary, EPICS could be a solution for controlling LHC detectors, but a sizeable support team would have to add the functionality needed, help to set up the system at the various sites, train and guide the developers of the sub-detectors and help in trouble shooting. However, the investment in capital would be small.   

8. GLOSSARY

ATLAS: A Toroidal LHC AparatuS BESSY: Berliner Elektronenspeicherring-Gesellschaft für Synchrotronstrahlung BURT: Back Up and Restore Tool CA: Channel Access CAN: Controller Area Network CAL: CAN Application Layer CERN: European Laboratory for Particle Physics CMS: Compact Muon Solenoid DCT: Database Configuration Tool DESY: Deutsches Elektronen-Synchrotron DM: Display Manager ECP/CO: Electronics and Computing for Physics / Controls Group EDD: Editor Display Design EPICS: Experimental Physics and Industrial Control System 

FSM: Finite State Machine GDCT: Graphical Database Configuration Tool IOC: Input/Output Controller IP: Internet Protocol LAN: Local Area Network MEDM: Motif Editor and Display Manager OPI: OPerator Interface TCP: Transport Control Protocol SDS: Smart Distributed System SNL: State Notation Language UDP: User Datagram Protocol

  

 

9. REFERENCES  

[1] EPICS home page, http://epics.aps.anl.gov/asd/controls/epics    [2] Overview of the Experimental Physics and Industrial Control System (EPICS). http://csg.lbl.gov/EPICS/OverView.html [3] EPICS: A brief description, http://www.syfit.lsnl.gov/sot8/EPICS/EPICSdescription.html [4] EPICS architecture. L.R. Dalesio (LANL), M.R. Kraimer (ANL), A.J. Kozubal (LANL), http://www.aps.anl.gov/asd/controls/epics/EpicsDocumentation/EpicsGeneral/EPICS_Architecture.ps [5] APS controls overview, http://www.aps.anl.gov/asd/controls/epics/EpicsDocumentation/APSManuals/aps_ctls_ov.html [6] LHC Controls and EPICS, D.R. Myers, http://ecpcowww.cern.ch/EPICS/epics2-1.html [7] A Common Control System for the LHC experiments, D.R. Myers, http://ecpcowww.cern.ch/EPICS/lcb1-1.html    [8] Le bus CAN, Controller Area Network, D. PARET, DUNOD Paris 1996 [9] Recommendations for the use of field buses at CERN, G. Baribaud et al., http://ecpcowww.cern.ch/fieldbus/report1.html [10] CAN-Bus Interface for WIENER Crate Remote Control, A. Ruben, A. Köster, , Wiener fan tray manual. [11] CAN in Automation ( CiA), http://www.can-cia.de [12] Design of a distributed Application Process with CAL, K. Etschberger, 1st international CAN Conference (ICC94), Mainz 1994. [13] Open communication and interoperability within CAL-based networks, K. Etschberger, 2nd international CAN Conference (ICC95), London 1995.

10. AUTHORS  

Helfried Burckhart                CERN, PPE Division, OPD group David Fernández Carreiras   CERN, ECP Division Manuel González Bergés      CERN, ECP Division, CO group Herve Milcent                      CERN, ECP Division, CO group David R. Myer s                    CERN, ECP Division, CO group