The CERN CMS Tracker Control SystemF. Drouhin, L. Gross, D. Vintache, A. Marchioro, C. Paillard, C. Ljuslin, L. Mirabito, P.G. Verdini

Abstract— Due to the high integration level of the experimentsplanned for the Large Hadron Collider (LHC) of the European Or-ganization for Nuclear Research (CERN), the data acquisition andthe control systems need complex developments both in hardwareand software. The purpose of this paper is to describe the controlsystem of a sub-detector of one of the CERN experiments, theTracker of the Compact Muon Solenoid (CMS). The CMS Trackercontrol system is based on dedicated hardware and software.The hardware is based on the Front End Controller (FEC), aninterface board that hosts token rings for the communication withthe Control and Communication Unit (CCU) modules. These inturn contain dedicated I/O channels for the front end readoutand control chips. The software is built in layers: one devicedriver, a C++ dedicated application program interface (API) plusa database for the storage of all the information needed for thefront end electronics. This system will also be adopted in someother CMS sub-detectors.

Index Terms— Control System, Software, Oracle Database,ASIC

I. INTRODUCTION

THE CMS apparatus [1] is a multi-purpose detector de-signed for operation at the CERN LHC [2] and optimized

for the identification of the Higgs boson and the measurementof its properties over a wide mass range, as well as for thesearch of new physics phenomena.

A high-precision muon detection and tracking system isinstalled around a four Tesla superconduction solenoid. In-side the magnet are located the hadronic and electromagneticcalorimeters, and the Central Tracker. The CMS Tracker [3]consists of an inner Pixel system and of an outer Silicon StripTracker (SST); the control system of the latter is described inthis paper.

The SST pseudorapidity coverage ranges up to η ≤2.5.It will provide at least ten points per track, resulting in atransverse momentum resolution for isolated tracks in thecentral region better than δp⊥ ≈ (15×p⊥+0.5)%, where p⊥ ismeasured in TeV/c. This in turn will result in a reconstructionefficiency better than 98 % for muon tracks over the full

F. Drouhin is with Universite de Haute-Alsace, Mulhouse, France. Cor-responding author: Frederic.Drouhin@cern.ch, Institut de Recherches Sub-atomiques de Strasbourg, Batiment 21, 23 rue du Loess, BP28, F67037Strasbourg Cedex

L. Gross is with Institut de Recherches Subatomiques de Strasbourg, FranceD. Vintache is with Institut de Recherches Subatomiques de Strasbourg,

FranceA. Marchioro is with European Organization for Nuclear Research, Geneva,

SwitzerlandC. Paillard is with European Organization for Nuclear Research, Geneva,

SwitzerlandC. Ljuslin is with European Organization for Nuclear Research, Geneva,

SwitzerlandL. Mirabito is with Institut de Physique Nucleaire, Lyon, FranceP.G. Verdini is with Istituto Nazionale di Fisica Nucleare, Pisa, Italy

pseudorapidity range, and better than 90 % for high energyelectrons, while keeping the occupancy and the amount of“ghost” tracks at the percent level. These features require asingle-hit resolution of ≈ 20µm, well within the possibilitiesof the technology chosen.

The SST has been designed to guarantee stable operatingconditions for ten years of LHC running, nominally. One ofthe critical issues for the viability of the system is the long-term survivability after heavy irradiation. The CMS SST willoperate in a hostile environment, where the integrated particlefluence, expressed in 1 MeV neutron equivalent, is expectedto reach 1.6 × 1014cm−2. In order to minimize the negativeeffects of annealing of radiation-induced damage to the Siliconbulk, and to prevent thermal runaway in the later stages ofoperation, the ambient temperature in the SST will be keptaround −20o C, when the saturation vapour pressure of waterover ice corresponds to 3000 ppm.

The need to compensate for accrued radiation damage inthe sensors and in the front-end electronics has led to thedevelopment of complex, programmable readout electronics;the extreme environmental conditions have prompted thedevelopment of a radiation-hard ASIC which can be deployedon every individual module and which can be used to monitorthe temperature of the sensors and of the front-end hybrid,the low voltage levels and the bias currents drawn by thesensors. The control system for the CMS SST must becapable of monitoring the temperature, humidity, voltage andcurrent measurements coming from 16,000 modules, and atthe same time ensure the download of correct parameters tothe front-end electronics while checking for the occurrenceof Single Event Upset phenomena in the chip registers. Adedicated control system, both in hardware and in software,has been developed to face this challenge.

With increasing radiation exposure, the response of thesilicon sensors as well as that the readout electronics is bound tochange. Therefore, for the CMS Tracker special electronics hasbeen developed whose mode of operation can be modified byprogramming specific registers. The Single Event Upset (SEU)phenomenon can change register values (bit changes).

There are five kinds of front end chips shown in figure 1:- The APV [4]: front end chip for the multiplexed readout

of 128 strips;- The APV Multiplexer (APV Mux): chip that multiplexes

the outputs of two APV on a single optical link;- The PLL: programmable delay and clock rephaser;- The Laserdriver: used for the transfer of data between the

front end electronics and the Front End Driver (analogueto digital converter with memory) over an optical fiber;

0-7803-8701-5/04/$20.00 (C) 2004 IEEE

Fig. 1. A Tracker module with two silicon sensors and the electronic hybrid that hosts the front end electronics

- The DCU (Detector Control Unit): this chip is very impor-tant in the CMS Tracker control system because it givescondition parameters for voltage, current and temperature.It contains an eight channel analog to digital converter,two constant current sources and a temperature sensor. Itwill monitor two sets of thermistors, one on the sensorand one on the hybrid, its own internal temperature, thesilicon detector leakage current and the two (1.25 V and2.5 V) low voltages. So there are three temperatures, twovoltages and one current measurement for each hybrid.Moreover, each DCU has a unique hardware identificationnumber that can be read by software. This identificationnumber will be used in order to identify in the finalTracker construction the position of each module andto act as a link between the construction database thatstores the module information (pedestal, calibration, ...),the condition database and the online parameter database.

The APV chips (4 or 6 per module), APV Mux, PLLand DCU (in the following text referenced to generically as“devices”) each have several registers to configure. All registersare accessed over an Inter Integrated Circuit (I2C) bus with thestandard modes described in the specification [5] and a specificaddressing mode for the APV with a non standard extension ofthe I2C extended mode, the RAL mode.

Before starting data taking through the data acquisition sys-tem [6], the front end electronics must be configured properlyand each device register must be set through the I2C bus.

For the Silicon Strip Tracker, about 16,000 modules willbe produced and installed. Thus the potential problems ofthe system (both in hardware and software) come from itscomplexity and from its dimensions. In fact, the parameterset and therefore the number of registers to access is about1,680,000. Most of the device registers must be tuned in orderto find the best conditions for taking physics data.

In parallel to this electronic design, a control system wasdeveloped to set and retrieve all the device parameters. The

next sections describe the hardware and software architecture.Note that this hardware and software control architecture isalso used in other parts of the CMS experiment with the sameFEC/CCU rings but different front end electronic chips.

II. HARDWARE ARCHITECTURE

A. Hardware description

i2c channels

CCU

CCU CCU

CCU

CCU

CCUCCU

Token Ring

16 − i2c channels

FEC

1 − Memory channel

1 − JTAG channel

1 − Trigger channel

Hybrids / Devices

4 − PIO channel

CCU: Control and Communication UnitFEC: Front End Controller

Fig. 2. Hardware architecture. Note that one FEC can manage up to 8 tokenrings.

Figure 2 shows the hardware architecture of the controlsystem. This architecture is based on a specific board, the FrontEnd Controller (FEC) [7], shown in figure 3, which hosts atoken ring with modules, Communication and Control Units(CCU) [8], shown in figure 4. Each CCU manages several kindsof channels:

0-7803-8701-5/04/$20.00 (C) 2004 IEEE

- 16 I2C channels for the access to the front end chips. Eachchannel can be configured for different speeds (100, 200,400 kHz and 1 MHz).

- 4 Input/Output-like parallel bus controllers (Parallel inputoutput (PIO)) such as the ones used in the Motorola PIA,etc. These channels are used in the Tracker in order toreset the front end readout electronics;

- 1 memory channel, memory-like bus controller to accessdevices such as static memories, analog digital converters,etc.

- 1 JTAG1 master controller ;- 1 trigger distribution controller ;- 1 node controller (the CCU control itself is seen as a

special channel capable, for instance, to report the statusof the other CCU channels).

Fig. 3. Current prototype of the PMC Front End Controller Board for PCIsupport

The memory, JTAG and the trigger channels are not used inthe Tracker control system, but some other parts of the CMSexperiment use the memory channel and check the possibilityto use the trigger channel.

The CCU is built to be radiation resistant but SEU can causebit changes, so a Cyclic Redundancy Checksum (CRC) and anauto-correction algorithm are implemented in case of errors.For SEU and noise reasons, the connexion between the FECand the ring is optical with laserdrivers. In the Tracker, theCCU is completed with a DCU for the voltage and temperaturemeasurements which must also be read.

In order to prevent any problem due to the loss of a CCUthat would break the ring and consequently lead to lose allthe CCU on the ring, an additional set of connection, are used

1IEEE Standard 1149.1 (JTAG), IEEE Standard Test Access Port andBoundary-Scan Architecture

Fig. 4. Current prototype of the Control and Command Unit module with 6I2C channels and the two rings A/B for reliability

to implement a second, redundant ring. This redundant ringmakes it possible to bypass one CCU by configuring the inputand output of each CCU and FEC.

Once the ring is built, for each access, whatever the channelis, a frame is built with a FEC number, a CCU address, achannel, a transaction and a command number. The channelnumber specifies one of the CCU channels. The transactionnumber allows to recover the action which initiates the frame,for example, a read command. The command number gives thecommand to be executed (read, write, read modify write). Forthe I2C accesses, the frame must also contain an address andpossibly some data for write or read-modify-write operations.Note that the ring number is not given; it is configured via aspecific register into the FEC and CCU.

For research and development reasons, two versions of theFEC have been developed, one with an electrical and one withan optical connection to the ring. Both FEC are developed underthe Peripheral Component Interconnect (PCI) bus but the finalversion of the FEC will be on the VME bus with eight opticalrings. An intermediate version will be available in mid-2004.For the same reasons two versions of CCU exist (CCU6 andCCU25).

III. SOFTWARE ARCHITECTURE

The software architecture presented in figure III is built uponthe hardware architecture presented in the previous section. Thesoftware is split up into several parts:

- Run Control [9]: controls the data acquisition and controlsystem with the help of the Detector Control System.

- Information Message service (IMS) integrated in the runcontrol: dedicated task that receives all the messages(information, error, ...) belonging to the run control.

- FEC device driver: a Linux device driver for the FEC.- FEC supervisor (C++ API, the FecSoftware API): central

part of the control, it manages several FECs in order toread and write values into the front end electronics.

0-7803-8701-5/04/$20.00 (C) 2004 IEEE

Hybird / devicesFEC

Hybird / devicesFEC

Hybird / devicesFEC

Hybird / devicesFEC

Hybird / devicesFEC

Hybird / devicesFEC

Device driver access (software: open, read, write, ioctl, close)

Database access through Oracle Call C++ Interface (OCCI)

Finite state machine: SOAP message

Hardware access

subscribe

Database

Oracle

CMS General Run Control

OCCI C++ API

FEC Supervisor

C++ API

FEC Supervisor

C++ API

FEC Supervisor

Device Driver

Device Driver

Device Driver

Device Driver

Device Driver

Device Driver

DCU Analyser

i2c accessPCI bus

(Finite State Machine)SOAP

Diagnosis system

IMS

Fig. 5. FecSoftware Architecture

- Database: Oracle database used to store all the valuesneeded for the hardware. It manages all the parametersto set the hardware with the help of a versioning system.

- DCU analyzer: dedicated process to analyze all the valuesprovided by the DCU.

- diagnostic system: system that diagnoses the source oferrors and possibly takes a decision and signals it to therun control system.

The software control is built around the FEC device driver,database, FEC C++ API (FecSoftware API), DCU analyzer andthe diagnostic system.

The control software issue is divided in two actions:

- Download and upload parameters in the front end elec-tronics: these two operations require that all FEC / CCU/ channels (PIO and I2C channels) needed for the access,are enabled and configured. Once the control hardware isconfigured, the software sends the value needed to eachregister of the front end electronics (download). At theend of the download operation, the parameters set areread back from each device and compared to the valuesdownloaded. If a difference appears, the faulty deviceregister is stored in the database and an error is reported tothe run control through the IMS. A similar operation canbe undertaken directly between the event builder and theFEC control processes (FecSupervisor) to make calibrationor a delay curve. During this kind of operation, the eventbuilder asks the control system to set PLL or APV values.The download/upload system must be done in a relativelyshort time (about 2 minutes for the current measurements)

- Control the environmental parameters: this operation con-sists in reading all DCU on the hybrid and CCU and tosend all this information to the IMS. By a mechanism of

web subscription, the information is sent to the DCU ana-lyzer to check the correctness of the parameters (voltagesand temperatures). These readings are performed everythirty seconds but this rate can be changed dynamicallydepending on the needs of the system. This system mustdetect a failure in any system in terms of temperaturesand power. A second system exists but it is controlledthrough hardware locks to switch off all the Tracker toprevent, burning for example. Note that the DCU systemis important to avoid the switching off of the completesystem. If a problem occurs in temperature or humidity,the DCU analyzer must detect the divergence of the systemand must send a message or take the decision to switchoff the system correctly one part at a time.

Each part of the software control system will be describedwith its requirements, the mode of operation and performances.The components are: FEC device driver, Database, C++ Fec-Software API, DCU analyzer and finally the diagnostic system.

IV. CONCLUSION

A. FecSoftware summary

Throughout this paper, all the interfaces needed to makesystem accesses and configurations will be presented:

- FEC/Ring/CCU hardware: physical layer;- FEC device driver: low level software layer;- FecDevice class: interface to the device driver;- FecAccess class: concurrent access management;- DeviceAccess class and its sub-classes: specific device

access;- FecSupervisor and FecAccessManager classes: software

application.

0-7803-8701-5/04/$20.00 (C) 2004 IEEE

All the tools and the API are completely documented withthe Doxygen tool [11] and this documentation is available onthe web page [10]. It gives developers the ability to developand use the FEC software for new front end electronic chipaccesses and new CCU channels. So the tools developed (devicedriver, C++ FEC API) are used in some other parts of the CMSexperiment: the Preshower, the µ chambers and the Electro-magnetic Calorimeter for the time being. All software and toolsdescribed here are used in several Tracker production centers(Aachen, Karlsruhe, Louvain la Neuve, IPN Lyon, INFN Pisa,IReS Strasbourg, etc.) for the tests and the production of thedetectors (silicon sensors for the Tracker).

B. Performance

This system has been fully tested in beam tests (2002, 2003)and has proved to fulfill all the requirements needed by the finalexperiment in terms of performance and functionalities for theTracker.

The download performance is good, for the whole Trackerin the worst case (only one FEC supervisor and one databasemachine) the complete Tracker parameters settings are down-loaded in 205 s.

Concerning the upload performance, the time needed is notso good: about 2000 s is required (always in the worst case) dueessentially to the database upload time. But this time is onlyvalid if the complete Tracker front end electronics is in a badstate, in case of a power cut for example. In the best case (noerrors), the upload time will only cost about 5 s to retrieve theparameters from the hardware and do a comparison, knowingthat only the devices with errors in settings will be stored inthe database.

Note that these numbers come from the different curves thatwill be shown in the paper.

Note that the performances described in this paper are givenfor hardware currently used. Due to choices of the Trackerand CMS collaboration, the Tracker will use the VME busand personal computers for crate controllers with a PCI/VMElinks (see next paragraph on the evolution). We are obviouslyplanning to repeat the measurements for the final hardware.

C. Evolution

The next upgrade of the system will be of the hardware. Infact, the final format of the FEC board will be the VME andthe software must be changed in order to take care of this bus.

The CMS collaboration has chosen to use PCI/VME link,essentially for two reasons:

- A Personal Computer with standard PCI interface ischeaper than a VME-board computer and has a moremodern, powerfull processor

- All CMS online software is developed on Linux PC thattakes advantage of recent Linux kernel developments andall available Linux tools.

The tests will be performed on a PCI/VME link, but thefinal link has not been chosen yet. The goal of this upgrade

is to manage the FEC system through the same API, and theinterfaces must be the same when using PCI bus or VME bus.

Another upgrade is to develop the diagnostic system for theTracker control. The use of an expert system is justified bythe complex interconnections of the system. For example if thedetector’s data are not correct (detected by the data acquisitionsystem), the diagnostic system must ask the control system ifall the parameters are correct (power, values correctness in thefront end chips) and an expert system can take the decisionto recover the system by switching one of the FEC or CCUto the redundancy channel, resetting the FEC or simply tore-downloading the values. The current plans are to developthis system and see how far we can automatize the solution ofproblems, i.e. whether this will be just and aid to the expert onchift or an automatic recovery tool.

REFERENCES

[1] CMS Collaboration, “Technical Proposal”,CERN/LHCC 94-38, December 1994

[2] http://lhc-new-homepage.web.cern.ch/lhc-new-homepage/“The Large Hadron Collider (LHC) project of CERN”,

[3] CMS Collaboration, “The Tracker Projet - Technical Design Report”,CERN/LHCC 98-6, April 1998

[4] P. Placidi, A. Marchioro, P. Moreira, K. Kloukinas,“A 40 MHz Clock and Trigger Recovery Circuit for the CMS TrackerFabricated in a 0.25µm CMOS Technology and Using a Self calibrationTechnique”, presented at the 5th Workshop on Electronics for LHCExperiments, Snowmass, CO, USA, September 1999.

[5] Philips Semiconductors,“The I2C-Bus Specification”, Version 2.1, document order number: 939839340011, January 2000

[6] L. Mirabito et al.,“Tracker data acquisition for beamtest and integration”, Internal note,CMS IN 2003-021, May 2003

[7] A. Marchioro, C. LJustin, C. Paillard,“Optical FEC, Front End Control Unit for Embedded Slow Control”,Draft 0.16, not pulished, February 2003

[8] A. Marchioro, C. LJustin, C. Paillard,“CCU25, Communication and Control Unit for Embedded Slow Control”,Draft 2.0, not pulished, May 2002

[9] M. Bellato et al.“Run Control and Monitor System for the CMS Experiment”, Submittedto Computing in High-Energy and Nuclear Physics, La Jolla CA, USA ,March 24-28, 2003,

[10] F. Drouhin, L. Gross, D. VintacheThe FecSoftware documentation, http://cmsdoc.cern.ch/cms/cmt/System aspects/Fe

[11] Doxygen web site, http://www.doxygen.org/index.html

0-7803-8701-5/04/$20.00 (C) 2004 IEEE