Fusion Engineering and Design 60 (2002) 285–289

128 Channel PCI-based data acquisition system forMDSplus

Xavier Llobet *, Basil P. DuvalCRPP-EPFL, CH-1015 Lausanne, Switzerland

Abstract

With the increasing demand for analogue channel acquisition on the TCV tokamak, a new PCI based acquisitionhas been specified, designed, built and installed into our MDSplus acquisition environment. The design criteria wereto not only improve the cost/channel, as compared to our conventional hub based acquisition (CAMAC), but toprovide some distributed processing power to avoid the associated acquisition server saturation, both in terms of CPUand network bandwidth. These units were initially intended to satisfy the requirements of general variable rateacquisition from a variety of sources, and many channel acquisition from modern multi-channel diagnostics. Hostedby a i386-Linux PC in a crate with four available PCI slots, each single-PCI slot 32-channel digitiser features samplingfrequencies up to 200 kHz, and 64 MB of memory, providing 1 Msample of 16-bit data per channel. The local harddisk is used for immediate local storage of all the acquired data from the selected channels into a local MDSplusdatabase. The host is then accessed as a MDS/IP server that provides, on demand, down-sampled and softwarefiltered traces. The local hard disk capacity is used for medium to long-term storage and availability of the full dataset with optional mirror technology to guard against hard disk failure. We have thus obtained a general solution forhigh resolution, multi-channel routine acquisition using the multi-platform MDSplus environment, in which differentsoftware and hardware architectures are intelligently linked across a standard TCP/IP network. The implementationpresented here uses ONLY standard components of the MDSplus environment. © 2002 Elsevier Science B.V. Allrights reserved.

Keywords: Data acquisition; MDSplus; Linux; PCI-bus; Multi-channel

www.elsevier.com/locate/fusengdes

1. Introduction

As in all experimental installations, the de-mands on the acquisition system, both in terms ofprecision and data volume, keep increasing, and

TCV is no exception. With the typical 2–4 s pulsetime of the TCV machine, a typical requirementfor multi-channel diagnostics is to be able todigitise at up to 100 kHz, for the full duration ofthe shot. This paper presents our approach toimplementing these requirements using, as an ex-ample, the recent instrumentation of the TCVpower supply rectifiers: 16 channels for each ofthe 19 rectifiers, with a requested bandwidth of�100 kHz.

* Corresponding author. Tel.: +41-21-693-6005; fax: +41-21-693-5176.

E-mail addresses: xavier.llobet@epfl.ch (X. Llobet),basil.duval@epfl.ch (B.P. Duval).

0920-3796/02/$ - see front matter © 2002 Elsevier Science B.V. All rights reserved.

PII: S0 920 -3796 (02 )00022 -4

X. Llobet, B.P. Du�al / Fusion Engineering and Design 60 (2002) 285–289286

As described in Duval et al. [1], highly modularsystems like CAMAC are expensive and requireheavy installation costs for single systems thatmay require hundreds of acquired channels. In thedesign of a new acquisition system, a high channeldensity, high ADC precision (greater than 14 bit),simple calibration and low cost were consideredthe most important optimisation criteria. A firstattempt at such a system was implemented on aSunOS CPU module in a VME rack, with polledADCs transferring data to computer memory,built by Pentland [2]. Although a high channelcount was obtained, the available CPU power andmemory made all but the simplest remote calcula-tion arduous. The advent of cheap powerful PCbased computers able to run reliable low costoperating systems (Linux) with high speed I/Obuses (PCI) encouraged us to develop a newsystem based on cheaper technology using TCP/IP for all control and data transfer.

In view of the limited acquisition time, and toavoid messy real-time operating system problems,these cards were to initially acquire to local on-board memory, which could then be accessedafter the acquisition sequence where slight trans-fer delays would be of no consequence.

The ACQ32 boards from D-TACQ [3] wereused in the first implementation of this solution,the DT100, and feature 32 channels of 16bit-200kHz acquisition with 64 MB (128 MB nowavailable) of onboard memory, permitting 1Msample of acquisition per channel on a singlePCI card. An internal bus permits up to four suchboards to share timing signals in a single PC, andeach board has a full complement of offset correc-tion and gain control logic so that calibration canbe performed in-situ. Electrically, space con-straints limit the card to unipolar +/−10 Vinputs and although different input gains areavailable, differential inputs and/or signal filteringmust be implemented on an external board, ifrequired.

From our experience with our previous acquisi-tion systems, the DT100 was designed as a three-state machine (STOP, ARM, RUN) with theinternal memory configured as a ring buffer inwhich the number of pre-trig and post-trig sam-ples can be programmed. Control and data trans-

fer is achieved by a Linux-hosted device-driverwith DMA capabilities. Commands sent to thisdriver control the acquisition settings and ma-chine state, and data and current status can beread from the appropriate device.

The acquisition boards are installed into a 19’’rack-mounting industrial PC (500 MHz AMDprocessor, with �128 MB of RAM) runningLinux and networked via a 10/100 Base-T card.Module clocking may be provided internally orsynced to an external source as at TCV. Even thisrelatively modest CPU has significant processingpower that may be used to analyse data beforetransfer to the central data acquisition databases.

In contrast to our present system, it was re-alised that a complete data set was not alwaysrequired in the main TCV databases, but thatlower re-sampled or filtered traces would be ac-ceptable if the complete raw data set could bekept available for a finite time.

Since TCV originally chose to use MDSplus onVMS, which has been very successful, it wasnatural to use the Unix-port of MDSplus on ourLinux acquisition boxes. We thus chose the MD-Splus road: to implement a local MDSplus data-base and data server to manage the data on theacquisition machine and provide a MDS/IPclient/server communication path to our centralcomputing facilities for control and data-transfer.These acquisition systems thus are now capable ofacquiring data from a local or remote request,store the data locally in high performance data-bases, and serve these data to local and remoteclients in raw or processed form.

This rather ambitious result is now not only acomplete acquisition system in its own right whichcan locally run high level analysis language pro-grams, like Matlab and IDL, but can then beseamlessly linked to our mainstream and legacyacquisition system using a single set of multi-plat-form software tools.

2. A short overview of the MDSplus world

MDSplus [4,5] is a data acquisition softwaresuite, developed in the MIT, the IGI and LANL,and based on a hierarchical structure (tree) whose

X. Llobet, B.P. Du�al / Fusion Engineering and Design 60 (2002) 285–289 287

nodes describe all the hardware and softwareparameters, and also hold the acquired data. Thedescription is written into a ‘‘model’’ tree, and forevery acquisition cycle (shot), a copy of the modelis made with an identifying shot number and putin a specific directory (which belongs to a tree-specific path). At the start of the shot cycle, amaster process (dispatcher) scans the tree, collectsthe desired software actions (if enabled), andlaunches them in pre-determined circumstances(timing or other environmental conditions). Theseactions include the initialisation of the acquisitionmodules with the requested parameters for theshot: sampling frequencies and durations, opera-tion modes, active channels, etc. Following theacquisition cycle, ‘‘store actions’’ transfer the datafrom the devices to the database disk files, writingthem in the specified tree nodes. These storedvalues range from simple numbers to full-fledged‘‘signals’’ (containing further information such astime-base, units, etc.), multidimensional arrayswith units and associated vectors of indices (forexample, the time of each sample). Users maythen open the tree files and read the values ofthese nodes by providing a shot-number/node-name pair.

Tree Data Interface (TDI) is the hidden trea-sure in MDSplus [5]. All the above actions aredescribed in the TDI language, which is parsedand executed in a similar way to high level Mat-lab/IDL languages.

Originally, this parser could evaluate simplefunctions in the tree nodes such as:

MY–DEVICE:GAIN * MY–DEVICE:CHANNEL–1

to return the product of the values held in thosetwo nodes. Equipped with good mathematical andcontrol functions, libraries of routines—whichcould be stored outside the trees—can be devel-oped to manipulate vector and matricial data,with special functionality provided to manipulateMDSplus traces. These C-like syntax TDI subrou-tines are scriptable and interpreted and are thusrelatively easy to maintain.

Finally, TDI has the ability to dynamically callshareable library routines, by value, reference, ordescriptor, allowing the access to system and userlibraries. (The descriptor data form permits data-

overloading in which data stored in a particularform—short, integer, floating, double precisionetc.—can be mixed without any specific conver-sion by the user).

The last relevant aspect is MDS/IP (MDS overIP). This communication layer provides server/client features [6] such that a remote (or local)client can request the evaluation of a functionfrom the server and obtain the result. The rela-tively simple client requests data in its own binarydata format which is converted to the server’slocal binary format at the server which thenreplies in the client’s binary format. The result isthat when a given function is implemented at theserver, any client, from any machine, can requestthe evaluation of this function from the serverand obtain the reply in its own binary format.This service provides enormous flexibility: a re-mote user has exactly the same possibilities as alocal user and, for example, a calculation thatrequires many resources can be transparently re-quested from a mainframe, or any client hasaccess to the hardware implemented on anyserver.

3. DT100 TCV implementation

A set of TDI functions was implemented on theacquisition machine which set and control theinstalled acquisition boards through the devicedriver which, in standard Unix fashion, behaveslike a file: a command is written to the driver andthe result is obtained by reading from it.

This is used to initialise and activate/deactivatethe DT100 channels: only the onboard memorylimits the number of samples, so having, for ex-ample, 16 active channels instead of 32 allows 2Msamples per channel. In the same way we set thenumber of acquisition samples, ARM the boardand wait for the external clock (which also func-tions as the trigger). At TCV the DT100s are usedin external clocking mode where the clocks areprovided by the TCV timing system, which ismaintained synchronous to 1 �s across the wholeplant.

X. Llobet, B.P. Du�al / Fusion Engineering and Design 60 (2002) 285–289288

On the DT100, a simple local tree is used tostore the data: four nodes, one per possible board,each with 32 numeric sub-nodes, corresponding tothe channels, such that the data are stored withnames as .BOARD–001:CHANNEL–032 .

A TDI function is used to initialise the boardsand, after ARMing them, spawns a process thatpolls the boards every 30 s to see if they are in theSTOP state. Once an acquisition sequence is com-pleted, this MDS/IP client task requests the cur-rent shot number from the TCV plant, creates alocal tree with that shot number, then reads thedata from the boards and stores them in the tree,as signed long integers. If the TCV shot is abortedand the modules are re-initialised without havingbeing triggered, the poll process is killed beforethe next sequence is initiated.

In the main VMS-hosted TCV cluster, a devicemodule in the TCV MDS tree describes each32-channel board. Although each board can beindependent, we have chosen to chain them in amaster/slaves configuration. Thus, in the tree,only the parameters of the first board need to beset, although a separate channel activation tablein maintained for each board.

During an acquisition cycle from the TCV com-puters, the TCV side works as client and theDT100 as server, so the INIT and STORE actionsoccur under the control of the main TCV se-quence. The INIT action determines the TCP/IPname of the remote DT100 device (kept in alogical table for flexibility), opens a MDS/IP con-nection to this machine, and sends the instruc-tions to initialise the boards with the requiredparameters and to arm them. In general, all chan-nels of the activated boards are run at 200 kHzfor simplicity.

It is the STORE action that shows the flexibil-ity of our approach. If there is no need for thecomplete data set, with say 16 GB of local diskavailable and a typical 3× compression factor,�200 full length acquisitions can be locallystored. Thus, the complete data set is available fora relatively long time. If a particular trace isrequired shortly after the acquisition sequence,the standard STORE action can open a connec-tion to the DT100 machine, check acquisition iscomplete, decimate the data by a factor of 10, and

write the result into the TCV tree, with the appro-priate timing information. Later, a user may de-cide that a more detailed acquisition is required,and opens a new connection to the server request-ing the appropriate data, that may even be nu-merically processed from the complete raw trace(e.g. numerical filtering using all the acquired datapoints). Provision is made in the main TCV treesto store these data in a separate node. In theexample of TCV power-supply monitoring, if ananomaly appears in the power supply perfor-mance, the engineers may return and study thefull trace around the time of the anomaly, andsearch preceding shots to obtain evaluativeinformation.

Interestingly enough, since the DT100 machinesare full MDSplus servers, all the provided visuali-sation tools can request data directly from theDT100 server without storing unnecessary infor-mation in the main TCV databases. This opensthe scope to acquire large amounts of informationabout complex systems without having to returnlarge quantities of data to the main databaseswhere their storage and backup will impose in-evitable waste. This information could be ob-tained directly from the acquisition machine fordisplay in the ‘‘scope’’ or even routed transpar-ently through the main server machine for a re-mote user.

These local trees are kept until the disk space isneeded. The polling task ensures that at least 10%of the disk is available for the next acquisitioncycle and erases older data according to a user-defined algorithm. Fiducial shots may be explic-itly moved to another directory that is nevererased where they still remain available to theMDS/IP server. A secondary disk can be usedeither as a mirror of the primary, for disasterrecovery, or as a disk extension, providing anincreased ‘‘data holding’’ period.

4. Conclusion

A multi-channel, high performance and lowcost acquisition system has been designed andinstalled at TCV. Using only ‘‘standard’’ MD-Splus tools, these systems were configured as both

X. Llobet, B.P. Du�al / Fusion Engineering and Design 60 (2002) 285–289 289

‘‘stand-alone’’ acquisition systems for separate ex-periments or system development, and as an inte-gral component of the TCV computerenvironment and acquisition systems. The systemcan be remotely piloted from the main TCV ac-quisition system such that traces can be requestedand the returned data will appear in the standardTCV acquisition trees like any other signals.

Local storage of the complete data set on theacquisition machine, accessible with the sametools used for TCV data tree analysis, allows areduced data set to be saved in the main TCVdatabases whilst maintaining a purely disk-sizelimited buffer of the complete data set, whichwould otherwise not be available.

By using a platform based on widely availablehigh performance hardware combined with anopen-source high performance operating system,and a symmetric acquisition software suite, com-plete acquisition-analysis ‘‘bricks’’ can be con-structed which may be then used to build bothsimple and highly complex acquisition systems.

References

[1] B.P. Duval, X. Llobet, P.F. Isoz, J.B. Lister, B. Marletaz,Ph. Marmillod, J.-M. Moret, Evolution not Revolution inthe TCV Tokamak Control and Acquisition System, in:Proceedings of the 21st Symposium on Fusion Technology,Madrid, 11–15 September 2000, in press.

[2] Pentland Systems, 1b Young Square, Brucefield IndustrialPark, Livingston, Lothian EH54 9BX, Scotland. Website:www.pentlandsys.com

[3] D-TACQ Solutions Limited, James Watt Building, Scot-tish Enterprise Technology Park, East Kilbride, G75 0QD,Scotland. Website: www.d-tacq.com

[4] G. Flor, G. Manduchi, T.W. Fredian, J.A. Stillerman,K.A. Klare, MDS-Plus-A comprehensive data acquisitionand analysis system, Proceedings of the 16th Sym-posium on Fusion Technology, London, 1990, pp. 1272–1276.

[5] J.A. Stillerman, T.W. Fredian, K.A. Klare, G. Manduchi,MDSplus data acquisition system, Review of ScientificInstruments 68 (1) (1997) 939–942.

[6] J. Stillerman, T. Fredian, M. Greenwald, Uniform DataAccess for Remote Participation, in: Proceedings of theWorkshop on Remote Participation in Fusion Experi-ments, Satellite Meeting to the 25th EPS 1998 ICPP,Prague, pp. 22–25.