the architecture of lux trigger control software november/06 2007

LUX Trigger Control Software. Wojtek Skulski University of Rochester, November 2007

The architecture of

LUX Trigger Control Software

November/06 2007

Wojtek Skulski

LUX Collaboration University of Rochester

skulski _at_ pas . rochester . edu


The goals of this presentation

• I will present the architecture and software components which I used

to develop several DAQ programs under BlackBox Component

Builder.

• The DAQ programs ran for thousands of hours in mission-critical

experiments without either crashing or leaking a single byte of

memory.

• The architecture can be used to develop the LUX Trigger Control

software running under BlackBox Component Builder.

• In order to make this presentation self-contained, I am also

explaining the hardware architecture of the LUX trigger boards.


• Development of BlackBox DAQ & control software: 2001-2007.

• Duties of the LUX trigger DAQ and control system.

• Data display and monitoring.

• Board control.

• Reconfiguration of FPGAs.

• Architecture of the LUX trigger system.

• Architecture of the BlackBox DAQ & control system.

• Parts of the BlackBox DAQ & control system:

• Object management POM.

• Waveform graphics Gr.

• Scientific Library Lib (graphics, matrices, special functions, digital filters, etc.)

• DAQ script DataLogger.

• Board control GUI panels : see my “without a sting” presentation

• Low-level hardware interface libraries: see my “without a sting” presentation.

Outline


NIM pulser

Analog signal reconstruction: digital FIR filter output

Control & waveform display:BlackBox and Gr

DDC-8

DDC-x development system under BlackBox


Development of BlackBox DAQ & control software: 2001-2007

• I have been developing and refining the BlackBox-based DAQ and control since

2001 with my DDC-x waveforms digitizers.

• Several students used and extended the DDC software.

• I use the same DAQ software package at LLE on a daily basis with the following

hardware: capacitance sensors (interfaced via the same USB controller as the DDC

cards), VME-based distance-measuring interferometers, National Instruments A/D

cards, sliding-scale sensors (interfaced via a serial port), humidity and temperature

sensors (interfaced via a serial port).

• All my experiments work smoothly. The BlackBox software never caused any

significant problems or delays in my experimental work.

• Software components described in this presentation were tested over thousands of

hours in mission-critical experiments, some involving hundreds of thousand $

worth of equipment.

• BlackBox is the best and the most robust development software I have ever used.


HW architecture of the trigger

The boards are described to give the context.

Block diagrams of the trigger board connections.

Block diagrams of the boards (the DDC and the EB).


Hardware architecture of the LUX trigger boards

• LUX trigger is composed of two kinds of boards.

• The front-end DDC-8 LUX is a waveform digitizer and signal processor.

• The back-end Trigger Event Builder is a communication board.

• Both boards use the same programmable chips:

• Xilinx FPGAs perform real-time signal processing.

• Cypress microcontroller manages the FPGA and provides the USB-2 data

link with the PC.

• PC is running BlackBox control software, which monitors and

controls the trigger boards.


Trigger consists of several DDC boards and one Event Builder board

Trigger system architecture

NIM pulses to DAQ system(event yes/no)

Up to 8 front-enddigital pulse processors

A single Level-2 decision board (Trigger Event Builder)

Fast data linksUp to 200 MB/s per link

Data readoutUSB 2.050 MB/s to the PC


Architecture of the front-end DDC boards

Xilinx

Cypress

The new DDC-LUX board will also have a fast data link, 200 MB/s


Architecture of the Event Builder boards

Xilinx FPGA

Cypress controller

The Trigger Event Builder is a new board designed for the LUX trigger

Eight fast data linksconnected to DDC

Up to 200 MB/s per link

USB 2.050 MB/s to the PC

Clock IN * 1NIM IN * 5NIM OUT * 4


Xilinx ISE/WebPack

Three development systems for LUX board programming

PC

Cypress EZ-USBdevelopment SW

DAQ and control SW Component Pascal

running under BlackBox Component Builder

Trigger boards

Xilinx FPGA

Cypresscontroller

FPGA configuration

(VHDL)

controllerprogram

(C)

Board controldatagrams

Cypress controlleris running board control

firmware

1

2

3


On-board firmware development for trigger boards

• FPGA development software.

• The FPGA is a massively parallel signal processor with 20 parallel multiply-

accumulate units. It implements real-time signal processing and decision logic.

• The original DDC-8 firmware was developed in 2004 and extensively tested. The

current LUX firmware is built upon this original firmware.

• Xilinx FPGAs type Spartan 3E-500 are programmed under Xilinx ISE WebPack.

• Currently we use high-level language VHDL to implement the firmware. Other

approaches (e.g., graphical System Generator) are possible, but not used yet.

• Microcontroller development software.

• Cypress controller manages the FPGA and provides the USB-2 data link with

the PC. Its processing power is relatively low (about 4 MIPS).

• The controller is programmed in C.


LUX Trigger Control

software architecture

Running on a PC under BlackBox

Data display and monitoring

Board control

Remote FPGA reconfiguration


Duties of the LUX Trigger Control software

• Data acquisition and writing to disk (optionally).

• Data display and monitoring. • One-dimensional waveforms (raw data from the ADC).

• Transformed waveforms (e.g., smoothed or filtered with some sort of “matched filter”).

• Two-dimensional correlations (e.g. time versus amplitude).

• Scalers (e.g., numbers of rejected and accepted events).

• Board control.• Thresholds, digital filter parameters, input DC offsets, etc.

• Several boards have to be controlled.

• Reconfiguration of the trigger system.

• The trigger operation is almost entirely defined by the FPGA. The FPGA has to be “loaded”

before the DDC boards can operate.

• The FPGA configuration has to be performed remotely, without unplugging the DDC boards.


Components of LUX Trigger Control Software

• Graphics.• Integer-valued 1D waveform graphics Gr (raw ADC data).

• Real-valued 1D vector graphics Lib (transformed data).

• Real-valued 2D matrix graphics Lib (correlations, e.g., amplitude vs time).

• Object management.• A package named Persistent Object Manager (POM) manages the above-

mentioned graphical objects.

• Numerical values such as scalers.• Numbers can be entered or shown in graphical panels.

•Data-acquisition script.• A “script” for performing DAQ is compiled and runs at full speed.

• USB-2 communication with trigger boards.• The component for USB communication is very well tested.


Waveform graphics GrShows raw ADC data.

Real-valued graphics LibMath and graphics library.Shows transformed data

and 2D matrices.

Persistent Object Manager POMManages graphical objects on screen.

Saves histograms and matrices to disk.

Simplified structure of DAQ software running under BlackBox.

USB communicationReceives data from the trigger boards.Sends control commands to the boards.

Data Loggeracquisition “script”

Compiled, runs at full speed.

Diagram of LUX Trigger Control Software

Management

Histogramming

Readout


Presentation of the software

components

which can be used in the

Trigger Software

Gr, POM, Lib, and the DAQ “script”


Waveform graphics Gr*

Easy to use histogram and waveform display package.

Designed for data acquisition (DAQ).

Extensively used for DAQ display:

DDC-1 and DDC-8 waveform digitizers, capacitance sensors,

temperature and humidity monitoring, CAMAC, and student projects.

*Developed by Wojtek Skulski.


Why Gr was needed?

Why needed: A typical DAQ program uses either histograms, waveforms, or time

“histories” to record integer-valued ADC data. I found BlackBox Component Builder a

tremendous tool to develop DAQ software. However, the waveform graphics was

missing when I started using BlackBox circa 1999. Therefore, I decided to develop the

histograming and display package for BlackBox. As a basis for development I used my

previous package developed for Oberon System-3, because the two languages (Oberon

and Component Pascal) are very similar to each other. I finished the development and

released Gr to public domain in September 2001.

To my own surprise, I have been using Gr in its almost original 2001 form ever since

its release. I made only very few additions to the original version. It turned out that the

design principle “make it as simple as possible, but not simpler” has yielded a software

package which is useful, robust, and maintainable. The Gr package became my “secret

weapon” which enabled me to develop robust DAQ software for my many projects.

Students, who also used Gr, have found it useful for their own projects as well.


When to use Gr

When to use Gr:

• Integer-valued data, e.g., acquired from a digitizer.• One-dimensional data. E.g. temperature recordings, etc.• The data is not an image (which would require 2D display).• The display has to be “live” and interactive.• Convenient zooming and scaling is implemented.

When to use Lib rather than Gr:

• Real-valued data (64-bit floating point numbers).• Two-dimensional display (e.g., time vs amplitude).• Lots of math, matrix algebra, or vector operations on data.• Special functions and/or digital filtering.


Gr has a reasonable GUI and is very easy to use.


Several DAQ examples

and a tutorial are provided

with the Gr package.

Example data acquisition panel.

Detailed instructions how to use the Gr package.

Gr User Guide


Persistent Object Manager POM*

It manages and organizes any objects.

Designed for organizing data acquisition display.

Extensively used in mission-critical experiments:

Tiled-grating assembly, capacitance sensors, temperature and humidity monitoring.

*Developed by Daniel Miner (with some W.S. help).


Why POM was needed?

Why needed: A typical DAQ program may use a large number of histograms,

waveforms, matrices, or other data objects. Typically, the histograms are initialized and

incremented under program control. Users want to display the content of these objects

while the DAQ program is running. (The “post-mortem” examination is inconvenient,

though often used.) This poses a problem of “dual ownership” of the data objects,

which are “owned” by the DAQ program, but need to be displayed by the GUI.

The Persistent Object Manager (POM) was developed to implement this feature. The

DAQ program will be running, processing data, and incrementing the histograms, and

yet you can display and save the histograms without stopping the DAQ. The histograms

are managed as a “tree” (similar to a Windows Explorer). You can double-click on any

histogram’s name, and you can examine its content. You can also save the entire tree.

Not just histograms: any object whatsoever can be managed by POM when the wrapper

for this object is available. We plan to write wrappers for several types of objects, such

as vectors, matrices, etc., as needed by the LUX Trigger project.


POM organizes Gr histograms into a tree structure.

We will implement management of other arbitrary objects, as needed.

Graphical object management of Gr histograms

POMwindows

HistosApplication-specific

panel


Another POM screenshot

Show another POM screenshot from the LLE experiment

Point out that POM was used for thousands of hours and never

crashed


POM “to do” list

Currently, POM can only manage Gr objects.

• The “Gr wrapper” is implemented.

Other objects (e.g., Lib objects) need to be managed as well.

• We need to develop the wrappers similar to the Gr wrapper, using the Gr wrapper

as the template for development.

• It is very straightforward to develop such wrappers.

• Any object can be managed by POM when the wrapper is available.

• The most immediate need is for Lib objects (real-valued vectors and matrices).

• A text object (implementing a formatted BlackBox text) will be a very attractive

addition. Such text objects can hold run-time comments and remarks, including

graphics. Using the text objects, the experimenter’s comments will never be lost.


DAQ DataLogger “script”*

It receives and processes experimental data.

Designed for organizing data acquisition process.

Extensively used in mission-critical experiments:

Tiled-grating assembly, capacitance sensors, temperature and humidity monitoring.

*Developed by Wojtek Skulski.


Why the “script” was needed?

Why needed: Writing a DAQ program may seem straightforward. Just write a piece of

code with whatever actions you want... However, the DAQ development will benefit

from having a template program divided into typical sections, such as initialization of

histograms, the DAQ “subroutine”, and possibly others.

The Data Logger “script” was developed to help writing typical DAQ programs. There

are sections to initialize histograms, a hook to perform the DAQ actions, and the

interface to the BlackBox multitasker, which makes the DAQ run.

And of course, the Data Logger uses POM to manage histograms (and other data

objects in the near future).

NB: The “script” is not a script. It is a compiled “module”, which means two things: (1)

It runs at full speed. (2) The compiler performs full consistency checking with the rest

of the DAQ system. I use the name “script” because functionally it is like a script, but

its implementation is vastly superior to any kind of a scripting language (such as cint,

I beg your pardon). The LUX Trigger software will be another example of this “script”.


Example experiment using the Data Logger + POM

Interferometer panel

(application-specific)

Example data: interferometer history recordings (popped up by clicking POM)

POM panel

This experiment was performed using POM and the Data Logger “script”

Data Logger panel

(start/stop, etc.)


Scientific and engineering library Lib*

1D and 2D display package.

Matrix and vector calculus, digital filters, and special functions.

Designed for data analysis, theoretical calculations, and modeling.

Extensively used for mission-critical applications.

Airborne antenna systems (BAE Systems), adaptive optics (UofR Laser Lab).

*Developed by Robert Campbell, BAE Systems.


Why Lib is needed?

Why needed: Writing a numerical program may seem straightforward... Until you

actually start doing it. Then you realize you need a library. Lib is a tremendous

computational library with over 6000 items. It covers vector and matrix calculus,

Fourier transforms, special functions, digital filtering, and many other areas.

In addition, Lib provides a very decent plotting utility which can render both 1D and

2D data. The latter is of course of interest to us (e.g., time-amplitude correlations).

Lib graphics and Gr graphics are complementary. I mostly use my own Gr graphics to

display raw data. However, when it comes to showing processed data, I often use Lib.

And of course, when I need to plot 2D data, I always use Lib because Gr only provides

1D display.

NB: You may ask why there are both Gr and Lib graphics? Part of this is history. The

two packages were developed independently. However, as I often use both, I appreciate

that Gr is great for displaying raw data, while Lib has a more “official look”.


When to use Lib libraries

When to use:• Real-valued data (64-bit floating point numbers).• Two-dimensional display (e.g., camera images, or time vs amplitude plots).• Lots of math, matrix algebra, or vector operations on data.

The list of subsystems:

Lib - Engineering & Scientific library, including 1D, 2D, and 3D plotting.

Algebra - Computational Algebra library.

Multi - Arbitrary ultra high precision arithmetic.

Filter - Digital and other filter design and analysis.

Nav - Navigation & Geodesic Coordinate transform modules and tools.

Wands - A collection of general purpose tools (Wand: a tool used by a magician).

Cpc and Ctls- several general-purpose tools and utilities.


Lib: math and plotting facilities

Vectors & Matrices, including 0-length cases.

Polynomial forms of vectors, matrix determinant, inverse, singular-value

decomposition, etc…

Complex numbers and complex functions.

Special Functions & curve fitting.

Random numbers: several r.n. generators.

Numerical methods.

Numerical integration, interpolation, root finding, and minimisation.

1D, 2D, and 3D plotting.

Includes interface to OpenGL.


Scientific plotting and computation library Lib


A mission-critical application using Lib

Adaptive optics control package, including hardware control.


Conclusion:

Tools to develop

LUX Trigger Control Software

The tools are ready: Gr, Lib, POM, Data Logger.

Other tools are available from BlackBox community (e.g. web server).

Several experiments were performed using these software tools.

The experiments ran successfully for thousands of hours.

Some development will be required (as is always the case).

the architecture of lux trigger control software november/06 2007

Documents

blackbox control software

blackbox software

lux trigger daq

blackbox daq control

lux trigger boards lux

ddc software

lux trigger system

blackboxbased daq