virgo data acquisition d. verkindt, lapp

6-10 Oct 2002 GREX 2002, Pisa D. Verkindt, LAPP

1

Virgo Data AcquisitionD. Verkindt, LAPP

• DAQ Purpose

• DAQ Architecture

• Data Acquisition examples

• Connection to DAQ and monitoring tools

• Data Streams

• Online analysis tools


2

DAQ purpose

DAQ requirements:

• collection of distributed data (timing system, optical links)

• flexibility in data flow (frame format)

• reliability (at least 1 month without crash)

• easyness of use and restart (DAQ graphical client)

DAQ requirements:

• collection of distributed data (timing system, optical links)

• flexibility in data flow (frame format)

• reliability (at least 1 month without crash)

• easyness of use and restart (DAQ graphical client)

Laser


3

DAQ purpose

Control BuildingCentral Building

Get data from various synchronized sources, sometimes 3 km away

North Building


4

Data Acquisition

DetectionEnvironment Controls

DAQ purpose

Collect distributed data from:• ITF environment• ITF controls• ITF output detection

Collect distributed data from:• ITF environment• ITF controls• ITF output detection

Env. monitoringEnv. monitoringSuspension control

Output MC BenchDetection Bench


5

DAQ architecture

Central data collectionCentral data collection

9 MB/s (compressed = 4MB/s)

local data collector

2.7 MB/s

frames

Input bench monitoring, Vacuum monitoring, Environment monitoring

3.3 MB/s


Suspensions dataLocking and alignment data

frames

Environment Monitoring

3.0 MB/s

frames


Photodiodes datadet. Bench monitoring

DetectionControls


6

DAQ architecture

FbF1 Gx FbF

PhotodiodesAlignement

FbS

Susp. CtrlGlobal Ctrl

6 Gx4 Fbf

Laser + env + towers+ tubes + calib.itf

FbF 3 Gx

local Main Frame Builder

Central Main Frame BuilderCentral Main Frame Builder

FbS

Det. Bench Ctrl

FbS

3.0 MB/s3.3 MB/s2.7 MB/s

9 MB/s (compressed = 4MB/s)

Environment Monitoring Controls Detection

frames framesframes

local Main Frame Builderlocal Main Frame Builder

DOLDOLDOL > 30 VME crates > 30 VME crates


7

DAQ architecture

More than 30 VME crates, but a reduced set of standard tools:

• Digital Optical Links (DOL) for controls• Fast Ethernet and Gbit Ethernet for central data collection

• VME crates for front-end data acquisition• Workstations for central data collection

• Standard format for data collection : frames encapsulated in Ethernet messages


8

Frame format

Frame = elementary time slice of dataFrame = elementary time slice of data

GW signal

channel 1

frame 1 frame 2 ...

Time

channel 2

channel 3

channel n

Contains:• GPS time stamp• ITF informations• raw data channels• processed data• events

Contains:• GPS time stamp• ITF informations• raw data channels• processed data• events

Common format of several gravitational waves detectors

Common format of several gravitational waves detectors


9

Timing system overview

GPSTiming

Laser

Data Acquisition

DetectionEnvironment Controls

Timing Distributor Crate User’s Timing CratesUser’s Timing Crates

ADC,DACCamera,DOL

Timing

CPU

Coax Cables

ADC,DACCamera,DOL

Timing

CPU

ADC,DACCamera,DOL

Timing

CPU

ADC,DACCamera,DOL

Timing

CPU

OPT/TTLRun

OPT/TTLFrame

OPT/TTLSampling

OPT/TTLFast Clock

TTL/OPTFrame

Sampling


ADC,DACCamera,DOL

Timing

CPU

Coax Cables

ADC,DACCamera,DOL

Timing

CPU

ADC,DACCamera,DOL

Timing

CPU

ADC,DACCamera,DOL

Timing

CPU

OPT/TTLRun

OPT/TTLFrame

OPT/TTLSampling

OPT/TTLFast Clock

TTL/OPTFrame

Sampling


ADC,DACCamera,DOL

Timing

CPU

Coax Cables

ADC,DACCamera,DOL

Timing

CPU

ADC,DACCamera,DOL

Timing

CPU

ADC,DACCamera,DOL

Timing

CPU

OPT/TTLRun

OPT/TTLFrame

OPT/TTLSampling

OPT/TTLFast Clock

TTL/OPTFrame

Sampling


ADC,DACCamera,DOL

Timing

CPU

Coax Cables

ADC,DACCamera,DOL

Timing

CPU

ADC,DACCamera,DOL

Timing

CPU

ADC,DACCamera,DOL

Timing

CPU

OPT/TTLRun

OPT/TTLFrame

OPT/TTLSampling

OPT/TTLFast Clock

TTL/OPTFrame

Sampling

Optical fibers

Timing Information (Cm)

all VME crates synchronized by Master clock• Fast Clock (2.5 Mhz)• Sampling (20 kHz)• Frame (1 Hz)

Monitoring & Control PartMonitoring & Control Part

Generator & Distributor PartGenerator & Distributor Part

GPS

CPU

TTL/OPTRun

TTL/OPTFrame

TTL/OPTFast Clock

Timing

TTL/OPTSampling

OPT/TTLSampling

OPT/TTLFrame

Build. Return Timing

Return GPS

GPS

Thanks to A. Masserot

Purpose• Synchronization (of controls)• Frame and sampling numbers• GPS time stamp for data exchange


10

SMS dataMain Frame Builder

frames

timing info

Slow Frame Builder

GPSTiming

timing info

Data acquisition examples

Slow Monitoring Stations

query

Sensor (temp. pressure…)


11

accelerometers, microphones, … Fast Frame Builder

BNC cables

GPSTiming

timing signals

Main Frame Builder

frames

Eth. 100 Mbps

timing info



12

Fast Frame Builder

Optical line

(DOL)

Main Frame Builderframes

Eth. 100 Mbps

timing info


Photodiode

Pre-ampli , demodulation&

filtering

Photodiode Readout

GPSTiming

timing signals

Optical lineInterferometer controls

(DOL)


13

Connection to DAQ

Slow Frame BuilderSlow Frame Builder

Main Frame Builder

Consumer 2Consumer 1

Fast Frame BuildersFast Frame Builders

Central Main Frame Builder

Central Main Frame Builder

Main data stream

Producer

DAQ world

Data Storage

Shared Memory

Online Processing Online Processing

Main Frame Builder:

Use shared memory and 2 processes

• Producer: merge input frames and put result in shared memory

• Consumer: read frames in shared memory and send them on network

Main Frame Builder:

Use shared memory and 2 processes

• Producer: merge input frames and put result in shared memory

• Consumer: read frames in shared memory and send them on network

dataDisplay dataDisplay

Monitoring worldrequested data request

Dynamical connection

• connect: send request with list of channels

• disconnect: automatic

• minimal perturbation on main stream.

Dynamical connection

• connect: send request with list of channels

• disconnect: automatic

• minimal perturbation on main stream.


14

Online Monitoring using dataDisplay


15

Offline use of dataDisplay


16

DAQ control and monitoring


17

Web DAQ Monitoring


18

DAQ Performances

nADC

nBytes

• Run almost continuously since Sept. 2001

• DAQ efficiency during last engineering runs > 99.8%

• Minimized latency --> DAQ can be used for online control


19

Current data streams

Raw data frames:

• most of channels sampled at 20 kHz or 10 kHz

• frame = 1 sec of raw data = 4MB (day=345 GB year=120 TB)

Raw data frames:

• most of channels sampled at 20 kHz or 10 kHz

• frame = 1 sec of raw data = 4MB (day=345 GB year=120 TB)

50Hz data frames: 3% of raw data storage

• provide fast access to raw data in low frequency band

• resampling at 50Hz (with filtering) all the fast data channels

• frame = 10 sec of resampled data = 1.1 MB (day=9 GB year=3300 GB)

50Hz data frames: 3% of raw data storage

• provide fast access to raw data in low frequency band

• resampling at 50Hz (with filtering) all the fast data channels

• frame = 10 sec of resampled data = 1.1 MB (day=9 GB year=3300 GB)

Trend data frames: 0.1% of raw data storage

• provide fast access to long (weeks) stretch of data

• trend data = min, max, mean, rms computed for each fast sampled channel, over one frame

• frame = 30mn of trend data = 9.6 MB (day=460 MB year=170 GB)

Trend data frames: 0.1% of raw data storage

• provide fast access to long (weeks) stretch of data

• trend data = min, max, mean, rms computed for each fast sampled channel, over one frame

• frame = 30mn of trend data = 9.6 MB (day=460 MB year=170 GB)


20

Trend data acquisition

Trend Frames Disks

Trend Frame Builder

Full FrameStorage (disks)

Main Frame Builder

ControlsFrame Builder

DetectionFrame Builder

Env. MoniFrame Builder

Vega DB

(Root)

Web


21

Online Monitoring using trend data

Example 1 : output of ITF over 8 hours, during engineering run E4 (min, max, mean)

Use of Vega tool

and Web browser


22

Offline use of trend dataExample 2 : max of output of ITF, building temp. and seismic motion near north tower over 3 days


23

50Hz data acquisition

Trend Frames Disks

Trend Frame Builder


Main Frame Builder

ControlsFrame Builder

DetectionFrame Builder

Env. MoniFrame Builder

50Hz Frames Disks

50 HzFrame Builder

50Hz processing

50Hz processing

50Hz processing

Vega DB

(Root)

Web


24

Online monitoring using 50 Hz dataExample 1 : monitoring of seismic activity over 8 hours, in 3 frequency bands


25

Offline use of 50 Hz data

Example 2 : spectral density of output of ITF over 3 hours of data (made in 30 sec)


26

Online analysis tools

GAI (General Algorithm Interface):

• A software tool to interface algorithms to online processing stream of data.

• Used to run online algorithms during engineering runs

• Used also offline to analyse engineering runs data

• Improved thanks to requests and comments from users and algorithm developers

Some of the algorithms developed up to now with GAI for online and offline analysis:

• Algorithm 1 : monitoring of spectral lines in ITF output channels

• Algorithm 2 : search of glitches in ITF output channels

• Algorithm 3 : monitoring of the stationarity and gaussianity of the ITF output.


27


Gai library

Disk Shared Memory

Ethernet

GAI processAlgorithm

Disk Shared Memory

Ethernet

frames

frames


28


Disk

Shared Mem

Ethernet

Algorithm2

Disk

Shared Mem

Ethernet Algorithm5

Algorithm4

Algorithm3

Algorithm1

: data under frame format


29

Online Analysis: current scheme


Main Frame BuilderOnline ProcessingFrame Distributor

Algo1 datastorage

Algorithm 2 Algorithm 3 Algorithm 1

Algo2 datastorage

Algo3 datastorage

frames

raw data frames


30

Online Analysis: futur scheme


Main Frame Builder Trigger manager

Algorithm 2

Algorithm 3

Algorithm 1 frames

Processed datastorage

frames

raw data frames


31

Conclusion

Virgo DAQ and online monitoring tools like dataDisplay or Vega+Web have been extensively used since year 2001.

DAQ has shown to be:

• modular (lego pieces with standard connections between them)

• reliable and quite easy to use (and to restart)

• flexible and evolutive

• latency minimized

Beyond DAQ:

• Useful data streams (raw data, trend data, 50Hz data, processed data, …) are under definition

• Online analysis has started

virgo data acquisition d. verkindt, lapp

Documents

data exchangegrex

lappdaq purposeget data

data eventscommon format

lappcurrent data streamsgrex

virgo data acquisitiond

lappdaq architecturemore

lappdaq control

lappdaq performances