Simons Observatory DAQ Overview

Brian Koopman Yale Universityfor the DAQ Team

CMB-S4 Meeting

October 17, 2019

Brian Koopman SO DAQ Overview October 17, 2019 1 / 15

SO DAQ Team

+ Test Users!!

Brian Koopman SO DAQ Overview October 17, 2019 2 / 15

Software Overview

Combination of custom written and open sources tools are being usedfor control, data collection, and live monitoring.

Designed to be very modular, allowing different components to actindependently.

SO Developed Software:

OCS/SOCS - The Observatory Control System/Simons OCSI A framework for coordinating operation of devices distributed across

the site

sisock - Backend for HK live monitoring

so3g - SO specific integrations with SPT3G software

Externally Developed (Custom) Software:

Lyrebird - Custom tool from SPT/POLARBEAR for live monitoringof detector timestreams. Forked and being adapted for SO.

SPT3G - Used for the data storage format.

Brian Koopman SO DAQ Overview October 17, 2019 3 / 15

Software Overview

Also using several open source software packages:

Docker - Delivers software w/dependencies

Docker Compose - Facilitates using Docker

Grafana - Live monitor web plotting

Developing publicly and open-source resources:

Docker Hub - Public distribution of Docker images

Travis CI - Automated build pipelines

readthedocs - Publicly hosted Sphinx docs

Brian Koopman SO DAQ Overview October 17, 2019 4 / 15

OCS: Observatory Control System

Coordinates control/readout ofall devices from multiple controlpoints

Requires three components:

1 Device specific code, i.e.I Lakeshore 372/240I SMuRFI HWP EncodersI Bluefors log reader

2 OCS Agent – interact withdevice code

3 OCS Client – interact with anyOCS Agent

OCS Architecture

Brian Koopman SO DAQ Overview October 17, 2019 5 / 15

crossbar.io

crossbar.io - Open-source connectivity toolkit for RPC + Pub/Sub

Brian Koopman SO DAQ Overview October 17, 2019 6 / 15

OCS: Observatory Control System

Figure Credit: Simone Aiola

Brian Koopman SO DAQ Overview October 17, 2019 7 / 15

OCS: Observatory Control System

Existing Agents:HK Aggregator - writes HK data to .g3 file

Registry - tracks running agents

Lakeshore 372 - 100 mK thermometry

Lakeshore 240 - > 1K thermometry

HWP Simulator - Example agent, simulates HWP encoder signal

Bluefors - tracks Bluefors generated logs

Cryomech CPA1114 - PT Compressor health

Keithley PSU - PSU used for load curves on cryogenics

PySMuRF Controller - Commands SMuRF crate

PySMuRF Monitor - Listens for UDP packets from controller and acts accordingly

PySMuRF Archiver - Copies misc files from pysmurf to archive

Timestream Aggregator - Writes G3Frames to disk from detectors

In Development Agents:CHWP Control - Control and readout encoders for HWP

ACU - Point the telescope

SMuRF PSU - Interface with SMuRF PSU

And more · · ·

Brian Koopman SO DAQ Overview October 17, 2019 8 / 15

OCS: SMuRF

OCS interface with SMuRFreadout system

Detector data collected,packaged, and dumped to diskin .g3 format

SMuRF HK data also retained

Docker images for SO builtupon the images made by SLAC

For more details, see Jack Lashner’stalk on Saturday!

SMuRF Software Stack

Brian Koopman SO DAQ Overview October 17, 2019 9 / 15

OCS: Client Example

Brian Koopman SO DAQ Overview October 17, 2019 10 / 15

so3g

Extensions to spt3g

Intervals - pairs of objects of thesame type

I Operations to supportintersection, union, andsubtraction

G3IndexedReader - Seek/Tell onfiles

General place for SO compiledcode

Brian Koopman SO DAQ Overview October 17, 2019 11 / 15

sisock: Simons Observatory Data over Websockets

Allows streaming data over theweb from various data servers

Data servers not necessarilycolocated

Useful for quick look at HK data

Tested deployment on NERSCSpinUp program

Currently mostly used forGrafana backend

kst support currently indevelopment

Brian Koopman SO DAQ Overview October 17, 2019 12 / 15

Grafana: HK Live Monitor

Web-based data viewinginterface

Allows flexibility in what theuser wants to plot

Supports many different datasources (i.e. Database, sisock,etc.)

Supports alerting on definedmetrics (for supported datasources)

Custom data source interfacessisock backend with Grafana

Anything in sisock can befetched in Grafana, i.e. live dataor data on disk

Example Grafana Dashboard

Brian Koopman SO DAQ Overview October 17, 2019 13 / 15

Lyrebird: Detector Live Monitor

Tool from SPT for live viewingfocal plane

Useful for:I First light sanity checksI FTS supportI Lab optical tests?

Forked by SO to add:I Generic input format

(non-dfmux)I Playback of archival data at

variable speedI Plugins for building simple

pipelines (filters, S/Nestimators)

Brian Koopman SO DAQ Overview October 17, 2019 14 / 15

Alarms/Alerting

Alarm system still indevelopment

Based on ACT system

redis/sqlite backend

Different states can triggerdifferent actions, i.e. email, sms,call

I Call/SMS supported via twilio

Distributed scripts forchecking/publishing state

ACT Alarm Page

Brian Koopman SO DAQ Overview October 17, 2019 15 / 15