Slide 1

1iTOP Electronics EffortLynn WoodPACIFIC NORTHWEST NATIONAL LABORATORYJULY 17, 2013TopicsBelle II DAQ SystemCOPPER/FINESSE

iTOP DAQ SystemFINESSE firmwareCOPPER software

Next StepsBASF2COPPER-IIIFINESSE redesign2Belle II DAQ SystemGoal: unified architectureSmooth transition from Belle ICommon readout hardware:COPPERCommon data link protocol:Belle2LinkCommon readout software framework: roobasf/basf2

Goal: scalabilityLuminosity at start of experiment will be several times lower than design luminosity, but trigger rate may be just as highNominal L1 trigger rate: 20 kHz, design average rate set as 30 kHz

3

Brief Belle II DAQ HistoryBelle I had dead-time issues with original LeCroy FastBus-based DAQ systemStarted in-house design in 2002

COPPER-I demo successful, took latency from 29us down to 3usReplaced all LeCroy systems in Belle I with COPPER-II (2005-08)

For Belle-II, expect to use a combination of COPPER-II and redesigned COPPER III4

COPPER-II5COPPER = COmmon Pipelined Platform for Electronics Readout9U VME boardFour detector front-end slotsRemote boot (no local filesystem)

COPPER-III redesign for Belle II will replace obsolete components, add new CPU and Gigabit Ethernet, fix bugs

COPPER-II CPUCPU is Linux-based embedded PCPCI Mezzanine Card = PMC

COPPER-II uses 800MHz Pentium-IIIIssues: 512MB, lack of support

COPPER-III will use 1.6 GHz Intel AtomNew features: VGA, USB, GbE, etc.Issues: power requirements6

COPPER-II FINESSE InterfaceInterface to specific detectors handled by custom daughtercardsFINESSE = Front-end INstrumentation Entity for Sub-detector Specific Electronics

Standard interface to COPPER FIFOs for data and local bus for control/status

(NOTE: most detectors settled on common HSLB (High Speed Logic Board) FINESSE design for Belle II)

7

COPPER-II Other featuresGlobal clock and trigger comein on TTRX boardSignals from FTSW (FrontendTiming Switch) boardDistributes deskewed clock and trigger to multiple destinations

FIFO control/status on dedicated COPPER FPGA

VME, secondary Ethernet available, but not commonly used

Second PMC slot for expansion (not pictured)

8

COPPER-II Data FlowData read into FINESSE

Processed then stored in 1MB FIFOs on local bus

DMA on COPPER CPU monitors FIFO status, reads out data when full/etc.

COPPER CPU sends datato host PC

9iTOP DAQ for Cosmic/Beam TestsPrevious testing (CERN, Fermilab, bench) used custom PCIe board from U. Hawaii for readout

DAQ goal for 2012-13 cosmic ray/beam tests: demonstrate COPPER-based readout

Requirements:Base hardware: COPPER-II, custom 9U VME crateiTOP-specific FINESSE (hardware, firmware)COPPER-based readout software for iTOP FINESSE

10Trigger sequence:TOF trigger via NIM logicTrigger passed to SCRODs via FTSWData readout from SCROD by FINESSE/COPPERTrigger data readout by USB daughtercard on COPPERTrigger clear from FINESSE back to NIM bin

iTOP DAQ for Cosmic/Beam Tests11USBiTOP barCOPPER CPU(PC1)CAMAC crateTOFNIMFTSWFiberTTLCOPPER server(PC2)USBCMD Tx/RxFINESSE BFINESSE AEthernetFIFOFIFOSCRODSCRODSCRODSCRODRemote boot/Data ReadoutTOF Data ReadoutTrigger ClearLocal busFINESSE Support for Cosmic/Beam TestsU. Hawaii has DSP_FINESSE designUp to 4 fiber links to SCRODsSpartan-6 FPGATwo dual-core BlackFin DSPs

Only basic example firmware and code no fiber readout, data handing, etc.Minimal DMA exampleNo existing COPPER drivers, only examples from other FINESSE boardsCOPPER runs old (2.4), customized version of Linux

Effort during 2011-12: wrote COPPER drivers to communicate with FINESSEAdded COPPER DMA support for FIFO readoutWrote FINESSE firmware to read data from SCRODs, send to COPPER FIFOsWrote demonstration code to loop data through DSP before writing to COPPER

12

COPPER-II Software for Cosmic/Beam TestsTo provide data-taking capabilities for cosmic and beam tests, overall framework was developed in 2012-13Python-based scriptsSpeed-critical portions (readout) in C librariesExperiment directory structure records all configuration settings, log files, and data for each run

Scripts generated for pedestal, pulser, laser, and beam runsUsed to debug electronics at KEK in 2013Used for data-taking at KEK and LEPS

Scripts updated during electronics bring-up and during LEPS data-taking to eliminate bugs, better match expected behavior and utilization

Multiple staff present at KEK/LEPS almost continuously in Jan-Jun 2013

13Next StepsConversion to BASF2 readoutBelle II readout settling on BASF2 framework on COPPER CPUPlan to use current DAQ software for next beam test(s), but will start porting on BASF2 readout

FINESSE redesignProcessing required for real-time analysis is not yet clearUnclear that current configuration will workSpartan-6 on SCROD packed to the gillsData transfer rate between FPGA and DSP on FINESSE limited to 75MHz

Transition to COPPER-IIICurrently developing with COPPER-II, but iTOP will use COPPER-IIICOPPER-III boards in short supply (all at KEK)Hardware should be largely transparent, but CPU will be running new version of Linux (2.6 kernel) possible non-trivial driver changes

14Future Processing RequirementsPossible changes:Replace FPGA on SCROD (likely to happen anyway)If we replace Spartan-6 with Virtex-7 enough to handle all processing?Could then use standard FINESSE design (HSLB)Replace DSP_FINESSE with new designSome redesign already required (obsolete components)

We have started investigating FINESSE redesignResearched different processing units: Vertex-7, Vertex-5 w. PowerPC, Zynq SoC (FPGA + dual-core ARM), quad-core ARM processorWrote draft design document based on ZynqKintex-7 FPGA (125-350K logic cells)0.7-1GHz dual-core ARM Cortex-A9256KB on-chip memory, 8 DMA controllersHigh-speed interconnect between CPU and FPGA

Starting on benchmarking effort with dev kit

15

Data HandlingCombine FPGA-based and CPU-based processingFPGA receives data andstores to memory via DMA, notifies CPUCPU manages data,passes commands to FPGA for dedicated processingFPGA writes data toCOPPER FIFO

CPU can make decisionsbased on data rates, quality metrics, etc.16

Looking into using existing Zynq module instead of full custom designSimplifies layout, memory, etc.Possibility of easy upgrade/downgrade later

Challenges:Only a few vendors with larger Zynq on modulesAdds peripherals unnecessary for our applicationClocking requirements may not match Belle II requirementsCost trade-offs: up-front design cost vs. higher per-unit cost

In unofficial discussion with EnclustraSODIMM form factor (68 x 30mm)Xilinx Zynq XC6Z030512MB SDRAM, 512MB flash

Potential Design Simplification17

SCROD

SCROD

Finesse

FPGA Scrod Interface

On-Chip Mem

DDR

FPGA Scrod Interface

DMAC

FPGA LB Interface

FPGA FIFO Interface

Copper

FPGA DSP

Fiber

Fiber

AXI Bus

AX Bus

PCIe

ARM9 CPU

Raw Event Data

Processed Event Data

Status/Control Data