Belle DAQ SystemBelle DAQ System

Takeo HiguchiIPNS, KEK

Belle DAQ group

2009/01/07 SLACAdvanced Instrumentation Seminars

Introduction to the Belle DAQ

Belle DAQ Overviewtrigger

decision Main coverage by “DAQ”decision

systemclock

triggerclock F/O

g y• Data flow: signal digitizer storage• Timing: trigger timing distribution

detectorfrontend TDCTDCTDC Readout

PC

detectorfrontend TDCTDCTDC Readout

PCEventbuilder

detectorfrontend

… TDCTDC

… …

TDC ReadoutPC Event

builderOnline

Analysis PC

detectorf t d

…

TDC

… …TDC Readout

PCEventb ildfrontend TDCTDC PC builder

Belle DAQ Diagramg

Coverage list R t lE t b ildiCoverage list• Signal digitization• Digitized data readout

• Run control• HV control• Data quality monitor

• Event building• Online data analysis• Timing distribution

Belle DAQ Virtual Tour – 1• Signal digitization and data transmissiong g

Signal digitizer (TDC)Signal digitizer (TDC) PMC-sized CPU

signal digital

via LAN

Belle DAQ Virtual Tour – 2• Trigger timing redistributiongg g

signal digitaltim

ing

Trigger Timing Redistributortiming gg gg

Belle DAQ Virtual Tour – 3• COPPER

• TDC module × 2• Trigger timing redistributor

VME9U-sized Platform“COPPER”• Trigger timing redistributor

• PMC-sized CPU“COPPER”

Detail of the COPPER is described later.

Belle DAQ Virtual Tour – 4• COPPER crate

+ =

After cablingof input signalsof input signals

Belle DAQ Virtual Tour – 5• Crate readout

Digitized data readoutby the PMC-sized CPU

are sent to crate readoutare sent to crate readoutPCs via a network switch.

Belle DAQ Virtual Tour – 6• Event building PCs / Online analysis PCsg y

Fragmented data fromeach crate readout PC

are sent to event buildingPCs to be combined toa single event record.

Finally, the data aresent to online analysis

PCs and recordedto hard disks.

Belle DAQ Virtual Tour – 7• Trigger timing distribution – again tim

ing

gg g g

• Trigger timing distributionS t l k di t ib ti• System clock distribution

• Busy collection

min

g

Fanout

tim

timing

crate rear

Q-to-T System

Drift chamber readout

y

Trigger latency = 2 1 µs

Drift charge tothe sense wire t1

Trigger latency 2.1 µs

Qevent L1-trigger

the sense wire

T(Q) t2

0 tDRIFT t

Q-to-T conv.

tSTOP t

• t1 = Drift time Q-to-T enables ADC+TDC

Q to T conv.

• t1 – t2 = Drift chargeQ to T enables ADC+TDCsimultaneously.

Belle readout is based on Q-to-T + TDC

Introduction to the COPPER

COPPER Systemy• We have developed several excellent DAQ p

technologies by ourselves.

• Among them, today, we emphasize the “COPPER System” for its …y

– High flexibility to fit your experiment,– Wide acceptance of L1 rate up to 30kHz,p p ,– Broad bandwidth of > 80MB/s,– Less DAQ deadtime with equipped pipeline, andQ q pp p p ,– Less requirement of knowledge in writing readout software.

Why We Developed COPPER?y p• KEKB / Belle upgrade planpg p

– KEKB luminosity increases: 1.7×1034 8×1035 cm-2s-1

more L1 rate: 500 Hz 10-30 kHz– Belle upgrades

more readout channels: 40 kB/ev 200-300 kB/ev

• Limit of the present Belle DAQ– Deadtime fraction of the present FASTBUS-based DAQ

will be extrapolated to ~20% even @ L1=1kHz.

Call for new DAQ accommodates with L1=30kHz.

COPPERFINESSE×4Add on modules to the

Online CPUAdd on module to the COPPERAdd-on modules to the

COPPER. Responsibleto digitize the input signals

d t t th t i li

Add-on module to the COPPER.Responsible to readout the COPPERFIFO, to process and format the data,

dt d th d t t t l PC iand output them to pipelineFIFOs on the COPPER.

RadiS s EPC 6315

andto send the data to external PC viathe network.

RadiSys EPC-6315–Intel P3 800 MHz–256 MB memory–Network boot–RedHat Linux 9

100BaseT port×2Trigger timing moduleAdd on module to the COPPER Responsiblep

Data transmission lineand control line.

Add-on module to the COPPER. Responsibleto receive trigger timing and system clock fromupstream and to deliver them to the FINESSEs.

COPPER Block Diagramg

s

add-onmodules PMC modules

COPPERFIFO

cal b

us

CI b

us

tect

or FINESSE

al P

CPMC CPU

Loc PC

ub-d

et FINESSE

xter

na

dge Network

I/F

from

su

FINESSE to e

x

Bri I/F

Timingfr

FINESSETiming Module

FINESSE

SuperBelle: composite of different kinds of sub detectorsSuperBelle: composite of different kinds of sub detectors ecto

r

SuperBelle: composite of different kinds of sub-detectorsSuperBelle: composite of different kinds of sub-detectors

Detector ARequires ADC

Detector ARequires ADC

Detector BRequires TDC

Detector BRequires TDC

Detector CRequires buffer

Detector CRequires buffer de

sign

the

dete

Requires ADCRequires ADC Requires TDCRequires TDC Requires bufferRequires buffer

Flex

ible

fit

ting

to

ADC FINESSEADC FINESSE TDC FINESSETDC FINESSE Buffer FINESSEBuffer FINESSE

COPPER COPPER COPPER

F f

Predefinedprotocol

eado

utN

ES

SE

mm

on re

er th

e FI

N

The COPPER/FINESSE system enables one to

Com

afteconcentrate on the digitizer part without worrying

about implementation of the readout part.

FINESSE Catalogue 2008g• Several functions of

FINESSEs are readyfor hitchhikers. 65 MHz 8ch Digital Buffer65 MHz 8ch

FADC FINESSEDigital Buffer

FINESSE

TDC FINESSE 500 MH 2 h USB I/F FINESSETDC FINESSEw/ AMT3 TDC chip

500 MHz 2chFADC FINESSE

USB I/F FINESSE(Debug use)

TDC FINESSEw/ HPTDC chip

Pixel ReadoutFINESSE

Dry runFINESSE

TDC FINESSEw/ Vertrx5 FPGA

Inside the COPPER – FINESSE• Typical FINESSE designyp g

FINESSEfrom detector

186x76 mm2

FIFO fullFINESSEfrom detector

to COPPERFIFO

digitizer

Analog / digitalsignal

L1 pipeline

tizer

Localb

UntriggeredS li

bus

garbage

L1 trigger

Samplingclock

L1 trigger

Busy responseSystem clock

42.3 MHz

Inside the COPPER – FINESSE I/F• Data Transfer from FINESSE to COPPER

– Data are fetched at ‘FWCLK’ timing during ‘FWEN’==L- ‘FWCLK’ and ‘FWEN’ are operated by the FINESSE to the

COPPER FIFO.

‘FWEN’ also indicates a single event sequence– FWEN also indicates a single event sequence.- Event-end marker is needed to build up the 4-FIFO data to an

single event record. g- # of ‘FWCLK’ during ‘FWEN’==L is also counted by the COPPER

FPGA to determine the DMA start timing from the FIFO to CPU. ‘No word’ event is also OKNo-word event is also OK.

FWEN

FWCLKFWCLK

FF00-FF31Firmware limit#-of words / FIFO / ev < 256kB

Inside the COPPER – Local Bus

L l1MB DMALocalbus

PCI busPCI-local bridge

1MB DMA

addr

z

PLX-9054

DMA controller32

• Cascaded FIFO

8 M

B a

33 M

H

FIFO ctrl

C l

• Cascaded FIFO

ss

JP

CycloneEP1C12Q240C6

Address decoder

IDT72V36110512k

bypa

s

Address decoderFIFO empty/full monitor

4-FIFO event builderData formatter IDT72V36110

FINESSEData formatter

DMA start interrupterIDT72V36110

512k

Inside the COPPER – PCI Bus

PCI VME b idPCI-VME bridgeSpecial DPM

4kB dual port CPU boot,100Base-T

CPU Eth tp

memory Run controlCPU EthernetIntel 82557

A32

D32

33 M

Hz

CPUBridge

PCI-PCI BridgeTI PCI-2050EPC 6315

EthernetIntel 82559EPC-6315

100Base-TPCI busDMA

PCI-local bridgePLX-9054

PCI-local bridgePLX-9054Local L1 trigger

TT-RX

Localbus DMA controller

PMC: PCI Mezzanine Card Standard

100% Compliant w/ PCI RadiSys EPC 6315• 100% Compliant w/ PCI

• Good for a high density

• RadiSys EPC-6315– PMC sized CPU module.– Equipped with Intel PentiumIII• Good for a high density

applications– Equipped with Intel PentiumIII

800 MHz w/ 512 MB memory.– RedHat Linux 7.3 or 9, or

F d C 1 Li it• Many commercial

products

FedoraCore 1 Linux run on it.– Price = ~¥120,000 / module.

– Ethernet cards, CPUs, GbE cards, memory modules, etc.

Bootable from CF card or from network.PCI NIC PMC NIC

Inside the COPPER – DMA

• The Cyclone FPGAData FIFOs

FWCLK Almost full

PCI

intr

• The Cyclone FPGA issues DMA start interrupt onto PCI bus when

FWCLKFWEN

FWCLK

Almost full

when– One of FIFOs is almost

full,

FWCLKFWEN

FWCLK

DMA startinterrupter

– #-of events in FIFOs exceeds a threshold, or

FWCLKFWEN

FWCLK

interrupter

– #-of words in FIFOs exceeds a threshold (blocked until all event

FWCLKFWEN

#-of-words#-of-events chunk has transferred

to the FIFO.)151011#-of-words

#-of-eventscounter

#-of-words FIFO

Inside the COPPER – DMA• The CPU starts DMA upon the PCI interrupt.p p

– Counts up #-of words in the FIFO together with header/trailer words.

– Setup DMA and start.

• Then, the Cyclone FPGA controls the FIFO RE/RCLKRE/RCLK.

– ‘Chained’ readout to realize chained DMA.

Timing Distribution for COPPERg

VME6U VME6UVME6U VME6U VME6UVME6U

CAT-5 serial bus -SW

T-R

X

FINESSE

FINESSE

T-IOEQRG

SEQ COPPER

connection TT-

TT

FINESSE

FINESSETTSTR

• SEQ– Event sequence controller.

• TT-IO

COPPER

TT-RX– Clock / trigger repeater from the SEQ.– Busy collection.

• TT-SW

TT-RX…

– Clock / trigger fan-out. Busy fan-in.– TT-SW can be cascaded.

• TT-RXKEK-VME9U Crate

TT-RX

…

– Clock / trigger fan-out to the FINESSEs.– Busy collection from FINESSEs. M.Nakao

Trigger-Busy Handshakegg y• Every L1 trigger to the FINESSE must be replied by busy.

– To avoid possible event loss.– Until the busy is cleared,

the next L1 trigger will be blocked

L1 trigger

the next L1 trigger will be blocked.- L1 trigger delivery resumes 2 SCLKs

after the busy cleared.Busy TT-RXFINESSE

– Busy response: no later than 2 sec or no busy error happens.– Busy duration: 1.5 SCLKs or more, but less than 5 sec or keeping

busy error happensbusy error happens.

• In case of FIFO almost full (remained buffer == event size max)– Do not issue busy immediately or unexpected busy error happens,– but keep busy raised at the response to the next L1 trigger.

COPPER Performance StudyyCOPPER Compressed @ 416 bytes/ev/FINESSE

(t i l d t i f S B ll CDC)Dry run

FINESSE CPU

by 10% (typical data size for SuperBelle CDC)

40

z]

Dry runFINESSE

Trigger Timing Mod.

Design Max L1 rate@ SuperBelle

20

30

rate

[kH

z

Dry runFINESSE

Eth t

Typical L1 rate@ SuperBelle10

20

pted

L1 r

Dry runFINESSE

Ethernet @ Supe e e

0 10 20 30 40Input L1 rate [kHz]

Acc

epThe COPPER works well even

d th t L1 t

Input L1 rate [kHz]

under the severest L1 rate (>30kHz) of the SuperBelle.

AMT-3 FINESSE

AMT-3 FINESSE• Motivations

– First practicable FINESSE.– COPPER in-situ study in the working DAQ.y g Q

• Boundary conditions of R&DBoundary conditions of R&D– Compatibility with the working DAQ (FASTBUS TDC).– Tolerance under the SuperBelle operationTolerance under the SuperBelle operation.

• Our choice of the TDC chip = AMT3Our choice of the TDC chip AMT3

What’s AMT-3 ?• ATLAS MUON TDC

– A TDC chip to readoutthe ATLAS muon chamber.

– http://atlas.kek.jp/tdc/amt3/index.html

– Y. Arai, M. Ikeno, S. Iri, T. Sofue, M. Sagara and M. Ohta, “Development of a new TDC LSI and a VME module,”IEEE T N l S i 49 1164 (2002)IEEE Trans. Nucl. Sci. 49, 1164 (2002).

– Y. Arai, “Development of front-end electronics and TDC LSI for the ATLAS MDT ” Nucl Instrum Meth A 453 365LSI for the ATLAS MDT, Nucl. Instrum. Meth. A 453, 365 (2000).

Why AMT-3 ?y• The AMT-3 satisfies all requirements for the q

SuperBelle drift chamber.Requirement AMT 3

For TrackingP i i l i 130 27

Requirement AMT-3

Position resolution < 130 µm 27 µmFor dE/dx measurement

Dynamic range 10 bit 17 bitLinearity < 0.5 – 1.0% 0.49%

Other boundary conditionsSingle rate 200 kHz

Total channel # ~ 15 k 24 ch/chip

Snapshot of the AMT-3 FINESSEp• “Tandem” FINESSE

– FASTBUS TDC (manufactured by LeCroy) has6 cable-connectors × 16 channels = 96 channels.

– Not to change the cables configuration, we developed “tandem” FINESSE (occupying 2 slots), with 3 connectors.

• AMT-3 TDC chip• Spartan3 FPGASpartan3 FPGA

– AMT3 register control.– Data readout from the AMT-3 output FIFO– Data readout from the AMT 3 output FIFO.– Data formatting (header/footer etc.).

Data output to the COPPER FIFO– Data output to the COPPER FIFO.– I/F to the COPPER local bus.

FINESSE Connector / Inputp• Connector ch#00

#

IN-IN+

– Normal pitch 17x2 pins– 3 connectors

ch#01ch#02ch#03

ch#14h#15

• Input

ch#15GND

de

LEDp– 48 ECL inputs / tandem

- Receiver chip = SN65LVDT348PW

ch#16ch#17ch#18ch#19

PP

ER

sid

– 3 GNDsch#30ch#31GNDC

OP

LEDch#32ch#33ch#34ch#35

Connector pin assignment iscompletely compatible with

ch#46ch#47GND

p y pthe LeCroy 1877S.

Pipelines in AMT-3AMT-3 output

p

R d t FIFOAMT-3

Nevergets full

Readout FIFO(64 edges)

g

L1 pipeline FIFO(256 edges) Trigger matching

Channel bufferChannel bufferChannel buffer Trigger timing FIFO

×24

(4 edges)(4 edges)Channel buffer(4 edges)

Trigger timing FIFO(8 events)

24-channel inputs Level-1 trigger inputs

System Clock y• Bunch crossing rateg

– LHC = 40 MHz– KEKB = 508 MHz

• To operate the AMT-3 with a similar SCLK as the pLHC, we choose SuperBelle SCLK = 42.33 MHz.

– 42.33 MHz = 508 MHz / 12.

AMT-3 Timing Measurementg• Dynamic range = 17 bit: coarse + fine counters

Coarse counter: 12-bit bunch crossing counter(40MHz @ LHC 42 3MHz @ Belle)(40MHz @ LHC 42.3MHz @ Belle)

Fine counter: 5-bit time memory cell1 SCLK / 24

– Edge timings are measured with coarse + fine.T i ti i i h t– Trigger timing is synchronous to coarse.

• LSB = 0 74 ns @ 42 3 MHz SCLK• LSB = 0.74 ns @ 42.3 MHz SCLK– Comparable to the LeCroy TDC (0.5ns)

AMT-3 Trigger Matching• How it works ?

gg g

preset L1 latency

preset searchwindow

tL1 pipelinet=0 L1

Edges are storedin the L1-pipelineL1 trigger comesto the AMT-3

Determine t=0from a presetSearch edgeswithin a preset

Output the foundedges to thein the L1 pipelineto the AMT 3from a preset

L1 latencywithin a presetsearch window

edges to thereadout FIFO

Preset Parameters for CDC

Preset L1 latency (virtual latency)8.1 us

Edge search region8.1 us == whole coverage

Real L1 laetncy2.2 us Delay by TTRX

1.9 us

t=0 SEQ L1 triggerTT-IOTT-RX

to AMT-3

Red numbers are preset based on a request by the CDC group.

AMT-3 Readout

DREADY

CS DSPACE

AMT-3 Spartan3CS, DSPACE

GETDATA

0xa0......0x30......0x30......0x30......0xc0......

D t t f i t ll d bData transfer is controlled byDREADY/GETDATA handshake.

AMT-3 Event Format• start with 0xa• TDC id

header

TDC id• event #• trigger timing

edge data #0

edge data #1• start with 0x3• TDC id• edge type (up/down)

edge data #2

:

• edge timing (relative from L1)• error

edge data #n

error (if happens)

• start with 0x6• TDC id• error type (most likely L1 full)

trailer• start with 0xc• TDC id

event #The AMT 3 can output more information • event #• # of words

(incl. header & trailer)

The AMT-3 can output more informationin principle, but the Spartan3 doesnot recognize other format than this.

AMT-3 Spartan3 COPPERp• AMT-3 Spartan3p

– The AMT-3 outputs are stored in a 32-bit FIFO within the Spartan3.

• Spartan3 COPPER Driven bySpartan3 COPPER– FINESSE header– AMT-3 outputs

ythe Spartan3

AMT 3 outputs– FINESSE trailer

w/ check sum

COPPER_FIFO_WENB

COPPER_FIFO_WCKL

SCLK# of WCLKs within WENB gate is considered as# of words / ev by the COPPER FPGA.

Spartan3 Output Formatp p

0xffaa 0x0000

event# tag Event# (24) counts from 0 /E t t (8)

CP

U.

header

edge data #0

event# tag Event tag (8)

war

eO

PP

ER

edge data #0

:

edge data #ny a

softw

ever

y C

O

edge data #n

trailer

error (if happens)XOR checksum (16)

mov

ed b

yni

ng o

n e

trailer0xff55 checksum

XOR’s of all Spartan3 outputs([31..16] xor [15..0])word=0xor

( )

Rem

runn

([31..16] xor [15..0])word=1:

AMT-3 Device Driver

When loaded by ‘insmod’ • close()• When loaded by insmod– Registers the driver itself to the

Linux kernel.

• close()

– Does nothing special.close(fd);

– Resets the AMT-3.– Presets AMT-3 registers to default

values.• ioctl()

ioctl(fd, COMMAND, arg);

• When unloaded by ‘rmmod’Unregisters the driver

– Resets the AMT-3.– Manipulates AMT-3 parameters.– Views AMT-3 status registers.– Unregisters the driver.

• open()

g– Manipulates Spartan3 registers.

– Rejects duplicated device open.

int fd = open(“/dev/copper/amt3:?”,O_RDWR);

j p p– Does nothing else.

System Test

Full Scale Test BenchNetworkswitches

COPPER/AMT3Pulse Generator

(signal & trigger source) COPPER/AMT36 crates

(signal & trigger source)

• Full-scale data flow simulation from the PG to the readout PCs.D t il d t d f th AMT3 b h i (i l di d b )• Detailed study of the AMT3 behavior (including debug).

• Establishment of the global control scheme.

In-Situ Study with Belle CDCy• Word-by-word comparison of readout data by y p y

the FASTBUS and the COPPER-II/AMT3.COPPER×16

CDC

Q-to-Tfrom CDC

4Residual distribution

AMT3Q to T

2

4

STB

US

3 LS

B)

0

T3 –

FAS

it: A

MT3

Resolution

FASTBUS

AMT3 FASTBUS

Daisy-chain

-4

-2

AM

T(u

ni Resolution0.61 LSB (RMS)

LeCroy TDC×16

AMT3 FASTBUS

Full Replacement of CDC DAQp6 COPPER crates

89 COPPERs Readout PCs

Boot SWTX SW

Q-to-T

Boot SWTX SW Local

event building PCQ-to-T

Boot SWTX SWQ-to-T

Boot SWTX SWQ-to-T

Boot SWTX SWQ-to-T

Boot SWTX SWQ-to-T

Reduction of DAQ Deadtime

Typical data sizeS.Y.Suzuki

t (µs

)

Typical data size

r eve

nt

～29.5 µsFASTBUS

me

per

We achieved deadtime reduction by 90% by installing pipelined

Dea

dtim

COPPER II/AMT3

DAQ with COPPER.

D

～2.8 µs COPPER-II/AMT3

# of hits/TDC

Performance: x-t Curve

x-t curve w/ COPPER/AMT3

ns]

Belle real dataExp=55, Run=1526(Dec.11,2006 15:13 -)

ft tim

e [n

( )

Dri Memo

x-t curve w/ LeCroy

Track-wire distance [cm]

Future

COPPER-3• COPPER-II COPPER-3

– Replacement of terminating parts with recent ones.– Fix jumper patches by pattern layout.j p p y p y– Upgrade onboard Ethernet chip to Gbit-Ethernet.– and others.and others.

We confirmed the COPPER 3 is fully compatible• We confirmed the COPPER-3 is fully compatible with the COPPER-II using AMT3 FINESSE.

Brief Prospects of SuperBelle DAQp p• Preliminary designy gDetector Digitizer Location FPGA link FINESSEPixel ? On hybrid Yes Link RXPixel ? On hybrid Yes Link RXSVD APV25 EH No SpecialCDC ASD based In detector Yes Link RXRICH/TOP Special ASIC In detector Yes Link RXECL Waveform Sa. On detector Yes Link RXKLM ? In detector Yes Link RX

On/In detector Electronics hut

digitizer FPGA Link RXCOPPER

Data and timing link

On/In detector Electronics hut

digitizer FPGA FINESSERocket I/O?

Advertisement …

COPPER Is Not Difficult• Several functions of FINESSEs are ready for you.y y

• Thanks to an excellent device driver, the ,COPPER can be read out quite easily.

int main(int c char *v[])int main(int c, char *v[]){int fd;

fd = open(“/dev/copper”, O_RDONLY);

/* put your FINESSE initialization here */

while(1){static char buf[1024*1024];read(fd, buf, sizeof(buf)); /* data read */

}} That’s all.

COPPER Users• Belle: Half of the DAQ is COPPER’ized• SuperBelle: Planning to utilize the COPPER.

T2K: Beam monitor• T2K: Beam monitor• muSR

Please join us andPlease join us andenjoy data takinge joy data ta g

with the COPPER system.Contact me at higuchit@post.kek.jp

Summaryy• We developed a new readout system “COPPER” toward the

higher luminosity HEP experiment.• We developed a TDC FINESSE equipped with an AMT3 chip

to readout the Belle CDC; In the in-situ study, we found the COPPER/FINESSE/AMT3 system showed high compatibility to the FASTBUS DAQ systemto the FASTBUS DAQ system.

• The COPPER is quite easy to operate. We introduce you to join us and to be a member of COPPER user teamjoin us and to be a member of COPPER user team.

• We are designing COPPER-3. Design of a new FINESSE for SuperBelle has also been started. In SuperBelle, one of key p p , yroles of the FINESSE will be data RX.