Read-Out Driver for ATLAS Silicon Detectors

Atlas Wisconsin/LBNL Group

March 21st 2007

ATLAS Pixel B-Layer Upgrade Workshop

2

Control Path:

ROD Controller FPGAMaster DSP

Program Reset Manager FPGA

Operation, Commands, Triggers

96

Ba

ck

of

Cra

te C

ard

/ F

ron

t E

nd

Ele

ctr

on

ics S

Lin

kC

ard

Clo

ck

F

an

ou

t

Triggers and Event ID Data

Tim

ing

Inte

rfa

ce

Mo

du

le

BOCSetup Bus

Serial Input Links

Xon/Xoff

ROD Busy

DSP Farm (4)

Event Trapping, Histograms

Data Path:Formatter, Event Fragment Builder & Router FPGA,Diagnostic Memory, S-Link Interface & SDSP DMA

DMA 4

...

S-Link

48

System Clock

Serial Output Links

VME64x Signals

VM

E

Bu

s

40

DMA 1

ClocksHost Port

Silicon Read Out Driver (SiROD)

3

Silicon RevE(F) SiROD

MasterDSP

ControllerFPGA

FormatterFPGAs

Router FPGA

EventFragmentBuilderFPGA

SlaveDSPs

9U VME

VME Interface

Clock Distribution

4

• FE Module Interface:

• 96 FE Data input connections available

• 48 FE Command output connection available

• 2048x32 Data FIFO for each Module (32 total)– Pixel B-Layer: 6-7 modules

• 12-14 80MHz input data links

• 6- 7 40MHz output command links

– Pixel Barrel L2: 26 modules

• 26 40MHz input data links

• 26 40MHz output command links

– Pixel Disks: 13 modules (26 ROD channels)

• 13 80MHz input data links

• 13 40MHz output command links

ROD Overview (1)

5

• Diagnostic components included for stand alone testing

• Maintain data throughput at 100kHz trigger rate

• Flexible calibration processes• MDSP Control

• ROD Controller FPGA Control

• TIM Control

ROD Overview (2)

6

32 MB SDRAM

VME Slave FPGA

Configuration Controller

ROD BUSY Histogram

16Mb Flash Memory

Master DSP

TI 320C6201

160MHz

ROD CONTROLLER

FPGA

FORMATTER FPGA

8x

EFB FPGA

ROUTER FPGA

DSP FARM TI 320C6713

220MHz 256MB SDRAM

4x

DIAGNOSTIC FIFOs

MDSP EMIF

RODBus

VME Bus

Back Of Crate (BOC)

CNFG DATA to All FPGAs

RESET

ROD BUSY

BOOT ROM 512KB

VME Bandwidth:

~8MB/s MDSP~4MB/s SDSP

~22MB/s VME Slave

All DSPs:

16 bit Host Port Interface

D16

D32

A32/D32

D16

ROD Control/VME Path

7

RCF RODBus Interface

(FPGAs, DSP Farm)

MDSP EMIF Data (32)

Control & Status Registers

RODBus Data (16)

RODBus Address

RODBus Control

MDSP EMIF Address

MDSP EMIF Control

ROD CONTROLLER FPGA: Control Functions

Diagnostic FIFO Interface

DSP Interrupt Interface

Reset Command Interface

MDSP Interrupts SDSP Interrupts

MDSP Serial Port Interface

INPUT FIFO (32K x 96)

EVENTMEM FIFO (16K x 48)

To VME/PRM FPGA

To MDSP SP0/SP1 ports

8

MDSP: Primitive List State Machine• Primitive List execution: Host-to-MDSP and MDSP-to-SDSP

Primitive List Execution

IDLE

PAUSED EXECUTING(one primitive per

Iteration)PREPAREREPLY

ACKNOWLEDGE List Inserted?

YES

Pause?

Finished or Abort?

NO

YES

Abort?

Resume?

YES

inList Bit ClearedBy Host?

NONO

YES

NO

NOYES

NO

YES

9

Text Buffer State Machine

• Text Buffer reply: MDSP-to-Host and SDSP-to-MDSPError, Info, Diagnostic and Transfer (MDSP is a way-station for SDSP text buffers)

Text Buffer Transmission

Buffer Occupied?Buffer Read?

Read Request?

IDLE

WAITINGFROZEN

YES

YES

NO

NO

NO

YES

10

ROD CONTROLLER

FPGA

XC2S600-6FG676

FORMATTER FPGA

EFB FPGA

XC2S400-6FG676

ROUTER FPGA

XC2S400-6FG456

DSP FARM TI 320C6713

220MHz 256MB SDRAM

4x

Xoff

BOC

FORMATTER FPGA

FORMATTER FPGA

FORMATTER FPGA

FORMATTER FPGA

FORMATTER FPGA

FORMATTER FPGA

FORMATTER FPGA

INPUT FIFO 32K Deep

S-Link

EVENTFIFOs

216Kx48

A

B

Event Fragments

Xoff

Xoff

ROD BUSY

TTC

EVENT ID

READOUT

TIM

MDSP SP

BOCFE CMD (48)

XC2S600-6FG456

SiROD DATA PATH

Can sustain 100KHz Trigger Rate into S-Link

11

RCF RODBus Interface

MDSP EMIF Data (32)

Control & Status Registers

RODBus Data (16)

RODBus Address

RODBus Control

MDSP EMIF Address

MDSP EMIF Control

Trigger Processor &

Counter

MDSP Ser Port 0

MDSP Ser Port 1

TTC L1 Trigger

FE Command Processor &

Mask

Event Processor

Dynamic MasksMode Bits

Event ID DataTrigger Type

Internal Scan Processor

FE Event Counter

FE CMD (48)

TTC EVT

EFB EVT (16)

FMT MB (12)

Diagnostics Generator

FMT Trailer Detect

ROD CONTROLLER FPGA: Real Time Functions

12

RODBus Interface

Control & Status Registers (32)

RODBus Data (16)

RODBus Address

RODBus Control

Link Input MUX

&

Half Clock

Counters

IN0

ModeBits FIFO 512x24

Link FIFO 2048 x 32

Link Decoder: 40/80/160MHz

Read Out Controller

DATA OUT (40)

TOKEN

FMT MB (12)

Link FIFO 2048 x 32

Link Decoder: 40/80MHz

Link FIFO 2048 x 32

Link Decoder: 40MHz

Link FIFO 2048 x 32

Link Decoder: 40MHz

IN1

IN2

IN3

IN4

IN5

IN6

IN7

IN8

IN9

IN10

IN11

FMT Type

FMT ID

HT LIMIT

ROD BUSY

Trailer DetectTRAILER (12)

PIXEL FORMATTER FPGA (8x)

13

EVENT FRAGMENT BUILDER FPGA

EventMemA

FIFO16Kx48

Engine 0

Engine 1

EventMemC To

/Fro

m R

ou

terT

o/F

rom

Fo

rmat

ter

To

/Fro

m F

orm

atte

rT

o/F

rom

Co

ntr

olle

r

Output FIFO Data (43)

Header and Trailer Data (32)

Halt Output

Halt Output

Header/TrailerGenerator

Error Summary Word

Error Summary Word

Fo

rma

t &

Co

un

tF

orm

at

& C

ou

nt

L1

/BC

ID

Ch

ec

kL

1/B

C I

D C

he

ck

Err

or

Ch

ec

kE

rro

r C

he

ck

INC

/DE

C L

1ID

L1 & BC IDs

Event Data& Trigger Type

Dynamic Mask

Link Num. (6 bits)

Link Num. (6 bits)

Time-out Error

Time-out Error

L1ID Error

L1ID Error

BCID Error

BCID Error

Data (32 bits)

Data (32 bits)

Formatter Num. (2 bits)

Formatter Num. (2 bits)

Link Num. (4 bits)

Link Num. (4 bits)

Data (32 bits)

Data (32 bits)

Data Valid

Data Valid

Time Out

Time Out

INC

/DE

C L

1ID

Purpose: Collect Formatter output, check L1 and BC

IDs, count errors, generate Event Header & Trailer

FIFO Controller Xoff

EventMemB

FIFO16Kx48

14

ROD Router (FPGA) & Slave DSPs

SDSP 3SDSP 2SDSP 1SDSP 0

Read, formatand direct the

event data

2 TrapsEvent Type: ATLAS,

ROD or TIM

2 TrapsEvent Type: ATLAS,

ROD or TIM

2 TrapsEvent Type: ATLAS,

ROD or TIM

2 TrapsEvent Type: ATLAS,

ROD or TIM

DMA Transfer Engine1024 32-bit FIFO

DMA Transfer Engine DMA Transfer Engine DMA Transfer Engine

S-L

INK

To

/Fro

m R

OS

To

/Fro

m E

ven

t F

rag

men

t B

uil

der

XOn / XOff S-Link

S-Link DataDsp Halt Output

Halt Output

Error Format Data

Data Valid

Output FIFO (43b)

Event Header andTrailer Data (32b)

Texas Instruments 6713 floating point DSPs running at 220 MHz for monitoring and calibration histogramming

Purpose: Route formatted data to Level-2 and/or Slave DSPs (histogram)

1024 32-bit FIFO 1024 32-bit FIFO 1024 32-bit FIFO

15

Additional Slides

16

The Silicon Read-Out Driver (ROD)

MODULE BOC

SBCTIM

ROS ROD

Controller& Master DSP

Formatters Event FragmentBuilder

Router& Slave DSPs

(Read Out Subsystem)

(TIming Module)

(Back Of Crate)

(Single Board Computer)

To Event-Builderand Level-2

CalibrationHistograms

• Secondary purpose: Calibrations / MonitoringFPGAs for time-critical functions (Event Data Path)

DSPs for configuration, ROD control, calibrations and monitoring

• Primary purpose: Module configuration, Trigger propagation, Data formattingA hybrid of FPGAs and DSPs

• The Master DSP (MDSP) has ROD and BOC registers connected to one of its EMIFs (External Memory InterFace)

Event DataConfiguration & Triggers

Backpressure / Halt Output S

-Lin

k

17

Pixel Calibration Scans• A couple of common Pixel Scans:

Threshold (on chip charge-injection for each individual pixel; scan the number of hits for each injected charge to obtain the discriminator threshold)

Noise (a threshold scan without charge injection to measure noise)

Threshold Scan resultshown for a

single Pixel Module

Threshold for each pixel(note the 16 FE ICs)

18

SCT Calibration Scans• Some of the commonly run SCT Calibration Scans:

Rx Threshold Test (optimize the Rx threshold value in the BOC data-receiver chip)

NMask Test (demonstrate that the chip mask register functions properly)

Pipeline Test (scan the chip pipeline for defects; stuck on or off)

Full Bypass Test (does the chip bypass feature function properly)

Noise Occupancy Test (a threshold scan without charge injection to measure noise)

Synchronous Trigger Noise Test (triggers distributed synchronously across the system)

N-Mask Scan resultshown for a

single SCT Module

19

ROD use during detector production / assembly• Calibration scans have been routinely used during module production

Verify module functionality before and after mounting or transport

Classification of modules• Rank by the number of defects; only mount and install the very best

• This phase also gave feedback to ROD developers (and still continues to!)A few DSP software and FPGA Firmware problems have been uncovered this way

• Some unique bugs appeared when stressing the ROD with a full complement of modules

20

Cosmics Data-Taking• TRT/SCT combined run in May 2006

ROD performed well; successfully collected more than 400k events

Useful DAQ/Detector shake-down, data for efficiency and alignment studies

• On-ROD Histogramming used for timing• in the SCT during cosmics running• (i.e., delay the signal by the correct amount)

Transmission Delay(bunch-crossings)

Number of coincident hitson top and bottom strips

Noise

SCT

Transition Radiation Tracker (TRT)

A total of 504 SCT Modules into 12 RODs were used during the combined cosmics data-taking run

20 cm of concrete

Scintillator

Scintillator

Scintillator

21

SCT RODs in USA-15• Production RODs in their final location for ATLAS running