fpga firmware  of dc5 fee

1

FPGA firmware of DC5 FEE

Chih-Hsun Lin, Yu-Sheng Teng, Ming-Lee Chu,Chia-Yu Hsieh, Takahiro Sawada, Wen-Chen Chang

Institute of Physics, Academia Sinica, Taiwan

Tobias Grussenmeyer, Horst FischerInstitute of Physics, University of Freiburg, Germany

DC5 FPGA firmware review 2014/5/15

2

Outline

• System architecture– Module connection– Signal/CLK connection

• FPGA design– FEM

• TDC measurement• Trigger match

– DCM• Data packing and transfer

• List of identified problems during the on-site test• Summary

FEM-DCM Architecture

3

• FPGA in FEM (LFXP2-8E-6FT256C)• 12 EBR Blocks (221184 bits)

• FPGA in DCM (Spartan-6 xc6slx45t-3fgg484)• 116 18Kb RAM Blocks = 232 9Kb RAM block

4

Module Connection

Signal and Clock rate

— Optical Tx and Rx 3.1104 Gbps— Master CLK 38.88 MHz (was 155.52MHz)— Command 38.88 Mbps (was 155.52MHz)— FLT (First level Trigger) Pulse sync to 38.88MHz CLK — Data 155.52 Mbps

Command & FLTGANDALF DCM FEM

Optical fiberEthernet

cable

Master CLK

Data

FEM

……

Optical Tx

Optical Rx

x1 x8 x20

Totally1 GANDALF8 DCM160 FEM

5

Experimental Parameters

• Highest hit rate per wire = 200 kHz• Noise rate per wire = 10 kHz (conservative estimation based

on the measured value at 4 fC threshold)• Overall hit occupancy = 10 %• Trigger matching window = 200 ns• Highest instantaneous trigger rate = 100 kHz, i.e. minimum

time between two consecutive triggers = 10 μs.• Trigger latency = 8 μs

• We will use these parameters to evaluate the proper specifications of FEE.

6

Data Transfer• FEM DCM

– Data format : 32 bits- Header : 2 bits- FEM ID (0-19): 6 bits- channel ID (0-15): 4 bits- Time (range 70 s, finest 1-ns resolution): 16 bits- PLL locked: 4 bits

- Speed = 155.52 Mbps - Maximum transmission time per trigger = 32bits/word * (16+2) words / 155.52 Mbps

~ 576 bits/ 155.52Mbps ~ 3.7 μs < 10 μs trigger period – Minimum cyclic buffer size : (200+10) kHz *16 hits * 8μs ~ 30 hits < 512 hits buffer

size

• DCM GANDALF- Speed = 3.11 Gbps - Maximum transmission time per trigger = 32bits/word * [(16+2) words *20 FEM +

5 S-link header words]/ 3.11 Gbps

~ 11680 bits/ 3.11 Gbps ~ 3.8 μs < 10 μs trigger period

• GANDALF ROB- Speed : 160 MByte/s = 1.28 Gbps- Maximum transmission time per trigger = 32bits/word * [(16+2) words *20 FEM +

5 S-link header words]* 8 DCM *10% occupancy/ 1.28 Gbps

~ 9344 bits/ 1.28 Gbps ~ 7.3 μs < 10 μs trigger period

7

Clock Structure

GANDALF

DCM FEM

Optical Tx

Optical RxXilinx

Transceiver

OSC155.52M

Rx clock155.52M

AlignCommaK28.1 k28.5

Deformater PLL1

PLL2

155.52M

FEM Ctrl Logic

77.76M

FEM2DCMtransmit

155.52MB

TDC

Clock 38.88M

233.28M

233.28M 90°

2M CMAD setting

• Deformater is a VHDL code from T. Grussenmeyer.— Master 38.88MHz CLK is generated and phase adjusted by RX CLK and

command from GANDALF.• Two TDC CLKs will be 233.28MHz = 38.88MHz x 6. (248MHz now)• 155.52MB is a phase adjustable CLK for data output.

8

First Level Trigger

• In DCM, First level trigger is generated according to a specific command/data pattern from GANDALF.

• A FLT pulse is distributed to FEMs sync to master CLK.

GANDALFOptical Tx

Optical RxXilinx

Transceiver

Rx clock155.52M

FLTCommaK28.0

Deformater

Clock 38.88M

FLT(first level trigger)

9

TCS Reset signal

• TCS reset will be distributed to FEMs by a dedicated command sync to master CLK.– Implemented on DCM and FEM.• A command to FEMs to reset TDC counters in the

current implementation.

– Need inputs about TCS Reset command/pattern from GANDALF.

10

Trig Time

FEM TDC Block Diagram

CMADx16 inputs TDC

x16

FLT trigger TDC

x1TriggerLogic

Flag

Reset

TriggerMatch

Trig FlagEventFIFO(512 x 32bit)

DCM & FEM(8b/10b)Link logic

BufferCtrl

Logic

x16 Time

x16 Flag

x16 Reset

Cycling buffer

(512 hits)

Trig Flag

Write point

# of TDC hits

0100110110Serial data

Commandhandler

It will be4096x32 bit.

Trig Time

Data

11

TDC counter

• TDC value for each hit is 16 bit.– MSB 14 bit is from a counter by 233MHz CLK.– LSB 2 bit is determined by a four bit pattern

latched with 233MHz CLK and 233MHz 90o CLK.

14 bits 2 bits

233.28M

233.28M 90°

14 bits ≈ 70.3us

1 unit = 1.07ns

12

Trigger Match• Data and control parameters:

– TDC time = 16 bit (1ns lsb) [maximum 70 s]– Trigger Latency = 12 bit (4ns lsb) [maximum 16 s]– Matching Window = 12 bit (4ns lsb) [maximum 16 s]

• (Ttrig–Tlatency–Twindow) < Thit < (Ttrig–Tlatency+Twindow)

• Matching process stops at either one of the following three conditions:– 16 matched hits.– No more TDC hit for matching(Max hits for matching process is 255).– 4 unmatched hits after last matched hit.All the parameters for these three conditions could be adjusted and optimized according to the realistic experimental conditions.

wire

FLT

TlatencyThit Ttrig

Twindow

13

DCM block diagram

010011Serial data

DCM & FEM(8b/10b)

Transmitter

x20

DCM to FEMCommand

FIFO

x20

FEM to DCMDataFIFO

(512 x 32bit)

x20

Commandhandler

GANDALFDCM

Link logic

Transceiver 010011Serial data

TCS info

command

FLT(first level trigger)

1bit Event Counting

FIFO

x20data

packing

dataDCM & FEM

(8b/10b)Receiver

x20

010011Serial data

14

Current data-packing procedure in DCM

Idle

Any FEM framevalid

No

Wait up to 4 system clocks

Yes

Timeoutor

all FEM valid

No

Send S-link begin mark

Yes

Send S-link header

Scan FEM FIFO

framevalid

Readout FEM data

EOF wordNo

All FIFOs scaned

Yes

Yes

No

Send S-link end mark

Yes

Power on reset

No

15

List of Identified Problemsduring the on-site test

• Data loss (DCM to GANDALF)• Command error (DCM to FEM)• Command lost (PC with USB connection to GANDALF)

16

Data Loss issue(DCM to GANDALF)

• Previous version of DCM FPGA design, the same state machine controls both command flow and data flow.– Whenever a command arrives in DCM, the data packing

and transmission is interrupted. This is the found cause of the data loss.

• Solution: The DCM FPGA design is modified to have independent control for:– Command flow (GANDALF DCM FEMs)– Data flow (FEMs DCM GANDALF)

17

Latest Data transmission Test(with new DCM firmware)

2. DCM generates100k trigger in one sec.

GANDALF DCM FEM

5. Use counter check validdata frame number from FEM

Optical fibre

x1 x1 x1

Count mode

USB Ethernet cable

1. Pass command Trigger_on to DCM

3. FEM send 1 data frameto DCM per trigger

4. Packing data frame andpass to GANDALF

6. Pass data frame to PC7. Save data into fileand use program analysis

• Test result– DCM did receive 100k data frames from FEM.– There is a loss of 1.5% data frame for the data recorded in PC and the

cause is likely the current USB I/O capability. The data rate for 100k Hz trigger test is 100k*4 byte per word* 23 words per FEM = 9.2 MB/sec.

18

Command Error issue(DCM to FEM)

• A timing issue, long operational logic path due to– 8b/10b encoding– Multiplexing of commands and fill pattern

• Solution: Pipeline/FIFO is added in the DCM FPGA design to reduce logic path.– Test done. All 20 FEM ports work correctly.

19

Command lost issue(PC with USB connection to GANDALF)

• DCM could not 100% successfully receive the control command issued from PC via GANDALF if PC is reading data from GANDALF and sending commands to GANDALF simultaneously.This is due to the handshaking of USB port between PC and GANDALF. It is not a valid issue for the reality where the data control is via the VME backplane.

• Solution: Make sure that the trigger is properly stopped before sending any other control commands.

20

Summary

• The proper functioning of clock/command/data chain in the current FPGA firmware is verified and the optimization study will continue.

• All identified problems of digital transmission during the on-site test have been investigated and solved.

• We trust that the current FPGA firmware and the hardware design overall do fulfill the experimental requirements of DC5 FEE. It should be reasonable to move toward the stage of mass production in this project.

21

BACKUP

22

Command lost (DCM to FEM)

DCM to FEMCommand FIFO

• Timing issue when do 8b10b encoder– Original structure

8b10bencoder

control

command

Idle(K28.5) Trig func

Mode controlstatus

0101001100

ERROR

control

Controller + Serializer

23

Fix Command lost (DCM to FEM)

• New version

DCM to FEMCMD FIFO

CMD CMD / idleSelector

32b to 8bFIFO

8b10bencoder

Serializer

Trig modefunc

Mode select with command for DCM0101001100

FLT pulse signal

24

CMAD Control (1) Functionality Description Command Note

ThresholdSetting (Threshold , Ch_id)

Set Threshold for 1 channel

dumy = Threshold And 0x3FFByte (0) = 0x80 + (dumy And 0xF)dm = (dumy And 0x3F0) / 16Byte (1) = 0x90 + 4 + (dm And 0x1)Byte (2) = 0xA0 + (id And 0xF) Byte (3) = 0xB0 dm = (dumy And 0x3E0) / 32Byte (4) = 0x80 + (dm And 0xF)dm = (dumy And 0x200) / 512Byte (5) = 0x90 + (dm And 0x1)Byte (6) = 0xA0 + (id And 0xF)Byte (7) = 0xB0

• Control 8 bytes• Threshold 0-650 digit• Ch_id 0-8

25

CMAD Control (2) Functionality Description Command Note

BaselineSetting(Baseline,ch_id)

Set gain for 1 channel

dumy = Baseline And 0x3FF Byte(0) = 0x80 + (dumy And 0xF)dm = (dumy And 0x3F0) / 16Byte(1) = 0x90 + 6 + (dm And 0x1)Byte(2) = 0xA0 + (id And 0xF)Byte(3) = 0xB0dm = (dumy And 0x3E0) / 32Byte(4) = 0x80 + (dm And 0xF) dm = (dumy And 0x200) / 512Byte(5) = 0x90 + 2 + (dm And 0x1)Byte(6) = 0xA0 + (id And 0xF)Byte(7) = 0xB0

• Control 8 bytes• Baseline ~650 digit

26

CMAD Control (3) Functionality Description Command Note

Gain_Setting(Gain,Ch_id)

Set gain for 1 channel

dumy = Sgain And 0xF Byte(0) = 0x80 + (dumy And 0xF)Byte(1) = 0x90 + 8Byte(2) = 0xA0 + (id And 0xF)Byte(3) = 0xB0Byte(4) = 0x80 + (dumy And 0xF)Byte(5) = 0x90 + 12Byte(6) = 0xA0 + (id And 0xF)Byte(7) = 0xB0

• Control 8 bytes• Gain 0-15 digit

27

CMAD Control (4) Functionality Description Control Note.

Setting (OneShot, Ch0TestOutput, SlowRateLimit, ThresVoltageOutput)

Bit 7-4 : 0xC0

Ex. We usually set“One Shot” On +”Ch0 Test Output” Off+” Slew Rate Limit” On+”Thres Voltage Output” OffSetting(0,0,1,0)0xC0+ b0010 = 0xC2

OneShot - Advantage: Reduce “bounce” of output signal after Dis. because of noise.- Disadvantage: Decrease the sensitivity. (see backup slide)

Bit 3 On = 0Off =1( def = 1)

Ch0TestOutput - Output signals of ch0 after preampand before discriminator. - Signal of Ch0 after Dis. will be turnoff if we turn on the Ch0TestOutput.

Bit 2 On =1Off =0 ( def = 0)

SlewRateLimit - Advantage: Avoid the oscillation and hold the baseline. - Disadvantage: Decrease the bandwidth of chip. (see backup slide)

Bit 1 On = 1Off = 0 (def = 1 )

ThresVoltageOutput - Output the voltage of threshold to test pins.

- If we turn of ThresVoltageOutput, the output of all channels will be turn of.

Bit 0 On =1Off = 0 (def = 0)

28

CMAD Control(5)

SelfTest(OddCh, EvenCh)

Bit 7-4 : 0x1

Ex.Test both odd and even channels SelfTest(1,1) 0x18

Ex.Test both only odd channels SelfTest(1,0) 0x13

OddCh Output clock signals generated by FPGA to Pin4 of CMAD (Test odd channels of CMAD)

Bit 0-1 :0x3(def = 0x0)

EvenCh Output clock signals generated by FPGA to Pin6 of CMAD (Test even channels of CMAD)

Bit 2-3:0xC(def = 0x0)

Functionality Description Control Note.

29

FEMID(6)

ChID(4)

DCM ID(10)

ChID(4)

FEMID(6)

30

S-Link Header(1)

must be defined.

31

S-Link Header(2)

chosen