Frank Lemke

Experiences and results using the CBMnet protocol including Experiences and results using the CBMnet protocol including precise time synchronization and clock distribution precise time synchronization and clock distribution

DPG – Frühjahrstagung Münster 2011

HK 20.7

University of Heidelberg

Computer Architecture Group

Frank Lemke, Sven Schenk, Ulrich Brüning

22.03.2011

Frank Lemke2

OutlineOutline

Motivation CBM Protocol Structure CBM Network Synchronization Beamtime results Dec. 2011 Adaptions heading towards CBMnet V2.0 Conclusion & Outlook

Frank Lemke3

MotivationMotivation

Standard Protocols abilities are only partly reasonable for the special demands within CBM DAQ System

An optimized design of a CBM protocol will result in

clearly better performance

CBM unique requirements considered under the topics of Space and cost limitations High bandwidth Specific system synchronization Reusability of protocol modules for different

Hierarchical Structure and Implementation

Frank Lemke4

CBM Network - Traffic ClassesCBM Network - Traffic Classes

• Data Transport Messages (DTM)• Data Messages with CRC• Only with error detection

• Detector Control Messages (DCM)• Control Messages with CRC• Retransmission on error

• Deterministic Latency Messages (DLM)• 1-Bit Error correction

• Additional Administration Packets• 1-Bit Error correction• IDLE, INIT, ACK, NACK …

Retransmission

CRC

CRC

Link_init

ACK/NACK

Arbiter

DLM

Data

Control

Link

Frank Lemke5

CBMnet V1.0 - Packet StructuresCBMnet V1.0 - Packet Structures

IDLE DLM IDLE… …16 Bit 16 Bit 16 Bit

IDLE INIT IDLE… …16 Bit 16 Bit 16 Bit

INIT… … ACK

16 Bit16 Bit

DTM:

DCM:

DLM:

Administrative packets:

SOSC Payload

16 Bit 4 - 64 Byte

R outing CRC EOP_C

16 Bit 16 Bit 16 Bit

R outing

16 Bit

SOP Payload

16 Bit 4 - 64 Byte

R outing CRC EOP

16 Bit 16 Bit 16 Bit

Frank Lemke6

CBMnet - Communication FlowCBMnet - Communication FlowC

omputing C

luster

Detector Frontend

ECS

FEEHub

DPB

Low

Bandw

idth

High B

andwidth

FEE

FEE

FEE

FEEHub

FEE

…

…

Control flow

Data flow

E/O

Frank Lemke7

DLM & Synchronization SpecificsDLM & Synchronization Specifics

System requirements Clock recovery and distribution for all system levels Guaranteed deterministic timing during runtime at all times

Special features provided by DLM usage Heterogenous synchronized system initialization Enables periodic time synchronization for epoch markers Various special user defined event signalling

with deterministic latency

Implementation specifics Priority request insertion Deterministic and structural separated HDL coding Specific HW requirements - FPGA configuration scheme and settings

DATASOP CRC EOP

DLM

Frank Lemke8

CBMnet - Synchronization FlowCBMnet - Synchronization FlowC

omputing C

luster

Detector Frontend

ECS

FEEHub

DPB

FEE

FEE

FEE

FEEHub

FEE

…

…

Initial HeterogenousSynchronization

Periodic SynchronizationResynchronization

E/O

Frank Lemke

Cosy Beamtime December 2010Cosy Beamtime December 2010

9

DCB

Jitter cleaner device50 meter cable toread-out controller

3 meter cables toABB and ECS

Frank Lemke

CBMnet Beamtime ResultsCBMnet Beamtime Results

Detector Control Messages Worked without problems during complete beamtime tests

Data Transfer Messages Merging of 4 data-streams within DCB into a single stream No fatal data transmission error No data flow problems

Deterministic Latency Messages All counters values were reset synchronous Synchronization using DLMs showed no problems

10

Frank Lemke11

Packet Format CBMnet V2.0Packet Format CBMnet V2.0

SOP Payload

16 Bit 2 - 64 Byte

R outing LME CRC EOP

16 Bit 16 Bit 16 Bit 16 Bit

Long messages extension (LME) requires a small message definition Insertion of extension character Reduction of buffer space Balanced network message flow LME packet well defined No additional overhead for small messages Type field within LME code required Additional overhead circa 2% Total overhead 11%

Frank Lemke12

Long Message Extension (LME)Long Message Extension (LME)

TYPE MTAG DSN

4 Bit 4 Bit 8 Bit

Type: 4’b Identifier for LME, 1111 reserved

Message TAG (MTAG): 32 different long messages can be handled

Decrementing Sequence Number (DSN): Shows the amount of packets following

Maximal length = 256*64Byte = 16KByte

Frank Lemke13

LME - Data Flow InfluencesLME - Data Flow Influences

DLMs still easily insertable with priority request insertion into message streams without increasing complexity

DCMs and administration chars can be arbitrated inbetween LME packets

Numerous message streams can be merged into one aggregate data stream

SOP Payload

16 Bit 2 - 64 Byte

R outing LME CRC EOP

16 Bit 16 Bit 16 Bit 16 Bit

SOP Payload

16 Bit 2 - 64 Byte

R outing LME CRC EOP

16 Bit 16 Bit 16 Bit 16 Bit…

DLM

16 Bit

SOSC Payload

16 Bit 4 - 64 Byte

R outing CRC EOP_C

16 Bit 16 Bit 16 Bit

R outing

16 Bit

…

…SOP Payload

16 Bit 2 - 64 Byte

R outing LME CRC EOP

16 Bit 16 Bit 16 Bit 16 Bit

=> LME has no major influence on the network flow characteristics

Frank Lemke14

Features towards CBMnet 2.0Features towards CBMnet 2.0

CBMnet V1.0 Features Planned additional CBMnet V2.0 extensions

• Communication over one optical linksupporting DTM, DCM and DLM

• Optimized data utilization about 91 % ( about 73 % considering 8b/10b)

• Optimized easy to use Interface• Highly modular CBM LP code structure• Special adapted routing scheme• Fast and efficient administration packets• Retransmission for Control Packets• System wide clock recovery with low jitter• Deterministic link latency feature for well

defined Deterministic Latency Messages

• Lane handling for unbalanced communication

• Support for large messages within hardware

• Planned data loss strategy in cases of overloads at the end of epochs

• Communication also reliable for data stream

• Meta data detection adaption to find Meta data within message streams in early system stages

Frank Lemke15

Conclusion Conclusion && Outlook Outlook

Beamtime results show reliability and usability of CBMnet V1.0 Planned extensions fit into CBMnet scheme for V2.0

Interface stays almost the same Implementation complexity should be manageable

The next step will be to include CBMnet logic into frontend ASICs Also next generation FPGA ROCs and a possible read-out ASIC are

planned and will probably be implemented soon

Frank Lemke16

Thank you for your attention !

Questions ?