may 29, 2013km3net, clbv2 meeting peter jansweijer mesfin gebyehu nikhef amsterdam electronics-...

May 29, 2013 KM3NeT, CLBv2 Meeting

Peter JansweijerMesfin GebyehuNikhefAmsterdamElectronics- Technology

1

KM3NeT CLBv2



Visual Status

2

Rx_m

ac2

buf

I2C

Fifo

31 TDCsTDC

0

Management

& Control

Data

ControlWishbone bus

RxPacket

Buffer64KB

IP/UDP Packet BufferStream Selector (IPMUX)

Rx_b

uf2

data

RxPort 1RxPort 2

RxPort_m

Management

& Config.

Tx_p

kt2

mac

Tx_d

ata

2b

uf

TxPort 1TxPort 2

TxPort_m

Flags

Rx

Str

eam

S

ele

ct

TxPacket

Buffer32KB

Flags

Tx

Str

eam

S

ele

ct

31

PM

Ts

UTC time & Clock (PPS, 125 MHz)

Pause Frame

ADC

Management

& Control

Hyd

rop

hon

e

Fifo

TDC30

Fifo

NanoBeacon

GPIODebug LEDsI2C

Debug RS232

Temp Compass

TiltPoint to Point interconnection

XilinxKintex-7

Start Time Slice UTC &Offset counter since

Tim

e S

lice S

tart

MEMS

2nd CPULM32

M

M

WB Crossbar(1x7)

WB Crossbar(3x2)

SM

SM

M

S

S

MM

M

S SSUART

S

M

M

S

S

MM

Sta

te M

ach

ine

SPIS

M

SPIFlash



Done:◦ LM32 + WB-Crossbar + DPRAM + UART◦ Soft-PLL FMC layout◦ WR without PCI-express

Currently:◦ Deterministic PHY => First shot White Rabbit in KC705◦ Soft PLL (hardware + software). First goal: lock onto a 125

MHz xtal and phase shift under control of LM32 via UART To do (in order of priority):

◦ Endpoint (= MAC) <= Complex!◦ Mini-nic <= Complex!◦ Fabric redirector <= probably less complex◦ PPS generator <= relatively straightforward◦ 1-wire, SysCon <= easy?

Status Listing

3



Integration 1 month from now… Ouch!Currently:

◦ Soft PLL (hardware + software) => 2 month?◦ Endpoint (= MAC) <= Complex! => 1 month?◦ Mini-nic <= Complex => 1 month?◦ Fabric redirector <= probably less complex◦ PPS generator <= relatively straightforward◦ 1-wire, SysCon <= easy?◦ Connection Endpoint IP-MUX => 2 weeks?

Estimation: ~4,5 Month (if we are lucky)!◦Please also note: Peter is involved in another

project the coming months!

Planning (White Rabbit + IP-Mux)

4



Integration of other objects◦White Rabbit + IP-Mux◦2nd LM32 system◦TDC’s / FIFOs / State Machine◦Hydrophone

=> Create complete design (hardware, software) + test environment (simulation)

Planning Intergration

5



GTXE2_CHANNELIBUFDS_GTE2

gtx_

dedi

cate

d_cl

k

fpga_pll_ref_clk_101_p_i

fpga_pll_ref_clk_101_n_i TXOUTCLK_OUTclk_gtx_i

BUFGRXOUTCLK_OUT

rx_rec_clkRXUSRCLK_IN

RXUSRCLK2_IN

TXUSRCLK_INTXUSRCLK2_IN

GTREFCLK0_IN

rx_r

bclk

_o

wr_gtx_phy_kintex7.vhdEntity: wr_gtx_phy_kintex7

Kc705_top.vhdEntity: kc705_top

IBUFGDS

clk_

125m

_pllr

ef

clk_125m_pllref_p_i

clk_125m_pllref_n_i

PLL_BASECmp_dmtd_clk_pll

pllout_clk_dmtd

Gc_extend_pulse?Clk_i

SPEC = FPGA_CLK_P/NSpartan6 pin G9/F10

SPEC =Spartan6 pin C11/D11

SoftPLL FMC

DAC1

DAC2 VCXO20MHz

VCXO25MHz

CLK20_VCXO

CDCM61004

fpga_pll_ref_clk_123_p_i

fpga_pll_ref_clk_123_n_i

CPLLRESET_IN

dac_dpll

dac_hpll

BUFG

PLL_BASECmp_sys_clk_pll pllout_clk_sys

Clk_20m_vcxo_i

Timing reference (125 MHz)

xwr_core.vhdEntity: xwr_core

clk_ref_i(0)

clk_fb_i(0)

clk_dmtd_i

clk_ref_i

phy_ref_clk_i

wr_core.vhdEntity: wr_core

????.vhdEntity: xwr_softpll_ng

clk_ref_i

clk_sys_iclk_dmtd_i

xwr_endpoint.vhdEntity: xwr_endpoint

clk_sys_i

BUFG

clk_

dmtd

_i

phy_ref_clk_i

HPC FMC only!

HPC FMC only!

Gtp_bitslide.vhdEntity: gtp_bitslide

rst_done_n

rst_i

clk_ref, phy_ref_clk = TXOUTCLK (62.5 MHz)Clk_sys = used for synchronizing reset signals => must run always! (62.5 MHz)

BUFG

FPGA_CLK_P/N

BUFG

tx_out_clk_o

clk_dmtd(62.5x MHz)

62.5 MHz

PLL25DAC1_SYNC_N

PLL25DAC_DINPLL25DAC_SCLK

PLL25DAC2_SYNC_N

White Rabbit for Kintex7Slave Clock distribution



Tried first shot for White Rabbit on Kintex7Solved issue with synchronous reset (reset

switched off the clock )Only found this through simulating the

design!!KC705_top should do exactly the same as

SPEC_top:

First shot KC705_top

7

KC705_TOP hangs!Why?ÞSherlock Holmes(Software/Hardware)



1. Switch Routing Table (software?) needs to be adjusted.2. PTP timestamps t1 for all DOMs are equal and reside in outgoing port (needs firmware- or

software-change or both)3. MAC Control-Level multicast MAC addresses; such as “pause frames” for flow control

need to be handled correctly. Example:◦ “DOM-B” request “Pause” = Okay (request over point to point link), but…◦ “Shore station” request “Pause” may be problematic (request over broadcast link, all ports are

stalled). Address single DOM?

4. Other surprises?

Shore Station Broadcast brainstorm

DOMA

DOMB

DOMC

DOMD

BufferBufferBufferBuffer

Port-2SFP

Broadcast

OpticalNetwork

Start

Tx

t4 Stop1

ReferenceClock

PTP

Time Stamp

t1

Time Stamp

t4

t4 Stop2Time

Stamp t4

t4 Stop3Time

Stamp t4

t4 Stop4Time

Stamp t4Shore Station interface

8

Rxj: DDMTD

Rxj: DDMTD

Rxj: DDMTD

Rxj: DDMTD

Port-1SFP

Main Electrical Optical Cable

Port-3SFP

Port-4SFP

Port-5SFP



IEEE802.3 Clause 31◦ For example “Pause frames” for flow control

(IEEE802.3 Annex 31B) MAC Destination Address

◦ multicast 01-80-C2-00-00-01 or unicast? (see Annex31B 310.0.1)

MAC Source Address Length Type

◦ 88-08 for “this is a MAC Control Frame” MAC Control Opcode

◦ IEEE802.3 Annex31a => 00-01 for “pause” MAC Control Parameter

◦ Pause Quanta (1 Quanta = 512 bit times)

Shore Station BroadcastMAC Control Level

9

may 29, 2013km3net, clbv2 meeting peter jansweijer mesfin gebyehu nikhef amsterdam electronics-...

Documents