ongoing developments for the atlas lar subdetector readout

Ongoing Developments for the ATLAS LAr Subdetector Readout

Possible replacement of VME for the upgrades of ATLAS

12 July 2012Guy Perrot on behalf of the LAr upgrade collaboration

• Context• US Developments (BNL/Stony Brook & U. of Arizona)

– ATCA Development Blade– AMC ROD Injector

• European Developments (Dresden & LAPP)– 10 GbE on Fabric interface – IPM Controller & ATCA Test Blade– ROD Evaluator

• LAr Today’s Path for 2018• Answers to your questions• Summary

Outline

12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 2

Current & Future Architectures


Currentsystem

1600 FEBs192 RODs~120 TBBs

Phase I 2018Digital Trigger

320 scells/LTDB

~120 LTDBs

~ 30 ATCA DPS Blades

~24 Tbps IN~24 Tbps OUT

Up to 1.5 TbpsMonitoring

Phase II 2021FEB & ROD Change

128 cells/FEB~1600 FEBs

~100 ATCA ROD blades ?

150 Tbps IN0.4 Tbps OUT

• BNL ATCA Development Blade– No IPM Controller (was

supposed to be DESY Firmware but lack of time)

– Shortcuts to get power on.– Room for 2 Mezzanines (not

AMC)• Used by Dresden to test 10

GbE on Fabric

US developments (1)


Base Interface (Gigabit Ethernet)Fabric Interface (10Gig Ethernet)

Zone 3 (to transition module) LVL1 communication

Update ChannelGlobal Clocks

-48v Power Input and IPMI interface

12 fiber inputs SNAP12 format

DDR2SODIMM Memory

Virtex 5 FXT FPGA

Power Conversion

Serial Debug Port

• BNL DPS/ROD sPU AMC– In design phase– DESY/CPPM/CERM MMC (ATMEL

Processor)– Virtex7 FPGA– 48 10Gbps links

• U. of Arizona ROD Injector AMC – 2 x 12 channels @ 6.25 Gbps (Front)– Stratix EP4SGX230KF40– 12 channels @ 5 Gbps (GbE or PCIe) on

Fabric – MMC implemented using an ARM Processor– Future AMCs to use CPPM/CERN MMC– AMC only used in uTCA Shelf

US developments (2)


• Technische Universitat Dresden: 10 GbE through ATCA Fabric Interface– ATCA shelf: 20port 10GbE Switch, BNL SubROD (Virtex 5 FPGA)– Server: Dual 4-Core Xeon, 4x Dual Port Myricom 10GbE NIC– Link: optical between shelf and server

• Results:– 80 byte UDP packets: ~400Mbps only! with packet losses – 1500 byte packets: ~4.4Gbps with packet losses – 9000 byte jumbo packets: ~10Gbps without packet loss

European developments (TUD)


• IPM Controller– Communication with Shelf manager through

IPMBus A & B– Hot Swap, Power management etc..

• ATCA blade manager– Communication via Ethernet (front panel or

ATCA Base Interface)– User functions

• Firmware Upgrade • ATCA blade monitoring & configuration• User stuff….

• Software libraries available

• FMC (FPGA Mezzanine Card) compliant– up to 160 customizable links( including 74

differential links) – Low cost

• Features:– ARM cortex M3 processor– Xilinx Spartan 6 for highly configurable user IO– Ethernet / USB / JTAG interfaces

European Developments (LAPP) ATCA Controller Mezzanine


FPGASpartan 6

µCLM3S9B92

bus

FMCMezzanine Power Supplies

connector

EEPROM

IOUSB

JTAGJTAG JTAG

I2C IPMBusA&B

Ethernet

69mm

76.5mm

µC FPGA IO

Possible VME Replacement for the ATLAS upgrades G.Perrot 8

• ATCA Controller Mezzanine tests– IO connections (JTAG boundary scan

tests)– IPMI management with the Shelf

manager– Ethernet communication through

ATCA Base Interface • ROD Evaluator tests

– Check Blade configuration with the ATCA Controller Mezzanine (Firmware upgrade, Configuration upload etc..)

– Test ATCA compliant power supplies– Check FPGA Design

(communications with DDR3, Flash, configuration with Flash)

European Developments (LAPP)ATCA Test Blade

12/07/2012

• Smaller size: DDR3 VLP Mini-DIMM• Based on 2 µC ARM cortex M4 from ST

Microelectronics• IPMC features

– IPMB_0 with on board buffers, Hardware address detection– Hot Swap management with ATCA Leds and front panel

switch– Management of up to 8 AMC + RTM– On board Event LOG– FRU & SDR via I2C– Access to ATCA board sensors via I2C– Configurable User Signals for Payload management

• JTAG Master– JTAG master via Ethernet (ATCA board debugging,

firmware upgrade)

• Custom interface– Possibility to have a custom interface between the

Mezzanine and the ATCA board for custom functionalities

• Other– USB and UART interfaces (debugging etc..).

European Developments (LAPP) ATCA Controller Mezzanine Version 2

12/07/2012 Possible VME Replacement for the ATLAS upgrades G.Perrot 982.127 mm

18.30 mm

• ATCA Blade with 4 AMC slots.• MMC mezzanine • Designed to test all the

functionalities of the ATCA controller mezzanine.

• Could be used as “Reference Design” for users.

European Developments (LAPP)ATCA Test Board Version 2


Mezzanine

DC/DC PowerMgt

AMC 1

AMC 4

PWRMgt

I2CHAEtc..

RJ45switch ETh

BaseInterface

Led, switch

T°

MMC

FPGA

User IORSVD FRU

PWRMgt

• Fully compliant ATCA blade.• To evaluate high speed and high density

optical & electrical transmissions.• To evaluate high power DSP cells from

FPGA• Board management trough Ethernet

(Firmware upgrade, DSP configuration…)

- 48 optical Rx links @8.5Gbit/s- 48 optical Tx links @8.5Gbit/s- 8 Electrical Rx/Tx links @8.5Gbit/s

16 Electrical Rx/Tx LVDS links @ 1.6 Gbit/s between FPGAs

- Connection to Fabric & Base interface

European Developments (LAPP) ROD Evaluator


FPGA

CPLDFLASH

DDR3

SNAP12 Rx(Carrier)

SNAP12 Tx(Carrier)

ATCA IPMC(FMC carrier)

ATCA DC/DC

DC/DC :3.3V,2.5V,1.8V,1.5V

DC/DC : 0.9V

Phase I Architecture


Phase I 2018Digital Trigger

320 scells/LTDB

~120 LTDBs

~ 30 ATCA DPS Blades (LDPB)

~24 Tbps IN~24 Tbps OUT

Up to 1.5 TbpsMonitoring

LDPB Data Path Block Diagram


Data Processing

Memory

MemoryData Processing

DATA 40 MHz RESULTS 40 MHzFE

LTDBL1 CaloFEX

DATA + RESULTS 400 KHz MonitoringData

Collector

DATARESULTS 100 KHz

L0 400 KHzLatency 2.5-10 us

L1 100 KHzLatency 100s of us

LTDB (LAr Trigger Digitizer Board) : 118 ModulesLDPB (LAr Digital Processing Blade) : 29 ½ Blades (4 LTDB / LDPB)FE (Front End) & FEX (Feature Extractor)Monitoring Data Collector

820 Gbps

820 Gbps205 Gbps

24.2 Tbps

49 Gbps (8 Gbps Res. only)1.45 Tbps (240 Gbps)

2 Gbps60 Gbps

24.2 Tbps

MonitoringData

Collector

Back-End Block Diagram (In Work)


LTDB AMC

Data Processing & Memory

FEX

Switch10

GbE

SwitchGbE

AMCData

Processing & Memory

Switch10

GbE

SwitchGbE

Blade:LDPB

TTC Distribution

TTC

PM PC

MonitoringData

Collector

TTC LDPS

Partition(NEW!)

Shelf Manager

Busy

Scells Raw data

Results to FEXL1 Calo

TTC, Control, Monitoring

Raw data?, result on L0

Raw data?, result on L1

Control, Monitoring

Shelf

IPMC

Blade:Switch

Switch

GbE

TTC

Blade Ctrl.& Monitor.

• What is the expected usage of the backplane?– ATCA IPM control of blades.– Blade & FE LTDB configuration & monitoring via base interface.– Data collection via fabric interface.

• Do you use the fat pipes. If yes which protocol?– For the time being we are thinking 10 GbE.

• Would you agree for a control through Ethernet? – Yes, we plan it that way!

• Do you use synchronisation signals and common clocks? – No common clock for operation of blades.– TTC for clock transmission to FE and Evt signature at FE & BE.

• Do you plan to use redundancy and/or hot swapping– Hot swapping YES, redundancy NO.

• Do you plan full mesh or dual star backplane?– Full mesh, though we are using a dual star architecture, as it can do both.

Answers (1)


• Do you need a local processor?– For the time being we don’t think so.

• Which shelf manager do you plan to use?– The one that will come with the shelf (we have Pigeon Point).

• Which IPMC do you plan to use?– We are developing an IPMC and are planning to use it.

• Typical expected power consumption?– No idea yet, but probably quite a lot: 200 W/blade => 2.8 KW ???

• Which crate size to you plan to use (i.e. how many slots available)?– ATCA format with 14 slots + shelf manager.

• Do you plan implementation with AMC mezzanine and a mother board?– We are planning to have 4 AMCs on an ATCA carrier blade.

Answers (2)


• How sure are you that you need ATCA. Could you envisage µTCA?– We went straight for ATCA as we have a very big number of channels.– A study would be needed to understand data throughput on the uTCA backplane compared to

ATCA, nb of slots needed, xTCA switches and external switches. We haven’t done that!

• Do you need zero, one or more ATCA switch blades in your ATCA shelf? Which kind of switch?

– To get the bandwidth for data collection we need 2 switch blades (10GbE) 12 in (fabric) / 12 out (front).

– GbE for the control of blades via base interface.– TTC distribution ?????????

• Which MMC do you want to use on your AMC or in uTCA?– We are planning to use the DESY/CPPM/CERN design.

Answers (3)


• A basic BE system will be installed in 2013-14 using our ROD Evaluator (2 blades).

• The Digital Trigger system in 2018 (Phase I) will allow us to develop, test and master all the technologies needed for the Phase II developments (replacement of FEBs and RODs)

• It will also allow us to understand what is the best compromise in term of density, power consumption, data collection.

Summary


ongoing developments for the atlas lar subdetector readout

Documents

atlas upgrades g

gbps gbe

atca rod blades

fabricus developments

atca dps blades

upgrades of atlas12

gbps links

gbe niclink