options for next generation digital acquisition...
TRANSCRIPT
OPTIONS FOR NEXT GENERATION DIGITAL ACQUISITION SYSTEMS
A. Boccardi, CERN, Geneva, Switzerland∗
Abstract
Digital acquisition system designers have an always in-creasing number of options in terms of bus standards anddigital signal processing hardware among which to choose.This allows for high flexibility but also opens the door to aproliferation of different architectures, potentially limitingthe reusability and the design synergies among the variousinstrumentation groups.
This contribution illustrates the design trends in someof the major institutes around the world with design ex-amples including VME [1], PCI [2] and TCA [3] basedmodular systems using AMC [3] and/or FMC [4] mezza-nines. Some examples of FPGA design practices aimed atincreasing reusability of code will be mentioned togetherwith some of the tools already available to designers to im-prove the information exchange and collaboration, like theOpen Hardware Repository [5] project.
THE GENERAL ARCHITECTURE OF ADIGITAL ACQUISITION SYSTEM
The general architecture of a Digital Acquisitionboard/system (DAQ) (see Fig. 1 ), comprises the interfaceto the instrument Front-End (FE), a processing unit, usuallyimplemented as a Field Programmable Gate Array (FPGA),some memory, a clock recovery and distribution systemand the interface(s) to the system bus(es).
Figure 1: General architecture of a DAQ board/system.
It should be noted that there could be more than one sys-tem bus. It is indeed common in case of need for high datathroughput to have a dedicated link on each board and leavethe shared bus to serve as slow control and for configura-tion.
The Concept of Carrier and Mezzanine
For a given system bus the main difference betweenDAQs remains the FE interface and a modular approach
can minimize the variations even in this respect. Thelogic required to implement such interfaces could be imple-mented as a separate module, a mezzanine, to be pluggedon the main board, the carrier.
A modular approach reduces the risks and costs involvedin new designs as the complexity of the PCB to be devel-oped is reduced to the minimum. The software develop-ment is also speeded up by a modular approach as the sys-tem facilities’ interfaces, like the clock management unit,remain the same from system to system. Moreover the useof a common carrier simplifies the spare management in aninstrumentation group as several instruments can share themost complex and expensive part of their DAQs.
The use of standards for the mezzanines also allows us-ing modules developed in other institutes or for differentinstruments whenever the requirements are similar, and thiseven if the system bus chosen for the DAQs is not the same.
There are 2 emerging standard for the mezzanines: theFPGA Mezzanine Card (FMC or VITA-57.x) and the Ad-vanced Mezzanine Card (AMC or PICMG-AMC.x). Whilethe first one results in very simple designs which assume allthe processing and complex logic to be on the carrier, thesecond one requires the presence of a relatively complexFPGA and of some system management logic on the mez-zanine itself. The AMC mezzanine will be detailed in thesection dedicated to the options for carriers, as it is the ba-sis for the uTCA standard and is mostly used in the highenergy physics community in this form.
THE FPGA MEZZANINE CARD.
The FPGA Mezzanine Card (FMC) standard was de-fined to take full advantage of the great flexibility offeredby modern FPGAs’ IO blocks. The mezzanine is sup-posed to be directly connected to an FPGA and the func-tion, direction and electrical standard of the pins is de-fined at configuration time. The standard foresees that themezzanine plugged on the carrier could be identified us-ing IPMI [6] (Intelligent Platform Management Interface)commands over a I2C (Inter-Integrated Circuit) bus. Thiswould avoid the risk of loading incorrect firmware resultingin possible bus conflicts. The use of IPMI is a suggestionbut not mandatory in VITA-57.
The FMC comes in several flavors. It is possible to havemezzanine with:
• Low Pin Count (LPC) or High Pin Count (HPC) con-nectors.
• Single width and double width (a double width mez-zanine can have one or two connectors as in Fig. 2)
Proceedings of DIPAC2011, Hamburg, Germany TUOB01
09 Others 289 Cop
yrig
htc ○
2011
byth
ere
spec
tive
auth
ors—
ccC
reat
ive
Com
mon
sAtt
ribu
tion
3.0
(CC
BY
3.0)
Figure 2: Single and double width FMCs.
The difference between the high and low pin count con-nector is in the number of pins available, 160 in the LPCand 400 on the HPC, but mechanically they are the thesame, allowing a LPC mezzanine to work on a HPC car-rier. Designers using LPC have up to 34 differential (68single ended) user defined IO and 1 multi-gigabit data lane(2 differential pairs) available. The HPC version of the con-nector offers more user defined IO and 9 more multi-gigabitdata pairs. The mezzanine receives 3.3V and 12.0V powersupplies.
The FMC standard is rather flexible but offers limitedreal estate. A single width mezzanine is about 69mm by76mm, but this is usually acceptable as it is not foreseento have any complex logic on it. Another downside of thestandard is the lack of dedicated carrier to mezzanine clocklines. The few available will most probably be removed inthe next revision of the standard.
General purpose IO, ADC, DAC as well as TDC (time todigital converter) mezzanines are already available in FMCformat..
OPTIONS FOR CARRIERS
Most of the DAQs in the high energy physics communityhave been based up to now on VME or PCI, but now a fewother standards are being evaluated including xTCA (Ad-vanced Telecommunications Computing Architecture andits derivatives) and VXS (VME switched serial).
The reason to look for new standard is not in the actualbandwidth of the old busses. Indeed in case of need for highthroughput, having dedicated links on each DAQ board re-mains the best option to avoid the bottleneck at the concen-trator/switch. Instead, it is driven by in the need to havehigh bandwidth communication between boards. A typicalexample of DAQs with such needs are the trigger systemsof the experiments, where there is the need to recombinewith low latency the information collected by several DAQboards.
VME
VME, ANSI/VITA 1-1994, is very common standard inthe high energy physics community. It was first developedin the late 1970s by Motorola. All the boards share thesame bus, on which there is only one master at a time. TheVME bus supports several transfer protocols that have beenadded later to the standard while keeping the same pinout.Boards supporting different transfer cycles can share thesame crate. The maximum transfer speed on the VMEbusis achievable with a synchronous cycle called 2eSST and isof 320MB/s.
VME boards can have different form factors, but themost common is the 6U, on which it is possible to haveup to 3 FMC mezzanine on the front panel.
The standard allows to have Rear Transition Modules(RTM) connected to the carrier using the user defined pinsof the bottom connector (P2). The form factor of thoseRTMs is not defined in the standard and mostly depend onthe crate geometry.
One of the main advantages of this standard is its matu-rity: over the years many modules have been developed andare now available as COTS (Commercial Off-The- Shelf).
Figure 3: The VME FMC carrier developed at CERN.
Figure 3 shows the VME FMC Carrier (VFC), a boarddeveloped at CERN as part of a collaboration between thebeam instrumentation and the controls groups. The VFCis a carrier for 2 FMC mezzanines and an RTM module.It has 2 small form-factor pluggable (SFP) transceivers onthe front panel for clock/timing recovery and data transmis-sion. Those modules are connected to the Gbit lines of theSystem FPGA (SFPGA).
All the interfaces, monitoring elements and configura-tion devices are managed by the SFPGA, while the Appli-cation FPGA has direct access to the FMCs and the RTMas well as to 2 SRAMs.
PCI
The PCI (Peripheral Component Interconnect) is a stan-dard of PCI-SIG [2] (PCI Special Interest Group). PCI wasoriginally developed at Intel in the 1990s. It went troughseveral revision and changes in the years. Its latest form isa serial link: PCIe (PCI Express).
TUOB01 Proceedings of DIPAC2011, Hamburg, Germany
290Cop
yrig
htc ○
2011
byth
ere
spec
tive
auth
ors—
ccC
reat
ive
Com
mon
sAtt
ribu
tion
3.0
(CC
BY
3.0)
09 Others
PCIe is not a bus but a point to point link. The PCIe1.0 can achieve a bandwidth of up to 250MB/s per lane,each lane consisting of two LVDS (Low Voltage Differen-tial Swing) pairs, and each board can have up to 32 lanes.
PCIe is a very common standard for desktop computingand therefore many products can be found on the marketas COTS in this format. PCIe boards can be plugged instandard and crate computers.
Figure 4: The PCIe FMC carrier developed at CERN.
Each PCIe board in a standard form factor can have upto 1 FMC available on the front panel.
Figure 4 is a picture of the PCIe FMC Carrier (PFC) de-veloped at CERN in parallel to the VFC. It is a carrier for aFMC with an SFP for timing and data transfer on the frontpanel. It has 4 eSATA connectors on the back to implementcustom board to board communication protocols.
The 2 carriers have been designed having in mind toshare as much as possible in terms of architecture and com-ponents. This was done in order to give to the designers thepossibility to have the seamless possible port of their appli-cation firmware from one carrier to the other.
VXS
VXS, VITA-41, is one of the answers of VITA (VMEbusInternational Trade Association) to the need of fast inter-communication between boards in the same crate.
The standard redefines the VME middle connector (P0)and assigns to it 16 differential pairs with a bandwidthof 10GHz. The protocol of those lines is defined in sub-standards with PCIe and Ethernet being the most commonones in commercial components. The standard foresees asystem management based on IPMI, but this is just a rec-ommendation.
A very strong point of the VXS standard is that old VMEmodules which do not use the P0, the most common optionin COTS, can be plugged in VXS crates and work properly.
The network topology of the backplane is not definedin the standard and crates implementing it as star (Fig. 5)and mesh (Fig. 6) are both possible. The star topology,requiring a switch board, is the most common as the meshrequires too high densities of connections.
The standard has high potentiality but is not yet verypopular in the high energy physics community. An FMCcarrier for VXS crates is being developed by the radio fre-quency group of CERN, but is just one of a few designs.
Figure 5: In a star configuration all the boards are con-nected to a switch board, usually plugged in the middleslot of the crate, via one or more lanes (couple of differen-tial pairs). The switch than connects the boards that needto communicate between them in a dynamic way.
Figure 6: In a mesh configuration the boards are directlyconnected to each other. This reduces the latency but makesthe backplane more complex. In some cases the boardscould be connected in small groups to simplify the topol-ogy.
xTCA
xTCA is the collective name for ATCA (AdvancedTelecommunication Computing Architecture), uTCA (Mi-cro TCA) and MTCA.4 (Micro TCA for physics).
ATCA, PICMG 3.x, is a standard developed by thetelecommunication industries for their specific needs: highbackplane bandwidth and very high availability. The firsttarget is reached thanks to the very high amount of Gbitlines each card could have: up to 200. The second with avery extensive use of system management based on IPMIfor hot swap support and redundancy. Each board coulduse up to 200W, but available only from a -48V power sup-ply, requiring DC/DC converters on board to obtain all therequired voltages. Like in VXS neither the protocol of thedifferential pairs nor the backplane topology is defined inthe standard. ATCA boards are in 8U format and can accept8U rear transition modules (RTM).
The AMC (Advanced Mezzanine Card) was developedto increase the ATCA availability with the introduction ofhot swappable mezzanines. The connector was specificallystudied to allow an easy insertion and extraction from the
Proceedings of DIPAC2011, Hamburg, Germany TUOB01
09 Others 291 Cop
yrig
htc ○
2011
byth
ere
spec
tive
auth
ors—
ccC
reat
ive
Com
mon
sAtt
ribu
tion
3.0
(CC
BY
3.0)
front of the crate without the need to unplug the carrier. Aswell as ATCA the AMC makes intensive use of the IPMIprotocol. The mezzanines can self declare to the carrier atplug in time and inform the carrier when they are being re-moved. In this way all the required hardware and softwaresteps to have a safe hot swap can be performed.
Figure 7: A uTCA board for a timing distribution sys-tem developed in a collaboration between the Universityof Stockholm and DESY.
An AMC mezzanine has 20 LVDS Gbit lanes (coupleof differential pairs). The protocol of those lanes is notspecified in the standard. Each AMC mezzanine is 180mmdeep, but there are 6 form factors:
• Single (74mm) and double (149mm) width• 13mm, 18mm or 28mm deep
The power in an AMC comes from a 12V supply. The al-lowed dissipation of a mezzanine depends on the form fac-tor and can be at most 80W.
A single width AMC is about twice the size of an FMC,but is required to have a reasonably powerful FPGA on itas well as a few DC/DC converters, so that the real estatefor the application specific logic is about the same.
The AMC mezzanines are also the basic element of theuTCA standard, in which the AMCs are directly pluggedinto a backplane. The standard doesn’t specify the topologyof the backplane, but the most common ones are the singleand double star. Each crate require one, or two in caseof double star topology, MCH (MicroTCA Controller andHub), see Fig. 8. The MCH works as system manager andalso as switch for the Gbit lanes between the boards. Userscan connect to the MCH via ethernet and have access to theboards in this way.
Figure 8: A uTCA controller and HUB (MCH)
The uTCA standard is being evaluated at CERN by thexTCA interest group [7]. The interest group has reported
some interoperability issues between boards from differentmanufacturers due to some interpretable part of the IPMIstandard.
Figure 9: A uTCA crate. In the 1st slot there is the powersupply, in the second the MCH and in the 3rd a crate PC.
The last standard that will be introduced in this paperis MTCA.4 (Micro TCA for physics). This standard is aderivate of uTCA that is still under ratification but has al-ready been chosen for the XFEL control system at DESY.
MTCA.4 tries to reduce the freedom in the uTCA def-initions to obtain a more standard product. In this opticit defines more strictly the connections assigning some assynchronization signals and strongly suggests to use PCIeas protocol for the Gbit lanes.
The MTCA.4 boards are defined as double width AMCand one more connector has been added on the top of theboard to be used with a RTM (rear transition module).
Figure 10: A MTCA .4 FMC carrier developed at DESY
An example of FMC carrier in MTCA.4 is shown inFig. 10. This board is built around an FPGA that is con-nected to the LPC FMC, the rear transition module, a128MB DDR2 memory and 4 SFPs accessible on the frontpanel. More details about the development ongoing atDESY for the XFEL control system are given in anotherpaper from the same conference [8]
TUOB01 Proceedings of DIPAC2011, Hamburg, Germany
292Cop
yrig
htc ○
2011
byth
ere
spec
tive
auth
ors—
ccC
reat
ive
Com
mon
sAtt
ribu
tion
3.0
(CC
BY
3.0)
09 Others
Table 1: Comparison Between the Various Standards
VME PCIe VXS uTCA MTCA.4
FMC slots1 3 1 3 1-2 2RTM YES NO YES NO YESSystem Bus speed up to 320MB/s - up to 320MB/s - -Gbit lanes 0 up to 32 8 20 20Power available 108W up to 75W 108W up to 80W 80WHot-Swap Partially implemented NO Partially implemented YES YES
LOOKING FOR REUSABILITY ANDSYNERGIES
It is not possible to point to one standard as the optimalone. Which one suits best the system being developed de-pends on the specific needs but sometimes also on historicalreasons: a group having invested in a specific technologywill tend to reuse the same to have a common platform forits various systems. This doesn’t mean that there cannotbe synergies between groups: as the architecture of a DAQis almost always the same it is possible that with minorchanges the same board could be used by several institutesor groups. Sharing designs and opening them to externalreview and specification changes requests to fit a wider au-dience is exactly the purpose of the Open Hardware Repos-itory (OHR) project, of which the VFC and PFC carrier arepart.
The idea of OHR is that all the designs should be keptopen and free to be used also by private companies. TheOHR repository of a project is updated at each step, fromspecifications to production. Each developer willing to par-ticipate in the design or review phases can read and sub-scribe to the associated mailing lists giving inputs. Therepository is also open to private companies willing to ac-cept the terms of the OHR contract: each design shall bekept open and all the information made available. A pri-vate company can under the OHR contract, being preparedby the CERN legal office, produce and resell the boards inthe repository.
Synergies and reusability can be improved not only atboard level but also in FPGA firmware development. Theuse of a common internal bus, like the WishBone fromOpenCores [9] as suggested in OHR, would allow to ex-change FPGA blocks between designers, especially if themezzanine standard used is the same.
1FMC mezzanines that could be placed on a carrier and be accessible on the front panel
ACKNOWLEDGEMENTS
Special thanks to Markus Joos from the xTCA interestgroup for all the information he provided me and the veryinteresting conversations.
REFERENCES
[1] VME International Trade Association:http://www.vita.com.
[2] PCI special interest group: http://www.pcisig.com.
[3] PCI Industrial Computer Manufacturers Group:http://www.picmg.org.
[4] VITA-57: http://www.vita.com/fmc.html.
[5] OHWR website: http://www.ohwr.org.
[6] IPMI on the Intel website:http://www.intel.com/design/servers/ipmi.
[7] XTCA interest group web site: https://twiki.cern.ch/twiki/bin/view/XTCA/WebHome
[8] P. Gessler et Al. , “Next Generation Electronics based onTCA for Beam-Diagnostics at FLASH and XFEL”, DIPAC2011, Hamburg, May 2011, TUOB03
[9] WishBone specifications: http://cdn.opencores.org/
downloads/wbspec_b4.pdf
Proceedings of DIPAC2011, Hamburg, Germany TUOB01
09 Others 293 Cop
yrig
htc ○
2011
byth
ere
spec
tive
auth
ors—
ccC
reat
ive
Com
mon
sAtt
ribu
tion
3.0
(CC
BY
3.0)