The planned new system

The planned new systemTom HimelDec 1, 20101OutlineControls has multiple related projectsDecided to use mainly TCA architectureDescription of TCAAdvantages of TCAPros and Cons and our decision making processSummary

2Controls has multiple related projectsUpgrade LLRF for 1 LCLS RF stationFunded for $1.8M from AIPRequirements mostly done, design startedUpgrade 10 sectors (80 RF stations) of LLRF (could be 21-30 or 11-20)Will be proposed when above prototype is finished at cost of ~$25k per station = $2MDevelop and industrialize the new TCA standard and use it to instrument ILC 10 MW klystron - interlocks and feedbacksFunded for $2M from ILC about 60% doneStandard nearly signed, companies producing shelves and modules3Controls has multiple related projectsstation node projectUpgrade multiple systems near a single RF station: stripline BPM, toroid, vacuum, motion control?, temperatures? Profile monitors?Proposed $2.8M AIPThis project is immediate cause of this review.Upgrade sectors 11-20Will be proposed at cost of ~$10M when above prototype is doneAnother ~8M needed for related non-controls upgrades: SSSB, magnet PS, modulator PLCBuild LCLS-IIWant to coordinate these projects to take advantage of synergies (sharing of module types and space in crates)4Why upgrade the linac?If that isnt obvious to you after the last talk, we have invited the wrong reviewers.Want better reliability and maintainabilityWant components for which we can get repair parts Want mostly commercially available components5The architecture choiceWe have spent considerable time and many group meetings deciding on the architecture to use.Decided on TCAWill first describe TCA to youThen explain its advantagesThen share the pros and cons matrix that went into our decision making process

6Genealogy of TCAATCA (Advanced TeleCommunications Architecture) is a standard developed for the telecommunications industry. Emphasis was on high availability and high bandwidth.There are many commercial modules availableModules are physically large (~fastbus size)Connections to smaller daughter boards are part of the standard.7Genealogy of TCAThese daughter boards are called AMC (Advanced Mezzanine Cards).Several can be mounted on an ATCA carrier card.Often the carrier card must be customized for the particular AMCs used to route in the necessary I/O from the cables that go to the RTM (Rear Transition Module) attached to the ATCA card.Some small projects can be done with ONLY AMC cards.8Genealogy of TCAThis led to first the TCA standard and then to the TCA for physics standard. The physics standard is twice the size of the minimum sized TCA (AMC) card and has a connector for an RTM. It is backplane compatible with a standard TCA card and simply defines the use of some spare lines on the backplane.It is the TCA for physics we plan to use and it will hereafter be referred to simply as TCA.The TCA standard was developed as an international industry/lab committee under the auspices of PICMG.The standard is virtually complete, but final signatures are still awaited.A small but growing number of commercial products are available.9TCA featuresIPMI: Standard out-of-band network to monitor temperatures, fan speeds, voltages of both crates and modules. Allows remote control of power to individual modules. Standard software available to implement all of this.Redundant hot-swappable fans allowing this most commonly failing component to be replaced without program interruptionAbility to have redundant power supplies and network hubsTiming distribution provided on backplane10TCA featuresTruly hot swappable modules and RTMs.Allows bad modules to be swapped without added degradation of the control system.This in turn allows more modules to be in a crate and hence fewer crates without degrading system availability.Split between AMC and RTM allows an AMC module to be used for several purposes by having different relatively simple RTM cards. E.g put the ADC on the AMC module and have RTM cards with different signal shaping for BPMs and toroids.Uses point-to-point communications instead of busses. Allows for high bandwidth and avoids subtle bus problems where a problem in one module causes problems in another.Low noise environment suitable for analog electronics.Solid, well tested mechanical and connector designs11TCA backplane12

AMC & RTM Modules TCA.413RTMAMCPowerSystem MgmtKeyingStandard AMC Connector and BackplaneUser I/OIndustry Prototypes:6-Payload Shelf14

RTMRTMDevelopment Shelf 6-Slot

Physics Backplane

Non-Redundant MCH, Fans, Power Module TCA.4 Development Platform15

SLAC Linac controls upgrade

6-Slot Prototype Shelf w/ MCH, Processor, Interim Timing System, power module, built-in fans

PMC Event Receiver (EVR) on double TCA Adapter

Shelf non-redundant

All rear I/O access

12-Payload Shelf16

Full TCA.4 Compatibility

Fully redundant MCH, power, fans

Courtesy K. Rehlich DESYDESY XFEL will use TCA in the tunnel17Prototype for XFEL. We have one.18

+ 2 16 bit DACSSlow I/O can be done with IP cards19

There is also a PMC carrier that we presently use for our timing cardTCA summaryScaleable modern architectureFrom 5 slot TCA full mesh ATCAGbit serial communication linksHigh speed and no single point of failureStandard PCIe, Ethernet communicationPCIe and Ethernet is part of Operating SystemRedundant system optionUp to 99.999% availabilityWell defined managementA must for large systems and for high availabilityHot-swapSafe against hardware damage and software crashes

20

Our Architecture decisionWe briefly looked at many standardsCarefully compared network attached devices (rack mounted chassis with Ethernet ports), VME and TCA.We expect to end up with a mixed system, so really deciding what standard to use for new and improved things.21Why not simply clone what we just did for LCLS-ISome parts not done at all (linac LLRF)Unhappy with other parts (stripline BPM) more laterWe likely will clone some of the parts like PLC system for vacuum and perhaps Beckhoff for temperatures and misc I/O22LCLS BPM rack23

FrontRearLCLS BPM rear close-up24

LI20 LCLS network rack25

BPM chassisEach has:4 signal cables (unavoidable)A trigger at beam time cableA calibration trigger cableAn ethernet port for channel accessAn ethernet port used to pass raw data at 120 Hz to a VME IOC for processing as the internal CPU is too slowA serial connection to a terminal server to allow viewing of the IOC consoleA power cable to an ethernet controlled power strip so power can be cycled to perform a remote reset.26BPM chassisThis was a design kludged together from available parts in 4 months when originally planned design for LCLS failed.Was then propagated to 10 linac sectors as didnt have time to do a proper redesign and wanted its improved analog performance.It works! Physicists are quite happy. But REALLY dont want to propagate this again! Needs a design using a crate e.g. TCA.27BPM in TCAEach module has:4 signal cables (unavoidable)A trigger at beam time cableA calibration trigger cableAn ethernet port for channel accessAn ethernet port used to pass raw data at 120 Hz to a VME IOC for processing as the internal CPU is too slowA serial connection to a terminal server to allow viewing of the IOC consoleA power cable to an ethernet controlled power strip so power can be cycled to perform a remote reset.28On backplaneOn backplanePCIe on backplane to CPUPCIe on backplane to CPUOnly CPU has oneIPMI handles thisVME situationVME is almost 30 years old: our system should operate for another 20-30 years.Number of new developments is decreasing, sales are still constantBus technology has speed limitationsWide busses create a lot of noise in analog channelsNo standard management on crate levelNo management on module levelSo far no extension bus survivedOne damaged bus line stops a whole crateAddress and interrupt misconfigurations are hard to find

29Reasons for choosing TCAUse an industry standard to share with othersRedundant fans and hot swappable fans and modules allows for troubleshooting and maintenance during user runs and improving reliabilityCable plant reduction compared to network attached devicesSystems can share crates with minimal impact30Reasons for choosing TCAFirmware can be remotely loaded (presently we bring each module to the lab for this)Standard system to monitor temperatures and voltagesA new standard rather than one nearing retirementModular so pieces of it can be upgradedAllows use of new technology that allows us to challenge and keep good engineers.31Decision spreadsheetThe presentation so far has been one-sided, listing the advantages of TCA and no disadvantages.We were much more balanced in our decision making process.The spreadsheet at the same site as this talk contains the detailed pros and cons list that was a key part of our decision making process. https://slacspace.slac.stanford.edu/sites/reviews/ad/linaccontrol_dec_2010/Pages/default.aspx 32A few design detailsHaving made the major architectural decision, there are still many design decisions to be made.We have a proposed AIP project which is mostly R&D where most of those decision will be made.This review doesnt really have time to go into design details anyway, but Ill list some of the next level decisions we have made.33A few design detailsWe will have 1 TCA shelf near each klystron that will have the LLRF, 0-3 BPMs and whatever else needs handling that is nearby.It will be a 12 slot shelf.There will be an air to water heat exchanger in the rack with air that circulates through it and the crate.We will reuse the current racks but bring them up to todays standards34A few design detailsWe are working with Struck and Vadatech to modify existing ADC designs they have to meet our requirements for LLRF and BPMWe expect all our AMCs to be commercial but we will design a variety of RTMsThere will be some rack mounted chassis to handle large cable plants, high power or high voltage.We have a list of all our systems and how we plan to handle them. Some plans are tentative depending on availability of resources.35SummaryThe linac control system clearly needs upgradingIt would be missed opportunity to not do it during FACET era with runs of only 4 months per yearTCA is the best choice for the architectureWe should get started as there is a lot of work to do.If we go fast enough, most of the LCLS-II upgrade can also be done with TCA36