ATCA based LLRF system design review

3.12.2007 DESY

Control servers for ATCA based LLRF Control servers for ATCA based LLRF system system

Piotr Pucyk - DESY, Warsaw University of Technology

Jaroslaw Szewinski – Warsaw University of Technology

2ATCA based LLRF system design review

3.12.2007 DESY

Agenda

• Requirements• LLRF servers classification• ATCA computation power for LLRF servers• Servers topology in ATCA based LLRF system • Possible control systems• Development environment• Time schedule and summary

3ATCA based LLRF system design review

3.12.2007 DESY

Requirements

• What servers should do ?

– Should make possible remote access to the applications (update parameters, readout data, etc.)

– Should support hardware in computation with given timing constraints (recalculate tables between pules, etc.)

– Servers should guarantee reliable data transfers to the history storage

– Should provide, for local applications, interface and interaction with external systems (other servers, foreign hardware, etc.)

– Should provide management, control and synchronization of multiple subsystems (procedures, FSMs, automation)

4ATCA based LLRF system design review

3.12.2007 DESY

What control software do we need?

• Finite state machines for automation• Front-end servers for hardware

maintenance, configuration and diagnostics

• Front-end servers for controller and low level applications (execution nodes for state machines)

• Middle layer servers for high level applications

• DAQ servers or interfaces to DAQ• GUI panels, interfaces to Matlab, C,

etc.

Diagnostics maintenance

Diagnostics maintenance

End-node for FSM, controller server

End-node for FSM, controller server

High level apps, FSM

High level apps, FSM

DAQ

GUI, external apps

GUI, external apps

5ATCA based LLRF system design review

3.12.2007 DESY

Different CPUs in ATCA based LLRF systemEmbedded processors

AMC module processors

ATCA blades mainframes

Computation power

• Huge processing power

• DAQ, storage

• Post processing

• GbE

+ server class processing power

- No PCIe until now

+ serious processing power

- uses AMC slot on carrier

+ PCIe, GbE

- Low processing power

- FPGA resources

+ close to the hardware

6ATCA based LLRF system design review

3.12.2007 DESY

FPGA

Possible topology of LLRF servers

AMC CPU

IO AMC (FPGA)

FPGA embedded

CPU

ATCA Carrier board

CPU

ATCA CPU blade

CPU

Mainframe blades

CPU CPU CPU CPU

CPU CPU CPU CPU

CPU CPU CPU CPU

CPU CPU CPU CPU

PCIe

GbE

GbE

Front-end middle layerSimple FSM

Front-end middle layerSimple FSM

Front-end

Front-end

Middle layer FSMFront-end

Middle layer FSMFront-end

FSM, DAQ

FSM, DAQ

7ATCA based LLRF system design review

3.12.2007 DESY

Possible control systems

• DOOCS

– Used for the FLASH control

• EPICS

– Widely used, low hardware requirements

• TINE

– Heterogeneous/portable, runs on different platforms (DOS, Windows, various UNIXes, VxWorks, etc), can use different network protocols (TCP/IP, IPX/SPX, etc.)

• Dedicated software

– custom servers, often embedded

• Hybrid configurations

– Standard control system server (on ATCA CPU board) as gateway for small, lightweight custom embedded servers in different ATCA facilities (AMC, Carrier, etc.)

8ATCA based LLRF system design review

3.12.2007 DESY

What we can reuse, what we have to develop

• Reuse

– Old communication scheme (one software interface – many hardware drivers), memory description map

– Tools for debugging and configuration

– Some existing server’s source code

• New development

– Diagnostics, maintenance interfaces (inc. IPMI, crate management)

– FSM framework for high level apps

– New communication channels

9ATCA based LLRF system design review

3.12.2007 DESY

Development environment & tools

• Platforms - preferred free, open systems like Linux or FreeBSD. Linux is widely used on various systems (from embedded to mainframe and cluster – can run in different facilities of ATCA). It has good hardware support - excellent platform for development.

• Usage of real time (commercial) systems (like VxWorks) – only where Linux can not be used.

• Technologies - 'keep it simple when possible ' - preferred well known, widely portable technologies like:

– C/C++

– BSD sockets + TCP/IP

– Pthreads

– etc.

10ATCA based LLRF system design review

3.12.2007 DESY

Schedule, manpower

• Before we start:– Development environment, control system, communication

libraries, FSM framework.• Development schedule strongly depends on other tasks

– Configuration and maintenance servers (when carrier board and at least one AMC is debugged and ready for firmware implementation - 3-5.2008 ?)

– Finite state machines and procedures – starting from 1.2008 ? – Controller server, and low level applications interface servers

(parallel to controller development and low level applications)• Minimum Manpower

– 1 fulltime programmer for front-ends, 1 fulltime for FSM and high level apps, 1-2 students for help

– A lot of support from MCS group !!!

11ATCA based LLRF system design review

3.12.2007 DESY

Backup slides

12ATCA based LLRF system design review

3.12.2007 DESY

Development & Design guidelines

• Usage open-source, free tools when possible • Projects organized by Makefiles, to enable automatic build of all sub-

components, including cross-builds for different targets (CPUs on AMCs, DSPs, PPCs in FPGA, ATCA CPU boards, etc.)

• Codes organized by version control tools (CVS, SVN)• Codes documented by tools like Doxygen• Common coding conventions & policy• Separated mechanism (platform dependent) and policy (platform

independent)• Avoid platform depended, hardly portable solutions• Avoid fancy external tools/libraries/technologies which development

or support my be stopped one day.

13ATCA based LLRF system design review

3.12.2007 DESY

Old System Scheme

FPGA

LPT

VME

???

Memory map descriptionFile / parser

TCP Server

Hardware Channels EngineUser

Applications

Inte

rnal In

terfa

ce

Eth

OtherApplication

s

TCP/IP

14ATCA based LLRF system design review

3.12.2007 DESY

Linux on PowerPC

User applications access hardware through the driver

Kernel mode driver has access to the FPGA

FPGA has defined hardware interface

Virtex II PRO

Linuxon PPC

Hardware

FPGA II Core

Client onremote

machine

Kernelmode

Internal Interface bus

driver

Usermode

TCPserver

TC

P/IP

15ATCA based LLRF system design review

3.12.2007 DESY

DOOCS patterns

http://flash.desy.de/sites/site_vuvfel/content/e403/e1644/e1136/e1137/infoboxContent1796/tesla2006-10.pdf

16ATCA based LLRF system design review

3.12.2007 DESY

DOOCS patterns (2)