Controlling embedded hardware

Erlang in global radio astronomy

Harro Verkouter/Joint Institute for VLBI in Europe

Joint Institute for VLBI in Europe

VeryLongBaselineInterferometry

VeryLongBaselineInterferometry

Ever larger telescopes

15cm

5m 100m

Electromagnetic radiation

(m) D (m) /D (rad)

smallest optical 10-7 10-1 10-6

biggest radio 10-1 100 10-3

wavelength diameter pixelsize

A neat trick

100m

A neat trick

100m

20m

correlator

10,000km

A different ball game!

(m) D (m) /D (rad)

largest optical (IR) 10-6 10 10-6

VLBI (radio) 10-1 107 10-8

Science

Science

Recording the data

Record up to 4096 Mbps (512 Mbyte/s)

Number of combinations to be computed as function of n

~n2

16 telescopesat

1024 Mbps

UniBoard

QuickTime™ and a decompressor

are needed to see this picture.

4 frontnode

4 backnode

1Gbit/s ethernet

16x 10Gbps

16x 10Gbps

>20 Gbps/link

Erlang?!

• typical languages used– 80% C++– 15% Python– PHP, Fortran, perl

• database– MySQL

• needed to convince management– pilot projects!

Decoding data files

• VLBI has framed data format(s)

QuickTime™ and a decompressor

are needed to see this picture.

• Look for ‘syncword’ – different flavours, same syncword, different framesize

• Other format uses different syncword

(not quite) surprising little code

• 212 lines of code– first dabblings ...

• about a week’s work• recognizes + decodes 3 formats• decodes the BCD timestamps• supports distributed system

– reading file on one machine– decoding on other machine

• management duely impressed– got permission to do a larger pilot!

Soft real time monitoring website

• webcams at stations– webpage with http links = #FAIL– unreliable links

• really high latency (China, Chile)• don’t want broken images

– low bandwidth• cache in Europe: multiple users eat all bandwidth

• parse telescope logfiles– immediate antenna calibration info– generate timeseries plots as observation progresses

• stored in MySQL

Soft real time monitoring website

• webcam URLs + status stored in MySQL– supervisor starts webcam monitors

• one gen_server for each webcam– loads info from MySQL– uses http:request/1– strict checking on return value

• only HTTP 200 status replies• content_length header set?• explode if problem

– use a port to ImageMagick to scale image– write binary JPEG data into mnesia

Soft real time monitoring website

• monitor all participating telescopes• telescope log file (ASCII text)

– retrieved via python + ssh – read into Erlang using a port– low data rate after start up

• parses lines– update MySQL

• generate plots using gnuplot – for each telescope– query MySQL every so many minutes– using a port– write .png binary data to mnesia

Soft real time monitoring website

• patch it all together using webtool• cache complete webpage in mnesia

– too old? generate new one

• serve .png and .jpg data from mnesia– image URL points back to webtool

• extremely simple to cache + serve data– brilliant idea: mapping URL to erlang fn call!

• in total used ~3000 lines of code– webcam, logfile monitoring; plotting; serving web pages– excellent test case for OTP and distributed system

• got the green light for the real thing!

UniBoardEthernetSwitch

FPGA

Nios2CPU

1Gbit PHY

FPGA

Nios2CPU

1Gbit PHY

FPGA

Nios2CPU

1Gbit PHY

FPGA

Nios2CPU

1Gbit PHY

Control Computerwindows/unix

EthernetSwitch

VHDLHardwareSoftware

FPGA

1Gbit PHY

VHDLHardwareSoftware

Polyphase Filterbank

registers

Delay Module

registers

Nios2CPU

registers

registers

VHDLHardwareSoftware

Control Computerwindows/unix

Nios2CPU

Client

Server

UDP/IPv4 over ethernet

Simple binary protocolRead N 32bit words starting from START ADDRESS

(Over)Write N 32bit words from the packet to START ADDRESS

Read/Modify/Write N 32bit words from START ADDRESS + packet to START ADDRESS

32 bits 32 bits 32 bits 0x01 N START ADDRESS

32 bits 32 bits 32 bits N*32 bits 0x02 N START ADDRESS N 32 bit words of DATA

32 bits 32 bits 32 bits N*32 bits OPCODE N START ADDRESS N 32bit words of MASK

OPCODE 0x03 0x04 0x05

BITWISE OPERATION AND OR XOR

QuickTime™ and a decompressor

are needed to see this picture.

FPGA client library

Registermap:

• list of register definitions

• type of the register (details follow)

• name of the register (symbolic)

• address of the register in the memory map

one controller one fpga

FPGA client library

fpga:bit( <name>, <bit #>, <address> )

fpga:bitrange( <name>, <startbit>, <nbit>, <address> )

fpga:word( <name>, <address> )

fpga:range( <name>, <# words>, <address> )

Available registerdefinition types for the controller

FPGA client library

fpga:read( <name> )

fpga:write( <name>, <value> )

fpga:or_( <name>, <value> )

fpga:and_( <name>, <value> )

fpga:xor_( <name>, <value> )

Available register commands

Example

-module(firfilter). %% this module is called firfilter, driving one of those-behaviour(personality).-export([registers/0, start_filter/2]). %% this module implements an FPGA personality

%% Called automatically by the FPGA control frameworkregisters() -> {ok, [

fpga:bitrange(num_tap, 5, 3, 16#24),fpga:bit(pps_enable, 28, 16#24),fpga:word(control_status, 16#abcc),

].

%% define a high-level command for this personalitystart_filter(FPGA, NumTap) ->

%% disable the PPSfpga:execute(FPGA, fpga:write(pps_enable, 0)),%% read the control status registercase fpga:execute(FPGA, fpga:read(control_status)) of

1 -> fpga:execute(FPGA, fpga:xor(control_status, 42));_ -> ok

end,%% however, you can easily execute a number of commands in one go:fpga:execute(FPGA, [fpga:write(num_tap, NumTap),

fpga:write(pps_enable, 1)]).

Safety

2> fpga:execute(FPGA, fpga:write(pps_enable, 2)).

OH NOES! error:"Value '2' does not fit in one bit (pps_enable)"

FPGAControl terminating [{{badmatch, {reply, {error, {"Value '2' does not fit in one bit (pps_enable)", [{fpga,err,2, [{fil

Full correlator control system

• besides single fpga control– parallel control of eight fpga’s

• parsing + validating configuration files– using yecc

• decoding + sending over UDP of binary data files– data as well as polynomial coefficients

• capture network data to file• configure and control the hardware• MySQL read/write• two weeks VHDL coding = < 1 day Erlang coding

data reader

data reader

data reader

data writerUniBoardUniBoard

command/control

Full correlator control system

The good

• cross platform– we have Linux, Mac OSX and Solaris

• distributed– typically many machines involved

• non-variability of ‘variables’• conciseness / amount of work per LOC• connects to MySQL

– not easy to connect other languages to mnesia

• hard crash on errors– about the most valuable property: can’t ignore!

• binary pattern matching– every language should be jealous of this

• re-use of development/testing code– it’s so easy to re-use code!

The bad

• our VHDL engineers struggle– language simple, but– concepts like recursion + distribution are ‘hard’

• conciseness / amount of work per LOC– takes effort to read code written by someone who’s better at

it than you

• deploying a developing distributed system– we haven’t found a simple way yet– dynamic remote code loading could be awsome

• it works for individual modules• useless if you have dynamic dependencies ...

– erlang:apply/3

Thank you for your attention!