ngVLA Correlator with 2015ish Technology

Jack Hickish, Dan Werthimer, and CASPER Collaboration

http://casper.berkeley.edu

CASPER Correlator Collaboration

PAPER, HERA (FPGA F, GPU X)

MeerKAT (FPGA F, FGPA X)

AMI

GMRT

Bologna

ISI (Infrared)

SKADS (Oxford)

SMA (CFA, ASIAA)

MIT FFT direct imaging correlator

FASR

Baryon Acoustic Oscillation experiments

LEDA (EOR)

Allen Telescope Array (90 uS imaging)

Berkeley Correlator Team• Dan Werthimer (28 years correlator design)

• Matt Dexter (20 years correlator design)

• David McMahon (10 years correlator design)

• Aaron Parsons (6 years correlator design)

• Rick Raffanti (ADC and RF/analog board designer – 30 years)

• Dave Deboer (project manager)

• Andrew Siemion (Post Doc) – correlators, transient, pulsars, SETI

• Hong Chen (Astr. Undergrad – parameterized FPGA designs)

• Jack Hickish – Cambridge/UCB Postdoc – HERA correlator

• Stevo Bailey – ASIC Design – Hurricane Correlator

3

HERA Array 350 x 15 meter dishes

Simultaneous Digital Backends Piggyback, Commensal, Sky

Surveys

Signal Splitt

er

Pulsar Spectromet

er

Galactic Spectrome

terExtra

Galactic Spectrome

terSETI

Spectrometer

Baseband Data

Recorder

Analog Power Splitters

or

Digital Data Splitter

Commercial off-the-shelfMulticast 10 Gbps (10GE

or InfiniBand) Switch

PFBADCFPGA DSP

Module

FPGA DSPModule

FPGA DSPModule

FPGA DSPModule

FPGA DSPModule

FPGA DSPModule

General-purpose CPUs

PFB

PFB

.

.

.

Correlator

Beamformers/Spectrometers

Pulsar timer

.

.

.

ReconfigurableCompute Cluster

ADC

ADC

PolyphaseFilter Banks

.

.

.

.

.

.

Beowulf Cluster Like General Purpose ArchitechtureDynamic Allocation of Resources, need not be FPGA based

FX Correlator

Antenna 2F engine

Antenna 3F engine

Frequency Band 1X engine

Frequency Band 2X engine

Antenna 1F engine

Frequency Band 3X engine

8

CASPER FXB Correlator/Beamformer (correlator needed to calibrate beamformer)

F Engine 0

10GbE Switch

F Engine 1

F Engine N-1

X Engine 0

X Engine 1

X Engine N-1

. . .

. . .

. . .

Correlators and Beamformers• Globally Asynchronous (like a computer

cluster)

• Data is time stamped with 1 PPS at ADC

• Locally Synchronous, Globally Asynchronous

• Solve problem of correlator/beamformer interconnect problem by using 10/40 Gbe switches (for both interconnect and fast readout)

• No need for high density complex boards10

CASPER FPGA Packetized Correlator

Packetized FX Correlator

Heterogeneous Correlator

ngVLA Correlator Specs ?

• 256 antenna dual pol (512 antpols)

• 50 GHz bandwidth (100 Gsps)

• 2015 technology and pricing

(cost, complexity, and power consumption

will decrease)

ADC’s

26 Gsps 3.5 bit Hittite Dave Hawkins: board and Altera interface Rick Raffanti: FMC board and Xilinx Interface see CASPER Memo 53

55 Gsps 8 bit Fujitsu (die) 32 Gsps 6 bit Micram 80 Gsps 8 bit Keysight 80 Gsps 8 bit Berkeley (freq) 80 Gsps 8 bit Berkeley (time)

Jack Hickish ngVLA Design

• Hittite 26 Gsps ADC (four ADC’s per antpol)

• NetFPGA-SUME boards for both F and X

• Mellanox SX1036 40Gbe switches

(ten 5 GHz correlators)

• $30M

NetFGPA Board Virtex7 XC7V690T

ngVLA correlator 2015 cost $28M ADC’s boards 2048 x $3K/hittite

$6M

F engine boards: 512 x XCKU11P $2M

X engine boards: 1200 x XCVU13 $4M

Switches (50 128 port QSFP) $5M

51,200 DWDM transceivers (80 km) $5M

racks, power, UPS, cables, cooling $1M

labor (depends on Nmodes) $5M

2015 Power Consumption, Size F engines (at the antenna) 60KW

(includes ADCs, FPGA boards, transceivers)

(about 250 watts per telescope)

X engine 170KW

(120KW/FGPA boards, 50KW/switches)

12 racks – 10U per 1GHz correlator

(4U switch, 6U FPGA boards)

21 lags 300kHz clock

discrete transistors

$19,000

1960 – First Radio Astronomy Digital Correlator

Sandy Weinreb

Correlator processing power

DLB

103

102

10

104

105

106

DXB

70 75 9085 80 95 2000 05 10 2015

VLA

GFlops

1

DCB

LOFARSMA

DAS

EVN/WSRT

107

103

106

109

ALMA

SKA

.EVLA

source: Arnold van Ardenne

Ray Escoffier“With correlator performance having gone up by a factor of 922,000 over the last 30 years, its only fair that correlator design engineers' salaries should have gone up by a similar factor!!”