technology and functionality of the evla correlator (next corr workshop, june 27, 2006)
TRANSCRIPT
Technology and Functionality of the EVLA Correlator
(Next Corr Workshop, June 27, 2006)
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Outline
• EVLA project overview.
• Correlator capabilities.
• Correlator architecture.
• Technology.
• Schedule.
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
EVLA Project Overview
• NRAO:
– Upgrade VLA from the current ~100 MHz BW to 16 GHz (8 GHz/pol’n).
– New receivers: 1-50 GHz contiguous frequency coverage.
– Replace waveguide system with digital fiber system.
– 3-bit (8x2 GHz) and 8-bit (4x1 GHz) sampling.
• New correlator (NRC Canada).
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Correlator Capabilities
• 16 GHz/antenna (8 GHz/poln). 32-antenna system…expandable.
• Can tradeoff BW for #ants and #beams; can proc multiple narrow (<1 GHz) bands; VLBI ready.
• 3/8-bit ADC (at antenna), 4/7-bit re-quantization and correlation…more bits/less BW possible.
• All digital +/-1/32 sample delay tracking; 0.26 sec delay range.
• 3-level phase rotation in correlator chip: sub-sample delay, LO offset removal, earth-rotation phase, bias/birdie/aliasing de-correlation.
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Correlator Capabilities
• 16k channels/baseline wideband; up to 4 Mchannels/baseline narrowband with “recirculation”.
• Up to 144 tunable multi-stage digital filters per antenna…’slot’ restrictions in 1st stage of filter.
• Use filter logic for sub-band multi-beaming and/or narrowband delay.
• ~60 dB spectral dynamic range.
• “WIDAR” technique (hybrid).
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Correlator Capabilities• High-speed pulsar phase binning; 2000 bins/product, min ~15 usec bin width.
• High-speed dumping. 1 Gbps Ethernet/board standard. Up to 10 Gbps/board. 25 Mbytes/sec to archive.
• All digital phased output…1 GHz…expandable to full BW.
• Unlimited sub-arrays.
• Multiple VSI I/O.
• Configurable for virtually any radio telescope configuration…including auto-correlation.
• Uses VLBI standard frequencies (256 MHz…). Can incorporate VLBA into “excess” correlator capacity.
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Correlator Architecture• Station Board:
– 6U Fiber receiver mezzanine card (NRAO).– Cross-bar switch FPGA.– Delay module mezzanine card (+/-0.5 samples delay).– Wideband autocorrelator.– Multi-stage digital filters (1 FPGA per sub-band).
• Cross-bar switch and sub-and multi-beaming delay memory.
• 4 stages…128 MHz…31.25 kHz BW out.
• Power measurements for sub-band stitching.
• Real-time RFI blanking per sub-band.
• CPU-settable scaling factors/re-quantization for max sensitivity/dynamic range.
• State counts, lag-0 power, phase-cal.
• Narrowband sub-sample delay with 16-step FIR interpolation.
– Cross-bar switch and mux to 1 Gbps for data transport to Baseline Board.
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
FIR Interpolator 1
FIR Interpolator 2
FIR Interpolator 3
FIR Interpolator 4
FIR Interpolator 5
FIR Interpolator 16
Fractionaldelay model
Output
Input
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Correlator Architecture• Fanout Board:
– Dual 1-8 fanout of signals (4-wafer ‘stack’, 1 sub-band, all basebands, one antenna) for baseline rack signal distribution.
• Baseline Board:– 8x8 matrix of corr chips fed by 8 ‘X’ and 8 ’Y’ FPGAs
(recirculation, phase generation, 7-bit handling, cross-bar).– Corr chip: 2048 c-lags, in 16 128 c-lag “cells”. Full 4-bit
multiply, no trunc, 10-500 sec int. Cells can be concatenated.– Dedicated LTA FPGA +256 Mbit RAM for each corr chip; high-
speed phase binning and recirculation.– Output 1 Gbps Ethernet via FPGA.– Can set corr chip and board for autocorr mode: 64 independent
2048 channel, 128 MHz autocorrelations.– Can process up to 32 stations (1024 correlations), 128 MHz, 1
poln product, at 64 channels/baseline.
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
RCRC RC
CCRC
Baseline Board“i”
in the “k”th sub-band correlator
wafer 0
Filter Bank1
(18 sub-bandfilters)
Filter Bank0
(18 sub-bandfilters)
DelayModule
DelayModule
Fib
er R
ecei
ver
Mo
dule
Station Board
0
17
k
0
17
k
0
17
0
17
Filter Bank1
(18 sub-bandfilters)
Filter Bank0
(18 sub-bandfilters)
DelayModule
DelayModule
Fib
er R
ece
iver
Mo
dule
Station Board
0
17
k
0
17
k
0
17
0
17
Filter Bank1
(18 sub-bandfilters)
Filter Bank0
(18 sub-bandfilters)
DelayModule
DelayModule
Fib
er R
ece
iver
Mo
dule
Station Board
0
17
k
0
17
k
0
17
0
17
Filter Bank1
(18 sub-bandfilters)
Filter Bank0
(18 sub-bandfilters)
DelayModule
DelayModule
Fib
er R
ece
iver
Mod
ule
Station Board 0of quad “n”
0
17
k
0
17
k
0
17
0
17
4 wafer cablebundle fromstation “n”,
physical output“k”, sub-band “s”
s
s
4 wafer cablebundle fromstation “m”,
physical output“k”, sub-band “s”
wafer 1
wafer 2
wafer 3
BB-0,1
BB-2,3
BB-4,5
BB-6,7
LTA
COTS GigESwitch “i"
GigE FPGA
BaselineBrd
“i",”k”output
k+1
BE CPU BE CPU
Ant
enn
a “
n”,
BB
-0,1
X
Y
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Technology
• Use Xilinx Virtex-IV, and Altera Stratix II/GX/Cyclone FPGAs.
• Standard cell correlator chip, 4 million gate, 0.13 m. PD~3.7 W each @ 256 MHz, 1 V. Use ‘Accel’ point-of-load reg to minimize power, track chip speed with time.
• BGA packages throughout, some gull-wing memories/drivers.
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Technology• 8 different boards in total:
• Station Board, Baseline Board: 12U x 400 mm, 28 layer, 0.0035” trace-’n-space, 0.125” thick. Phasing Board: ~6U x 160 mm(?).
• Fanout Board: 6U x 100 mm, with re-sync Stratix II FPGA.
• Common Backplane/midplane (3U x 25 mm); signal I/O, power, ID…used everywhere…no monolithic backplanes. HM 2.0 mm straight-thru. All press-fit.
• Delay module (~5”x3.5”)mezzanine card: FPGA+DDR SDRAMs
• PC/104+ COTS CPU + PCMC card.
• RPMIB (very simple; diodes, optos) for rack/fan power monitor and control
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Technology
• 1 Gbps station-baseline data transport on high-density Meritec 2 mm hardmetric cable assemblies. Make correlator highly configurable for various correlator configurations (e.g. e-MERLIN).
• Use Altera’s 1 Gbps mux/demux with dynamic phase alignment…built into Stratix chips.
• About 500 rack-to-rack cables, 2500 intra-rack cables (<1 m) in EVLA correlator. All cables plug into 3U x 25mm Common Backplane.
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Baseline Board (12Ux400 mm) PCB-back
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Baseline Board PCB-front
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Thermal vias “pocket” for corr chip heatsink
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Technology
• Use central -48 VDC COTS power supply, DC-DC supplies on each board. Provides battery backup (5 min req).
• 24, 24” racks (2.5’ x 3’ x 7’), each one holding up to 16 large boards in two crates, and Fanout Boards in 6U crate.
• All hot-swappable, including fans.
• On-line detection of communications errors, temps, voltages…remote shutdown/power cycle of any board through central PXI chassis/CPU.
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Thermal mock-up test rack
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Technology
• Software:– Embedded PC/104+ COTS CPU on each large board. Cheap,
fast, easily available, easily replaceable, many vendors.– RT Linux…”Unix-style” device drivers.– System…board…chip-level GUIs for initial testing. Useful
during operations for continued debugging, low-level access.– MCCC—central host computer maps high-level requests from
EVLA M&C to embedded processors, via XML messages.– CPCC—central power control computer.– Backend—array of COTS PCs connected to Baseline Boards
via commercial GigE switches.
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Schedule• Slipped about 1.5 yrs from original 2001 schedule…various reasons.
• 10 proto corr chips arrived June 20/06.
• Baseline Board…probably August/06…some PCB fab problems have delayed delivery.
• Station Board…signed-off design for fab quotation…probably get assembled board late September/06.
• Depending on how well testing goes:
– 1st prototype corrs at VLA/Jodrell Bank…mid ’07.– Full production mid 2008. Full installation early 2009.– Turn off old correlator end of 2009.
Next Gen Corr Workshop: EVLA Correlator—Carlson
June 27, 2006
Summary Response• Type of correlations: all• Output req: spectral + continuum• Special processing req: tunable sub-bands, high spec res, pulsar, 4/7-bit corr, RFI
blanking/robust; RFI excision in backend CPUs eventually; possible RT RFI cancel.• Input BW + digitization: 8x2GHz 3-bit; 4x1GHz 8-bit; dedicated fiber I/F on mezzanine
card; Other BW/digitization supported; VSI I/O.• #spec channels at max BW: 16k, increases by 2X with each decrease in BW by ½.• #baselines correlated: 528 baselines, 163,840 cross-corrs• Integration times: min ~15 usec with binning. 11 msec standard…scalable with backend
and O/P link; unlimited max.• Dynamic range: ~60 dB spectral.• Scale: large, near limits of technology but not “bleading edge”.• Technology: FPGAs, custom hardware, 1 ASIC, COTS CPUs/network, standard form
factor.• Scalability: unlimited but high flexibility not necessarily best approach for ‘large N’. 4X
baselines possible with new ASIC, same architecture.• Flexibility: high: BW/Nant/Nchan/Nbeam/Nbits; RT, nRT VLBI, auto, connected element,
integration/dumping independent of system timing.• Architecture: “WIDAR” (hybrid XF).• RFI mitigation: Nbits, RT blanking, high dynamic range.