ASKAP Signal Processing OverviewDIFX Users and Developers Meeting

CSIRO ASTRONOMY AND SPACE SCIENCE

Dr. Grant Hampson | ADE Team Leader4 September 2012

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

ASKAP Overview• 36x12-metre antennas

• 630 baselines

• 188-port PAF receiver• “Tuneable” over 0.7-1.8GHz

• Digitisation • Provides at least 300MHz processed BW

• 30deg2 FoV• 36-dual pol. Beams (varies with frequency)• Carries through to correlator• Tied array outputs also

2 |

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

ASKAP Differences• BETA (Eng. Test Array)

• Dual heterodyne receiver • ADC at antenna• Dragonfly-2 digitiser (IF sampling)• Redback-2 DSP board inside ATCA chassis

– Xilinx Virtex-6 FPGAs

• ADE (ASKAP Design Enhancements)• RFOF (RF over fibre) from PAF to central site• All digital electronics in central site• Dragonfly-3 digitiser (direct sampling)• Redback-3 DSP board in 1U chassis

– Xilinx Kintex-7 FPGAs

3 |

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

Digital Back-end Blocks

4 |

Correlator

Beamformer

Digital Receiver

1Beamformer

Correlator

FineFilter Bank

PAFCross

Connect

AntennaCross

Connect

192

36

PAF ADC PhaseSwitch

CoarseFilter Bank

ChannelSelect

ArrayCovariance

MatrixCoarseDelay Fringe

Tied Array

Long TermAccumlator

1

36-antennas192-ports2x16-bits300 MHz=66Tbps=6600 x 10G

36-antennas2x36-beams2x16-bits300 MHz=25Tbps=2500 x 10G

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

PAF Electronics – The Domino• The “Domino” combines:

• Low noise amplifier • RF filtering – defines 3 bands• RF over SM fibre transmitter

• Each Domino contains electronics for 2 PAF ports:• 1.8W power per port• 250grams per port

• Backplanes provide power,control and monitoring

• Duplex LC/APC connections to MTP elite, connect RF to central site

5 |

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

ADE Backend – Digital Receiver• Upgrade to Xilinx Series-7 FPGAs

• Kintex-325 FPGAs (840-DSP/RAM)• Contains first stage coarse filter bank

• Direct sampling ADCs• NatSemi 12-bit 1600MSPS ADCs

• Avago optical MiniPODs • Multimode 120Gbps modules• FPGA connect directly at 10Gbps

• 1U chassis packaging • No front or rear to boards• 16 ports per 1U• ~10W per PAF port

6 |

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

ASKAP (BETA) Processed Spectrum

7 |

Sampled Bandwidth = 384MHz

Processed Bandwidth = 304MHz

384MHz 768MHz

Second Nyquist zone

54 x 18.52kHz fine channels (= 1MHz)sent to the correlator

First-stage filterbank output:304 x 1MHz channels

oversampled by 32/27

Second-stage filterbank output:64 x 18.52kHz channels per 1MHz

critically sampled

Stage 1 filterbank

Beamformer

Stage 2 filterbank

-1 -0.5 0 0.5 1 1.5 2-100

-90

-80

-70

-60

-50

-40

-30

-20

-10

0

Frequency (MHz)

Mag

nitu

de (d

B)

Filterbank

Critically sampled filterbank

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

Frequencies?• Can’t have a single 768MHz channel – need to channelise

• Beamformer requires ~1MHz resolution to apply weights• RFI – can remove frequencies that are troublesome – few MHz• Allows looking at frequencies that are well separated• Allows different simultaneous zooms on the same frequencies

• Zooms• Fine delays/fringe stopping require ~100kHz to avoid smearing at band edges• To avoid aliasing near edges of 1MHz channels, and frequency rolloff

– The 1MHz channels are oversampled by 32/27=18.5%

• Tied Array / VLBI• Fine channels are stitched back together to form larger BWs (e.g., 16MHz)• To avoid aliasing and frequency rolloff, the fine channels are planned to also be

oversampled• Don’t require sample rates to be a power of 2

8 |

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

ADE DSP: Beamformer and Correlator• Common hardware platform

• Xilinx Kintex-420 Series-7 FPGAs• ~1680 DSP/RAM per FPGA• 2 DDR3 modules @ 1066MTS• 120Gbps optical transceiver

• Direct 10Gbps from FPGA to optics transceivers simplifies design

• 1U chassis packaging • No front or rear to boards• 6 FPGAs per Redback board

• Use of optical backplanes• Low capital cost – no power• 1.5Tbps optical backplane• Acts like “fixed” network switch• Scalable and modular

9 |

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

Why use FPGAs for Beamforming?• Why use FPGAs?

• 10k multipliers per Redback-3 board (@300MHz = 3Tera ops/sec)• We are using 36x6=216 Redback-3 boards in beamformer • Beamforming, ACM and Fringe• Large processed BW – 300MHz with 36 dual pol beams is significant• Large bitwidths – linear up to beamformer output – preserving dynamic range• Lots of IO – guaranteed IO and lots of it• Power efficient

• Why not use a GPU?• Network switch is a killer: 6600 input ports + 6600 output ports

– size, price, power, cabling, ???• Are GPU’s IO constrained for such an application? • How many GPUs per PC?

10 |

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

Why use FPGAs for Correlator?

• We are using 75 Redback-3 boards in correlator (225TOPS)

• Why not use a GPU?• Network switch: 2500 input ports + 2500 output ports

– size, price, power, cabling, ???• Processing still very regular

• Correlation cells are not that different to beamformers (CMAC with DDR3)• Have to also manage Tied Array outputs, inverse filterbank

• The balance of on-chip resources: logic, RAM, DSP units along with the high IO bandwidth and relatively low power consumption still makes the FPGA a compelling choice in obtaining a cost effective solution for these types of processing applications.

11 |

DIFX Users and Developers Meeting | Grant Hampson | 25 Sep 12

My two bits ...• For single pixel feeds

• No beamformer• Correlator is so much smaller• Pretty simple – GPU?

• However, single pixel feeds can be wider BW• Maybe a factor of 10 larger? How many more dishes?

• Issues with GPUs???• Network switch ports and BW• Watching bit widths carefully• Conversion fixed-floating point• Data rates into GPU cards?• Density of processing?• Software tools

12 |

• Issues with FPGAs???• Communications• Programming • Software tools• Fixed interconnections?• Compilation times• Test benches

• ASKAP ~1 Peta Op• Data rates going up

• 10G to 25G• 10GbE pretty slow

• 100GbE needsto become standard

• Density going up• Better cooling

CSIRO ASTRONOMY AND SPACE SCIENCE

Thank you

We acknowledge the Wajarri Yamatji people as the traditional owners of the Observatory site.

CSIRO Astronomy and Space ScienceDr. Grant HampsonADE Team Leadert +61 2 9372 4647e grant.hampson@csiro.auw www.csiro.au/cass