RFI Mitigation Techniquesat the ATA

Garrett “Karto” Keating

RFI2010 – Groningen, NetherlandsMarch 31st, 2010

A Quick (Re)Introduction

Data volume is destined to become HUGE

20 ft/6.1 m Primary

Currently 42 Dishes

Currently 4 independent IFs and 3 independent FX

64-input correlators

Planned to move to 350 dishes, producing 61,425

unique baselines

Wideband (0.5 to 11 GHz)

Welcome to RAPIDRapid Automated Processing and Rapid Automated Processing and

Imaging of Data.Imaging of Data.

Flag, Calibrate, Image, Repeat.Flag, Calibrate, Image, Repeat.

RAPID and ARTIS

• “Offline” data reduction package for ATA data• Development began in January 2008

• “Online” data reduction package for ATA data• Offshoot of the RAPID project• First light in March 2008• In regular use since Feb 2009

The truth will set you free…

“Effective” troubleshooting

I am not a software/computer engineer

A “real” engineer would program in a “real” language

I “program” using shell scripts (awk/grep/sed) wrapped around MIRIAD

MacGyver would be proud““Like building a radio telescope from a Swiss Like building a radio telescope from a Swiss

army knife and duct tape” army knife and duct tape” –Anonymous ATA Engineer–Anonymous ATA Engineer

Limitations arise from programming choices:Limited

math support Messy codeSlow

processing

The impact of these limitationscan be minimized!

Overall Processing Philosophy

Assume Nothing• Nothing is every truly “static” – LNAs, antennas, RFI, correlators and even objects in the sky are all dynamic

Pragmatic Processing• Processing choices must maximize time utility

“Fallback” Processing• Assume tasks will fail to do everything correctly• Allow for recovery at each processing stage (RFI/Calibration/Imaging)

System Overview

CorrelatorCorrelatorAntennasAntennas CatcherCatcher

MasterObs

MasterObs

ARTISARTIS

FlagFlag CalibrateCalibrate ImageImage

ArchiveArchive

The RFI “cube”Time

Intensity

Long

Medium

Short

NarrowMultichannel

Broadband

“Axes of RFI”• Most RFI cases can be adequately described by its length, bandwidth and intensity • Each case requires a unique strategy and multiple layers of mitigation• Not to be confusedwith Rubik’s cube

Spectral OccupancyPo

wer

Frequency

“Cou

nts”

Frequency

Spectral Occupancy

Spectral Occupancy

x100

Sca

le 0.1

0.2

0.1

Spectral OccupancyData is broken up into temporal windows - the more spectra processed, the better/deeper the RFI processing can go

With enough spectra, extremely weak RFI can be identified and excised. Bad channels and surrounding channels are flagged.

BenefitsNo gains solutions required

• Most spectra can be processing without having solved for the gains solution first

Results can be easily combined• Spectral occupancy results can be combined across antennas and correlator dumps, or “windowed” to limit processing to only select antennas, baselines, etc.

Processing is fast/computationally inexpensive• Achieves an 8:1 observing to processing ratio at the ATA

DrawbacksSNR Limitations

• Results dependant on noise-to-bandpass feature ratio• Normally becomes an issue with bright objects• Can be corrected by normalizing datasets• Can also be corrected with a good gains solution and sky model

DrawbacksIs it really RFI?

• Results can confuse wanted and unwanted RF emissions• Can be solved by creating “good frequency range” masks when flagging• Can also be corrected for during the imaging stage

DrawbacksResults require a large number of samples

• The smaller the array, the higher the required dump rate or the lower the sigma threshold

Multiple iterations sometimes required• More powerful RFI may need to be culled first before reaching weaker RFI

Trouble with “burst” RFI• RFI with an extremely short duration not likely to be picked up as consistently

Threshold RFI Removal

Real

BenefitsFast with calibrated data

• Hours of data can be processed with this method in a matter of minutes

Effective against “burst” RFI• Good at catching RFI left behind by spectral occupancy flagging

No iterative processing generally required• With calibrated data, generally one pass is adequate

DrawbacksSlower with uncalibrated data

• When used in conjunction with gains-solving processes, normally a 1:2 observing to processing ratio is achieved

Iterative processing with uncalibrated data• Process must be repeated after a new gains solution is built

Vulnerable against weaker RFI• RFI below the noise threshold difficult to catch

WRATH RFI RemovalContinuum Image

Channel by channel

WRATH RFI RemovalPre-WRATH Post-WRATH

WRATH flagging stands as the “last guard” against RFI

Dynamic range and fidelity in images can easily double following use of the WRATH mechanism

BenefitsExcellent “last guard”

• Can generally catch the weakest of RFI

Fast processing, not dependent on data size•Processing scales with number of spectral channels, relatively invariant to the size of the dataset

Robust against calibration errors• Gains-solution errors usually affect all channels, thus don’t bias results

Who needs HI?

Who needs HI?Spectral line problems

• Spectral line features can (again) be misrecognized as RFI• Not a lost cause! •Spectral line channels can be masked a priori • Spectral line features should still image with “better” deconvolution, RFI should not

DrawbacksRelatively slow for small datasets

• Again, processing time is primarily proportional to number of channels

Vulnerable against “burst” RFI• RFI with an extremely short duration not likely to be picked up

“Baby and bathwater” dilemma• Highest risk for throwing away good data, since imaging artifacts may only be due to a single datapoint

SummaryCurrent system is robust against most RFI

• Still some trouble with transient/”burst” RFI, but future upgrades to fix that

Processing able to keep up with observing• As processing speed increases, more processing tasks and choices can be made to render better images• Processing is highly scalable

Good astronomer acceptance• Most astronomers using ATA data use the RAPID package, particularly the RFI flagging routines

Source code available atSource code available at svn.hcro.org/mmm/karto/RAPIDBetasvn.hcro.org/mmm/karto/RAPIDBeta

karto@hcro.orgkarto@hcro.orgOffice number: 1-530-335-2364Office number: 1-530-335-2364

Allen Telescope Array/Hat Creek Radio ObservatoryAllen Telescope Array/Hat Creek Radio Observatory