lofar offline interference detection

LOFAR offline interference detection

André Offringa1

Ger de Bruyn1,2

Saleem Zaroubi1

Michael Biehl3

1Kapteyn Astronomical Institute, Groningen2ASTRON

3Faculty of Computing Science, University of Groningen

Contents

● About interference detection algorithms● Results for LOFAR● Perspectives for SKA

About detecting interference

● Motivation for automated detection:– Enormous data sizes in new telescopes

such as LOFAR

– No possibilities for manually “browsing” the data for bad baselines, bad antennae, contaminated frequencies, or any other bad dimension of the cube

About RFI detection (2)

● Many methods for RFI removal:– Blanking before correlation at highest time

resolution

– Spatial filtering before correlation

– Flagging after correlation

– RFI modeling and subtraction before and/or after correlation

● After: e.g. fringe fitting● Before: e.g. cyclostationair filtering, satellite

subtraction

● Most of the methods are complementary

Flagging methods

Approach of automatic (baseline) flaggers:● Estimate the value of a sample using its

neighboring time/frequency samples.

● If this deviates substantially from its real value, flag it.

Flagging methods

Approach of automatic (baseline) flaggers:● Estimate the value of a sample using its

neighboring time/frequency samples.– Median value, polynomial fit, weighted

Gaussian filter

● If this deviates substantially from its real value, flag it.

– By thresholding, line detection or combinatorial thresholding

See: Bhat et al. (2005), Winkel et al. (2006), Offringet al. (2010)

Freq

Time (Offringa et al., MNRAS, 2010, in press)

Flagging methods

● Combinatorial thresholding strategy● Idea:

– Sum samples and use different thresholds

A > threshold1? → FLAG AA+B > threshold2? → FLAG A, BA+B+C > threshold3? → FLAG A, B, CA+B+C+D > threshold4? → FLAG A, B, C, DA+E > threshold2? → FLAG A, EA+E+F > threshold3? → FLAG A, E, FA+E+F+G > threshold4? → FLAG A, E, F, GB > threshold1? → FLAG BB+C > threshold2? → FLAG B, C.......

Method comparison

● Compare methods with the help of test sets and ROC curves

● One of the test sets:

(Offringa et al., MNRAS, 2010, in press)

Without flagging After flagging:

LOFAR results

● Data quality is very good

● Flagging somewhat between 1% to 3%

● No show-stopping RFI

LOFAR results

● We see some in sito RFI, but it is very weak

● Too weak to subtract– Done at the GMRT

● Effects after long time integration not yet known

(Athreya, AJ, 2009)

LOFAR results (2)

● Many different flaggers are needed

Different needs per KSP for flagger sensitivity and flagging strategy:

– EoR needs a very stringent approach

– Pulsars need an approach not based on time smoothness

– For transients the same for frequency

– Other option is to quickly scan for RFI to compress the data immediately after

SKA and RFI

SKA and RFI

● The SKA will also be influenced by RFI:– Satellites, aeroplanes

– Lightning

– In situ RFI

– Car engines, (imperfect) transmitters, electrical fences

– Classes of RFI currently not observed

● ...and more detection requirements... :– Failing hardware and software

SKA and RFI

● We are not sure yet what it takes to flag LOFAR data sufficiently

● >>5002 cross-correlations with 10 x LOFAR time resolution and 50 x LOFAR beams is not trivial, if at all possible in 2020

● The order of the data might be a crucial problem:

– Accurate flagging needs large chunks of the cubes in T/F directions

– Calibration needs chunks in the baseline-direction

SKA and RFI

● If necessary, detectors can always trade off speed for accuracy, e.g.:

– Flag on time/frequency integrated data

– Flag on auto correlations

– Flag on groups of antennae

● But is it wise to build more antennae than we can possible process?

Conclusions

● LOFAR can very well deal with its RFI● For SKA, processing will be a major issue● IO and transport might be problematic● The more antennae, the less accurate you

can deal with RFI.● Carefully watch the pathfinders...

● If anyone wants to automatically flag data... => offringa@astro.rug.nl