Managing Data from Avian Radar Systems

Edwin Herricks, PhDSiddhartha Majumdar

Introduction

• 24/7 operation generates significant amounts of data• Ability to review archived avian radar data is

essential• Data must be organized, distributed, and

manipulated into useful data products• At each step of processing information is gained but

data is lost

Radar Data Generation

Analog Data Generated by Radars on airfields

Rutter Card Digitizes Analog Data for the Radar Data Processor

Radar Data Processor extracts plots from energy data and links them to form tracks

Data flow diagram of an Accipiter Radar Technologies, Inc. system

Radar Physics1) Radar transmits electromagnetic energy pulses.2) Energy reflects off all physical objects in it’s path. 3) Reflected energy is received by the radar. 4) Timing and intensity of reflections is recorded by the radar.

•Material•Size•Orientation relative to the radar beam•Environment

The Radar Equation

RCS affected by:

Analog Data• Intensity of reflected

energy is projected on axes of azimuth and range, or B-Scan Data

• Coordinate transformation is necessary to convert to a polar plot, or Scan-converted Data

Digitization• Radars receive a constant stream of

electromagnetic energy.• That stream must be quantified so

that it is in a binary format that can be understood by the Radar Data Processor.

• The rate at which analog values are measured and digitized is called the sampling rate.

• Higher sampling rate gives more detail, and higher file sizes.

• Commercial digitizers typically record at hundreds to thousands of samples per second.

• Some loss of data inherent to the process.

Detection Extraction• Radar Data Processor takes

digitized data and looks for potential targets.

• Simplest form of extraction sets a constant minimum energy level required to constitute a detection.

• More complex forms of extraction can vary that minimum energy level in space and time.

• Gained – Potential targets of interest.Lost – Ability to reprocess with different “sensitivity.”

• Any information contained below the threshold is lost.

Tracking

• Radar Data Processor uses tracking algorithms to compare detections of successive scans in order to predict future scans.

• Successful predictions lead to identifying tracks.

• Additional parameters are recorded– Target history, heading, speed, etc

• “False” tracks can be created from regularly occurring detections resulting from clutter.

• No reprocessing can be done after this step!

Tradeoffs of Processing Data

File size

Information available

Flexibility to reprocess

DigitizationDetection Extraction Tracking

Each stage of processing presents an opportunity to save data, as well as some level of data loss.

Reprocessing Data

• After Digitization, analog resolution is lost.• After Detection Extraction, ability to reprocess

with different sensitivity is lost.• After Tracking, no reprocessing can be done.

Data Management

Data Generation• Remote access to radar• Transfer of data• Storage of data• Generation of data products• Dissemination of data products

Remote Access

• Radars are typically installed relatively inaccessible areas.

• Connectivity must be established to allow remote control and data transfer.

• CEAT ASMP employs Virtual Network Computing (VNC) software to manipulate radar settings and coordinate data transfers.

VNC Access

Close to real-time controlLimited by connection speed

Remote Framebuffer ProtocolPixel by pixel video stream and simple action commands

Cross platform compatibilityLow use of system resources

Data Transfer

• Given the potential size and number of files created, it is best to automate these processes.

• Typically only detection and track data is transferred.– Data available immediately after digitization can be prohibitively large.

• Connectivity must allow enough throughput to keep up with generation of data.– Ideally connection is fast enough to transfer only during off-hours.– CEAT ASMP has found 1-2 megabytes per second is sufficient for most

single-radar installations.

Data Storage

• Amount of storage space necessary is unpredictable.– Amount of data proportional to amount of activity

tracked by radar.

• Ability to increase storage dynamically is important.

• Data must be organized to facilitate future playback and manipulation.

CEAT ASMP Data Storage ServerOriginally 2 Terabytes, now at 6 Terabytes capacity with 2.2 Terabytes full.Located in University managed datacenter– Video surveillance, 24/7 monitoring– Redundant power and temperature control– Rolling 14 days of automatic incremental

backup– Campus firewall

•12 Gigabyte/second fibrenetwork connection

Data Product Generation

• Data products must be designed with the end-user in mind.– What information do they need?– How quickly can it be understood by a non-

expert?– Where and how will it be viewed?

CEAT ASMP Data Products

• Google Earth Enterprise Server– Live data available to client software via internet– Up to 1 hour stored for historical context

• 24/7 Archived Data– 1 Hour historical Summaries– 8 Hour video playback

Google Earth Enterprise“Live” data display

Google Earth EnterpriseHistorical data layer

Historical Summaries

• Tracks accumulate red trails for 1 hour segments

• History cleared at the beginning of each segment

Video Playback

• Variable playback delay

• Data rate of playback dependant on computer processing speed

CEAT ASMP Data Product Generation

• Detection and Track data generated locally at each radar

• Data product generation automated wherever possible

Data Product Distribution

• Security– How sensitive is the information specific to your

tracked targets?– Radars track birds, planes, vehicles, etc, anything

that moves.

• Ease of Access– Can the end-user install client software?

• Clear Organization of Data– By location, then radar, then date.

CEAT ASMP Data Product Distribution

• Secure FTP Server– Requires client software to access– High security

• Web Server– No client software required. Good for end-users

with limited computer access. – Limited security

• Google Earth Enterprise– Requires client software to access

Key Points

• With each step of processing, information is gained but ability to reprocess is lost.

• Data products must be tailored for the end-user.

• Good organization of data and data products is vital to their future utility.

• Network and computer limitations of end-users can limit data distribution options.

Questions?