applications of enterprise integration patterns to near-real time radar data processing

© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448© 2014 The MITRE Corporation. All Rights Reserved.

Applications of Spring Integration to

Near-Real Time Radar Data ProcessingG. Wampole – MITRE Corporation

Approved for Public Release; Distribution Unlimited. Case Number 14-2448

© 2014 The MITRE Corporation. All rights reserved. Approved for Public Release; Distribution Unlimited. Case Number 14-2448

NOTICE

This technical data was produced for the U.S. Government

under Contract No. W15P7T-13-C-A802, and is subject to

the Rights in Technical Data-Noncommercial Items clause

at DFARS 252.227-7013 (FEB 2012)

© 2014 The MITRE Corporation. All Rights Reserved.


Your Presenter

• Garrett Wampole– Hometown: Allentown, PA

– Alum: Rochester Inst. Tech.

– Been Doing This: ~15 years

• Currently– Prin. Engineer @ MITRE

– Radars, Battle Management, Remote Sensing, Imagery Exploitation, U-2, JSTARS

• Spare Time– iPhone & Windows Phone

(VoiceNotes anyone?)

– Root for New York Mets


What is MITRE?

• A not-for-profit company that operates FFRDCs(Federally Funded Research and Development Center)

• Presently administers six centers

– National Security Engineering Center (DoD)

– Center for Advanced Aviation System Development (FAA)

– Center for Enterprise Modernization (IRS)

– Homeland Security Systems Eng. and Development Institute (DHS)

– Judiciary Engineering and Modernization Center (Federal Judiciary)

– CMS Alliance to Modernize Healthcare (DHHS)

• Approximately 7000 employees (80.4% engineers)

• Two main headquarters: Bedford, MA and McLean, VA

– ~50 domestic sites, 10 international

• http://www.mitre.org

– Fun Fact: First name registered in the .org TLD (July 1985)

http://www.mitre.org/


What is Radar?

Image source: NOAA

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=0W2WgWtpW1122M&tbnid=VrspP5B8CzDlyM:&ved=0CAUQjRw&url=http://www.srh.noaa.gov/jetstream/doppler/how.htm&ei=Hj8zUtvhO7bh4AP_pYCoCA&bvm=bv.52164340,d.dmg&psig=AFQjCNGHrTyewpjLOkyKQVoLbK1Kglh_-g&ust=1379176601624399

http://www.google.com/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&docid=0W2WgWtpW1122M&tbnid=VrspP5B8CzDlyM:&ved=0CAUQjRw&url=http://www.srh.noaa.gov/jetstream/doppler/how.htm&ei=Hj8zUtvhO7bh4AP_pYCoCA&bvm=bv.52164340,d.dmg&psig=AFQjCNGHrTyewpjLOkyKQVoLbK1Kglh_-g&ust=1379176601624399


Simplest Case – Monostatic Radar

Mover


Fusion of Multiple Monostatic Radars


Trade-Offs

• Multistatic

+ Less complicated hardware, smaller sensor footprint

+ Cheaper to Build

– Need multiple sensor locations

– Need coordinated central processing

• Phased Array

+ Everything in-situ, one (bigger) box

+ Potential for greater accuracy

– More complex sensor setup

– Development costs have three commas


It’s a Data Reduction Problem

Mover Reports

Secondary

Processing


Pipeline Processing

> wget -qO - www.cnn.com | grep -i spring | wc -l

1. HTTP GET

2. Grep

3. Count

1. Coordinate Timing

2. Find Movers

3. Track Movers over Time

4. Geolocate and Report


Technical Challenges

• Matching up data from multiple sensors that may arrive at different times

• Coordinating stateful pipeline steps that depend on prior data from each sensor

• Keeping up with near-real-time processing when unforeseen processing hiccups occur

• Supporting multiple types of deployment

– One machine, multiple laptops, servers on different networks

• Support different types of sensor input and product output

– Sensors hardwired or wireless, provide reports as KML or STANAG 4607 to file or network or …

• Provision for a not overly burdensome development environment

• Save intermediate processing products for later analysis


Pipeline Processing Steps

• Service Activator which manages a set of MATLAB objects

maintained in a Matlab Compiler Runtime instance

MCR

service()

onNewData()

blockonProccessingComplete()


Multiple Input/Output Types

• Channel Adapter abstraction is key

– Doesn’t change the rest of the processing pipeline

PbufFileIn/OutboundChannelAdapter

CgmtiFile/UDPOutboundChannelAdapter

PbufTCPIn/OutboundChannelAdapter

PbufUDPIn/OutboundChannelAdapter

PbufUDTIn/OutboundChannelAdapter

ReversePbufFileInboundChannelAdapter

Processing


Development Environment

• Accurately recreating what happened during near-real-

time processing is difficult

– But key for debugging problems and for testing new

algorithms against a representative data set

setRealTimePlaybackEnabled( true )

Recorded

Data

Poller blocked


Flexible Deployments

• One Spring Context / Matlab Runtime combo per JVM

• Use network-based Channel Adapters as IPC

Generate Products Process

Generate Products

Both deployments supported, no recompile

– Pick an XML context file to start

Process


Deployments

Receive Data From Sensors

Coordinate Pre-Processing

Find and Track Movers

Geolocate and Create Products

Do It All At Once


Coordinated Processing

• How to coordinate processing steps that have to occur in proximity, but might be on different processors according to one of many deployments?

– Decorate the service activator with a inter-process cyclic barrier, or “Network Phaser”

Time

arriveAtPhase( 0 )

arriveAtPhase( 0 )

waitForPhase( 1 )

arriveAtPhase( 1 )

blocked

JVM1

JVM2

Phase

Server

Phase

Announcements


Keeping Up

• Fixed-size queue of data inside a message channel

– How to handle out-of-order data over unreliable networks?

Sort according to when sensor produced data


This Stuff Works

http://www.mitre.org/publications/project-stories/bringing-covert-aircraft-out-of-the-shadows


Thoughts

• Enterprise Integration is not the same as B2BIt can work within near-real-time constraints

Spring Integration == ESB == Slow

• SI’s lightweight implementation enables flexibility by not forcing you into too many paradigms

– It also enables performance

• The Spring IoC container eases management and deployment

– Easy deployment is a driver for quick-response ISR systems

• You guys made this possible, thanks!

Not True

applications of enterprise integration patterns to near-real time radar data processing

Documents