May 8, 2017 Sam Siewert, GPU Tech 2017 –

Drone Net Concept

“Drone Net”

Using Tegra for Multi-Spectral Detection and Tracking in

Shared Air Space

Significance

2

Motivation – Large Numbers of sUAS – Droneii, FAA, Sandia, ASSURE – Counter UAS Challenge – senseFly Catalog of Uses

Problem – Default solution – Part 107 for sUAS and beyond – ADS-B for sUAS insufficient, infeasible – RADAR/LIDAR feasibility

Drone Net hypothesis – Networked, multi-modal

(passive/active), information and sensor data fusion

– EO/IR + acoustic, spectral fusion, machine learning

– Compare to and validate with LIDAR/RADAR, ADS-B

The sUAS Market is Vigorous [Droneii]

Proposed and State of Art

4

Counter UAS [DARPA, ONR, SRC, Mitre] – RADAR/LIDAR, IFF – Neutralization, Jamming, Interception

Security [FAA, DHS, FBI, Sandia] – EO/IR, RADAR/LIDAR, Geo-fencing – Detection Field Testing [FBI at JFK]

Safety and Compliance [DOT FAA] – Part 107 - sUAS Registration, Pilot

Certification, Classification – ADS-B NAS, See/Sense-and-avoid – PrecisionHawk LATAS (Low-Altitude Traffic

and Airspace Safety) – NASA / Verizon UTM [UAS Traffic

Management]

Goals and Objectives

5

EO/IR sUAS Detection Feasibility – Baseline – UAS, GA, Wildlife, Insects, … – DNN, DBN, SVM Machine Vision and

Learning Classification, Identification

Passive, Passive + Active – Performance Evaluation – ROC, PR, F-measure, Confusion Matrices – Data, Image, Information Fusion – Acoustic Camera, LIDAR [Next Steps]

Complimentary Spatial, Temporal, Spectral Resolution – flightradar24.com – Enhanced aggregation – Network of Sensor Fusion Nodes – Long baseline [optical and acoustic

localization] – Campus Drone Net – Geo-Net [Florida, Alaska, Arizona,

Colorado]

Tegra K1, X1,X2

Method – Information Fusion

6

EO/IR Multi-spectral imaging – Visible, NIR to LWIR – Pixel Level and Feature Level Registration – Lower Cost than common bore-sight – Test sUAS with “Flash Pop” Stimulators

Link Drone Net Nodes [wireless, wired] ADS-B aggregation

– Drone Net receivers - Garmin, Appareo, etc. – Test UAS transceivers– e.g. uAvionix ping1090,

ping2020

Acoustic cameras and cues [improve classification] Active LIDAR [RADAR] GP-GPU Real-time Processing

Method – Machine Vision & Learning

7

Machine Vision using SoC Linux – Salient Object Detection

Shape, Behavior and Contrast/Color/Texture in Multiple Bands Performance [ROC, PR, F-measure, confusion matrices]

– Real-Time Detection, Segmentation, Tracking, Classification, Identification

Machine Learning (Traditional, ANN) – Expert systems – Bayesian inference, Dempster-Shafer

reasoning [DBN] – PCA [Principal Component Analysis] – SVM [Support Vector Machines] – Clustering [e.g. K-means] – GPU Accelerated DNN (cuDNN) – Supervised, Unsupervised learning

Leverage Open Source: ROS, OpenCV, PyBrain, PyML, MLpack, cuDNN, Caffe

Conceptual Configuration

Sam Siewert 8

Panchromatic, NIR, RGB

Jetson Tegra X1 With GP-GPU Co-Processing

Cloud Analytics and

Machine Learning

Flash SD Card (local

database) LWIR

Saliency & Behavioral Assessment Thermal Fusion Assessment

Many multi-spectral focal planes …

2D/3D Spatial Assessment

Experimental System Block Diagram 2 Watts at Idle, Plus 1.5 Watts per Camera = 6.5W E.g. Sobel, 30Hz, 20 Mega Pixels/Sec/Watt, 7.3W Peak – SPIE Sensor Tech + Apps

Sam Siewert 9

1) Sync’d Capture

2) Resolution Match

3) Image Registration

4) Detection 5) Classification 6) Identification

Needs Debugging – Literally! Many Insects Detected in Visible to LWIR Opportunity to work on Bird / Aviation Interaction Testing

Sam Siewert 10

2015/16 – ADAC & ERAU Sponsored UAA – ADAC, SmartCam ERAU (Undergraduate Research Team) – Dr. Sam Siewert, PI, Assistant Professor – Demi Matthew Vis – AE/SE Student – Ryan Claus – SE Student – Nicholas DiPinto – SE Student – Arctic Power Team – Power Team Poster

CU Boulder – Embedded Systems Engineering Graduate Program – Ram Krishnamurthy – MS EE – Surjith Singh – MS, ESE – Akshay Singh – ME, ESE – Shivasankar Gunasekaran – ME,ESE – Swaminath Badrinath – ME, ESE

Industry Advising/Collaboration Participants – Randall Myers, Mentor Graphics

Sam Siewert 11

This material is based upon work supported by the U.S. Department of Homeland Security under Grant Award Number, DHS-14-ST-061-COE-001A-02. The views and conclusions contained in this document are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of the U.S. Department of Homeland Security.

2016/17 Team – ERAU Sponsored ERAU – Drone Net – Dr. Sam Siewert, PI, Assistant Prof. – Dr. Iacopo Gentilini, Co-I – Dr. Stephen Bruder, Co-I – Dr. Mehran Andalibi, Co-I – Demi Matthew Vis – AE/SE Student – Ryan Claus – SE Student

CU Boulder – Embedded Systems Graduate – Ram Krishnamurthy – MS EE – Surjith Singh – MS, ESE – Akshay Singh – ME, ESE – Shivasankar Gunasekaran – ME,ESE – Omkar Seelam – ME, ESE

Industry Advising/Collaboration Participants – Randall Myers, Mentor Graphics (PCB, CAD, Systems) – Joe Butler, Intel Corporation (IoT)

Sam Siewert 12

Detection Experiments for Aircraft and UAS

Preliminary Roof-top Field Trials at ERAU Prescott

Sam Siewert

13

AIAA – Drone Net Feasibility Results

Sam Siewert

14

S. Siewert, M. Vis, R. Claus, R. Krishnamurthy, S. B. Singh, A. K. Singh, S. Gunasekaran, “Image and Information Fusion Experiments with a Software-Defined Multi-Spectral Imaging System for Aviation and Marine Sensor Networks”, AIAA SciTech 2017, Grapevine, Texas, January 2017.

Open Reference SDMSI Configuration 2 Basler Pulse Visible Cameras 1 FLIR Vue LWIR Camera with ZnSe Window Jetson TK1, Panda Wireless, USB3 Hub, Power, NEMA Enclosure

Sam Siewert 15

Smart Camera Deployment - Aerial UAV Systems - ERAU ICARUS Group Experimental Aviation and Small Aircraft - ERAU Kite Aerial Photography, Balloon Missions (ERAU, UAA, CU Boulder)

Sam Siewert 16

Sam Siewert – ERAU ICARUS Group

Actual - Roof Mount Experiment Starting point – evolve to aircraft, buoy and UAS later Embry Riddle flight line provides lots of light aircraft traffic Simple UAS testing in Campus (semi-Urban) environment Wildlife – insects, bats, birds, etc. Sam Siewert 17

Information Fusion Concepts Integration and System of Systems Between ADS-B and S-AIS for Vessel / Aircraft / UAS Awareness Smart Cameras Can Monitor and Plan Uplink Opportunity as Well as Wake up and Uplink

Sam Siewert 18

System Fusion For Uplink

Baseline Motion Trigger Detection

Difference Images over Time (adjustable) Threshold - Statistically Significant Pixel Change Filters (Atmospheric, Cloud, Constant Background Motion) – Dispersion of Changes Detection Performance – ROC, PR-Curve, F-measure [TP, FP, FN, TN analysis] Classification/Identification - Confusion Matrix

Sam Siewert 19

https://en.wikipedia.org/wiki/Precision_and_recall

PR best for Image Retrieval E.g. https://images.google.com/ ROC best for Target Detection

Frame by Frame Analysis TP – Determined by Human Review Frame by Frame Alternative is by Physical Experiment Design “Autoit” Program to Analyze Sam Siewert 20

Aircraft Detection Performance - Baseline Video Links – Aircraft, Bugs, FP, TP+FP, [TN], [Full]

Sam Siewert 21

UAS Detection Performance – Baseline Video Link – UAS+Aircraft, Bugs, FP, TP+FP, [TN], [Full]

Sam Siewert 22

Candidate SOD (BinWang14) - Aircraft Modified to Run BinWang14 SOD => MD Baseline Video Links – TP+FP, [TN], [Full]

Sam Siewert 23

Candidate SOD (BinWang14) - UAS Modified to Run BinWang14 SOD => MD Baseline Video Links – TP+FP, [TN], [Full]

Sam Siewert 24

AIAA – Drone Detection Conclusions

Drone Net Likely Requires Custom Detection – SOD Classification Based on Shape, Behavior and Contrast/Color/Texture in Multiple Bands (RGB, NIR, LWIR) Considering Acoustic Cue Fusion Cross Check with ADS-B, RADAR/LIDAR Data Produce Improved flightradar24.com Meta-data Find Ghost UAS and Aircraft [Non-compliant], Log Others

Sam Siewert 25

SPIE – Benchmark Results

Sam Siewert

26

S. Siewert, V. Angoth, R. Krishnamurthy, K. Mani, K. Mock, S. B. Singh, S. Srivistava, C. Wagner, R. Claus, M. Vis, “Software Defined Multi-Spectral Imaging for Arctic Sensor Networks”, SPIE Proceedings, Volume 9840, Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXII, Baltimore, Maryland, April 2016.

FPGA Results - Sobel ALUTs: 10187 Registers: 13,561 Logic utilization: 7,427 / 32,070 ( 23 % )

Sam Siewert 27

Resolution Transform (Watts) (Pixel/sec) per Watt

Saturation FPS Bus transfer rate (MB/sec)

320x240 5.655 2,050,716 151 11.06

640x480 5.700 2,107,284 39.1 11.46

1280x960 5.704 2,143,506 9.95 11.66

2560x1920 5.696 2,157,303 2.50 11.72

Table 2. Sobel Continuous Transform Power Consumption by Cyclone V FPGA

FPGA Results – Pyramidal ALUTs: 24456 Registers: 34,062 Logic utilization: 17,721 / 32,070 ( 55 % ) ( 55 % )

Sam Siewert 28

Table 3. Pyramidal Laplacian Resolution Up-Conversion Continuous Transform Power

Resolution Transform (Watts) (Pixel/sec) per Watt

Saturation FPS Bus transfer rate (MB/sec)

320x240 6.009 889,546 69.6 5.10

640x480 6.013 904,281 17.7 5.19

1280x960 6.038 905,624 4.45 5.21

2560x1920 6.192 889,054 1.12 5.25

Table 4. Pyramidal Gaussian Resolution Down-Conversion Continuous Transform Power Resolution Continuous Transform

Power (Watts) (Pixel/sec) / Watt Saturation

FPS Bus transfer rate (MB/sec)

320x240 5.968 2,445,040 190 13.92 640x480 6.018 2,399,202 47.0 13.77 1280x960 6.023 2,427,813 11.9 13.95 2560x1920 6.109 2,309,154 2.87 13.45

GP-GPU Results - Sobel

Sam Siewert 29

Table 5. Sobel Continuous Transform Power Resolution Continuous

Power at 1Hz (Watts)

Continuous Power at 30Hz (Watts)

(pixels/sec) per Watt @ 1Hz

(pixels/sec) per Watt @ 30Hz

Saturation FPS

320x240 4.241 4.932 18,109 467,153 1624 640x480 4.256 4.984 72,180 1,849,117 840 1280x960 4.266 5.142 288,045 7,169,195 237 2560x1920 4.325 7.326 1,136,462 20,127,764 55

GP-GPU Results - Pyramidal

Sam Siewert 30

Table 6. Pyramidal Up and Down Conversion Continuous Transform Power Resolution Continuous

Power at 1Hz (Watts)

Continuous Power at 20Hz (Watts)

(pixels/sec) / Watt @ 1Hz

(pixels/sec) / Watt @ 20Hz

Saturation FPS

320x240 4.104 4.824 18,713 477,612 1120 640x480 4.116 5.460 74,636 1,687,912 325 1280x960 4.152 6.864 295,954 5,370,629 82 2560x1920 4.224 13.44 1,163,636 10,971,429 20

SPIE On-going Work We Have Completed Hough Lines Continuous Transform Test, Available on GitHub Hough Power Curves Not Yet Produced – In Progress Goal to Identify all Continuous Transform Primitives Used in Infrared + Visible Fusion and 3D Mapping Pixel Level Emphasis, But Also Plan to Review Feature Level – Camera Extrinsic and Intrinsic Transformations – Registration – Resolution and AR Matching – Methods of Pixel Level Fusion in Review [10] [11], [12], [14]

Sam Siewert 31

SPIE Benchmark Conclusions Please Download our Benchmarks – https://github.com/siewertserau/fusion_coproc_benchmarks – MIT License

Test on NVIDIA GP-GPU or FPGA SoCs (Altera, Xilinx) Share Results Back Please Help Us Add Benchmarks Critical to Continuous 3D Mapping and Infrared + Visible Fusion (Suite of Primitives) Open Source Hardware, Firmware, Software for Multispectral Smart Camera Applications

Sam Siewert 32

Research References

Sam Siewert

33

References

Sam Siewert 34

1S. Siewert, V. Angoth, R. Krishnamurthy, K. Mani, K. Mock, S. B. Singh, S. Srivistava, C. Wagner, R. Claus, M. Demi Vis, “Software Defined Multi-Spectral Imaging for Arctic Sensor Networks”, SPIE Algorithms and Technologies for Multispectral, Hyperspectral, and Ultraspectral Imagery XXII, Baltimore, Maryland, April 2016. 2 S. Siewert, J. Shihadeh, Randall Myers, Jay Khandhar, Vitaly Ivanov, “Low Cost, High Performance and Efficiency Computational Photometer Design”, SPIE Sensing Technology and Applications, SPIE Proceedings, Volume 9121, Baltimore, Maryland, May 2014. 3Piella, G. (2003). A general framework for multiresolution image fusion: from pixels to regions. Information fusion, 4(4), 259-280. 4Blum, R. S., & Liu, Z. (Eds.). (2005). Multi-sensor image fusion and its applications. CRC press. 5Liu, Z., Blasch, E., Xue, Z., Zhao, J., Laganiere, R., & Wu, W. (2012). Objective assessment of multiresolution image fusion algorithms for context enhancement in night vision: a comparative study. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 34(1), 94-109. 6Simone, G., Farina, A., Morabito, F. C., Serpico, S. B., & Bruzzone, L. (2002). Image fusion techniques for remote sensing applications. Information fusion, 3(1), 3-15. 7Mitchell, H. B. (2010). Image fusion: theories, techniques and applications. Springer Science & Business Media. 8Szeliski, R. (2010). Computer vision: algorithms and applications. Springer Science & Business Media. 9Sharma, G., Jurie, F., & Schmid, C. (2012, June). Discriminative spatial saliency for image classification. In Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on (pp. 3506-3513). IEEE. 10Toet, A. (2011). Computational versus psychophysical bottom-up image saliency: A comparative evaluation study. Pattern Analysis and Machine Intelligence, IEEE Transactions on, 33(11), 2131-2146.

References

Sam Siewert 35

11Valenti, R., Sebe, N., & Gevers, T. (2009, September). Image saliency by isocentric curvedness and color. In Computer Vision, 2009 IEEE 12th International Conference on (pp. 2185-2192). IEEE. 12Wang, M., Konrad, J., Ishwar, P., Jing, K., & Rowley, H. (2011, June). Image saliency: From intrinsic to extrinsic context. In Computer Vision and Pattern Recognition (CVPR), 2011 IEEE Conference on (pp. 417-424). IEEE. 13Liu, F., & Gleicher, M. (2006, July). Region enhanced scale-invariant saliency detection. In Multimedia and Expo, 2006 IEEE International Conference on (pp. 1477-1480). IEEE. 14Cheng, M. M., Mitra, N. J., Huang, X., & Hu, S. M. (2014). Salientshape: Group saliency in image collections. The Visual Computer, 30(4), 443-453. 15http://global.digitalglobe.com/sites/default/files/DG_WorldView2_DS_PROD.pdf 16http://www.spaceimagingme.com/downloads/sensors/datasheets/DG_WorldView3_DS_2014.pdf 17Richards, Mark A., James A. Scheer, and William A. Holm. Principles of modern radar. SciTech Pub., 2010. 18Brown, Christopher D., and Herbert T. Davis. "Receiver operating characteristics curves and related decision measures: A tutorial." Chemometrics and Intelligent Laboratory Systems 80.1 (2006): 24-38. 19Wang, Bin, and Piotr Dudek. "A fast self-tuning background subtraction algorithm." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 2014. 20Panagiotakis, Costas, et al. "Segmentation and sampling of moving object trajectories based on representativeness." IEEE Transactions on Knowledge and Data Engineering 24.7 (2012): 1328-1343.

References

Sam Siewert 36

21Public SDMSI shared data web site for video sequences captured and used in two experiments presented in this paper - http://mercury.pr.erau.edu/~siewerts/extra/papers/AIAA-SDMSI-data-2017/ 22Perazzi, Federico, et al. "Saliency filters: Contrast based filtering for salient region detection." Computer Vision and Pattern Recognition (CVPR), 2012 IEEE Conference on. IEEE, 2012. 23Achanta, Radhakrishna, et al. "SLIC superpixels compared to state-of-the-art superpixel methods." IEEE transactions on pattern analysis and machine intelligence 34.11 (2012): 2274-2282. 24Hou, Xiaodi, and Liqing Zhang. "Saliency detection: A spectral residual approach." 2007 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2007. 25Global Contrast based Salient Region Detection. Ming-Ming Cheng, Niloy J. Mitra, Xiaolei Huang, Philip H. S. Torr, Shi-Min Hu. IEEE Transactions on Pattern Analysis and Machine Intelligence (IEEE TPAMI), 37(3), 569-582, 2015. 26flightradar24.com, ADS-B, primary/secondary RADAR flight localization and aggregation services. 27Birch, Gabriel Carisle, John Clark Griffin, and Matthew Kelly Erdman. UAS Detection Classification and Neutralization: Market Survey 2015. No. SAND2015-6365. Sandia National Laboratories (SNL-NM), Albuquerque, NM (United States), 2015.

Drone Detection and Neutralization Companies

37

Leading Drone Detection Companies

• https://www.blacksagetech.com/ • https://www.droneshield.com/ • http://www.dedrone.com/en/

List of Drone Detection Companies and Experiments

• Test at JFK by FBI - https://www.faa.gov/news/updates/?newsId=85546

• https://www.x20.org/uav-drone-detection-anti-drone-defense-systems/

• http://industrialcamera.net/security-surveillance-system-products-services/uav-detection/

• https://www.hgh-infrared.com/es/Aplicaciones/Seguridad/UAV-Detection

• http://www.cerbair.com/en/ • http://www.israeldefense.co.il/en/content/rafael-unveils-drone-dome-

drone-detection-and-neutralization-system